Model splitting¶
Some language models are too large for a single machine, or even a single cluster. Model splitting cuts such a model into smaller pieces (splits), deploys each piece as its own block, and connects them with a vDAG. Through the vDAG controller, clients then call the whole chain as if it were one model. This is pipeline parallelism across the network.
At a glance
Who this is for: developers serving models that don't fit on one node.
You'll learn: how splitting works, how to write a splitter with the SDK, and the APIs for running split jobs and tracking reusable layers.
Source: services/applications/llm-model-splits/ in the repository.
A framework, not a turnkey tool
AIOS provides the framework for splitting: running split jobs, tracking results and reusing layers. Because every model architecture and framework splits differently, you write the splitting logic for your model.
How it works¶
flowchart LR
A["1. Write splitter code<br/><small>Model splitter SDK</small>"] --> B["2. Run a split job<br/><small>on a split runner</small>"]
B --> C["3. Model plan<br/><small>block specs + vDAG spec</small>"]
C --> D["4. Deploy blocks<br/>and the vDAG"]
B -. "registers each layer" .-> R[("Model layers<br/>registry")]
D --> V["Call it through the<br/>vDAG controller"]
- Write your splitter with the model splitter SDK and package it as a container.
- Submit a split job to a split runner on a cluster. The runner runs your container as a Kubernetes Job and tracks it in the Tasks DB.
- Your splitter produces a model plan: one block specification per split, plus a vDAG specification that connects them. It also registers each layer in the model layers registry.
- Deploy the blocks and the vDAG, then start a vDAG controller.
Pros and cons¶
| Advantages | Disadvantages |
|---|---|
| Splits can run on different nodes or different clusters, enabling distributed inference with pipeline parallelism. | Not every model architecture can be split effectively. |
| Models that don't fit on any single machine or cluster can still be served. | Latency can increase, especially when splits are in different clusters and intermediate data crosses the internet. |
| Each split is an independent block, so it can be scaled on its own, avoiding wasted resources. | Moving large intermediate tensors between splits can be slow. |
| GPUs and compute from many contributors can be pooled for one model. | Writing splitting logic can be complex and may take experimentation. |
| Splits can be reused by several vDAGs. | Onboarding a split model end to end takes time and effort. |
Model splitting task runners¶
A cluster can host a split runner, a service that accepts split jobs and runs them on that cluster as Kubernetes Jobs, using your splitter container. Each job gets a task ID, and its progress is recorded in the Tasks DB (task type model_splits).
To deploy a split runner on a cluster, see Join a cluster: model splitter or use the split runners registry API below.
Start a split job¶
POST /start_model_split_job: creates a model_splits task in the Tasks DB and launches a Kubernetes Job to perform the split.
| Field | Type | Required | Description |
|---|---|---|---|
task_data |
object | Yes | What to split and how (see below). |
container_image |
string | Yes | Your splitter container image. |
storage_size |
string | No | Size of the job's persistent volume. Default: "1Gi". |
task_data fields
| Field | Type | Required | Description |
|---|---|---|---|
model_name |
string | Yes | Name of the model to split. |
model_download_url |
string | Yes | Where to download the model weights. |
world_size |
int | Yes | Number of splits to produce. |
model_split_config |
object | Yes | How to split. Interpreted by your splitter code. |
Example request
POST /start_model_split_job
Content-Type: application/json
{
"task_data": {
"model_name": "gpt2",
"model_download_url": "https://example.com/models/gpt2.tar.gz",
"world_size": 4,
"model_split_config": {
"split_strategy": "tensor_parallel",
"tp_degree": 2
}
},
"container_image": "myregistry/model-split-job:latest",
"storage_size": "2Gi"
}
Responses
Use the task_id to track the job in the Tasks DB.
Split runners registry¶
The split runners registry lists the split runners available in the network. Creating an entry also deploys the runner onto a cluster, and deleting it removes the runner.
| Endpoint | Method | What it does |
|---|---|---|
/split-runner |
POST |
Deploys a split runner to a cluster (using cluster_k8s_config) and registers it. |
/split-runner/{runner_id} |
GET |
Gets a split runner by ID. |
/split-runner/{runner_id} |
PUT |
Updates a split runner (MongoDB-style update). |
/split-runner/{runner_id} |
DELETE |
Removes a split runner from the registry and from Kubernetes. |
/split-runners |
POST |
Searches split runners with a MongoDB-style filter. |
The registry is served by splits-runners-registry-svc on the management cluster (internal port 8001, NodePort 30111).
curl -X POST http://<server-url>:8001/split-runner \
-H "Content-Type: application/json" \
-d '{
"cluster_k8s_config": { /* Kubernetes config dict */ },
"split_runner_public_host": "192.168.0.106",
"split_runner_metadata": {"key": "value"},
"split_runner_tags": ["tag1", "tag2"]
}'
Response:
Response:
curl -X PUT http://<server-url>:8001/split-runner/runner-id \
-H "Content-Type: application/json" \
-d '{
"$set": {
"split_runner_metadata.version": "2.0"
},
"$addToSet": {
"split_runner_tags": "auto-scaled"
}
}'
Response:
Response:
Model layers registry¶
The model layers registry records every layer produced by splitting. Each layer is identified by the MD5 hash of its file. That makes reuse possible: blocks carry the layer hash in their metadata, so the system can tell when a block is already serving a given layer, and share it instead of creating a duplicate.
Schema¶
@dataclass
class ModelLayerObject:
model_layer_hash: str = ''
model_asset_id: str = ''
model_component_registry_uri: str = ''
model_layer_public_url: str = ''
model_layer_metadata: List[Dict[str, Any]] = field(default_factory=list)
model_layer_rank: int = 0
model_world_size: int = 0
| Field | Type | Required | Description |
|---|---|---|---|
model_layer_hash |
string | Yes | Primary key. MD5 hash of the serialized layer. |
model_asset_id |
string | Yes | ID of the original model asset the layer came from. |
model_component_registry_uri |
string | Yes | URI of the component spec or metadata used for this layer. |
model_layer_public_url |
string | Yes | Where the layer file is hosted (for example S3 or HTTP). |
model_layer_metadata |
array of dict | No | Any extra details, such as shape, precision or configuration. |
model_layer_rank |
integer | Yes | Position of this layer in the model's pipeline. |
model_world_size |
integer | Yes | Total number of splits the model was cut into. |
APIs¶
The registry is served by layers-registry-svc on the management cluster (internal port 8002, NodePort 30110).
| Endpoint | Method | What it does |
|---|---|---|
/model-layer |
POST |
Registers a new layer. Used by split jobs. |
/model-layer/<model_layer_hash> |
GET |
Gets a layer by hash. |
/model-layer/<model_layer_hash> |
PUT |
Updates a layer (MongoDB-style, for example $set). |
/model-layer/<model_layer_hash> |
DELETE |
Deletes a layer. |
/model-layers |
POST |
Searches layers with a MongoDB-style filter. |
# Register a layer
curl -X POST http://<server-url>:8002/model-layer \
-H "Content-Type: application/json" \
-d '{
"model_layer_hash": "abc123...",
"model_asset_id": "gpt2-base",
"model_component_registry_uri": "registry://component/decoder",
"model_layer_public_url": "https://example.com/layers/abc123",
"model_layer_metadata": [{"precision": "fp16"}],
"model_layer_rank": 0,
"model_world_size": 4
}'
# Get a layer
curl -X GET http://<server-url>:8002/model-layer/abc123...
# Update a layer
curl -X PUT http://<server-url>:8002/model-layer/abc123... \
-H "Content-Type: application/json" \
-d '{
"$set": {
"model_layer_public_url": "https://example.com/layers/abc123-v2",
"model_layer_metadata": [{"precision": "fp32"}]
}
}'
# Delete a layer
curl -X DELETE http://<server-url>:8002/model-layer/abc123...
# Search layers
curl -X POST http://<server-url>:8002/model-layers \
-H "Content-Type: application/json" \
-d '{
"model_asset_id": "gpt2-base",
"model_layer_rank": 0
}'
Model splitter SDK¶
The aios_model_splitter library is the framework for writing your splitter. You write one class with three stages (begin, main, finish). The SDK reads the job's settings into a Context object, runs your stages in order, and reports the outcome to the Tasks DB.
The Context object¶
The SDK builds a Context from environment variables and passes it to your class:
| Environment variable | Context field |
Type | Description |
|---|---|---|---|
TASK_ID |
task_id |
str |
ID of the task, used to report status. |
TASK_DATA |
task_data |
dict |
The job's task_data (see Start a split job). |
TASK_STATUS_UPDATE_URL |
task_status_update_url |
str |
Where to report task status in the Tasks DB. |
MODEL_LAYERS_REGISTRY_URL |
model_layers_registry_url |
str |
The model layers registry. |
BLOCK_DB_URL |
block_db_url |
str |
The block registry. |
Example task_data. Your code decides what model_split_config means:
{
"model_name": "gpt2",
"model_download_url": "https://example.com/models/gpt2.tar.gz",
"world_size": 4,
"model_split_config": {
"split_strategy": "tensor_parallel",
"tp_degree": 2
}
}
Your splitter class¶
| Stage | Signature | Typical work | Returns |
|---|---|---|---|
| 1 | begin() |
Set up: download the model, prepare directories. | (True, extra_data) or (False, error) |
| 2 | main(extra_data) |
Split the model according to task_data["model_split_config"]. Register each layer. |
(True, extra_data) or (False, error) |
| 3 | finish(extra_data) |
Save outputs, upload artifacts, optionally clean up, and return the final result, such as the model plan. | (True, result_dict) or (False, error) |
extra_data lets you pass information from one stage to the next.
class SampleContainerClass:
def __init__(self, context: Context):
self.context = context
def begin(self):
# Set up, download the model, etc.
return True, {"download_path": "/tmp/gpt2.tar.gz"}
def main(self, extra_data):
# Split the model according to task_data['model_split_config']
split_info = {"splits": ["split1", "split2"]}
return True, {**extra_data, **split_info}
def finish(self, extra_data):
# Final logging, upload metadata, or generate the model plan
return True, {"model_plan_data": ""}
The dictionary returned by finish() is stored as the task's result. The SDK calls GlobalTasksDB.update_task(..., "complete", result_dict). For example:
return True, {
"status": "done",
"model_name": "gpt2",
"split_strategy": "tensor_parallel",
"tp_degree": 2,
"output_dir": "/mnt/splits/gpt2/",
"artifact": "/mnt/splits/gpt2/registry.json"
}
Package your splitter¶
1. Build the SDK base image (once):
It uses this Dockerfile:
FROM python:3.10-slim
WORKDIR /app
COPY . /app
RUN pip3 install --upgrade pip && pip3 install -e .
CMD ["python3"]
2. Build your splitter on top of it:
FROM aios_model_splitter:base
COPY my_splitter_code/ /splitter
WORKDIR /splitter
CMD ["python3", "my_splitter_entrypoint.py"]
with an entry point that hands your class to the SDK:
# my_splitter_entrypoint.py
from aios_model_splitter.executor import execute_model_splitter
from my_splitter_code.splitter_container import MyCustomModelSplitter
if __name__ == "__main__":
execute_model_splitter(MyCustomModelSplitter)
Push the image, then pass it as container_image when you start a split job.
SDK utilities¶
The SDK includes helpers for common splitter tasks.
| Utility | Purpose | Key methods |
|---|---|---|
S3Manager |
Upload and download files to and from Amazon S3 | upload_file, download_file |
FileDownloader |
Download files over HTTP/HTTPS | download |
LayerHash |
Compute the MD5 hash of a layer file | get_hash |
ModelLayerClient |
Register, fetch and search layers in the model layers registry | create_model_layer, get_model_layer, query_model_layers |
Blocks |
Find running blocks that serve a given layer | query |
S3Manager¶
Upload and download files with Amazon S3, using boto3.
| Method | Description | Returns |
|---|---|---|
upload_file(file_path: str, bucket_name: str, object_key: str) |
Uploads a local file to bucket_name/object_key. |
True on success, False on failure |
download_file(bucket_name: str, object_key: str, destination_path: str) |
Downloads bucket_name/object_key to a local path. |
True on success, False on failure |
s3 = S3Manager("ACCESS_KEY", "SECRET_KEY")
s3.upload_file("/tmp/model.bin", "my-models", "gpt2/model.bin")
s3.download_file("my-models", "gpt2/model.bin", "/tmp/model.bin")
FileDownloader¶
Downloads a file from a URL, for example model weights in begin(). It creates missing directories, downloads in 4 MB chunks, and logs success or failure.
| Method | Description | Returns |
|---|---|---|
download(url: str, target_path: str) |
Downloads url to target_path. |
True on success, False on failure |
downloader = FileDownloader()
downloader.download("https://example.com/model.tar.gz", "/tmp/model.tar.gz")
LayerHash¶
Computes the MD5 hash of a file, which is the key used in the model layers registry. It reads the file in 4 KB chunks, and returns None (logging an error) if the file doesn't exist.
| Method | Description | Returns |
|---|---|---|
get_hash(file_path: str) |
MD5 hex digest of the file. | str, or None on failure |
ModelLayerClient¶
A client for the model layers registry. It reads the registry address from the MODEL_LAYERS_REGISTRY_URL environment variable.
| Method | Description | Returns |
|---|---|---|
create_model_layer(payload: dict) |
Registers a new layer. | The created entry; raises on failure |
get_model_layer(layer_hash: str) |
Gets a layer by its MD5 hash. | The entry; raises if not found |
query_model_layers(query_filter: dict) |
Searches with a MongoDB-style filter. | A list of entries; raises on failure |
client = ModelLayerClient()
client.create_model_layer({
"layer_hash": "abc123",
"layer_metadata": {
"model_name": "gpt2",
"split_index": 0,
"size_mb": 45
}
})
client.query_model_layers({"layer_metadata.model_name": "gpt2"})
Blocks¶
Finds running blocks that already serve a given layer, identified by component URI and layer hash, so you can reuse them instead of deploying a duplicate.
| Method | Description | Returns |
|---|---|---|
query(component_uri: str, hash: str) |
Running blocks using this layer. | A list of block records; raises on failure |
blocks = Blocks()
running_blocks = blocks.query("split-runner.torch-split-runner:v0.0.1-beta", "abc123")
This returns every running block of component split-runner.torch-split-runner:v0.0.1-beta that serves layer abc123.