Skip to content

Model splitting

Some language models are too large for a single machine, or even a single cluster. Model splitting cuts such a model into smaller pieces (splits), deploys each piece as its own block, and connects them with a vDAG. Through the vDAG controller, clients then call the whole chain as if it were one model. This is pipeline parallelism across the network.

At a glance

Who this is for: developers serving models that don't fit on one node.
You'll learn: how splitting works, how to write a splitter with the SDK, and the APIs for running split jobs and tracking reusable layers.
Source: services/applications/llm-model-splits/ in the repository.

A framework, not a turnkey tool

AIOS provides the framework for splitting: running split jobs, tracking results and reusing layers. Because every model architecture and framework splits differently, you write the splitting logic for your model.

How it works

flowchart LR
    A["1. Write splitter code<br/><small>Model splitter SDK</small>"] --> B["2. Run a split job<br/><small>on a split runner</small>"]
    B --> C["3. Model plan<br/><small>block specs + vDAG spec</small>"]
    C --> D["4. Deploy blocks<br/>and the vDAG"]
    B -. "registers each layer" .-> R[("Model layers<br/>registry")]
    D --> V["Call it through the<br/>vDAG controller"]
  1. Write your splitter with the model splitter SDK and package it as a container.
  2. Submit a split job to a split runner on a cluster. The runner runs your container as a Kubernetes Job and tracks it in the Tasks DB.
  3. Your splitter produces a model plan: one block specification per split, plus a vDAG specification that connects them. It also registers each layer in the model layers registry.
  4. Deploy the blocks and the vDAG, then start a vDAG controller.

Pros and cons

Advantages Disadvantages
Splits can run on different nodes or different clusters, enabling distributed inference with pipeline parallelism. Not every model architecture can be split effectively.
Models that don't fit on any single machine or cluster can still be served. Latency can increase, especially when splits are in different clusters and intermediate data crosses the internet.
Each split is an independent block, so it can be scaled on its own, avoiding wasted resources. Moving large intermediate tensors between splits can be slow.
GPUs and compute from many contributors can be pooled for one model. Writing splitting logic can be complex and may take experimentation.
Splits can be reused by several vDAGs. Onboarding a split model end to end takes time and effort.

Model splitting task runners

A cluster can host a split runner, a service that accepts split jobs and runs them on that cluster as Kubernetes Jobs, using your splitter container. Each job gets a task ID, and its progress is recorded in the Tasks DB (task type model_splits).

To deploy a split runner on a cluster, see Join a cluster: model splitter or use the split runners registry API below.

Start a split job

POST /start_model_split_job: creates a model_splits task in the Tasks DB and launches a Kubernetes Job to perform the split.

Field Type Required Description
task_data object Yes What to split and how (see below).
container_image string Yes Your splitter container image.
storage_size string No Size of the job's persistent volume. Default: "1Gi".

task_data fields

Field Type Required Description
model_name string Yes Name of the model to split.
model_download_url string Yes Where to download the model weights.
world_size int Yes Number of splits to produce.
model_split_config object Yes How to split. Interpreted by your splitter code.

Example request

POST /start_model_split_job
Content-Type: application/json

{
  "task_data": {
    "model_name": "gpt2",
    "model_download_url": "https://example.com/models/gpt2.tar.gz",
    "world_size": 4,
    "model_split_config": {
      "split_strategy": "tensor_parallel",
      "tp_degree": 2
    }
  },
  "container_image": "myregistry/model-split-job:latest",
  "storage_size": "2Gi"
}

Responses

{
  "success": true,
  "task_id": "abc123-task-id"
}
{
  "success": false,
  "error": "Missing required fields"
}

Use the task_id to track the job in the Tasks DB.


Split runners registry

The split runners registry lists the split runners available in the network. Creating an entry also deploys the runner onto a cluster, and deleting it removes the runner.

Endpoint Method What it does
/split-runner POST Deploys a split runner to a cluster (using cluster_k8s_config) and registers it.
/split-runner/{runner_id} GET Gets a split runner by ID.
/split-runner/{runner_id} PUT Updates a split runner (MongoDB-style update).
/split-runner/{runner_id} DELETE Removes a split runner from the registry and from Kubernetes.
/split-runners POST Searches split runners with a MongoDB-style filter.

The registry is served by splits-runners-registry-svc on the management cluster (internal port 8001, NodePort 30111).

curl -X POST http://<server-url>:8001/split-runner \
  -H "Content-Type: application/json" \
  -d '{
    "cluster_k8s_config": { /* Kubernetes config dict */ },
    "split_runner_public_host": "192.168.0.106",
    "split_runner_metadata": {"key": "value"},
    "split_runner_tags": ["tag1", "tag2"]
  }'

Response:

{
  "success": true,
  "data": {
    "message": "SplitRunner created",
    "id": "runner-id"
  }
}
curl -X GET http://<server-url>:8001/split-runner/runner-id

Response:

{
  "success": true,
  "data": {
    "split_runner_id": "runner-id",
    "split_runner_public_url": "http://split-runner-url",
    "split_runner_metadata": { "key": "value" },
    "split_runner_public_host": "<server-url>",
    "split_runner_tags": ["tag1", "tag2"]
  }
}
curl -X PUT http://<server-url>:8001/split-runner/runner-id \
  -H "Content-Type: application/json" \
  -d '{
    "$set": {
      "split_runner_metadata.version": "2.0"
    },
    "$addToSet": {
      "split_runner_tags": "auto-scaled"
    }
  }'

Response:

{
  "success": true,
  "data": {
    "message": "SplitRunner updated"
  }
}
curl -X DELETE http://<server-url>:8001/split-runner/runner-id

Response:

{
  "success": true,
  "data": {
    "message": "SplitRunner deleted"
  }
}
curl -X POST http://<server-url>:8001/split-runners \
  -H "Content-Type: application/json" \
  -d '{
    "split_runner_metadata.framework": "transformers",
    "split_runner_tags": { "$in": ["llm"] }
  }'

Response:

{
  "success": true,
  "data": [
    {
      "split_runner_id": "runner-id",
      "split_runner_public_url": "http://host:32286",
      "split_runner_metadata": { "framework": "transformers" },
      "split_runner_public_host": "<server-url>",
      "split_runner_tags": ["llm", "split"]
    }
  ]
}

Model layers registry

The model layers registry records every layer produced by splitting. Each layer is identified by the MD5 hash of its file. That makes reuse possible: blocks carry the layer hash in their metadata, so the system can tell when a block is already serving a given layer, and share it instead of creating a duplicate.

Schema

@dataclass
class ModelLayerObject:
    model_layer_hash: str = ''
    model_asset_id: str = ''
    model_component_registry_uri: str = ''
    model_layer_public_url: str = ''
    model_layer_metadata: List[Dict[str, Any]] = field(default_factory=list)
    model_layer_rank: int = 0
    model_world_size: int = 0
Field Type Required Description
model_layer_hash string Yes Primary key. MD5 hash of the serialized layer.
model_asset_id string Yes ID of the original model asset the layer came from.
model_component_registry_uri string Yes URI of the component spec or metadata used for this layer.
model_layer_public_url string Yes Where the layer file is hosted (for example S3 or HTTP).
model_layer_metadata array of dict No Any extra details, such as shape, precision or configuration.
model_layer_rank integer Yes Position of this layer in the model's pipeline.
model_world_size integer Yes Total number of splits the model was cut into.

APIs

The registry is served by layers-registry-svc on the management cluster (internal port 8002, NodePort 30110).

Endpoint Method What it does
/model-layer POST Registers a new layer. Used by split jobs.
/model-layer/<model_layer_hash> GET Gets a layer by hash.
/model-layer/<model_layer_hash> PUT Updates a layer (MongoDB-style, for example $set).
/model-layer/<model_layer_hash> DELETE Deletes a layer.
/model-layers POST Searches layers with a MongoDB-style filter.
# Register a layer
curl -X POST http://<server-url>:8002/model-layer \
  -H "Content-Type: application/json" \
  -d '{
    "model_layer_hash": "abc123...",
    "model_asset_id": "gpt2-base",
    "model_component_registry_uri": "registry://component/decoder",
    "model_layer_public_url": "https://example.com/layers/abc123",
    "model_layer_metadata": [{"precision": "fp16"}],
    "model_layer_rank": 0,
    "model_world_size": 4
  }'

# Get a layer
curl -X GET http://<server-url>:8002/model-layer/abc123...

# Update a layer
curl -X PUT http://<server-url>:8002/model-layer/abc123... \
  -H "Content-Type: application/json" \
  -d '{
    "$set": {
      "model_layer_public_url": "https://example.com/layers/abc123-v2",
      "model_layer_metadata": [{"precision": "fp32"}]
    }
  }'

# Delete a layer
curl -X DELETE http://<server-url>:8002/model-layer/abc123...

# Search layers
curl -X POST http://<server-url>:8002/model-layers \
  -H "Content-Type: application/json" \
  -d '{
    "model_asset_id": "gpt2-base",
    "model_layer_rank": 0
  }'

Model splitter SDK

The aios_model_splitter library is the framework for writing your splitter. You write one class with three stages (begin, main, finish). The SDK reads the job's settings into a Context object, runs your stages in order, and reports the outcome to the Tasks DB.

The Context object

The SDK builds a Context from environment variables and passes it to your class:

Environment variable Context field Type Description
TASK_ID task_id str ID of the task, used to report status.
TASK_DATA task_data dict The job's task_data (see Start a split job).
TASK_STATUS_UPDATE_URL task_status_update_url str Where to report task status in the Tasks DB.
MODEL_LAYERS_REGISTRY_URL model_layers_registry_url str The model layers registry.
BLOCK_DB_URL block_db_url str The block registry.

Example task_data. Your code decides what model_split_config means:

{
  "model_name": "gpt2",
  "model_download_url": "https://example.com/models/gpt2.tar.gz",
  "world_size": 4,
  "model_split_config": {
    "split_strategy": "tensor_parallel",
    "tp_degree": 2
  }
}

Your splitter class

Stage Signature Typical work Returns
1 begin() Set up: download the model, prepare directories. (True, extra_data) or (False, error)
2 main(extra_data) Split the model according to task_data["model_split_config"]. Register each layer. (True, extra_data) or (False, error)
3 finish(extra_data) Save outputs, upload artifacts, optionally clean up, and return the final result, such as the model plan. (True, result_dict) or (False, error)

extra_data lets you pass information from one stage to the next.

class SampleContainerClass:
    def __init__(self, context: Context):
        self.context = context

    def begin(self):
        # Set up, download the model, etc.
        return True, {"download_path": "/tmp/gpt2.tar.gz"}

    def main(self, extra_data):
        # Split the model according to task_data['model_split_config']
        split_info = {"splits": ["split1", "split2"]}
        return True, {**extra_data, **split_info}

    def finish(self, extra_data):
        # Final logging, upload metadata, or generate the model plan
        return True, {"model_plan_data": ""}

The dictionary returned by finish() is stored as the task's result. The SDK calls GlobalTasksDB.update_task(..., "complete", result_dict). For example:

return True, {
    "status": "done",
    "model_name": "gpt2",
    "split_strategy": "tensor_parallel",
    "tp_degree": 2,
    "output_dir": "/mnt/splits/gpt2/",
    "artifact": "/mnt/splits/gpt2/registry.json"
}

Package your splitter

1. Build the SDK base image (once):

cd services/applications/llm-model-splits/splits-sdk/
docker build -t aios_model_splitter:base .

It uses this Dockerfile:

FROM python:3.10-slim

WORKDIR /app

COPY . /app

RUN pip3 install --upgrade pip && pip3 install -e .

CMD ["python3"]

2. Build your splitter on top of it:

FROM aios_model_splitter:base

COPY my_splitter_code/ /splitter

WORKDIR /splitter

CMD ["python3", "my_splitter_entrypoint.py"]

with an entry point that hands your class to the SDK:

# my_splitter_entrypoint.py
from aios_model_splitter.executor import execute_model_splitter
from my_splitter_code.splitter_container import MyCustomModelSplitter

if __name__ == "__main__":
    execute_model_splitter(MyCustomModelSplitter)

Push the image, then pass it as container_image when you start a split job.


SDK utilities

The SDK includes helpers for common splitter tasks.

Utility Purpose Key methods
S3Manager Upload and download files to and from Amazon S3 upload_file, download_file
FileDownloader Download files over HTTP/HTTPS download
LayerHash Compute the MD5 hash of a layer file get_hash
ModelLayerClient Register, fetch and search layers in the model layers registry create_model_layer, get_model_layer, query_model_layers
Blocks Find running blocks that serve a given layer query

S3Manager

Upload and download files with Amazon S3, using boto3.

S3Manager(aws_access_key_id: str, aws_secret_access_key: str, region_name: str = 'us-east-1')
Method Description Returns
upload_file(file_path: str, bucket_name: str, object_key: str) Uploads a local file to bucket_name/object_key. True on success, False on failure
download_file(bucket_name: str, object_key: str, destination_path: str) Downloads bucket_name/object_key to a local path. True on success, False on failure
s3 = S3Manager("ACCESS_KEY", "SECRET_KEY")
s3.upload_file("/tmp/model.bin", "my-models", "gpt2/model.bin")
s3.download_file("my-models", "gpt2/model.bin", "/tmp/model.bin")

FileDownloader

Downloads a file from a URL, for example model weights in begin(). It creates missing directories, downloads in 4 MB chunks, and logs success or failure.

Method Description Returns
download(url: str, target_path: str) Downloads url to target_path. True on success, False on failure
downloader = FileDownloader()
downloader.download("https://example.com/model.tar.gz", "/tmp/model.tar.gz")

LayerHash

Computes the MD5 hash of a file, which is the key used in the model layers registry. It reads the file in 4 KB chunks, and returns None (logging an error) if the file doesn't exist.

Method Description Returns
get_hash(file_path: str) MD5 hex digest of the file. str, or None on failure
hasher = LayerHash()
md5 = hasher.get_hash("/mnt/models/split_1.pt")

ModelLayerClient

A client for the model layers registry. It reads the registry address from the MODEL_LAYERS_REGISTRY_URL environment variable.

Method Description Returns
create_model_layer(payload: dict) Registers a new layer. The created entry; raises on failure
get_model_layer(layer_hash: str) Gets a layer by its MD5 hash. The entry; raises if not found
query_model_layers(query_filter: dict) Searches with a MongoDB-style filter. A list of entries; raises on failure
client = ModelLayerClient()

client.create_model_layer({
    "layer_hash": "abc123",
    "layer_metadata": {
        "model_name": "gpt2",
        "split_index": 0,
        "size_mb": 45
    }
})

client.query_model_layers({"layer_metadata.model_name": "gpt2"})

Blocks

Finds running blocks that already serve a given layer, identified by component URI and layer hash, so you can reuse them instead of deploying a duplicate.

Method Description Returns
query(component_uri: str, hash: str) Running blocks using this layer. A list of block records; raises on failure
blocks = Blocks()
running_blocks = blocks.query("split-runner.torch-split-runner:v0.0.1-beta", "abc123")

This returns every running block of component split-runner.torch-split-runner:v0.0.1-beta that serves layer abc123.