Skip to content

Block creator guide

A block creator packages AI models, or any computation, so they can run on the network as blocks: scalable, load-balanced services that others can call.

At a glance

You will: choose a deployment method, package your code as a component, deploy it as a block, and manage it while it runs.
You need: Python, Docker, access to a container registry, and the network's Parser URL.


Choose a deployment method

Your situation Method Example
Your model or code fits on one node (one or several GPUs) Basic block: your code runs inside the block's instances YOLOv5 on one GPU, Mistral 7B on 3 GPUs
You already run a serving system (for example vLLM) and just want to plug it in Third-party, externally managed: the block forwards tasks to your system Block backed by vLLM
You want AIOS to install and remove that external system with the block Third-party, managed by AIOS: an init container sets it up Init container pattern
Your model is too large for one node, or even one cluster Model splitting: split the model into blocks connected by a vDAG Split an LLM

Tip

Not sure? Start with a basic block. You can switch to a third-party or split setup later, without changing how clients call you, as long as your inputs and outputs stay the same.


The workflow

Every method follows the same path from code to running service:

flowchart LR
    A["1. Write code<br/><small>Instance SDK</small>"] --> B["2. Build and push<br/>the image"]
    B --> C["3. Register a<br/>component"]
    C --> D["4. Create a block<br/><small>choose policies</small>"]
    D --> E["5. Manage it<br/><small>scale, configure, monitor</small>"]

1. Write your code

Implement your model server, or any processing logic, with the AIOS Instance SDK. You write a Python class with a few methods (initialize, preprocess, process, health, management), and the SDK handles queues, batching, metrics and communication. See Instance SDK.

For LLMs, the LLM SDK utilities provide ready-made helpers for Hugging Face Transformers and llama.cpp models.

2. Build and push the container image

Use aios_instance:v1 as the base image, then push the image to a container registry: your own, an external one, or one of the network's container registries.

3. Register a component

A component describes your image so it can be deployed: name, version, metadata, default settings and parameters, input/output protocol, and any sidecars or init containers. Submit it to the Parser:

curl -X POST http://<parser-url>/api/addComponent \
  -H "Content-Type: application/json" \
  -d @./component.json

The component gets a unique componentURI (name, version and release tag). See Component spec.

4. Create a block

A block specification names the component and the policies that control where the block runs and how it behaves:

curl -X POST http://<parser-url>/api/createBlock \
  -H "Content-Type: application/json" \
  -d @./block.json
Policy What it decides
clusterAllocator Which cluster the block is placed on. It filters and ranks candidate clusters, then picks one using feasibility scores.
resourceAllocator Which node and GPUs each instance uses, at creation, when scaling and when instances move.
loadBalancer Which instance handles each incoming task.
autoscaler When to add or remove instances, between minInstances and maxInstances.
stabilityChecker Whether the block's instances are healthy.

A block inherits its component's policies and settings, and can override any of them. See Block spec for every field and for how to write each policy.

5. Manage the running block

Once deployed, your block can be called through any inference server using its block_id. You can also:

The auto-scaler normally handles scaling, but you can add or remove instances yourself through the Cluster Controller Gateway. These actions must pass the scale_block pre-check policies.

# Add 2 instances; omit allocation_data to let the resource allocation policy choose
curl -X POST "http://<cluster-controller-gateway-url>/controller/block-scaling/<cluster-id>" \
     -H "Content-Type: application/json" \
     -d '{
           "operation": "scale",
           "block_id": "<block-id>",
           "instances_count": 2
         }'

# Remove specific instances
curl -X POST "http://<cluster-controller-gateway-url>/controller/block-scaling/<cluster-id>" \
     -H "Content-Type: application/json" \
     -d '{
           "operation": "downscale",
           "block_id": "<block-id>",
           "instances_list": ["<instance-id>"]
         }'

See Cluster Controller Gateway: scaling.

Change settings or talk to the block's policies at runtime, for example to adjust the auto-scaler or query the load balancer:

curl -X POST "http://<cluster-controller-gateway-url>/blocks/mgmt" \
  -H "Content-Type: application/json" \
  -d '{
    "block_id": "<block-id>",
    "service": "<service_name>",
    "action": "<mgmt_command>",
    "payload": {}
  }'

See Management commands and Block runtime: APIs.

Blocks report standard metrics automatically, and you can add custom metrics from your code. See Metrics system: block metrics.

Raise custom events from your code, such as "finished processing" or "error", and attach a policy to each. See Block events.