Block creator guide¶
A block creator packages AI models, or any computation, so they can run on the network as blocks: scalable, load-balanced services that others can call.
At a glance
You will: choose a deployment method, package your code as a component, deploy it as a block, and manage it while it runs.
You need: Python, Docker, access to a container registry, and the network's Parser URL.
Choose a deployment method¶
| Your situation | Method | Example |
|---|---|---|
| Your model or code fits on one node (one or several GPUs) | Basic block: your code runs inside the block's instances | YOLOv5 on one GPU, Mistral 7B on 3 GPUs |
| You already run a serving system (for example vLLM) and just want to plug it in | Third-party, externally managed: the block forwards tasks to your system | Block backed by vLLM |
| You want AIOS to install and remove that external system with the block | Third-party, managed by AIOS: an init container sets it up | Init container pattern |
| Your model is too large for one node, or even one cluster | Model splitting: split the model into blocks connected by a vDAG | Split an LLM |
Tip
Not sure? Start with a basic block. You can switch to a third-party or split setup later, without changing how clients call you, as long as your inputs and outputs stay the same.
The workflow¶
Every method follows the same path from code to running service:
flowchart LR
A["1. Write code<br/><small>Instance SDK</small>"] --> B["2. Build and push<br/>the image"]
B --> C["3. Register a<br/>component"]
C --> D["4. Create a block<br/><small>choose policies</small>"]
D --> E["5. Manage it<br/><small>scale, configure, monitor</small>"]
1. Write your code¶
Implement your model server, or any processing logic, with the AIOS Instance SDK. You write a Python class with a few methods (initialize, preprocess, process, health, management), and the SDK handles queues, batching, metrics and communication. See Instance SDK.
For LLMs, the LLM SDK utilities provide ready-made helpers for Hugging Face Transformers and llama.cpp models.
2. Build and push the container image¶
Use aios_instance:v1 as the base image, then push the image to a container registry: your own, an external one, or one of the network's container registries.
3. Register a component¶
A component describes your image so it can be deployed: name, version, metadata, default settings and parameters, input/output protocol, and any sidecars or init containers. Submit it to the Parser:
curl -X POST http://<parser-url>/api/addComponent \
-H "Content-Type: application/json" \
-d @./component.json
The component gets a unique componentURI (name, version and release tag). See Component spec.
4. Create a block¶
A block specification names the component and the policies that control where the block runs and how it behaves:
curl -X POST http://<parser-url>/api/createBlock \
-H "Content-Type: application/json" \
-d @./block.json
| Policy | What it decides |
|---|---|
clusterAllocator |
Which cluster the block is placed on. It filters and ranks candidate clusters, then picks one using feasibility scores. |
resourceAllocator |
Which node and GPUs each instance uses, at creation, when scaling and when instances move. |
loadBalancer |
Which instance handles each incoming task. |
autoscaler |
When to add or remove instances, between minInstances and maxInstances. |
stabilityChecker |
Whether the block's instances are healthy. |
A block inherits its component's policies and settings, and can override any of them. See Block spec for every field and for how to write each policy.
5. Manage the running block¶
Once deployed, your block can be called through any inference server using its block_id. You can also:
The auto-scaler normally handles scaling, but you can add or remove instances yourself through the Cluster Controller Gateway. These actions must pass the scale_block pre-check policies.
# Add 2 instances; omit allocation_data to let the resource allocation policy choose
curl -X POST "http://<cluster-controller-gateway-url>/controller/block-scaling/<cluster-id>" \
-H "Content-Type: application/json" \
-d '{
"operation": "scale",
"block_id": "<block-id>",
"instances_count": 2
}'
# Remove specific instances
curl -X POST "http://<cluster-controller-gateway-url>/controller/block-scaling/<cluster-id>" \
-H "Content-Type: application/json" \
-d '{
"operation": "downscale",
"block_id": "<block-id>",
"instances_list": ["<instance-id>"]
}'
Change settings or talk to the block's policies at runtime, for example to adjust the auto-scaler or query the load balancer:
curl -X POST "http://<cluster-controller-gateway-url>/blocks/mgmt" \
-H "Content-Type: application/json" \
-d '{
"block_id": "<block-id>",
"service": "<service_name>",
"action": "<mgmt_command>",
"payload": {}
}'
See Management commands and Block runtime: APIs.
Blocks report standard metrics automatically, and you can add custom metrics from your code. See Metrics system: block metrics.
Raise custom events from your code, such as "finished processing" or "error", and attach a policy to each. See Block events.
Related¶
- What a block is made of: Block runtime
- Integrating external systems: Third-party systems
- Writing policies: Policies
- Combining blocks into workflows: vDAG creator guide