Block spec¶
A block is a running deployment of a component. The block specification says which component to run, how many instances to allow, any settings that override the component's defaults, and the policies that decide where the block runs and how it behaves.
At a glance
Submit with: POST /api/createBlock on the Parser.
Before this: register the component.
Key rule: a block runs in one cluster, though its instances may be on several nodes of that cluster.
What a block is made of¶
| Part | Description |
|---|---|
| Instances | Copies of your component's container. Each runs on a node that meets the requirements; several instances may share a node, and even a GPU, if the resource allocation policy allows. Instances emit system and custom metrics, and are scaled by the auto-scaling policy. |
| Executor | The block's gateway and load balancer. It spreads tasks across instances with the load-balancer policy. |
| Mapper | Keeps all block services up to date with the current list of instances. |
| Auto-scaler | Adds or removes instances, using the auto-scaling policy. |
| Health checker | Polls instances and applies the health-check policy. |
| Metrics | Written to the cluster's local and the network's global metrics databases. |
| Sidecars (optional) | Helper pods that receive instance outputs, for example to save results to a database or alert someone. |
For how these behave at runtime, see Block runtime.
Structure¶
{
"header": {
"templateUri": "Parser/V1",
"parameters": {}
},
"body": {
"spec": {
"values": {
...
}
}
}
}
The Parser processes the spec (block_create_IR), filling in any values you leave out from the component.
Fields¶
| Field | Type | Required | Description |
|---|---|---|---|
blockComponentURI |
string | Yes | URI of the registered component to deploy. |
blockId |
string | No | Unique ID for the block. Generated if not given. |
minInstances |
number | Yes | Minimum number of instances. |
maxInstances |
number | Yes | Maximum number of instances. |
blockInitData |
object | No | Start-up data. Defaults to the component's componentInitData. |
initSettings |
object | No | Runtime settings. Defaults to the component's componentInitSettings. |
parameters |
object | No | Runtime or inference parameters. Defaults to the component's componentParameters. |
policyRulesSpec |
array | No | Policies to add or override. See below. |
What the block inherits from the component¶
If you don't set them in the block spec, these values are copied from the component:
| Block field | Copied from |
|---|---|
blockInitData |
componentInitData |
initSettings |
componentInitSettings |
parameters |
componentParameters |
blockMetadata |
componentMetadata |
inputProtocol |
componentInputProtocol |
outputProtocol |
componentOutputProtocol |
tags |
component.tags |
policies |
component.policies, then changed by policyRulesSpec |
Policy rules¶
policyRulesSpec adds or overrides policies for this block:
- The block starts with all of the component's policies.
- A block policy with the same
nameas a component policy replaces it. - A block policy with a new
nameis added. - If
policyRulesSpecis omitted, the component's policies are used unchanged.
Each entry has this shape:
{
"values": {
"name": "policyName",
"policyRuleURI": "policy.rule.uri",
"parameters": { ... },
"settings": { ... }
}
}
| Field | Type | Description |
|---|---|---|
name |
string | Which policy slot this fills, for example autoscaler. Used as the key for overriding. |
policyRuleURI |
string | The registered policy to use. |
parameters |
object | The policy's parameters. |
settings |
object | The policy's settings. |
Block policies¶
A block uses five policies:
Policy (name) |
What it decides | How to write it |
|---|---|---|
clusterAllocator |
Which cluster the block is placed on: it shortlists candidates, then picks one using feasibility scores. | Below |
resourceAllocator |
Which node and GPUs each instance uses: at creation, when scaling, and when an instance must move. Also scores clusters in dry-run mode. | Below |
loadBalancer |
Which instance handles each task, using live metrics. | Block runtime: load balancer |
stabilityChecker |
Whether the block's instances are healthy, based on monitoring data. | Block runtime: health checker |
autoscaler |
When to add or remove instances, from instance and node metrics. | Block runtime: auto-scaler |
Block allocation flow¶
When you create a block, the network chooses where it runs in two stages: first a cluster, then nodes and GPUs inside it.
sequenceDiagram
participant U as You
participant P as Parser
participant G as Cluster Controller Gateway
participant CA as clusterAllocator policy
participant C as Candidate clusters
participant CC as Chosen cluster's controller
U->>P: 1. Submit block spec
P->>G: 2. Validated spec
G->>G: 3–4. Run the allocator's filter → list of clusters
G->>CA: 5. action = "selection" (candidates)
CA-->>G: 6. Shortlist
G->>C: 7. resourceAllocator in dry_run mode (each shortlisted cluster)
C-->>G: Feasibility scores (0–1)
G->>CA: 8. action = "post_dry_run" (scores)
CA-->>G: One cluster
G->>CC: 9. Allocate the block
CC->>CC: resourceAllocator "allocation" → node and GPUs
- You submit the block spec to the Parser.
- Once validated, the spec data goes to the Cluster Controller Gateway.
- The Gateway reads the block's cluster allocator policy.
- It runs the policy's filter rule (in its
parameters.filter), which returns a list of clusters. - That list is passed to the cluster allocator policy with
"action": "selection". - The policy can filter further, and returns a shortlist of candidate clusters.
- Each candidate cluster runs the block's resource allocation policy in
dry_runmode. Each returns a feasibility score between 0 and 1; for example, 0.9 means the cluster can run the block with 90% feasibility. Running the feasibility check formaxInstancesis recommended. - The scores go to the cluster allocator policy with
"action": "post_dry_run", and it picks one cluster. - The block is sent to that cluster's controller, which runs the resource allocation policy for real (
allocation) to choose nodes and GPUs.
Actions each policy must handle:
| Policy | Action | Purpose |
|---|---|---|
| Cluster allocator | selection |
Rank or further filter the clusters returned by the filter step. |
post_dry_run |
Receive the feasibility scores and pick the final cluster. | |
| Resource allocator | dry_run |
Simulate placement on this cluster and return a feasibility score (0.0–1.0). |
allocation |
Place a new block's instance: return a node ID and GPU IDs. | |
scale |
Place an additional instance of an existing block: return a node ID and GPU IDs. | |
reassignment |
Place an existing instance that must move; can take its current placement into account. |
Writing a cluster allocator policy¶
selection: input contains filter_result, the clusters returned by the filter. Return {"clusters": [...]}.
post_dry_run: input contains clusters, a list of candidates, each with its dry-run score_data. Return the chosen entry.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
# Rule ID, configuration settings and external parameters
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Supported actions:
- "selection": choose candidate clusters after filtering (all or a subset).
- "post_dry_run": choose one cluster based on feasibility scores.
"""
# Action: selection
if input_data['action'] == "selection":
# Clusters returned by the filter step
filter_result = input_data['filter_result']
if len(filter_result) == 0:
raise Exception("no results found in the filter")
# Apply any custom filtering or ranking here.
# This example keeps all filtered clusters.
return {"clusters": filter_result}
# Action: post_dry_run
if input_data['action'] == "post_dry_run":
# Each entry looks like:
# {
# "score_data": {
# "score": 0.9, # feasibility, 0–1
# "node_info": {
# "node_id": <node_id>,
# "gpus": [....]
# }
# }
# }
selected_clusters = input_data['clusters']
# Pick the cluster with the highest feasibility score
highest_index = 0
current_highest = -1
for idx, entry in enumerate(selected_clusters):
score = entry["score_data"]["score"]
if score > current_highest:
current_highest = score
highest_index = idx
return selected_clusters[highest_index]
Note
As with every policy, the class must be named AIOSv1PolicyRule. That is the name the policy loader looks for.
Writing a resource allocation policy¶
The resource allocation policy receives {"action": ..., "payload": ...} and must handle dry_run, allocation, scale and reassignment. Every action receives the block, the cluster, current cluster metrics and the list of healthy nodes. scale and reassignment also receive the block's metrics, and reassignment receives the instance being moved.
| Action | Returns |
|---|---|
dry_run |
{"selection_score_data": {"score": 0.9, "node_info": {"node_id": ..., "gpus": [...]}}} |
allocation, scale, reassignment |
{"node_id": ..., "gpus": [...]} |
The full payload for each action, and a policy template, are in Cluster Controller: resource allocation policy.
Example block specification¶
This is the YOLOv5 example from the Quickstart. It places the block on a cluster from vendor dma-bangalore with more than 5 GPUs, a 15-minute CPU load below 10 and more than 9 GB of free GPU memory. It then uses one GPU per instance, on GPUs under 80% utilization with at least 5 GB free. It runs 1 to 5 instances.
{
"header": {
"templateUri": "Parser/V1",
"parameters": {}
},
"body": {
"spec": {
"values": {
"blockId": "detector-yolo-88",
"blockComponentURI": "model.object-detector:2.0.0-stable",
"minInstances": 1,
"maxInstances": 5,
"blockInitData": {
"weights_path": "/models/yolov5s.pt",
"device": "cuda"
},
"initSettings": {
"batch_size": 4,
"timeout_ms": 1000
},
"parameters": {
"confidence_threshold": 0.5,
"nms_threshold": 0.4
},
"policyRulesSpec": [
{
"values": {
"name": "clusterAllocator",
"policyRuleURI": "policy.clusterAllocator.yolo-cluster-selector:v1",
"parameters": {
"filter": {
"clusterMetricsQuery": {
"logicalOperator": "AND",
"conditions": [
{
"variable": "cluster.vcpu.load_15m",
"operator": "<",
"value": "10"
},
{
"variable": "cluster.gpu.totalFreeMem",
"operator": ">",
"value": "9000"
}
]
},
"clusterQuery": {
"logicalOperator": "AND",
"conditions": [
{
"variable": "gpus.count",
"operator": ">",
"value": "5"
},
{
"variable": "clusterMetadata.vendor",
"operator": "==",
"value": "dma-bangalore"
}
]
}
}
},
"settings": {
"max_candidates": 3
}
}
},
{
"values": {
"name": "resourceAllocator",
"policyRuleURI": "policy.resourceAllocator.yolo-gpu-scheduler:v0.0.1-beta",
"parameters": {
"gpus": 1,
"max_gpu_utilization": 0.80,
"min_gpu_free_memory": 5000
},
"settings": {
"selection_mode": "greedy"
}
}
},
{
"values": {
"name": "loadBalancer",
"policyRuleURI": "policy.loadBalancer.roundrobin:v0.0.1",
"parameters": {
"cache_sessions": true
},
"settings": {
"session_cache_size": 1000
}
}
},
{
"values": {
"name": "stabilityChecker",
"policyRuleURI": "policy.stabilityChecker.unhealthy-retry:v0.0.1",
"parameters": {
"unhealthy_threshold": 3
},
"settings": {
"check_interval_sec": 15
}
}
},
{
"values": {
"name": "autoscaler",
"policyRuleURI": "policy.autoscaler.gpu-based-autoscaler:v0.0.01",
"parameters": {
"target_cpu_utilization": 0.7
},
"settings": {
"scale_up_cooldown": 60,
"scale_down_cooldown": 120
}
}
}
]
}
}
}
}
The filter conditions use the query language described in Search and filter.
Create the block¶
curl -X POST http://<parser-url>/api/createBlock \
-H "Content-Type: application/json" \
-d @./block_spec.json
Once it's running, see Block creator guide: manage the running block.