Skip to content

Block spec

A block is a running deployment of a component. The block specification says which component to run, how many instances to allow, any settings that override the component's defaults, and the policies that decide where the block runs and how it behaves.

At a glance

Submit with: POST /api/createBlock on the Parser.
Before this: register the component.
Key rule: a block runs in one cluster, though its instances may be on several nodes of that cluster.

What a block is made of

Part Description
Instances Copies of your component's container. Each runs on a node that meets the requirements; several instances may share a node, and even a GPU, if the resource allocation policy allows. Instances emit system and custom metrics, and are scaled by the auto-scaling policy.
Executor The block's gateway and load balancer. It spreads tasks across instances with the load-balancer policy.
Mapper Keeps all block services up to date with the current list of instances.
Auto-scaler Adds or removes instances, using the auto-scaling policy.
Health checker Polls instances and applies the health-check policy.
Metrics Written to the cluster's local and the network's global metrics databases.
Sidecars (optional) Helper pods that receive instance outputs, for example to save results to a database or alert someone.

For how these behave at runtime, see Block runtime.


Structure

{
  "header": {
    "templateUri": "Parser/V1",
    "parameters": {}
  },
  "body": {
    "spec": {
      "values": {
        ...
      }
    }
  }
}

The Parser processes the spec (block_create_IR), filling in any values you leave out from the component.

Fields

Field Type Required Description
blockComponentURI string Yes URI of the registered component to deploy.
blockId string No Unique ID for the block. Generated if not given.
minInstances number Yes Minimum number of instances.
maxInstances number Yes Maximum number of instances.
blockInitData object No Start-up data. Defaults to the component's componentInitData.
initSettings object No Runtime settings. Defaults to the component's componentInitSettings.
parameters object No Runtime or inference parameters. Defaults to the component's componentParameters.
policyRulesSpec array No Policies to add or override. See below.

What the block inherits from the component

If you don't set them in the block spec, these values are copied from the component:

Block field Copied from
blockInitData componentInitData
initSettings componentInitSettings
parameters componentParameters
blockMetadata componentMetadata
inputProtocol componentInputProtocol
outputProtocol componentOutputProtocol
tags component.tags
policies component.policies, then changed by policyRulesSpec

Policy rules

policyRulesSpec adds or overrides policies for this block:

  • The block starts with all of the component's policies.
  • A block policy with the same name as a component policy replaces it.
  • A block policy with a new name is added.
  • If policyRulesSpec is omitted, the component's policies are used unchanged.

Each entry has this shape:

{
  "values": {
    "name": "policyName",
    "policyRuleURI": "policy.rule.uri",
    "parameters": { ... },
    "settings": { ... }
  }
}
Field Type Description
name string Which policy slot this fills, for example autoscaler. Used as the key for overriding.
policyRuleURI string The registered policy to use.
parameters object The policy's parameters.
settings object The policy's settings.

Block policies

A block uses five policies:

Policy (name) What it decides How to write it
clusterAllocator Which cluster the block is placed on: it shortlists candidates, then picks one using feasibility scores. Below
resourceAllocator Which node and GPUs each instance uses: at creation, when scaling, and when an instance must move. Also scores clusters in dry-run mode. Below
loadBalancer Which instance handles each task, using live metrics. Block runtime: load balancer
stabilityChecker Whether the block's instances are healthy, based on monitoring data. Block runtime: health checker
autoscaler When to add or remove instances, from instance and node metrics. Block runtime: auto-scaler

Block allocation flow

When you create a block, the network chooses where it runs in two stages: first a cluster, then nodes and GPUs inside it.

sequenceDiagram
    participant U as You
    participant P as Parser
    participant G as Cluster Controller Gateway
    participant CA as clusterAllocator policy
    participant C as Candidate clusters
    participant CC as Chosen cluster's controller
    U->>P: 1. Submit block spec
    P->>G: 2. Validated spec
    G->>G: 3–4. Run the allocator's filter → list of clusters
    G->>CA: 5. action = "selection" (candidates)
    CA-->>G: 6. Shortlist
    G->>C: 7. resourceAllocator in dry_run mode (each shortlisted cluster)
    C-->>G: Feasibility scores (0–1)
    G->>CA: 8. action = "post_dry_run" (scores)
    CA-->>G: One cluster
    G->>CC: 9. Allocate the block
    CC->>CC: resourceAllocator "allocation" → node and GPUs
  1. You submit the block spec to the Parser.
  2. Once validated, the spec data goes to the Cluster Controller Gateway.
  3. The Gateway reads the block's cluster allocator policy.
  4. It runs the policy's filter rule (in its parameters.filter), which returns a list of clusters.
  5. That list is passed to the cluster allocator policy with "action": "selection".
  6. The policy can filter further, and returns a shortlist of candidate clusters.
  7. Each candidate cluster runs the block's resource allocation policy in dry_run mode. Each returns a feasibility score between 0 and 1; for example, 0.9 means the cluster can run the block with 90% feasibility. Running the feasibility check for maxInstances is recommended.
  8. The scores go to the cluster allocator policy with "action": "post_dry_run", and it picks one cluster.
  9. The block is sent to that cluster's controller, which runs the resource allocation policy for real (allocation) to choose nodes and GPUs.

Actions each policy must handle:

Policy Action Purpose
Cluster allocator selection Rank or further filter the clusters returned by the filter step.
post_dry_run Receive the feasibility scores and pick the final cluster.
Resource allocator dry_run Simulate placement on this cluster and return a feasibility score (0.0–1.0).
allocation Place a new block's instance: return a node ID and GPU IDs.
scale Place an additional instance of an existing block: return a node ID and GPU IDs.
reassignment Place an existing instance that must move; can take its current placement into account.

Writing a cluster allocator policy

selection: input contains filter_result, the clusters returned by the filter. Return {"clusters": [...]}.

post_dry_run: input contains clusters, a list of candidates, each with its dry-run score_data. Return the chosen entry.

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        # Rule ID, configuration settings and external parameters
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Supported actions:
        - "selection": choose candidate clusters after filtering (all or a subset).
        - "post_dry_run": choose one cluster based on feasibility scores.
        """
        # Action: selection
        if input_data['action'] == "selection":
            # Clusters returned by the filter step
            filter_result = input_data['filter_result']
            if len(filter_result) == 0:
                raise Exception("no results found in the filter")

            # Apply any custom filtering or ranking here.
            # This example keeps all filtered clusters.
            return {"clusters": filter_result}

        # Action: post_dry_run
        if input_data['action'] == "post_dry_run":
            # Each entry looks like:
            # {
            #     "score_data": {
            #         "score": 0.9,            # feasibility, 0–1
            #         "node_info": {
            #             "node_id": <node_id>,
            #             "gpus": [....]
            #         }
            #     }
            # }
            selected_clusters = input_data['clusters']

            # Pick the cluster with the highest feasibility score
            highest_index = 0
            current_highest = -1
            for idx, entry in enumerate(selected_clusters):
                score = entry["score_data"]["score"]
                if score > current_highest:
                    current_highest = score
                    highest_index = idx

            return selected_clusters[highest_index]

Note

As with every policy, the class must be named AIOSv1PolicyRule. That is the name the policy loader looks for.

Writing a resource allocation policy

The resource allocation policy receives {"action": ..., "payload": ...} and must handle dry_run, allocation, scale and reassignment. Every action receives the block, the cluster, current cluster metrics and the list of healthy nodes. scale and reassignment also receive the block's metrics, and reassignment receives the instance being moved.

Action Returns
dry_run {"selection_score_data": {"score": 0.9, "node_info": {"node_id": ..., "gpus": [...]}}}
allocation, scale, reassignment {"node_id": ..., "gpus": [...]}

The full payload for each action, and a policy template, are in Cluster Controller: resource allocation policy.


Example block specification

This is the YOLOv5 example from the Quickstart. It places the block on a cluster from vendor dma-bangalore with more than 5 GPUs, a 15-minute CPU load below 10 and more than 9 GB of free GPU memory. It then uses one GPU per instance, on GPUs under 80% utilization with at least 5 GB free. It runs 1 to 5 instances.

{
    "header": {
        "templateUri": "Parser/V1",
        "parameters": {}
    },
    "body": {
        "spec": {
            "values": {
                "blockId": "detector-yolo-88",
                "blockComponentURI": "model.object-detector:2.0.0-stable",
                "minInstances": 1,
                "maxInstances": 5,
                "blockInitData": {
                    "weights_path": "/models/yolov5s.pt",
                    "device": "cuda"
                },
                "initSettings": {
                    "batch_size": 4,
                    "timeout_ms": 1000
                },
                "parameters": {
                    "confidence_threshold": 0.5,
                    "nms_threshold": 0.4
                },
                "policyRulesSpec": [
                    {
                        "values": {
                            "name": "clusterAllocator",
                            "policyRuleURI": "policy.clusterAllocator.yolo-cluster-selector:v1",
                            "parameters": {
                                "filter": {
                                    "clusterMetricsQuery": {
                                        "logicalOperator": "AND",
                                        "conditions": [
                                            {
                                                "variable": "cluster.vcpu.load_15m",
                                                "operator": "<",
                                                "value": "10"
                                            },
                                            {
                                                "variable": "cluster.gpu.totalFreeMem",
                                                "operator": ">",
                                                "value": "9000"
                                            }
                                        ]
                                    },
                                    "clusterQuery": {
                                        "logicalOperator": "AND",
                                        "conditions": [
                                            {
                                                "variable": "gpus.count",
                                                "operator": ">",
                                                "value": "5"
                                            },
                                            {
                                                "variable": "clusterMetadata.vendor",
                                                "operator": "==",
                                                "value": "dma-bangalore"
                                            }
                                        ]
                                    }
                                }
                            },
                            "settings": {
                                "max_candidates": 3
                            }
                        }
                    },
                    {
                        "values": {
                            "name": "resourceAllocator",
                            "policyRuleURI": "policy.resourceAllocator.yolo-gpu-scheduler:v0.0.1-beta",
                            "parameters": {
                                "gpus": 1,
                                "max_gpu_utilization": 0.80,
                                "min_gpu_free_memory": 5000
                            },
                            "settings": {
                                "selection_mode": "greedy"
                            }
                        }
                    },
                    {
                        "values": {
                            "name": "loadBalancer",
                            "policyRuleURI": "policy.loadBalancer.roundrobin:v0.0.1",
                            "parameters": {
                                "cache_sessions": true
                            },
                            "settings": {
                                "session_cache_size": 1000
                            }
                        }
                    },
                    {
                        "values": {
                            "name": "stabilityChecker",
                            "policyRuleURI": "policy.stabilityChecker.unhealthy-retry:v0.0.1",
                            "parameters": {
                                "unhealthy_threshold": 3
                            },
                            "settings": {
                                "check_interval_sec": 15
                            }
                        }
                    },
                    {
                        "values": {
                            "name": "autoscaler",
                            "policyRuleURI": "policy.autoscaler.gpu-based-autoscaler:v0.0.01",
                            "parameters": {
                                "target_cpu_utilization": 0.7
                            },
                            "settings": {
                                "scale_up_cooldown": 60,
                                "scale_down_cooldown": 120
                            }
                        }
                    }
                ]
            }
        }
    }
}

The filter conditions use the query language described in Search and filter.

Create the block

curl -X POST http://<parser-url>/api/createBlock \
  -H "Content-Type: application/json" \
  -d @./block_spec.json

Once it's running, see Block creator guide: manage the running block.