Block runtime¶
A block runs your workload (AI inference or any computation written with the Instance SDK) and wraps it with services that keep it fast, scaled and healthy. This page explains those services at runtime: the load balancer, the auto-scaler, the health checker, the APIs and the metrics. It also shows how to write the policies that drive them.
At a glance
Who this is for: block creators who want to control how their block behaves once deployed.
Related: how blocks are created and placed (cluster and resource allocation) is covered in the Block spec.
What a block gives you¶
- Runs one or more instances of your workload.
- A load balancer (the executor) spreads tasks across instances, using a load-balancing policy you write.
- An auto-scaler sidecar adds and removes instances, using an auto-scaling policy you write.
- A health checker sidecar checks instances and acts on problems, using a health-check policy you write.
- Management APIs let you change configuration and query the policies at runtime.
- Metrics APIs report on the instances and the executor.
- A gRPC proxy exposes the block's inference API to the outside world through the cluster's ingress.
Architecture¶

(Download the architecture diagram)
| Service | Role |
|---|---|
| Executor | The block's gateway. It load-balances incoming tasks across instances with the load-balancer policy and reports executor metrics. For third-party blocks, it always goes through the instances, never directly to the external system. |
| Mapper | Watches for instances being added or removed and sends the updated list to all other block services. |
| Auto-scaler | Periodically reads metrics and runs the auto-scaling policy to scale up or down. For third-party blocks, it scales only the block's instances, not the external system. |
| Health checker | Periodically calls every instance's health endpoint, runs the health-check policy and acts on any problems. For third-party blocks, the instances answer on behalf of the external system. |
| Metrics | Collected by the executor and instances, and written to the cluster's local metrics database and the network's global metrics databases. |
| Instances | Your container. For third-party blocks, each instance converts tasks to the external system's format and back. Instances also run vDAG pre- and post-processing policies, and push outputs to any sidecars. |
Calling a block directly¶
Usually you call blocks through an inference server. You can also call a block's own gRPC proxy directly, at http://<cluster-public-url>/block/<block-id>/rpc (internal port 50051).
For performance, the proxy takes a serialized AIOSPacket as raw bytes:
syntax = "proto3";
message InferenceMessage {
bytes rpc_data = 1;
}
message InferenceRespose {
bool message = 1;
}
// Define the gRPC service
service InferenceProxy {
rpc infer(InferenceMessage) returns (InferenceRespose);
}
The bytes in rpc_data are this message, serialized:
syntax = "proto3";
message FileInfo {
string metadata = 1; // JSON serialized metadata
bytes file_data = 2; // File data
}
message AIOSPacket {
string session_id = 1;
uint64 seq_no = 2; // Sequence number
bytes frame_ptr=3; // frame_ptr data
string data = 4; // Data - json serialized
double ts = 5; // Timestamp
string output_ptr = 6; // JSON serialized string for output pointer
repeated FileInfo files = 7; // Array of file structures
}
| Field | Required | Description |
|---|---|---|
session_id |
Yes | Identifies an inference session, so the block can keep per-session state. |
seq_no |
Yes | Sequence number of the task within the session. (session_id, seq_no) uniquely identifies a task and can be used to keep inputs in order. |
frame_ptr |
No | For very large inputs: a JSON reference to a file stored in FrameDB, with framedb_id (the FrameDB's ID) and key (the file's key). |
data |
Yes | The task input, usually a JSON string. |
ts |
Yes | When the task was created, as a UNIX timestamp (float). |
files |
No | Small files (images, audio, PDFs) as FileInfo entries: optional JSON metadata plus binary file_data. |
output_ptr |
No | Used only for graph inference. Leave it empty for direct block calls. |
import grpc
import time
from protos import inference_pb2, inference_pb2_grpc, payload_pb2
def create_aios_packet():
packet = payload_pb2.AIOSPacket()
packet.session_id = "session-123"
packet.seq_no = 1
packet.data = '{"input": "Hello Block"}'
packet.ts = time.time()
# Optional: add a small file
file_info = payload_pb2.FileInfo()
file_info.metadata = '{"type": "text", "description": "sample file"}'
file_info.file_data = b"Sample file content"
packet.files.append(file_info)
return packet
def run():
# Connect to the gRPC server
channel = grpc.insecure_channel("<server-url>:50051")
stub = inference_pb2_grpc.InferenceProxyStub(channel)
# Serialize the payload and wrap it in an InferenceMessage
serialized_data = create_aios_packet().SerializeToString()
request = inference_pb2.InferenceMessage(rpc_data=serialized_data)
response = stub.infer(request)
print("Inference response:", response.message)
if __name__ == "__main__":
run()
Load balancer (executor)¶
The executor decides which instance handles each task. There is no single best rule: different workloads have different latency, throughput and resource patterns. So you write the rule as a load-balancer policy. The policy receives the task and the current list of instances, and can call a function that returns live block and cluster metrics.

(Download the architecture diagram)
What the policy receives
| Where | Item | Description |
|---|---|---|
settings (in __init__) |
get_metrics |
A function you can call at any time. It returns {"block_metrics": ..., "cluster_metrics": ...}. See Metrics system for the structure. |
block_data |
This block's record. See Block registry. | |
cluster_data |
The cluster's record. See Cluster registry. | |
input_data (in eval) |
packet |
The task, as an AIOSPacket object. |
instances |
The current list of instance IDs, refreshed on every call. |
What it returns: {"instance_id": "<chosen instance>"}
Management commands it should support: health_check (returns the known instances and a status) and get_current_mapping (returns its session-to-instance mapping, if any).
Load-balancer policy template: random choice with session stickiness
import random
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
# get_metrics() is injected by the executor. Call it at any time to get
# current block and cluster metrics for load-balancing decisions.
self.metrics_function = self.settings["get_metrics"]
self.block_data = self.settings["block_data"]
self.cluster_data = self.settings["cluster_data"]
# Optional: remember which instance served each session. This speeds up
# decisions and keeps stateful sessions on the same instance.
self.session_ids_cache = {}
# The current instances are passed on every eval() call, so additions
# and removals are always reflected. Keeping a copy is optional.
self.current_instances = []
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated.
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
Returns:
dict: {"instance_id": <chosen instance>}
"""
# input_data["instances"] = ["instance-0", "instance-1", ...]
if len(input_data["instances"]) != len(self.current_instances):
# Instances were added or removed: reset the sessions cache (for example)
self.session_ids_cache.clear()
self.current_instances = input_data["instances"]
# packet is the AIOSPacket protobuf object
packet = input_data["packet"]
session_id = packet.session_id
if session_id in self.session_ids_cache:
return {"instance_id": self.session_ids_cache[session_id]}
# Current metrics:
# {"block_metrics": ..., "cluster_metrics": ...}
current_metrics = self.metrics_function()
# Do the actual computation here; this example just picks at random
random_instance_id = random.choice(input_data["instances"])
# Remember the choice for this session
self.session_ids_cache[session_id] = random_instance_id
return {"instance_id": random_instance_id}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
Required actions:
1. "health_check": return {"instances": self.current_instances, "status": "healthy"}
2. "get_current_mapping": return {"mapping": self.session_ids_cache}
"""
if action == "health_check":
return {"instances": self.current_instances, "status": "healthy"}
if action == "get_current_mapping":
return {"mapping": self.session_ids_cache}
return {"message": f"Unknown action: {action}"}
Auto-scaler¶
The auto-scaler runs as a sidecar. At an interval set when the block is created, it calls your auto-scaling policy. The policy can read current block and cluster metrics, and decides whether to add instances, remove idle ones or do nothing. Instance counts stay between the block's minInstances and maxInstances.

(Download the architecture diagram)
What the policy receives
| Where | Item | Description |
|---|---|---|
settings (in __init__) |
get_metrics |
A function that returns current block and cluster metrics. |
input_data (in eval) |
block_data |
This block's record. |
cluster_data |
The cluster's record. |
What it returns (one of):
| Decision | Return value |
|---|---|
| Do nothing | {"skip": True} |
| Add instances | {"skip": False, "operation": "upscale", "instances_count": 2} |
| Remove specific instances | {"skip": False, "operation": "downscale", "instances_list": ["instance-0", "instance-3"]} |
Auto-scaling policy template
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
# get_metrics() is injected by the auto-scaler. Call it at any time to get
# current block and cluster metrics for scaling decisions.
self.metrics_function = self.settings["get_metrics"]
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): {"block_data": ..., "cluster_data": ...}
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
"""
block_data = input_data["block_data"]
cluster_data = input_data["cluster_data"]
# Rule logic goes here
# Case 1: no change
return {"skip": True}
# Case 2: add instances
# return {"skip": False, "operation": "upscale", "instances_count": 2}
# Case 3: remove specific instances
# return {"skip": False, "operation": "downscale",
# "instances_list": ["instance-0", "instance-3"]}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
"""
# Implement custom management actions here
return {}
Health checker¶
The health checker is another sidecar. At fixed intervals it:
- Calls
http://{pod_ip}:18001/healthon every instance, with a timeout. This runs your instance'shealth()method. - Marks an instance unhealthy if it returns a non-200 status or times out.
- Calls your health-check policy with the result for every instance.
- Leaves it to the policy to decide what to do. For example, it could notify a developer on Slack, call the failure policy executor, or ask the Cluster Controller Gateway to replace the instance.
What the policy receives
| Where | Item | Description |
|---|---|---|
settings (in __init__) |
block_data, cluster_data |
This block's and its cluster's records. |
input_data (in eval) |
health_check_data |
{instance_id: True/False}, where True means healthy. |
Health-check policy template: escalate after repeated failures
import logging
import requests
def call_failure_policy(api_url: str, failure_policy_id: str, inputs: dict, parameters: dict):
try:
url = f"{api_url}/executeFailurePolicy"
payload = {
"failure_policy_id": failure_policy_id,
"inputs": inputs,
"parameters": parameters
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
return response.json()
except requests.RequestException as e:
logging.error(f"Request failed: {e}")
return {"success": False, "message": str(e)}
except Exception as ex:
logging.error(f"Unexpected error: {ex}")
return {"success": False, "message": str(ex)}
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
# Block and cluster data are passed in settings
self.block_data = self.settings["block_data"]
self.cluster_data = self.settings["cluster_data"]
# Count how many times each instance has been unhealthy
self.counter = {}
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
input_data["health_check_data"] maps instance IDs to health, for example:
{
"instance-00": True,
"instance-01": True,
"instance-02": False
}
"""
health_check_data = input_data["health_check_data"]
for instance, is_healthy in health_check_data.items():
if not is_healthy:
self.counter[instance] = self.counter.get(instance, 0) + 1
# For example: escalate if an instance was unhealthy more than 3 times
if self.counter[instance] > 3:
call_failure_policy(
self.settings["failure_policy_server_api_url"],
self.settings["failure_policy_id"],
inputs={
"block_id": self.block_data['id'],
"cluster_data": self.cluster_data['id'],
"instance_id": instance
},
parameters={}
)
self.counter[instance] = 0
return {}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
"""
# Implement custom management actions here
return {}
Endpoints and ports¶
When a block is created, the cluster's ingress gets a route at /block/<block-id>. From outside the cluster, a block is reachable at:
For example, block block-771a8 on a cluster whose public URL is my-home-cluster.clusters.aiosv1:
| Purpose | Internal port | Ingress suffix | Description |
|---|---|---|---|
| gRPC inference | 50051 | /rpc |
The executor's gRPC proxy for submitting tasks |
| Executor metrics | 18000 | /metrics |
Executor metrics |
| Executor management | 18001 | /executor |
Management commands for the load-balancer policy |
| Auto-scaler management | 10000 | /autoscaler |
Management commands for the auto-scaling policy |
| Health-checker management | 19001 | /health-checker |
Management commands for the health-check policy |
| Redis | 6379 | — | Internal caching and coordination |
| Mapping | 5000 | — | Internal use |
Management APIs¶
Each policy-driven service accepts management commands at its own /mgmt endpoint. The mgmt_action values that are available depend on what your policy's management() method implements.
You can also send management commands through the network's Parser or Gateway, which apply governance checks. See Management commands.
Executor metrics¶
| Metric | Description |
|---|---|
tasks_processed |
Number of tasks processed by the executor |
latency |
Average end-to-end latency of tasks, in seconds |
For instance-level and custom metrics, see Metrics system: block metrics.