Skip to content

Block runtime

A block runs your workload (AI inference or any computation written with the Instance SDK) and wraps it with services that keep it fast, scaled and healthy. This page explains those services at runtime: the load balancer, the auto-scaler, the health checker, the APIs and the metrics. It also shows how to write the policies that drive them.

At a glance

Who this is for: block creators who want to control how their block behaves once deployed.
Related: how blocks are created and placed (cluster and resource allocation) is covered in the Block spec.

What a block gives you

  1. Runs one or more instances of your workload.
  2. A load balancer (the executor) spreads tasks across instances, using a load-balancing policy you write.
  3. An auto-scaler sidecar adds and removes instances, using an auto-scaling policy you write.
  4. A health checker sidecar checks instances and acts on problems, using a health-check policy you write.
  5. Management APIs let you change configuration and query the policies at runtime.
  6. Metrics APIs report on the instances and the executor.
  7. A gRPC proxy exposes the block's inference API to the outside world through the cluster's ingress.

Architecture

Block architecture

(Download the architecture diagram)

Service Role
Executor The block's gateway. It load-balances incoming tasks across instances with the load-balancer policy and reports executor metrics. For third-party blocks, it always goes through the instances, never directly to the external system.
Mapper Watches for instances being added or removed and sends the updated list to all other block services.
Auto-scaler Periodically reads metrics and runs the auto-scaling policy to scale up or down. For third-party blocks, it scales only the block's instances, not the external system.
Health checker Periodically calls every instance's health endpoint, runs the health-check policy and acts on any problems. For third-party blocks, the instances answer on behalf of the external system.
Metrics Collected by the executor and instances, and written to the cluster's local metrics database and the network's global metrics databases.
Instances Your container. For third-party blocks, each instance converts tasks to the external system's format and back. Instances also run vDAG pre- and post-processing policies, and push outputs to any sidecars.

Calling a block directly

Usually you call blocks through an inference server. You can also call a block's own gRPC proxy directly, at http://<cluster-public-url>/block/<block-id>/rpc (internal port 50051).

For performance, the proxy takes a serialized AIOSPacket as raw bytes:

syntax = "proto3";

message InferenceMessage {
   bytes rpc_data = 1;
}

message InferenceRespose {
     bool message = 1;
}

// Define the gRPC service
service InferenceProxy {
    rpc infer(InferenceMessage) returns (InferenceRespose);
}

The bytes in rpc_data are this message, serialized:

syntax = "proto3";

message FileInfo {
    string metadata = 1; // JSON serialized metadata
    bytes file_data = 2; // File data
}

message AIOSPacket {
    string session_id = 1;
    uint64 seq_no = 2;           // Sequence number
    bytes frame_ptr=3;           // frame_ptr data
    string data = 4;             // Data - json serialized
    double ts = 5;               // Timestamp
    string output_ptr = 6;       // JSON serialized string for output pointer
    repeated FileInfo files = 7; // Array of file structures
}
Field Required Description
session_id Yes Identifies an inference session, so the block can keep per-session state.
seq_no Yes Sequence number of the task within the session. (session_id, seq_no) uniquely identifies a task and can be used to keep inputs in order.
frame_ptr No For very large inputs: a JSON reference to a file stored in FrameDB, with framedb_id (the FrameDB's ID) and key (the file's key).
data Yes The task input, usually a JSON string.
ts Yes When the task was created, as a UNIX timestamp (float).
files No Small files (images, audio, PDFs) as FileInfo entries: optional JSON metadata plus binary file_data.
output_ptr No Used only for graph inference. Leave it empty for direct block calls.
import grpc
import time
from protos import inference_pb2, inference_pb2_grpc, payload_pb2

def create_aios_packet():
    packet = payload_pb2.AIOSPacket()
    packet.session_id = "session-123"
    packet.seq_no = 1
    packet.data = '{"input": "Hello Block"}'
    packet.ts = time.time()

    # Optional: add a small file
    file_info = payload_pb2.FileInfo()
    file_info.metadata = '{"type": "text", "description": "sample file"}'
    file_info.file_data = b"Sample file content"
    packet.files.append(file_info)

    return packet

def run():
    # Connect to the gRPC server
    channel = grpc.insecure_channel("<server-url>:50051")
    stub = inference_pb2_grpc.InferenceProxyStub(channel)

    # Serialize the payload and wrap it in an InferenceMessage
    serialized_data = create_aios_packet().SerializeToString()
    request = inference_pb2.InferenceMessage(rpc_data=serialized_data)

    response = stub.infer(request)
    print("Inference response:", response.message)

if __name__ == "__main__":
    run()

Load balancer (executor)

The executor decides which instance handles each task. There is no single best rule: different workloads have different latency, throughput and resource patterns. So you write the rule as a load-balancer policy. The policy receives the task and the current list of instances, and can call a function that returns live block and cluster metrics.

Load balancer architecture

(Download the architecture diagram)

What the policy receives

Where Item Description
settings (in __init__) get_metrics A function you can call at any time. It returns {"block_metrics": ..., "cluster_metrics": ...}. See Metrics system for the structure.
block_data This block's record. See Block registry.
cluster_data The cluster's record. See Cluster registry.
input_data (in eval) packet The task, as an AIOSPacket object.
instances The current list of instance IDs, refreshed on every call.

What it returns: {"instance_id": "<chosen instance>"}

Management commands it should support: health_check (returns the known instances and a status) and get_current_mapping (returns its session-to-instance mapping, if any).

Load-balancer policy template: random choice with session stickiness
import random

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

        # get_metrics() is injected by the executor. Call it at any time to get
        # current block and cluster metrics for load-balancing decisions.
        self.metrics_function = self.settings["get_metrics"]
        self.block_data = self.settings["block_data"]
        self.cluster_data = self.settings["cluster_data"]

        # Optional: remember which instance served each session. This speeds up
        # decisions and keeps stateful sessions on the same instance.
        self.session_ids_cache = {}

        # The current instances are passed on every eval() call, so additions
        # and removals are always reflected. Keeping a copy is optional.
        self.current_instances = []

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated.
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.

        Returns:
            dict: {"instance_id": <chosen instance>}
        """
        # input_data["instances"] = ["instance-0", "instance-1", ...]
        if len(input_data["instances"]) != len(self.current_instances):
            # Instances were added or removed: reset the sessions cache (for example)
            self.session_ids_cache.clear()
            self.current_instances = input_data["instances"]

        # packet is the AIOSPacket protobuf object
        packet = input_data["packet"]
        session_id = packet.session_id
        if session_id in self.session_ids_cache:
            return {"instance_id": self.session_ids_cache[session_id]}

        # Current metrics:
        # {"block_metrics": ..., "cluster_metrics": ...}
        current_metrics = self.metrics_function()

        # Do the actual computation here; this example just picks at random
        random_instance_id = random.choice(input_data["instances"])

        # Remember the choice for this session
        self.session_ids_cache[session_id] = random_instance_id

        return {"instance_id": random_instance_id}

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.

        Required actions:
        1. "health_check":         return {"instances": self.current_instances, "status": "healthy"}
        2. "get_current_mapping":  return {"mapping": self.session_ids_cache}
        """
        if action == "health_check":
            return {"instances": self.current_instances, "status": "healthy"}
        if action == "get_current_mapping":
            return {"mapping": self.session_ids_cache}
        return {"message": f"Unknown action: {action}"}

Auto-scaler

The auto-scaler runs as a sidecar. At an interval set when the block is created, it calls your auto-scaling policy. The policy can read current block and cluster metrics, and decides whether to add instances, remove idle ones or do nothing. Instance counts stay between the block's minInstances and maxInstances.

Auto-scaler architecture

(Download the architecture diagram)

What the policy receives

Where Item Description
settings (in __init__) get_metrics A function that returns current block and cluster metrics.
input_data (in eval) block_data This block's record.
cluster_data The cluster's record.

What it returns (one of):

Decision Return value
Do nothing {"skip": True}
Add instances {"skip": False, "operation": "upscale", "instances_count": 2}
Remove specific instances {"skip": False, "operation": "downscale", "instances_list": ["instance-0", "instance-3"]}
Auto-scaling policy template
class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

        # get_metrics() is injected by the auto-scaler. Call it at any time to get
        # current block and cluster metrics for scaling decisions.
        self.metrics_function = self.settings["get_metrics"]

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): {"block_data": ..., "cluster_data": ...}
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.
        """
        block_data = input_data["block_data"]
        cluster_data = input_data["cluster_data"]

        # Rule logic goes here

        # Case 1: no change
        return {"skip": True}

        # Case 2: add instances
        # return {"skip": False, "operation": "upscale", "instances_count": 2}

        # Case 3: remove specific instances
        # return {"skip": False, "operation": "downscale",
        #         "instances_list": ["instance-0", "instance-3"]}

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.
        """
        # Implement custom management actions here
        return {}

Health checker

The health checker is another sidecar. At fixed intervals it:

  1. Calls http://{pod_ip}:18001/health on every instance, with a timeout. This runs your instance's health() method.
  2. Marks an instance unhealthy if it returns a non-200 status or times out.
  3. Calls your health-check policy with the result for every instance.
  4. Leaves it to the policy to decide what to do. For example, it could notify a developer on Slack, call the failure policy executor, or ask the Cluster Controller Gateway to replace the instance.

What the policy receives

Where Item Description
settings (in __init__) block_data, cluster_data This block's and its cluster's records.
input_data (in eval) health_check_data {instance_id: True/False}, where True means healthy.
Health-check policy template: escalate after repeated failures
import logging
import requests

def call_failure_policy(api_url: str, failure_policy_id: str, inputs: dict, parameters: dict):
    try:
        url = f"{api_url}/executeFailurePolicy"
        payload = {
            "failure_policy_id": failure_policy_id,
            "inputs": inputs,
            "parameters": parameters
        }
        headers = {"Content-Type": "application/json"}

        response = requests.post(url, json=payload, headers=headers)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        logging.error(f"Request failed: {e}")
        return {"success": False, "message": str(e)}
    except Exception as ex:
        logging.error(f"Unexpected error: {ex}")
        return {"success": False, "message": str(ex)}


class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

        # Block and cluster data are passed in settings
        self.block_data = self.settings["block_data"]
        self.cluster_data = self.settings["cluster_data"]

        # Count how many times each instance has been unhealthy
        self.counter = {}

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        input_data["health_check_data"] maps instance IDs to health, for example:
        {
            "instance-00": True,
            "instance-01": True,
            "instance-02": False
        }
        """
        health_check_data = input_data["health_check_data"]

        for instance, is_healthy in health_check_data.items():
            if not is_healthy:
                self.counter[instance] = self.counter.get(instance, 0) + 1

                # For example: escalate if an instance was unhealthy more than 3 times
                if self.counter[instance] > 3:
                    call_failure_policy(
                        self.settings["failure_policy_server_api_url"],
                        self.settings["failure_policy_id"],
                        inputs={
                            "block_id": self.block_data['id'],
                            "cluster_data": self.cluster_data['id'],
                            "instance_id": instance
                        },
                        parameters={}
                    )
                    self.counter[instance] = 0

        return {}

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.
        """
        # Implement custom management actions here
        return {}

Endpoints and ports

When a block is created, the cluster's ingress gets a route at /block/<block-id>. From outside the cluster, a block is reachable at:

http://<cluster-public-url>/block/<block-id>

For example, block block-771a8 on a cluster whose public URL is my-home-cluster.clusters.aiosv1:

http://my-home-cluster.clusters.aiosv1/block/block-771a8
Purpose Internal port Ingress suffix Description
gRPC inference 50051 /rpc The executor's gRPC proxy for submitting tasks
Executor metrics 18000 /metrics Executor metrics
Executor management 18001 /executor Management commands for the load-balancer policy
Auto-scaler management 10000 /autoscaler Management commands for the auto-scaling policy
Health-checker management 19001 /health-checker Management commands for the health-check policy
Redis 6379 — Internal caching and coordination
Mapping 5000 — Internal use

Management APIs

Each policy-driven service accepts management commands at its own /mgmt endpoint. The mgmt_action values that are available depend on what your policy's management() method implements.

curl -X POST http://<cluster-public-url>/block/<block-id>/executor/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "<action>",
    "mgmt_data": {}
  }'
curl -X POST http://<cluster-public-url>/block/<block-id>/autoscaler/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "<action>",
    "mgmt_data": {}
  }'
curl -X POST http://<cluster-public-url>/block/<block-id>/health-checker/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "<action>",
    "mgmt_data": {}
  }'

You can also send management commands through the network's Parser or Gateway, which apply governance checks. See Management commands.

Executor metrics

curl -X GET http://<cluster-public-url>/block/<block-id>/metrics
Metric Description
tasks_processed Number of tasks processed by the executor
latency Average end-to-end latency of tasks, in seconds

For instance-level and custom metrics, see Metrics system: block metrics.