Skip to content

Instance SDK

The AIOS Instance SDK is the Python library you use to write the code that runs inside a block. You write one class with a few methods (set up, prepare input, process, report health), and the SDK handles everything around it: receiving tasks, batching, metrics, management commands and sending results on.

It is general-purpose: use it for AI model inference, data processing, web backends or any other computation. Instances can also make outbound network calls to external services.

At a glance

Who this is for: developers building blocks.
You'll learn: how to write, test, package and run a block instance.
Source: services/applications/aios-sdk/instance-sdk in the repository.

What the SDK gives you

Feature What it means for you
Clear processing stages Split your logic into preprocessing and processing (inference) methods.
Management commands Handle custom runtime commands, such as reset or reload, with one callback.
Live parameter updates Change parameters such as thresholds at runtime, without restarting.
Muxing Merge related inputs from several upstream blocks into one task, which is useful in vDAGs.
Batching Group tasks for efficient batched inference.
GPU support Use GPUs from your code as usual.
State helpers In-memory and Redis-backed state dictionaries for stateful workflows.
Metrics Define custom Prometheus-style metrics. Load balancers and auto-scalers can use them.
Events Raise events that trigger policies in the cluster controller.
Sidecars Push outputs to helper services deployed alongside the block, for example for auditing or saving results.
Local testing Run your class locally with BlockTester, with no infrastructure.
LLM extension The LLM SDK utilities add helpers for serving large language models.

How an instance processes a task

flowchart LR
    IN["Incoming task<br/>(AIOSPacket)"] --> MUX{"Muxer?<br/>get_muxer()"}
    MUX -->|"no"| PRE["on_preprocess(packet)"]
    MUX -->|"merge N related<br/>packets"| PRE
    PRE -->|"one or more<br/>PreProcessResult"| DATA["on_data(entry)"]
    DATA -->|"OnDataResult"| OUT["Result sent on<br/>(client, next block, sidecars)"]
    CTRL["Management and<br/>health calls"] -.-> MGMT["management() / health() / on_update()"]

Quick start

1. Install the SDK

From the repository root:

cd services/applications/aios-sdk/instance-sdk
pip3 install -e .

2. Write your block class

Create main.py with a class that implements the required methods. This minimal example checks whether an input value passes a threshold:

import json
from aios_instance import Block, PreProcessResult, OnDataResult


class SimpleBlock:
    def __init__(self, context):
        """Initializes the block with its initial parameters."""
        self.context = context
        self.threshold = context.block_init_parameters.get("threshold", 0.5)

    def on_preprocess(self, packet):
        """Parses the JSON data from the packet."""
        try:
            data = json.loads(packet.data)
            return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
        except Exception as e:
            return False, str(e)

    def on_data(self, preprocessed_entry):
        """Processes the input and returns a result based on a simple threshold."""
        try:
            input_data = preprocessed_entry.extra_data["input"]
            value = input_data.get("value", 0)

            result = {
                "value": value,
                "passed": value > self.threshold
            }
            return True, OnDataResult(output=result)
        except Exception as e:
            return False, str(e)

    def on_update(self, updated_parameters):
        """Updates the threshold value at runtime."""
        try:
            if "threshold" in updated_parameters:
                self.threshold = updated_parameters["threshold"]
            return True, updated_parameters
        except Exception as e:
            return False, str(e)

    def health(self):
        """Returns basic health status."""
        return {"status": "healthy"}

    def management(self, action, data):
        """Handles simple management commands."""
        if action == "get_threshold":
            return {"threshold": self.threshold}
        elif action == "reset":
            self.threshold = 0.5
            return {"message": "Threshold reset to default"}
        else:
            return {"message": f"Unknown action: {action}"}

    def get_muxer(self):
        """No muxing needed for this example."""
        return None


if __name__ == "__main__":
    block = Block(SimpleBlock)
    block.run()

3. Test it locally

Use BlockTester to run your class without Redis, metrics or servers. See Testing.

from aios_instance.testing import BlockTester
from main import SimpleBlock

tester = BlockTester(SimpleBlock)
print(tester.run({"value": 0.9}))   # [{"value": 0.9, "passed": True}]

4. Build the container image

First build the AIOS base image, once, from the SDK directory:

cd services/applications/aios-sdk/instance-sdk
docker build . -f Dockerfile -t aios_instance:v1

Then build your own image on top of it:

FROM aios_instance:v1

COPY . /app
WORKDIR /app

# Install your dependencies
RUN pip3 install -r requirements.txt

ENTRYPOINT ["python3", "main.py"]

Push the image to a container registry. Next, register it as a component and create a block. The Block creator guide walks through the whole process.


The block class

Your class must implement these methods:

Method Called when Receives Returns
__init__ Once, when the instance starts context —
on_preprocess For every incoming task (or merged task) AIOSPacket (True, [PreProcessResult, ...]) or (False, "error")
on_data For each result of on_preprocess PreProcessResult (True, OnDataResult) or (False, "error")
on_update When parameters are changed at runtime dict (True, dict) or (False, "error")
health On every health check (/health) — any JSON-serializable value
management On every management command (/mgmt) action: str, data: dict any JSON-serializable value
get_muxer At start-up — a Muxer or None

Register the class and start the instance with:

from aios_instance import Block

if __name__ == "__main__":
    block = Block(YourCustomBlock)
    block.run()

__init__(self, context)

Called once when the instance starts. Use it to load models, set up caches and register metrics.

The context object contains:

Attribute What it is
context.block_data The block's full record from the Blocks database.
context.block_init_data Initial, usually static, configuration for the block.
context.block_init_settings Runtime settings for the block.
context.block_init_parameters Parameters that can be changed at runtime (see on_update).
context.allocation_info Where this instance runs: {"gpus": [], "node_id": "", "cluster_id": ""}.
context.metrics An AIOSMetrics object for custom metrics.
context.events A BlockEvents object for raising events.
context.sessions A SessionsManager for keeping per-session data across requests.
context.side_cars A BlockSideCars object for talking to sidecars.
def __init__(self, context):
    self.context = context
    # Read a parameter, with a default
    self.threshold = context.block_init_parameters.get("threshold", 0.5)

on_preprocess(self, packet)

Called for every incoming task. Use it to parse and validate the input. It can also split one task into several items to process.

  • Input: an AIOSPacket. Normally packet.data is a JSON string. If a Muxer is active, it is a dictionary of the form {"inputs": [...]}.
  • Returns: (True, [PreProcessResult, ...]), where each PreProcessResult wraps the packet plus any extra data you add in extra_data; or (False, "error message").
def on_preprocess(self, packet):
    try:
        data = packet.data
        if isinstance(data, str):
            data = json.loads(data)

        if "inputs" in data:
            # Merged by a Muxer: one result per input
            return True, [
                PreProcessResult(packet=packet, extra_data={"input": item})
                for item in data["inputs"]
            ]
        else:
            # Normal single input
            return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
    except Exception as e:
        return False, str(e)

on_data(self, preprocessed_entry)

Called once for each PreProcessResult. This is where your main logic, such as model inference, runs.

  • Input: a PreProcessResult with .packet (the original AIOSPacket) and .extra_data (what you added in on_preprocess).
  • Returns: (True, OnDataResult(output=...)), where output is a JSON-serializable dictionary; or (False, "error message") to log the error and skip the task.

The output dictionary is JSON-encoded into the data field of the outgoing packet. The packet's output_ptr decides where it is sent next.

def on_data(self, preprocessed_entry):
    try:
        result = {
            "processed_at": time.time(),
            "input": preprocessed_entry.extra_data["input"],
            "status": "ok"
        }
        return True, OnDataResult(output=result)
    except Exception as e:
        return False, str(e)

on_update(self, updated_parameters)

Called when the block's parameters are changed at runtime, either through the Redis parameter-update queue or an HTTP POST to /setParameters. Use it for things like thresholds, flags or runtime settings.

  • Returns: (True, updated_parameters) on success; (False, "error message") if the update is invalid.
def on_update(self, updated_parameters):
    try:
        if "threshold" in updated_parameters:
            self.threshold = updated_parameters["threshold"]
        return True, updated_parameters
    except Exception as e:
        return False, str(e)

health(self)

Answers the /health endpoint, which the block's health checker polls. Return any JSON-serializable value. Don't raise an exception unless the instance is really unhealthy.

def health(self):
    return {"status": "healthy", "timestamp": time.time()}

Third-party systems

If your block forwards work to an external system, health() should check that system too, because your instance answers health checks on its behalf.

management(self, action, data)

Handles custom commands sent to the /mgmt endpoint, for example to reset state, reload a model or report status. See Management commands for how to send them.

def management(self, action, data):
    try:
        if action == "reset":
            self.context.sessions.clear()
            return {"message": "Session state reset"}
        elif action == "status":
            return {"message": "All systems operational"}
        else:
            return {"message": f"Unknown action '{action}'"}
    except Exception as e:
        return {"error": str(e)}

get_muxer(self)

Return a Muxer to merge related packets before on_preprocess, or None if you don't need one. See Muxer.

from aios_instance.tools import Muxer

def get_muxer(self):
    return Muxer(N=4)  # wait for 4 packets with the same session_id and seq_no, then merge them

AIOSPacket: the task format

Every task reaches your block as an AIOSPacket:

message AIOSPacket {
    string session_id = 1;      // Session identifier; enables stateful logic and graph routing
    uint64 seq_no = 2;          // Sequence number within the session
    string data = 4;            // JSON-encoded input payload (merged into {"inputs": [...]} by a Muxer)
    double ts = 5;              // Optional Unix timestamp (used for latency metrics and ordering)
    string output_ptr = 6;      // JSON-encoded routing to downstream blocks
    repeated FileInfo files = 7;// Optional attached files
}

message FileInfo {
    string metadata = 1;        // JSON string describing the file
    bytes file_data = 2;        // Raw file content
}

Muxing and batching

You can group packets before or during processing:

Strategy What it does What your code receives
Muxer(N) Merges N packets with the same (session_id, seq_no) into one. packet.data = {"inputs": [<data1>, <data2>, ...]}
Batcher(N) Collects any N packets in arrival order. A list of AIOSPackets (not merged)
TimeBasedBatcher(N, T) Emits a batch when N packets arrive or T seconds pass. A list of AIOSPackets (not merged)

Only the Muxer changes the structure of packet.data.

A Muxer waits until it has N packets with the same (session_id, seq_no), then merges them into one packet: their data becomes {"inputs": [...]} and their files are combined. Use it when one step needs inputs from several upstream steps, for example a multi-camera model, an ensemble, or a vDAG node with several parents.

How the Muxer works (source)
class Muxer:
    def __init__(self, N: int):
        self.N = N
        self.store = defaultdict(list)
        self.counts = defaultdict(int)

    def process_packet(self, packet):
        if self.N == 1:
            return packet  # No merging needed

        key = (packet.session_id, packet.seq_no)
        self.store[key].append(packet)
        self.counts[key] += 1

        if self.counts[key] == self.N:
            merged_packet = self._merge_packets(self.store[key])
            del self.store[key]
            del self.counts[key]
            return merged_packet

        return None

    def _merge_packets(self, packets):
        merged_data = {"inputs": [json.loads(p.data) for p in packets]}
        merged_files = [file for p in packets for file in p.files]

        base_packet = copy.deepcopy(packets[0])
        base_packet.data = merged_data
        base_packet.files = merged_files

        return base_packet

To use it, return it from get_muxer(), and make sure on_preprocess handles {"inputs": [...]} (see the example above). With N=1, packets pass through unchanged.

Batcher: group tasks for batch inference

A Batcher collects N packets and returns them as a list, without changing them. Use it when your model can process several inputs at once, for example image batches or transformer batches.

Unlike the Muxer, the Batcher is not plugged in automatically: you call it yourself.

How the Batcher works (source)
class Batcher:
    def __init__(self, N: int):
        self.N = N
        self.batch = []

    def add_to_batch(self, packet):
        self.batch.append(packet)

        if len(self.batch) >= self.N:
            batch_copy = self.batch
            self.batch = []
            return batch_copy

        return None

Typical use inside on_data: add each processed packet, and emit a result only when the batch is full. See the full example.

Which one should I use?

Muxer Batcher
Output One merged AIOSPacket A list of AIOSPackets
Groups N packets with the same session and seq_no Any N packets
Changes the packet Yes (data becomes {"inputs": [...]}) No
Plugged in Automatically, via get_muxer() Manually, in your code
Use for Combining several inputs for one task Mini-batching many independent tasks

Tip

Handle the edge cases: groups that never complete, invalid JSON, oversized batches and expired sessions.


Metrics

AIOSMetrics (available as context.metrics) lets you define your own Prometheus-compatible metrics. They are exposed over HTTP for scraping and pushed to Redis for central collection, so that load-balancing and auto-scaling policies can use them.

Type Use for Example
Counter A total that only goes up images processed
Gauge A value that goes up and down queue length, FPS
Histogram A distribution of values inference latency

Register metrics (in __init__)

metrics = context.metrics

metrics.register_counter("images_processed_total", "Total number of images processed")
metrics.register_counter("detections_emitted_total", "Total number of detected objects")
metrics.register_gauge("input_queue_length", "Jobs in Redis input queue")
metrics.register_histogram("inference_latency_seconds", "Time taken for model inference",
                           buckets=[0.05, 0.1, 0.2, 0.5, 1, 2])
metrics.register_histogram("objects_per_image", "Number of detections per image",
                           buckets=[0, 1, 2, 3, 5, 10, 20])
Method Purpose
register_counter(name, documentation, labelnames=None) Create a counter.
register_gauge(name, documentation, labelnames=None) Create a gauge.
register_histogram(name, documentation, labelnames=None, buckets=None) Create a histogram. Default latency buckets are used if buckets is not given.

Update metrics (while processing)

# Count the image
metrics.increment_counter("images_processed_total")

# Measure inference time
start = time.time()
results = model.detect_objects(image)
metrics.observe_histogram("inference_latency_seconds", time.time() - start)

# Record detection statistics
metrics.increment_counter("detections_emitted_total")
metrics.observe_histogram("objects_per_image", len(results))

# Queue monitoring
metrics.set_gauge("input_queue_length", redis_client.llen("block_inputs"))
Method Purpose
increment_counter(name) Add one to a counter.
set_gauge(name, value) Set a gauge to a value.
observe_histogram(name, value) Add a sample to a histogram.

Blocks also report standard metrics automatically.


Events

BlockEvents (available as context.events) lets your code raise events, such as "inference started" or "something failed". Each event is pushed to a Redis queue (EVENTS_QUEUE by default) and handled by the block events processor in the cluster controller, which runs the policy you attached to that event.

Each event contains:

  • event_name: the type of event
  • block_id: filled in automatically from the BLOCK_ID environment variable
  • event_data: any extra information you include
self.context.events.push_event(
    event_name="inference_started",
    event_data={
        "session_id": packet.session_id,
        "seq_no": packet.seq_no,
        "ts": packet.ts
    }
)

The block does not wait for the event to be handled.


Sidecars

Sidecars are helper services deployed alongside a block, for example a feature generator, a results saver or an auditor. BlockSideCars (available as context.side_cars) sends Protobuf messages, such as an AIOSPacket, to them through Redis queues. Sidecars are configured on the component; see Component spec.

push_to_sidecar(name, input_proto)

Sends a message to one sidecar.

Parameter Type Description
name str The sidecar's name, as configured on the component.
input_proto google.protobuf.Message The message to send, for example an AIOSPacket.

How it works:

  • It finds the sidecar's Redis address: internally <name>-<block_id>-svc.sidecars.svc.cluster.local, or external_redis_url from the configuration if the sidecar is marked external: true.
  • It serializes the message with .SerializeToString() and pushes it to the Redis list INPUTS.
  • If the push fails, it reconnects and retries once.
# Push a packet to the "feature_generator" sidecar
self.context.side_cars.push_to_sidecar(
    name="feature_generator",
    input_proto=packet  # must be a protobuf message, for example an AIOSPacket
)

push_to_all_sidecars(input_proto)

Sends the same message to every configured sidecar. If one fails, the error is logged and the others are still tried.

self.context.side_cars.push_to_all_sidecars(input_proto=packet)

Testing your block

BlockTester runs your class end to end (packet → on_preprocess → on_data) without Redis, metrics, batching infrastructure or HTTP servers. Use it for unit tests and debugging.

from aios_instance.testing import BlockTester
from your_blocks import YourBlockClass

tester = BlockTester(YourBlockClass)   # creates a mock TestContext and your block

# Run with a simple input (converted to an AIOSPacket for you)
result = tester.run({"value": 0.9})
print(result)

Attach files. Each file is a tuple of (metadata dict, raw bytes):

test_files = [
    ({"filename": "image.jpg", "type": "jpeg"}, b"\xff\xd8\xff...")
]

result = tester.run(data={"value": 1.0}, files=test_files)

Use a custom context, for example to set parameters or inject real metrics, sessions or events objects:

from aios_instance.testing import BlockTester, TestContext
from your_blocks import YourBlockClass
from your_runtime import AIOSMetrics, SessionsManager, BlockEvents

context = TestContext()
context.block_init_parameters = {"threshold": 0.75}
context.metrics = AIOSMetrics()
context.sessions = SessionsManager()
context.events = BlockEvents()

tester = BlockTester.init_with_context(YourBlockClass, context)
result = tester.run({"value": 0.6})

Output. run() returns a list of the dictionaries produced by on_data:

[
  {"value": 0.9, "passed": True}
]

If your block batches results, the output contains the batch:

[
  {
    "batch": [
      {"value": 0.6, "passed": True},
      {"value": 0.4, "passed": False},
      ...
    ]
  }
]

Example: Muxer and Batcher together

This block merges 3 related inputs with a Muxer, applies a threshold to each, and emits results in batches of 4.

import json
from aios_instance import PreProcessResult, OnDataResult
from aios_instance.tools import Muxer, Batcher


class BlockWithMuxerAndBatcher:
    def __init__(self, context):
        """
        Sets up parameters and a Batcher.

        - threshold: a configurable float parameter used for the check.
        - batcher: collects 4 processed packets before emitting a result.
        """
        self.context = context
        self.threshold = context.block_init_parameters.get("threshold", 0.5)
        self.batcher = Batcher(N=4)

    def on_preprocess(self, packet):
        """Handles muxed ({"inputs": [...]}) and single-packet inputs."""
        try:
            data = packet.data
            if isinstance(data, str):
                data = json.loads(data)

            if "inputs" in data:
                return True, [
                    PreProcessResult(packet=packet, extra_data={"input": item})
                    for item in data["inputs"]
                ]
            else:
                return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
        except Exception as e:
            return False, str(e)

    def on_data(self, preprocessed_entry):
        """
        Processes one entry and adds it to the Batcher.

        Returns:
        - (True, OnDataResult) when a full batch is emitted.
        - (True, None) while the batch is still filling.
        - (False, error message) on failure.
        """
        try:
            value = preprocessed_entry.extra_data["input"].get("value", 0)
            result = {"value": value, "passed": value > self.threshold}

            packet = preprocessed_entry.packet
            packet.data = json.dumps(result)

            batch = self.batcher.add_to_batch(packet)
            if batch:
                return True, OnDataResult(output={"batch": [json.loads(p.data) for p in batch]})

            # Waiting for the batch to fill
            return True, None
        except Exception as e:
            return False, str(e)

    def on_update(self, updated_parameters):
        try:
            if "threshold" in updated_parameters:
                self.threshold = updated_parameters["threshold"]
            return True, updated_parameters
        except Exception as e:
            return False, str(e)

    def health(self):
        return {"status": "healthy"}

    def management(self, action, data):
        if action == "get_threshold":
            return {"threshold": self.threshold}
        elif action == "reset":
            self.threshold = 0.5
            return {"message": "Threshold reset to default"}
        return {"message": f"Unknown action: {action}"}

    def get_muxer(self):
        """Merge 3 packets with the same session ID and sequence number."""
        return Muxer(N=3)

Next steps