Instance SDK¶
The AIOS Instance SDK is the Python library you use to write the code that runs inside a block. You write one class with a few methods (set up, prepare input, process, report health), and the SDK handles everything around it: receiving tasks, batching, metrics, management commands and sending results on.
It is general-purpose: use it for AI model inference, data processing, web backends or any other computation. Instances can also make outbound network calls to external services.
At a glance
Who this is for: developers building blocks.
You'll learn: how to write, test, package and run a block instance.
Source: services/applications/aios-sdk/instance-sdk in the repository.
What the SDK gives you¶
| Feature | What it means for you |
|---|---|
| Clear processing stages | Split your logic into preprocessing and processing (inference) methods. |
| Management commands | Handle custom runtime commands, such as reset or reload, with one callback. |
| Live parameter updates | Change parameters such as thresholds at runtime, without restarting. |
| Muxing | Merge related inputs from several upstream blocks into one task, which is useful in vDAGs. |
| Batching | Group tasks for efficient batched inference. |
| GPU support | Use GPUs from your code as usual. |
| State helpers | In-memory and Redis-backed state dictionaries for stateful workflows. |
| Metrics | Define custom Prometheus-style metrics. Load balancers and auto-scalers can use them. |
| Events | Raise events that trigger policies in the cluster controller. |
| Sidecars | Push outputs to helper services deployed alongside the block, for example for auditing or saving results. |
| Local testing | Run your class locally with BlockTester, with no infrastructure. |
| LLM extension | The LLM SDK utilities add helpers for serving large language models. |
How an instance processes a task¶
flowchart LR
IN["Incoming task<br/>(AIOSPacket)"] --> MUX{"Muxer?<br/>get_muxer()"}
MUX -->|"no"| PRE["on_preprocess(packet)"]
MUX -->|"merge N related<br/>packets"| PRE
PRE -->|"one or more<br/>PreProcessResult"| DATA["on_data(entry)"]
DATA -->|"OnDataResult"| OUT["Result sent on<br/>(client, next block, sidecars)"]
CTRL["Management and<br/>health calls"] -.-> MGMT["management() / health() / on_update()"]
Quick start¶
1. Install the SDK¶
From the repository root:
2. Write your block class¶
Create main.py with a class that implements the required methods. This minimal example checks whether an input value passes a threshold:
import json
from aios_instance import Block, PreProcessResult, OnDataResult
class SimpleBlock:
def __init__(self, context):
"""Initializes the block with its initial parameters."""
self.context = context
self.threshold = context.block_init_parameters.get("threshold", 0.5)
def on_preprocess(self, packet):
"""Parses the JSON data from the packet."""
try:
data = json.loads(packet.data)
return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
except Exception as e:
return False, str(e)
def on_data(self, preprocessed_entry):
"""Processes the input and returns a result based on a simple threshold."""
try:
input_data = preprocessed_entry.extra_data["input"]
value = input_data.get("value", 0)
result = {
"value": value,
"passed": value > self.threshold
}
return True, OnDataResult(output=result)
except Exception as e:
return False, str(e)
def on_update(self, updated_parameters):
"""Updates the threshold value at runtime."""
try:
if "threshold" in updated_parameters:
self.threshold = updated_parameters["threshold"]
return True, updated_parameters
except Exception as e:
return False, str(e)
def health(self):
"""Returns basic health status."""
return {"status": "healthy"}
def management(self, action, data):
"""Handles simple management commands."""
if action == "get_threshold":
return {"threshold": self.threshold}
elif action == "reset":
self.threshold = 0.5
return {"message": "Threshold reset to default"}
else:
return {"message": f"Unknown action: {action}"}
def get_muxer(self):
"""No muxing needed for this example."""
return None
if __name__ == "__main__":
block = Block(SimpleBlock)
block.run()
3. Test it locally¶
Use BlockTester to run your class without Redis, metrics or servers. See Testing.
from aios_instance.testing import BlockTester
from main import SimpleBlock
tester = BlockTester(SimpleBlock)
print(tester.run({"value": 0.9})) # [{"value": 0.9, "passed": True}]
4. Build the container image¶
First build the AIOS base image, once, from the SDK directory:
Then build your own image on top of it:
FROM aios_instance:v1
COPY . /app
WORKDIR /app
# Install your dependencies
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python3", "main.py"]
Push the image to a container registry. Next, register it as a component and create a block. The Block creator guide walks through the whole process.
The block class¶
Your class must implement these methods:
| Method | Called when | Receives | Returns |
|---|---|---|---|
__init__ |
Once, when the instance starts | context |
— |
on_preprocess |
For every incoming task (or merged task) | AIOSPacket |
(True, [PreProcessResult, ...]) or (False, "error") |
on_data |
For each result of on_preprocess |
PreProcessResult |
(True, OnDataResult) or (False, "error") |
on_update |
When parameters are changed at runtime | dict |
(True, dict) or (False, "error") |
health |
On every health check (/health) |
— | any JSON-serializable value |
management |
On every management command (/mgmt) |
action: str, data: dict |
any JSON-serializable value |
get_muxer |
At start-up | — | a Muxer or None |
Register the class and start the instance with:
from aios_instance import Block
if __name__ == "__main__":
block = Block(YourCustomBlock)
block.run()
__init__(self, context)¶
Called once when the instance starts. Use it to load models, set up caches and register metrics.
The context object contains:
| Attribute | What it is |
|---|---|
context.block_data |
The block's full record from the Blocks database. |
context.block_init_data |
Initial, usually static, configuration for the block. |
context.block_init_settings |
Runtime settings for the block. |
context.block_init_parameters |
Parameters that can be changed at runtime (see on_update). |
context.allocation_info |
Where this instance runs: {"gpus": [], "node_id": "", "cluster_id": ""}. |
context.metrics |
An AIOSMetrics object for custom metrics. |
context.events |
A BlockEvents object for raising events. |
context.sessions |
A SessionsManager for keeping per-session data across requests. |
context.side_cars |
A BlockSideCars object for talking to sidecars. |
def __init__(self, context):
self.context = context
# Read a parameter, with a default
self.threshold = context.block_init_parameters.get("threshold", 0.5)
on_preprocess(self, packet)¶
Called for every incoming task. Use it to parse and validate the input. It can also split one task into several items to process.
- Input: an
AIOSPacket. Normallypacket.datais a JSON string. If a Muxer is active, it is a dictionary of the form{"inputs": [...]}. - Returns:
(True, [PreProcessResult, ...]), where eachPreProcessResultwraps the packet plus any extra data you add inextra_data; or(False, "error message").
def on_preprocess(self, packet):
try:
data = packet.data
if isinstance(data, str):
data = json.loads(data)
if "inputs" in data:
# Merged by a Muxer: one result per input
return True, [
PreProcessResult(packet=packet, extra_data={"input": item})
for item in data["inputs"]
]
else:
# Normal single input
return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
except Exception as e:
return False, str(e)
on_data(self, preprocessed_entry)¶
Called once for each PreProcessResult. This is where your main logic, such as model inference, runs.
- Input: a
PreProcessResultwith.packet(the originalAIOSPacket) and.extra_data(what you added inon_preprocess). - Returns:
(True, OnDataResult(output=...)), whereoutputis a JSON-serializable dictionary; or(False, "error message")to log the error and skip the task.
The output dictionary is JSON-encoded into the data field of the outgoing packet. The packet's output_ptr decides where it is sent next.
def on_data(self, preprocessed_entry):
try:
result = {
"processed_at": time.time(),
"input": preprocessed_entry.extra_data["input"],
"status": "ok"
}
return True, OnDataResult(output=result)
except Exception as e:
return False, str(e)
on_update(self, updated_parameters)¶
Called when the block's parameters are changed at runtime, either through the Redis parameter-update queue or an HTTP POST to /setParameters. Use it for things like thresholds, flags or runtime settings.
- Returns:
(True, updated_parameters)on success;(False, "error message")if the update is invalid.
def on_update(self, updated_parameters):
try:
if "threshold" in updated_parameters:
self.threshold = updated_parameters["threshold"]
return True, updated_parameters
except Exception as e:
return False, str(e)
health(self)¶
Answers the /health endpoint, which the block's health checker polls. Return any JSON-serializable value. Don't raise an exception unless the instance is really unhealthy.
Third-party systems
If your block forwards work to an external system, health() should check that system too, because your instance answers health checks on its behalf.
management(self, action, data)¶
Handles custom commands sent to the /mgmt endpoint, for example to reset state, reload a model or report status. See Management commands for how to send them.
def management(self, action, data):
try:
if action == "reset":
self.context.sessions.clear()
return {"message": "Session state reset"}
elif action == "status":
return {"message": "All systems operational"}
else:
return {"message": f"Unknown action '{action}'"}
except Exception as e:
return {"error": str(e)}
get_muxer(self)¶
Return a Muxer to merge related packets before on_preprocess, or None if you don't need one. See Muxer.
from aios_instance.tools import Muxer
def get_muxer(self):
return Muxer(N=4) # wait for 4 packets with the same session_id and seq_no, then merge them
AIOSPacket: the task format¶
Every task reaches your block as an AIOSPacket:
message AIOSPacket {
string session_id = 1; // Session identifier; enables stateful logic and graph routing
uint64 seq_no = 2; // Sequence number within the session
string data = 4; // JSON-encoded input payload (merged into {"inputs": [...]} by a Muxer)
double ts = 5; // Optional Unix timestamp (used for latency metrics and ordering)
string output_ptr = 6; // JSON-encoded routing to downstream blocks
repeated FileInfo files = 7;// Optional attached files
}
message FileInfo {
string metadata = 1; // JSON string describing the file
bytes file_data = 2; // Raw file content
}
Muxing and batching¶
You can group packets before or during processing:
| Strategy | What it does | What your code receives |
|---|---|---|
Muxer(N) |
Merges N packets with the same (session_id, seq_no) into one. |
packet.data = {"inputs": [<data1>, <data2>, ...]} |
Batcher(N) |
Collects any N packets in arrival order. | A list of AIOSPackets (not merged) |
TimeBasedBatcher(N, T) |
Emits a batch when N packets arrive or T seconds pass. | A list of AIOSPackets (not merged) |
Only the Muxer changes the structure of packet.data.
Muxer: merge related inputs¶
A Muxer waits until it has N packets with the same (session_id, seq_no), then merges them into one packet: their data becomes {"inputs": [...]} and their files are combined. Use it when one step needs inputs from several upstream steps, for example a multi-camera model, an ensemble, or a vDAG node with several parents.
How the Muxer works (source)
class Muxer:
def __init__(self, N: int):
self.N = N
self.store = defaultdict(list)
self.counts = defaultdict(int)
def process_packet(self, packet):
if self.N == 1:
return packet # No merging needed
key = (packet.session_id, packet.seq_no)
self.store[key].append(packet)
self.counts[key] += 1
if self.counts[key] == self.N:
merged_packet = self._merge_packets(self.store[key])
del self.store[key]
del self.counts[key]
return merged_packet
return None
def _merge_packets(self, packets):
merged_data = {"inputs": [json.loads(p.data) for p in packets]}
merged_files = [file for p in packets for file in p.files]
base_packet = copy.deepcopy(packets[0])
base_packet.data = merged_data
base_packet.files = merged_files
return base_packet
To use it, return it from get_muxer(), and make sure on_preprocess handles {"inputs": [...]} (see the example above). With N=1, packets pass through unchanged.
Batcher: group tasks for batch inference¶
A Batcher collects N packets and returns them as a list, without changing them. Use it when your model can process several inputs at once, for example image batches or transformer batches.
Unlike the Muxer, the Batcher is not plugged in automatically: you call it yourself.
How the Batcher works (source)
Typical use inside on_data: add each processed packet, and emit a result only when the batch is full. See the full example.
Which one should I use?¶
| Muxer | Batcher | |
|---|---|---|
| Output | One merged AIOSPacket |
A list of AIOSPackets |
| Groups | N packets with the same session and seq_no |
Any N packets |
| Changes the packet | Yes (data becomes {"inputs": [...]}) |
No |
| Plugged in | Automatically, via get_muxer() |
Manually, in your code |
| Use for | Combining several inputs for one task | Mini-batching many independent tasks |
Tip
Handle the edge cases: groups that never complete, invalid JSON, oversized batches and expired sessions.
Metrics¶
AIOSMetrics (available as context.metrics) lets you define your own Prometheus-compatible metrics. They are exposed over HTTP for scraping and pushed to Redis for central collection, so that load-balancing and auto-scaling policies can use them.
| Type | Use for | Example |
|---|---|---|
| Counter | A total that only goes up | images processed |
| Gauge | A value that goes up and down | queue length, FPS |
| Histogram | A distribution of values | inference latency |
Register metrics (in __init__)¶
metrics = context.metrics
metrics.register_counter("images_processed_total", "Total number of images processed")
metrics.register_counter("detections_emitted_total", "Total number of detected objects")
metrics.register_gauge("input_queue_length", "Jobs in Redis input queue")
metrics.register_histogram("inference_latency_seconds", "Time taken for model inference",
buckets=[0.05, 0.1, 0.2, 0.5, 1, 2])
metrics.register_histogram("objects_per_image", "Number of detections per image",
buckets=[0, 1, 2, 3, 5, 10, 20])
| Method | Purpose |
|---|---|
register_counter(name, documentation, labelnames=None) |
Create a counter. |
register_gauge(name, documentation, labelnames=None) |
Create a gauge. |
register_histogram(name, documentation, labelnames=None, buckets=None) |
Create a histogram. Default latency buckets are used if buckets is not given. |
Update metrics (while processing)¶
# Count the image
metrics.increment_counter("images_processed_total")
# Measure inference time
start = time.time()
results = model.detect_objects(image)
metrics.observe_histogram("inference_latency_seconds", time.time() - start)
# Record detection statistics
metrics.increment_counter("detections_emitted_total")
metrics.observe_histogram("objects_per_image", len(results))
# Queue monitoring
metrics.set_gauge("input_queue_length", redis_client.llen("block_inputs"))
| Method | Purpose |
|---|---|
increment_counter(name) |
Add one to a counter. |
set_gauge(name, value) |
Set a gauge to a value. |
observe_histogram(name, value) |
Add a sample to a histogram. |
Blocks also report standard metrics automatically.
Events¶
BlockEvents (available as context.events) lets your code raise events, such as "inference started" or "something failed". Each event is pushed to a Redis queue (EVENTS_QUEUE by default) and handled by the block events processor in the cluster controller, which runs the policy you attached to that event.
Each event contains:
event_name: the type of eventblock_id: filled in automatically from theBLOCK_IDenvironment variableevent_data: any extra information you include
self.context.events.push_event(
event_name="inference_started",
event_data={
"session_id": packet.session_id,
"seq_no": packet.seq_no,
"ts": packet.ts
}
)
The block does not wait for the event to be handled.
Sidecars¶
Sidecars are helper services deployed alongside a block, for example a feature generator, a results saver or an auditor. BlockSideCars (available as context.side_cars) sends Protobuf messages, such as an AIOSPacket, to them through Redis queues. Sidecars are configured on the component; see Component spec.
push_to_sidecar(name, input_proto)¶
Sends a message to one sidecar.
| Parameter | Type | Description |
|---|---|---|
name |
str |
The sidecar's name, as configured on the component. |
input_proto |
google.protobuf.Message |
The message to send, for example an AIOSPacket. |
How it works:
- It finds the sidecar's Redis address: internally
<name>-<block_id>-svc.sidecars.svc.cluster.local, orexternal_redis_urlfrom the configuration if the sidecar is markedexternal: true. - It serializes the message with
.SerializeToString()and pushes it to the Redis listINPUTS. - If the push fails, it reconnects and retries once.
# Push a packet to the "feature_generator" sidecar
self.context.side_cars.push_to_sidecar(
name="feature_generator",
input_proto=packet # must be a protobuf message, for example an AIOSPacket
)
push_to_all_sidecars(input_proto)¶
Sends the same message to every configured sidecar. If one fails, the error is logged and the others are still tried.
Testing your block¶
BlockTester runs your class end to end (packet → on_preprocess → on_data) without Redis, metrics, batching infrastructure or HTTP servers. Use it for unit tests and debugging.
from aios_instance.testing import BlockTester
from your_blocks import YourBlockClass
tester = BlockTester(YourBlockClass) # creates a mock TestContext and your block
# Run with a simple input (converted to an AIOSPacket for you)
result = tester.run({"value": 0.9})
print(result)
Attach files. Each file is a tuple of (metadata dict, raw bytes):
test_files = [
({"filename": "image.jpg", "type": "jpeg"}, b"\xff\xd8\xff...")
]
result = tester.run(data={"value": 1.0}, files=test_files)
Use a custom context, for example to set parameters or inject real metrics, sessions or events objects:
from aios_instance.testing import BlockTester, TestContext
from your_blocks import YourBlockClass
from your_runtime import AIOSMetrics, SessionsManager, BlockEvents
context = TestContext()
context.block_init_parameters = {"threshold": 0.75}
context.metrics = AIOSMetrics()
context.sessions = SessionsManager()
context.events = BlockEvents()
tester = BlockTester.init_with_context(YourBlockClass, context)
result = tester.run({"value": 0.6})
Output. run() returns a list of the dictionaries produced by on_data:
If your block batches results, the output contains the batch:
Example: Muxer and Batcher together¶
This block merges 3 related inputs with a Muxer, applies a threshold to each, and emits results in batches of 4.
import json
from aios_instance import PreProcessResult, OnDataResult
from aios_instance.tools import Muxer, Batcher
class BlockWithMuxerAndBatcher:
def __init__(self, context):
"""
Sets up parameters and a Batcher.
- threshold: a configurable float parameter used for the check.
- batcher: collects 4 processed packets before emitting a result.
"""
self.context = context
self.threshold = context.block_init_parameters.get("threshold", 0.5)
self.batcher = Batcher(N=4)
def on_preprocess(self, packet):
"""Handles muxed ({"inputs": [...]}) and single-packet inputs."""
try:
data = packet.data
if isinstance(data, str):
data = json.loads(data)
if "inputs" in data:
return True, [
PreProcessResult(packet=packet, extra_data={"input": item})
for item in data["inputs"]
]
else:
return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
except Exception as e:
return False, str(e)
def on_data(self, preprocessed_entry):
"""
Processes one entry and adds it to the Batcher.
Returns:
- (True, OnDataResult) when a full batch is emitted.
- (True, None) while the batch is still filling.
- (False, error message) on failure.
"""
try:
value = preprocessed_entry.extra_data["input"].get("value", 0)
result = {"value": value, "passed": value > self.threshold}
packet = preprocessed_entry.packet
packet.data = json.dumps(result)
batch = self.batcher.add_to_batch(packet)
if batch:
return True, OnDataResult(output={"batch": [json.loads(p.data) for p in batch]})
# Waiting for the batch to fill
return True, None
except Exception as e:
return False, str(e)
def on_update(self, updated_parameters):
try:
if "threshold" in updated_parameters:
self.threshold = updated_parameters["threshold"]
return True, updated_parameters
except Exception as e:
return False, str(e)
def health(self):
return {"status": "healthy"}
def management(self, action, data):
if action == "get_threshold":
return {"threshold": self.threshold}
elif action == "reset":
self.threshold = 0.5
return {"message": "Threshold reset to default"}
return {"message": f"Unknown action: {action}"}
def get_muxer(self):
"""Merge 3 packets with the same session ID and sequence number."""
return Muxer(N=3)
Next steps¶
- Package your code as a component: Component spec
- Deploy it: Block spec
- Serve LLMs: LLM SDK utilities
- Delegate to an external system: Third-party systems