vDAG controller¶
A vDAG controller is the front door of a vDAG workflow. Clients send tasks to it, it runs them through every block in the graph, and it returns the final result. Along the way it enforces the workflow owner's quotas, watches the health of every block involved, samples outputs for quality checks, and reports metrics.
At a glance
Who this is for: vDAG creators, and developers calling vDAGs.
You'll learn: how the controller works, how to create one, how to call it, and how to write its three policies.
Related: the vDAG creator guide for the end-to-end workflow, and the vDAG spec for defining the graph.
What it does¶
- Gateway. Accepts tasks for the vDAG, runs them, and returns the results.
- Quotas and rate limits. Lets the vDAG owner limit usage, per session or globally, with a quota checker policy.
- Health checks. Periodically checks every block in the vDAG, across the network, with a health checker policy.
- Quality checks. Periodically compares outputs with inputs, using a quality checker policy. It can forward samples to an external QA system or for manual review.
- Management APIs. Lets you send management commands to the three policies.
- Metrics. Reports latency, throughput and more.
Architecture¶

(Download the architecture diagram)
What happens to a task¶
sequenceDiagram
participant C as Client
participant R as RequestsProcessor
participant Q as QuotaChecker
participant P as PreProcessor
participant S as Ad-hoc inference server
participant B as Blocks in the vDAG
participant QC as QualityCheckProcessor
C->>R: 1. Submit task (gRPC)
R->>Q: 2. Check quota (policy)
Q-->>R: allowed / rejected
R-->>C: 3. Rejected, if over quota
R->>P: 4. Convert to internal format, build runtime graph
P->>S: 5. Submit graph task
S->>B: run each step
S-->>P: 6–8. Final output (via Redis queue), converted back
P->>QC: 9–10. Queue a quality-check sample (async)
P-->>R: 11. Output
R-->>C: 12. Response (same gRPC call)
- A client submits a task through the controller's gRPC API.
- The RequestsProcessor receives it and runs the quota checker policy, which can enforce per-session or global limits using Redis-backed counters.
- If the quota check fails, the client gets an immediate rejection. Otherwise the task continues.
- The PreProcessor converts the input into the format used by the ad-hoc inference server. It works out the entry
block_idand turns the vDAG's static graph into a runtime graph. - The task is sent to the ad-hoc inference server through its gRPC client.
- A Redis queue listener waits for the result, which the inference server pushes to a queue when it finishes.
- The listener picks up the result.
- The PreProcessor converts it back into a vDAG response.
- The response goes to the QualityCheckProcessor.
- The QualityCheckProcessor queues the input and output as a quality-check sample. A background worker evaluates samples with the quality policy, without slowing down the request.
- The output returns to the RequestsProcessor.
- The response is sent back to the client over the same gRPC connection.
Modules¶
| Module | What it does |
|---|---|
| Metrics server | Collects latency, throughput and FPS from the RequestsProcessor, and periodically writes them to the global vDAG metrics database. |
| Quota checker | Runs the quota policy, backed by a Redis QuotaTable that tracks usage per session or globally. |
| Quality checker | A background thread that reads samples from a queue and runs the quality policy on them. |
| Health checker | Periodically fetches the health of every block in the vDAG through the block management APIs, and runs the health policy on the results. |
| vDAG DB client | Reads the vDAG's structure, configuration and version from the vDAG registry. |
| Controller and API server | REST APIs for managing metrics, health checks, quotas, quality checks and other controller operations. |
Create a controller¶
A controller serves one vDAG, which must already exist in the registry. You can deploy it on any cluster in the network, through the Cluster Controller Gateway:
POST /vdag-controller/<cluster_id>: forwards the request to the /vdag-controller endpoint of the chosen cluster.
curl -X POST http://<cluster-controller-gateway-url>/vdag-controller/<cluster-id> \
-H "Content-Type: application/json" \
-d '{
"action": "create_controller",
"payload": {
"vdag_controller_id": "<controller-id>",
"vdag_uri": "<vdag-uri>",
"config": {
"policy_execution_mode": "local",
"replicas": 2
}
}
}'
| Parameter | Type | Description |
|---|---|---|
cluster_id (in the URL) |
string | Cluster to deploy the controller on. |
action |
string | create_controller. |
vdag_controller_id |
string | A unique ID for the controller. |
vdag_uri |
string | The vDAG this controller serves. |
config.policy_execution_mode |
string | Where the controller runs its policies, for example local. |
config.replicas |
integer | Number of controller replicas. |
Once created, the controller is reachable at:
and its management server at:
For example, controller vdag-123 on cluster us-west-1.aiosv1.cluster.net:
http://us-west-1.aiosv1.cluster.net/vdag-123 # gateway
http://us-west-1.aiosv1.cluster.net/vdag-123/server # management server
Remove a controller¶
curl -X POST http://<cluster-controller-gateway-url>/vdag-controller/<cluster_id> \
-H "Content-Type: application/json" \
-d '{
"action": "remove_controller",
"payload": {
"vdag_controller_id": "<controller-id>"
}
}'
| Parameter | Type | Description |
|---|---|---|
cluster_id (in the URL) |
string | Cluster the controller runs on. |
action |
string | remove_controller. |
vdag_controller_id |
string | The controller to remove. |
Scaling (scale_controller) and listing (list_controllers) are described in Cluster Controller: vDAG controller actions.
Inference gRPC API¶
Clients send tasks to the controller with this gRPC service:
// File metadata and content
message vDAGFileInfo {
string metadata = 1; // Metadata describing the file (e.g., filename, content type, etc.)
bytes file_data = 2; // Actual binary data of the file
}
// Used for both vDAG requests and responses
message vDAGInferencePacket {
string session_id = 3; // Unique identifier for the inference session
uint64 seq_no = 4; // Sequence number for ordering messages in the session
bytes frame_ptr = 5; // Pointer to frame data stored in the frame DB
string data = 6; // Input JSON serialized as a string; contains the output in a response
double ts = 8; // Timestamp of the packet (typically epoch time)
repeated vDAGFileInfo files = 9; // Files associated with the packet
}
// gRPC service for vDAG inference
service vDAGInferenceService {
// Runs inference on the packet and returns a response packet
rpc infer(vDAGInferencePacket) returns (vDAGInferencePacket);
}
| Field | Type | Description |
|---|---|---|
session_id |
string |
Identifies the inference session. Quotas can be applied per session. |
seq_no |
uint64 |
Keeps packets in order within a session. |
frame_ptr |
bytes (JSON) |
Optional. For large inputs stored in FrameDB: {"framedb_id": ..., "key": ...}. |
data |
string |
Input as a JSON string. In the response, it holds the output. |
ts |
double |
Timestamp, typically UNIX epoch time. |
files |
repeated vDAGFileInfo |
Attached files, each with metadata and binary data. |
Python example¶
import grpc
import json
import time
# Import the generated gRPC classes
import vdag_pb2
import vdag_pb2_grpc
# Create a gRPC channel to the vDAG controller
channel = grpc.insecure_channel("<vdag-controller-public-url>")
stub = vdag_pb2_grpc.vDAGInferenceServiceStub(channel)
session_id = "session-5678"
seq_no = 1
# Area of interest (bounding box coordinates)
area_of_interest = {
"region_of_interest": {
"x_min": 120,
"y_min": 200,
"x_max": 450,
"y_max": 650
}
}
# Load the image to send
with open("example_image.jpg", "rb") as img_file:
image_bytes = img_file.read()
image_file = vdag_pb2.vDAGFileInfo(
metadata=json.dumps({
"filename": "example_image.jpg",
"content_type": "image/jpeg"
}),
file_data=image_bytes
)
packet = vdag_pb2.vDAGInferencePacket(
session_id=session_id, # Unique session ID
seq_no=seq_no, # Sequence number for packet ordering
data=json.dumps(area_of_interest), # Input as JSON
ts=time.time(), # Timestamp
files=[image_file] # Attached files
)
# Send the task
response = stub.infer(packet)
print("Response received.")
print("Data:", response.data)
# Save any returned files
for f in response.files:
metadata = json.loads(f.metadata)
print(f"Returned file: {metadata.get('filename')} (type: {metadata.get('content_type')})")
with open(f"output_{metadata.get('filename')}", "wb") as out_file:
out_file.write(f.file_data)
To find a controller's address, see Inference user guide: find the vDAG's controller.
Policies¶
The controller uses three optional policies. Each one follows the standard policy shape (__init__, eval, management) described in Policies.
| Policy | When it runs | Receives | Returns |
|---|---|---|---|
| Quota checker | On every task, before processing | The quota table, the task, the session's new quota count and the session ID | {"allowed": true/false} |
| Quality checker | In the background, on sampled tasks | vDAG info, plus the request and response | Ignored |
| Health checker | Periodically, at the interval set in the vDAG configuration | vDAG info, plus health data for every block | Ignored (custom data) |
Quota checker policy¶
Lets the vDAG owner limit usage of the vDAG, globally or per session_id. It runs every time a task is submitted.
Quota counts are kept in Redis and managed through a QuotaManagement object, which is passed to the policy:
QuotaManagement methods
class QuotaManagement:
def __init__(self, redis_host="<server-url>", redis_port=6379, redis_db=0, quota_ttl=None):
"""
Connects to Redis (and tests the connection). An optional TTL makes quota
counts expire.
"""
def increment(self, session_id: str, amount: int = 1) -> int:
"""
Increments the session's quota count by `amount` and returns the new value.
Refreshes the expiry time if a TTL is set.
"""
def get(self, session_id: str) -> int:
"""Returns the session's current quota count, or 0 if not found."""
def reset(self, session_id: str) -> None:
"""Resets the session's quota count to zero."""
def clean(self) -> None:
"""Clears all quotas from Redis."""
def remove(self, session_id: str) -> None:
"""Deletes the session's quota entry."""
def exists(self, session_id: str) -> bool:
"""Returns True if a quota entry exists for the session."""
Input:
{
// QuotaManagement object, for reading and changing the quota table
"quota_table": <object of QuotaManagement>,
// the task input
"input": <object of vDAGInferencePacket - task input>,
// the session's quota count including this request (current + 1)
"quota": <number (int) representing the updated quota, i.e current_quota + 1>,
// session_id of the task
"session_id": <string - session_id>
}
Output: {"allowed": true} to accept the task, or {"allowed": false} to reject it.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
input_data = {
"quota_table": <QuotaManagement object>,
"input": <vDAGInferencePacket>,
"quota": <current quota + 1>,
"session_id": <session_id>
}
"""
# Example: allow at most 1000 requests per session
# return {"allowed": input_data["quota"] <= 1000}
return {
"allowed": True
}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
"""
# Implement custom management actions here
return {}
Quota APIs (on the controller's management server):
| Endpoint | Method | What it does |
|---|---|---|
/quota/<session_id> |
GET |
Current quota value for a session. |
/quota/reset/<session_id> |
POST |
Resets a session's quota. |
/quota/exists/<session_id> |
GET |
Whether a quota record exists for a session. |
/quota/<session_id> |
DELETE |
Removes a session's quota record. |
/quota/mgmt |
POST |
Sends a management command to the quota policy. |
curl -X GET http://<server-url>/quota/session-1234
curl -X POST http://<server-url>/quota/reset/session-1234
curl -X GET http://<server-url>/quota/exists/session-1234
curl -X DELETE http://<server-url>/quota/session-1234
curl -X POST http://<server-url>/quota/mgmt \
-H "Content-Type: application/json" \
-d '{
"mgmt_action": "",
"mgmt_data": {}
}'
Quality checker policy¶
Audits the vDAG's output quality by looking at samples of requests and their responses. The policy can judge quality itself, or act as a client of an external QA system where samples are stored and reviewed manually.
It runs in the background, so it never slows down requests.
Input:
{
// vDAG information, as stored in the registry
"vdag_info": <vdag information>,
"input_data": {
// the task request
"request": <request data - object of vDAGInferencePacket class>,
// the task response
"response": <response - object of vDAGInferencePacket class>
}
}
Output: ignored.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
input_data = {
"vdag_info": <vdag information>,
"input_data": {
"request": <vDAGInferencePacket>,
"response": <vDAGInferencePacket>
}
}
"""
# The return value is ignored
return {}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
"""
# Implement custom management actions here
return {}
Management API:
curl -X POST http://<server-url>/quality-check/mgmt \
-H "Content-Type: application/json" \
-d '{
"mgmt_action": "",
"mgmt_data": {}
}'
Health checker¶
Runs periodically, at the interval set in the vDAG configuration, on fresh health data from every block in the vDAG.
Input:
{
// vDAG information, as stored in the registry
"vdag": <vDAG information>,
"health_check_data": {
// a healthy block
<block-id>: {
"success": true,
"data": {"instances": [list of running instances of that block]}
},
// an unhealthy block
<block-id>: {
"success": false,
"data": {
// why it is unhealthy (see the table below)
"mode": <unhealthy-reason-string>,
// a message (can be ignored)
"data": <message - message string (can be ignored)>
}
}
}
}
mode |
Meaning |
|---|---|
api_internal_error |
The block's health API answered with status 200 but reported failure, for example "success": False. Usually an internal error in the block. |
network_error |
The check failed because of persistent network problems (timeouts, request exceptions) after all retries. |
general_error |
An unexpected error outside the request itself, for example a bad URL or configuration. |
timeout_error |
A request timed out. Used during retries, and may become network_error if all retries fail. |
Output: any custom data. The controller does not process it.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
input_data = {
"vdag": <vDAG information>,
"health_check_data": {
<block-id>: {"success": True, "data": {"instances": [...]}},
<block-id>: {"success": False, "data": {"mode": "<reason>", "data": "<message>"}}
}
}
"""
# Return custom data (not processed by the vDAG controller)
return {}
def management(self, action: str, data: dict) -> dict:
"""
Executes a custom management command, for example to update settings,
read internal state, or run diagnostics.
"""
# Implement custom management actions here
return {}
Health APIs:
| Endpoint | Method | What it does |
|---|---|---|
/health/check |
GET |
Runs a health check on every block in the vDAG now, evaluates it with the policy, and returns the result. |
/health/mgmt |
POST |
Sends a management command to the health policy. |
curl -X GET http://<server-url>/health/check
curl -X POST http://<server-url>/health/mgmt \
-H "Content-Type: application/json" \
-d '{
"mgmt_action": "",
"mgmt_data": {}
}'
Metrics¶
Each vDAG controller reports these metrics at fixed intervals to the network's global vDAG metrics database:
| Metric | Description |
|---|---|
inference_requests_total |
Total inference requests processed |
inference_fps |
Inference throughput, in frames per second |
inference_latency_seconds |
Latency per request, in seconds |
Query them from the global vDAG metrics database (global-vdag-metrics on the management cluster, NodePort 30203):
| Endpoint | Method | What it does |
|---|---|---|
/vdag/<vdag_id> |
GET |
Metrics for one vDAG. |
/vdag/query |
POST |
Search metrics with a MongoDB-style filter ($eq, $gt, $in, ...). |
curl -X GET http://<server-url>/vdag/<vdag-id>
curl -X POST http://<server-url>/vdag/query \
-H "Content-Type: application/json" \
-d '{
"status": { "$eq": "healthy" },
"metrics.latency": { "$lt": 200 }
}'
See also Metrics system: vDAG metrics.