Skip to content

vDAG controller

A vDAG controller is the front door of a vDAG workflow. Clients send tasks to it, it runs them through every block in the graph, and it returns the final result. Along the way it enforces the workflow owner's quotas, watches the health of every block involved, samples outputs for quality checks, and reports metrics.

At a glance

Who this is for: vDAG creators, and developers calling vDAGs.
You'll learn: how the controller works, how to create one, how to call it, and how to write its three policies.
Related: the vDAG creator guide for the end-to-end workflow, and the vDAG spec for defining the graph.

What it does

  1. Gateway. Accepts tasks for the vDAG, runs them, and returns the results.
  2. Quotas and rate limits. Lets the vDAG owner limit usage, per session or globally, with a quota checker policy.
  3. Health checks. Periodically checks every block in the vDAG, across the network, with a health checker policy.
  4. Quality checks. Periodically compares outputs with inputs, using a quality checker policy. It can forward samples to an external QA system or for manual review.
  5. Management APIs. Lets you send management commands to the three policies.
  6. Metrics. Reports latency, throughput and more.

Architecture

vDAG controller architecture

(Download the architecture diagram)

What happens to a task

sequenceDiagram
    participant C as Client
    participant R as RequestsProcessor
    participant Q as QuotaChecker
    participant P as PreProcessor
    participant S as Ad-hoc inference server
    participant B as Blocks in the vDAG
    participant QC as QualityCheckProcessor
    C->>R: 1. Submit task (gRPC)
    R->>Q: 2. Check quota (policy)
    Q-->>R: allowed / rejected
    R-->>C: 3. Rejected, if over quota
    R->>P: 4. Convert to internal format, build runtime graph
    P->>S: 5. Submit graph task
    S->>B: run each step
    S-->>P: 6–8. Final output (via Redis queue), converted back
    P->>QC: 9–10. Queue a quality-check sample (async)
    P-->>R: 11. Output
    R-->>C: 12. Response (same gRPC call)
  1. A client submits a task through the controller's gRPC API.
  2. The RequestsProcessor receives it and runs the quota checker policy, which can enforce per-session or global limits using Redis-backed counters.
  3. If the quota check fails, the client gets an immediate rejection. Otherwise the task continues.
  4. The PreProcessor converts the input into the format used by the ad-hoc inference server. It works out the entry block_id and turns the vDAG's static graph into a runtime graph.
  5. The task is sent to the ad-hoc inference server through its gRPC client.
  6. A Redis queue listener waits for the result, which the inference server pushes to a queue when it finishes.
  7. The listener picks up the result.
  8. The PreProcessor converts it back into a vDAG response.
  9. The response goes to the QualityCheckProcessor.
  10. The QualityCheckProcessor queues the input and output as a quality-check sample. A background worker evaluates samples with the quality policy, without slowing down the request.
  11. The output returns to the RequestsProcessor.
  12. The response is sent back to the client over the same gRPC connection.

Modules

Module What it does
Metrics server Collects latency, throughput and FPS from the RequestsProcessor, and periodically writes them to the global vDAG metrics database.
Quota checker Runs the quota policy, backed by a Redis QuotaTable that tracks usage per session or globally.
Quality checker A background thread that reads samples from a queue and runs the quality policy on them.
Health checker Periodically fetches the health of every block in the vDAG through the block management APIs, and runs the health policy on the results.
vDAG DB client Reads the vDAG's structure, configuration and version from the vDAG registry.
Controller and API server REST APIs for managing metrics, health checks, quotas, quality checks and other controller operations.

Create a controller

A controller serves one vDAG, which must already exist in the registry. You can deploy it on any cluster in the network, through the Cluster Controller Gateway:

POST /vdag-controller/<cluster_id>: forwards the request to the /vdag-controller endpoint of the chosen cluster.

curl -X POST http://<cluster-controller-gateway-url>/vdag-controller/<cluster-id> \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create_controller",
    "payload": {
      "vdag_controller_id": "<controller-id>",
      "vdag_uri": "<vdag-uri>",
      "config": {
        "policy_execution_mode": "local",
        "replicas": 2
      }
    }
  }'
Parameter Type Description
cluster_id (in the URL) string Cluster to deploy the controller on.
action string create_controller.
vdag_controller_id string A unique ID for the controller.
vdag_uri string The vDAG this controller serves.
config.policy_execution_mode string Where the controller runs its policies, for example local.
config.replicas integer Number of controller replicas.

Once created, the controller is reachable at:

http://<cluster-public-url>/<vdag_controller_id>

and its management server at:

http://<cluster-public-url>/<vdag_controller_id>/server

For example, controller vdag-123 on cluster us-west-1.aiosv1.cluster.net:

http://us-west-1.aiosv1.cluster.net/vdag-123          # gateway
http://us-west-1.aiosv1.cluster.net/vdag-123/server   # management server

Remove a controller

curl -X POST http://<cluster-controller-gateway-url>/vdag-controller/<cluster_id> \
  -H "Content-Type: application/json" \
  -d '{
    "action": "remove_controller",
    "payload": {
      "vdag_controller_id": "<controller-id>"
    }
  }'
Parameter Type Description
cluster_id (in the URL) string Cluster the controller runs on.
action string remove_controller.
vdag_controller_id string The controller to remove.

Scaling (scale_controller) and listing (list_controllers) are described in Cluster Controller: vDAG controller actions.


Inference gRPC API

Clients send tasks to the controller with this gRPC service:

// File metadata and content
message vDAGFileInfo {
    string metadata = 1;   // Metadata describing the file (e.g., filename, content type, etc.)
    bytes file_data = 2;   // Actual binary data of the file
}

// Used for both vDAG requests and responses
message vDAGInferencePacket {
    string session_id = 3;       // Unique identifier for the inference session
    uint64 seq_no = 4;           // Sequence number for ordering messages in the session
    bytes frame_ptr = 5;         // Pointer to frame data stored in the frame DB
    string data = 6;             // Input JSON serialized as a string; contains the output in a response
    double ts = 8;               // Timestamp of the packet (typically epoch time)
    repeated vDAGFileInfo files = 9; // Files associated with the packet
}

// gRPC service for vDAG inference
service vDAGInferenceService {
    // Runs inference on the packet and returns a response packet
    rpc infer(vDAGInferencePacket) returns (vDAGInferencePacket);
}
Field Type Description
session_id string Identifies the inference session. Quotas can be applied per session.
seq_no uint64 Keeps packets in order within a session.
frame_ptr bytes (JSON) Optional. For large inputs stored in FrameDB: {"framedb_id": ..., "key": ...}.
data string Input as a JSON string. In the response, it holds the output.
ts double Timestamp, typically UNIX epoch time.
files repeated vDAGFileInfo Attached files, each with metadata and binary data.

Python example

import grpc
import json
import time

# Import the generated gRPC classes
import vdag_pb2
import vdag_pb2_grpc

# Create a gRPC channel to the vDAG controller
channel = grpc.insecure_channel("<vdag-controller-public-url>")
stub = vdag_pb2_grpc.vDAGInferenceServiceStub(channel)

session_id = "session-5678"
seq_no = 1

# Area of interest (bounding box coordinates)
area_of_interest = {
    "region_of_interest": {
        "x_min": 120,
        "y_min": 200,
        "x_max": 450,
        "y_max": 650
    }
}

# Load the image to send
with open("example_image.jpg", "rb") as img_file:
    image_bytes = img_file.read()

image_file = vdag_pb2.vDAGFileInfo(
    metadata=json.dumps({
        "filename": "example_image.jpg",
        "content_type": "image/jpeg"
    }),
    file_data=image_bytes
)

packet = vdag_pb2.vDAGInferencePacket(
    session_id=session_id,              # Unique session ID
    seq_no=seq_no,                      # Sequence number for packet ordering
    data=json.dumps(area_of_interest),  # Input as JSON
    ts=time.time(),                     # Timestamp
    files=[image_file]                  # Attached files
)

# Send the task
response = stub.infer(packet)

print("Response received.")
print("Data:", response.data)

# Save any returned files
for f in response.files:
    metadata = json.loads(f.metadata)
    print(f"Returned file: {metadata.get('filename')} (type: {metadata.get('content_type')})")
    with open(f"output_{metadata.get('filename')}", "wb") as out_file:
        out_file.write(f.file_data)

To find a controller's address, see Inference user guide: find the vDAG's controller.


Policies

The controller uses three optional policies. Each one follows the standard policy shape (__init__, eval, management) described in Policies.

Policy When it runs Receives Returns
Quota checker On every task, before processing The quota table, the task, the session's new quota count and the session ID {"allowed": true/false}
Quality checker In the background, on sampled tasks vDAG info, plus the request and response Ignored
Health checker Periodically, at the interval set in the vDAG configuration vDAG info, plus health data for every block Ignored (custom data)

Quota checker policy

Lets the vDAG owner limit usage of the vDAG, globally or per session_id. It runs every time a task is submitted.

Quota counts are kept in Redis and managed through a QuotaManagement object, which is passed to the policy:

QuotaManagement methods
class QuotaManagement:
    def __init__(self, redis_host="<server-url>", redis_port=6379, redis_db=0, quota_ttl=None):
        """
        Connects to Redis (and tests the connection). An optional TTL makes quota
        counts expire.
        """

    def increment(self, session_id: str, amount: int = 1) -> int:
        """
        Increments the session's quota count by `amount` and returns the new value.
        Refreshes the expiry time if a TTL is set.
        """

    def get(self, session_id: str) -> int:
        """Returns the session's current quota count, or 0 if not found."""

    def reset(self, session_id: str) -> None:
        """Resets the session's quota count to zero."""

    def clean(self) -> None:
        """Clears all quotas from Redis."""

    def remove(self, session_id: str) -> None:
        """Deletes the session's quota entry."""

    def exists(self, session_id: str) -> bool:
        """Returns True if a quota entry exists for the session."""

Input:

{
    // QuotaManagement object, for reading and changing the quota table
    "quota_table": <object of QuotaManagement>,
    // the task input
    "input": <object of vDAGInferencePacket - task input>,
    // the session's quota count including this request (current + 1)
    "quota": <number (int) representing the updated quota, i.e current_quota + 1>,
    // session_id of the task
    "session_id": <string - session_id>
}

Output: {"allowed": true} to accept the task, or {"allowed": false} to reject it.

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        input_data = {
            "quota_table": <QuotaManagement object>,
            "input": <vDAGInferencePacket>,
            "quota": <current quota + 1>,
            "session_id": <session_id>
        }
        """
        # Example: allow at most 1000 requests per session
        # return {"allowed": input_data["quota"] <= 1000}

        return {
            "allowed": True
        }

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.
        """
        # Implement custom management actions here
        return {}

Quota APIs (on the controller's management server):

Endpoint Method What it does
/quota/<session_id> GET Current quota value for a session.
/quota/reset/<session_id> POST Resets a session's quota.
/quota/exists/<session_id> GET Whether a quota record exists for a session.
/quota/<session_id> DELETE Removes a session's quota record.
/quota/mgmt POST Sends a management command to the quota policy.
curl -X GET http://<server-url>/quota/session-1234
curl -X POST http://<server-url>/quota/reset/session-1234
curl -X GET http://<server-url>/quota/exists/session-1234
curl -X DELETE http://<server-url>/quota/session-1234

curl -X POST http://<server-url>/quota/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "",
    "mgmt_data": {}
  }'

Quality checker policy

Audits the vDAG's output quality by looking at samples of requests and their responses. The policy can judge quality itself, or act as a client of an external QA system where samples are stored and reviewed manually.

It runs in the background, so it never slows down requests.

Input:

{
    // vDAG information, as stored in the registry
    "vdag_info": <vdag information>,
    "input_data": {
        // the task request
        "request": <request data - object of vDAGInferencePacket class>,
        // the task response
        "response": <response - object of vDAGInferencePacket class>
    }
}

Output: ignored.

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        input_data = {
            "vdag_info": <vdag information>,
            "input_data": {
                "request": <vDAGInferencePacket>,
                "response": <vDAGInferencePacket>
            }
        }
        """
        # The return value is ignored
        return {}

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.
        """
        # Implement custom management actions here
        return {}

Management API:

curl -X POST http://<server-url>/quality-check/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "",
    "mgmt_data": {}
  }'

Health checker

Runs periodically, at the interval set in the vDAG configuration, on fresh health data from every block in the vDAG.

Input:

{
    // vDAG information, as stored in the registry
    "vdag": <vDAG information>,
    "health_check_data": {
        // a healthy block
        <block-id>: {
            "success": true,
            "data": {"instances": [list of running instances of that block]}
        },
        // an unhealthy block
        <block-id>: {
            "success": false,
            "data": {
                // why it is unhealthy (see the table below)
                "mode": <unhealthy-reason-string>,
                // a message (can be ignored)
                "data": <message - message string (can be ignored)>
            }
        }
    }
}
mode Meaning
api_internal_error The block's health API answered with status 200 but reported failure, for example "success": False. Usually an internal error in the block.
network_error The check failed because of persistent network problems (timeouts, request exceptions) after all retries.
general_error An unexpected error outside the request itself, for example a bad URL or configuration.
timeout_error A request timed out. Used during retries, and may become network_error if all retries fail.

Output: any custom data. The controller does not process it.

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        input_data = {
            "vdag": <vDAG information>,
            "health_check_data": {
                <block-id>: {"success": True, "data": {"instances": [...]}},
                <block-id>: {"success": False, "data": {"mode": "<reason>", "data": "<message>"}}
            }
        }
        """
        # Return custom data (not processed by the vDAG controller)
        return {}

    def management(self, action: str, data: dict) -> dict:
        """
        Executes a custom management command, for example to update settings,
        read internal state, or run diagnostics.
        """
        # Implement custom management actions here
        return {}

Health APIs:

Endpoint Method What it does
/health/check GET Runs a health check on every block in the vDAG now, evaluates it with the policy, and returns the result.
/health/mgmt POST Sends a management command to the health policy.
curl -X GET http://<server-url>/health/check

curl -X POST http://<server-url>/health/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "",
    "mgmt_data": {}
  }'

Metrics

Each vDAG controller reports these metrics at fixed intervals to the network's global vDAG metrics database:

Metric Description
inference_requests_total Total inference requests processed
inference_fps Inference throughput, in frames per second
inference_latency_seconds Latency per request, in seconds

Query them from the global vDAG metrics database (global-vdag-metrics on the management cluster, NodePort 30203):

Endpoint Method What it does
/vdag/<vdag_id> GET Metrics for one vDAG.
/vdag/query POST Search metrics with a MongoDB-style filter ($eq, $gt, $in, ...).
curl -X GET http://<server-url>/vdag/<vdag-id>

curl -X POST http://<server-url>/vdag/query \
  -H "Content-Type: application/json" \
  -d '{
    "status": { "$eq": "healthy" },
    "metrics.latency": { "$lt": 200 }
  }'

See also Metrics system: vDAG metrics.