Cluster Controller¶
The Cluster Controller is the set of services that manages a single cluster. It is installed automatically on every cluster that joins a network. It creates and removes blocks, places their instances on nodes, watches node and pod health, recovers from failures, and enforces the cluster's own rules.
At a glance
Runs on: every cluster in the network (one controller per cluster).
Talks to: the network's Cluster Controller Gateway, which sends it most requests.
You'll use this page to: write cluster pre-check, stability and event policies, and look up the controller's APIs.
What it does¶
- Creates and removes the infrastructure for blocks in the cluster.
- Relays administrative and management commands to blocks.
- Allocates compute and storage for new instances when a block scales.
- Works with the Cluster Controller Gateway when a block is first placed.
- Creates, scales and removes vDAG controllers.
- Adds and removes nodes.
- Continuously monitors nodes and pods, and moves workloads off failed nodes.
- Runs pre-check policies before infrastructure changes, such as creating a block or adding a node.
Architecture¶

(Download the architecture diagram)
| Module | What it does |
|---|---|
| Blocks data API | Create, read, update and delete operations on the Blocks database, the record of every block in the cluster. Used by the other modules. |
| Block events processor | Runs the policies attached to events that blocks raise. See Block events. |
| Dry-run and resource allocator | Runs a block's resource allocation policy to choose nodes and GPUs, for initial placement, scaling and (in dry-run mode) feasibility checks. See Resource allocation policy. |
| Infrastructure manager | Talks to the Kubernetes API to create, change and remove resources for blocks and vDAG controllers. Runs pre-check policies before acting. |
| Dynamic infrastructure scanner | Periodically checks every block instance pod. When it finds a problem, it tells the infrastructure manager, which can reschedule the pods onto healthy nodes. |
| Source controller | Schedules and manages GStreamer-based video ingestion streams for inference workloads. |
| Management command executor | Routes management commands to blocks or their services, after checking them against policies. |
| Stability checker | Runs the cluster's stability policy on data from the infrastructure scanner. It can call external webhooks or notification systems when something is wrong. See Stability checker policy. |
Pre-check policies¶
Pre-check policies are Python rules that run before an action and decide whether it may proceed. They let a cluster admin enforce rules specific to this cluster. These rules apply in addition to the network's rules, which are enforced by the Cluster Controller Gateway.
| Action | Checked when... |
|---|---|
add_node |
a node asks to join the cluster |
remove_node |
a node is to be removed |
update_cluster |
cluster configuration or metadata changes |
scale_block |
a block's instances are scaled |
add_block |
a new block is to be created |
remove_block |
a block is to be removed |
add_vdag_controller |
a vDAG controller is to be created |
remove_vdag_controller |
a vDAG controller is to be removed |
block_mgmt |
a management command is sent to a block |
cluster_mgmt |
a cluster management command is run |
Writing a pre-check policy¶
A pre-check policy returns a dictionary with two keys:
| Key | Meaning |
|---|---|
allowed |
True to let the action proceed, False to stop it. |
input_data |
If allowed: the request data, either unchanged or modified by your policy. If not allowed: a message or dictionary explaining why. |
# Allowed (input_data may be modified)
{
"allowed": True,
"input_data": input_data
}
# Not allowed
{
"allowed": False,
"input_data": <message or dict explaining why the action was not allowed>
}
Policy template:
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated.
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
"""
# The input_data dict can be modified by the policy.
# Make input_data modifications here.
return {
"allowed": True,
"input_data": input_data
}
After writing a policy, register it with the Policies System and attach it as described in the Cluster admin guide.
Stability checker policy¶
The stability checker periodically decides whether the cluster as a whole is stable. It runs your stability policy on two inputs: the cluster's registry entry, and the latest health data for each node from the infrastructure scanner.
Input:
{
"cluster_data": <data-of-the-cluster>, // cluster entry from the cluster registry
"nodes_data": <metrics data of each node>
}
Full input example
{
"cluster_data": {
"id": "cluster-west-vision-001",
"regionId": "us-west-2",
"status": "live",
"nodes": {
"count": 2,
"nodeData": [
{
"id": "node-1",
"gpus": {
"count": 2,
"memory": 32768,
"gpus": [
{ "modelName": "NVIDIA A100", "memory": 16384 },
{ "modelName": "NVIDIA A100", "memory": 16384 }
],
"features": ["fp16", "tensor_cores"],
"modelNames": ["NVIDIA A100"]
},
"vcpus": { "count": 32 },
"memory": 131072,
"swap": 8192,
"storage": {
"disks": 2,
"size": 1048576
},
"network": {
"interfaces": 2,
"txBandwidth": 10000,
"rxBandwidth": 10000
}
},
{
"id": "node-2",
"gpus": {
"count": 1,
"memory": 16384,
"gpus": [
{ "modelName": "NVIDIA V100", "memory": 16384 }
],
"features": ["fp16"],
"modelNames": ["NVIDIA V100"]
},
"vcpus": { "count": 16 },
"memory": 65536,
"swap": 4096,
"storage": {
"disks": 1,
"size": 524288
},
"network": {
"interfaces": 1,
"txBandwidth": 5000,
"rxBandwidth": 5000
}
}
]
},
"gpus": {
"count": 3,
"memory": 49152
},
"vcpus": {
"count": 48
},
"memory": 196608,
"swap": 12288,
"storage": {
"disks": 3,
"size": 1572864
},
"network": {
"interfaces": 3,
"txBandwidth": 15000,
"rxBandwidth": 15000
},
"config": {
"policyExecutorId": "policy-exec-007",
"policyExecutionMode": "local",
"customPolicySystem": {
"name": "AdvancedPolicyRunner",
"version": "2.1.0"
},
"publicHostname": "cluster-west-vision-001.company.net",
"useGateway": true,
"actionsPolicyMap": {
"onScaleUp": "evaluate-gpu-availability",
"onFailure": "notify-admin"
},
// these fields are populated by the system:
"urlMap": {
"controllerService": "http://cluster-west-vision-001.company.net:32000/controller",
"metricsService": "http://cluster-west-vision-001.company.net:32000/metrics",
"blocksQuery": "http://cluster-west-vision-001.company.net:32000/blocks",
"publicGateway": "http://cluster-west-vision-001.company.net:32000",
"parameterUpdater": "http://cluster-west-vision-001.company.net:32000/mgmt"
}
},
"tags": ["gpu", "production", "ml", "vision", "us-west"],
"clusterMetadata": {
"name": "Sample cluster",
"description": "Dedicated to serving large-scale computer vision models in production.",
"owner": "AI Infrastructure Team",
"email": "ai-infra@company.net",
"countries": ["USA", "Canada"],
"miscContactInfo": {
"pagerDuty": "https://sample-website/ai-clusters",
"slack": "#ml-infra"
},
"additionalInfo": {
}
},
"reputation": 94
},
"nodes_data": [
{
"node_id": "node-1",
"status": "healthy",
"cpu_utilization_percent": 35.0,
"memory": {
"total": 16384,
"allocatable": 10240,
"usage_percent": 37.5
},
"disk": {
"total": 512000,
"allocatable": 409600,
"usage_percent": 20.0
}
},
{
"node_id": "node-2",
"status": "unhealthy",
"cpu_utilization_percent": 25.0,
"memory": {
"total": 8192,
"allocatable": 6144,
"usage_percent": 25.0
},
"disk": {
"total": 256000,
"allocatable": 204800,
"usage_percent": 20.0
}
}
]
}
Output: your policy returns is_stable, plus any data you want to record.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
'''
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
'''
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
'''
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated.
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
input_data = {
"cluster_data": <cluster-data>,
"nodes_data": <nodes-data>
}
'''
# handle the logic here
return {
"is_stable": True,
# custom data of the policy
"stability_check_data": {}
}
Resource allocation policy¶
A block's resource allocation policy decides which node and GPUs each of its instances uses. The block's specification names the policy. The controller's dry-run and resource allocator module runs it whenever a block is created, scaled or has an instance moved. See also the Block spec.
Your policy must handle four actions:
| Action | When it runs | Must return |
|---|---|---|
dry_run |
While the network is choosing a cluster. Simulates placement on this cluster. | A feasibility score from 0.0 to 1.0, plus the node and GPUs it would use. |
allocation |
When a new block is created on this cluster. | The node_id and gpus for the instance. |
scale |
When a new instance is added to an existing block. | The node_id and gpus for the new instance. |
reassignment |
When an existing instance must move, for example after a node failure. | The new node_id and gpus. |
Input always has this shape:
{
"action": "dry_run", // or "allocation", "scale", "reassignment"
"payload": {} // depends on the action; see below
}
Payload:
{
// data of the block that is being created
"block": <complete-block-entry>,
// data of the current cluster
"cluster": <complete-cluster-entry>,
// metrics of the current cluster at that point in time
"cluster_metrics": <metrics of the cluster>,
// list of nodes that are healthy in the current cluster at that point in time
"healthy_nodes": <list of node IDs that are healthy>
}
Return:
Payload:
{
// data of the block that is being created
"block": <complete-block-entry>,
// data of the current cluster
"cluster": <complete-cluster-entry>,
// metrics of the current cluster at that point in time
"cluster_metrics": <metrics of the cluster>,
// list of nodes that are healthy in the current cluster at that point in time
"healthy_nodes": <list of node IDs that are healthy>
}
Return:
Payload:
{
// data of the block being scaled
"block": <complete-block-entry>,
// data of the current cluster
"cluster": <complete-cluster-entry>,
// metrics of the current cluster at that point in time
"cluster_metrics": <metrics of the cluster>,
// metrics of the block for which the new instance is being created
"block_metrics": <metrics of the block>,
// list of nodes that are healthy in the current cluster at that point in time
"healthy_nodes": <list of node IDs that are healthy>
}
Return:
Payload:
{
// data of the block whose instance is being moved
"block": <complete-block-entry>,
// data of the current cluster
"cluster": <complete-cluster-entry>,
// metrics of the current cluster at that point in time
"cluster_metrics": <metrics of the cluster>,
// metrics of the block
"block_metrics": <metrics of the block>,
// list of nodes that are healthy in the current cluster at that point in time
"healthy_nodes": <list of node IDs that are healthy>,
// ID of the instance being reassigned
"instance_id": <instance-id>,
// name of the instance pod being reassigned
"pod_name": <name of the pod being re-assigned>
}
Return:
Policy template:
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
'''
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
'''
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
'''
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated.
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
input_data = {
"action": "dry_run" / "allocation" / "scale" / "reassignment",
"payload": <as described above for each action>
}
'''
# handle the logic here
if input_data['action'] in ["allocation", "scale", "reassignment"]:
return {
# node_id where the instance will be scheduled
"node_id": <node_id>,
# gpu_ids that will be assigned to the instance
"gpus": [....]
}
elif input_data['action'] == "dry_run":
return {
"selection_score_data": {
# feasibility score between 0 and 1
"score": 0.9,
"node_info": {
# node_id where the instance would be scheduled
"node_id": <node_id>,
# gpu_ids that would be assigned to the instance
"gpus": [....]
}
}
}
Block events processor¶
Blocks can raise custom events, such as "processing finished" or "uncaught exception", and attach a policy to each one. The block events processor runs inside the Cluster Controller pod and executes the matching policy for each event.
Events travel from the block to the processor through a Redis queue and are handled one at a time. The block does not wait for the event to be processed.
Typical uses:
- Notify users when a condition is met.
- Report an internal failure, and have the event policy trigger a failure-handling policy.
- Forward an event to an external system for further processing.
1. Write the event policy¶
The policy receives:
{
"block_id": <id of the block that raised the event>,
"event_name": <name of the event>,
"event_data": <dict containing the event data>
}
Its return value is ignored.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
'''
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
'''
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
'''
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated.
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
input_data = {
"block_id": <id of the block that raised the event>,
"event_name": <name of the event>,
"event_data": <dict containing the event data>
}
'''
# handle the event here
# the return value is ignored
2. Register the event on the block¶
Map each event name to a policy rule URI in the block's blockInitData.events field:
{
"blockInitData": {
"events": {
"send-processed-notification": "policies.event-notifications.custom-notifier:v0.0.1-stable",
"uncaught-exception": "policies.exception-handlers.custom-exception-handler:v0.0.1-stable"
}
}
}
3. Raise the event from your instance code¶
Use the BlockEvents class from the Instance SDK:
from aios_instance.events import BlockEvents
# Create a BlockEvents client
block_events = BlockEvents()
# Push an event
block_events.push_event("block_started", {"step": 1, "message": "Execution started"})
API reference¶
Base URL
<server-url> below is the cluster's controller address. Through the cluster gateway it is http://<cluster-public-host>:32000/controller; it is also exposed on NodePort 32300. See What gets exposed.
Block actions¶
Internal API
These actions are normally called by the Cluster Controller Gateway, a block's auto-scaler or the controller's own modules. Don't call them directly; use the Cluster Controller Gateway instead.
POST /executeAction: runs an action on blocks in this cluster.
curl -X POST http://<server-url>/executeAction \
-H "Content-Type: application/json" \
-d '{
"action": "<your-action>",
"data": {}
}'
| Action | What it does | Called by |
|---|---|---|
create_block |
Creates a new block with the given configuration. | Cluster Controller Gateway |
remove_block |
Removes a block. | Cluster Controller Gateway |
parameter_update |
Updates the parameters of an existing block. | Management command executor |
scale |
Adds or removes instances. | Block auto-scaler, Cluster Controller Gateway |
dry_run |
Simulates an allocation without performing it. | Cluster Controller Gateway |
remove_instance |
Removes one instance. | Cluster Controller Gateway |
init_create_status_update |
Updates status while an init container runs. See Third-party systems. | Cluster Controller Gateway |
query_init_container_data |
Returns status or data from an init container. See Third-party systems. | Cluster Controller Gateway |
reassign-instances |
Moves instances, for example away from a failed node. | Dynamic infrastructure scanner |
vDAG controller actions¶
POST /vdag-controller: creates, scales, lists or removes vDAG controllers in this cluster. cluster_id and public_url are filled in automatically when a controller is created.
curl -X POST http://<server-url>/vdag-controller \
-H "Content-Type: application/json" \
-d '{
"action": "create_controller",
"payload": {
"vdag_controller_id": "vdag-controller-123",
"vdag_uri": "<vdag-uri>",
"config": {
"policy_execution_mode": "local",
"replicas": 2
},
"search_tags": []
}
}'
| Action | What it does | Example payload |
|---|---|---|
create_controller |
Creates a vDAG controller. | { "vdag_controller_id": "<id>", "vdag_uri": "<uri>", "config": { "policy_execution_mode": "local", "replicas": 2 }, "search_tags": [] } |
remove_controller |
Removes a controller and deletes its registry entry. | { "vdag_controller_id": "<id>" } |
scale_controller |
Changes the number of replicas. | { "vdag_controller_id": "<id>", "replicas": 3 } |
list_controllers |
Lists all controllers. | {} |
These are normally called through the Cluster Controller Gateway. See the vDAG creator guide.
Cluster actions¶
Internal API
Used internally through the Cluster Controller Gateway.
| Endpoint | Method | What it does |
|---|---|---|
/cluster-actions/add-node |
POST |
Adds a node, using the node data in the body. |
/cluster-actions/remove-node/<node_id> |
DELETE |
Removes a node. |
/cluster-actions/get-cluster |
GET |
Returns the cluster's configuration and nodes. |
/cluster-actions/update-cluster |
PATCH |
Updates the cluster's configuration. |
# Add a node
curl -X POST http://<server-url>/cluster-actions/add-node \
-H "Content-Type: application/json" \
-d '{}'
# Remove a node
curl -X DELETE http://<server-url>/cluster-actions/remove-node/<node-id>
# Get cluster data
curl -X GET http://<server-url>/cluster-actions/get-cluster
# Update cluster data (MongoDB-style update)
curl -X PATCH http://<server-url>/cluster-actions/update-cluster \
-H "Content-Type: application/json" \
-d '{
"$set": {
"tags": ["gpu", "updated"],
"reputation": 97
}
}'
Query blocks¶
| Endpoint | Method | What it does |
|---|---|---|
/blocks/<block_id> |
GET |
Returns one block. |
/blocks |
GET |
Returns all blocks in the cluster. |
/blocks/query |
POST |
Filters blocks with a MongoDB-style query ($in, $and, $or, ...). |
curl -X GET http://<server-url>/blocks/<block-id>
curl -X GET http://<server-url>/blocks
curl -X POST http://<server-url>/blocks/query \
-H "Content-Type: application/json" \
-d '{
"query": {
"cluster.reputation": { "$gt": 90 },
"policies.autoscaler.policyRuleURI": { "$ne": "" }
},
"options": {
"sort": { "id": 1 },
"limit": 10
}
}'
Update a block¶
PUT /block/update/<block_id>: updates a block's configuration or metadata using a MongoDB-style update ($set, $unset, ...).
curl -X PUT http://<server-url>/block/update/<block-id> \
-H "Content-Type: application/json" \
-d '{
"$set": {
"blockMetadata.description": "Updated description for object detection block",
"minInstances": 2
}
}'
Node health (infrastructure scanner)¶
| Endpoint | Method | What it returns |
|---|---|---|
/healthy_nodes |
GET |
Nodes that are currently healthy. |
/nodes_status |
GET |
Health status of every node, healthy or not. |
Stability checker¶
| Endpoint | Method | What it does |
|---|---|---|
/check_stability |
GET |
Runs a cluster-wide stability check now, using the stability policy. |
/mgmt |
POST |
Sends a management command to the stability policy. |
curl -X GET http://<server-url>/check_stability
curl -X POST http://<server-url>/mgmt \
-H "Content-Type: application/json" \
-d '{
"mgmt_action": "<action>",
"mgmt_data": {}
}'
Management commands¶
POST /mgmt (management command executor) sends a management command to one of a block's services, or to the cluster's stability checker.
- With a
block_id: the command goes to the namedserviceof that block. - With an empty
block_idandserviceset tostability_checker: the command goes to the cluster's stability checker.
curl -X POST http://<server-url>/mgmt \
-H "Content-Type: application/json" \
-d '{
"block_id": "",
"service": "stability_checker",
"mgmt_action": "<action>",
"mgmt_data": {}
}'
service |
Where the command goes |
|---|---|
instances |
All of the block's instance pods, each by its own IP address. |
executor |
The block's executor (load balancer). |
autoscaler |
The block's auto-scaler. |
health |
The block's health checker. |
stability_checker |
The cluster's stability checker (leave block_id empty). |
See also Management commands.