Skip to content

Cluster Controller

The Cluster Controller is the set of services that manages a single cluster. It is installed automatically on every cluster that joins a network. It creates and removes blocks, places their instances on nodes, watches node and pod health, recovers from failures, and enforces the cluster's own rules.

At a glance

Runs on: every cluster in the network (one controller per cluster).
Talks to: the network's Cluster Controller Gateway, which sends it most requests.
You'll use this page to: write cluster pre-check, stability and event policies, and look up the controller's APIs.

What it does

  1. Creates and removes the infrastructure for blocks in the cluster.
  2. Relays administrative and management commands to blocks.
  3. Allocates compute and storage for new instances when a block scales.
  4. Works with the Cluster Controller Gateway when a block is first placed.
  5. Creates, scales and removes vDAG controllers.
  6. Adds and removes nodes.
  7. Continuously monitors nodes and pods, and moves workloads off failed nodes.
  8. Runs pre-check policies before infrastructure changes, such as creating a block or adding a node.

Architecture

Cluster Controller architecture

(Download the architecture diagram)

Module What it does
Blocks data API Create, read, update and delete operations on the Blocks database, the record of every block in the cluster. Used by the other modules.
Block events processor Runs the policies attached to events that blocks raise. See Block events.
Dry-run and resource allocator Runs a block's resource allocation policy to choose nodes and GPUs, for initial placement, scaling and (in dry-run mode) feasibility checks. See Resource allocation policy.
Infrastructure manager Talks to the Kubernetes API to create, change and remove resources for blocks and vDAG controllers. Runs pre-check policies before acting.
Dynamic infrastructure scanner Periodically checks every block instance pod. When it finds a problem, it tells the infrastructure manager, which can reschedule the pods onto healthy nodes.
Source controller Schedules and manages GStreamer-based video ingestion streams for inference workloads.
Management command executor Routes management commands to blocks or their services, after checking them against policies.
Stability checker Runs the cluster's stability policy on data from the infrastructure scanner. It can call external webhooks or notification systems when something is wrong. See Stability checker policy.

Pre-check policies

Pre-check policies are Python rules that run before an action and decide whether it may proceed. They let a cluster admin enforce rules specific to this cluster. These rules apply in addition to the network's rules, which are enforced by the Cluster Controller Gateway.

Action Checked when...
add_node a node asks to join the cluster
remove_node a node is to be removed
update_cluster cluster configuration or metadata changes
scale_block a block's instances are scaled
add_block a new block is to be created
remove_block a block is to be removed
add_vdag_controller a vDAG controller is to be created
remove_vdag_controller a vDAG controller is to be removed
block_mgmt a management command is sent to a block
cluster_mgmt a cluster management command is run

Writing a pre-check policy

A pre-check policy returns a dictionary with two keys:

Key Meaning
allowed True to let the action proceed, False to stop it.
input_data If allowed: the request data, either unchanged or modified by your policy. If not allowed: a message or dictionary explaining why.
# Allowed (input_data may be modified)
{
    "allowed": True,
    "input_data": input_data
}

# Not allowed
{
    "allowed": False,
    "input_data": <message or dict explaining why the action was not allowed>
}

Policy template:

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated.
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.
        """
        # The input_data dict can be modified by the policy.
        # Make input_data modifications here.

        return {
            "allowed": True,
            "input_data": input_data
        }

After writing a policy, register it with the Policies System and attach it as described in the Cluster admin guide.


Stability checker policy

The stability checker periodically decides whether the cluster as a whole is stable. It runs your stability policy on two inputs: the cluster's registry entry, and the latest health data for each node from the infrastructure scanner.

Input:

{
    "cluster_data": <data-of-the-cluster>, // cluster entry from the cluster registry
    "nodes_data": <metrics data of each node>
}
Full input example
{
    "cluster_data": {
        "id": "cluster-west-vision-001",
        "regionId": "us-west-2",
        "status": "live",
        "nodes": {
            "count": 2,
            "nodeData": [
            {
                "id": "node-1",
                "gpus": {
                "count": 2,
                "memory": 32768,
                "gpus": [
                    { "modelName": "NVIDIA A100", "memory": 16384 },
                    { "modelName": "NVIDIA A100", "memory": 16384 }
                ],
                "features": ["fp16", "tensor_cores"],
                "modelNames": ["NVIDIA A100"]
                },
                "vcpus": { "count": 32 },
                "memory": 131072,
                "swap": 8192,
                "storage": {
                "disks": 2,
                "size": 1048576
                },
                "network": {
                "interfaces": 2,
                "txBandwidth": 10000,
                "rxBandwidth": 10000
                }
            },
            {
                "id": "node-2",
                "gpus": {
                "count": 1,
                "memory": 16384,
                "gpus": [
                    { "modelName": "NVIDIA V100", "memory": 16384 }
                ],
                "features": ["fp16"],
                "modelNames": ["NVIDIA V100"]
                },
                "vcpus": { "count": 16 },
                "memory": 65536,
                "swap": 4096,
                "storage": {
                "disks": 1,
                "size": 524288
                },
                "network": {
                "interfaces": 1,
                "txBandwidth": 5000,
                "rxBandwidth": 5000
                }
            }
            ]
        },
        "gpus": {
            "count": 3,
            "memory": 49152
        },
        "vcpus": {
            "count": 48
        },
        "memory": 196608,
        "swap": 12288,
        "storage": {
            "disks": 3,
            "size": 1572864
        },
        "network": {
            "interfaces": 3,
            "txBandwidth": 15000,
            "rxBandwidth": 15000
        },
        "config": {
            "policyExecutorId": "policy-exec-007",
            "policyExecutionMode": "local",
            "customPolicySystem": {
            "name": "AdvancedPolicyRunner",
            "version": "2.1.0"
            },
            "publicHostname": "cluster-west-vision-001.company.net",
            "useGateway": true,
            "actionsPolicyMap": {
            "onScaleUp": "evaluate-gpu-availability",
            "onFailure": "notify-admin"
            },

            // these fields are populated by the system:
            "urlMap": {
            "controllerService": "http://cluster-west-vision-001.company.net:32000/controller",
            "metricsService": "http://cluster-west-vision-001.company.net:32000/metrics",
            "blocksQuery": "http://cluster-west-vision-001.company.net:32000/blocks",
            "publicGateway": "http://cluster-west-vision-001.company.net:32000",
            "parameterUpdater": "http://cluster-west-vision-001.company.net:32000/mgmt"
            }
        },
        "tags": ["gpu", "production", "ml", "vision", "us-west"],
        "clusterMetadata": {
            "name": "Sample cluster",
            "description": "Dedicated to serving large-scale computer vision models in production.",
            "owner": "AI Infrastructure Team",
            "email": "ai-infra@company.net",
            "countries": ["USA", "Canada"],
            "miscContactInfo": {
            "pagerDuty": "https://sample-website/ai-clusters",
            "slack": "#ml-infra"
            },
            "additionalInfo": {

            }
        },
        "reputation": 94
    },
    "nodes_data": [
        {
            "node_id": "node-1",
            "status": "healthy",
            "cpu_utilization_percent": 35.0,
            "memory": {
            "total": 16384,
            "allocatable": 10240,
            "usage_percent": 37.5
            },
            "disk": {
            "total": 512000,
            "allocatable": 409600,
            "usage_percent": 20.0
            }
        },
        {
            "node_id": "node-2",
            "status": "unhealthy",
            "cpu_utilization_percent": 25.0,
            "memory": {
            "total": 8192,
            "allocatable": 6144,
            "usage_percent": 25.0
            },
            "disk": {
            "total": 256000,
            "allocatable": 204800,
            "usage_percent": 20.0
            }
        }
    ]
}

Output: your policy returns is_stable, plus any data you want to record.

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        '''
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        '''
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        '''
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated.
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.

        input_data = {
            "cluster_data": <cluster-data>,
            "nodes_data": <nodes-data>
        }
        '''

        # handle the logic here

        return {
            "is_stable": True,
            # custom data of the policy
            "stability_check_data": {}
        }

Resource allocation policy

A block's resource allocation policy decides which node and GPUs each of its instances uses. The block's specification names the policy. The controller's dry-run and resource allocator module runs it whenever a block is created, scaled or has an instance moved. See also the Block spec.

Your policy must handle four actions:

Action When it runs Must return
dry_run While the network is choosing a cluster. Simulates placement on this cluster. A feasibility score from 0.0 to 1.0, plus the node and GPUs it would use.
allocation When a new block is created on this cluster. The node_id and gpus for the instance.
scale When a new instance is added to an existing block. The node_id and gpus for the new instance.
reassignment When an existing instance must move, for example after a node failure. The new node_id and gpus.

Input always has this shape:

{
    "action": "dry_run", // or "allocation", "scale", "reassignment"
    "payload": {}        // depends on the action; see below
}

Payload:

{
    // data of the block that is being created
    "block": <complete-block-entry>,
    // data of the current cluster
    "cluster": <complete-cluster-entry>,
    // metrics of the current cluster at that point in time
    "cluster_metrics": <metrics of the cluster>,
    // list of nodes that are healthy in the current cluster at that point in time
    "healthy_nodes": <list of node IDs that are healthy>
}

Return:

{
    "selection_score_data": {
        // feasibility score between 0 and 1: 1 = fully feasible, 0 = the block cannot be placed here
        "score": 0.9,
        "node_info": {
            // node_id where the instance would be scheduled
            "node_id": <node_id>,
            // gpu_ids that would be assigned to the instance
            "gpus": [....]
        }
    }
}

Payload:

{
    // data of the block that is being created
    "block": <complete-block-entry>,
    // data of the current cluster
    "cluster": <complete-cluster-entry>,
    // metrics of the current cluster at that point in time
    "cluster_metrics": <metrics of the cluster>,
    // list of nodes that are healthy in the current cluster at that point in time
    "healthy_nodes": <list of node IDs that are healthy>
}

Return:

{
    // node_id where the instance will be scheduled
    "node_id": <node_id>,
    // gpu_ids that will be assigned to the instance
    "gpus": [....]
}

Payload:

{
    // data of the block being scaled
    "block": <complete-block-entry>,
    // data of the current cluster
    "cluster": <complete-cluster-entry>,
    // metrics of the current cluster at that point in time
    "cluster_metrics": <metrics of the cluster>,
    // metrics of the block for which the new instance is being created
    "block_metrics": <metrics of the block>,
    // list of nodes that are healthy in the current cluster at that point in time
    "healthy_nodes": <list of node IDs that are healthy>
}

Return:

{
    // node_id where the instance will be scheduled
    "node_id": <node_id>,
    // gpu_ids that will be assigned to the instance
    "gpus": [....]
}

Payload:

{
    // data of the block whose instance is being moved
    "block": <complete-block-entry>,
    // data of the current cluster
    "cluster": <complete-cluster-entry>,
    // metrics of the current cluster at that point in time
    "cluster_metrics": <metrics of the cluster>,
    // metrics of the block
    "block_metrics": <metrics of the block>,
    // list of nodes that are healthy in the current cluster at that point in time
    "healthy_nodes": <list of node IDs that are healthy>,
    // ID of the instance being reassigned
    "instance_id": <instance-id>,
    // name of the instance pod being reassigned
    "pod_name": <name of the pod being re-assigned>
}

Return:

{
    // node_id where the instance will be scheduled
    "node_id": <node_id>,
    // gpu_ids that will be assigned to the instance
    "gpus": [....]
}

Policy template:

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        '''
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        '''
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        '''
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated.
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.

        input_data = {
            "action": "dry_run" / "allocation" / "scale" / "reassignment",
            "payload": <as described above for each action>
        }
        '''

        # handle the logic here

        if input_data['action'] in ["allocation", "scale", "reassignment"]:
            return {
                # node_id where the instance will be scheduled
                "node_id": <node_id>,
                # gpu_ids that will be assigned to the instance
                "gpus": [....]
            }
        elif input_data['action'] == "dry_run":
            return {
                "selection_score_data": {
                    # feasibility score between 0 and 1
                    "score": 0.9,
                    "node_info": {
                        # node_id where the instance would be scheduled
                        "node_id": <node_id>,
                        # gpu_ids that would be assigned to the instance
                        "gpus": [....]
                    }
                }
            }

Block events processor

Blocks can raise custom events, such as "processing finished" or "uncaught exception", and attach a policy to each one. The block events processor runs inside the Cluster Controller pod and executes the matching policy for each event.

Events travel from the block to the processor through a Redis queue and are handled one at a time. The block does not wait for the event to be processed.

Typical uses:

  1. Notify users when a condition is met.
  2. Report an internal failure, and have the event policy trigger a failure-handling policy.
  3. Forward an event to an external system for further processing.

1. Write the event policy

The policy receives:

{
    "block_id": <id of the block that raised the event>,
    "event_name": <name of the event>,
    "event_data": <dict containing the event data>
}

Its return value is ignored.

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        '''
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        '''
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        '''
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated.
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.

        input_data = {
            "block_id": <id of the block that raised the event>,
            "event_name": <name of the event>,
            "event_data": <dict containing the event data>
        }
        '''

        # handle the event here
        # the return value is ignored

2. Register the event on the block

Map each event name to a policy rule URI in the block's blockInitData.events field:

{
    "blockInitData": {
        "events": {
            "send-processed-notification": "policies.event-notifications.custom-notifier:v0.0.1-stable",
            "uncaught-exception": "policies.exception-handlers.custom-exception-handler:v0.0.1-stable"
        }
    }
}

3. Raise the event from your instance code

Use the BlockEvents class from the Instance SDK:

from aios_instance.events import BlockEvents

# Create a BlockEvents client
block_events = BlockEvents()

# Push an event
block_events.push_event("block_started", {"step": 1, "message": "Execution started"})

API reference

Base URL

<server-url> below is the cluster's controller address. Through the cluster gateway it is http://<cluster-public-host>:32000/controller; it is also exposed on NodePort 32300. See What gets exposed.

Block actions

Internal API

These actions are normally called by the Cluster Controller Gateway, a block's auto-scaler or the controller's own modules. Don't call them directly; use the Cluster Controller Gateway instead.

POST /executeAction: runs an action on blocks in this cluster.

curl -X POST http://<server-url>/executeAction \
  -H "Content-Type: application/json" \
  -d '{
    "action": "<your-action>",
    "data": {}
  }'
Action What it does Called by
create_block Creates a new block with the given configuration. Cluster Controller Gateway
remove_block Removes a block. Cluster Controller Gateway
parameter_update Updates the parameters of an existing block. Management command executor
scale Adds or removes instances. Block auto-scaler, Cluster Controller Gateway
dry_run Simulates an allocation without performing it. Cluster Controller Gateway
remove_instance Removes one instance. Cluster Controller Gateway
init_create_status_update Updates status while an init container runs. See Third-party systems. Cluster Controller Gateway
query_init_container_data Returns status or data from an init container. See Third-party systems. Cluster Controller Gateway
reassign-instances Moves instances, for example away from a failed node. Dynamic infrastructure scanner

vDAG controller actions

POST /vdag-controller: creates, scales, lists or removes vDAG controllers in this cluster. cluster_id and public_url are filled in automatically when a controller is created.

curl -X POST http://<server-url>/vdag-controller \
  -H "Content-Type: application/json" \
  -d '{
    "action": "create_controller",
    "payload": {
      "vdag_controller_id": "vdag-controller-123",
      "vdag_uri": "<vdag-uri>",
      "config": {
        "policy_execution_mode": "local",
        "replicas": 2
      },
      "search_tags": []
    }
  }'
Action What it does Example payload
create_controller Creates a vDAG controller. { "vdag_controller_id": "<id>", "vdag_uri": "<uri>", "config": { "policy_execution_mode": "local", "replicas": 2 }, "search_tags": [] }
remove_controller Removes a controller and deletes its registry entry. { "vdag_controller_id": "<id>" }
scale_controller Changes the number of replicas. { "vdag_controller_id": "<id>", "replicas": 3 }
list_controllers Lists all controllers. {}

These are normally called through the Cluster Controller Gateway. See the vDAG creator guide.

Cluster actions

Internal API

Used internally through the Cluster Controller Gateway.

Endpoint Method What it does
/cluster-actions/add-node POST Adds a node, using the node data in the body.
/cluster-actions/remove-node/<node_id> DELETE Removes a node.
/cluster-actions/get-cluster GET Returns the cluster's configuration and nodes.
/cluster-actions/update-cluster PATCH Updates the cluster's configuration.
# Add a node
curl -X POST http://<server-url>/cluster-actions/add-node \
  -H "Content-Type: application/json" \
  -d '{}'

# Remove a node
curl -X DELETE http://<server-url>/cluster-actions/remove-node/<node-id>

# Get cluster data
curl -X GET http://<server-url>/cluster-actions/get-cluster

# Update cluster data (MongoDB-style update)
curl -X PATCH http://<server-url>/cluster-actions/update-cluster \
  -H "Content-Type: application/json" \
  -d '{
    "$set": {
      "tags": ["gpu", "updated"],
      "reputation": 97
    }
  }'

Query blocks

Endpoint Method What it does
/blocks/<block_id> GET Returns one block.
/blocks GET Returns all blocks in the cluster.
/blocks/query POST Filters blocks with a MongoDB-style query ($in, $and, $or, ...).
curl -X GET http://<server-url>/blocks/<block-id>

curl -X GET http://<server-url>/blocks

curl -X POST http://<server-url>/blocks/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "cluster.reputation": { "$gt": 90 },
      "policies.autoscaler.policyRuleURI": { "$ne": "" }
    },
    "options": {
      "sort": { "id": 1 },
      "limit": 10
    }
  }'

Update a block

PUT /block/update/<block_id>: updates a block's configuration or metadata using a MongoDB-style update ($set, $unset, ...).

curl -X PUT http://<server-url>/block/update/<block-id> \
  -H "Content-Type: application/json" \
  -d '{
    "$set": {
        "blockMetadata.description": "Updated description for object detection block",
        "minInstances": 2
    }
  }'

Node health (infrastructure scanner)

Endpoint Method What it returns
/healthy_nodes GET Nodes that are currently healthy.
/nodes_status GET Health status of every node, healthy or not.
curl -X GET http://<server-url>/healthy_nodes
curl -X GET http://<server-url>/nodes_status

Stability checker

Endpoint Method What it does
/check_stability GET Runs a cluster-wide stability check now, using the stability policy.
/mgmt POST Sends a management command to the stability policy.
curl -X GET http://<server-url>/check_stability

curl -X POST http://<server-url>/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "mgmt_action": "<action>",
    "mgmt_data": {}
  }'

Management commands

POST /mgmt (management command executor) sends a management command to one of a block's services, or to the cluster's stability checker.

  • With a block_id: the command goes to the named service of that block.
  • With an empty block_id and service set to stability_checker: the command goes to the cluster's stability checker.
curl -X POST http://<server-url>/mgmt \
  -H "Content-Type: application/json" \
  -d '{
    "block_id": "",
    "service": "stability_checker",
    "mgmt_action": "<action>",
    "mgmt_data": {}
  }'
service Where the command goes
instances All of the block's instance pods, each by its own IP address.
executor The block's executor (load balancer).
autoscaler The block's auto-scaler.
health The block's health checker.
stability_checker The cluster's stability checker (leave block_id empty).

See also Management commands.