Skip to content

Cluster Controller Gateway

The Cluster Controller Gateway is the network's control point for its clusters. It sits between users (and other management services, such as the Parser) and every cluster's Cluster Controller. Through it, you install cluster services, place and scale blocks, manage nodes and vDAG controllers, send management commands and read cluster metrics. Every change first passes the network's governance (pre-check) policies.

At a glance

Runs on: the management cluster, as cluster-controller-gateway (NodePort 30600).
Owned by: the network admin.
You'll use this page to: write network-wide governance policies and look up the Gateway's APIs.

What it does

For users and admins

  1. Read cluster and node metrics.
  2. Update or delete a cluster's registry entry.
  3. Install or remove a cluster's AIOS services (given the cluster's kubeconfig).
  4. Add or remove nodes.
  5. Create or remove vDAG controllers.
  6. Send management commands to blocks (parameter updates, policy parameters, SDK settings) and to cluster controllers.
  7. Manually scale a block's instances.

For the Parser

  1. Place (allocate) blocks.
  2. Remove blocks.
  3. Dry-run block placement.
  4. Create cluster entries.

Architecture

Module What it does
API server Internal APIs (used by the Parser, the search server and others) and external APIs (exposed to users through the management ingress).
Pre-check policy executor Every write or update passes through it. It runs the governance policy mapped to the action, which can allow, reject or modify the request. The action-to-policy mapping can be changed at any time through the API.
Block allocator and dry-run Used by the Parser when creating a block. It loads the block's allocation policies, filters clusters through the cluster DB, asks candidate clusters for feasibility scores, and has the chosen cluster's controller place the block. It runs asynchronously: a task is created in the Tasks DB and its ID returned, so you can check progress. It also handles block removal.
Cluster infrastructure manager Installs or removes AIOS services on a cluster. Runs asynchronously, with a task ID (see below).
Node manager Adds and removes nodes by forwarding requests to the cluster's controller. Used by the node onboarding scripts.
vDAG controller manager Creates, removes and scales vDAG controllers by forwarding requests to the target cluster's controller.
Clients Internal clients for the cluster controllers, the clusters' metrics services and the global cluster DB.

What "install cluster services" does

When you call create-cluster-infra, the Gateway:

  1. creates the cluster's local metrics databases on the target nodes, and waits for them to be ready
  2. creates the metrics reader and writer services that store collected metrics locally
  3. creates the metrics collector DaemonSet, which runs on every node and collects hardware metrics
  4. creates the Cluster Controller, configured from the cluster's registry entry
  5. sets up services, ingress routes and port mappings, so the services are reachable from outside
  6. records the services' URLs in the cluster's registry entry, so other services can find them

This runs in the background: you get a task ID to check in the Tasks DB. Removal runs the same steps in reverse.


Pre-check policies

The network admin attaches a pre-check (governance) policy to any of these actions:

Action Runs when someone tries to...
add_node add a node to a cluster
remove_node remove a node from a cluster
add_cluster add a new cluster
remove_cluster remove a cluster
add_block create a block on a cluster
remove_block remove a block
update_cluster update a cluster's details
scale_block scale a block up or down
block_mgmt send a management command to a block
cluster_mgmt send a management command to a cluster

Actions with no policy are not checked.

Setting the policies

There are two ways:

  • At start-up, with the CLUSTER_CONTROLLER_GATEWAY_ACTIONS_MAP environment variable: a JSON object mapping each action to a policy rule URI. Leave out actions you don't want checked.

    {
        "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.1-stable",
        "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
        "add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.1-stable",
        "remove_cluster": "aiosv1.policies.cluster-controller-gateway.remove-cluster:v0.0.1-stable",
        "add_block": "aiosv1.policies.cluster-controller-gateway.add-block:v0.0.1-stable",
        "remove_block": "aiosv1.policies.cluster-controller-gateway.remove-block:v0.0.1-stable",
        "update_cluster": "aiosv1.policies.cluster-controller-gateway.update-cluster:v0.0.1-stable",
        "scale_block": "aiosv1.policies.cluster-controller-gateway.scale-block:v0.0.1-stable",
        "block_mgmt": "aiosv1.policies.cluster-controller-gateway.block-mgmt:v0.0.1-stable",
        "cluster_mgmt": "aiosv1.policies.cluster-controller-gateway.cluster-mgmt:v0.0.1-stable"
    }
    
  • At runtime, with the update API. See also the Network admin guide.

Writing a pre-check policy

Return allowed and input_data:

# Allowed (input_data may be modified)
{
    "allowed": True,
    "input_data": input_data
}

# Not allowed
{
    "allowed": False,
    "input_data": <message or dict explaining why the action was not allowed>
}

If the policy changes input_data, the action proceeds with the changed data. If it makes no changes, return input_data unchanged.

class AIOSv1PolicyRule:

    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule.
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        Args:
            parameters (dict): The current parameters.
            input_data (any): The input data to be evaluated (see "Pre-check policy inputs").
            context (dict): Context (external cache), which can be used to
                store and access state across multiple runs.
        """
        # The input_data dict can be modified by the policy.
        # Make input_data modifications here.

        return {
            "allowed": True,
            "input_data": input_data
        }

Pre-check policy inputs

What each action's policy receives as input_data:

Action input_data
add_node {"node_data": ..., "cluster_data": ..., "cluster_metrics": ...}: the new node (in node spec format), its cluster, and the cluster's metrics.
remove_node {"node_id": ..., "cluster_data": ..., "cluster_metrics": ..., "block_instances": [...]}: the node, its cluster, the cluster's metrics, and the blocks with instances on that node.
add_cluster {"cluster_data": ...}: the new cluster (in cluster spec format).
remove_cluster {"cluster_data": ..., "cluster_metrics": ..., "cluster_blocks": [...]}: the cluster, its metrics, and the blocks running on it.
add_block {"block_data": ..., "cluster_data": ..., "cluster_metrics": ...}: the block, its target cluster, and the cluster's metrics.
remove_block {"block_data": ..., "cluster_data": ..., "cluster_metrics": ..., "block_metrics": ..., "block_instance_nodes": [...]}: the block, its cluster, metrics, and the nodes its instances run on.
update_cluster {"cluster_data": ...}: the cluster with the requested changes.
scale_block {"block_data": ..., "input": ...}: the block, and the scaling request.
block_mgmt {"block_data": ..., "input": ...}: the block, and the management command.
cluster_mgmt {"cluster_data": ..., "input": ...}: the cluster, and the management command.
Sample node_data (add_node)
{
  "clusterId": "cluster-west-vision-001",
  "id": "node-1",
  "gpus": {
    "count": 2,
    "memory": 32768,
    "gpus": [
      { "modelName": "NVIDIA A100", "memory": 16384 },
      { "modelName": "NVIDIA A100", "memory": 16384 }
    ],
    "modelNames": ["NVIDIA A100"],
    "features": ["tensor_cores"]
  },
  "vcpus": { "count": 32 },
  "memory": 131072,
  "swap": 8192,
  "storage": { "disks": 2, "size": 1048576 },
  "network": { "interfaces": 2, "txBandwidth": 0, "rxBandwidth": 0 },
  "tags": ["gpu", "fp16", "production"],
  "nodeMetadata": {
    "vendor": "Supermicro",
    "location": "Rack 2 - DC1",
    "notes": "Installed 2024-12"
  }
}
Sample cluster_data (add_cluster, update_cluster, ...)

See the full example in Cluster spec. Once registered, the record also contains system-filled fields such as config.urlMap.

Sample block_data (add_block, remove_block, scale_block, block_mgmt)

A block record contains the block spec fields plus the full component it was created from:

{
  "blockId": "detector-yolo-88",
  "blockComponentURI": "model.object-detector:2.0.0-stable",
  "component": {
    "componentId": {
      "name": "vllm-chat",
      "version": "1.0.0",
      "releaseTag": "stable"
    },
    "componentType": "model",
    "containerRegistryInfo": {
      "containerImage": "registry.ai-platform.com/models/vllm-external-chat:2.0.0",
      "containerRegistryId": "ai-platform-registry",
      "containerImageMetadata": {
        "author": "llm-team",
        "description": "LLM team"
      },
      "componentMode": "third_party",
      "initContainer": {
        "containerImage": "registry.ai-platform.com/init-containers/vLLM-deployer:2.0.0",
        "containerRegistryId": "ai-platform-registry",
        "containerImageMetadata": {
          "author": "vision-team",
          "description": "Init container to deploy vLLM system"
        }
      }
    },
    "componentMetadata": {
      "usecase": "chat-completion",
      "framework": "external-api",
      "hardware": "external"
    },
    "componentInitData": {
      "vllm_server_url": "http://vllm-service:8080",
      "model_name": "mistralai/Mistral-7B-Instruct-v0.1"
    },
    "componentInitParametersProtocol": {
      "temperature": { "type": "number", "description": "Sampling temperature", "min": 0, "max": 2 }
    },
    "componentInitSettingsProtocol": {
      "max_new_tokens": { "type": "number", "description": "Maximum number of tokens to generate", "min": 1, "max": 2048 }
    },
    "componentInputProtocol": {
      "message": { "type": "string", "description": "Input message from the user" },
      "session_id": { "type": "string", "description": "Unique identifier for the chat session" }
    },
    "componentOutputProtocol": {
      "reply": { "type": "string", "description": "Chat reply from the vLLM model" }
    },
    "componentParameters": { "temperature": 0.7 },
    "componentInitSettings": { "max_new_tokens": 256 },
    "componentManagementCommandsTemplate": {
      "reset": { "description": "Clears all stored chat sessions", "args": {} }
    },
    "tags": ["vllm", "chat", "openai-compatible", "mistral", "llm"]
  },
  "minInstances": 1,
  "maxInstances": 5,
  "blockInitData": {
    "weights_path": "/models/yolov5s.pt",
    "device": "cuda"
  },
  "initSettings": {
    "batch_size": 4,
    "timeout_ms": 1000
  },
  "parameters": {
    "confidence_threshold": 0.5,
    "nms_threshold": 0.4
  },
  "policyRulesSpec": [
    "... see the Block spec example ..."
  ]
}

The full policyRulesSpec is shown in the Block spec example.


API reference

<server-url> is the Gateway's address, for example http://<management-node-ip>:30600.

1. Cluster records

Endpoint Method What it does Pre-check
/clusters/read/<cluster-id> GET Get a cluster's record. —
/clusters/update/<cluster-id> PATCH Update one or more fields. update_cluster
/clusters/delete/<cluster-id> DELETE Delete a cluster's record. remove_cluster
# Read
curl -X GET http://<server-url>/clusters/read/cluster-123

# Update
curl -X PATCH http://<server-url>/clusters/update/cluster-123 \
     -H "Content-Type: application/json" \
     -d '{
           "clusterMetadata.email": "ai-infra@company.net",
           "regionId": "us-west-2",
           "tags": ["ai-cluster", "llm", "datacenter-grade-1"]
        }'

# Delete
curl -X DELETE http://<server-url>/clusters/delete/cluster-123

2. Cluster metrics

Endpoint Method Returns
/cluster-metrics/<cluster_id> GET Metrics for a cluster.
/cluster-metrics/node/<cluster-id>/<node-id> GET Metrics for one node.
curl -X GET "http://<server-url>/cluster-metrics/cluster-123" -H "Content-Type: application/json"
curl -X GET "http://<server-url>/cluster-metrics/node/cluster-123/node-2" -H "Content-Type: application/json"

For the metrics' structure, see Metrics system.

3. Scale a block

POST /controller/block-scaling/<cluster-id>: forwarded to the cluster where the block runs. Must pass the scale_block pre-check.

Set instances_count. allocation_data is optional: give one entry per new instance (node and GPUs) to place them yourself, or leave it out to let the block's resource allocation policy decide.

curl -X POST "http://<server-url>/controller/block-scaling/cluster-123" \
     -H "Content-Type: application/json" \
     -d '{
           "operation": "scale",
           "block_id": "block-787sj",
           "instances_count": 2,
           "allocation_data": [
               {
                   "node_id": "node-1",
                   "gpu_ids": [0, 2]
               },
               {
                   "node_id": "node-5",
                   "gpu_ids": [1, 2]
               }
           ]
         }'

List the instances to remove in instances_list.

curl -X POST "http://<server-url>/controller/block-scaling/cluster-123" \
     -H "Content-Type: application/json" \
     -d '{
           "operation": "downscale",
           "block_id": "block-787sj",
           "instances_list": ["instance-7347a"]
         }'

4. Cluster infrastructure

AIOS does not store your kubeconfig, so you pass it each time. This lets cluster owners install and remove AIOS services whenever they want, without re-registering the cluster.

Endpoint Method What it does
/create-cluster-infra POST Install AIOS cluster services.
/remove-cluster-infra POST Remove them.
curl -X POST "http://<server-url>/create-cluster-infra" \
    -H "Content-Type: application/json" \
    -d '{
    "cluster_id": "cluster-123",
    "kube_config_data": {}
    }'

curl -X POST "http://<server-url>/remove-cluster-infra" \
    -H "Content-Type: application/json" \
    -d '{
        "cluster_id": "cluster-123",
        "kube_config_data": {}
    }'

kube_config_data is the content of your kubeconfig file. Ready-made Python scripts that read ~/.kube/config and call these APIs are in Join a cluster: step 3 and Remove a cluster's services.

5. vDAG controllers

POST /vdag-controller/<cluster_id>: forwards a vDAG controller action to the cluster.

curl -X POST "http://<server-url>/vdag-controller/<cluster_id>" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "<action_value>",
    "payload": <payload_data>
  }'

Actions (create_controller, remove_controller, scale_controller, list_controllers) and their payloads are listed in Cluster Controller: vDAG controller actions.

6. Block management commands

POST /blocks/mgmt: sends a management command to a block's policies, services, or your instance code. Must pass the block_mgmt pre-check.

curl -X POST "http://<server-url>/blocks/mgmt" \
  -H "Content-Type: application/json" \
  -d '{
    "block_id": "<block_id_value>",
    "service": "<service_name>",
    "action": "<mgmt_command>",
    "payload": <payload_data>
  }'

service is one of instances, executor, autoscaler or health. See Management commands.

7. Cluster management commands

POST /cluster/mgmt: sends a management command to a cluster controller's policies. Must pass the cluster_mgmt pre-check.

curl -X POST "http://<server-url>/cluster/mgmt" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster_id": "<cluster_id_value>",
    "service": "<service_name>",
    "action": "<mgmt_command>",
    "payload": <payload_data>
  }'

8. Add or remove nodes

Internal API

These are called by the node onboarding scripts. Use Join a node instead of calling them directly.

# Add a node
curl -X POST http://<server-url>/nodes/add-node-to-cluster/<cluster_id> \
    -H "Content-Type: application/json" \
    -d '{}'

# Remove a node
curl -X GET http://<server-url>/nodes/remove-node-from-cluster/<cluster_id>/<node_id>

9. Update pre-check policies

POST /pre-check-policies/update: attach, replace or remove governance policies.

curl -X POST "http://<server-url>/pre-check-policies/update" \
    -H "Content-Type: application/json" \
    -d '{
        "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable",
        "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
        "add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.2-stable",
        "remove_cluster": "",
        "add_block": ""
    }'
  • An action with a URI gets that policy.
  • An action with "" has its policy removed.
  • Actions not mentioned keep their current policy.

10. Init container status

POST /initContainer/queryInitContainerData/<cluster_id>: query the status of third-party system init containers on a cluster, with a MongoDB-style query.

curl -X POST http://<server-url>/initContainer/queryInitContainerData/<cluster_id> \
     -H "Content-Type: application/json" \
     -d '{
           "status": "failed"
         }'