Cluster Controller Gateway¶
The Cluster Controller Gateway is the network's control point for its clusters. It sits between users (and other management services, such as the Parser) and every cluster's Cluster Controller. Through it, you install cluster services, place and scale blocks, manage nodes and vDAG controllers, send management commands and read cluster metrics. Every change first passes the network's governance (pre-check) policies.
At a glance
Runs on: the management cluster, as cluster-controller-gateway (NodePort 30600).
Owned by: the network admin.
You'll use this page to: write network-wide governance policies and look up the Gateway's APIs.
What it does¶
For users and admins
- Read cluster and node metrics.
- Update or delete a cluster's registry entry.
- Install or remove a cluster's AIOS services (given the cluster's kubeconfig).
- Add or remove nodes.
- Create or remove vDAG controllers.
- Send management commands to blocks (parameter updates, policy parameters, SDK settings) and to cluster controllers.
- Manually scale a block's instances.
For the Parser
- Place (allocate) blocks.
- Remove blocks.
- Dry-run block placement.
- Create cluster entries.
Architecture¶
| Module | What it does |
|---|---|
| API server | Internal APIs (used by the Parser, the search server and others) and external APIs (exposed to users through the management ingress). |
| Pre-check policy executor | Every write or update passes through it. It runs the governance policy mapped to the action, which can allow, reject or modify the request. The action-to-policy mapping can be changed at any time through the API. |
| Block allocator and dry-run | Used by the Parser when creating a block. It loads the block's allocation policies, filters clusters through the cluster DB, asks candidate clusters for feasibility scores, and has the chosen cluster's controller place the block. It runs asynchronously: a task is created in the Tasks DB and its ID returned, so you can check progress. It also handles block removal. |
| Cluster infrastructure manager | Installs or removes AIOS services on a cluster. Runs asynchronously, with a task ID (see below). |
| Node manager | Adds and removes nodes by forwarding requests to the cluster's controller. Used by the node onboarding scripts. |
| vDAG controller manager | Creates, removes and scales vDAG controllers by forwarding requests to the target cluster's controller. |
| Clients | Internal clients for the cluster controllers, the clusters' metrics services and the global cluster DB. |
What "install cluster services" does¶
When you call create-cluster-infra, the Gateway:
- creates the cluster's local metrics databases on the target nodes, and waits for them to be ready
- creates the metrics reader and writer services that store collected metrics locally
- creates the metrics collector DaemonSet, which runs on every node and collects hardware metrics
- creates the Cluster Controller, configured from the cluster's registry entry
- sets up services, ingress routes and port mappings, so the services are reachable from outside
- records the services' URLs in the cluster's registry entry, so other services can find them
This runs in the background: you get a task ID to check in the Tasks DB. Removal runs the same steps in reverse.
Pre-check policies¶
The network admin attaches a pre-check (governance) policy to any of these actions:
| Action | Runs when someone tries to... |
|---|---|
add_node |
add a node to a cluster |
remove_node |
remove a node from a cluster |
add_cluster |
add a new cluster |
remove_cluster |
remove a cluster |
add_block |
create a block on a cluster |
remove_block |
remove a block |
update_cluster |
update a cluster's details |
scale_block |
scale a block up or down |
block_mgmt |
send a management command to a block |
cluster_mgmt |
send a management command to a cluster |
Actions with no policy are not checked.
Setting the policies¶
There are two ways:
-
At start-up, with the
CLUSTER_CONTROLLER_GATEWAY_ACTIONS_MAPenvironment variable: a JSON object mapping each action to a policy rule URI. Leave out actions you don't want checked.{ "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.1-stable", "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable", "add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.1-stable", "remove_cluster": "aiosv1.policies.cluster-controller-gateway.remove-cluster:v0.0.1-stable", "add_block": "aiosv1.policies.cluster-controller-gateway.add-block:v0.0.1-stable", "remove_block": "aiosv1.policies.cluster-controller-gateway.remove-block:v0.0.1-stable", "update_cluster": "aiosv1.policies.cluster-controller-gateway.update-cluster:v0.0.1-stable", "scale_block": "aiosv1.policies.cluster-controller-gateway.scale-block:v0.0.1-stable", "block_mgmt": "aiosv1.policies.cluster-controller-gateway.block-mgmt:v0.0.1-stable", "cluster_mgmt": "aiosv1.policies.cluster-controller-gateway.cluster-mgmt:v0.0.1-stable" } -
At runtime, with the update API. See also the Network admin guide.
Writing a pre-check policy¶
Return allowed and input_data:
# Allowed (input_data may be modified)
{
"allowed": True,
"input_data": input_data
}
# Not allowed
{
"allowed": False,
"input_data": <message or dict explaining why the action was not allowed>
}
If the policy changes input_data, the action proceeds with the changed data. If it makes no changes, return input_data unchanged.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule.
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
Args:
parameters (dict): The current parameters.
input_data (any): The input data to be evaluated (see "Pre-check policy inputs").
context (dict): Context (external cache), which can be used to
store and access state across multiple runs.
"""
# The input_data dict can be modified by the policy.
# Make input_data modifications here.
return {
"allowed": True,
"input_data": input_data
}
Pre-check policy inputs¶
What each action's policy receives as input_data:
| Action | input_data |
|---|---|
add_node |
{"node_data": ..., "cluster_data": ..., "cluster_metrics": ...}: the new node (in node spec format), its cluster, and the cluster's metrics. |
remove_node |
{"node_id": ..., "cluster_data": ..., "cluster_metrics": ..., "block_instances": [...]}: the node, its cluster, the cluster's metrics, and the blocks with instances on that node. |
add_cluster |
{"cluster_data": ...}: the new cluster (in cluster spec format). |
remove_cluster |
{"cluster_data": ..., "cluster_metrics": ..., "cluster_blocks": [...]}: the cluster, its metrics, and the blocks running on it. |
add_block |
{"block_data": ..., "cluster_data": ..., "cluster_metrics": ...}: the block, its target cluster, and the cluster's metrics. |
remove_block |
{"block_data": ..., "cluster_data": ..., "cluster_metrics": ..., "block_metrics": ..., "block_instance_nodes": [...]}: the block, its cluster, metrics, and the nodes its instances run on. |
update_cluster |
{"cluster_data": ...}: the cluster with the requested changes. |
scale_block |
{"block_data": ..., "input": ...}: the block, and the scaling request. |
block_mgmt |
{"block_data": ..., "input": ...}: the block, and the management command. |
cluster_mgmt |
{"cluster_data": ..., "input": ...}: the cluster, and the management command. |
Sample node_data (add_node)
{
"clusterId": "cluster-west-vision-001",
"id": "node-1",
"gpus": {
"count": 2,
"memory": 32768,
"gpus": [
{ "modelName": "NVIDIA A100", "memory": 16384 },
{ "modelName": "NVIDIA A100", "memory": 16384 }
],
"modelNames": ["NVIDIA A100"],
"features": ["tensor_cores"]
},
"vcpus": { "count": 32 },
"memory": 131072,
"swap": 8192,
"storage": { "disks": 2, "size": 1048576 },
"network": { "interfaces": 2, "txBandwidth": 0, "rxBandwidth": 0 },
"tags": ["gpu", "fp16", "production"],
"nodeMetadata": {
"vendor": "Supermicro",
"location": "Rack 2 - DC1",
"notes": "Installed 2024-12"
}
}
Sample cluster_data (add_cluster, update_cluster, ...)
See the full example in Cluster spec. Once registered, the record also contains system-filled fields such as config.urlMap.
Sample block_data (add_block, remove_block, scale_block, block_mgmt)
A block record contains the block spec fields plus the full component it was created from:
{
"blockId": "detector-yolo-88",
"blockComponentURI": "model.object-detector:2.0.0-stable",
"component": {
"componentId": {
"name": "vllm-chat",
"version": "1.0.0",
"releaseTag": "stable"
},
"componentType": "model",
"containerRegistryInfo": {
"containerImage": "registry.ai-platform.com/models/vllm-external-chat:2.0.0",
"containerRegistryId": "ai-platform-registry",
"containerImageMetadata": {
"author": "llm-team",
"description": "LLM team"
},
"componentMode": "third_party",
"initContainer": {
"containerImage": "registry.ai-platform.com/init-containers/vLLM-deployer:2.0.0",
"containerRegistryId": "ai-platform-registry",
"containerImageMetadata": {
"author": "vision-team",
"description": "Init container to deploy vLLM system"
}
}
},
"componentMetadata": {
"usecase": "chat-completion",
"framework": "external-api",
"hardware": "external"
},
"componentInitData": {
"vllm_server_url": "http://vllm-service:8080",
"model_name": "mistralai/Mistral-7B-Instruct-v0.1"
},
"componentInitParametersProtocol": {
"temperature": { "type": "number", "description": "Sampling temperature", "min": 0, "max": 2 }
},
"componentInitSettingsProtocol": {
"max_new_tokens": { "type": "number", "description": "Maximum number of tokens to generate", "min": 1, "max": 2048 }
},
"componentInputProtocol": {
"message": { "type": "string", "description": "Input message from the user" },
"session_id": { "type": "string", "description": "Unique identifier for the chat session" }
},
"componentOutputProtocol": {
"reply": { "type": "string", "description": "Chat reply from the vLLM model" }
},
"componentParameters": { "temperature": 0.7 },
"componentInitSettings": { "max_new_tokens": 256 },
"componentManagementCommandsTemplate": {
"reset": { "description": "Clears all stored chat sessions", "args": {} }
},
"tags": ["vllm", "chat", "openai-compatible", "mistral", "llm"]
},
"minInstances": 1,
"maxInstances": 5,
"blockInitData": {
"weights_path": "/models/yolov5s.pt",
"device": "cuda"
},
"initSettings": {
"batch_size": 4,
"timeout_ms": 1000
},
"parameters": {
"confidence_threshold": 0.5,
"nms_threshold": 0.4
},
"policyRulesSpec": [
"... see the Block spec example ..."
]
}
The full policyRulesSpec is shown in the Block spec example.
API reference¶
<server-url> is the Gateway's address, for example http://<management-node-ip>:30600.
1. Cluster records¶
| Endpoint | Method | What it does | Pre-check |
|---|---|---|---|
/clusters/read/<cluster-id> |
GET |
Get a cluster's record. | — |
/clusters/update/<cluster-id> |
PATCH |
Update one or more fields. | update_cluster |
/clusters/delete/<cluster-id> |
DELETE |
Delete a cluster's record. | remove_cluster |
# Read
curl -X GET http://<server-url>/clusters/read/cluster-123
# Update
curl -X PATCH http://<server-url>/clusters/update/cluster-123 \
-H "Content-Type: application/json" \
-d '{
"clusterMetadata.email": "ai-infra@company.net",
"regionId": "us-west-2",
"tags": ["ai-cluster", "llm", "datacenter-grade-1"]
}'
# Delete
curl -X DELETE http://<server-url>/clusters/delete/cluster-123
2. Cluster metrics¶
| Endpoint | Method | Returns |
|---|---|---|
/cluster-metrics/<cluster_id> |
GET |
Metrics for a cluster. |
/cluster-metrics/node/<cluster-id>/<node-id> |
GET |
Metrics for one node. |
curl -X GET "http://<server-url>/cluster-metrics/cluster-123" -H "Content-Type: application/json"
curl -X GET "http://<server-url>/cluster-metrics/node/cluster-123/node-2" -H "Content-Type: application/json"
For the metrics' structure, see Metrics system.
3. Scale a block¶
POST /controller/block-scaling/<cluster-id>: forwarded to the cluster where the block runs. Must pass the scale_block pre-check.
Set instances_count. allocation_data is optional: give one entry per new instance (node and GPUs) to place them yourself, or leave it out to let the block's resource allocation policy decide.
curl -X POST "http://<server-url>/controller/block-scaling/cluster-123" \
-H "Content-Type: application/json" \
-d '{
"operation": "scale",
"block_id": "block-787sj",
"instances_count": 2,
"allocation_data": [
{
"node_id": "node-1",
"gpu_ids": [0, 2]
},
{
"node_id": "node-5",
"gpu_ids": [1, 2]
}
]
}'
4. Cluster infrastructure¶
AIOS does not store your kubeconfig, so you pass it each time. This lets cluster owners install and remove AIOS services whenever they want, without re-registering the cluster.
| Endpoint | Method | What it does |
|---|---|---|
/create-cluster-infra |
POST |
Install AIOS cluster services. |
/remove-cluster-infra |
POST |
Remove them. |
curl -X POST "http://<server-url>/create-cluster-infra" \
-H "Content-Type: application/json" \
-d '{
"cluster_id": "cluster-123",
"kube_config_data": {}
}'
curl -X POST "http://<server-url>/remove-cluster-infra" \
-H "Content-Type: application/json" \
-d '{
"cluster_id": "cluster-123",
"kube_config_data": {}
}'
kube_config_data is the content of your kubeconfig file. Ready-made Python scripts that read ~/.kube/config and call these APIs are in Join a cluster: step 3 and Remove a cluster's services.
5. vDAG controllers¶
POST /vdag-controller/<cluster_id>: forwards a vDAG controller action to the cluster.
curl -X POST "http://<server-url>/vdag-controller/<cluster_id>" \
-H "Content-Type: application/json" \
-d '{
"action": "<action_value>",
"payload": <payload_data>
}'
Actions (create_controller, remove_controller, scale_controller, list_controllers) and their payloads are listed in Cluster Controller: vDAG controller actions.
6. Block management commands¶
POST /blocks/mgmt: sends a management command to a block's policies, services, or your instance code. Must pass the block_mgmt pre-check.
curl -X POST "http://<server-url>/blocks/mgmt" \
-H "Content-Type: application/json" \
-d '{
"block_id": "<block_id_value>",
"service": "<service_name>",
"action": "<mgmt_command>",
"payload": <payload_data>
}'
service is one of instances, executor, autoscaler or health. See Management commands.
7. Cluster management commands¶
POST /cluster/mgmt: sends a management command to a cluster controller's policies. Must pass the cluster_mgmt pre-check.
curl -X POST "http://<server-url>/cluster/mgmt" \
-H "Content-Type: application/json" \
-d '{
"cluster_id": "<cluster_id_value>",
"service": "<service_name>",
"action": "<mgmt_command>",
"payload": <payload_data>
}'
8. Add or remove nodes¶
Internal API
These are called by the node onboarding scripts. Use Join a node instead of calling them directly.
# Add a node
curl -X POST http://<server-url>/nodes/add-node-to-cluster/<cluster_id> \
-H "Content-Type: application/json" \
-d '{}'
# Remove a node
curl -X GET http://<server-url>/nodes/remove-node-from-cluster/<cluster_id>/<node_id>
9. Update pre-check policies¶
POST /pre-check-policies/update: attach, replace or remove governance policies.
curl -X POST "http://<server-url>/pre-check-policies/update" \
-H "Content-Type: application/json" \
-d '{
"add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable",
"remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
"add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.2-stable",
"remove_cluster": "",
"add_block": ""
}'
- An action with a URI gets that policy.
- An action with
""has its policy removed. - Actions not mentioned keep their current policy.
10. Init container status¶
POST /initContainer/queryInitContainerData/<cluster_id>: query the status of third-party system init containers on a cluster, with a MongoDB-style query.