Cluster admin guide¶
The cluster admin owns a Kubernetes cluster and contributes it to a network. You decide what your cluster accepts, keep it healthy, and can offer extra services, such as registries or an inference server, to everyone on the network.
At a glance
You will: join your cluster to a network, set its own rules, monitor its stability, keep its details up to date and, optionally, host add-on services.
You need: kubectl and Helm access to your cluster, and the network's Parser and Cluster Controller Gateway URLs.
Your tasks¶
- Join your cluster to a network
- Set the cluster's rules (pre-check policies)
- Choose where policies run
- Monitor stability
- Update cluster details
- Host add-on services
Two layers of rules
Every action on your cluster is checked twice: first against the network's rules (set by the network admin), then against your cluster's rules (set by you). Both must allow it.
1. Join your cluster to a network¶
Follow Join a cluster. In short:
cd k8s/onboarding/cluster
# On every GPU node: publish GPU details
sh ./gpu_info.sh
# Edit config.json, then register the cluster with the network
./onboard.sh config.json --onboard --api-url <parser-url>
Then install the cluster services with the Gateway's create-cluster-infra API (step 3).
2. Set the cluster's rules (pre-check policies)¶
Pre-check policies are Python rules that run before an action and can allow it, reject it, or modify the request. Use them to enforce your cluster's own constraints, for example "only accept nodes with at least 32 GB of RAM" or "only allow blocks from approved authors".
| Action | Checked when... |
|---|---|
add_node |
a node asks to join |
remove_node |
a node is to be removed |
update_cluster |
cluster configuration or metadata changes |
scale_block |
a block's instances are scaled |
add_block |
a new block is to be created |
remove_block |
a block is to be removed |
add_vdag_controller |
a vDAG controller is to be created |
remove_vdag_controller |
a vDAG controller is to be removed |
block_mgmt |
a management command is sent to a block |
cluster_mgmt |
a cluster management command is run |
To add a policy:
- Write it. See Cluster Controller: writing a pre-check policy.
- Register it with the Policies System. See Prepare a policy for onboarding.
-
Attach it to the action with your cluster controller's update API. Map each action to a policy rule URI:
curl -X POST "http://<cluster-controller-url>/pre-check-policies/update" \ -H "Content-Type: application/json" \ -d '{ "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable", "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable", "remove_block": "", "add_vdag_controller": "" }'In the request Result Action with a URI ( add_node,remove_node)That policy is attached, replacing any previous one. Action with an empty string ( remove_block,add_vdag_controller)Any existing policy for that action is removed. Action not mentioned Its current policy, if any, is unchanged.
3. Choose where policies run¶
By default, the cluster controller runs governance policies locally. To save compute on your cluster, you can offload them to a remote policy executor.
Two environment variables on the <cluster_id>-controller deployment (namespace controllers, container index 1) control this:
| Variable | Local (default) | Remote |
|---|---|---|
POLICY_EXECUTION_MODE |
local |
remote |
POLICY_SYSTEM_EXECUTOR_ID |
"" (empty) |
the ID of the policy executor to use |
kubectl patch deployment <cluster_id>-controller \
-n controllers \
--type='json' \
-p='[
{
"op": "add",
"path": "/spec/template/spec/containers/1/env",
"value": []
},
{
"op": "add",
"path": "/spec/template/spec/containers/1/env/-",
"value": {
"name": "POLICY_EXECUTION_MODE",
"value": "remote"
}
},
{
"op": "add",
"path": "/spec/template/spec/containers/1/env/-",
"value": {
"name": "POLICY_SYSTEM_EXECUTOR_ID",
"value": "<executor_id>"
}
}
]'
Check the current setting:
kubectl -n controllers get deployment <cluster_id>-controller -o jsonpath="{.spec.template.spec.containers[1].env}"
Warning
These patches replace the container's whole environment variable list. If you added other variables to this container, add them back afterwards.
4. Monitor stability¶
The cluster's stability checker periodically looks at every node's health and decides whether the cluster as a whole is stable. You define what "stable" means with a stability checker policy. The policy can also notify you, for example through a webhook, when something goes wrong.
- Write the policy. See Cluster Controller: stability checker policy.
- Register it with the Policies System. See Prepare a policy for onboarding.
-
Run a check on demand, or send the policy a management command:
# Run a stability check now curl -X GET http://<cluster-controller-url>/check_stability # Send a management command to the stability checker policy curl -X POST http://<cluster-controller-url>/mgmt \ -H "Content-Type: application/json" \ -d '{ "block_id": "", "service": "stability_checker", "mgmt_action": "<action>", "mgmt_data": {} }'Leave
block_idempty and setservicetostability_checker. Whichmgmt_actionvalues exist depends on your policy.
To see the health of individual nodes, use GET /healthy_nodes and GET /nodes_status. See Node health.
5. Update cluster details¶
You can change your cluster's registry entry, such as region, tags or contact details, through the network's Cluster Controller Gateway. Changes must pass the network's update_cluster pre-check policy, if there is one.
curl -X PATCH http://<cluster-controller-gateway-url>/clusters/update/<cluster-id> \
-H "Content-Type: application/json" \
-d '<updated-fields-json>'
Example: update the contact email, region and tags:
curl -X PATCH http://<cluster-controller-gateway-url>/clusters/update/cluster-123 \
-H "Content-Type: application/json" \
-d '{
"clusterMetadata.email": "ai-infra@company.net",
"regionId": "us-west-2",
"tags": ["ai-cluster", "llm", "datacenter-grade-1"]
}'
6. Host add-on services¶
Your cluster can host optional services and offer them to the network:
- a container registry for images
- an assets registry for model weights and files
- an inference server, a gateway for users' tasks
- a policy executor, to run other people's policies
- a model splitter, to split large models
See Deploying add-on services.