Skip to content

Cluster admin guide

The cluster admin owns a Kubernetes cluster and contributes it to a network. You decide what your cluster accepts, keep it healthy, and can offer extra services, such as registries or an inference server, to everyone on the network.

At a glance

You will: join your cluster to a network, set its own rules, monitor its stability, keep its details up to date and, optionally, host add-on services.
You need: kubectl and Helm access to your cluster, and the network's Parser and Cluster Controller Gateway URLs.

Your tasks

  1. Join your cluster to a network
  2. Set the cluster's rules (pre-check policies)
  3. Choose where policies run
  4. Monitor stability
  5. Update cluster details
  6. Host add-on services

Two layers of rules

Every action on your cluster is checked twice: first against the network's rules (set by the network admin), then against your cluster's rules (set by you). Both must allow it.


1. Join your cluster to a network

Follow Join a cluster. In short:

cd k8s/onboarding/cluster

# On every GPU node: publish GPU details
sh ./gpu_info.sh

# Edit config.json, then register the cluster with the network
./onboard.sh config.json --onboard --api-url <parser-url>

Then install the cluster services with the Gateway's create-cluster-infra API (step 3).


2. Set the cluster's rules (pre-check policies)

Pre-check policies are Python rules that run before an action and can allow it, reject it, or modify the request. Use them to enforce your cluster's own constraints, for example "only accept nodes with at least 32 GB of RAM" or "only allow blocks from approved authors".

Action Checked when...
add_node a node asks to join
remove_node a node is to be removed
update_cluster cluster configuration or metadata changes
scale_block a block's instances are scaled
add_block a new block is to be created
remove_block a block is to be removed
add_vdag_controller a vDAG controller is to be created
remove_vdag_controller a vDAG controller is to be removed
block_mgmt a management command is sent to a block
cluster_mgmt a cluster management command is run

To add a policy:

  1. Write it. See Cluster Controller: writing a pre-check policy.
  2. Register it with the Policies System. See Prepare a policy for onboarding.
  3. Attach it to the action with your cluster controller's update API. Map each action to a policy rule URI:

    curl -X POST "http://<cluster-controller-url>/pre-check-policies/update" \
        -H "Content-Type: application/json" \
        -d '{
            "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable",
            "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
            "remove_block": "",
            "add_vdag_controller": ""
        }'
    
    In the request Result
    Action with a URI (add_node, remove_node) That policy is attached, replacing any previous one.
    Action with an empty string (remove_block, add_vdag_controller) Any existing policy for that action is removed.
    Action not mentioned Its current policy, if any, is unchanged.

3. Choose where policies run

By default, the cluster controller runs governance policies locally. To save compute on your cluster, you can offload them to a remote policy executor.

Two environment variables on the <cluster_id>-controller deployment (namespace controllers, container index 1) control this:

Variable Local (default) Remote
POLICY_EXECUTION_MODE local remote
POLICY_SYSTEM_EXECUTOR_ID "" (empty) the ID of the policy executor to use
kubectl patch deployment <cluster_id>-controller \
  -n controllers \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/1/env",
      "value": []
    },
    {
      "op": "add",
      "path": "/spec/template/spec/containers/1/env/-",
      "value": {
        "name": "POLICY_EXECUTION_MODE",
        "value": "remote"
      }
    },
    {
      "op": "add",
      "path": "/spec/template/spec/containers/1/env/-",
      "value": {
        "name": "POLICY_SYSTEM_EXECUTOR_ID",
        "value": "<executor_id>"
      }
    }
  ]'
kubectl patch deployment <cluster_id>-controller \
  -n controllers \
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/1/env",
      "value": [
        {
          "name": "POLICY_EXECUTION_MODE",
          "value": "local"
        },
        {
          "name": "POLICY_SYSTEM_EXECUTOR_ID",
          "value": ""
        }
      ]
    }
  ]'

Check the current setting:

kubectl -n controllers get deployment <cluster_id>-controller -o jsonpath="{.spec.template.spec.containers[1].env}"

Warning

These patches replace the container's whole environment variable list. If you added other variables to this container, add them back afterwards.


4. Monitor stability

The cluster's stability checker periodically looks at every node's health and decides whether the cluster as a whole is stable. You define what "stable" means with a stability checker policy. The policy can also notify you, for example through a webhook, when something goes wrong.

  1. Write the policy. See Cluster Controller: stability checker policy.
  2. Register it with the Policies System. See Prepare a policy for onboarding.
  3. Run a check on demand, or send the policy a management command:

    # Run a stability check now
    curl -X GET http://<cluster-controller-url>/check_stability
    
    # Send a management command to the stability checker policy
    curl -X POST http://<cluster-controller-url>/mgmt \
      -H "Content-Type: application/json" \
      -d '{
        "block_id": "",
        "service": "stability_checker",
        "mgmt_action": "<action>",
        "mgmt_data": {}
      }'
    

    Leave block_id empty and set service to stability_checker. Which mgmt_action values exist depends on your policy.

To see the health of individual nodes, use GET /healthy_nodes and GET /nodes_status. See Node health.


5. Update cluster details

You can change your cluster's registry entry, such as region, tags or contact details, through the network's Cluster Controller Gateway. Changes must pass the network's update_cluster pre-check policy, if there is one.

curl -X PATCH http://<cluster-controller-gateway-url>/clusters/update/<cluster-id> \
    -H "Content-Type: application/json" \
    -d '<updated-fields-json>'

Example: update the contact email, region and tags:

curl -X PATCH http://<cluster-controller-gateway-url>/clusters/update/cluster-123 \
    -H "Content-Type: application/json" \
    -d '{
          "clusterMetadata.email": "ai-infra@company.net",
          "regionId": "us-west-2",
          "tags": ["ai-cluster", "llm", "datacenter-grade-1"]
       }'

6. Host add-on services

Your cluster can host optional services and offer them to the network:

  • a container registry for images
  • an assets registry for model weights and files
  • an inference server, a gateway for users' tasks
  • a policy executor, to run other people's policies
  • a model splitter, to split large models

See Deploying add-on services.