Skip to content

Network admin guide

The network admin creates a network and decides its rules: who may join, what may run, and which actions are allowed.

At a glance

You will: install the management cluster, write governance policies, and keep them up to date.
You need: a Kubernetes cluster for management services, and some Python, for writing policies.

Your tasks

  1. Set up the management cluster
  2. Write governance (pre-check) policies
  3. Update governance policies
  4. Choose where governance policies run

1. Set up the management cluster

The management cluster hosts every management service and database for your network, and exposes the network's core APIs. Follow Create a network. In short:

kubectl label node <registry-node-name> registry=yes --overwrite
kubectl label node <metrics-node-name> metrics-db=yes --overwrite
bash k8s/install_mgmt_services.sh

When it is done, share your Parser URL (port 30500) and Cluster Controller Gateway URL (port 30600) with cluster owners who want to join.


2. Write governance (pre-check) policies

Governance policies, also called pre-check policies, are how you enforce the network's rules. The Cluster Controller Gateway runs the matching policy before every action on a cluster or block. The policy can allow the action, reject it, or modify the request.

For example, an add_cluster policy could accept only clusters in certain regions or with a minimum number of GPUs, and an add_block policy could block certain kinds of workloads.

Actions you can govern

Policy name Runs when someone tries to...
add_node add a node to a cluster
remove_node remove a node from a cluster
add_cluster add a new cluster to the network
remove_cluster remove a cluster
update_cluster change a cluster's details
add_block create a block on a cluster
remove_block remove a block
scale_block scale a block up or down
block_mgmt run a management command on a block
cluster_mgmt run a management command on a cluster

How to add one

  1. Write the policy in Python. The input each policy receives, and the expected output, are described in Cluster Controller Gateway: pre-check policies.
  2. Register it with the Policies System, which gives it a policy rule URI. See Policies: prepare a policy for onboarding.
  3. Attach it to the action, as shown in the next section.

3. Update governance policies

Use the Gateway's update API to attach, replace or detach policies. Map each action to a policy rule URI:

curl -X POST "http://<cluster-controller-gateway-url>/pre-check-policies/update" \
    -H "Content-Type: application/json" \
    -d '{
        "add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable",
        "remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
        "add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.2-stable",
        "remove_cluster": "",
        "add_block": ""
    }'

How this request is interpreted:

In the request Result
Action with a URI (add_node, remove_node, add_cluster) That policy is attached, replacing any previous one.
Action with an empty string (remove_cluster, add_block) Any existing policy for that action is removed.
Action not mentioned Its current policy, if any, is unchanged.

4. Choose where governance policies run

By default the Gateway runs governance policies locally, inside its own process. If policies are heavy, or you want to free up the management cluster, you can offload them to a remote policy executor.

Two environment variables on the cluster-controller-gateway deployment (namespace services) control this:

Variable Local (default) Remote
POLICY_EXECUTION_MODE local remote
POLICY_SYSTEM_EXECUTOR_ID "" (empty) the ID of the policy executor to use

You need kubectl access to the management cluster.

kubectl patch deployment cluster-controller-gateway \
  -n services \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/env",
      "value": []
    },
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/env/-",
      "value": {
        "name": "POLICY_EXECUTION_MODE",
        "value": "remote"
      }
    },
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/env/-",
      "value": {
        "name": "POLICY_SYSTEM_EXECUTOR_ID",
        "value": "<executor_id>"
      }
    }
  ]'
kubectl patch deployment cluster-controller-gateway \
  -n services \
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/env",
      "value": [
        {
          "name": "POLICY_EXECUTION_MODE",
          "value": "local"
        },
        {
          "name": "POLICY_SYSTEM_EXECUTOR_ID",
          "value": ""
        }
      ]
    }
  ]'

Check the current setting:

kubectl -n services get deployment cluster-controller-gateway -o jsonpath="{.spec.template.spec.containers[*].env}"

Warning

The "switch to remote" patch replaces the container's whole environment variable list. If you have added other environment variables to this deployment, add them back afterwards.