Network admin guide¶
The network admin creates a network and decides its rules: who may join, what may run, and which actions are allowed.
At a glance
You will: install the management cluster, write governance policies, and keep them up to date.
You need: a Kubernetes cluster for management services, and some Python, for writing policies.
Your tasks¶
- Set up the management cluster
- Write governance (pre-check) policies
- Update governance policies
- Choose where governance policies run
1. Set up the management cluster¶
The management cluster hosts every management service and database for your network, and exposes the network's core APIs. Follow Create a network. In short:
kubectl label node <registry-node-name> registry=yes --overwrite
kubectl label node <metrics-node-name> metrics-db=yes --overwrite
bash k8s/install_mgmt_services.sh
When it is done, share your Parser URL (port 30500) and Cluster Controller Gateway URL (port 30600) with cluster owners who want to join.
2. Write governance (pre-check) policies¶
Governance policies, also called pre-check policies, are how you enforce the network's rules. The Cluster Controller Gateway runs the matching policy before every action on a cluster or block. The policy can allow the action, reject it, or modify the request.
For example, an add_cluster policy could accept only clusters in certain regions or with a minimum number of GPUs, and an add_block policy could block certain kinds of workloads.
Actions you can govern¶
| Policy name | Runs when someone tries to... |
|---|---|
add_node |
add a node to a cluster |
remove_node |
remove a node from a cluster |
add_cluster |
add a new cluster to the network |
remove_cluster |
remove a cluster |
update_cluster |
change a cluster's details |
add_block |
create a block on a cluster |
remove_block |
remove a block |
scale_block |
scale a block up or down |
block_mgmt |
run a management command on a block |
cluster_mgmt |
run a management command on a cluster |
How to add one¶
- Write the policy in Python. The input each policy receives, and the expected output, are described in Cluster Controller Gateway: pre-check policies.
- Register it with the Policies System, which gives it a policy rule URI. See Policies: prepare a policy for onboarding.
- Attach it to the action, as shown in the next section.
3. Update governance policies¶
Use the Gateway's update API to attach, replace or detach policies. Map each action to a policy rule URI:
curl -X POST "http://<cluster-controller-gateway-url>/pre-check-policies/update" \
-H "Content-Type: application/json" \
-d '{
"add_node": "aiosv1.policies.cluster-controller-gateway.add-node:v0.0.12-stable",
"remove_node": "aiosv1.policies.cluster-controller-gateway.remove-node:v0.0.1-stable",
"add_cluster": "aiosv1.policies.cluster-controller-gateway.add-cluster:v0.0.2-stable",
"remove_cluster": "",
"add_block": ""
}'
How this request is interpreted:
| In the request | Result |
|---|---|
Action with a URI (add_node, remove_node, add_cluster) |
That policy is attached, replacing any previous one. |
Action with an empty string (remove_cluster, add_block) |
Any existing policy for that action is removed. |
| Action not mentioned | Its current policy, if any, is unchanged. |
4. Choose where governance policies run¶
By default the Gateway runs governance policies locally, inside its own process. If policies are heavy, or you want to free up the management cluster, you can offload them to a remote policy executor.
Two environment variables on the cluster-controller-gateway deployment (namespace services) control this:
| Variable | Local (default) | Remote |
|---|---|---|
POLICY_EXECUTION_MODE |
local |
remote |
POLICY_SYSTEM_EXECUTOR_ID |
"" (empty) |
the ID of the policy executor to use |
You need kubectl access to the management cluster.
kubectl patch deployment cluster-controller-gateway \
-n services \
--type='json' \
-p='[
{
"op": "add",
"path": "/spec/template/spec/containers/0/env",
"value": []
},
{
"op": "add",
"path": "/spec/template/spec/containers/0/env/-",
"value": {
"name": "POLICY_EXECUTION_MODE",
"value": "remote"
}
},
{
"op": "add",
"path": "/spec/template/spec/containers/0/env/-",
"value": {
"name": "POLICY_SYSTEM_EXECUTOR_ID",
"value": "<executor_id>"
}
}
]'
Check the current setting:
kubectl -n services get deployment cluster-controller-gateway -o jsonpath="{.spec.template.spec.containers[*].env}"
Warning
The "switch to remote" patch replaces the container's whole environment variable list. If you have added other environment variables to this deployment, add them back afterwards.
Related¶
- Cluster Controller Gateway reference
- Policies
- Cluster admin guide: clusters can add their own rules on top of yours.