Resource Allocator¶
The Resource Allocator runs resource allocation policies. Given a policy and some inputs, it gathers candidate clusters, their details and live metrics, runs your policy, and returns the best cluster, node and GPUs for a workload. It is generic: you plug in your own allocation logic.
At a glance
Runs on: the management cluster, as resource-allocator (WebSocket on port 8765, NodePort 30151).
Protocol: WebSocket, because answers can take a while when the search space is large.
Use it for: placement decisions and manual "where would this run?" dry runs.
How it works¶

(Download the architecture diagram)
- A client sends a request over the WebSocket.
- If the request lists cluster IDs, those clusters are the candidates.
- If not, the request can include a filter or a similarity search query (see Search and filter).
- In that case, the Resource Allocator asks the search server for matching clusters.
- It fetches each candidate's record from the cluster registry, and its live metrics from the cluster metrics API.
- It loads your policy, giving it a cluster registry client and a cluster metrics client (see below).
- It runs the policy with the clusters' data and metrics.
- It returns the result over the WebSocket.
Send a request¶
Connect a WebSocket client and send one JSON message:
{
"adhoc": true,
"policy_rule_uri": "string",
"clusters": [/* optional list of clusters */],
"inputs": { /* required input parameters */ },
"filter": { /* optional filter criteria */ },
"search": { /* optional search criteria */ },
"parameters": { /* optional parameters to customize allocation */ },
"settings": { /* optional execution settings */ }
}
| Field | Type | Required | Description |
|---|---|---|---|
adhoc |
boolean | Yes | Must be true. |
policy_rule_uri |
string | Yes | The allocation policy to run (registered in the Policies System). |
clusters |
array | No | Candidate cluster IDs, if you already know them. |
filter |
object | No | A filter query to find candidates, if you don't. |
search |
object | No | A similarity search query to find candidates, if you don't. |
inputs |
object | Yes | Inputs for your policy, for example the workload's requirements. |
parameters |
object | No | Overrides for the policy's parameters. |
settings |
object | No | Overrides for the policy's settings. |
Python example¶
import asyncio
import websockets
import json
import logging
logging.basicConfig(level=logging.INFO)
async def send_resource_allocation_request():
uri = "ws://<server-url>:8765"
sample_payload = {
"adhoc": True,
"policy_rule_uri": "",
"clusters": [],
"inputs": {},
"filter": {},
"search": {},
"parameters": {},
"settings": {}
}
try:
async with websockets.connect(uri) as websocket:
# Send the request
await websocket.send(json.dumps(sample_payload))
logging.info("Request sent to server.")
# Wait for the response
response = await websocket.recv()
logging.info(f"Response received: {response}")
except Exception as e:
logging.error(f"Error in WebSocket client: {e}")
if __name__ == "__main__":
asyncio.run(send_resource_allocation_request())
Write an allocation policy¶
Your policy follows the standard policy structure. It does not need to implement management commands.
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
"""
Initializes an AIOSv1PolicyRule instance.
Args:
rule_id (str): Unique identifier for the rule.
settings (dict): Configuration settings for the rule
(includes the injected clients; see below).
parameters (dict): Parameters defining the rule's behavior.
"""
self.rule_id = rule_id
self.settings = settings
self.parameters = parameters
def eval(self, parameters, input_data, context):
"""
Evaluates the policy rule.
input_data = {
"cluster_data": {<cluster_id>: {"data": ..., "cluster_metrics": ...}, ...},
"inputs": <the "inputs" from the request>
}
Returns:
dict: {"cluster_id": ..., "node_id": ..., "gpus": [...]}
"""
# Allocation logic goes here
return {}
def management(self, action: str, data: dict) -> dict:
"""Management commands are not required for this policy."""
return {}
Input¶
{
"cluster_data": {<cluster_id>: {"data": ..., "cluster_metrics": ...}, ...},
"inputs": <the "inputs" dict from the request>
}
cluster_data has one entry per candidate cluster:
{
"cluster-0": {
"data": <cluster-data>,
"cluster_metrics": <cluster-metrics>
},
"cluster-1": {
"data": <cluster-data>,
"cluster_metrics": <cluster-metrics>
}
}
| Key | Type | Description |
|---|---|---|
<cluster_id> |
string | One entry per candidate cluster. |
data |
dict | The cluster's record: hardware, labels, configuration. See Cluster registry. |
cluster_metrics |
dict | Live metrics: GPU availability, load, memory use. See Metrics system. |
Output¶
Example:
This means: use GPUs 0 and 2 on node-0 in cluster-123.
Clients available to your policy¶
The Resource Allocator adds two clients to your policy's settings:
settings.update({
'cluster_client': self.cluster_client,
'cluster_metrics_client': self.metrics_client
})
Use them like this:
# Cluster registry client
cluster_client = self.settings['cluster_client']
ok, cluster = cluster_client.read_cluster('cluster-123')
# Cluster metrics client
cluster_metrics_client = self.settings['cluster_metrics_client']
metrics = cluster_metrics_client.get_cluster('cluster-123')
| Client | Method | Returns |
|---|---|---|
cluster_client |
read_cluster(cluster_id) |
(True, record) or (False, error message) |
execute_query(query) |
(True, results) or (False, error message), for a MongoDB-style query |
|
cluster_metrics_client |
get_cluster(cluster_id) |
The cluster's metrics. |
query_clusters(query_params=None) |
Metrics for clusters matching a query. |
Client source
class ClusterClient:
def __init__(self):
self.base_url = os.getenv(
"CLUSTER_SERVICE_URL", "http://<server-url>:3000")
def read_cluster(self, cluster_id):
try:
response = requests.get(f"{self.base_url}/clusters/{cluster_id}")
response.raise_for_status()
return True, response.json()
except requests.exceptions.HTTPError as http_err:
return False, f"HTTP error occurred: {http_err}"
except Exception as err:
return False, f"Error occurred: {err}"
def execute_query(self, query):
try:
response = requests.post(
f"{self.base_url}/clusters/query", json={"query": query})
response.raise_for_status()
return True, response.json()
except requests.exceptions.HTTPError as http_err:
return False, f"HTTP error occurred: {http_err}"
except Exception as err:
return False, f"Error occurred: {err}"
class GlobalClusterMetricsClient:
def __init__(self):
self.base_url = os.getenv("GLOBAL_CLUSTER_METRICS_URL", "http://<server-url>:8888").rstrip('/')
logger.info(
f"GlobalClusterMetricsClient initialized with base URL: {self.base_url}")
def _handle_response(self, response):
try:
response.raise_for_status()
json_data = response.json()
if not json_data.get("success", False):
raise APIError(json_data.get(
"error", "Unknown error"), response.status_code)
return json_data.get("data")
except requests.RequestException as e:
logger.error(f"HTTP error: {e}")
raise
except ValueError as e:
logger.error(f"Invalid JSON response: {e}")
raise
def get_cluster(self, cluster_id):
url = f"{self.base_url}/cluster/{cluster_id}"
response = requests.get(url)
return self._handle_response(response)
def query_clusters(self, query_params=None):
url = f"{self.base_url}/cluster/query"
response = requests.get(url, params=query_params)
return self._handle_response(response)
Related¶
- How blocks are placed during creation: Block spec: allocation flow
- Node and GPU allocation inside a cluster: Cluster Controller: resource allocation policy