Skip to content

Resource Allocator

The Resource Allocator runs resource allocation policies. Given a policy and some inputs, it gathers candidate clusters, their details and live metrics, runs your policy, and returns the best cluster, node and GPUs for a workload. It is generic: you plug in your own allocation logic.

At a glance

Runs on: the management cluster, as resource-allocator (WebSocket on port 8765, NodePort 30151).
Protocol: WebSocket, because answers can take a while when the search space is large.
Use it for: placement decisions and manual "where would this run?" dry runs.

How it works

Resource Allocator architecture

(Download the architecture diagram)

  1. A client sends a request over the WebSocket.
  2. If the request lists cluster IDs, those clusters are the candidates.
  3. If not, the request can include a filter or a similarity search query (see Search and filter).
  4. In that case, the Resource Allocator asks the search server for matching clusters.
  5. It fetches each candidate's record from the cluster registry, and its live metrics from the cluster metrics API.
  6. It loads your policy, giving it a cluster registry client and a cluster metrics client (see below).
  7. It runs the policy with the clusters' data and metrics.
  8. It returns the result over the WebSocket.

Send a request

Connect a WebSocket client and send one JSON message:

{
  "adhoc": true,
  "policy_rule_uri": "string",
  "clusters": [/* optional list of clusters */],
  "inputs": { /* required input parameters */ },
  "filter": { /* optional filter criteria */ },
  "search": { /* optional search criteria */ },
  "parameters": { /* optional parameters to customize allocation */ },
  "settings": { /* optional execution settings */ }
}
Field Type Required Description
adhoc boolean Yes Must be true.
policy_rule_uri string Yes The allocation policy to run (registered in the Policies System).
clusters array No Candidate cluster IDs, if you already know them.
filter object No A filter query to find candidates, if you don't.
search object No A similarity search query to find candidates, if you don't.
inputs object Yes Inputs for your policy, for example the workload's requirements.
parameters object No Overrides for the policy's parameters.
settings object No Overrides for the policy's settings.

Python example

import asyncio
import websockets
import json
import logging

logging.basicConfig(level=logging.INFO)

async def send_resource_allocation_request():
    uri = "ws://<server-url>:8765"

    sample_payload = {
        "adhoc": True,
        "policy_rule_uri": "",
        "clusters": [],
        "inputs": {},
        "filter": {},
        "search": {},
        "parameters": {},
        "settings": {}
    }

    try:
        async with websockets.connect(uri) as websocket:
            # Send the request
            await websocket.send(json.dumps(sample_payload))
            logging.info("Request sent to server.")

            # Wait for the response
            response = await websocket.recv()
            logging.info(f"Response received: {response}")

    except Exception as e:
        logging.error(f"Error in WebSocket client: {e}")

if __name__ == "__main__":
    asyncio.run(send_resource_allocation_request())

Write an allocation policy

Your policy follows the standard policy structure. It does not need to implement management commands.

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        """
        Initializes an AIOSv1PolicyRule instance.

        Args:
            rule_id (str): Unique identifier for the rule.
            settings (dict): Configuration settings for the rule
                (includes the injected clients; see below).
            parameters (dict): Parameters defining the rule's behavior.
        """
        self.rule_id = rule_id
        self.settings = settings
        self.parameters = parameters

    def eval(self, parameters, input_data, context):
        """
        Evaluates the policy rule.

        input_data = {
            "cluster_data": {<cluster_id>: {"data": ..., "cluster_metrics": ...}, ...},
            "inputs": <the "inputs" from the request>
        }

        Returns:
            dict: {"cluster_id": ..., "node_id": ..., "gpus": [...]}
        """
        # Allocation logic goes here
        return {}

    def management(self, action: str, data: dict) -> dict:
        """Management commands are not required for this policy."""
        return {}

Input

{
    "cluster_data": {<cluster_id>: {"data": ..., "cluster_metrics": ...}, ...},
    "inputs": <the "inputs" dict from the request>
}

cluster_data has one entry per candidate cluster:

{
    "cluster-0": {
        "data": <cluster-data>,
        "cluster_metrics": <cluster-metrics>
    },
    "cluster-1": {
        "data": <cluster-data>,
        "cluster_metrics": <cluster-metrics>
    }
}
Key Type Description
<cluster_id> string One entry per candidate cluster.
data dict The cluster's record: hardware, labels, configuration. See Cluster registry.
cluster_metrics dict Live metrics: GPU availability, load, memory use. See Metrics system.

Output

{
    "cluster_id": <cluster_id>,
    "node_id": <node_id>,     (optional)
    "gpus": [<gpu_id>]        (optional)
}

Example:

{
    "cluster_id": "cluster-123",
    "node_id": "node-0",
    "gpus": ["0", "2"]
}

This means: use GPUs 0 and 2 on node-0 in cluster-123.

Clients available to your policy

The Resource Allocator adds two clients to your policy's settings:

settings.update({
    'cluster_client': self.cluster_client,
    'cluster_metrics_client': self.metrics_client
})

Use them like this:

# Cluster registry client
cluster_client = self.settings['cluster_client']
ok, cluster = cluster_client.read_cluster('cluster-123')

# Cluster metrics client
cluster_metrics_client = self.settings['cluster_metrics_client']
metrics = cluster_metrics_client.get_cluster('cluster-123')
Client Method Returns
cluster_client read_cluster(cluster_id) (True, record) or (False, error message)
execute_query(query) (True, results) or (False, error message), for a MongoDB-style query
cluster_metrics_client get_cluster(cluster_id) The cluster's metrics.
query_clusters(query_params=None) Metrics for clusters matching a query.
Client source
class ClusterClient:
    def __init__(self):
        self.base_url = os.getenv(
            "CLUSTER_SERVICE_URL", "http://<server-url>:3000")

    def read_cluster(self, cluster_id):
        try:
            response = requests.get(f"{self.base_url}/clusters/{cluster_id}")
            response.raise_for_status()
            return True, response.json()
        except requests.exceptions.HTTPError as http_err:
            return False, f"HTTP error occurred: {http_err}"
        except Exception as err:
            return False, f"Error occurred: {err}"

    def execute_query(self, query):
        try:
            response = requests.post(
                f"{self.base_url}/clusters/query", json={"query": query})
            response.raise_for_status()
            return True, response.json()
        except requests.exceptions.HTTPError as http_err:
            return False, f"HTTP error occurred: {http_err}"
        except Exception as err:
            return False, f"Error occurred: {err}"


class GlobalClusterMetricsClient:
    def __init__(self):
        self.base_url = os.getenv("GLOBAL_CLUSTER_METRICS_URL", "http://<server-url>:8888").rstrip('/')
        logger.info(
            f"GlobalClusterMetricsClient initialized with base URL: {self.base_url}")

    def _handle_response(self, response):
        try:
            response.raise_for_status()
            json_data = response.json()
            if not json_data.get("success", False):
                raise APIError(json_data.get(
                    "error", "Unknown error"), response.status_code)
            return json_data.get("data")
        except requests.RequestException as e:
            logger.error(f"HTTP error: {e}")
            raise
        except ValueError as e:
            logger.error(f"Invalid JSON response: {e}")
            raise

    def get_cluster(self, cluster_id):
        url = f"{self.base_url}/cluster/{cluster_id}"
        response = requests.get(url)
        return self._handle_response(response)

    def query_clusters(self, query_params=None):
        url = f"{self.base_url}/cluster/query"
        response = requests.get(url, params=query_params)
        return self._handle_response(response)