Skip to content

Metrics system

The metrics system collects measurements from every part of the network, stores the latest values, and makes them available to policies (for load balancing, auto-scaling and placement) and to people (for monitoring).

At a glance

Three kinds of metrics: hardware (nodes and clusters), blocks, and vDAGs.
Two levels of storage: a local database in each cluster, and global databases on the management cluster.
Note: these databases hold the current values, not history. For history, connect the optional Prometheus deployment, and Grafana for dashboards.

What is measured

Kind Examples Collected by
Hardware metrics CPU, memory, disk, network and GPU usage of each node A DaemonSet on every node
Block metrics Latency, throughput, queue length; plus any custom metrics you add Each block's executor and instances
vDAG metrics End-to-end latency, throughput Each vDAG controller

Architecture

Metrics system architecture

(Download the architecture diagram)

flowchart LR
    subgraph Cluster
        D["Node metrics DaemonSet<br/><small>every node</small>"] --> L[("Local metrics DB")]
        B["Block executor + instances<br/><small>metrics SDK</small>"] --> L
    end
    L -->|"periodic"| G[("Global cluster and<br/>block metrics DBs")]
    V["vDAG controllers"] --> GV[("Global vDAG metrics DB")]
    L -. "read by" .-> P1["Block policies<br/><small>load balancer, auto-scaler</small>"]
    G -. "read by" .-> P2["Placement policies,<br/>dashboards, users"]

In each cluster

Component What it does
Node metrics DaemonSet Runs on every node in host mode and collects CPU, memory, disk, network and GPU usage.
Metrics SDK and executor metrics Built into the Instance SDK and the block executor, providing the standard metrics plus your custom metrics. Pushes them to the local database periodically.
Local metrics database Holds the current metrics of every block and node in the cluster. Optionally backed by Prometheus, which can keep history for as long as storage allows, and Grafana for charts.

On the management cluster

Component What it does
Global metrics databases Hold the current metrics reported by every cluster (clusters, blocks) and every vDAG controller. Optionally backed by Prometheus and Grafana.

Hardware metrics

Node metrics DaemonSets

Every node that joins a cluster runs a metrics pod as a DaemonSet, in privileged mode so that it can read the host's hardware. At fixed intervals it collects:

  • CPU, memory, storage and network metrics (using psutil)
  • GPU metrics, if GPUs are present (using NVIDIA NVML)

and pushes them to the cluster's local metrics database.

Full list of hardware metrics (click to expand)

CPU

Name Description
load1m System load average over the last minute
load5m System load average over the last 5 minutes
load15m System load average over the last 15 minutes
runningThreads Currently running threads
runningProcesses Currently running processes
totalThreads Total threads
totalProcesses Total processes

Memory

Name Description
freeMem Free memory
usedMem Used memory
averageUtil Average memory utilization (%)
usedSwap Swap in use
freeSwap Free swap
pageFaultsPerSec Page faults per second

Disk

Name Description
memory.freeMem Free disk space
memory.usedMem Used disk space
memory.maxDisk Maximum disk space available
memory.minDisk Minimum disk space available
iops.blocksPerSec Blocks read or written per second
iops.readBytes Bytes read
iops.writeBytes Bytes written
iops.total Total I/O operations per second
iops.active Active I/O operations
iops.response Average I/O response time

Storage

Name Description
freeMem Free storage
usedMem Used storage

Network

Name Description
txBytesTotal Total bytes sent
rxBytesTotal Total bytes received

Per GPU

Name Description
gpu_id GPU ID
usedMem Used GPU memory (MB)
freeMem Free GPU memory (MB)
totalMem Total GPU memory (MB)
memUtilization GPU memory utilization (%)
cudaUtilization CUDA core utilization (%)
powerUtilization Power utilization (currently always 0)
temperature GPU temperature

All GPUs on a node (aggregated)

Name Description
totalUsedMem Used memory across all GPUs (MB)
totalFreeMem Free memory across all GPUs (MB)
totalMem Total memory across all GPUs (MB)
avgMemUtilization Average memory utilization (%)
avgCudaUtilization Average CUDA utilization (%)
avgPowerUtilization Average power utilization
avgTemperature Average temperature
count Number of GPUs

Local metrics database

Each cluster has its own local metrics database. It stores the latest hardware metrics from every node, and is what services such as a block's load balancer and auto-scaler read when making decisions. It is reachable through the cluster gateway at /metrics, or on NodePort 32301.

Endpoint Method Returns
/node/<node_id> GET Metrics for one node.
/node/query POST Nodes matching a MongoDB-style filter ($eq, $gt, $in, ...).
/cluster GET All nodes' metrics plus cluster-wide aggregates. Used by many internal components.
curl -X GET http://<server-url>/node/<node-id>

# Nodes whose 15-minute load is below 2
curl -X POST http://<server-url>/node/query \
  -H "Content-Type: application/json" \
  -d '{ "metrics.resource.node.vcpu.load_15m": { "$lt": 2 } }'

curl -X GET http://<server-url>/cluster

Global cluster metrics database

Every cluster's local database periodically reports its latest aggregated metrics to the global cluster metrics database (global-cluster-metrics, NodePort 30202), for network-wide monitoring and placement decisions.

Endpoint Method Returns
/cluster/<cluster_id> GET Metrics for one cluster.
/cluster/query POST Clusters matching a MongoDB-style filter.
curl -X GET http://<server-url>/cluster/<cluster-id>

# Clusters with more than ~50 GB of free memory
curl -X POST http://<server-url>/cluster/query \
  -H "Content-Type: application/json" \
  -d '{
    "cluster.memory.freeMem": {"$gt": 50000}
  }'

Block metrics

Collected from each block's executor and instances.

Standard metrics

Always available for every block, with no code needed.

Executor

Name Description
tasks_processed Tasks processed by the executor
latency Average end-to-end latency of tasks (seconds)

Instances

Name Description
on_preprocess_latency Time spent in on_preprocess
on_data_latency Time spent in on_data
end_to_end_latency Total processing time per task
on_preprocess_count Number of on_preprocess calls
on_data_count Number of on_data calls
end_to_end_count Number of tasks processed
on_preprocess_fps on_preprocess calls per second
on_data_fps on_data calls per second
end_to_end_fps Tasks processed per second
queue_length Length of the block's input Redis queue

Custom metrics

Add your own metrics with AIOSMetrics, available in your block as context.metrics. It registers and updates Prometheus counters, gauges and histograms, exposes them over HTTP, and publishes them through Redis. The Instance SDK guide shows how to use it inside a block.

AIOSMetrics(block_id: Optional[str] = None)

block_id defaults to the BLOCK_ID environment variable, or "test-block".

Environment variable Purpose Default
BLOCK_ID Block ID used to tag metrics —
INSTANCE_ID Instance ID "instance-001"
METRICS_REDIS_HOST Redis host for publishing metrics —
Method Description
register_counter(name, documentation, labelnames=None) Create a counter.
register_gauge(name, documentation, labelnames=None) Create a gauge.
register_histogram(name, documentation, labelnames=None, buckets=None) Create a histogram. Default buckets: [0.1, 0.2, 0.5, 1, 2, 5, 10].
increment_counter(name, labelnames=None) Add one to a counter.
set_gauge(name, value, labelnames=None) Set a gauge.
observe_histogram(name, value, labelnames=None) Record a value in a histogram.

Note

The labelnames argument of the update methods is currently ignored.

import os
from aios_metrics import AIOSMetrics

# Set environment variables if needed
os.environ["BLOCK_ID"] = "block-123"
os.environ["INSTANCE_ID"] = "instance-xyz"

metrics = AIOSMetrics()

# Register
metrics.register_counter("tasks_processed", "Number of tasks processed")
metrics.register_gauge("latency", "End-to-end latency of task processing")
metrics.register_histogram("inference_duration", "Histogram of inference durations")

# Update
metrics.increment_counter("tasks_processed")
metrics.set_gauge("latency", 0.432)
metrics.observe_histogram("inference_duration", 1.2)

Local block metrics database

Holds the current metrics (standard and custom) of every block and instance in the cluster, for local decision-making.

Endpoint Method Returns
/block/<block_id> GET All metrics for one block.
/block/query POST Blocks matching a MongoDB-style filter.
curl -X GET http://<server-url>/block/<block-id>

curl -X POST http://<server-url>/block/query \
  -H "Content-Type: application/json" \
  -d '{
    "metrics.latency.latency": {"$lt": 3}
  }'

Global block metrics database

Each cluster's local database periodically reports to the global block metrics database (global-block-metrics, NodePort 30201), which covers every block in the network.

Endpoint Method Returns
/block/<block_id> GET Metrics for one block.
/block/query POST Blocks matching a MongoDB-style filter.
curl -X GET http://<server-url>/block/<block-id>

curl -X POST http://<server-url>/block/query \
  -H "Content-Type: application/json" \
  -d '{
    "metrics.latency.latency": {"$lt": 3}
  }'

vDAG metrics

Each vDAG controller reports directly to the global vDAG metrics database (global-vdag-metrics, NodePort 30203), at fixed intervals.

Name Description
inference_requests_total Total inference requests processed
inference_fps Inference throughput (frames per second)
inference_latency_seconds Latency per request (seconds)
Endpoint Method Returns
/vdag/<vdag_id> GET Metrics for one vDAG.
/vdag/query POST vDAGs matching a MongoDB-style filter.
curl -X GET http://<server-url>/vdag/<vdag-id>

curl -X POST http://<server-url>/vdag/query \
  -H "Content-Type: application/json" \
  -d '{
    "status": { "$eq": "healthy" },
    "metrics.latency": { "$lt": 200 }
  }'