Architecture¶
This page shows how the services of AIOS fit together. If you are new to the terms, read Key concepts first.

(Download the architecture diagram)
Three layers¶
AIOS is organized into three layers:
| Layer | Where it runs | Purpose |
|---|---|---|
| Management services | The network's management cluster (one per network) | Create, schedule, govern and keep track of everything in the network. |
| Cluster services | Every cluster that joins the network | Run and monitor workloads on that cluster's nodes. |
| Application layer | On clusters, wherever workloads are placed | The blocks, workflows and gateways that do the actual AI work and serve users. |
flowchart TB
U["Developers, admins<br/>and users"]
subgraph MGMT["Management cluster"]
P["Parser"]
CCG["Cluster Controller Gateway"]
RA["Resource Allocator"]
PS["Policies System"]
REG[("Registries")]
GM[("Global metrics")]
end
subgraph C1["Cluster (one of many)"]
CC["Cluster Controller"]
LM[("Local metrics")]
B1["Block"]
B2["Block"]
VC["vDAG controller"]
IS["Inference server"]
end
U -->|"specs (JSON)"| P
P --> CCG
CCG --> RA
CCG --> CC
CC --> B1
CC --> B2
P --> REG
PS -. "policies" .- CCG
PS -. "policies" .- CC
B1 --> LM
LM --> GM
U -->|"inference tasks"| IS
U -->|"workflow tasks"| VC
IS --> B1
VC --> B1
VC --> B2
What happens when you deploy a block¶
Following one request end to end is the easiest way to see how the services cooperate.
- You submit a block specification to the Parser. The spec names a component and the policies to use for placement and scaling.
- The Parser validates the spec, using a custom template from the Template store if you asked for one, and passes the result to the Cluster Controller Gateway.
- The Gateway runs the network's
add_blockpre-check policy. If the policy rejects the request, it stops here. - Shortlisting clusters. The block's cluster allocator policy filters and ranks clusters, using cluster data and live metrics.
- Feasibility check. Each shortlisted cluster runs the block's resource allocation policy in dry-run mode and returns a feasibility score between 0 and 1. The cluster allocator policy uses these scores to pick one cluster.
- Placement. The chosen cluster's Cluster Controller runs its own pre-check policy, then runs the resource allocation policy for real to choose nodes and GPUs, and creates the block's services and instances.
- The block is recorded in the Block registry. From then on it reports metrics and can be scaled, managed and called.
- Users send tasks to the block through an inference server, or through a vDAG controller if the block is part of a workflow.
The full details are in Block spec: allocation flow.
Management services¶
These run once per network, on the management cluster.
| Service | What it does | Docs |
|---|---|---|
| Parser | The entry point for creating things. It accepts JSON specifications for blocks, vDAGs, clusters and registry entries. It applies the standard spec format by default, and supports custom formats through templates and policies. | How the Parser works |
| Cluster Controller Gateway | Manages every cluster in the network. It sets up cluster infrastructure, schedules blocks together with each cluster's controller, relays management commands for blocks and clusters, sets up vDAG controllers and enforces pre-check policies. | Cluster Controller Gateway |
| Resource Allocator | Runs resource allocation policies to choose the cluster, node and GPUs for each block. It also supports manual dry runs. | Resource Allocator |
| Policies System | A database, registry, executors and Kubernetes interfaces for storing, finding and running policies, or deploying them as jobs or functions. | Policies |
| Spec and Template stores | Store reusable specifications and the custom parser templates the Parser uses for parsing and validation. | Spec store, Template store |
| Container registry index | A distributed system for storing container images on any cluster or machine, with a central index for discovery and upload. | Container registry |
| Assets registry index | A distributed store for assets such as model files, policy code archives, images and videos. Blocks can reference these assets. Asset stores can be hosted on any cluster or machine with object storage. The central index lists them all. | Assets registry |
| Component registry | Stores components that can be deployed as blocks. Each entry holds the instance image, any sidecar images, author metadata, input/output templates and default parameters and settings. | Component registry |
| Inference server registry | Lists every inference server deployed in the network, with its public endpoint and metadata, so users can find one and submit tasks. | Inference server |
| Runtime registries | Track what is live right now: clusters, blocks, vDAGs and vDAG controllers. | Registries |
| Global metrics | Central databases that hold metrics from every cluster, block and vDAG controller. | Metrics system |
| Tasks DB | Tracks the status of long-running, asynchronous operations. | Tasks DB |
| Failure policy executor | Runs policies that report failures to external systems. | Failure policy executor |
Cluster services¶
These are installed automatically on every cluster that joins the network.
| Service | What it does | Docs |
|---|---|---|
| Cluster Controller | Places blocks on nodes, provisions resources, monitors deployments and node health, and works with the Cluster Controller Gateway on scheduling. It enforces the cluster's own pre-check policies. | Cluster Controller |
| Local metrics system | A hardware metrics DaemonSet on every node, plus local databases for block, cluster and vDAG metrics, so policies can query them quickly. | Metrics system |
A cluster can also host optional add-ons (container registry, assets registry, policy executor, inference server, model splitter) and offer them to the network. See Deploy add-on services.
Application layer¶
This is where AI work actually happens.
| Piece | What it does | Docs |
|---|---|---|
| Block | A running deployment of a component, built with the AIOS SDK. It runs on any cluster and comes with auto-scaling, load balancing, health checks and metrics. | Block runtime, Instance SDK |
| Inference server | A gateway for sending tasks directly to blocks. Anyone can deploy one on any cluster and register it for public discovery. | Inference server |
| vDAG and vDAG controller | A vDAG is a workflow across blocks anywhere in the network. Its controller accepts tasks for the workflow and enforces health checks, quotas and quality audits through policies. | vDAG controller, vDAG spec |
| LLM support | The LLM SDK utilities build on the base SDK to serve large language models as blocks. Large models can be split and spread across nodes or clusters as a vDAG (pipeline parallelism). | LLM SDK, Model splitting |
| Third-party systems | Blocks can delegate work to external services, either deployed independently or installed next to the block by an init container. | Third-party systems |
Next steps¶
- Set up your own network: Create a network
- Walk through a full example: Quickstart tutorial
- Look up a term: Glossary