Skip to content

Quickstart tutorial

This tutorial walks through the most common tasks in AIGr.id using ready-made examples. Along the way you will set up infrastructure, deploy AI models as blocks, chain blocks into a vDAG workflow, and send tasks to both.

At a glance

Who this is for: developers and operators trying AIGr.id for the first time.
You need: the prerequisites from Get started, and access to a network (yours or someone else's).
Examples used: YOLOv5 object detection, Mistral 7B chat, a vLLM-backed chat block and a pose-estimation vDAG. All live in services/applications/examples/.

Work in progress

Some examples refer to sample policies and a sample container registry (registry.ai-platform.com) that will be published with the testnet release. Until then, write your own policies and use your own container registry.

What you'll do

Part Task Typical role
Part 1: Set up infrastructure Create a network, join a cluster, join a node Network and cluster admins
Part 2: Deploy AI blocks YOLOv5 on one GPU, Mistral 7B across GPUs, a block backed by vLLM Block creators
Part 3: Compose a workflow Deploy a vDAG and send it tasks vDAG creators, app developers
Part 4: Advanced Install an external system with an init container, split an LLM across the network Advanced developers

Already have a network to use? Skip to Part 2.


Set your environment

The example scripts read the network's addresses from environment variables. Set them once in your terminal. Values are host:port, without http://:

# The network's Parser (port 30500 on the management cluster)
export SERVER_URL=<management-node-ip>:30500

# The network's Cluster Controller Gateway (port 30600 on the management cluster)
export CLUSTER_CONTROLLER_GATEWAY_URL=<management-node-ip>:30600

Part 1: Set up infrastructure

Create a new network

Skip this if you are joining an existing network.

A network begins with a management cluster that hosts all management services. On a prepared Kubernetes cluster:

# 1. Choose and label nodes for the registry and metrics databases
kubectl label node <registry-node-name> registry=yes --overwrite
kubectl label node <metrics-node-name> metrics-db=yes --overwrite

# 2. (Optional) set replica counts and sizes at the top of k8s/install_mgmt_services.sh

# 3. Install everything (asks before each step)
bash k8s/install_mgmt_services.sh

Full instructions, prerequisites and the list of installed services: Create a network.

Join a cluster to the network

Prepare the cluster first: install Ambassador on NodePort 32000 and label each node with a nodeID. Both steps are explained in Join a cluster: prerequisites. Then:

cd k8s/onboarding/cluster

# 1. On every GPU node: publish GPU details as a node annotation
sh ./gpu_info.sh

# 2. Edit config.json (cluster ID, region, contact details, configuration)

# 3. Register the cluster with the network
./onboard.sh config.json --onboard --api-url http://$SERVER_URL

Then install the AIOS cluster services through the Gateway's create-cluster-infra API. See Join a cluster: step 3.

Tip

Run ./onboard.sh config.json without --onboard to preview the final cluster description (written to <cluster_id>_populated.json) without registering.

Join a node to a cluster

On the machine you want to add, with kubeadm installed at the cluster's version:

cd k8s/onboarding/node

./join-node.sh \
  --node-id node-001 \
  --cluster-id cluster-abc \
  --api-url http://$SERVER_URL \
  --kubeadm-join-cmd "<value of clusterMetadata.additionalInfo.joinCommand>"

Details and network requirements: Join a node.


Part 2: Deploy AI blocks

Deploying a model always follows the same two steps:

flowchart LR
    A["Your code + Dockerfile"] -->|"build_and_push_image.sh"| B["Container image<br/>in a registry"]
    B -->|"create_component.sh<br/>(Parser: addComponent)"| C["Component<br/>registered"]
    C -->|"create_block.sh<br/>(Parser: createBlock)"| D["Block running<br/>on a cluster"]
    D -->|"gRPC"| E["Your app"]
  1. Onboard the component. Build the container image, push it, and register the component (component.json).
  2. Create the block. Submit block.json. Its policies choose a cluster, nodes and GPUs, and set scaling limits.

Before running an example

Each example's build_and_push_image.sh tags the image as registry.ai-platform.com/.... Change the tag to your own registry, and update the image name in component.json to match.

Deploy a block on a single GPU (YOLOv5)

An object-detection block. Each instance uses one GPU, and the block can grow to five instances.

1. Onboard the component

cd services/applications/examples/yolov5-block

# Build and push the container image
sh ./build_and_push_image.sh

# Register the component
sh ./create_component.sh

2. Create the block

sh ./create_block.sh

What the example's specification asks for:

Decision Rule
Cluster At least 5 GPUs in total, from vendor dma-bangalore; average 15-minute CPU load below 10; at least 9 GB of free GPU memory
Node / GPU GPUs with less than 80% utilization and at least 5 GB free memory
Scaling 1 to 5 instances

If it succeeds, a block with ID detector-yolo-88 is created.

3. Send it tasks

Use any available inference server as the gateway. This script sends two images, each with a region of interest:

import grpc
import time
import inference_pb2
import inference_pb2_grpc
import json

SERVER_ADDRESS = "<server-url>"  # use any of the available inference server gateways

def load_image(file_path):
    """Loads an image as binary data."""
    with open(file_path, "rb") as f:
        return f.read()

def create_yolo_request(image_bytes, bbox, file_name, session_id, seq_no):
    """
    Creates a YOLOv5 inference request with region of interest and one image file.
    """
    x1 = bbox["x"]
    y1 = bbox["y"]
    x2 = x1 + bbox["width"]
    y2 = y1 + bbox["height"]

    data_dict = {
        "roi": {
            "x1": x1,
            "y1": y1,
            "x2": x2,
            "y2": y2
        }
    }

    request = inference_pb2.BlockInferencePacket(
        block_id="detector-yolo-88",  # Target block ID
        session_id=session_id,
        seq_no=seq_no,
        data=json.dumps(data_dict),
        ts=time.time(),
        query_parameters="",
        output_ptr="",
        files=[
            inference_pb2.FileInfo(
                metadata=json.dumps({"file_name": file_name}),
                file_data=image_bytes
            )
        ]
    )

    return request

def main():
    # Sample bounding boxes for each image
    bounding_boxes = [
        {"file_path": "image1.jpg", "file_name": "image1.jpg", "x": 50, "y": 100, "width": 200, "height": 150},
        {"file_path": "image2.jpg", "file_name": "image2.jpg", "x": 30, "y": 60, "width": 180, "height": 140}
    ]

    # Create gRPC channel and stub
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    session_id = "sess-67890"

    for i, bbox in enumerate(bounding_boxes):
        image_bytes = load_image(bbox["file_path"])
        request = create_yolo_request(
            image_bytes=image_bytes,
            bbox=bbox,
            file_name=bbox["file_name"],
            session_id=session_id,
            seq_no=i + 1
        )

        response = stub.infer(request)

        print(f"\nInference Response for image {i + 1}:")
        print("Session ID:", response.session_id)
        print("Sequence No:", response.seq_no)
        print("Data:", response.data)
        print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

Deploy a block across multiple GPUs (Mistral 7B)

A chat model that spans 3 GPUs on one node. It is built with the AIOS Instance SDK and the LLM SDK utilities.

1. Onboard the component

cd services/applications/examples/mistral7b

# Build and push the container image
sh ./build_and_push_image.sh

# Register the component
sh ./create_component.sh

2. Create the block

sh ./create_block.sh

What the example's specification asks for:

Decision Rule
Cluster At least 3 GPUs in total, from vendor dma-bangalore; average 15-minute CPU load below 10; at least 48 GB of free GPU memory
Node / GPU GPUs with less than 90% utilization and at least 16 GB free memory
Scaling 1 to 3 instances

If it succeeds, a block with ID mistral-chat-88 is created.

3. Send it tasks

import grpc
import time
import inference_pb2
import inference_pb2_grpc
import json

# Replace with a valid AIOS inference server address
SERVER_ADDRESS = "<server-url>"

def create_chat_request(message, session_id, seq_no):
    """
    Creates a Mistral 7B chat inference request.
    """
    data_dict = {
        "message": message,
        "session_id": session_id
    }

    request = inference_pb2.BlockInferencePacket(
        block_id="mistral-chat-88",  # Target block ID
        session_id=session_id,
        seq_no=seq_no,
        data=json.dumps(data_dict),
        ts=time.time(),
        query_parameters="",
        output_ptr="",
        files=[]
    )

    return request

def main():
    # Sample user inputs
    messages = [
        "Hello! What can you do?",
        "Can you explain the concept of quantum entanglement?",
        "What’s the capital of Japan?"
    ]

    # Create gRPC channel and stub
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    session_id = "sess-chat-001"

    for i, msg in enumerate(messages):
        request = create_chat_request(
            message=msg,
            session_id=session_id,
            seq_no=i + 1
        )

        response = stub.infer(request)

        print(f"\nResponse for message {i + 1}:")
        print("Session ID:", response.session_id)
        print("Sequence No:", response.seq_no)
        print("Reply:", json.loads(response.data).get("reply", ""))
        print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

Connect a block to an external vLLM server

Here the block does no inference itself. Its instances forward each chat message to a vLLM server that you run separately, and return the reply. This is the externally managed third-party system pattern.

Before you start: run a vLLM server in OpenAI-compatible mode (/v1/chat/completions), serving mistralai/Mistral-7B-Instruct-v0.1. The block expects it at:

"vllm_server_url": "http://vllm-service:8080"

It must be reachable from inside the cluster. If it isn't, expose it through an internal service or a tunnel, or change vllm_server_url in block.json.

1. Onboard the component

cd services/applications/examples/mistral7b-vllm

# Build and push the container image
sh ./build_and_push_image.sh

# Register the component
sh ./create_component.sh

2. Create the block

sh ./create_block.sh

What the example's specification asks for:

Decision Rule
Resources CPU only, no GPU needed: at least 2 CPUs and about 2 GB RAM per instance
Scaling 1 to 2 instances, CPU-based auto-scaling
Load balancing Round-robin, with session caching

If it succeeds, a block with ID vllm-chat-01 is created.

3. Send it tasks

import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc

# Replace with the actual gateway host:port
SERVER_ADDRESS = "<inference-gateway-host>:<port>"

BLOCK_ID = "vllm-chat-01"  # Block ID from block.json
SESSION_ID = "chat-session-001"

def create_chat_request(message: str, session_id: str, seq_no: int):
    """
    Creates a chat inference request for the vLLMChatBlock.
    """
    data_dict = {
        "message": message,
        "session_id": session_id
    }

    request = inference_pb2.BlockInferencePacket(
        block_id=BLOCK_ID,
        session_id=session_id,
        seq_no=seq_no,
        data=json.dumps(data_dict),
        ts=time.time(),
        query_parameters="",
        output_ptr="",
        files=[]
    )

    return request

def main():
    # Example user messages
    messages = [
        "Hello, what is your name?",
        "Can you explain quantum physics in simple terms?",
        "What's the capital of France?"
    ]

    # Create gRPC channel and stub
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    for i, msg in enumerate(messages):
        request = create_chat_request(msg, SESSION_ID, seq_no=i + 1)

        try:
            response = stub.infer(request)
            print(f"\nResponse {i + 1}:")
            print("Session ID:", response.session_id)
            print("Sequence No:", response.seq_no)
            print("Reply:", json.loads(response.data).get("reply", "<no reply>"))
            print("Timestamp:", response.ts)
        except grpc.RpcError as e:
            print(f"\nError during inference: {e}")

if __name__ == "__main__":
    main()

Part 3: Compose a workflow (vDAG)

Deploy a vDAG and send it tasks

A vDAG chains existing blocks into one workflow. This example builds a pose-estimation pipeline from an object detector and a tracker.

Assumptions: blocks object-det-1 and tracker-122 already exist, and the policies the spec refers to are available.

1. Create the vDAG

cd services/applications/examples/vdag-pose-estimation
sh ./create_vdag.sh

This creates the vDAG pose-estimator:v0.0.1-stable.

2. Deploy a controller for it

A vDAG cannot accept tasks until it has a controller. Deploy one on the cluster of your choice:

sh ./create_controller.sh

This creates controller estimator-001 on cluster cluster-123. Edit the script to choose a different cluster.

3. Send it tasks

import grpc
import json
import time

# Import the generated gRPC classes
import vdag_pb2
import vdag_pb2_grpc

# Create a gRPC channel to the vDAG inference service
channel = grpc.insecure_channel("http://<cluster-public-ip>:32000/vdag/estimator-001")
stub = vdag_pb2_grpc.vDAGInferenceServiceStub(channel)

# Example session and sequence information
session_id = "session-5678"
seq_no = 1

# Define the area of interest (bounding box coordinates)
# Format: [x_min, y_min, x_max, y_max]
area_of_interest = {
    "region_of_interest": {
        "x_min": 120,
        "y_min": 200,
        "x_max": 450,
        "y_max": 650
    }
}

# Load the image file to send as part of inference
with open("example_image.jpg", "rb") as img_file:
    image_bytes = img_file.read()

# Create vDAGFileInfo message with image
image_file = vdag_pb2.vDAGFileInfo(
    metadata=json.dumps({
        "filename": "example_image.jpg",
        "content_type": "image/jpeg"
    }),
    file_data=image_bytes
)

# Construct the vDAGInferencePacket
packet = vdag_pb2.vDAGInferencePacket(
    session_id=session_id,              # Unique session ID
    seq_no=seq_no,                      # Sequence number for packet ordering
    data=json.dumps(area_of_interest),  # Area of interest in JSON
    ts=time.time(),                     # Timestamp
    files=[image_file]                  # List of attached files
)

# Send inference request to the server
response = stub.infer(packet)

# Handle the response
print("Response received.")
print("Data:", response.data)

# Print details of any returned files
for f in response.files:
    metadata = json.loads(f.metadata)
    print(f"Returned file: {metadata.get('filename')} (type: {metadata.get('content_type')})")
    # Save returned file if needed
    with open(f"output_{metadata.get('filename')}", "wb") as out_file:
        out_file.write(f.file_data)

Learn more: vDAG spec and vDAG controller.


Part 4: Advanced patterns

Note

Ready-to-run examples for this part will be published with the testnet release. The steps below show the workflow.

Deploy an external system with an init container

In the vLLM example you started the external server yourself. AIOS can instead install it for you when the block is created, and remove it when the block is deleted. To do that, you attach an init container to the component. An init container is a small Python program, built with the SDK, that contains the installation logic.

  1. Write the init container with the Python SDK. See Third-party systems.
  2. Create a component that names the init container. See Component spec with init container.
  3. Create the block as usual. AIOS deploys the external system alongside it and passes the system's URL to the block in the blockInitData.initContainer field. See Accessing the third-party system.

Split an LLM across the network

A model too large for one node, or even one cluster, can be split into chunks that run as separate blocks connected by a vDAG. Because different frameworks and architectures split differently, AIOS lets you write your own splitting logic with the model splitter SDK. The splitter produces a model plan: a block specification for each chunk, plus a vDAG specification that connects them.

  1. Write your splitting code with the model splitter SDK.
  2. Run it as a splitting task. See Model splitting task runners. The result is the model plan.
  3. Deploy the blocks (Block spec) and the vDAG (vDAG spec) from the plan.

Next steps