Quickstart tutorial¶
This tutorial walks through the most common tasks in AIGr.id using ready-made examples. Along the way you will set up infrastructure, deploy AI models as blocks, chain blocks into a vDAG workflow, and send tasks to both.
At a glance
Who this is for: developers and operators trying AIGr.id for the first time.
You need: the prerequisites from Get started, and access to a network (yours or someone else's).
Examples used: YOLOv5 object detection, Mistral 7B chat, a vLLM-backed chat block and a pose-estimation vDAG. All live in services/applications/examples/.
Work in progress
Some examples refer to sample policies and a sample container registry (registry.ai-platform.com) that will be published with the testnet release. Until then, write your own policies and use your own container registry.
What you'll do¶
| Part | Task | Typical role |
|---|---|---|
| Part 1: Set up infrastructure | Create a network, join a cluster, join a node | Network and cluster admins |
| Part 2: Deploy AI blocks | YOLOv5 on one GPU, Mistral 7B across GPUs, a block backed by vLLM | Block creators |
| Part 3: Compose a workflow | Deploy a vDAG and send it tasks | vDAG creators, app developers |
| Part 4: Advanced | Install an external system with an init container, split an LLM across the network | Advanced developers |
Already have a network to use? Skip to Part 2.
Set your environment¶
The example scripts read the network's addresses from environment variables. Set them once in your terminal. Values are host:port, without http://:
# The network's Parser (port 30500 on the management cluster)
export SERVER_URL=<management-node-ip>:30500
# The network's Cluster Controller Gateway (port 30600 on the management cluster)
export CLUSTER_CONTROLLER_GATEWAY_URL=<management-node-ip>:30600
Part 1: Set up infrastructure¶
Create a new network¶
Skip this if you are joining an existing network.
A network begins with a management cluster that hosts all management services. On a prepared Kubernetes cluster:
# 1. Choose and label nodes for the registry and metrics databases
kubectl label node <registry-node-name> registry=yes --overwrite
kubectl label node <metrics-node-name> metrics-db=yes --overwrite
# 2. (Optional) set replica counts and sizes at the top of k8s/install_mgmt_services.sh
# 3. Install everything (asks before each step)
bash k8s/install_mgmt_services.sh
Full instructions, prerequisites and the list of installed services: Create a network.
Join a cluster to the network¶
Prepare the cluster first: install Ambassador on NodePort 32000 and label each node with a nodeID. Both steps are explained in Join a cluster: prerequisites. Then:
cd k8s/onboarding/cluster
# 1. On every GPU node: publish GPU details as a node annotation
sh ./gpu_info.sh
# 2. Edit config.json (cluster ID, region, contact details, configuration)
# 3. Register the cluster with the network
./onboard.sh config.json --onboard --api-url http://$SERVER_URL
Then install the AIOS cluster services through the Gateway's create-cluster-infra API. See Join a cluster: step 3.
Tip
Run ./onboard.sh config.json without --onboard to preview the final cluster description (written to <cluster_id>_populated.json) without registering.
Join a node to a cluster¶
On the machine you want to add, with kubeadm installed at the cluster's version:
cd k8s/onboarding/node
./join-node.sh \
--node-id node-001 \
--cluster-id cluster-abc \
--api-url http://$SERVER_URL \
--kubeadm-join-cmd "<value of clusterMetadata.additionalInfo.joinCommand>"
Details and network requirements: Join a node.
Part 2: Deploy AI blocks¶
Deploying a model always follows the same two steps:
flowchart LR
A["Your code + Dockerfile"] -->|"build_and_push_image.sh"| B["Container image<br/>in a registry"]
B -->|"create_component.sh<br/>(Parser: addComponent)"| C["Component<br/>registered"]
C -->|"create_block.sh<br/>(Parser: createBlock)"| D["Block running<br/>on a cluster"]
D -->|"gRPC"| E["Your app"]
- Onboard the component. Build the container image, push it, and register the component (
component.json). - Create the block. Submit
block.json. Its policies choose a cluster, nodes and GPUs, and set scaling limits.
Before running an example
Each example's build_and_push_image.sh tags the image as registry.ai-platform.com/.... Change the tag to your own registry, and update the image name in component.json to match.
Deploy a block on a single GPU (YOLOv5)¶
An object-detection block. Each instance uses one GPU, and the block can grow to five instances.
1. Onboard the component
cd services/applications/examples/yolov5-block
# Build and push the container image
sh ./build_and_push_image.sh
# Register the component
sh ./create_component.sh
2. Create the block
What the example's specification asks for:
| Decision | Rule |
|---|---|
| Cluster | At least 5 GPUs in total, from vendor dma-bangalore; average 15-minute CPU load below 10; at least 9 GB of free GPU memory |
| Node / GPU | GPUs with less than 80% utilization and at least 5 GB free memory |
| Scaling | 1 to 5 instances |
If it succeeds, a block with ID detector-yolo-88 is created.
3. Send it tasks
Use any available inference server as the gateway. This script sends two images, each with a region of interest:
import grpc
import time
import inference_pb2
import inference_pb2_grpc
import json
SERVER_ADDRESS = "<server-url>" # use any of the available inference server gateways
def load_image(file_path):
"""Loads an image as binary data."""
with open(file_path, "rb") as f:
return f.read()
def create_yolo_request(image_bytes, bbox, file_name, session_id, seq_no):
"""
Creates a YOLOv5 inference request with region of interest and one image file.
"""
x1 = bbox["x"]
y1 = bbox["y"]
x2 = x1 + bbox["width"]
y2 = y1 + bbox["height"]
data_dict = {
"roi": {
"x1": x1,
"y1": y1,
"x2": x2,
"y2": y2
}
}
request = inference_pb2.BlockInferencePacket(
block_id="detector-yolo-88", # Target block ID
session_id=session_id,
seq_no=seq_no,
data=json.dumps(data_dict),
ts=time.time(),
query_parameters="",
output_ptr="",
files=[
inference_pb2.FileInfo(
metadata=json.dumps({"file_name": file_name}),
file_data=image_bytes
)
]
)
return request
def main():
# Sample bounding boxes for each image
bounding_boxes = [
{"file_path": "image1.jpg", "file_name": "image1.jpg", "x": 50, "y": 100, "width": 200, "height": 150},
{"file_path": "image2.jpg", "file_name": "image2.jpg", "x": 30, "y": 60, "width": 180, "height": 140}
]
# Create gRPC channel and stub
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
session_id = "sess-67890"
for i, bbox in enumerate(bounding_boxes):
image_bytes = load_image(bbox["file_path"])
request = create_yolo_request(
image_bytes=image_bytes,
bbox=bbox,
file_name=bbox["file_name"],
session_id=session_id,
seq_no=i + 1
)
response = stub.infer(request)
print(f"\nInference Response for image {i + 1}:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
Deploy a block across multiple GPUs (Mistral 7B)¶
A chat model that spans 3 GPUs on one node. It is built with the AIOS Instance SDK and the LLM SDK utilities.
1. Onboard the component
cd services/applications/examples/mistral7b
# Build and push the container image
sh ./build_and_push_image.sh
# Register the component
sh ./create_component.sh
2. Create the block
What the example's specification asks for:
| Decision | Rule |
|---|---|
| Cluster | At least 3 GPUs in total, from vendor dma-bangalore; average 15-minute CPU load below 10; at least 48 GB of free GPU memory |
| Node / GPU | GPUs with less than 90% utilization and at least 16 GB free memory |
| Scaling | 1 to 3 instances |
If it succeeds, a block with ID mistral-chat-88 is created.
3. Send it tasks
import grpc
import time
import inference_pb2
import inference_pb2_grpc
import json
# Replace with a valid AIOS inference server address
SERVER_ADDRESS = "<server-url>"
def create_chat_request(message, session_id, seq_no):
"""
Creates a Mistral 7B chat inference request.
"""
data_dict = {
"message": message,
"session_id": session_id
}
request = inference_pb2.BlockInferencePacket(
block_id="mistral-chat-88", # Target block ID
session_id=session_id,
seq_no=seq_no,
data=json.dumps(data_dict),
ts=time.time(),
query_parameters="",
output_ptr="",
files=[]
)
return request
def main():
# Sample user inputs
messages = [
"Hello! What can you do?",
"Can you explain the concept of quantum entanglement?",
"What’s the capital of Japan?"
]
# Create gRPC channel and stub
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
session_id = "sess-chat-001"
for i, msg in enumerate(messages):
request = create_chat_request(
message=msg,
session_id=session_id,
seq_no=i + 1
)
response = stub.infer(request)
print(f"\nResponse for message {i + 1}:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Reply:", json.loads(response.data).get("reply", ""))
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
Connect a block to an external vLLM server¶
Here the block does no inference itself. Its instances forward each chat message to a vLLM server that you run separately, and return the reply. This is the externally managed third-party system pattern.
Before you start: run a vLLM server in OpenAI-compatible mode (/v1/chat/completions), serving mistralai/Mistral-7B-Instruct-v0.1. The block expects it at:
It must be reachable from inside the cluster. If it isn't, expose it through an internal service or a tunnel, or change vllm_server_url in block.json.
1. Onboard the component
cd services/applications/examples/mistral7b-vllm
# Build and push the container image
sh ./build_and_push_image.sh
# Register the component
sh ./create_component.sh
2. Create the block
What the example's specification asks for:
| Decision | Rule |
|---|---|
| Resources | CPU only, no GPU needed: at least 2 CPUs and about 2 GB RAM per instance |
| Scaling | 1 to 2 instances, CPU-based auto-scaling |
| Load balancing | Round-robin, with session caching |
If it succeeds, a block with ID vllm-chat-01 is created.
3. Send it tasks
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc
# Replace with the actual gateway host:port
SERVER_ADDRESS = "<inference-gateway-host>:<port>"
BLOCK_ID = "vllm-chat-01" # Block ID from block.json
SESSION_ID = "chat-session-001"
def create_chat_request(message: str, session_id: str, seq_no: int):
"""
Creates a chat inference request for the vLLMChatBlock.
"""
data_dict = {
"message": message,
"session_id": session_id
}
request = inference_pb2.BlockInferencePacket(
block_id=BLOCK_ID,
session_id=session_id,
seq_no=seq_no,
data=json.dumps(data_dict),
ts=time.time(),
query_parameters="",
output_ptr="",
files=[]
)
return request
def main():
# Example user messages
messages = [
"Hello, what is your name?",
"Can you explain quantum physics in simple terms?",
"What's the capital of France?"
]
# Create gRPC channel and stub
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
for i, msg in enumerate(messages):
request = create_chat_request(msg, SESSION_ID, seq_no=i + 1)
try:
response = stub.infer(request)
print(f"\nResponse {i + 1}:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Reply:", json.loads(response.data).get("reply", "<no reply>"))
print("Timestamp:", response.ts)
except grpc.RpcError as e:
print(f"\nError during inference: {e}")
if __name__ == "__main__":
main()
Part 3: Compose a workflow (vDAG)¶
Deploy a vDAG and send it tasks¶
A vDAG chains existing blocks into one workflow. This example builds a pose-estimation pipeline from an object detector and a tracker.
Assumptions: blocks object-det-1 and tracker-122 already exist, and the policies the spec refers to are available.
1. Create the vDAG
This creates the vDAG pose-estimator:v0.0.1-stable.
2. Deploy a controller for it
A vDAG cannot accept tasks until it has a controller. Deploy one on the cluster of your choice:
This creates controller estimator-001 on cluster cluster-123. Edit the script to choose a different cluster.
3. Send it tasks
import grpc
import json
import time
# Import the generated gRPC classes
import vdag_pb2
import vdag_pb2_grpc
# Create a gRPC channel to the vDAG inference service
channel = grpc.insecure_channel("http://<cluster-public-ip>:32000/vdag/estimator-001")
stub = vdag_pb2_grpc.vDAGInferenceServiceStub(channel)
# Example session and sequence information
session_id = "session-5678"
seq_no = 1
# Define the area of interest (bounding box coordinates)
# Format: [x_min, y_min, x_max, y_max]
area_of_interest = {
"region_of_interest": {
"x_min": 120,
"y_min": 200,
"x_max": 450,
"y_max": 650
}
}
# Load the image file to send as part of inference
with open("example_image.jpg", "rb") as img_file:
image_bytes = img_file.read()
# Create vDAGFileInfo message with image
image_file = vdag_pb2.vDAGFileInfo(
metadata=json.dumps({
"filename": "example_image.jpg",
"content_type": "image/jpeg"
}),
file_data=image_bytes
)
# Construct the vDAGInferencePacket
packet = vdag_pb2.vDAGInferencePacket(
session_id=session_id, # Unique session ID
seq_no=seq_no, # Sequence number for packet ordering
data=json.dumps(area_of_interest), # Area of interest in JSON
ts=time.time(), # Timestamp
files=[image_file] # List of attached files
)
# Send inference request to the server
response = stub.infer(packet)
# Handle the response
print("Response received.")
print("Data:", response.data)
# Print details of any returned files
for f in response.files:
metadata = json.loads(f.metadata)
print(f"Returned file: {metadata.get('filename')} (type: {metadata.get('content_type')})")
# Save returned file if needed
with open(f"output_{metadata.get('filename')}", "wb") as out_file:
out_file.write(f.file_data)
Learn more: vDAG spec and vDAG controller.
Part 4: Advanced patterns¶
Note
Ready-to-run examples for this part will be published with the testnet release. The steps below show the workflow.
Deploy an external system with an init container¶
In the vLLM example you started the external server yourself. AIOS can instead install it for you when the block is created, and remove it when the block is deleted. To do that, you attach an init container to the component. An init container is a small Python program, built with the SDK, that contains the installation logic.
- Write the init container with the Python SDK. See Third-party systems.
- Create a component that names the init container. See Component spec with init container.
- Create the block as usual. AIOS deploys the external system alongside it and passes the system's URL to the block in the
blockInitData.initContainerfield. See Accessing the third-party system.
Split an LLM across the network¶
A model too large for one node, or even one cluster, can be split into chunks that run as separate blocks connected by a vDAG. Because different frameworks and architectures split differently, AIOS lets you write your own splitting logic with the model splitter SDK. The splitter produces a model plan: a block specification for each chunk, plus a vDAG specification that connects them.
- Write your splitting code with the model splitter SDK.
- Run it as a splitting task. See Model splitting task runners. The result is the model plan.
- Deploy the blocks (Block spec) and the vDAG (vDAG spec) from the plan.
Next steps¶
- Build your own block: Instance SDK
- Control placement, scaling and access: Policies
- Find the guide for your role: Guides by role