Inference server¶
An inference server is a gateway for sending tasks to blocks and getting results back. You can target one block by ID, let a search pick the block for you, or chain several blocks into an ad-hoc graph. The blocks can be on any cluster in the network.
Anyone with the infrastructure and a public IP or DNS name can run an inference server. Servers are listed in a network-wide inference server registry, so users can find them.
At a glance
Who this is for: application developers sending tasks (see also the Inference user guide), and operators running their own inference server.
Protocol: gRPC.
To deploy one: see Join a cluster: inference server.
Features¶
- gRPC API for submitting tasks.
- Three ways to choose a target: a fixed
block_id, a search query, or an ad-hoc graph of blocks. - Graphs can span clusters in the same network.
- Quotas, through a quota checker policy.
- Optional request logging to TimescaleDB, with query APIs.
- Public discovery through the inference server registry.
Architecture¶

(Download the architecture diagram)
How a task flows¶
- A client submits a task over gRPC. The task has a
session_idand a sequence number, plus a targetblock_id, a search query, or a graph. - The quota checker policy decides whether the request is allowed.
- If a search query was given, the server runs the search to find a block, and caches the result for the session.
- If a graph was given, the server looks up connection details for every block in it and caches them.
- The task is executed. The results collector waits for the result on a Redis queue. Connections are cached for reuse.
- The result is returned to the client.
Components¶
| Component | What it does |
|---|---|
| gRPC API | Accepts tasks in a defined protobuf format. |
| Inference query executor | Runs the quota checker policy. If it passes, the task goes to the query resolver; otherwise an error is returned. |
| Query resolver | Works out the target block. It is skipped if block_id is given. For a search query, it uses the cache to find the block for this session. For a graph, it uses the cache to resolve every block in the graph. |
| Cache | Session cache: maps session_id → (block_id, connection_url), so a search runs only once per session. On a cache miss, it runs the search on the similarity search server, then reads the block's connection details from the blocks DB. Graph cache: stores the resolved connection details for every block in a graph, per session, so the graph isn't re-resolved on every task. |
| Block connector and client | Opens and caches a connection to each block (block_id → connection object), then sends tasks over it. It uses Redis when the block is in the same cluster as the inference server, and gRPC otherwise. |
| Results collector | Picks up finished results from the Redis queue and passes them back to the client. |
| Request logging (optional) | Stores every request in TimescaleDB, with a REST API for querying the logs by time range and other filters. |
gRPC inference guide¶
syntax = "proto3";
message AIOSPacket {
string session_id = 1; // Session identifier; useful for stateful inference
uint64 seq_no = 2; // Sequence number, for sequential or stateful inference
string data = 4; // Data (output data in a response)
double ts = 5; // Timestamp in UNIX epoch format (optional)
string output_ptr = 6; // The graph structure (optional; see "Inference using graphs")
repeated FileInfo files = 7; // Files, each a FileInfo (optional)
}
message FileInfo {
string metadata = 1; // JSON serialized metadata
bytes file_data = 2; // File data
}
message BlockInferencePacket {
string block_id = 1; // Optional: the target block, for single-block inference
string session_id = 3; // Session identifier; useful for stateful inference
uint64 seq_no = 4; // Sequence number, for sequential or stateful inference
bytes frame_ptr = 5; // Frame pointer (optional; see below)
string data = 6; // Input data
string query_parameters = 7; // Search query (used if block_id is not given)
double ts = 8; // Timestamp in UNIX epoch format
repeated FileInfo files = 9; // Files, each a FileInfo (optional)
string output_ptr = 10; // The graph structure (optional; see "Inference using graphs")
}
service BlockInferenceService {
rpc infer(BlockInferencePacket) returns (AIOSPacket);
}
BlockInferencePacket fields¶
| Field | Required | Description |
|---|---|---|
block_id |
No | The block to run the task. Use it when you know the block in advance. |
session_id |
Yes | Identifies an inference session, so blocks can keep per-session state. |
seq_no |
Yes | Sequence number of the task within the session. (session_id, seq_no) uniquely identifies a task and can be used to keep inputs in order. |
frame_ptr |
No | For very large inputs: a JSON reference to a file in FrameDB, with framedb_id (the FrameDB's ID) and key (the file's key). |
data |
Yes | The task input, usually a JSON string. |
query_parameters |
No | Instead of block_id: a JSON similarity search query that selects a block dynamically. See Search and filter. |
ts |
Yes | When the task was created, as a UNIX timestamp (float). |
files |
No | Small files (images, audio, PDFs) as FileInfo entries: optional JSON metadata plus binary file_data. |
output_ptr |
No | Only for graph inference. |
Inference examples¶
1. Inference with specified block_id¶
The simplest case: you know which block you want.
import grpc
import time
import inference_pb2
import inference_pb2_grpc
SERVER_ADDRESS = "<server-url>"
def create_inference_request():
request = inference_pb2.BlockInferencePacket(
block_id="blk-weugwtns", # Fixed block ID
session_id="sess-12345", # Unique session ID
seq_no=1, # First message in the sequence
data='''{
"messages": [
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "How does photosynthesis work?"},
{"role": "assistant", "content": "Photosynthesis is the process plants use to convert sunlight into energy."},
{"role": "user", "content": "Can you explain it in simpler terms?"}
]
}''', # Chat messages with history
ts=time.time(), # Current UNIX timestamp
query_parameters="", # Not needed since block_id is fixed
output_ptr="", # No graph inference
files=[] # No files attached
)
return request
def main():
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
response = stub.infer(create_inference_request())
print("Inference Response:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
2. Inference with block_id and files attached as FileInfo elements¶
Attach images, audio or documents to the task.
import grpc
import time
import inference_pb2
import inference_pb2_grpc
SERVER_ADDRESS = "<server-url>"
def load_image(file_path):
"""Loads an image as binary data."""
with open(file_path, "rb") as f:
return f.read()
def create_object_detection_request():
"""Creates an object detection request with bounding boxes and image files."""
data_json = '''{
"task": "object_detection",
"bounding_boxes": [
{"file_index": 0, "x": 50, "y": 100, "width": 200, "height": 150},
{"file_index": 1, "x": 30, "y": 60, "width": 180, "height": 140}
]
}'''
image1 = load_image("image1.jpg")
image2 = load_image("image2.jpg")
request = inference_pb2.BlockInferencePacket(
block_id="blk-xhhwhwu", # Fixed block ID
session_id="sess-67890", # Unique session ID
seq_no=1, # First task in sequence
data=data_json, # Bounding boxes
ts=time.time(), # Current timestamp
query_parameters="", # Not needed since block_id is fixed
output_ptr="", # No graph inference
files=[
inference_pb2.FileInfo(metadata='{"file_name": "image1.jpg"}', file_data=image1),
inference_pb2.FileInfo(metadata='{"file_name": "image2.jpg"}', file_data=image2)
]
)
return request
def main():
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
response = stub.infer(create_object_detection_request())
print("Inference Response:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
3. Inference with query_parameters using similarity search¶
Don't know the block ID? Describe what you need, and a search, run by a ranking policy over a filter, picks a matching block. The result is cached for the session, so the search runs only once per session. See Search and filter for the query format.
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc
SERVER_ADDRESS = "<server-url>"
def create_chat_request():
"""Creates an LLM chat request that selects its block by search."""
data_json = json.dumps({
"task": "llm_chat",
"messages": [
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": "Hello! How are you?"},
{"role": "assistant", "content": "I'm doing well! How can I assist you today?"},
{"role": "user", "content": "Tell me about black holes."}
]
})
# Similarity search: pick an LLM block whose model name contains "gpt", in cluster-123
query_parameters_json = json.dumps({
"body": {
"values": {
"matchType": "cluster",
"rankingPolicyRule": {
"values": {
"executionMode": "code",
"policyRuleURI": "policies.search.object-det-selector:v0.0.01-stable",
"settings": {},
"parameters": {
"filterRule": {
"matchType": "block",
"filter": {
"clusterQuery": {
"variable": "id",
"operator": "==",
"value": "cluster-123"
},
"blockQuery": {
"logicalOperator": "AND",
"conditions": [
{
"variable": "component.metadata.algorithmType",
"operator": "==",
"value": "llm"
},
{
"variable": "component.metadata.model",
"operator": "LIKE",
"value": "gpt"
}
]
}
}
}
}
}
}
}
}
})
request = inference_pb2.BlockInferencePacket(
session_id="sess-12345", # Unique session ID
seq_no=1, # First message in sequence
data=data_json, # Chat history
ts=time.time(), # Current timestamp
query_parameters=query_parameters_json, # Block selection
output_ptr="", # No graph inference
files=[] # No files attached
)
return request
def main():
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
response = stub.infer(create_chat_request())
print("Inference Response:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
4. Inference with FrameDB by specifying frame_ptr¶
For very large inputs, such as video, store the data in FrameDB first and send only a pointer.
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc
SERVER_ADDRESS = "<server-url>:50051"
def create_video_detection_request():
"""Creates an object detection request for part of a video stored in FrameDB."""
# Which part of the video to process, in seconds
data_json = json.dumps({
"task": "object_detection",
"video_interval": {
"start_time": 90, # 1 min 30 sec
"end_time": 210 # 3 min 30 sec
}
})
# FrameDB reference for the video
frame_ptr_json = json.dumps({
"framedb_id": "framdb-123",
"key": "video-abcdef-123"
})
request = inference_pb2.BlockInferencePacket(
block_id="blk-wexaans", # Fixed block ID
session_id="sess-67890", # Unique session ID
seq_no=1, # First sequence
frame_ptr=frame_ptr_json.encode("utf-8"), # FrameDB reference
data=data_json, # Video processing details
ts=time.time(), # Current timestamp
output_ptr="", # No graph inference
files=[] # No direct file attachments
)
return request
def main():
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
response = stub.infer(create_video_detection_request())
print("Inference Response:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
Inference using graphs¶
You can chain known blocks for a single request by putting a graph in output_ptr. Each key is a block ID, and its value lists the blocks that receive its output. Set block_id to the first block.
{
"graph": {
"blk-ksshxpiy": ["blk-bfl3gbd5"],
"blk-bfl3gbd5": ["blk-tsonq3qr"],
"blk-tsonq3qr": []
}
}
This means: blk-ksshxpiy → blk-bfl3gbd5 → blk-tsonq3qr.
Ad-hoc graphs vs vDAGs
Ad-hoc graphs are quick to use, but unlike vDAGs they have no pre- and post-processing policies, quotas or health checks. Each block's output must already be usable as the next block's input.
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc
SERVER_ADDRESS = "<server-url>"
def create_graph_inference_request():
"""Object detection → tracking → pose estimation, as an ad-hoc graph."""
data_json = json.dumps({
"task": "object_detection_tracking_pose_estimation",
"video_interval": {
"start_time": 90, # 1 min 30 sec
"end_time": 210 # 3 min 30 sec
}
})
# FrameDB reference for the video
frame_ptr_json = json.dumps({
"framedb_id": "framdb-123",
"key": "video-abcdef-123"
})
# The graph
graph_json = json.dumps({
"graph": {
"blk-ksshxpiy": ["blk-bfl3gbd5"], # Object detection → tracking
"blk-bfl3gbd5": ["blk-tsonq3qr"], # Tracking → pose estimation
"blk-tsonq3qr": [] # End of pipeline
}
})
request = inference_pb2.BlockInferencePacket(
block_id="blk-ksshxpiy", # First block (object detection)
session_id="sess-67890", # Unique session ID
seq_no=1, # First sequence
frame_ptr=frame_ptr_json.encode("utf-8"), # FrameDB reference
data=data_json, # Video processing details
ts=time.time(), # Current timestamp
output_ptr=graph_json, # The graph
files=[] # No direct file attachments
)
return request
def main():
channel = grpc.insecure_channel(SERVER_ADDRESS)
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
response = stub.infer(create_graph_inference_request())
print("Inference Response:")
print("Session ID:", response.session_id)
print("Sequence No:", response.seq_no)
print("Data:", response.data)
print("Timestamp:", response.ts)
if __name__ == "__main__":
main()
Request logging¶
If logging is enabled, every request is stored in a TimescaleDB table. Files are not logged, to save storage.
CREATE TABLE IF NOT EXISTS block_inference (
request_id UUID PRIMARY KEY,
session_id TEXT,
block_id TEXT,
seq_no INTEGER,
ts TIMESTAMPTZ,
frame_ptr BYTEA,
data BYTEA,
query_parameters BYTEA,
output_ptr BYTEA
);
-- Time-series index on ts
SELECT create_hypertable('block_inference', 'ts', if_not_exists => TRUE);
Log query APIs¶
| Endpoint | Method | Returns |
|---|---|---|
/logs/session/<session_id> |
GET |
All logs for a session |
/logs/request/<request_id> |
GET |
Logs for one request |
/logs/block/<block_id> |
GET |
All logs for a block |
/logs/session_block?session_id=<session_id>&block_id=<block_id> |
GET |
Logs for a session on a given block |
curl -X GET "<server-url>/logs/session/<session_id>"
curl -X GET "<server-url>/logs/request/<request_id>"
curl -X GET "<server-url>/logs/block/<block_id>"
curl -X GET "<server-url>/logs/session_block?session_id=<session_id>&block_id=<block_id>"
Inference server registry¶
The registry lists inference servers so that users can find them. Add your server to it if you want it to be publicly usable. The registry is served by adhoc-server-registry-svc on the management cluster (NodePort 30105).
Schema¶
@dataclass
class InferenceServer:
inference_server_id: str = field(default_factory=lambda: str(uuid.uuid4()))
inference_server_name: str = ''
inference_server_metadata: Dict[str, str] = field(default_factory=dict)
inference_server_tags: List[str] = field(default_factory=list)
inference_server_public_url: str = ''
| Field | Description |
|---|---|
inference_server_id |
Unique ID, generated automatically (UUID). |
inference_server_name |
A human-readable name. |
inference_server_metadata |
Extra details, such as region, availability zone, provider, cluster ID, hardware and quota limits. |
inference_server_tags |
Tags for categorizing and searching. |
inference_server_public_url |
The public address clients connect to. |
APIs¶
| Method | Endpoint | Description |
|---|---|---|
POST |
/inference_server |
Register a server |
GET |
/inference_server/<server_id> |
Get a server |
PUT |
/inference_server/<server_id> |
Update a server (MongoDB-style update) |
DELETE |
/inference_server/<server_id> |
Remove a server |
POST |
/inference_servers |
Search servers (MongoDB-style query) |
curl -X POST <server-url>/inference_server \
-H "Content-Type: application/json" \
-d '{
"inference_server_name": "inference-server-us-east-1",
"inference_server_metadata": {
"region": "us-east-1",
"availability_zone": "us-east-1a",
"provider": "AWS",
"cluster_id": "cluster-123",
"quota_management_data": {
"requests_per_second_total": 100,
"requests_per_second_per_session": 10,
"requests_per_second_per_block_id": 10,
"requests_per_session_id": 1000,
"requests_per_block_id": 100
}
},
"inference_server_tags": ["NLP", "Transformer", "BERT"],
"inference_server_public_url": "https://us-east-1.inference.example.com"
}'
# Change region and provider
curl -X PUT <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd \
-H "Content-Type: application/json" \
-d '{
"$set": {
"inference_server_metadata.region": "us-west-2",
"inference_server_metadata.provider": "Google Cloud"
}
}'
# Add a tag
curl -X PUT <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd \
-H "Content-Type: application/json" \
-d '{
"$push": {
"inference_server_tags": "LLM"
}
}'
# Servers in us-east-1a with GPUs
curl -X POST <server-url>/inference_servers \
-H "Content-Type: application/json" \
-d '{
"inference_server_metadata.availability_zone": "us-east-1a",
"inference_server_metadata.gpu_available": true
}'
# Servers hosted on AWS
curl -X POST <server-url>/inference_servers \
-H "Content-Type: application/json" \
-d '{
"inference_server_metadata.provider": "AWS"
}'
# Servers tagged NLP or LLM
curl -X POST <server-url>/inference_servers \
-H "Content-Type: application/json" \
-d '{
"inference_server_tags": { "$in": ["NLP", "LLM"] }
}'
# Servers registered after 2025-01-01
curl -X POST <server-url>/inference_servers \
-H "Content-Type: application/json" \
-d '{
"registered_at": { "$gte": "2025-01-01T00:00:00Z" }
}'