Skip to content

Inference server

An inference server is a gateway for sending tasks to blocks and getting results back. You can target one block by ID, let a search pick the block for you, or chain several blocks into an ad-hoc graph. The blocks can be on any cluster in the network.

Anyone with the infrastructure and a public IP or DNS name can run an inference server. Servers are listed in a network-wide inference server registry, so users can find them.

At a glance

Who this is for: application developers sending tasks (see also the Inference user guide), and operators running their own inference server.
Protocol: gRPC.
To deploy one: see Join a cluster: inference server.

Features

  • gRPC API for submitting tasks.
  • Three ways to choose a target: a fixed block_id, a search query, or an ad-hoc graph of blocks.
  • Graphs can span clusters in the same network.
  • Quotas, through a quota checker policy.
  • Optional request logging to TimescaleDB, with query APIs.
  • Public discovery through the inference server registry.

Architecture

Inference server architecture

(Download the architecture diagram)

How a task flows

  1. A client submits a task over gRPC. The task has a session_id and a sequence number, plus a target block_id, a search query, or a graph.
  2. The quota checker policy decides whether the request is allowed.
  3. If a search query was given, the server runs the search to find a block, and caches the result for the session.
  4. If a graph was given, the server looks up connection details for every block in it and caches them.
  5. The task is executed. The results collector waits for the result on a Redis queue. Connections are cached for reuse.
  6. The result is returned to the client.

Components

Component What it does
gRPC API Accepts tasks in a defined protobuf format.
Inference query executor Runs the quota checker policy. If it passes, the task goes to the query resolver; otherwise an error is returned.
Query resolver Works out the target block. It is skipped if block_id is given. For a search query, it uses the cache to find the block for this session. For a graph, it uses the cache to resolve every block in the graph.
Cache Session cache: maps session_id → (block_id, connection_url), so a search runs only once per session. On a cache miss, it runs the search on the similarity search server, then reads the block's connection details from the blocks DB. Graph cache: stores the resolved connection details for every block in a graph, per session, so the graph isn't re-resolved on every task.
Block connector and client Opens and caches a connection to each block (block_id → connection object), then sends tasks over it. It uses Redis when the block is in the same cluster as the inference server, and gRPC otherwise.
Results collector Picks up finished results from the Redis queue and passes them back to the client.
Request logging (optional) Stores every request in TimescaleDB, with a REST API for querying the logs by time range and other filters.

gRPC inference guide

syntax = "proto3";

message AIOSPacket {
    string session_id = 1;       // Session identifier; useful for stateful inference
    uint64 seq_no = 2;           // Sequence number, for sequential or stateful inference
    string data = 4;             // Data (output data in a response)
    double ts = 5;               // Timestamp in UNIX epoch format (optional)
    string output_ptr = 6;       // The graph structure (optional; see "Inference using graphs")
    repeated FileInfo files = 7; // Files, each a FileInfo (optional)
}

message FileInfo {
    string metadata = 1; // JSON serialized metadata
    bytes file_data = 2; // File data
}

message BlockInferencePacket {
    string block_id = 1;          // Optional: the target block, for single-block inference
    string session_id = 3;        // Session identifier; useful for stateful inference
    uint64 seq_no = 4;            // Sequence number, for sequential or stateful inference
    bytes frame_ptr = 5;          // Frame pointer (optional; see below)
    string data = 6;              // Input data
    string query_parameters = 7;  // Search query (used if block_id is not given)
    double ts = 8;                // Timestamp in UNIX epoch format
    repeated FileInfo files = 9;  // Files, each a FileInfo (optional)
    string output_ptr = 10;       // The graph structure (optional; see "Inference using graphs")
}

service BlockInferenceService {
    rpc infer(BlockInferencePacket) returns (AIOSPacket);
}

BlockInferencePacket fields

Field Required Description
block_id No The block to run the task. Use it when you know the block in advance.
session_id Yes Identifies an inference session, so blocks can keep per-session state.
seq_no Yes Sequence number of the task within the session. (session_id, seq_no) uniquely identifies a task and can be used to keep inputs in order.
frame_ptr No For very large inputs: a JSON reference to a file in FrameDB, with framedb_id (the FrameDB's ID) and key (the file's key).
data Yes The task input, usually a JSON string.
query_parameters No Instead of block_id: a JSON similarity search query that selects a block dynamically. See Search and filter.
ts Yes When the task was created, as a UNIX timestamp (float).
files No Small files (images, audio, PDFs) as FileInfo entries: optional JSON metadata plus binary file_data.
output_ptr No Only for graph inference.

Inference examples

1. Inference with specified block_id

The simplest case: you know which block you want.

import grpc
import time
import inference_pb2
import inference_pb2_grpc

SERVER_ADDRESS = "<server-url>"

def create_inference_request():
    request = inference_pb2.BlockInferencePacket(
        block_id="blk-weugwtns",   # Fixed block ID
        session_id="sess-12345",   # Unique session ID
        seq_no=1,                  # First message in the sequence
        data='''{
            "messages": [
                {"role": "system", "content": "You are a helpful AI assistant."},
                {"role": "user", "content": "How does photosynthesis work?"},
                {"role": "assistant", "content": "Photosynthesis is the process plants use to convert sunlight into energy."},
                {"role": "user", "content": "Can you explain it in simpler terms?"}
            ]
        }''',                      # Chat messages with history
        ts=time.time(),            # Current UNIX timestamp
        query_parameters="",       # Not needed since block_id is fixed
        output_ptr="",             # No graph inference
        files=[]                   # No files attached
    )
    return request

def main():
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    response = stub.infer(create_inference_request())

    print("Inference Response:")
    print("Session ID:", response.session_id)
    print("Sequence No:", response.seq_no)
    print("Data:", response.data)
    print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

2. Inference with block_id and files attached as FileInfo elements

Attach images, audio or documents to the task.

import grpc
import time
import inference_pb2
import inference_pb2_grpc

SERVER_ADDRESS = "<server-url>"

def load_image(file_path):
    """Loads an image as binary data."""
    with open(file_path, "rb") as f:
        return f.read()

def create_object_detection_request():
    """Creates an object detection request with bounding boxes and image files."""

    data_json = '''{
        "task": "object_detection",
        "bounding_boxes": [
            {"file_index": 0, "x": 50, "y": 100, "width": 200, "height": 150},
            {"file_index": 1, "x": 30, "y": 60, "width": 180, "height": 140}
        ]
    }'''

    image1 = load_image("image1.jpg")
    image2 = load_image("image2.jpg")

    request = inference_pb2.BlockInferencePacket(
        block_id="blk-xhhwhwu",    # Fixed block ID
        session_id="sess-67890",   # Unique session ID
        seq_no=1,                  # First task in sequence
        data=data_json,            # Bounding boxes
        ts=time.time(),            # Current timestamp
        query_parameters="",       # Not needed since block_id is fixed
        output_ptr="",             # No graph inference
        files=[
            inference_pb2.FileInfo(metadata='{"file_name": "image1.jpg"}', file_data=image1),
            inference_pb2.FileInfo(metadata='{"file_name": "image2.jpg"}', file_data=image2)
        ]
    )
    return request

def main():
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    response = stub.infer(create_object_detection_request())

    print("Inference Response:")
    print("Session ID:", response.session_id)
    print("Sequence No:", response.seq_no)
    print("Data:", response.data)
    print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

Don't know the block ID? Describe what you need, and a search, run by a ranking policy over a filter, picks a matching block. The result is cached for the session, so the search runs only once per session. See Search and filter for the query format.

import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc

SERVER_ADDRESS = "<server-url>"

def create_chat_request():
    """Creates an LLM chat request that selects its block by search."""

    data_json = json.dumps({
        "task": "llm_chat",
        "messages": [
            {"role": "system", "content": "You are an AI assistant."},
            {"role": "user", "content": "Hello! How are you?"},
            {"role": "assistant", "content": "I'm doing well! How can I assist you today?"},
            {"role": "user", "content": "Tell me about black holes."}
        ]
    })

    # Similarity search: pick an LLM block whose model name contains "gpt", in cluster-123
    query_parameters_json = json.dumps({
        "body": {
            "values": {
                "matchType": "cluster",
                "rankingPolicyRule": {
                    "values": {
                        "executionMode": "code",
                        "policyRuleURI": "policies.search.object-det-selector:v0.0.01-stable",
                        "settings": {},
                        "parameters": {
                            "filterRule": {
                                "matchType": "block",
                                "filter": {
                                    "clusterQuery": {
                                        "variable": "id",
                                        "operator": "==",
                                        "value": "cluster-123"
                                    },
                                    "blockQuery": {
                                        "logicalOperator": "AND",
                                        "conditions": [
                                            {
                                                "variable": "component.metadata.algorithmType",
                                                "operator": "==",
                                                "value": "llm"
                                            },
                                            {
                                                "variable": "component.metadata.model",
                                                "operator": "LIKE",
                                                "value": "gpt"
                                            }
                                        ]
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    })

    request = inference_pb2.BlockInferencePacket(
        session_id="sess-12345",                 # Unique session ID
        seq_no=1,                                # First message in sequence
        data=data_json,                          # Chat history
        ts=time.time(),                          # Current timestamp
        query_parameters=query_parameters_json,  # Block selection
        output_ptr="",                           # No graph inference
        files=[]                                 # No files attached
    )
    return request

def main():
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    response = stub.infer(create_chat_request())

    print("Inference Response:")
    print("Session ID:", response.session_id)
    print("Sequence No:", response.seq_no)
    print("Data:", response.data)
    print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

4. Inference with FrameDB by specifying frame_ptr

For very large inputs, such as video, store the data in FrameDB first and send only a pointer.

import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc

SERVER_ADDRESS = "<server-url>:50051"

def create_video_detection_request():
    """Creates an object detection request for part of a video stored in FrameDB."""

    # Which part of the video to process, in seconds
    data_json = json.dumps({
        "task": "object_detection",
        "video_interval": {
            "start_time": 90,   # 1 min 30 sec
            "end_time": 210     # 3 min 30 sec
        }
    })

    # FrameDB reference for the video
    frame_ptr_json = json.dumps({
        "framedb_id": "framdb-123",
        "key": "video-abcdef-123"
    })

    request = inference_pb2.BlockInferencePacket(
        block_id="blk-wexaans",                    # Fixed block ID
        session_id="sess-67890",                   # Unique session ID
        seq_no=1,                                  # First sequence
        frame_ptr=frame_ptr_json.encode("utf-8"),  # FrameDB reference
        data=data_json,                            # Video processing details
        ts=time.time(),                            # Current timestamp
        output_ptr="",                             # No graph inference
        files=[]                                   # No direct file attachments
    )
    return request

def main():
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    response = stub.infer(create_video_detection_request())

    print("Inference Response:")
    print("Session ID:", response.session_id)
    print("Sequence No:", response.seq_no)
    print("Data:", response.data)
    print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

Inference using graphs

You can chain known blocks for a single request by putting a graph in output_ptr. Each key is a block ID, and its value lists the blocks that receive its output. Set block_id to the first block.

{
    "graph": {
        "blk-ksshxpiy": ["blk-bfl3gbd5"],
        "blk-bfl3gbd5": ["blk-tsonq3qr"],
        "blk-tsonq3qr": []
    }
}

This means: blk-ksshxpiy → blk-bfl3gbd5 → blk-tsonq3qr.

Ad-hoc graphs vs vDAGs

Ad-hoc graphs are quick to use, but unlike vDAGs they have no pre- and post-processing policies, quotas or health checks. Each block's output must already be usable as the next block's input.

import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc

SERVER_ADDRESS = "<server-url>"

def create_graph_inference_request():
    """Object detection → tracking → pose estimation, as an ad-hoc graph."""

    data_json = json.dumps({
        "task": "object_detection_tracking_pose_estimation",
        "video_interval": {
            "start_time": 90,   # 1 min 30 sec
            "end_time": 210     # 3 min 30 sec
        }
    })

    # FrameDB reference for the video
    frame_ptr_json = json.dumps({
        "framedb_id": "framdb-123",
        "key": "video-abcdef-123"
    })

    # The graph
    graph_json = json.dumps({
        "graph": {
            "blk-ksshxpiy": ["blk-bfl3gbd5"],  # Object detection → tracking
            "blk-bfl3gbd5": ["blk-tsonq3qr"],  # Tracking → pose estimation
            "blk-tsonq3qr": []                 # End of pipeline
        }
    })

    request = inference_pb2.BlockInferencePacket(
        block_id="blk-ksshxpiy",                   # First block (object detection)
        session_id="sess-67890",                   # Unique session ID
        seq_no=1,                                  # First sequence
        frame_ptr=frame_ptr_json.encode("utf-8"),  # FrameDB reference
        data=data_json,                            # Video processing details
        ts=time.time(),                            # Current timestamp
        output_ptr=graph_json,                     # The graph
        files=[]                                   # No direct file attachments
    )
    return request

def main():
    channel = grpc.insecure_channel(SERVER_ADDRESS)
    stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

    response = stub.infer(create_graph_inference_request())

    print("Inference Response:")
    print("Session ID:", response.session_id)
    print("Sequence No:", response.seq_no)
    print("Data:", response.data)
    print("Timestamp:", response.ts)

if __name__ == "__main__":
    main()

Request logging

If logging is enabled, every request is stored in a TimescaleDB table. Files are not logged, to save storage.

CREATE TABLE IF NOT EXISTS block_inference (
    request_id UUID PRIMARY KEY,
    session_id TEXT,
    block_id TEXT,
    seq_no INTEGER,
    ts TIMESTAMPTZ,
    frame_ptr BYTEA,
    data BYTEA,
    query_parameters BYTEA,
    output_ptr BYTEA
);

-- Time-series index on ts
SELECT create_hypertable('block_inference', 'ts', if_not_exists => TRUE);

Log query APIs

Endpoint Method Returns
/logs/session/<session_id> GET All logs for a session
/logs/request/<request_id> GET Logs for one request
/logs/block/<block_id> GET All logs for a block
/logs/session_block?session_id=<session_id>&block_id=<block_id> GET Logs for a session on a given block
curl -X GET "<server-url>/logs/session/<session_id>"
curl -X GET "<server-url>/logs/request/<request_id>"
curl -X GET "<server-url>/logs/block/<block_id>"
curl -X GET "<server-url>/logs/session_block?session_id=<session_id>&block_id=<block_id>"

Inference server registry

The registry lists inference servers so that users can find them. Add your server to it if you want it to be publicly usable. The registry is served by adhoc-server-registry-svc on the management cluster (NodePort 30105).

Schema

@dataclass
class InferenceServer:
    inference_server_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    inference_server_name: str = ''
    inference_server_metadata: Dict[str, str] = field(default_factory=dict)
    inference_server_tags: List[str] = field(default_factory=list)
    inference_server_public_url: str = ''
Field Description
inference_server_id Unique ID, generated automatically (UUID).
inference_server_name A human-readable name.
inference_server_metadata Extra details, such as region, availability zone, provider, cluster ID, hardware and quota limits.
inference_server_tags Tags for categorizing and searching.
inference_server_public_url The public address clients connect to.

APIs

Method Endpoint Description
POST /inference_server Register a server
GET /inference_server/<server_id> Get a server
PUT /inference_server/<server_id> Update a server (MongoDB-style update)
DELETE /inference_server/<server_id> Remove a server
POST /inference_servers Search servers (MongoDB-style query)
curl -X POST <server-url>/inference_server \
     -H "Content-Type: application/json" \
     -d '{
           "inference_server_name": "inference-server-us-east-1",
           "inference_server_metadata": {
               "region": "us-east-1",
               "availability_zone": "us-east-1a",
               "provider": "AWS",
               "cluster_id": "cluster-123",
               "quota_management_data": {
                    "requests_per_second_total": 100,
                    "requests_per_second_per_session": 10,
                    "requests_per_second_per_block_id": 10,
                    "requests_per_session_id": 1000,
                    "requests_per_block_id": 100
                }
           },
           "inference_server_tags": ["NLP", "Transformer", "BERT"],
           "inference_server_public_url": "https://us-east-1.inference.example.com"
         }'
curl -X GET <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd
# Change region and provider
curl -X PUT <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd \
     -H "Content-Type: application/json" \
     -d '{
           "$set": {
               "inference_server_metadata.region": "us-west-2",
               "inference_server_metadata.provider": "Google Cloud"
           }
         }'

# Add a tag
curl -X PUT <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd \
     -H "Content-Type: application/json" \
     -d '{
           "$push": {
               "inference_server_tags": "LLM"
           }
         }'
curl -X DELETE <server-url>/inference_server/d4e8c3f0-7c8d-4a2e-b23e-2d4b6789abcd
# Servers in us-east-1a with GPUs
curl -X POST <server-url>/inference_servers \
     -H "Content-Type: application/json" \
     -d '{
           "inference_server_metadata.availability_zone": "us-east-1a",
           "inference_server_metadata.gpu_available": true
         }'

# Servers hosted on AWS
curl -X POST <server-url>/inference_servers \
     -H "Content-Type: application/json" \
     -d '{
           "inference_server_metadata.provider": "AWS"
         }'

# Servers tagged NLP or LLM
curl -X POST <server-url>/inference_servers \
     -H "Content-Type: application/json" \
     -d '{
           "inference_server_tags": { "$in": ["NLP", "LLM"] }
         }'

# Servers registered after 2025-01-01
curl -X POST <server-url>/inference_servers \
     -H "Content-Type: application/json" \
     -d '{
           "registered_at": { "$gte": "2025-01-01T00:00:00Z" }
         }'