Skip to content

Inference user guide

An inference user sends tasks to AI services on the network and receives results. You might be an application developer integrating AI into a product, or a user calling a model directly.

At a glance

You will: find a gateway, send a task, and read the result.
You need: Python with grpcio, the generated gRPC client classes, and the ID of the block or vDAG you want to call.

Two ways to call AI

You want to call... Send tasks to... Identify the target by
A single block (one model or service) an inference server block_id, or a search query
A vDAG (a multi-step workflow) that vDAG's controller the controller's address
flowchart LR
    App["Your application"] -->|"gRPC: BlockInferencePacket"| IS["Inference server"]
    IS --> B1["Block"]
    App -->|"gRPC: vDAGInferencePacket"| VC["vDAG controller"]
    VC --> B2["Block"] --> B3["Block"]

1. Send tasks to a block

Step 1: Find an inference server

Inference servers are listed in the network's inference server registry. Anyone can run one and list it there. Each entry looks like this:

{
    "inference_server_name": "inference-server-us-east-1",
    "inference_server_metadata": {
        "region": "us-east-1",
        "availability_zone": "us-east-1a",
        "provider": "AWS",
        "cluster_id": "cluster-123",
        "quota_management_data": {
            "requests_per_second_total": 100,
            "requests_per_second_per_session": 10,
            "requests_per_second_per_block_id": 10,
            "requests_per_session_id": 1000,
            "requests_per_block_id": 100
        }
    },
    "inference_server_tags": ["NLP", "Transformer", "BERT"],
    "inference_server_public_url": "https://us-east-1.inference.example.com"
}

Search the registry with a MongoDB-style query. For example, to find all servers in cluster-123:

curl -X POST <inference-servers-registry>/inference_servers \
     -H "Content-Type: application/json" \
     -d '{
           "inference_server_metadata.cluster_id": "cluster-123"
         }'

Use the chosen server's inference_server_public_url as your gRPC endpoint. More query examples: Inference server registry.

Step 2: Choose how to send the task

All block tasks use the same gRPC message, BlockInferencePacket. Pick the pattern that fits your data:

Pattern Use it when Example code
By block_id You know which block you want. This is the simplest option. Example
With attached files (FileInfo) You need to send images, audio or documents with the task. Example
By search (query_parameters) You don't know the block ID. A similarity search picks a matching block for you. See Search and filter. Example
Via FrameDB (frame_ptr) Your input is very large. Store it in FrameDB first and send only a pointer. Example
As an ad-hoc graph You want to chain a few known blocks for one request, without creating a vDAG. Note: ad-hoc graphs don't support pre/post-processing policies, so the blocks' inputs and outputs must already be compatible. Example

The full message format is described in the gRPC inference guide.

Minimal example: call a block by ID
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc

channel = grpc.insecure_channel("<inference-server-address>")
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)

request = inference_pb2.BlockInferencePacket(
    block_id="mistral-chat-88",
    session_id="sess-001",
    seq_no=1,
    data=json.dumps({"message": "Hello!", "session_id": "sess-001"}),
    ts=time.time(),
    query_parameters="",
    output_ptr="",
    files=[]
)

response = stub.infer(request)
print(response.data)

2. Send tasks to a vDAG

Step 1: Find the vDAG's controller

Look up the controller for a vDAG URI in the vDAG controller registry:

curl -X GET http://<vdag-registry>/vdag-controllers/by-vdag-uri/sample-vdag:1.0-stable

Each controller entry has this structure:

@dataclass
class vDAGController:
    # Unique identifier for the vDAG controller instance
    vdag_controller_id: str = ''
    # Associated vDAG URI this controller is managing
    vdag_uri: str = ''
    # Publicly accessible URL for interacting with the controller
    public_url: str = ''
    # Identifier of the cluster where the controller is deployed
    cluster_id: str = ''
    # Arbitrary metadata for storing additional information
    metadata: Dict[str, Any] = field(default_factory=dict)
    # Configuration parameters used by the controller
    config: Dict[str, Any] = field(default_factory=dict)
    # Tags used for search and discovery of the controller
    search_tags: List[str] = field(default_factory=list)

Use the controller's public_url as your gRPC endpoint. More about the registry: vDAG registry.

Step 2: Send the task

vDAG tasks use the vDAGInferencePacket gRPC message. The controller runs the task through the whole workflow and returns the final result.

Quotas

A vDAG's owner may set quotas that limit how many requests you can make, globally or per session_id. If your requests are rejected, check with the vDAG owner.