Inference user guide¶
An inference user sends tasks to AI services on the network and receives results. You might be an application developer integrating AI into a product, or a user calling a model directly.
At a glance
You will: find a gateway, send a task, and read the result.
You need: Python with grpcio, the generated gRPC client classes, and the ID of the block or vDAG you want to call.
Two ways to call AI¶
| You want to call... | Send tasks to... | Identify the target by |
|---|---|---|
| A single block (one model or service) | an inference server | block_id, or a search query |
| A vDAG (a multi-step workflow) | that vDAG's controller | the controller's address |
flowchart LR
App["Your application"] -->|"gRPC: BlockInferencePacket"| IS["Inference server"]
IS --> B1["Block"]
App -->|"gRPC: vDAGInferencePacket"| VC["vDAG controller"]
VC --> B2["Block"] --> B3["Block"]
1. Send tasks to a block¶
Step 1: Find an inference server¶
Inference servers are listed in the network's inference server registry. Anyone can run one and list it there. Each entry looks like this:
{
"inference_server_name": "inference-server-us-east-1",
"inference_server_metadata": {
"region": "us-east-1",
"availability_zone": "us-east-1a",
"provider": "AWS",
"cluster_id": "cluster-123",
"quota_management_data": {
"requests_per_second_total": 100,
"requests_per_second_per_session": 10,
"requests_per_second_per_block_id": 10,
"requests_per_session_id": 1000,
"requests_per_block_id": 100
}
},
"inference_server_tags": ["NLP", "Transformer", "BERT"],
"inference_server_public_url": "https://us-east-1.inference.example.com"
}
Search the registry with a MongoDB-style query. For example, to find all servers in cluster-123:
curl -X POST <inference-servers-registry>/inference_servers \
-H "Content-Type: application/json" \
-d '{
"inference_server_metadata.cluster_id": "cluster-123"
}'
Use the chosen server's inference_server_public_url as your gRPC endpoint. More query examples: Inference server registry.
Step 2: Choose how to send the task¶
All block tasks use the same gRPC message, BlockInferencePacket. Pick the pattern that fits your data:
| Pattern | Use it when | Example code |
|---|---|---|
By block_id |
You know which block you want. This is the simplest option. | Example |
With attached files (FileInfo) |
You need to send images, audio or documents with the task. | Example |
By search (query_parameters) |
You don't know the block ID. A similarity search picks a matching block for you. See Search and filter. | Example |
Via FrameDB (frame_ptr) |
Your input is very large. Store it in FrameDB first and send only a pointer. | Example |
| As an ad-hoc graph | You want to chain a few known blocks for one request, without creating a vDAG. Note: ad-hoc graphs don't support pre/post-processing policies, so the blocks' inputs and outputs must already be compatible. | Example |
The full message format is described in the gRPC inference guide.
Minimal example: call a block by ID
import grpc
import time
import json
import inference_pb2
import inference_pb2_grpc
channel = grpc.insecure_channel("<inference-server-address>")
stub = inference_pb2_grpc.BlockInferenceServiceStub(channel)
request = inference_pb2.BlockInferencePacket(
block_id="mistral-chat-88",
session_id="sess-001",
seq_no=1,
data=json.dumps({"message": "Hello!", "session_id": "sess-001"}),
ts=time.time(),
query_parameters="",
output_ptr="",
files=[]
)
response = stub.infer(request)
print(response.data)
2. Send tasks to a vDAG¶
Step 1: Find the vDAG's controller¶
Look up the controller for a vDAG URI in the vDAG controller registry:
Each controller entry has this structure:
@dataclass
class vDAGController:
# Unique identifier for the vDAG controller instance
vdag_controller_id: str = ''
# Associated vDAG URI this controller is managing
vdag_uri: str = ''
# Publicly accessible URL for interacting with the controller
public_url: str = ''
# Identifier of the cluster where the controller is deployed
cluster_id: str = ''
# Arbitrary metadata for storing additional information
metadata: Dict[str, Any] = field(default_factory=dict)
# Configuration parameters used by the controller
config: Dict[str, Any] = field(default_factory=dict)
# Tags used for search and discovery of the controller
search_tags: List[str] = field(default_factory=list)
Use the controller's public_url as your gRPC endpoint. More about the registry: vDAG registry.
Step 2: Send the task¶
vDAG tasks use the vDAGInferencePacket gRPC message. The controller runs the task through the whole workflow and returns the final result.
- Message format and full client example: vDAG controller: inference gRPC API
- Worked example: Quickstart: deploy a vDAG and send it tasks
Quotas
A vDAG's owner may set quotas that limit how many requests you can make, globally or per session_id. If your requests are rejected, check with the vDAG owner.