LLM SDK utilities¶
The LLM utilities extend the Instance SDK with ready-made helpers for loading and serving large language models (LLMs) as blocks. They cover model loading, text generation, chat sessions, tokenization and LLM-specific metrics, so your block code stays short.
At a glance
Who this is for: developers serving LLMs on AIGr.id.
Source: services/applications/aios-sdk/llm-utils/ in the repository.
Choose a library¶
| Library | Built on | Best for | Package |
|---|---|---|---|
TransformersUtils |
Hugging Face Transformers (PyTorch) | Larger models, including models spread across several GPUs on one node (tensor parallelism) | aios_transformers |
LLAMAUtils |
llama.cpp |
Simpler, single-GPU or CPU inference with GGUF models | aios_llama_cpp |
Both include LLMMetrics for standard LLM performance metrics. You can also use your own library.
Model too big for one node?
These utilities serve a model inside one block, on one node. For models that don't fit on one node, see Model splitting.
TransformersUtils (Hugging Face Transformers)¶
A wrapper around Hugging Face transformers for inference. It supports single-GPU and multi-GPU execution (tensor parallelism with device_map="auto"), text generation, chat sessions, tokenization, embeddings and optional metrics.
Install¶
# Interface to Transformers for LLM functionality
from aios_transformers import TransformersUtils
# LLM metrics (see "LLM metrics" below)
from aios_transformers import LLMMetrics
Create¶
TransformersUtils(
model_name: str = "gpt2",
device: str = None,
metrics=None,
tensor_parallel: bool = False,
quantize: bool = False,
generation_config: dict = {}
)
| Parameter | Description |
|---|---|
model_name |
Hugging Face model ID. Default: "gpt2". |
device |
"cuda" or "cpu". Detected automatically if None. |
metrics |
Optional metrics object, such as LLMMetrics. |
tensor_parallel |
True to spread the model across all available GPUs (device_map="auto"). |
quantize |
Placeholder for future quantization support; currently unused. |
generation_config |
Default generation parameters. |
Methods¶
| Group | Method | Description |
|---|---|---|
| Loading | load_model(model_name: str, extra_args: dict = {}) |
Loads the model and tokenizer (AutoModelForCausalLM) and creates a text-generation pipeline. With tensor_parallel=True, the model is sharded across GPUs automatically; otherwise it loads onto device. |
reload_model() |
Reloads the last loaded model. | |
set_generator(generator: Pipeline) |
Uses a pipeline you created yourself. | |
| Configuration | set_generation_config(**kwargs) |
Updates default generation parameters, such as max_new_tokens, top_k and temperature. |
| Generation | generate(prompt: str, **kwargs) |
Generates text with the pipeline. Merges your arguments with the defaults, and records prompt, response and timing metrics if metrics are enabled. |
generate_tokens(prompt: str, **kwargs) |
Generates token IDs directly with model.generate(). |
|
| Tokenizer | tokenize(text: str) |
Tokenizes text. |
decode(token_ids) |
Turns token IDs back into text. | |
| Embeddings | get_embeddings(text: str) |
Returns the last hidden state (embeddings), using AutoModel. Works with or without tensor parallelism. |
| Chat | create_chat_session(session_id: str, system_message: str = "") |
Starts a chat session, optionally with a system prompt. |
add_message_to_chat(session_id: str, message: str, role: str = "user") |
Adds a user or assistant message. | |
run_chat_inference(session_id: str, **kwargs) |
Generates a reply from the full conversation and adds it to the history. | |
remove_chat_session(session_id: str) |
Deletes a session. | |
| Info | get_device_info() |
Returns hardware and model details (see below). |
get_device_info() returns:
{
"device": "cuda" or "cpu",
"cuda_available": True or False,
"num_gpus": <int>,
"tensor_parallel": True or False,
"model": "<model_name>"
}
Example¶
from aios_transformers import TransformersUtils
utils = TransformersUtils(
model_name="Qwen/Qwen1.5-0.5B-Chat", # small, fast model for testing
tensor_parallel=True # spread across all available GPUs
)
utils.load_model()
prompt = "hey!"
# Generate text
generated = utils.generate(prompt)
print("Generated Text:\n", generated)
# Tokenize and decode
tokens = utils.tokenize(prompt)
print("\nToken IDs:", tokens["input_ids"])
print("Decoded Text:", utils.decode(tokens["input_ids"][0]))
# Raw token output
token_output = utils.generate_tokens(prompt, max_new_tokens=20)
print("\nGenerated Token IDs:", token_output.tolist()[0])
print(utils.get_device_info())
# Chat
utils.create_chat_session('123', "You are a chat bot, you respond to only what is asked")
utils.add_message_to_chat('123', "Tell me about yourself")
print(utils.run_chat_inference('123'))
LLAMAUtils (llama.cpp)¶
A wrapper around llama_cpp for model loading, tokenization, streaming and batch inference, and chat, with optional metrics and GPU support.
Install¶
# Interface to llama.cpp for LLM functionality
from aios_llama_cpp import LLAMAUtils
# LLM metrics (see "LLM metrics" below)
from aios_llama_cpp import LLMMetrics
Create¶
| Parameter | Description |
|---|---|
model_path |
Path to the .gguf or .bin model file. |
use_gpu |
Use GPU acceleration, if llama_cpp supports it. |
gpu_id |
Which GPU to use when use_gpu is True. |
metrics |
Optional metrics object, such as LLMMetrics. |
Default generation settings. Override any of them per call, for example max_tokens=100:
Methods¶
| Group | Method | Description |
|---|---|---|
| Loading | load_model() -> bool |
Loads the model from model_path with the configured device. Returns True on success. |
| Inference | run_inference(prompt: str, stream: bool = False, **kwargs) |
Runs a prompt. With stream=True, yields tokens as they are generated. Records latency and token metrics. |
generate_text(prompt: str, num_sequences: int = 1, **kwargs) |
Generates several completions for the same prompt. | |
stream_inference(prompt: str, **kwargs) |
Streams output token by token to stdout; for CLI or interactive use. | |
| Chat | supports_chat() -> bool |
Whether the model supports chat completion (create_chat_completion()). |
create_chat_session(session_id: str, system_message: str = "", tools_list: list = None, tools_choice: dict = None) |
Starts a session, optionally with a system message and tool configuration (for tool-using models). | |
add_message_to_chat(session_id: str, message: str, role: str = "user") |
Adds a message to the session. | |
run_chat_inference(session_id: str, **kwargs) -> str |
Generates the assistant's reply. Supports tools and streaming. Returns only the reply text. | |
remove_chat_session(session_id: str) |
Deletes a session. | |
| Tokens | tokenize(text: str) |
Text to token IDs. |
detokenize(tokens: List[int]) |
Token IDs to text. | |
| Utilities | save_model(save_path: str) -> bool |
Saves the model state, if supported. |
get_model_info() -> dict |
Model details such as size, layers and architecture, if available. | |
set_seed(seed: int) -> bool |
Sets the random seed, for repeatable output. |
Example¶
from aios_llama_cpp import LLAMAUtils
model_path = "<path>/qwen1_5-0_5b-chat-q2_k.gguf" # update to the real path
llama = LLAMAUtils(model_path=model_path, use_gpu=True, gpu_id=0, metrics=None)
if llama.load_model():
# Basic inference
result = llama.run_inference("What is the capital of France?")
print("\nResult:", result["choices"][0]["text"].strip() if result else "No result")
# Streaming inference
print("\nStreaming:")
llama.run_inference("Write a haiku about space.", stream=True)
# Chat session
session_id = "chat123"
llama.create_chat_session(session_id, system_message="You are a helpful assistant.")
llama.add_message_to_chat(session_id, "What's the weather like on Mars?")
print("\n\nChat Response:", llama.run_chat_inference(session_id))
llama.remove_chat_session(session_id)
LLM metrics¶
LLMMetrics registers and updates standard LLM performance metrics on top of your block's metrics. Create it from the AIOSMetrics object in your block's context:
Then pass it as metrics= to TransformersUtils or LLAMAUtils. They record most values automatically.
Metrics collected¶
| Metric | Type | Description | Unit / buckets |
|---|---|---|---|
llm_prompts_total |
Counter | Prompts received and processed | Count |
llm_tokens_generated_total |
Counter | Tokens generated | Count |
llm_prompt_tokens_total |
Counter | Tokens received in prompts | Count |
llm_active_sessions |
Gauge | Active chat sessions | Count |
llm_inference_duration_seconds |
Histogram | Time for a full inference | Seconds: [0.01, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] |
llm_time_to_first_token_seconds |
Histogram | Time from prompt to first token (TTFT) | Seconds: [0.01, 0.05, 0.1, 0.2, 0.5, 1, 2] |
llm_time_per_output_token_seconds |
Histogram | Time per output token (TPOT) | Seconds: [0.01, 0.05, 0.1, 0.2, 0.5] |
llm_tokens_per_second |
Gauge | Generation throughput | Tokens/second |
llm_cpu_utilization |
Gauge | CPU use during inference | Percent (0–100) |
llm_gpu_utilization |
Gauge | GPU use during inference | Percent (0–100) |
llm_memory_usage_bytes |
Gauge | Memory used during inference | Bytes |
llm_inference_errors_total |
Counter | Inference errors | Count |
Methods¶
| Method | Description |
|---|---|
__init__(metrics, block_id=None) |
Registers all metrics. metrics is a Prometheus-compatible metrics manager, normally context.metrics. block_id is optional. |
log_prompt(prompt_token_count: int) |
Records a prompt and its token count. |
log_response(generated_token_count: int) |
Records the number of generated tokens. |
observe_inference_time(start_time: float) |
Records total inference time, from start_time (a time.time() value). |
observe_time_to_first_token(start_time: float) |
Records time to first token. |
observe_time_per_output_token(start_time: float, token_count: int) |
Records average time per output token. |
update_tokens_per_second(tokens_generated: int, duration_seconds: float) |
Updates throughput. |
update_resource_utilization(cpu_percent=None, gpu_percent=None, memory_bytes=None) |
Sets the resource gauges. |
increment_inference_errors() |
Counts an error. |
increase_active_sessions() / decrease_active_sessions() |
Adjusts the active session count. |
Build a container image¶
Each library has its own base image, built on top of aios_instance:v1. Build the Instance SDK base image first.
Use the resulting image as the FROM base for your block's image, then register it as a component.
Full example: a chat block with llama.cpp¶
A complete block that serves a chat model (for example qwen1.5-0.5B-chat) and keeps a separate conversation per session:
import json
from aios_instance import Block, PreProcessResult, OnDataResult
from aios_llama_cpp import LLAMAUtils, LLMMetrics
class ChatBlock:
def __init__(self, context):
"""Loads the model and sets up metrics."""
self.context = context
llm_metrics = LLMMetrics(context.metrics)
self.llama = LLAMAUtils(
model_path=context.block_init_parameters.get("model_path", "./"),
use_gpu=True,
metrics=llm_metrics
)
if not self.llama.load_model():
raise RuntimeError("Failed to load LLaMA model.")
self.default_temperature = context.block_init_parameters.get("temperature", 0.8)
def on_preprocess(self, packet):
"""Parses the user message and session ID."""
try:
data = packet.data
if isinstance(data, str):
data = json.loads(data)
if "inputs" in data:
return True, [
PreProcessResult(packet=packet, extra_data={"input": item})
for item in data["inputs"]
]
return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
except Exception as e:
return False, str(e)
def on_data(self, preprocessed_entry):
"""Runs chat inference for the session."""
try:
input_data = preprocessed_entry.extra_data["input"]
user_message = input_data.get("message")
session_id = input_data.get("session_id", "default")
# Create the chat session if it doesn't exist
if session_id not in self.llama.chat_sessions:
self.llama.create_chat_session(session_id, system_message="You are a helpful assistant.")
self.llama.add_message_to_chat(session_id, user_message, role="user")
response = self.llama.run_chat_inference(
session_id,
temperature=self.default_temperature
)
return True, OnDataResult(output={"reply": response})
except Exception as e:
return False, str(e)
def on_update(self, updated_parameters):
try:
if "temperature" in updated_parameters:
self.default_temperature = updated_parameters["temperature"]
return True, updated_parameters
except Exception as e:
return False, str(e)
def health(self):
return {"status": "healthy", "model_loaded": self.llama.model is not None}
def management(self, action, data):
try:
if action == "reset":
self.llama.chat_sessions.clear()
return {"message": "Chat sessions cleared."}
return {"message": f"Unknown action '{action}'"}
except Exception as e:
return {"error": str(e)}
def get_muxer(self):
return None # or Muxer(N=3) to merge packets
if __name__ == "__main__":
Block(ChatBlock).run()
For a deployable, multi-GPU example, see the Mistral 7B block in the Quickstart (services/applications/examples/mistral7b).
Using other libraries¶
You are not limited to these two libraries. To use any other framework, build your image from the plain Instance SDK base image (aios_instance:v1) and install your library in it. See Instance SDK.