Skip to content

LLM SDK utilities

The LLM utilities extend the Instance SDK with ready-made helpers for loading and serving large language models (LLMs) as blocks. They cover model loading, text generation, chat sessions, tokenization and LLM-specific metrics, so your block code stays short.

At a glance

Who this is for: developers serving LLMs on AIGr.id.
Source: services/applications/aios-sdk/llm-utils/ in the repository.

Choose a library

Library Built on Best for Package
TransformersUtils Hugging Face Transformers (PyTorch) Larger models, including models spread across several GPUs on one node (tensor parallelism) aios_transformers
LLAMAUtils llama.cpp Simpler, single-GPU or CPU inference with GGUF models aios_llama_cpp

Both include LLMMetrics for standard LLM performance metrics. You can also use your own library.

Model too big for one node?

These utilities serve a model inside one block, on one node. For models that don't fit on one node, see Model splitting.


TransformersUtils (Hugging Face Transformers)

A wrapper around Hugging Face transformers for inference. It supports single-GPU and multi-GPU execution (tensor parallelism with device_map="auto"), text generation, chat sessions, tokenization, embeddings and optional metrics.

Install

cd services/applications/aios-sdk/llm-utils/aios_transformers
pip3 install -e .
# Interface to Transformers for LLM functionality
from aios_transformers import TransformersUtils

# LLM metrics (see "LLM metrics" below)
from aios_transformers import LLMMetrics

Create

TransformersUtils(
    model_name: str = "gpt2",
    device: str = None,
    metrics=None,
    tensor_parallel: bool = False,
    quantize: bool = False,
    generation_config: dict = {}
)
Parameter Description
model_name Hugging Face model ID. Default: "gpt2".
device "cuda" or "cpu". Detected automatically if None.
metrics Optional metrics object, such as LLMMetrics.
tensor_parallel True to spread the model across all available GPUs (device_map="auto").
quantize Placeholder for future quantization support; currently unused.
generation_config Default generation parameters.

Methods

Group Method Description
Loading load_model(model_name: str, extra_args: dict = {}) Loads the model and tokenizer (AutoModelForCausalLM) and creates a text-generation pipeline. With tensor_parallel=True, the model is sharded across GPUs automatically; otherwise it loads onto device.
reload_model() Reloads the last loaded model.
set_generator(generator: Pipeline) Uses a pipeline you created yourself.
Configuration set_generation_config(**kwargs) Updates default generation parameters, such as max_new_tokens, top_k and temperature.
Generation generate(prompt: str, **kwargs) Generates text with the pipeline. Merges your arguments with the defaults, and records prompt, response and timing metrics if metrics are enabled.
generate_tokens(prompt: str, **kwargs) Generates token IDs directly with model.generate().
Tokenizer tokenize(text: str) Tokenizes text.
decode(token_ids) Turns token IDs back into text.
Embeddings get_embeddings(text: str) Returns the last hidden state (embeddings), using AutoModel. Works with or without tensor parallelism.
Chat create_chat_session(session_id: str, system_message: str = "") Starts a chat session, optionally with a system prompt.
add_message_to_chat(session_id: str, message: str, role: str = "user") Adds a user or assistant message.
run_chat_inference(session_id: str, **kwargs) Generates a reply from the full conversation and adds it to the history.
remove_chat_session(session_id: str) Deletes a session.
Info get_device_info() Returns hardware and model details (see below).

get_device_info() returns:

{
    "device": "cuda" or "cpu",
    "cuda_available": True or False,
    "num_gpus": <int>,
    "tensor_parallel": True or False,
    "model": "<model_name>"
}

Example

from aios_transformers import TransformersUtils

utils = TransformersUtils(
    model_name="Qwen/Qwen1.5-0.5B-Chat",  # small, fast model for testing
    tensor_parallel=True                  # spread across all available GPUs
)

utils.load_model()

prompt = "hey!"

# Generate text
generated = utils.generate(prompt)
print("Generated Text:\n", generated)

# Tokenize and decode
tokens = utils.tokenize(prompt)
print("\nToken IDs:", tokens["input_ids"])
print("Decoded Text:", utils.decode(tokens["input_ids"][0]))

# Raw token output
token_output = utils.generate_tokens(prompt, max_new_tokens=20)
print("\nGenerated Token IDs:", token_output.tolist()[0])

print(utils.get_device_info())

# Chat
utils.create_chat_session('123', "You are a chat bot, you respond to only what is asked")
utils.add_message_to_chat('123', "Tell me about yourself")
print(utils.run_chat_inference('123'))

LLAMAUtils (llama.cpp)

A wrapper around llama_cpp for model loading, tokenization, streaming and batch inference, and chat, with optional metrics and GPU support.

Install

cd services/applications/aios-sdk/llm-utils/aios_llama_cpp
pip3 install -e .
# Interface to llama.cpp for LLM functionality
from aios_llama_cpp import LLAMAUtils

# LLM metrics (see "LLM metrics" below)
from aios_llama_cpp import LLMMetrics

Create

LLAMAUtils(
    model_path: str,
    use_gpu: bool = False,
    gpu_id: int = 0,
    metrics = None
)
Parameter Description
model_path Path to the .gguf or .bin model file.
use_gpu Use GPU acceleration, if llama_cpp supports it.
gpu_id Which GPU to use when use_gpu is True.
metrics Optional metrics object, such as LLMMetrics.

Default generation settings. Override any of them per call, for example max_tokens=100:

{
    "max_tokens": 50,
    "temperature": 1.0,
    "top_p": 1.0,
    "stop": ["Q:", "\n"]
}

Methods

Group Method Description
Loading load_model() -> bool Loads the model from model_path with the configured device. Returns True on success.
Inference run_inference(prompt: str, stream: bool = False, **kwargs) Runs a prompt. With stream=True, yields tokens as they are generated. Records latency and token metrics.
generate_text(prompt: str, num_sequences: int = 1, **kwargs) Generates several completions for the same prompt.
stream_inference(prompt: str, **kwargs) Streams output token by token to stdout; for CLI or interactive use.
Chat supports_chat() -> bool Whether the model supports chat completion (create_chat_completion()).
create_chat_session(session_id: str, system_message: str = "", tools_list: list = None, tools_choice: dict = None) Starts a session, optionally with a system message and tool configuration (for tool-using models).
add_message_to_chat(session_id: str, message: str, role: str = "user") Adds a message to the session.
run_chat_inference(session_id: str, **kwargs) -> str Generates the assistant's reply. Supports tools and streaming. Returns only the reply text.
remove_chat_session(session_id: str) Deletes a session.
Tokens tokenize(text: str) Text to token IDs.
detokenize(tokens: List[int]) Token IDs to text.
Utilities save_model(save_path: str) -> bool Saves the model state, if supported.
get_model_info() -> dict Model details such as size, layers and architecture, if available.
set_seed(seed: int) -> bool Sets the random seed, for repeatable output.

Example

from aios_llama_cpp import LLAMAUtils

model_path = "<path>/qwen1_5-0_5b-chat-q2_k.gguf"  # update to the real path
llama = LLAMAUtils(model_path=model_path, use_gpu=True, gpu_id=0, metrics=None)

if llama.load_model():
    # Basic inference
    result = llama.run_inference("What is the capital of France?")
    print("\nResult:", result["choices"][0]["text"].strip() if result else "No result")

    # Streaming inference
    print("\nStreaming:")
    llama.run_inference("Write a haiku about space.", stream=True)

    # Chat session
    session_id = "chat123"
    llama.create_chat_session(session_id, system_message="You are a helpful assistant.")
    llama.add_message_to_chat(session_id, "What's the weather like on Mars?")
    print("\n\nChat Response:", llama.run_chat_inference(session_id))

    llama.remove_chat_session(session_id)

LLM metrics

LLMMetrics registers and updates standard LLM performance metrics on top of your block's metrics. Create it from the AIOSMetrics object in your block's context:

llm_metrics = LLMMetrics(context.metrics)

Then pass it as metrics= to TransformersUtils or LLAMAUtils. They record most values automatically.

Metrics collected

Metric Type Description Unit / buckets
llm_prompts_total Counter Prompts received and processed Count
llm_tokens_generated_total Counter Tokens generated Count
llm_prompt_tokens_total Counter Tokens received in prompts Count
llm_active_sessions Gauge Active chat sessions Count
llm_inference_duration_seconds Histogram Time for a full inference Seconds: [0.01, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]
llm_time_to_first_token_seconds Histogram Time from prompt to first token (TTFT) Seconds: [0.01, 0.05, 0.1, 0.2, 0.5, 1, 2]
llm_time_per_output_token_seconds Histogram Time per output token (TPOT) Seconds: [0.01, 0.05, 0.1, 0.2, 0.5]
llm_tokens_per_second Gauge Generation throughput Tokens/second
llm_cpu_utilization Gauge CPU use during inference Percent (0–100)
llm_gpu_utilization Gauge GPU use during inference Percent (0–100)
llm_memory_usage_bytes Gauge Memory used during inference Bytes
llm_inference_errors_total Counter Inference errors Count

Methods

Method Description
__init__(metrics, block_id=None) Registers all metrics. metrics is a Prometheus-compatible metrics manager, normally context.metrics. block_id is optional.
log_prompt(prompt_token_count: int) Records a prompt and its token count.
log_response(generated_token_count: int) Records the number of generated tokens.
observe_inference_time(start_time: float) Records total inference time, from start_time (a time.time() value).
observe_time_to_first_token(start_time: float) Records time to first token.
observe_time_per_output_token(start_time: float, token_count: int) Records average time per output token.
update_tokens_per_second(tokens_generated: int, duration_seconds: float) Updates throughput.
update_resource_utilization(cpu_percent=None, gpu_percent=None, memory_bytes=None) Sets the resource gauges.
increment_inference_errors() Counts an error.
increase_active_sessions() / decrease_active_sessions() Adjusts the active session count.

Build a container image

Each library has its own base image, built on top of aios_instance:v1. Build the Instance SDK base image first.

cd services/applications/aios-sdk/llm-utils/aios_llama_cpp
docker build . -t aios_llama_cpp:v1
cd services/applications/aios-sdk/llm-utils/aios_transformers
docker build . -t aios_transformers:v1

Use the resulting image as the FROM base for your block's image, then register it as a component.


Full example: a chat block with llama.cpp

A complete block that serves a chat model (for example qwen1.5-0.5B-chat) and keeps a separate conversation per session:

import json
from aios_instance import Block, PreProcessResult, OnDataResult
from aios_llama_cpp import LLAMAUtils, LLMMetrics


class ChatBlock:
    def __init__(self, context):
        """Loads the model and sets up metrics."""
        self.context = context

        llm_metrics = LLMMetrics(context.metrics)

        self.llama = LLAMAUtils(
            model_path=context.block_init_parameters.get("model_path", "./"),
            use_gpu=True,
            metrics=llm_metrics
        )

        if not self.llama.load_model():
            raise RuntimeError("Failed to load LLaMA model.")

        self.default_temperature = context.block_init_parameters.get("temperature", 0.8)

    def on_preprocess(self, packet):
        """Parses the user message and session ID."""
        try:
            data = packet.data
            if isinstance(data, str):
                data = json.loads(data)

            if "inputs" in data:
                return True, [
                    PreProcessResult(packet=packet, extra_data={"input": item})
                    for item in data["inputs"]
                ]
            return True, [PreProcessResult(packet=packet, extra_data={"input": data})]
        except Exception as e:
            return False, str(e)

    def on_data(self, preprocessed_entry):
        """Runs chat inference for the session."""
        try:
            input_data = preprocessed_entry.extra_data["input"]
            user_message = input_data.get("message")
            session_id = input_data.get("session_id", "default")

            # Create the chat session if it doesn't exist
            if session_id not in self.llama.chat_sessions:
                self.llama.create_chat_session(session_id, system_message="You are a helpful assistant.")

            self.llama.add_message_to_chat(session_id, user_message, role="user")

            response = self.llama.run_chat_inference(
                session_id,
                temperature=self.default_temperature
            )

            return True, OnDataResult(output={"reply": response})
        except Exception as e:
            return False, str(e)

    def on_update(self, updated_parameters):
        try:
            if "temperature" in updated_parameters:
                self.default_temperature = updated_parameters["temperature"]
            return True, updated_parameters
        except Exception as e:
            return False, str(e)

    def health(self):
        return {"status": "healthy", "model_loaded": self.llama.model is not None}

    def management(self, action, data):
        try:
            if action == "reset":
                self.llama.chat_sessions.clear()
                return {"message": "Chat sessions cleared."}
            return {"message": f"Unknown action '{action}'"}
        except Exception as e:
            return {"error": str(e)}

    def get_muxer(self):
        return None  # or Muxer(N=3) to merge packets


if __name__ == "__main__":
    Block(ChatBlock).run()

For a deployable, multi-GPU example, see the Mistral 7B block in the Quickstart (services/applications/examples/mistral7b).


Using other libraries

You are not limited to these two libraries. To use any other framework, build your image from the plain Instance SDK base image (aios_instance:v1) and install your library in it. See Instance SDK.