Skip to content

Assets registry

The assets registry stores the files that blocks and policies need, such as model weights, policy code archives, images and videos, together with metadata, versions and tags so they can be found and reused.

At a glance

Optional: any public storage with a downloadable URL works just as well. The assets registry adds versioning, search and a network-wide index.
Deploy your own: see Join a cluster: assets registry.

Two services

Service What it does Who runs it
Assets DB Stores asset files and their metadata. Anyone. Each one is backed by S3-compatible storage (Amazon S3, Ceph Object Storage, MinIO, ...) plus a database for metadata, which can be the network's global DB if you want assets to be searchable network-wide. It needs a public URL.
Assets DB registry A network-wide index of assets DBs, with search and health checks. Register your assets DB here to make it discoverable. The network (assets-db-registry-svc, NodePort 30104).
flowchart LR
    U["You"] -->|"upload / register"| A1[("Assets DB A<br/><small>S3 / Ceph / MinIO</small>")]
    U --> A2[("Assets DB B")]
    A1 -->|"listed in"| R["Assets DB registry<br/><small>network-wide index</small>"]
    A2 --> R
    B["Blocks and policies"] -->|"download by URL"| A1

Assets DB

Asset schema

@dataclass
class AssetObject:
    asset_uri: str = ''
    asset_name: str = ''
    asset_id: str = ''
    assets_db_id: str = ''
    asset_version: Dict[str, str] = field(
        default_factory=lambda: {"version": "", "tag": ""})
    asset_metadata: Dict[str, str] = field(default_factory=dict)
    asset_public_url: str = ''
    asset_file_info: Dict[str, str] = field(default_factory=dict)
    asset_tags: List[str] = field(default_factory=list)
Field Type Description
asset_uri str Unique ID, generated as asset_name:version-tag, for example SampleAsset:1.0-initial. Empty if any part is missing.
asset_name str The asset's name.
asset_id str Internal storage ID, assigned when you upload the file to the assets DB (rather than giving your own public URL).
assets_db_id str Which assets DB holds the asset. Defaults to the DB the asset was uploaded to. Useful when several DBs share a global metadata store.
asset_version dict version and tag. Lets several versions of an asset share a name.
asset_metadata dict Free-form details: description, author, dates and so on.
asset_public_url str The asset's download URL. For uploaded files: http://<asset-db-url>/asset-object/<asset-id>.
asset_file_info dict File details, such as size, format or checksum.
asset_tags list Tags, for search.

APIs

Endpoint Method What it does
/upload_asset POST Upload a file with its metadata (multipart form).
/asset POST Register an asset that is already hosted elsewhere (JSON, with its asset_public_url).
/asset/<asset_uri> GET Get an asset's record.
/asset/<asset_uri> PUT Update an asset's record.
/asset/<asset_uri> DELETE Delete an asset.
/assets POST Search assets with a MongoDB-style filter.
/asset-object/<asset_id> GET Download an uploaded file.
/health GET Check that the service is running.
curl -X POST http://<server-url>/upload_asset \
     -H "Content-Type: multipart/form-data" \
     -F "asset=@/path/to/your/file.png" \
     -F 'asset_metadata={
           "asset_name": "SampleAsset",
           "asset_version": { "version": "1.0", "tag": "initial" },
           "asset_metadata": { "description": "Sample asset upload" },
           "asset_tags": ["image", "sample"]
         }'
curl -X POST http://<server-url>/asset \
     -H "Content-Type: application/json" \
     -d '{
           "asset_name": "Sample Asset",
           "asset_version": {"version": "1.0", "tag": "initial"},
           "asset_metadata": {"description": "Sample asset for testing"},
           "asset_public_url": "https://example.com/files/sample-asset.bin",
           "asset_tags": ["test", "sample"]
         }'
curl -X GET http://<server-url>/asset/SampleAsset:1.0-initial

curl -X PUT http://<server-url>/asset/SampleAsset:1.0-initial \
     -H "Content-Type: application/json" \
     -d '{
           "asset_metadata": {"description": "Updated asset description"},
           "asset_tags": ["updated", "sample"]
         }'

curl -X DELETE http://<server-url>/asset/SampleAsset:1.0-initial
# Simple match
curl -X POST http://<server-url>/assets \
     -H "Content-Type: application/json" \
     -d '{
           "asset_tags": ["sample"],
           "asset_metadata": {"type": "image"}
         }'

# MongoDB operators
curl -X POST http://<server-url>/assets \
     -H "Content-Type: application/json" \
     -d '{
           "asset_tags": { "$in": ["sample", "image"] },
           "asset_metadata.type": "image",
           "asset_version.version": { "$gte": "1.0" }
         }'
curl -X GET http://<server-url>/asset-object/<asset_id> -o downloaded_asset

curl -X GET http://<server-url>/health

Assets DB registry

Schema

@dataclass
class AssetsDBRegistry:
    asset_registry_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    asset_registry_name: str = ''
    asset_registry_metadata: Dict[str, str] = field(default_factory=dict)
    asset_registry_tags: List[str] = field(default_factory=list)
    asset_registry_public_url: str = ''
Field Type Description
asset_registry_id str Unique ID (UUID), generated automatically.
asset_registry_name str Human-readable name.
asset_registry_metadata dict Free-form details: version, owner, framework and so on.
asset_registry_tags list Tags, for search and grouping.
asset_registry_public_url str The assets DB's public URL.

APIs

Endpoint Method What it does
/asset_registry POST Register an assets DB.
/asset_registry/<asset_db_id> GET Get an entry.
/asset_registry/<asset_db_id> PUT Update an entry.
/asset_registry/<asset_db_id> DELETE Delete an entry.
/asset_registries POST Search entries with a MongoDB-style filter.
/check_health/<asset_db_id> GET Check that a registered assets DB is reachable and healthy.
# Register
curl -X POST "http://<server-url>/asset_registry" \
     -H "Content-Type: application/json" \
     -d '{
           "asset_registry_name": "ResNet Models registry",
           "asset_registry_metadata": {
             "framework": "PyTorch",
             "version": "1.0",
             "author": "Example Org",
             "license": "Apache-2.0"
           },
           "asset_registry_tags": ["image-classification", "deep-learning", "pytorch"],
           "asset_registry_public_url": "https://example.com/models/resnet-models-registry"
         }'

# Get
curl -X GET "http://<server-url>/asset_registry/123e4567-e89b-12d3-a456-426614174000"

# Update
curl -X PUT "http://<server-url>/asset_registry/123e4567-e89b-12d3-a456-426614174000" \
     -H "Content-Type: application/json" \
     -d '{
           "asset_registry_metadata": {
             "framework": "TensorFlow",
             "version": "2.0"
           },
           "asset_registry_tags": ["neural-network", "tensorflow"]
         }'

# Delete
curl -X DELETE "http://<server-url>/asset_registry/123e4567-e89b-12d3-a456-426614174000"

# Search
curl -X POST "http://<server-url>/asset_registries" \
     -H "Content-Type: application/json" \
     -d '{
           "$and": [
             { "asset_registry_tags": { "$in": ["deep-learning"] } },
             { "asset_registry_metadata.framework": "PyTorch" }
           ]
         }'

# Health check
curl -X GET "http://<server-url>/check_health/123e4567-e89b-12d3-a456-426614174000"