Skip to content

Instantly share code, notes, and snippets.

@mikezupper

mikezupper/livepeer-byoc-openai-onboarding-guide.md

Created April 3, 2026 10:56
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mikezupper/c5c34201ce6376a83caffeab4ae11a01 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mikezupper/c5c34201ce6376a83caffeab4ae11a01 to your computer and use it in GitHub Desktop.
Download ZIP
Livepeer BYOC OpenAI - BlueClaw Orchestrator Onboarding Guide
Raw
livepeer-byoc-openai-onboarding-guide.md

BlueClaw GPU Provider Onboarding Guide

For Livepeer AI Orchestrators joining the BlueClaw Network

Version: 1.3 — March 29, 2026 BlueClaw: https://blueclaw.network Base URL: https://openai.blueclaw.network/v1

Overview

BlueClaw is an OpenAI-compatible inference gateway running on the Livepeer GPU network. It routes LLM, text embedding, and image generation requests to Livepeer AI Orchestrators via the BYOC (Bring Your Own Compute or Bring Your Own Container) runner framework.

This guide walks you through connecting your GPU infrastructure to BlueClaw so you can serve inference jobs and earn fees.

Platform: Linux only. Docker-based deployment. Windows is not supported. macOS may work but is not verified.

Extensible: This guide demonstrates Ollama and vLLM as inference backends, but any custom container can be implemented as a BYOC runner. See https://github.com/Cloud-SPE/livepeer-byoc-suite for additional runner implementations including custom reranking, video models, and more.

Architecture

BlueClaw Gateway
    │
    ▼
AI Service Registry (on-chain, Arbitrum)
    │
    ▼
Your AI Orchestrator (go-livepeer)
    │
    ▼ (via Cloudflare Tunnel)
BYOC Runners (chat completions / embeddings / image gen)
    │
    ▼
Inference Backend (Ollama, vLLM, or custom container)

How it works:

  1. The BlueClaw gateway discovers your orchestrator via the on-chain AI Service Registry
  2. When a request comes in, the gateway routes it to your orchestrator
  3. Your orchestrator forwards the job to your BYOC runner (via Cloudflare Tunnel)
  4. The BYOC runner proxies the request to your inference backend (Ollama or vLLM)
  5. Response flows back through the same path

Key components you'll deploy:

  • AI Orchestrator — go-livepeer registered on-chain
  • Cloudflare Tunnel — secure HTTPS connectivity (valid SSL certificates are required; self-signed certs are NOT acceptable). You can use non-Cloudflare tunnels as long as they provide valid HTTPS certificates.
  • Inference Backend — Ollama, vLLM, or any custom container
  • BYOC Runner(s) — proxy containers that register capabilities with your orch
  • WebUI (optional) — Open WebUI for local model testing

Prerequisites

  • Linux server with NVIDIA GPU(s) and Docker installed (Linux only — Windows is not supported, macOS is unverified)
  • Latest NVIDIA drivers installed
  • NVIDIA Container Toolkit (nvidia-container-toolkit) — not nvidia-docker2
  • An existing Livepeer AI Orchestrator with stake on Arbitrum One
  • Your orchestrator registered on the AI Service Registry (call setServiceURI with https://your-orch-domain:port)
  • A domain name you control
  • A Cloudflare account (free tier works)
  • ETH on Arbitrum One (for gas fees on registration transactions)
  • A HuggingFace token (if running vLLM or image generation models)

GPU Requirements by Capability

Capability Minimum GPU VRAM Recommended Models
Chat (small models) RTX 3090 24GB qwen3:8b, gemma-3-4b-it
Chat (medium models) RTX 4090/5090 24-32GB Qwen2.5-14B-AWQ
Chat (large models) A100/H100 or multi-GPU 48GB+ Llama-3.3-70B-FP8
Text Embeddings RTX 3090+ 24GB nomic-embed-text, SFR-Embedding-2_R
Image Generation (RealVisXL) RTX 4090+ 24GB+ SG161222/RealVisXL_V4.0_Lightning
Image Generation (FLUX.1) RTX 4090/5090+ 24GB+ black-forest-labs/FLUX.1-dev

Important notes:

  • Chat completions + text embeddings together use ~14GB VRAM on a 3090. Image generation requires a separate, dedicated GPU unless you have a 5090 or larger.
  • Running multiple Ollama models concurrently requires sufficient VRAM for all loaded models. If your GPU cannot run them all at once, you will get errors. Plan your model selection accordingly.
  • Depending on how many models you run or test, disk usage can grow considerably (models are multiple GB each). Plan storage accordingly.

Step 1: Create Docker Network and Volumes

All services share a Docker network for inter-container communication.

# Create shared network
docker network create ingress

# Create volumes for Ollama (if using Ollama backend)
docker volume create ollama
docker volume create open-webui

# Create volumes for vLLM (if using vLLM backend)
docker volume create huggingface-model-cache

# Create volumes for image generation (if applicable)
docker volume create ai-image-models
docker volume create ai-image-kernel_cache

# Create volume for orchestrator data
docker volume create ai-lpData

Step 2: Deploy the AI Orchestrator

2a. Create aiModels.json

The orchestrator requires an aiModels.json file to start. Create an empty one:

# Copy into your lpData volume
docker run --rm -v ai-lpData:/data alpine sh -c 'echo "[]" > /data/aiModels.json'

2b. Create a ticket redemption wallet

The orchestrator needs an Ethereum keystore file for ticket redemption. For security, use a separate wallet — NOT your main orchestrator staking wallet. This wallet just needs some ETH on Arbitrum for gas.

Generate a new keystore file (or use an existing one):

# Copy your keystore JSON file into the lpData volume
docker run --rm -v ai-lpData:/data -v /path/to/your/keystore.json:/tmp/key.json alpine \
  cp /tmp/key.json /data/key.json

Tip: If you don't have a keystore file, you can generate one with geth account new or any Ethereum wallet tool. Fund it with a small amount of ETH on Arbitrum One for gas.

2c. Create the ETH password file

docker run --rm -v ai-lpData:/data alpine sh -c 'echo "YOUR_KEYSTORE_PASSWORD" > /data/eth-secret.txt'

2d. Deploy the Orchestrator

Create docker-compose.ai-orch.yml:

services:
  orchestrator:
    image: tztcloud/go-livepeer:latest
    container_name: "ai-orch"
    restart: unless-stopped
    volumes:
      - ai-lpData:/root/.lpData
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - YOUR_ORCH_PORT:YOUR_ORCH_PORT
    command: >
      -network=arbitrum-one-mainnet
      -ethUrl=YOUR_ARBITRUM_RPC_URL
      -orchestrator=true
      -monitor=true
      -orchSecret=YOUR_ORCH_SECRET
      -serviceAddr=YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      -orchAddr=YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      -cliAddr=ai-orch:7935
      -v=6
      -pricePerUnit=250
      -autoAdjustPrice=false
      -ethKeystorePath=/root/.lpData
      -ethPassword=/root/.lpData/eth-secret.txt
      -ethOrchAddr=YOUR_ETH_ORCH_ADDRESS
      -ticketEV=2999999999999
      -aiModels=/root/.lpData/aiModels.json

volumes:
  ai-lpData:
    external: true

networks:
  default:
    name: ingress
    external: true

Important: Use tztcloud/go-livepeer:latest — there are pending changes that must merge before using official livepeer/go-livepeer images. Specifically, Cloud-SPE/go-livepeer#3 adds options filtering so gateways can match orchestrators and runners to capabilities for incoming jobs. Worker options (model, VRAM, etc.) registered by BYOC workers are returned under each orchestrator in /getNetworkCapabilities as capability_options keyed by capability name. This is required for proper job routing.

docker compose -f docker-compose.ai-orch.yml up -d

2e. Register on the AI Service Registry

If you haven't already, register your orchestrator on the AI Service Registry smart contract:

This is how the BlueClaw gateway discovers your orchestrator.

Step 3: Set Up the Cloudflare Tunnel

Valid HTTPS with trusted SSL certificates is required for all communication between the gateway, orchestrator, and runners. Self-signed certificates are not accepted.

Cloudflare Tunnels are the easiest way to get valid HTTPS. You can use alternative tunnel providers or reverse proxies (e.g., Traefik, Caddy with Let's Encrypt) as long as they provide valid, trusted SSL certificates.

3a. Create the Tunnel

  1. Log into Cloudflare Zero Trust
  2. Click Networks → Connectors
  3. Click Create a Tunnel → select cloudflared
  4. Name it (your hostname works)
  5. Copy the tunnel token

3b. Deploy cloudflared

Create docker-compose.cloudflared.yml:

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    networks:
      - ingress
    command: tunnel --no-autoupdate run --token YOUR_CLOUDFLARE_TUNNEL_TOKEN

networks:
  ingress:
    external: true
docker compose -f docker-compose.cloudflared.yml up -d

3c. Configure Published Application Routes

In the Cloudflare Zero Trust UI, click on your tunnel → Published Application Routes → Add Routes.

You'll need routes for each capability you're serving. Use a subdomain on your domain (e.g., openai-runner.yourdomain.com). All capabilities can share a single subdomain, differentiated by path.

Subdomain Path Service Type URL
openai-runner.yourdomain.com /v1/chat/completions HTTP openai_chat_completion_runner:8080
openai-runner.yourdomain.com /openai-chat-completions/options HTTP openai_chat_completion_runner:8080
openai-runner.yourdomain.com /v1/embeddings HTTP byoc_embeddings_runner:8080
openai-runner.yourdomain.com /openai-text-embeddings/options HTTP byoc_embeddings_runner:8080
openai-runner.yourdomain.com /v1/images/generations HTTP byoc_image_runner:8080
openai-runner.yourdomain.com /openai-image-generation/options HTTP byoc_image_runner:8080

Multi-GPU servers: If you plan to run more than one GPU server, each server will need its own Cloudflare tunnel and its own subdomain. This may change in the future with Traefik-based proxying, but that is not documented at this time. Most operators start with a single tunnel and single subdomain.

Step 4: Deploy the Inference Backend

Choose Ollama or vLLM as your inference backend. Ollama is simpler to set up. vLLM provides higher throughput, supports quantized models (AWQ, FP8), and is required for larger models. Any custom OpenAI-compatible container can also be used.

Option A: Ollama

Create docker-compose.ollama.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ports:
      - 11434:11434
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_NUM_PARALLEL=1
      - OLLAMA_CONTEXT_LENGTH=16384
      - OLLAMA_KEEP_ALIVE=300

  # Optional: Open WebUI for local testing and model verification
  # Remove this service if you don't need a web interface
  ollama-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-webui
    restart: unless-stopped
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=CHANGE_THIS_SECRET
    ports:
      - 3000:8080
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
    external: true
  open-webui:
    external: true

networks:
  default:
    name: ingress
    external: true
docker compose -f docker-compose.ollama.yml up -d

Pull the models you'll be serving:

# Chat model
docker exec ollama ollama pull qwen3:8b

# Embedding model
docker exec ollama ollama pull nomic-embed-text:latest

# Optional: additional chat models (if VRAM allows)
docker exec ollama ollama pull gemma3:4b

Pin a model to keep it hot-loaded (optional):

Set OLLAMA_KEEP_ALIVE=0 in the environment to keep models loaded indefinitely, or use a specific value in seconds (e.g., 300 = 5 minutes).

Option B: vLLM (For Larger Models / Higher Throughput)

vLLM is required for models that need quantization (AWQ, FP8) or when you need higher concurrent throughput.

Create docker-compose.vllm.yml:

services:
  vllm_model_runner:
    image: vllm/vllm-openai:nightly
    container_name: vllm_model_runner
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - huggingface-model-cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=YOUR_HF_TOKEN
    ipc: host
    command: >
      Qwen/Qwen2.5-14B-Instruct-AWQ
      --quantization awq
      --dtype half
      --max-model-len 32768
      --gpu-memory-utilization 0.90
      --max-num-seqs 4
      --enable-auto-tool-choice
      --tokenizer_mode auto
      --tool-call-parser hermes
      --enable-log-requests

volumes:
  huggingface-model-cache:
    external: true

networks:
  default:
    name: ingress
    external: true

Tested vLLM configurations by GPU:

RTX 5090 (32GB) — Tested Working
Model Quantization dtype max-model-len gpu-memory-util max-num-seqs tool-call-parser
google/gemma-3-4b-it — bfloat16 131072 0.50 16 pythonic
google/gemma-3-12b-it — bfloat16 32768 0.90 8 pythonic
mistralai/Ministral-3-3B-Instruct-2512 — bfloat16 131072 0.70 32 mistral
mistralai/Ministral-3-8B-Instruct-2512 — — 65536 0.75 — mistral
mistralai/Ministral-3-8B-Reasoning-2512 — bfloat16 65536 0.90 — mistral
mistralai/Ministral-3-14B-Instruct-2512 — float16 32768 0.85 — mistral
meta-llama/Llama-3.1-8B-Instruct — float16 32768 0.90 — —
Qwen/Qwen2.5-Coder-32B-Instruct-AWQ awq_marlin auto 32768 0.95 — hermes
Qwen/Qwen2.5-32B-Instruct-AWQ awq_marlin auto 16384 0.90 — hermes

Not working on 5090: mistralai/Ministral-3-14B-Reasoning-2512

RTX 4090 (24GB) — Tested Working
Model Quantization dtype max-model-len gpu-memory-util tool-call-parser
Qwen/Qwen2.5-32B-Instruct-AWQ awq half 4096 0.95 hermes
Qwen/Qwen2.5-14B-Instruct-AWQ awq half 32768 0.90 hermes

Step 5: Deploy BYOC Runners

BYOC runners are lightweight proxy containers that register capabilities with your orchestrator and route requests to your inference backend.

5a. Chat Completions Runner

Create docker-compose.chat-completions.yml:

services:
  openai_chat_completion_runner:
    image: registry.livepeer.tools/livepeer-byoc-openai-chat-completion-runner:v0.1.0
    container_name: openai_chat_completion_runner
    environment:
      - RUNNER_ADDR=:8080
      - UPSTREAM_URL=http://ollama:11434/v1/chat/completions

  register_chat_capability:
    image: registry.livepeer.tools/livepeer-byoc-register-capability:v0.1.1
    container_name: register_chat_capability
    environment:
      - ORCH_URL=https://YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      - ORCH_SECRET=YOUR_ORCH_SECRET
      - CAPABILITY_NAME=openai-chat-completions
      - CAPABILITY_URL=https://YOUR_CLOUDFLARE_RUNNER_SUBDOMAIN
      - PRICE_PER_UNIT=2500
      - CAPACITY=5
      - RETRIES=30
      - RETRY_DELAY_SECONDS=3
      - PERIODIC_REGISTRATION_ENABLED=true
      - PERIODIC_REGISTRATION_INTERVAL_SECONDS=500
      - PERIODIC_REGISTRATION_RETRIES=3
      - PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS=2
      - UNREGISTER_ON_SHUTDOWN=true
    depends_on:
      - openai_chat_completion_runner

networks:
  default:
    name: ingress
    external: true

UPSTREAM_URL: Points to your inference backend.

  • Ollama on same box: http://ollama:11434/v1/chat/completions
  • vLLM on same box: http://vllm_model_runner:8000/v1/chat/completions
  • Backend on different box: http://LAN_IP:PORT/v1/chat/completions

5b. Text Embeddings Runner

Create docker-compose.text-embeddings.yml:

services:
  byoc_embeddings_runner:
    image: registry.livepeer.tools/livepeer-byoc-openai-embeddings-runner:v0.1.0
    container_name: byoc_embeddings_runner
    environment:
      - RUNNER_ADDR=:8080
      - UPSTREAM_URL=http://ollama:11434/v1/embeddings

  register_embeddings_capability:
    image: registry.livepeer.tools/livepeer-byoc-register-capability:v0.1.1
    container_name: register_embeddings_capability
    environment:
      - ORCH_URL=https://YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      - ORCH_SECRET=YOUR_ORCH_SECRET
      - CAPABILITY_NAME=openai-text-embeddings
      - CAPABILITY_URL=https://YOUR_CLOUDFLARE_RUNNER_SUBDOMAIN
      - WORKER_OPTIONS=[{"model":"nomic-embed-text:latest"}]
      - PRICE_PER_UNIT=1000
      - CAPACITY=10
      - RETRIES=30
      - RETRY_DELAY_SECONDS=3
      - PERIODIC_REGISTRATION_ENABLED=true
      - PERIODIC_REGISTRATION_INTERVAL_SECONDS=500
      - PERIODIC_REGISTRATION_RETRIES=3
      - PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS=2
      - UNREGISTER_ON_SHUTDOWN=true
    depends_on:
      - byoc_embeddings_runner

networks:
  default:
    name: ingress
    external: true

5c. Image Generation Runner (4090+ only)

Create docker-compose.image-generation.yml:

services:
  byoc_image_runner:
    image: tztcloud/livepeer-byoc-openai-image-generation-runner:v0.1.0
    container_name: byoc_image_runner
    runtime: nvidia
    environment:
      - HF_TOKEN=YOUR_HUGGINGFACE_TOKEN
      - MODEL_ID=SG161222/RealVisXL_V4.0_Lightning
      - MODEL_DIR=/models
      - RUNNER_PORT=8080
      - DEVICE=cuda
      - DTYPE=float16
      - MAX_QUEUE_SIZE=10
      - USE_TORCH_COMPILE=false
      - PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
    volumes:
      - ai-image-models:/models
      - ai-image-kernel_cache:/cache

  register_image_capability:
    image: registry.livepeer.tools/livepeer-byoc-register-capability:v0.1.1
    container_name: register_image_capability
    environment:
      - ORCH_URL=https://YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      - ORCH_SECRET=YOUR_ORCH_SECRET
      - CAPABILITY_NAME=openai-image-generation
      - CAPABILITY_URL=https://YOUR_CLOUDFLARE_RUNNER_SUBDOMAIN
      - PRICE_PER_UNIT=5000
      - CAPACITY=10
      - RETRIES=30
      - RETRY_DELAY_SECONDS=3
      - PERIODIC_REGISTRATION_ENABLED=true
      - PERIODIC_REGISTRATION_INTERVAL_SECONDS=500
      - PERIODIC_REGISTRATION_RETRIES=3
      - PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS=2
      - UNREGISTER_ON_SHUTDOWN=true
    depends_on:
      - byoc_image_runner
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
              driver: nvidia
              count: all

volumes:
  ai-image-models:
    external: true
  ai-image-kernel_cache:
    external: true

networks:
  default:
    name: ingress
    external: true

Image generation models by GPU:

Model DTYPE Min VRAM Notes
SG161222/RealVisXL_V4.0_Lightning float16 ~15GB Fast, good quality. 4090+
black-forest-labs/FLUX.1-dev bfloat16 ~24GB+ Best quality. Works on 4090/5090 with settings shown above. Needs HF token.

5d. Rerank Runner (Experimental — Future BlueClaw Capability)

Note: Reranking is not currently used by BlueClaw but will be added in the future. Set this up if you want to be ready or participate in testing.

The rerank runner provides a Cohere-compatible /v1/rerank endpoint powered by zerank-2 (4B CrossEncoder based on Qwen3-4B).

Source: github.com/Cloud-SPE/livepeer-byoc-rerank-runner

Pre-built images are available. You'll need to create a volume for model weights (the runner downloads them on first start).

docker volume create ai-rerank-models

Create docker-compose.rerank.yml:

services:
  byoc_rerank_runner:
    image: tztcloud/byoc-rerank-runner:v0.0.7
    container_name: byoc_rerank_runner
    runtime: nvidia
    environment:
      - MODEL_ID=zeroentropy/zerank-2
      - MODEL_DIR=/models
      - RUNNER_PORT=8080
      - DEVICE=cuda
      - DTYPE=bfloat16
      - MAX_QUEUE_SIZE=5
      - MAX_BATCH_SIZE=1000
      - INFERENCE_BATCH_SIZE=64
    volumes:
      - ai-rerank-models:/models
    ports:
      - "9999:8080"
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
              driver: nvidia
              count: all

  register_rerank_capability:
    image: registry.livepeer.tools/livepeer-byoc-register-capability:v0.1.1
    container_name: register_rerank_capability
    environment:
      - ORCH_URL=https://YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      - ORCH_SECRET=YOUR_ORCH_SECRET
      - CAPABILITY_NAME=cohere-rerank
      - CAPABILITY_URL=https://YOUR_CLOUDFLARE_RUNNER_SUBDOMAIN
      - PRICE_PER_UNIT=1000
      - CAPACITY=5
      - RETRIES=30
      - RETRY_DELAY_SECONDS=3
      - PERIODIC_REGISTRATION_ENABLED=true
      - PERIODIC_REGISTRATION_INTERVAL_SECONDS=500
      - PERIODIC_REGISTRATION_RETRIES=3
      - PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS=2
      - UNREGISTER_ON_SHUTDOWN=true
    depends_on:
      - byoc_rerank_runner

volumes:
  ai-rerank-models:
    external: true

networks:
  default:
    name: ingress
    external: true

Add a Cloudflare tunnel route:

Subdomain Path Service Type URL
openai-runner.yourdomain.com /v1/rerank HTTP byoc_rerank_runner:8080
openai-runner.yourdomain.com /cohere-rerank/options HTTP byoc_rerank_runner:8080

Test locally:

curl -X POST http://localhost:8080/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is deep learning?",
    "documents": ["Deep learning uses neural networks.", "The weather is nice."],
    "top_n": 2,
    "return_documents": true
  }'

5e. Video Generation Runner (Experimental — Future BlueClaw Capability)

Note: Video generation is not currently used by BlueClaw but is under active development and will be added in the future. This requires significant GPU resources.

The video runner generates multi-clip videos from text prompts using LightX2V (Wan 2.2 TI2V-5B), with Real-ESRGAN 4x upscaling, RIFE frame interpolation, and S3 delivery.

Source: github.com/Cloud-SPE/livepeer-byoc-video-runner

Requirements:

  • RTX 4090 / 5090 or larger GPU
  • S3-compatible object storage (AWS S3, MinIO, etc.) for video delivery
  • Significant disk space for model weights (~10GB+) and temp files

Pre-built images are available. You'll need to download model weights and deploy two stacks: the LightX2V inference server and the video runner itself.

Step 1: Download model weights

# Create volumes
docker volume create ai-video-models
docker volume create ai-video-temp

# Download Wan 2.2 TI2V-5B + RealESRGAN weights (~10GB)
docker run --rm -v ai-video-models:/models \
  tztcloud/livepeer-byoc-model-downloader:latest

Step 2: Deploy LightX2V inference server

Create docker-compose.lightx2v.yml:

services:
  lightx2v:
    image: tztcloud/livepeer-byoc-lightx2v:v0.0.7
    container_name: lightx2v
    runtime: nvidia
    ports:
      - 8000:8000
    command: >
      bash -c "python -m lightx2v.server
      --model_path $$MODEL_PATH
      --model_cls $$MODEL_CLS
      --task $$TASK
      --config_json $$CONFIG_JSON
      --host 0.0.0.0
      --port 8000"
    volumes:
      - ai-video-models:/models
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - MODEL_PATH=/models/Wan2.2-TI2V-5B
      - MODEL_CLS=wan2.2
      - TASK=i2v
      - CONFIG_JSON=/workspace/LightX2V/configs/wan_ti2v_i2v_4090.json
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
              driver: nvidia
              count: all

volumes:
  ai-video-models:
    external: true

networks:
  default:
    name: ingress
    external: true
docker compose -f docker-compose.lightx2v.yml up -d

Step 3: Deploy the video runner

Create docker-compose.video-runner.yml with your S3 and orchestrator settings:

services:
  byoc_video_runner:
    image: tztcloud/livepeer-byoc-video-runner:latest
    container_name: byoc_video_runner
    runtime: nvidia
    environment:
      - RUNNER_PORT=8080
      - LIGHTX2V_URL=http://lightx2v:8000
      - S3_ENDPOINT=YOUR_S3_ENDPOINT
      - S3_ACCESS_KEY=YOUR_S3_ACCESS_KEY
      - S3_SECRET_KEY=YOUR_S3_SECRET_KEY
      - S3_BUCKET=YOUR_S3_BUCKET
      - S3_REGION=us-east-1
      - S3_USE_SSL=true
      - S3_PUBLIC_URL=YOUR_S3_PUBLIC_URL
    volumes:
      - ai-video-models:/models
      - ai-video-temp:/tmp/video
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
              driver: nvidia
              count: all

  register_video_capability:
    image: registry.livepeer.tools/livepeer-byoc-register-capability:v0.1.1
    container_name: register_video_capability
    environment:
      - ORCH_URL=https://YOUR_ORCH_DOMAIN:YOUR_ORCH_PORT
      - ORCH_SECRET=YOUR_ORCH_SECRET
      - CAPABILITY_NAME=video-pipeline-generation
      - CAPABILITY_URL=https://YOUR_CLOUDFLARE_RUNNER_SUBDOMAIN
      - PRICE_PER_UNIT=250
      - CAPACITY=1
      - RETRIES=30
      - RETRY_DELAY_SECONDS=3
      - PERIODIC_REGISTRATION_ENABLED=true
      - PERIODIC_REGISTRATION_INTERVAL_SECONDS=500
      - PERIODIC_REGISTRATION_RETRIES=3
      - PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS=2
      - UNREGISTER_ON_SHUTDOWN=true
    depends_on:
      - byoc_video_runner

volumes:
  ai-video-models:
    external: true
  ai-video-temp:
    external: true

networks:
  default:
    name: ingress
    external: true
docker compose -f docker-compose.video-runner.yml up -d

This starts:

  • byoc_video_runner — video generation runner (port 8080), connects to lightx2v for inference
  • register_video_capability — registers video-pipeline-generation capability with the orch

Add Cloudflare tunnel routes:

Subdomain Path Service Type URL
openai-runner.yourdomain.com /v1/video/pipeline/generations HTTP byoc_video_runner:8080
openai-runner.yourdomain.com /v1/video/pipeline/generations/status HTTP byoc_video_runner:8080
openai-runner.yourdomain.com /video-pipeline-generation/options HTTP byoc_video_runner:8080

Key environment variables:

Variable Description
S3_ENDPOINT S3/MinIO endpoint URL
S3_ACCESS_KEY Access key ID
S3_SECRET_KEY Secret access key
S3_BUCKET Bucket name
S3_PUBLIC_URL Public URL prefix for generated video URLs

Test locally:

# Submit a job
curl -sS http://localhost:8080/v1/video/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A cat walking on a beach at sunset, cinematic",
    "num_clips": 2,
    "num_inference_steps": 30,
    "upscale": false,
    "interpolate": false
  }'
# Returns: {"job_id": "...", "status": "processing"}

# Poll for status
curl -sS http://localhost:8080/v1/video/generations/status \
  -H "Content-Type: application/json" \
  -d '{"job_id": "YOUR_JOB_ID"}'

Step 6: Start Everything

Recommended startup order:

# 1. Orchestrator
docker compose -f docker-compose.ai-orch.yml up -d

# 2. Cloudflare Tunnel
docker compose -f docker-compose.cloudflared.yml up -d

# 3. Inference Backend
docker compose -f docker-compose.ollama.yml up -d
# OR
docker compose -f docker-compose.vllm.yml up -d

# 4. Pull models (Ollama only)
docker exec ollama ollama pull qwen3:8b
docker exec ollama ollama pull nomic-embed-text:latest

# 5. BYOC Runners (start each one)
docker compose -f docker-compose.chat-completions.yml up -d
docker compose -f docker-compose.text-embeddings.yml up -d
# docker compose -f docker-compose.image-generation.yml up -d  # if applicable

Verify Runner Registration

When each BYOC runner stack starts, you'll see two containers:

  • The runner (e.g., openai_chat_completion_runner) — stays running
  • The capability registrar (e.g., register_chat_capability) — registers with the orch, then exits

Check the registrar logs to confirm:

docker logs register_chat_capability
# Should show: capability registered successfully

docker logs register_embeddings_capability
# Should show: capability registered successfully

With PERIODIC_REGISTRATION_ENABLED=true, the registrar will re-register every 500 seconds to keep the capability alive. It will also unregister on shutdown (UNREGISTER_ON_SHUTDOWN=true).

Step 7: Verify with BlueClaw

  1. Go to https://blueclaw.network
  2. Sign up and get verified
  3. Get your API key
  4. Go to the Playground and submit a request

Your orchestrator should receive the job. Check your orch logs:

docker logs ai-orch --tail 50 -f

You can also test via curl:

curl https://openai.blueclaw.network/v1/chat/completions \
  -H "Authorization: Bearer YOUR_BLUECLAW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Test your runner directly (before BlueClaw verification):

You can also test your inference backend and runner locally before going through BlueClaw:

# Test Ollama directly
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:8b", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 50}'

# Test the BYOC runner directly (if on same network)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:8b", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 50}'

You can also use the OpenAI Python SDK pointed at your local runner for more thorough testing.

Note: Enhanced end-to-end verification tooling is coming soon. For now, the BlueClaw playground is the primary way to verify your setup is reachable from the network.

BlueClaw Supported Models

Models currently enabled on BlueClaw (subject to change):

Chat Models

Model ID Name Backend Min GPU
qwen3:8b Qwen 3 8B Ollama 3090
google/gemma-3-4b-it Gemma 3 4B Ollama 3090
Qwen/Qwen2.5-14B-Instruct-AWQ Qwen 2.5 14B vLLM 4090
RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic Llama 3.3 70B vLLM A100/H100

Embedding Models

Model ID Name Backend Min GPU
nomic-embed-text Nomic Embed Text Ollama 3090
Salesforce/SFR-Embedding-2_R SFR Embedding 2 Ollama/vLLM 3090

Image Models

Model ID Name Backend Min GPU
SG161222/RealVisXL_V4.0_Lightning RealVisXL V4.0 Custom runner 4090
black-forest-labs/FLUX.1-dev FLUX.1 Dev Custom runner 4090+

Environment Variable Reference

BYOC Register Capability (v0.1.1)

Variable Description Example
ORCH_URL Your orchestrator's URL (https) https://your-orch:18935
ORCH_SECRET Orchestrator secret (matches -orchSecret) your-secret
CAPABILITY_NAME Capability identifier openai-chat-completions
CAPABILITY_URL Public URL via Cloudflare tunnel https://openai-runner.yourdomain.com
PRICE_PER_UNIT Price per unit for this capability 2500
CAPACITY Max concurrent requests 5
RETRIES Registration retry attempts 30
RETRY_DELAY_SECONDS Delay between retries 3
PERIODIC_REGISTRATION_ENABLED Re-register periodically true
PERIODIC_REGISTRATION_INTERVAL_SECONDS Re-registration interval 500
PERIODIC_REGISTRATION_RETRIES Retries per periodic registration 3
PERIODIC_REGISTRATION_RETRY_DELAY_SECONDS Delay between periodic retries 2
UNREGISTER_ON_SHUTDOWN Clean up capability on container stop true
WORKER_OPTIONS Model-specific options (embeddings) [{"model":"nomic-embed-text:latest"}]

Capability Names

Capability CAPABILITY_NAME Status
Chat Completions openai-chat-completions Active on BlueClaw
Text Embeddings openai-text-embeddings Active on BlueClaw
Image Generation openai-image-generation Active on BlueClaw
Reranking cohere-rerank Future — not yet on BlueClaw
Video Generation video-pipeline-generation Future — not yet on BlueClaw

Pricing

Pricing per capability is set via PRICE_PER_UNIT in the register capability container. Current reference values:

Capability Reference PRICE_PER_UNIT Notes
Chat Completions 2500 Varies by model size
Text Embeddings 1000
Image Generation 5000 Higher due to GPU intensity

Note: Pricing per model and per GPU type is subject to change as the network evolves. These values will be reviewed and standardized as more operators join.

Troubleshooting

"No orchestrators found for capability"

  • Your BYOC runner's capability registration failed or expired
  • Check register_* container logs
  • Verify CAPABILITY_NAME matches exactly (e.g., openai-chat-completions, not ollama-openai)
  • Ensure PERIODIC_REGISTRATION_ENABLED=true so capabilities don't expire

TLS handshake errors

  • Verify your Cloudflare tunnel is running and routes are configured
  • Ensure your orch's serviceAddr uses https:// format
  • Check that the domain in setServiceURI matches exactly

"Failed to get token from Orchestrator"

  • The gateway can reach your orch but auth failed
  • Verify your orch is registered on the AI Service Registry
  • Check that your orch is running and the port is accessible

Ollama unloads model automatically

  • Set OLLAMA_KEEP_ALIVE=0 for indefinite loading
  • Or set a high value in seconds (e.g., 3600 for 1 hour)

Livepeer won't start without aiModels.json

  • Create an empty JSON array file: echo "[]" > aiModels.json
  • Mount it in the orch volume at /root/.lpData/aiModels.json

Concurrent model errors with Ollama

  • If you load multiple models but your GPU can't run them all concurrently, you'll get errors
  • Check total VRAM usage: nvidia-smi
  • Either reduce the number of loaded models or set OLLAMA_NUM_PARALLEL=1 to serialize requests

Running out of disk space

  • Model files are large (4-30GB+ each). Monitor disk usage with df -h and docker system df
  • Clean unused models: docker exec ollama ollama rm MODEL_NAME
  • Clean Docker build cache: docker system prune

Quick Reference: 3090 Operator Setup

For operators with RTX 3090 GPUs, here's the minimal path:

  1. Orch — tztcloud/go-livepeer:latest registered on AI Service Registry
  2. Cloudflare Tunnel — one tunnel, one subdomain
  3. Ollama — pull qwen3:8b + nomic-embed-text:latest
  4. Chat Runner — UPSTREAM_URL=http://ollama:11434/v1/chat/completions
  5. Embeddings Runner — UPSTREAM_URL=http://ollama:11434/v1/embeddings
  6. No image generation — requires 4090+
  7. Verify — sign up at blueclaw.network, test via playground

BYOC Suite — Full Source Reference

All BYOC runner source code is open source under the Cloud-SPE organization:

Repository Description
livepeer-byoc-suite Monorepo (git submodules) aggregating all BYOC services
livepeer-byoc-openai-runners Chat completions + embeddings runners (Go)
livepeer-byoc-register-capabilities Capability registration init container (Go)
livepeer-byoc-rerank-runner Cohere-compatible reranking with zerank-2 (Python/FastAPI)
livepeer-byoc-video-runner Multi-clip video generation with LightX2V/Wan 2.2 (Python/FastAPI)
livepeer-byoc-transcode-runners Transcoding runners
livepeer-byoc-gateway-proxy Gateway proxy — OpenAI/Cohere-compatible API in front of Livepeer Gateway (Go)

You can build custom runners for any inference workload. The BYOC framework only requires:

  1. A container that exposes an HTTP endpoint for inference
  2. A register-capabilities sidecar to advertise the capability to the orchestrator

Version History

Date Version Changes
2026-03-29 1.3 Use pre-built registry images for rerank/video runners (no local builds). Added LightX2V compose. Video runner split into lightx2v + runner stacks.
2026-03-29 1.2 Added: ticket redemption wallet setup, pending go-livepeer PR for options filtering, rerank runner section, video generation runner section, full BYOC suite source reference
2026-03-29 1.1 Clarifications: BYOC = Bring Your Own Compute/Container, HTTPS cert requirements, single subdomain routing, FLUX.1 works on 4090/5090, disk/VRAM planning, local runner testing
2026-03-29 1.0 Initial guide — v0.1.0 BYOC runners, Ollama + vLLM backends
2026-02-20 (beta) v0.0.7 runners, capability name change from ollama-openai to openai-chat-completions
2026-02-09 (alpha) Original beta tester notes

BlueClaw — Decentralized AI inference on the Livepeer network. Questions? Reach out to the @mike_zoop team on Discord.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment