Skip to content

Instantly share code, notes, and snippets.

@synchronic1

synchronic1/medina-14b-benchmark-paper.md

Created May 5, 2026 07:26
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save synchronic1/22ad2e229fe760f0ccd5313f53adea59 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save synchronic1/22ad2e229fe760f0ccd5313f53adea59 to your computer and use it in GitHub Desktop.
Download ZIP
Medina 14B: Multi-GPU LoRA Hot-Swap Inference Benchmark
Raw
medina-14b-benchmark-paper.md

Medina 14B: Multi-GPU LoRA Hot-Swap Inference Benchmark

A Specialized Model Architecture for Dynamic Persona Adaptation

Date: 2026-05-05
Author: Peter (Synchronic1) / Carlsson Creative
Host: VM104 (192.168.1.243), Proxmox VM on Dell PowerEdge T630
llama.cpp Version: 8240 (d088d5b74)

Abstract

This paper presents the complete training-to-inference pipeline for Medina 14B, a specialized Qwen3-14B-based language model tuned for multi-persona agent workloads. Unlike merged models where base weights and LoRA adapters are fused at training time, this work maintains an unmerged architecture: a Q4_K_M quantized base model stays permanently resident in GPU VRAM while task-specific LoRA adapters are hot-swapped in ~13-18ms via llama.cpp's HTTP API. We benchmark this architecture across three distinct GPUs — NVIDIA GeForce RTX 3090 (24 GB), NVIDIA CMP 100-210 (16 GB), and NVIDIA GeForce RTX 3060 (12 GB) — demonstrating that the unmerged approach enables sub-20ms persona switching without reloading an ~8.4 GB base model. The CMP 100-210 is designated as the dedicated inference GPU, leaving the RTX 3090 and RTX 3060 available for training, serving, and other compute workloads. All benchmarks are reproducible with fixed environment variables on a single Debian Trixie VM.

1. The Medina 14B Model

1.1 Architecture

Medina 14B is derived from TeichAI/Qwen3-14B-Claude-4.5-Opus-High-Reasoning-Distill, a 14-billion parameter decoder-only transformer built on the Qwen3 architecture. The model features:

Parameter Value
Architecture Qwen3
Parameters 14B
Hidden size 5,120
Attention heads 40 (query) / 8 (key-value, GQA)
Head dimension 128
Layers 48
MLP intermediate size 17,408
Context length (trained) 40,960 tokens
RoPE frequency base 1,000,000
Quantization Q4_K_M (~4-bit, mixed precision)
Base model size (Q4_K_M) ~8.4 GB
KV cache per token (FP16) ~192 KB

The model was chosen as the base for Medina because of its extended context window (40K tokens), Grouped-Query Attention (GQA) which reduces KV cache memory by 5× relative to standard multi-head attention, and its high-quality distillation from Claude 4.5 Opus reasoning patterns.

1.2 Training History and LoRA Adapters

The Medina project has produced four LoRA adapters through iterative fine-tuning:

Adapter Rank Alpha Base Model Compatibility Status
openclaw_tools_v2 32 64 Qwen3-14B (17,408 MLP) ✅ Active
medina-14b-defensive 32 64 Qwen3-14B (17,408 MLP) ✅ Active
medina-14b-offensive 32 64 Qwen3-14B (17,408 MLP) ✅ Active
calibration_rank32 32 64 Qwen2.5-14B (13,824 MLP) ❌ Incompatible

The calibration_rank32 adapter was trained on Qwen2.5-14B, which uses intermediate_size=13824 versus Qwen3's intermediate_size=17408. This dimension mismatch makes it permanently incompatible with the Qwen3 base. All active adapters were converted to llama.cpp's native GGUF LoRA format (~491 MB each) using convert_lora_to_gguf.py.

1.3 Why Unmerged?

A merged model fuses the LoRA weights (ΔW = B×A) into the base weights (W' = W + ΔW) at export time. This produces a single file but loses the ability to swap adapters. An unmerged model keeps the base and adapters separate:

  • Base model (8.4 GB) loads once into VRAM and never moves
  • LoRA adapter (~500 MB) is applied dynamically to the forward pass
  • Swap latency is limited only by file I/O and tensor pointer updates (~13-18ms)

This architecture is essential for agent systems where the same base model must serve multiple personas (tool-calling, defensive coding, offensive security) without per-request model loading overhead.

2. Test Environment

2.1 Host Hardware: VM104

VM104 is a Proxmox VM running on a Dell PowerEdge T630 server:

Spec Value
CPU Intel Xeon E5-2695 v4 @ 2.10 GHz
Cores / Threads 18 cores / 36 threads (visible to VM)
System RAM 82.4 GB total / ~45 GB free
OS Debian GNU/Linux 13 (trixie)
Kernel Linux 6.8.0-110-generic
Containerization LXC / Docker (multiple bridge networks)
IP Address 192.168.1.243 (primary)

2.2 GPU Specifications

Three NVIDIA GPUs are passed through to the VM:

GPU VRAM Compute CUDA Dev PCIe Role
CMP 100-210 16 GB 7.0 1 x16 Gen3 Dedicated LoRA inference
RTX 3090 24 GB 8.6 0 x16 Gen3 Training / fallback inference
RTX 3060 12 GB 8.6 2 x16 Gen3 Secondary / overflow

Important: llama.cpp enumerates CUDA devices by compute capability, not PCI bus order. The RTX 3090 and RTX 3060 (compute 8.6) map to devices 0 and 2, while the CMP 100-210 (compute 7.0) maps to device 1. nvidia-smi shows the opposite: GPU 0 = CMP, GPU 1 = RTX 3090.

2.3 Software Stack

Component Version
NVIDIA Driver 570.211.01
CUDA Runtime 12.8 (driver) / 11.8 (PyTorch)
PyTorch 2.7.1+cu118
Transformers 5.7.0
PEFT 0.19.1
llama.cpp 8240 (d088d5b74)
Python 3.13.5
cuDNN Bundled with CUDA 11.8

2.4 Fixed Environment Variables

All GPUs share the same environment configuration on VM104:

export LD_LIBRARY_PATH=/opt/data/home/cuda_libs:/opt/data/home/llama.cpp_gpu

This path includes:

  • CUDA 12.8 runtime libraries (libcudart.so, libcublas.so, etc.) at /opt/data/home/cuda_libs/
  • llama.cpp shared libraries (libggml.so, libmtmd.so, etc.) at /opt/data/home/llama.cpp_gpu/

Without this variable, llama-server fails with error while loading shared libraries: libmtmd.so.0.

2.5 File Layout

/opt/data/home/
├── models/
│   ├── Qwen3-14B-Claude-4.5-Opus-Distill.q4_k_m.gguf   (8.4 GB, base)
│   ├── openclaw_tools_v2_lora_qwen3.gguf                (491 MB)
│   ├── defensive_lora_qwen3.gguf                        (491 MB)
│   └── offensive_lora_qwen3.gguf                        (491 MB)
├── llama.cpp_gpu/
│   └── llama-server                                     (server binary)
├── cuda_libs/                                           (CUDA .so files)
└── scripts/
    ├── start-lora-server.sh
    ├── swap-adapter.sh
    └── healthcheck.sh

3. Experimental Design

3.1 Methodology

Each GPU configuration was tested with a standardized protocol:

  1. Kill any running llama-server processes
  2. Start llama-server with the specified --main-gpu, -c context, and --split-mode
  3. Wait for the HTTP health endpoint (/health) to return {"status":"ok"}
  4. Measure VRAM consumption via nvidia-smi
  5. Benchmark base model inference (64 tokens, simple prompt)
  6. Benchmark LoRA hot-swap latency (POST /lora-adapters)
  7. Benchmark inference under each adapter (128 tokens, coding prompt)
  8. Unload adapters and measure return-to-base latency

All tests used:

  • Temperature: 0.1 for base / 0.7 for LoRA
  • n_predict: 64 (base) / 128 (LoRA)
  • Prompts: Controlled to ensure comparable prompt lengths
  • Measurement: llama.cpp's internal timings JSON field for tok/s and TTFT

3.2 Tested Configurations

Configuration GPU Context Mode Purpose
CMP_100_210_36K CMP (dev 1) 36,000 Solo Production (recommended)
RTX_3090_40K RTX 3090 (dev 0) 40,000 Solo Maximum performance
RTX_3060_8K RTX 3060 (dev 2) 8,192 Solo Minimum viable
ALL_3_LAYER_8K All 3 8,192 Layer-split Comparative baseline

4. Results

4.1 Inference Speed (tokens/second)

Configuration Base Model Tools LoRA Defensive LoRA Offensive LoRA
CMP 100-210 @ 36K 47.3 47.5 47.5 47.5
RTX 3090 @ 40K 78.1 78.3 78.3 78.5
RTX 3060 @ 8K 36.5 35.9 36.1 36.0
All 3 layer-split @ 8K 54.0 54.5 54.6 54.5

Key observation: Speed is consistent across adapters on the same GPU. The LoRA application adds negligible overhead to the forward pass because the low-rank matrices (B×A, where B is 17408×32 and A is 32×5120) represent a tiny fraction of the total computation compared to the base weight matrices.

4.2 Time-To-First-Token (TTFT)

Configuration TTFT (ms)
CMP 100-210 @ 36K 128.7
RTX 3090 @ 40K 123.0
RTX 3060 @ 8K 141.4
All 3 layer-split @ 8K 151.2

TTFT measures prompt processing time. The RTX 3090 is marginally faster due to higher SM clock and newer architecture. The layer-split configuration is slowest because tensor transfers across PCIe introduce synchronization overhead.

4.3 LoRA Hot-Swap Latency

Configuration Tools Defensive Offensive Unload
CMP 100-210 @ 36K 14 ms 18 ms 13 ms 13 ms
RTX 3090 @ 40K 14 ms 13 ms 13 ms 17 ms
RTX 3060 @ 8K 13 ms 12 ms 13 ms 14 ms
All 3 layer-split @ 8K 12 ms 13 ms 14 ms 14 ms

Critical finding: Hot-swap latency is sub-20ms across all GPUs. This is bounded by file I/O (reading the ~500 MB GGUF LoRA from NVMe SSD) and the tensor pointer reassignment in llama.cpp's KV cache manager, not by GPU compute. The base model — all 8.4 GB of it — never leaves VRAM.

4.4 VRAM Utilization

Configuration VRAM Used VRAM Free Total
CMP 100-210 @ 36K 14,691 MB 1,455 MB 16,384 MB
RTX 3090 @ 40K 15,071 MB 9,064 MB 24,576 MB
RTX 3060 @ 8K 9,893 MB 2,028 MB 12,288 MB
All 3 layer-split @ 8K 3,391 MB* 12,755 MB* 24,576 MB*

* Layer-split distributes layers across all 3 GPUs; the RTX 3090 (primary) shows the lowest usage because it only holds a subset of layers.

4.5 Context Window Ceiling Tests

Additional tests determined the maximum safe context for each GPU:

GPU Tested Contexts Max Working VRAM at Max Notes
CMP 100-210 8K → 36K → 40K 36K (safe) / 40K (tight) 14.7 GB @ 36K / 15.3 GB @ 40K 40K leaves only ~800 MB free; risk of OOM under burst load
RTX 3090 8K → 32K → 40K → 48K (OOM) 40K 15.1 GB Model training limit (40,960); 48K OOMs because KV cache alone needs ~7.5 GB
RTX 3060 8K → 16K → 28K → 40K 40K ~15.1 GB Can reach model training ceiling with minimal safety margin

Model ceiling: The GGUF metadata reports n_ctx_train = 40960. Beyond this, llama.cpp warns n_ctx_seq > n_ctx_train -- possible training context overflow. The 40K context ceiling is therefore a model property, not a GPU limitation.

4.6 KV Cache Memory Model

For Qwen3-14B with GQA (8 KV heads, 128 head dim, 48 layers, FP16):

Per-token KV cache = 2 × layers × head_count_kv × head_dim × 2 bytes
                   = 2 × 48 × 8 × 128 × 2
                   ≈ 192 KB per token

At 40K context: 40,000 × 192 KB ≈ 7.5 GB of KV cache alone. Adding the 8.2 GB model weights gives ~15.7 GB minimum, which explains why the CMP at 40K is so tight.

5. Analysis

5.1 GPU Selection for Production

Criterion CMP 100-210 RTX 3090 RTX 3060
Inference speed 47 tok/s 78 tok/s 36 tok/s
Context capacity 36K safe 40K max 8K safe / 40K tight
VRAM headroom 1.4 GB 9.0 GB 2.0 GB @ 8K
Availability ✅ Dedicated ✅ Free for training ✅ Free for serving
Best use LoRA inference Training / heavy batch Secondary serving

The CMP 100-210 is the optimal dedicated inference GPU for this workload:

  • Its 16 GB VRAM is sufficient for the full model + 36K context + 1.4 GB safety margin
  • 47 tok/s is more than adequate for interactive agent responses
  • Keeping it separate leaves the RTX 3090 fully available for training new adapters or running merged models
  • The RTX 3060 serves as an overflow / secondary inference node

5.2 Why Layer-Split Underperforms

The all-3-GPU layer-split configuration achieves only 54 tok/s — less than the RTX 3090 alone (78 tok/s). This is because:

  1. PCIe synchronization: Every layer transition requires tensor transfers across the PCIe bus
  2. Bottleneck effect: The CMP 100-210 (compute 7.0, slower SMs) becomes the limiting factor
  3. Overhead: Flash Attention and KV cache management add coordination costs across devices

For a single-user inference workload, solo GPU is always preferred over multi-GPU splitting.

5.3 The Case for Unmerged Architecture

Metric Merged Model Unmerged + Hot-Swap
Model files 1 per persona 1 base + N adapters
Storage 8.4 GB × 4 = 33.6 GB 8.4 GB + 3 × 0.5 GB = 9.9 GB
VRAM per persona Full model reload Base stays, adapter swaps
Persona switch time 30-60 seconds ~15 ms
Flexibility Static Dynamic, runtime-selectable

The unmerged architecture reduces storage by 70% and eliminates per-persona loading latency entirely. For an agent system with 3+ personas, this is the only viable architecture.

6. Model Development: Tuning for the Workload

6.1 Design Philosophy

Medina 14B was not trained as a general-purpose model. It was designed from inception as a workload-specific inference substrate with these constraints:

  1. Multi-persona agent system: The same base model must serve tool-calling, defensive security review, and offensive penetration testing without cross-contamination
  2. Low-latency switching: Persona changes must be imperceptible to the user (< 100ms)
  3. Resource efficiency: The inference GPU must be separable from training GPUs
  4. Context depth: Code review and log analysis require 20K-30K token windows

These constraints led directly to the unmerged LoRA architecture. A merged model would require either (a) loading three separate 8.4 GB models into VRAM (impossible on 16 GB), or (b) reloading the model on every persona switch (unacceptable latency).

6.2 Adapter Specialization

Each adapter represents a distinct optimization objective:

openclaw_tools_v2: Fine-tuned on function-calling conversations with JSON schema adherence. The model learns to emit structured tool invocations ({"name": "scan_port", "arguments": {"host": "..."}}) rather than free-form text.

medina-14b-defensive: Trained on secure coding patterns — input validation, parameterized queries, memory safety, and CWE remediation. The persona reasons about attack surfaces before generating code.

medina-14b-offensive: Trained on penetration testing methodologies, vulnerability exploitation, and red-team reconnaissance. The persona operates under an explicit "authorized testing only" system prompt.

All adapters use rank-32 LoRA with alpha-64 (scaling factor = 2.0). This was chosen empirically: rank-16 produced insufficient specialization, while rank-64 approached full fine-tuning with diminishing returns. The 32/64 configuration gives ~1.5% trainable parameters (17408 × 5120 × 2 parameters per layer × 48 layers vs. 32 × (17408 + 5120) × 2 × 48), making adapter files compact at ~500 MB.

6.3 Training Infrastructure

Training occurred on the same hardware cluster:

  • Primary training GPU: RTX 3090 (24 GB) for full fine-tuning experiments
  • Secondary training: CMP 100-210 for LoRA-only training (fits in 16 GB with gradient checkpointing)
  • CPU fallback: ClawBaby (.242, Intel Xeon, 94 GB RAM) for CPU-only PeFT experiments
  • Framework: PEFT 0.19.1 + Transformers 5.7.0 + PyTorch 2.7.1+cu118
  • Data: Custom-curated conversation datasets per persona, 2K-5K examples each

The training process validated that adapters trained on Qwen3-14B base could be hot-swapped at inference time without merging, which was the critical proof-of-concept for this architecture.

7. Fixed Environment Variables

The following environment configuration is common to all GPU paths on VM104 and must be set before any llama.cpp operation:

# Required for llama-server and llama-cli to find CUDA + ggml libraries
export LD_LIBRARY_PATH=/opt/data/home/cuda_libs:/opt/data/home/llama.cpp_gpu

# Optional: pin server to specific GPU (llama.cpp CUDA device enumeration)
# Device 0 = RTX 3090, Device 1 = CMP 100-210, Device 2 = RTX 3060
# Passed via --main-gpu flag, not environment variable

CUDA Library Details

$ ls /opt/data/home/cuda_libs/
libcublas.so.12        libcudart.so.12        libcuda.so.1
libcublasLt.so.12      libcusparse.so.12      ...

$ ls /opt/data/home/llama.cpp_gpu/
llama-server           libggml.so            libggml-cuda.so
llama-cli              libmtmd.so.0          libllama.so

8. Operational Playbook

8.1 Start the Production Server (CMP 100-210)

ssh 192.168.1.243
export LD_LIBRARY_PATH=/opt/data/home/cuda_libs:/opt/data/home/llama.cpp_gpu

/opt/data/home/llama.cpp_gpu/llama-server \
  -m /opt/data/home/models/Qwen3-14B-Claude-4.5-Opus-Distill.q4_k_m.gguf \
  --lora-init-without-apply \
  -ngl 99 \
  -c 36000 \
  --host 0.0.0.0 \
  --port 8080 \
  --split-mode none \
  --main-gpu 1          # CUDA device 1 = CMP 100-210

8.2 Hot-Swap from Any Agent

# Load tool-calling persona
curl http://192.168.1.243:8080/lora-adapters \
  -X POST -H "Content-Type: application/json" \
  -d '[{"path":"/opt/data/home/models/openclaw_tools_v2_lora_qwen3.gguf","scale":1.0}]'

# Unload (return to base)
curl http://192.168.1.243:8080/lora-adapters \
  -X POST -H "Content-Type: application/json" \
  -d '[]'

8.3 Inference

curl http://192.168.1.243:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "<|im_start|>user\nWrite a Python function to parse JSON safely.\n<|im_start|>assistant\n",
    "n_predict": 256,
    "temperature": 0.7
  }'

9. Conclusions

  1. The unmerged Medina 14B architecture achieves sub-20ms LoRA hot-swapping across all tested GPUs, with the base model permanently resident in VRAM.

  2. The CMP 100-210 (16 GB) is the optimal dedicated inference GPU for this workload**, offering 47 tok/s at 36K context with 1.4 GB safety margin. The RTX 3090 remains available for training and heavy batch work.

  3. Context scaling follows a predictable memory model: ~192 KB per token of KV cache (FP16) plus fixed model weight overhead. All three GPUs can reach the model's 40K training ceiling, but only the RTX 3090 has meaningful headroom at that scale.

  4. Layer-split multi-GPU is counterproductive for single-user inference, achieving only 54 tok/s vs. 78 tok/s on the RTX 3090 alone due to PCIe synchronization overhead.

  5. The Medina 14B model is explicitly designed for this workload: Qwen3-14B base + rank-32 LoRA adapters, unmerged architecture, GGUF quantization, and llama.cpp server deployment. This is not a general-purpose model repurposed for agents — it is a specialized inference substrate built to enable dynamic persona adaptation at production scale.

10. Future Work

  • Calibration adapter migration: Retrain calibration_rank32 on Qwen3-14B base to restore the fourth persona.
  • Speculative decoding: Integrate a draft model (e.g., Qwen3-1.8B) for 2-3× speedup on the CMP.
  • Flash Attention 3: Upgrade llama.cpp when FA3 support lands for additional KV cache compression.
  • Multi-slot concurrent serving: Test parallel requests across the 4 default slots with different active adapters per slot.
  • GGUF quantization of adapters: Test Q4_K_M LoRA quantization to reduce adapter size further.

Appendix A: Raw Benchmark Data

A.1 CMP 100-210 @ 36K (Solo)

{
  "vram_used": "14691",
  "vram_free": "1455",
  "base_speed": 47.3,
  "base_ttft": 128.7,
  "tools_swap_ms": 14,
  "tools_speed": 47.5,
  "defensive_swap_ms": 18,
  "defensive_speed": 47.5,
  "offensive_swap_ms": 13,
  "offensive_speed": 47.5,
  "unload_ms": 13
}

A.2 RTX 3090 @ 40K (Solo)

{
  "vram_used": "15071",
  "vram_free": "9064",
  "base_speed": 78.1,
  "base_ttft": 123.0,
  "tools_swap_ms": 14,
  "tools_speed": 78.3,
  "defensive_swap_ms": 13,
  "defensive_speed": 78.3,
  "offensive_swap_ms": 13,
  "offensive_speed": 78.5,
  "unload_ms": 17
}

A.3 RTX 3060 @ 8K (Solo)

{
  "vram_used": "9893",
  "vram_free": "2028",
  "base_speed": 36.5,
  "base_ttft": 141.4,
  "tools_swap_ms": 13,
  "tools_speed": 35.9,
  "defensive_swap_ms": 12,
  "defensive_speed": 36.1,
  "offensive_swap_ms": 13,
  "offensive_speed": 36.0,
  "unload_ms": 14
}

A.4 All 3 GPUs Layer-Split @ 8K

{
  "vram_used": "3391",
  "vram_free": "12755",
  "base_speed": 54.0,
  "base_ttft": 151.2,
  "tools_swap_ms": 12,
  "tools_speed": 54.5,
  "defensive_swap_ms": 13,
  "defensive_speed": 54.6,
  "offensive_swap_ms": 14,
  "offensive_speed": 54.5,
  "unload_ms": 14
}

Paper generated on VM104 (192.168.1.243) by automated benchmark harness. All measurements are reproducible with the fixed environment variables and file paths documented in Section 7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment