llama.cpp.md

AMD APU Optimization Guide for llama.cpp on SteamOS

This guide documents the process to optimize and run large language models on AMD APUs (specifically 7840U with Radeon 780M, Ayaneo Flip w/ 32GB RAM) using llama.cpp with Vulkan acceleration.

Prerequisites

System Preparation

# Disable read-only filesystem (SteamOS specific)
sudo steamos-readonly disable

# Install ROCm and development tools
sudo pacman -S rocm-hip-sdk rocm-device-libs
sudo pacman -S base-devel git cmake vulkan-headers vulkan-icd-loader vulkan-validation-layers

Building llama.cpp with Vulkan Support

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp/

# Build with Vulkan and ROCm support
cmake -B build -DGGML_VULKAN=1 -DGGML_HIPBLAS=on
cmake --build build --config Release

Kernel Configuration for VRAM Allocation

Edit the GRUB configuration to allocate sufficient VRAM:

sudo nano /etc/default/grub

Add the following parameters to GRUB_CMDLINE_LINUX_DEFAULT:

rocm.allowed_devices=3 ttm.pages_limit=6881280

Update GRUB and reboot:

sudo update-grub
sudo reboot

Model Setup

Place your GGUF model files in the models/ directory:

# Example model structure
llama.cpp/
├── models/
│   └── qwen2.5-coder-32b  # Your model file
└── build/
    └── bin/
        ├── llama-cli
        └── llama-server

Basic Usage

Command Line Interface

./build/bin/llama-cli -m "models/qwen2.5-coder-32b" -p "Hi you how are you" -ngl 99

API Server Setup

Create a startup script start_server.sh:

#!/bin/bash
./build/bin/llama-server \\
  -m "models/qwen2.5-coder-32b" \\
  -ngl 99 \\
  --host 0.0.0.0 \\
  --port 8080 \\
  --threads 8 \\
  --batch-size 2048

Make it executable:

chmod +x start_server.sh

API Usage Examples

Basic Chat Completion

curl http://localhost:8080/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "models/qwen2.5-coder-32b",
    "messages": [
      {"role": "user", "content": "Write a Python function to calculate factorial"}
    ],
    "temperature": 0.7,
    "max_tokens": 500
  }'

Streaming Response

curl http://localhost:8080/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "models/qwen2.5-coder-32b",
    "messages": [
      {"role": "user", "content": "Explain quantum computing"}
    ],
    "stream": true,
    "temperature": 0.7
  }'

Performance Optimization

Environment Variables

Add to your shell configuration (~/.bashrc or ~/.zshrc):

export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/radeon_icd.x86_64.json
export AMD_VULKAN_ASYNC_COMPUTE=1
export RADV_PERFTEST=aco,ngg,sam,rt

GPU Performance Tuning

Install PowerDeck and then create a tuning script gpu_tune.sh:

#!/bin/bash
# Set power limits for better memory bandwidth
sudo ryzenadj --stapm-limit=28000 --fast-limit=32000 --slow-limit=28000
sudo ryzenadj --tdc-limit=30000 --edc-limit=48000

# Force performance mode
echo "high" | sudo tee /sys/class/drm/card*/device/power_dpm_force_performance_level

Monitoring and Verification

Check VRAM Allocation

cat /sys/class/drm/card*/device/mem_info_vram_total

Verify Vulkan Support

vulkaninfo | grep -A 10 -B 5 "deviceLocal"

Monitor Performance

# GPU usage
radeontop

# System resources
htop

Troubleshooting

Common Issues

1. "Not enough memory for command submission": Increase ttm.pages_limit
2. "amdgpu version file missing": Use the provided tuning scripts
3. Low performance: Ensure GPU is in performance mode

Reset to Defaults

# Remove custom kernel parameters
sudo nano /etc/default/grub
# Remove: rocm.allowed_devices=3 ttm.pages_limit=6881280
sudo update-grub
sudo reboot

Notes

• The ttm.pages_limit=6881280 allocates approximately 26.25GB for GPU compute
• rocm.allowed_devices=3 enables consumer GPU support (gfx1103)
• Adjust batch size and threads based on your specific hardware capabilities
• For 32B models, expect 3-5 tokens/second generation speed

This configuration has been tested on AMD 7840U with Radeon 780M and provides stable performance for large language model inference.