RK3588 Local AI Setup Guide (Orange Pi 5 Plus)

gistfile1.md

Original article here: https://www.reddit.com/r/OrangePI/comments/1tlr6s2/comment/onhlvcw/

This document provides detailed instructions for setting up a local AI server with NPU acceleration on an RK3588 system using rk-llama.cpp.

1. System Specifications

Hardware : Orange Pi 5 Plus (RK3588) - 16GB RAM / 256GB eMMC

OS : Armbian 25.11.1 (Ubuntu 24.04 noble)

Kernel : 6.1.115-vendor-rk35xx

NPU Driver : v0.9.8

2. Prerequisites

Install the required build dependencies:

sudo apt-get update
sudo apt-get install -y build-essential cmake git libssl-dev pkg-config jq curl

3. Building rk-llama.cpp

We use the invisiofficial fork which contains the specialized RKNPU2 backend.

# Clone the repository to your home directory
cd ~
git clone https://github.com/invisiofficial/rk-llama.cpp.git
cd rk-llama.cpp


# Create build directory
mkdir build && cd build


# Configure with RKNPU and CURL support
cmake .. -DGGML_RKNPU2=ON -DLLAMA_CURL=ON


# Build the binaries (cli, server, bench)
make -j$(nproc) llama-cli llama-server llama-bench


# Install to system path
sudo cp bin/llama-* /usr/local/bin/

4. Model Selection (Recommended)

The Phi-4-mini-instruct (Q8_0) is the current "Golden Model" for this setup. It provides excellent reasoning and coding capabilities while fitting perfectly in the NPU's memory requirements.

mkdir -p ~/.cache/llama.cpp
cd ~/.cache/llama.cpp
# Download via llama-cli (securely via HTTPS)
llama-cli -hf unsloth/phi-4-mini-instruct-GGUF --hf-file Phi-4-mini-instruct.Q8_0.gguf -p "warmup" -n 1 --no-warmup

5. Systemd Service Configuration

To ensure the AI server runs in the background and starts on boot:

1. Create the environment file ~/.cache/llama.cpp/server.env. Replace <YOUR_HOME> with the output of echo $HOME:

MODEL_PATH=<YOUR_HOME>/.cache/llama.cpp/Phi-4-mini-instruct.Q8_0.gguf
MODEL_ALIAS=Phi-4-mini-instruct.Q8_0
CTX_SIZE=65536
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
BATCH_SIZE=512
UBATCH_SIZE=512

2. Create the service file /etc/systemd/system/llama-server.service:

[Unit]
Description=Llama.cpp Server (RK3588 NPU)
After=network.target


[Service]
Type=simple
LimitNOFILE=100000
PermissionsStartOnly=true
EnvironmentFile=%h/.cache/llama.cpp/server.env
ExecStartPre=/usr/bin/chmod 666 /dev/dri/renderD129
ExecStartPre=/usr/bin/chmod 666 /dev/dma_heap/system
ExecStartPre=/usr/bin/chmod 666 /dev/dma_heap/reserved
ExecStart=/usr/bin/taskset -c 4-7 /usr/local/bin/llama-server \
    -m ${MODEL_PATH} \
    --alias ${MODEL_ALIAS} \
    -t 4 \
    --host 0.0.0.0 \
    --port 11434 \
    --ctx-size ${CTX_SIZE} \
    --cache-type-k ${CACHE_TYPE_K} \
    --cache-type-v ${CACHE_TYPE_V} \
    --parallel 1 \
    --batch-size ${BATCH_SIZE} \
    --ubatch-size ${UBATCH_SIZE} \
    -ngl 100
Restart=always
RestartSec=3


[Install]
WantedBy=multi-user.target

Note: Use %h in the EnvironmentFile path or provide the absolute path.

3. Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server

6. Performance Tuning

Context Size : For 16GB RAM, 65536 works well with Phi-4-mini.

NPU Cores : Use taskset -c 4-7 to pin the process to the high-performance cores.

KV Cache : Keep at f16 for stability with Phi models. For Llama/Qwen models, q8_0 can save memory.

Frequency Lock : To push the NPU to its limit:

echo performance | sudo tee /sys/class/devfreq/fb000000.rknpu/governor
echo performance | sudo tee /sys/class/devfreq/dmc/governor

7. Management Script

Use the provided manage_ai.sh script to monitor load, switch models, and tune settings interactively.