Skip to content

Instantly share code, notes, and snippets.

@johnandersen777

johnandersen777/gemma-4-12B.md

Created June 6, 2026 14:04
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save johnandersen777/874b59ad2494cb7e74f84b2e810ec8a3 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save johnandersen777/874b59ad2494cb7e74f84b2e810ec8a3 to your computer and use it in GitHub Desktop.
Download ZIP
gemma-4-12B-it-qat-q4_0-gguf:Q4_0
Raw
gemma-4-12B.md

gemma-4-12B-it-qat-q4_0-gguf:Q4_0 local on RX 9070 XT

Opencode

{
  "$schema": "https://opencode.ai/config.json",
  "model": "llama.cpp/gemma-4-12b",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:12434/v1"
      },
      "models": {
        "gemma-4-12b": {
          "name": "google/gemma-4-12B-it-qat-q4_0-gguf:Q4_0",
          "limit": {
            "context": 131072,
            "output": 65536
          }
        }
      }
    }
  }
}

Stats and Logs

$ rocm-smi --showmeminfo vram -f -t -p -u | python -u ~/Downloads/parse_gpu_mem.py
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status

GPU VRAM Usage (GB)
===================================
GPU[0]: 10.32967 GB
GPU[1]: 0.01564 GB

docker run -d --name llama-gemma \
  --device /dev/kfd --device /dev/dri \
  --group-add video \
  --shm-size 16G \
  -v docker-model-runner-models:/models \
  -p 12434:12434 \
  ghcr.io/ggml-org/llama.cpp:server-rocm \
  -hf google/gemma-4-12B-it-qat-q4_0-gguf:Q4_0 \
  --host 0.0.0.0 --port 12434 \
  -c 131072 \
  -ngl 999 \
  --device ROCm0 \
  -fa on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --no-mmproj
llama-gemma
9390ac8cba495f0e5ac4b0f7caa525b92ff9bccfbbddfccaf2869864582456b5
(tmp.2204GaznhW) $ docker logs -f llama-gemma 
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
1.47.632.587 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
1.47.632.590 I device_info:
1.47.632.618 I   - ROCm0   : AMD Radeon RX 9070 XT (16304 MiB, 16166 MiB free)
1.47.632.741 I   - ROCm1   : AMD Ryzen 7 7700X 8-Core Processor (15587 MiB, 24902 MiB free)
1.47.632.746 I   - CPU     : AMD Ryzen 7 7700X 8-Core Processor (31174 MiB, 31174 MiB free)
1.47.632.838 I system_info: n_threads = 8 (n_threads_batch = 8) / 16 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
1.47.632.845 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
1.47.632.894 I srv          init: running without SSL
1.47.632.915 I srv          init: using 15 threads for HTTP server
1.47.633.001 I srv         start: binding port with default address family
1.47.634.233 I srv  llama_server: loading model
1.47.634.238 I srv    load_model: loading model '/root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf/snapshots/f6e7774e6148da3b7f201e42ba37cf084c1db35f/gemma-4-12b-it-qat-q4_0.gguf'
1.47.634.263 I common_init_result: fitting params to device memory ...
1.47.634.264 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
1.48.608.100 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
1.48.608.344 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
1.48.632.346 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
1.49.515.726 W llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
1.49.550.781 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
1.49.606.289 I srv    load_model: initializing slots, n_slots = 4
1.49.647.727 W common_speculative_init: no implementations specified for speculative decoding
1.49.647.732 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072
1.49.647.735 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 131072
1.49.647.736 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 131072
1.49.647.736 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 131072
1.49.647.765 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
1.49.647.766 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
1.49.647.767 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
1.49.647.767 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 256
1.49.647.781 I srv          init: idle slots will be saved to prompt cache and cleared upon starting a new task
1.49.652.202 I init: chat template, example_format: '<|turn>system
<|think|>
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
1.49.652.616 I srv          init: init: chat template, thinking = 1
1.49.652.682 I srv  llama_server: model loaded
1.49.652.692 I srv  llama_server: server is listening on http://0.0.0.0:12434
1.49.652.706 I srv  update_slots: all slots are idle
2.37.940.450 I srv  params_from_: Chat format: peg-gemma4
2.37.941.625 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
2.37.941.627 I srv  get_availabl: updating prompt cache
2.37.941.632 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
2.37.941.636 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 131072 tokens, 8589934592 est)
2.37.941.638 I srv  get_availabl: prompt cache update took 0.01 ms
2.37.941.678 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
2.37.986.881 I srv  params_from_: Chat format: peg-gemma4
2.38.016.579 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
2.38.016.581 I srv  get_availabl: updating prompt cache
2.38.016.585 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
2.38.016.587 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 131072 tokens, 8589934592 est)
2.38.016.588 I srv  get_availabl: prompt cache update took 0.01 ms
2.38.016.725 I slot launch_slot_: id  2 | task 3 | processing task, is_child = 0
2.41.040.087 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   6144, progress = 0.66, t =   3.02 s / 2032.18 tokens per second
2.42.225.535 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   8192, progress = 0.88, t =   4.21 s / 1946.40 tokens per second
2.42.225.619 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    546, progress = 0.98, t =   4.28 s / 127.45 tokens per second
2.42.538.768 I slot create_check: id  3 | task 0 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 545, n_tokens = 546, size = 90.652 MiB)
2.42.861.097 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   8804, progress = 0.94, t =   4.84 s / 1817.37 tokens per second
2.42.861.167 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =    552, progress = 0.99, t =   4.92 s / 112.21 tokens per second
2.43.262.985 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   9310, progress = 1.00, t =   5.25 s / 1774.60 tokens per second
2.43.325.163 I slot create_check: id  2 | task 3 | created context checkpoint 1 of 32 (pos_min = 5258, pos_max = 9309, n_tokens = 9310, size = 170.013 MiB)
2.43.364.298 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   9316, progress = 1.00, t =   5.35 s / 1742.10 tokens per second
2.43.428.363 I reasoning-budget: activated, budget=2147483647 tokens
2.45.733.114 I slot print_timing: id  3 | task 0 | n_decoded =    100, tg =  40.48 t/s
2.45.782.021 I reasoning-budget: deactivated (natural end)
2.45.782.040 I slot print_timing: id  2 | task 3 | n_decoded =    100, tg =  41.95 t/s
2.46.150.623 I slot print_timing: id  2 | task 3 | prompt eval time =    5381.35 ms /  9320 tokens (    0.58 ms per token,  1731.91 tokens per second)
2.46.150.625 I slot print_timing: id  2 | task 3 |        eval time =    2752.53 ms /   115 tokens (   23.94 ms per token,    41.78 tokens per second)
2.46.150.625 I slot print_timing: id  2 | task 3 |       total time =    8133.87 ms /  9435 tokens
2.46.150.630 I slot print_timing: id  2 | task 3 |    graphs reused =          0
2.46.151.029 I slot      release: id  2 | task 3 | stop processing: n_tokens = 9434, truncated = 0
2.48.745.251 I slot print_timing: id  3 | task 0 | n_decoded =    244, tg =  44.51 t/s
2.49.218.722 I slot print_timing: id  3 | task 0 | prompt eval time =    5321.26 ms /   556 tokens (    9.57 ms per token,   104.49 tokens per second)
2.49.218.725 I slot print_timing: id  3 | task 0 |        eval time =    5955.77 ms /   267 tokens (   22.31 ms per token,    44.83 tokens per second)
2.49.218.725 I slot print_timing: id  3 | task 0 |       total time =   11277.03 ms /   823 tokens
2.49.218.726 I slot print_timing: id  3 | task 0 |    graphs reused =          0
2.49.218.768 I slot      release: id  3 | task 0 | stop processing: n_tokens = 822, truncated = 0
2.49.218.773 I srv  update_slots: all slots are idle
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment