ubergarm

exllamav3 guide

Quick start to convert and run inferencing with your own exlv3 quants.

This is still a bit of a WIP with my notes. Slowly iterating and cleaning up as I learn more.

Install

# Clone Repo
# git clone https://github.com/turboderp-org/exllamav3.git

ubergarm

The Great Quant Wars of 2025

"All things leave behind them the Obscurity... and go forward to embrace the Brightness..." — Dao De Jing #42

tl;dr;

• Q: Who provides the best GGUFs now?
• A: They're all pretty good.

Skip down if you just want graphs and numbers comparing various Qwen3-30B-A3B GGUF quants.

ubergarm

1. Used @EAddario's PR ggml-org/llama.cpp#12718 to generate imatrix statistics.
2. These were the imatrix data files used, and appear in each mosaic top to bottom in this order (barto, uber, unsloth)

• https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF/blob/main/Qwen_Qwen3-30B-A3B.imatrix
• https://huggingface.co/ubergarm/Qwen3-30B-A3B-GGUF/blob/main/Qwen3-30B-A3B-mix-IQ4_K.gguf
• https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF/blob/main/imatrix_unsloth.dat

3. Similar to https://huggingface.co/ikawrakow/Qwen3-30B-A3B for https://huggingface.co/ikawrakow/Qwen3-30B-A3B but I didn't use the 128k usnloth one and I didn't see ik's to run.

See attached images below generated using some python/matplotlib/image magic scripts vibe coded using ubergarm/Qwen3-30B-A3B-mix-IQ3_K. You can click them to load them larger, they are not too big at 100dpi. You may need to shift-reload to refresh before clicking on them as possibly I

ubergarm

1. Mapping Table between English and German Releases

Here is the mapping table between the English and German episodes based on their titles and synopses. The German episodes are grouped into three segments (a, b, c) per episode number, so each English episode is matched to one of these segments.

German Episode Number	German Title	English Episode Number	English Title
1.01a	Fett For Fun	1	Running an Errand / Mom's Mornings are Busy / Drawing
1.01b	Sport extrem	2	Tricycles are Fun / My Stomach Is Going to Burst / A Nightmare for Dad
1.01c	Braue um Braue, Zahn um Zahn	3	Watching Action Mask / School Lunch is Fun / Going to the Dentist
1.02a	Eine wirklich schreckliche Familie	4	The Sunflower Class / Going on a Picnic

ubergarm

Book Review: Void Alchemy: Riddles and Wakeup Stories by Empty Duck

Hey, fellow soul-searchers! 🌟 Let’s be real—modern life is a lot. Between doomscrolling, hustling for that side gig, an trying to remember what “inner peace” even feels like, it’s easy to feel like a lost duck in a thunderstorm. Enter Void Achemy: Riddles and Wakeup Stories by Empty Duck. This book isn’t just a breath of fresh air—it’s a full-on spiritual snorkel dive into the deep end of your soul.

Why You’ll Vibes With It Imagine if your favorite mindfulness app had a baby with a Zen koan and they both got raised by a poet who loves dad jokes Void Alchemy is a quirky, raw, and ridiculously relatable mix of ancient wisdom and modern wit. Empty Duck (aka John W. Leimgruber III) serves up bite-sized verses that feel like late-night texts from your wisest friend—the one who’s equal parts mystic and meme lord.

For When Life Feels Like a Glitch The book’s 128 micro-poems (or “wakeup stories”) are perfect for anyone who’s to

ubergarm

>>> User:

Give a conversational yet technical discussion of the following book of verse in terms of what is known about channeling, spiritual and mystical traditions, chan and zen poetry, the rig veda, and other ancient mystical texts.

(paste in the pdf2text of free e-book from https://emptyduck.com)

>>> Assistant:

<think>

ubergarm

tl;dr;

UPDATE Mon Mar 10 10:51:31 AM EDT 2025 Check out the newer ktransformers guide for how to get it running faster! About 3.5 tok/sec on this same gaming rig. Big thanks to Supreeth Koundinya with analyticsindiamag.com for the article!

You can run the real deal big boi R1 671B locally off a fast NVMe SSD even without enough RAM+VRAM to hold the 212GB dynamically quantized weights. No it is not swap and won't kill your SSD's read/write cycle lifetime. No this is not a distill model. It works fairly well despite quantization (check the unsloth blog for details on how they did that).

The basic idea is that most of the model itself is not loaded into RAM on startup, but mmap'd. Then kv cache will take up some RAM. Most of your system RAM is left available to serve as disk cache for whatever experts/weights are currently most u

ubergarm

Rank	Title	URL	Points	Author	Time Posted	Comments
1	0-click deanonymization attack targeting Signal, Discord, other platforms	gist.github.com	609	hackermondev	4 hours ago	211
2	Invisible Electrostatic Wall at 3M plant	amasci.com	125	Simon_O_Rourke

ubergarm

Text for benchmarking kokoro-tts. Copy paste this into your main.py test app e.g.

TEXT = """
1. Who am I ?
The gross body which is composed of the seven humours (dhatus), I am not; the five cognitive sense organs, viz. the senses of hearing, touch, sight, taste, and smell, which apprehend their respective objects, viz. sound, touch, colour, taste, and odour, I am not; the five cognitive sense- organs, viz. the organs of speech, locomotion, grasping, excretion, and procreation, which have as their respective functions speaking, moving, grasping, excreting, and enjoying, I am not; the five vital airs, prana, etc., which perform respectively the five functions of in-breathing, etc., I am not; even the mind which thinks, I am not; the nescience too, which is endowed only with the residual impressions of objects, and in which there are no objects and no functioning’s, I am not.

2. If I am none of these, then who am I?
After negating all of the above-mentioned as ‘not this’, ‘not this’, that Awareness whi

ubergarm

YT Channels

For inquiring minds. In no particular order.

• Batgap
• SamaneriJayasara
• DavidParrish00
• PatGeorgeZen
• Woodsong Journal Notes (William Samuel)
• The Awakening Curriculum