ubergarm

tl;dr;

You can run the real deal big boi R1 671B locally off a fast NVMe SSD even without enough RAM+VRAM to hold the 212GB dynamically quantized weights. No it is not swap and won't kill your SSD's read/write cycle lifetime. No this is not a distill model. It works fairly well despite quantization (check the unsloth blog for details on how they did that).

The basic idea is that most of the model itself is not loaded into RAM on startup, but mmap'd. Then kv cache will take up some RAM. Most of your system RAM is left available to serve as disk cache for whatever experts/weights are currently most used. I can see the model slow down and cache dump and refill when the model switches over to counting words for example.

It is faster on my system using the GPU, but not by much. It may be overall faster to dedicate the GPU PCIe lanes to more NVMe storage in the theory. Curious if anyone has such a fast read IOPS array to try?

Notes and example generations below.

Model Reference

Rank	Title	URL	Points	Author	Time Posted	Comments
1	0-click deanonymization attack targeting Signal, Discord, other platforms	gist.github.com	609	hackermondev	4 hours ago	211
2	Invisible Electrostatic Wall at 3M plant	amasci.com	125	Simon_O_Rourke

Text for benchmarking kokoro-tts. Copy paste this into your main.py test app e.g.

TEXT = """
1. Who am I ?
The gross body which is composed of the seven humours (dhatus), I am not; the five cognitive sense organs, viz. the senses of hearing, touch, sight, taste, and smell, which apprehend their respective objects, viz. sound, touch, colour, taste, and odour, I am not; the five cognitive sense- organs, viz. the organs of speech, locomotion, grasping, excretion, and procreation, which have as their respective functions speaking, moving, grasping, excreting, and enjoying, I am not; the five vital airs, prana, etc., which perform respectively the five functions of in-breathing, etc., I am not; even the mind which thinks, I am not; the nescience too, which is endowed only with the residual impressions of objects, and in which there are no objects and no functioning’s, I am not.

2. If I am none of these, then who am I?
After negating all of the above-mentioned as ‘not this’, ‘not this’, that Awareness whi

YT Channels

For inquiring minds. In no particular order.

AI Summary of J.P. Ascher's Ph.D. Dissertation

To test long context LLM understanding of academic materials running locally on <= 24GB VRAM.

tl;dr;

I downloaded a complex ~450 page Ph.D. dissertation PDF, converted it to text, and prompted two LLMs to generate some summaries. Exact versions of llama.cpp and GGUFs used for inference are listed below. All tests performed locally on 3090TI w/ 24GB VRAM. Both models support ~128k context in their respective tokenization formats.

Mistral-Nemo-12B-Instruct-2407
- Tokenizes document into 51617 tokens

Not really full support for explicit system prompt.

Life on the Rim

As the shuttle crash-landed on the rim world planet, Wagner, Chad, and Gramps were thrown from their seats. The impact was intense, and the three baseliner xenotypes struggled to free themselves from the wreckage.

The first thing they noticed was the eerie silence. There were no birds chirping, no rustling of leaves, just an unsettling quiet. As they assessed their situation, they saw that they were in a dense forest, surrounded by towering trees that seemed to stretch up to the sky.

Wagner, ever the pragmatist, quickly took charge. "We need to assess our injuries and find shelter," he said, his voice calm and authoritative. Chad, the policeman, nodded in agreement, while Gramps, the herbalist, began to examine their surroundings, searching for any signs of danger or useful resources.

As they tended to their wounds, they realized that they were not alone on this planet. Strange creatures lurked just out of sight, their eyes glowing in the darkness. The three baseliners knew they had to

Nomad Scavanger 1.4 Mod Error Report

The RimWorld 1.4 Mod Nomad Scavanger causes an error that breaks the Assign Manage Outfits menu until you restart the game.

Steps to Recreate Bug

Use this ModPack https://steamcommunity.com/sharedfiles/filedetails/?id=3174706609
Start a new colony (singleplayer is fine).

	#Example RimLog Header and CSV data file
	#This example still has a dupe issue with LoadSave patch firing twice.
	#You can use `miller` or Python Pandas to parse this and generate AI Story Prompts
	#Example AI Generated Game Story: https://rentry.co/bzvb43mo
	# Definition of each of the columns:
	#timestamp in ticks - easy to convert: https://rimworldwiki.com/wiki/Time
	#type by logging function
	#defName if available or colony/pawn for quests
	#text description from POV of pawn or simple ShortDescription string
	# YY-MM-dd HH:mm:ss

	https://steamcommunity.com/sharedfiles/filedetails/?id=3225433074" # Rimworld Multiplayer 1.5
	https://steamcommunity.com/sharedfiles/filedetails/?id=3222890125" # 1.5 Multiplayer Pack
	https://steamcommunity.com/sharedfiles/filedetails/?id=3219845807" # Hail's Rim Multiplayer 1.5
	https://steamcommunity.com/sharedfiles/filedetails/?id=3233104905" # Multiplayer :3
	https://steamcommunity.com/sharedfiles/filedetails/?id=3232085004" # RimWorld Multiplayer
	https://steamcommunity.com/sharedfiles/filedetails/?id=3233792240" # Multiplayer w/ Rygiic
	https://steamcommunity.com/sharedfiles/filedetails/?id=3235546043" # Multiplayer Collection
	https://steamcommunity.com/sharedfiles/filedetails/?id=3241807816" # Multiplayer DokiDokiSquad
	https://steamcommunity.com/sharedfiles/filedetails/?id=3220302481" # Itchys Multiplayer Pain
	=== Most common mods between above 9 RimWorld 1.5 Zetrith's Multiplayer Modpacks ===

	# http://atlas.obs-hp.fr/hyperleda/fullsql.html
	# Query Description
	# Type of request: Extract data from a table
	# Name of the selected table: meandata
	# Restrict with the SQL clause: (de2000 > 27.3875) and (de2000 < 28.5177) and (al2000 > 12.9336) and (al2000 < 13.0655)
	# Display the fields: objname, al2000, de2000, objtype, logd25, vt, brief, agnclass
	# Order by: logd25 DESC
	# Description of each column
	# objname: ------------- Principal Name (characters)
	# al2000: ------------- RA (double decimal hours)