CaptainEmo craftgear

tl;dr;

UPDATE Mon Mar 10 10:51:31 AM EDT 2025 Check out the newer ktransformers guide for how to get it running faster! About 3.5 tok/sec on this same gaming rig. Big thanks to Supreeth Koundinya with analyticsindiamag.com for the article!

You can run the real deal big boi R1 671B locally off a fast NVMe SSD even without enough RAM+VRAM to hold the 212GB dynamically quantized weights. No it is not swap and won't kill your SSD's read/write cycle lifetime. No this is not a distill model. It works fairly well despite quantization (check the unsloth blog for details on how they did that).

The basic idea is that most of the model itself is not loaded into RAM on startup, but mmap'd. Then kv cache will take up some RAM. Most of your system RAM is left available to serve as disk cache for whatever experts/weights are currently most u

LLM Samplers Explained

Everytime a large language model makes predictions, all of the thousands of tokens in the vocabulary are assigned some degree of probability, from almost 0%, to almost 100%. There are different ways you can decide to choose from those predictions. This process is known as "sampling", and there are various strategies you can use which I will cover here.

OpenAI Samplers

Temperature

Temperature is a way to control the overall confidence of the model's scores (the logits). What this means is that, if you use a lower value than 1.0, the relative distance between the tokens will become larger (more deterministic), and if you use a larger value than 1.0, the relative distance between the tokens becomes smaller (less deterministic).
1.0 Temperature is the original distribution that the model was trained to optimize for, since the scores remain the same.
Graph demonstration with voiceover: https://files.catbox.moe/6ht56x.mp4

Kalomaze's Local LLM Glossary

Not super comprehensive (yet), but I think having up to date documentation like this should be quite helpful for those out of the loop. Things change all the time in local AI circles, and it can be dizzying to catch up from an outsider's perspective, especially if you are new to the more technical aspects of language models in general (and not just locally hosted LLMs).

Available Models

Llama

A language model series created by Meta. Llama 1 was originally leaked in February 2023; Llama 2 then officially released later that year with openly available model weights & a permissive license. Kicked off the initial wave of open source developments that have been made when it comes to open source language modeling. The Llama series comes in four distinct sizes: 7b, 13b, 34b (only Code Llama was released for Llama 2 34b), and 70b. As of writing, the hotly anticipated Llama 3 has yet to arrive.

Mistral

Mistral AI is a French company that also distributes open weight

Open ~/.mozilla/firefox/<your_profile>/chrome/userContent.css (create it if does not exist).

Paste the following content in it.

 @namespace url(http://www.w3.org/1999/xhtml);
 @-moz-document regexp("chrome://browser/content/devtools/**/.*"){
     .devtools-monospace {
         font-family: Consolas, monospace !important;
         font-size: 8pt !important;
     }

}

	{# =========================
	Merged Qwen Multimodal Chat Template from
	- https://github.com/allanchan339/vLLM-Qwen3-3.5-3.6-chat-template-fix
	- https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
	Features:
	- developer role supported (from froggeric)
	- <\|think_on\|> / <\|think_off\|> toggles (from froggeric)
	- Historical reasoning HIDDEN by default (from allanchan339)
	- String-form tool arguments parsed as JSON (from allanchan339)
	- Non-ASCII escaped in tools/args JSON (from froggeric)

	#!/usr/bin/env python3
	"""
	Transplant extra tensors (e.g. MTP layers) from one GGUF file into another,
	producing a mixed-quantization GGUF.

	Note: Tested with ik_llama.cpp GGUF Python module.

	Usage:
	python convert.py <target.gguf> <source.gguf> <output.gguf>

	// MIT License
	//
	// Copyright (c) 2019 Simon Lightfoot
	//
	// Permission is hereby granted, free of charge, to any person obtaining a copy
	// of this software and associated documentation files (the "Software"), to deal
	// in the Software without restriction, including without limitation the rights
	// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	// copies of the Software, and to permit persons to whom the Software is
	// furnished to do so, subject to the following conditions:

	// MIT License
	//
	// Copyright (c) 2019 Simon Lightfoot
	//
	// Permission is hereby granted, free of charge, to any person obtaining a copy
	// of this software and associated documentation files (the "Software"), to deal
	// in the Software without restriction, including without limitation the rights
	// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	// copies of the Software, and to permit persons to whom the Software is
	// furnished to do so, subject to the following conditions:

	### Flutter Generated

	# Miscellaneous
	*.class
	*.lock
	*.log
	*.pyc
	*.swp
	.DS_Store
	.atom/

	import Maybe from 'folktale/maybe';

	const Inc = 'Inc';
	const Dec = 'Dec';
	const IncBy = 'IncBy';

	const IncFn = state => state + 1;
	const DecFn = state => state - 1;
	const IncByFn = (state, action) => state + action.incBy;