ubergarm

Which GGUF is right for me? (Opinionated)

Good question! I am collecting human data on how quantization affects outputs. See here for more information: ggerganov/llama.cpp#5962

In the meantime, use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters.

llama.cpp feature matrix

See the wiki upstream: https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix

In most of deep learning projects, the training scripts always start with lines to load in data, which can easily take a handful minutes. Only after data ready can start testing my buggy code. It is so frustratingly often that I wait for ten minutes just to find I made a stupid typo, then I have to restart and wait for another ten minutes hoping no other typos are made.

In order to make my life easy, I devote lots of effort to reduce the overhead of I/O loading. Here I list some useful tricks I found and hope they also save you some time.

use Numpy Memmap to load array and say goodbye to HDF5.

I used to relay on HDF5 to read/write data, especially when loading only sub-part of all data. Yet that was before I realized how fast and charming Numpy Memmapfile is. In short, Memmapfile does not load in the whole array at open, and only later "lazily" load in the parts that are required for real operations.

Sometimes I may want to copy the full array to memory at once, as it makes later operations

Put slacktive.py somewhere on your system, then change the path in slacktive.plist
Put com.sean.python.slacktive.plist in ~/Library/LaunchAgents/
Add your token to com.sean.python.slacktive.plist (see file comments)
launchctl load ~/Library/LaunchAgents/com.sean.python.slactive.plist

	Log uploaded on Thursday, December 10, 2020, 12:16:49 AM
	Loaded mods:
	Harmony(brrainz.harmony)[mv:1.0.4.0]: 0Harmony(2.0.2), HarmonyMod(1.0.4)
	Core(Ludeon.RimWorld): (no assemblies)
	SRTS Expanded(smashphil.neceros.srtsexpanded)[mv:1.4.6]: 0Harmony(av:2.0.2,fv:1.2.0.1), SRTS(1.0.0)
	Royalty(Ludeon.RimWorld.Royalty): (no assemblies)
	HugsLib(UnlimitedHugs.HugsLib)[ov:8.0.1]: 0Harmony(av:2.0.2,fv:1.2.0.1), HugsLib(av:1.0.0,fv:8.0.1)
	JecsTools (Unofficial)(jecrell.jecstools)[mv:1.1.2.2]: 0JecsTools(1.1.2.2), AbilityUser(1.1.2.2), AbilityUserAI(1.1.2.2), CompActivatableEffect(1.1.2.2), CompAnimated(1.1.2.2), CompBalloon(1.1.2.2), CompBigBox(1.1.2.2), CompDeflector(1.1.2.2), CompDelayedSpawner(1.1.2.2), CompExtraSounds(1.1.2.2), CompInstalledPart(1.1.2.2), CompLumbering(1.1.2.2), CompOverlays(1.1.2.2), CompOversizedWeapon(1.1.2.2), CompSlotLoadable(1.1.2.2), CompToggleDef(1.1.2.2), CompVehicle(1.1.2.1), PawnShields(1.1.2.2), ThinkNodes(1.1.2.2)
	FSharp.Core(latta.fsharp.core)[mv:4.8.2]: FSharp.Core(av:4.7.0,fv:4.700.2

	Log uploaded on Monday, November 16, 2020, 2:10:08 AM
	Loaded mods:
	Harmony(brrainz.harmony)[mv:1.0.4.0]: 0Harmony(2.0.2), HarmonyMod(1.0.4)
	Core(Ludeon.RimWorld): (no assemblies)
	Royalty(Ludeon.RimWorld.Royalty): (no assemblies)
	HugsLib(UnlimitedHugs.HugsLib)[ov:8.0.1]: 0Harmony(av:2.0.2,fv:1.2.0.1), HugsLib(av:1.0.0,fv:8.0.1)
	KanbanStockpile(ubergarm.kanbanstockpile): 0Harmony(av:2.0.2,fv:2.0.4), 0MultiplayerAPI(av:0.2.0,fv:0.1.0), KanbanStockpile(1.0.7623.32571)

	Active Harmony patches:
	DebugWindowsOpener.DevToolStarterOnGUI: TRANS: HugsLib.Patches.DevToolStarterOnGUI_Patch.ExtendButtonsWindow

	"""
	WARNING: dont use loguru, use structlog
	https://gist.github.com/nkhitrov/38adbb314f0d35371eba4ffb8f27078f


	Configure handlers and formats for application loggers.
	"""
	import logging
	import sys
	from pprint import pformat

	# If you don‘t want to build it youself, you can try `docker pull killercai/postgres`.
	FROM healthcheck/postgres:latest

	# China debian mirror
	RUN sed -i s@/deb.debian.org/@/mirrors.aliyun.com/@g /etc/apt/sources.list
	RUN apt-get clean && apt-get update
	RUN apt-get install -y wget git build-essential libpq-dev python-dev postgresql-server-dev-all
	# SCWS (Simple Chinese Word Segmentation library)
	RUN cd /tmp && wget -q -O - http://www.xunsearch.com/scws/down/scws-1.2.1.tar.bz2 \| tar xjf - && cd scws-1.2.1 && ./configure && make install
	# zhpaser (postgres plugin)

	import crafttweaker.data.IData;
	import crafttweaker.item.IItemStack;

	val goldBlock=<minecraft:gold_block>;

	val sideItems = {
	efficiency: <minecraft:potion>.withTag({Potion: "minecraft:strong_swiftness"}),
	unbreaking: <minecraft:potion>.withTag({Potion: "minecraft:strong_strength"})
	} as IItemStack[string];

	#0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:135
	No locals.
	#1 0x00000000007cdd9b in DFInstanceLinux::read_raw (this=0x13797f0, addr=17974274650124877746, bytes=17179869184, buffer=0x7ffff6c593b0) at /build/src/dfinstancelinux.cpp:154
	bytes_read = -1
	__FUNCTION__ = "read_raw"
	#2 0x0000000000556629 in DFInstance::enum_vec<int> (this=0x13797f0, addr=140406800909548) at /build/src/dfinstance.h:110
	bytes_read = 4
	__FUNCTION__ = "enum_vec"
	out = {d = 0x7ffff6c59398}
	start = 17974274650124877746

	// Source: http://unix.stackexchange.com/a/251769/17594
	// Modifications: long long, exit codes, diagnostics

	#include <stdio.h>
	#include <stdlib.h>
	#include <limits.h>
	#include <sys/ptrace.h>
	#include <sys/socket.h>
	#include <arpa/inet.h>