Sam sammcj

Ollama GPU Selector: Customize GPU Usage for Ollama

If you want to run Ollama on a specific GPU or multiple GPUs, this tutorial is for you. By default, Ollama utilizes all available GPUs, but sometimes you may want to dedicate a specific GPU or a subset of your GPUs for Ollama's use. The idea for this guide originated from the following issue: Run Ollama on dedicated GPU.

Steps:

Create a script let's call it ollama_gpu_selector.sh:
```
nano ollama_gpu_selector.sh
```

Which GGUF is right for me? (Opinionated)

Good question! I am collecting human data on how quantization affects outputs. See here for more information: ggml-org/llama.cpp#5962

In the meantime, use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters.

llama.cpp feature matrix

See the wiki upstream: https://github.com/ggerganov/llama.cpp/wiki/Feature-matrix

Setup Docker on Amazon Linux 2023

The following guide is for setting up Docker with docker-compose v2 on Amazon Linux 2023. The steps are intendend for AL2023 on EC2 but should mostly work for the AL2023 VMs running on other hypervisors.

Install and configure Docker on Amazon Linux 2023

Install Base OS Packages

Install the following packages, which are good to have installed:

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

	#!/bin/bash

	# Define the main export folder
	export_folder=~/Desktop/iMessages_Export
	mkdir -p "$export_folder"

	# Part 1: Generate the CSV file
	echo "Generating CSV file..."

	sqlite3 ~/Library/Messages/chat.db <<EOF

	#!/usr/bin/env ruby

	# https://asmirnov.xyz/vram
	# https://vram.asmirnov.xyz

	require "fileutils"
	require "json"
	require "open-uri"

	# https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator/blob/main/index.html

	from txtai import Embeddings, LLM

	# Data to index
	data = [
	"US tops 5 million confirmed virus cases",
	"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg",
	"Beijing mobilises invasion craft along coast as Taiwan tensions escalate",
	"The National Park Service warns against sacrificing slower friends in a bear attack",
	"Maine man wins $1M from $25 lottery ticket",
	"Make huge profits without work, earn up to $100,000 a day"

	# https://github.com/oobabooga/text-generation-webui/blob/main/README.md
	# https://github.com/oobabooga/text-generation-webui/blob/main/docs/Spell-book.md

	# by default the Dockerfile specifies these versions: 3.5;5.0;6.0;6.1;7.0;7.5;8.0;8.6+PTX
	# https://developer.nvidia.com/cuda-gpus you can find the version for your card here
	# Tesla P100 = sm_60, sm_61, sm_62 and compute_60, compute_61, compute_62
	# TORCH_CUDA_ARCH_LIST=6.0,6.1,6.2,7.0,7.5,8.0,8.6+PTX
	# RTX3090 = sm_86 and compute_86 (PTX)
	#8.6+PTX
	# Tesla P100 = sm_60, sm_61, sm_62 and compute_60, compute_61, compute_62

	#------------------------------------------------------------------------------------#
	# PIN Schematics #
	# #
	# GPIO-00 MCU-BOOT #
	# GPIO-01 Speaker Mute-Status #
	# GPIO-02 I2S MCLK #
	# GPIO-03 Touch-Screen TT21100 Interrupt Pin #
	# GPIO-04 ILI92xxx Display DC-Pin (SPI: CLK-Pin) #
	# GPIO-05 ILI92xxx Display CS-Pin (SPI: MOSI-Pin) #
	# GPIO-06 ILI92xxx Display SDA #

	---
	blueprint:
	name: Conversation agent Agenda Notification
	description:
	Conversation agent generates a notification based on the upcoming calendar
	agenda, location, and weather information.
	domain: automation
	input:
	notify_time:
	name: Notification time