Costa Huang vwxyzjn

For typical use cases, I prefer using nvitop to view detailed information. This script offers a dependency-free implementation.

The script enhances the functionality of nvidia-smi and provides the following info:

This is useful on multi-user servers and can be used to quickly identify which user is using the GPU and running what kind of program.

Step 1 - Make a Discord Webhook

Find the Discord channel in which you would like to send commits and other updates
In the settings for that channel, find the Webhooks option and create a new webhook. Note: Do NOT give this URL out to the public. Anyone or service can post messages to this channel, without even needing to be in the server. Keep it safe!

	from transformers import AutoModelForCausalLM, AutoTokenizer, StaticCache
	import torch
	from typing import Optional
	device = "cuda"

	# Copied from the gpt-fast repo
	def multinomial_sample_one_no_sync(probs_sort): # Does multinomial sampling without a cuda synchronization
	q = torch.empty_like(probs_sort).exponential_(1)
	return torch.argmax(probs_sort / q, dim=-1, keepdim=True).to(dtype=torch.int)

	# A very useful script for giving credit where it is due: by replacing (most) variable names by the authors who wrote them.
	# Requirements: pip install gitpython
	# Usage: python3 assign-credits.py <input_file> <output_file>
	# Example: python3 assign-credits.py ./src/main.py ./src/main-credited.py

	# NOTE that this is an awful idea with an awful implementation. The "generated" code likely does not work
	# (e.g. typing stuff is skipped, class attribute names are replaced). The "author" is decided by the current HEAD of git repo, and whoever
	# defines the variable first will get the credits (hahaa dunno if even this is right).

	# It is getting late and I am tired of typing so I will let Github Copilot write something for me.

	# ==== Non-communicating processes

	# 4x 1 chip per process:
	os.environ["TPU_CHIPS_PER_PROCESS_BOUNDS"] = "1,1,1"
	os.environ["TPU_PROCESS_BOUNDS"] = "1,1,1"
	# Different per process:
	os.environ["TPU_VISIBLE_DEVICES"] = "0" # "1", "2", "3"

	# 1-liner for bash: TPU_CHIPS_PER_PROCESS_BOUNDS=1,1,1 TPU_PROCESS_BOUNDS=1,1,1 TPU_VISIBLE_DEVICES=0 TPU_MESH_CONTROLLER_ADDRESS=localhost:8476 TPU_MESH_CONTROLLER_PORT=8476

	model.zero_grad() # Reset gradients tensors
	for i, (inputs, labels) in enumerate(training_set):
	predictions = model(inputs) # Forward pass
	loss = loss_function(predictions, labels) # Compute loss function
	loss = loss / accumulation_steps # Normalize our loss (if averaged)
	loss.backward() # Backward pass
	if (i+1) % accumulation_steps == 0: # Wait for several backward steps
	optimizer.step() # Now we can do an optimizer step
	model.zero_grad() # Reset gradients tensors
	if (i+1) % evaluation_steps == 0: # Evaluate the model when we...