kyo kyo-takano

Xet-GC semi-automates the cleanup of Hugging Face repositories bloated by Xet.

With the "LFS Files" modal open in your settings, this script maps each row of the table to its corresponding "Possible filename," which is hidden within a tooltip mask. By parsing the DOM to extract this information, the script retains only the most recent version of each unique filename (assuming the first occurrence from top to bottom is the latest) and automatically selects all older duplicates.

Open the Settings of your Hugging Face repository and navigate to Storage Usage > List Large Files.

投資コストに対して得られる計算量は、GPUと演算強度（Arithmetic Intensity）によって大きく異なります。Runpodの価格設定に基づき、`batch_size=1024` (tokens) でLLMを事前学習した場合、0.5Bパラメータ未満のモデルではL4が、それ以上の規模ではH100が圧倒的なコストパフォーマンスを示します。

貧者のクラウドGPU選択ガイド

Compute obtained per dollar varies significantly by GPU and arithmetic intensity. According to Runpod's pricing, when pre-training LLMs with `batch_size=1024` (tokens), the L4 offers superior cost-performance for models under 0.5B parameters, while the H100 dominates for larger scales.

The Poor Man’s Guide to Cloud GPU Selection

Cheatsheet: Migrating a PyTorch script to a single TPU

Mar 10, 2024.

See pytorch.org/xla for up-to-date info and implementation with multiple TPUs

Installing PyTorch/XLA for TPU

# Usually pre-installed on TPU instances
pip install torch_xla[tpu] -f https://storage.googleapis.com/libtpu-releases/index.html

	#!/usr/bin/python

	import json
	import textwrap
	import time

	from openai import OpenAI

	client = OpenAI()

	init_std, lr = 0.02, 2e-5 # example

	""" Redundant """
	indices = df["init_std"] == init_std) & (df["lr"] == lr)
	print(df[(indices])

	""" Concise """
	query = f"{init_std=} & {lr=}".replace("=", "==")
	print(df.query(query)) # "init_std==0.02 & lr==2e-05"

	import gzip

	def gzip_search(query: str, candidate_chunks: list[str], top_k: int=1):
	"""
	文字列ベースで類似したテキストチャンクを推定するアルゴリズム.
	`query`, `chunk`, および`query + " " + chunk`をそれぞれgzipで圧縮し、編集距離のようなものをベースに評価する.

	Parameters:
	query (str): 検索クエリとして使用する文字列.
	top_k (int, optional): 返される類似チャンクの上位k個を指定する (default: 1).