Skip to content

Instantly share code, notes, and snippets.

View xrsrke's full-sized avatar
🎯
focus

XλRI-U5 xrsrke

🎯
focus
View GitHub Profile
@3outeille
3outeille / pipeline_parallel.py
Last active April 28, 2025 00:54
Self contained example of how pipeline parallel works (AFAB and 1F1B) in 200 LOC
#VERBOSE=0 torchrun --nproc_per_node 3 self_contained_pp_LOC.py
import os, random, numpy as np, torch, torch.nn as nn, torch.distributed as dist, torch.nn.functional as F
from torch.optim import AdamW
from torch.utils.data import DataLoader, DistributedSampler
from datasets import load_dataset
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
STEP, local_rank, world_size, verbose = 0, int(os.environ["LOCAL_RANK"]), int(os.environ["WORLD_SIZE"]), os.environ.get("VERBOSE", "0") == "1"
def set_all_seed(seed):
@Gaelan
Gaelan / README.md
Last active July 19, 2024 03:55
ChatGPT passes the 2022 APCSA free response section

ChatGPT passes the 2022 AP Computer Science A free response section

For fun, I had ChatGPT take the free response section of the 2022 AP Computer Science A exam. (The exam also has a multiple-choice section, but the College Board doesn't publish this.) It scored 32/36.

Methodology

  • For each question, I pasted in the full text of the question and took the response given.
  • I tried each question once and took the response given: no cherry-picking. For readability, I've added indentation in some cases, and included method signatures where they were provided in the question and ChatGPT only provided a body. I've added question numbers; any other comments are ChatGPT's.
  • Many questions have examples containing tables or diagrams; because those don't translate well to plain text, I excluded those tables/diagrams and any text that referenced them.
  • I excluded the initial instructions at the top of th
@wkcn
wkcn / fp8_gemm.py
Created October 17, 2022 09:50
FP8GEMM
import torch
import transformer_engine.pytorch.cpp_extensions as texcpp
from transformer_engine.pytorch.module import get_workspace
import transformer_engine_extensions as tex
scale = 1.0
meta = tex.FP8TensorMeta()
meta.scale = torch.ones(1,dtype=torch.float32, device="cuda") * scale
meta.scale_inv = torch.ones(1, dtype=torch.float32, device="cuda") / scale
@wkcn
wkcn / measure_fp8_speed.py
Created October 16, 2022 15:42
measure FP8 speed
import torch
import transformer_engine.pytorch as te
from transformer_engine.common.recipe import Format, DelayedScaling
import transformer_engine_extensions as tex
import copy
import math
from typing import Callable, Optional
def speedometer(
module: torch.nn.Module,
@sradc
sradc / autodiff.py
Last active November 13, 2024 14:11
Automatic Differentiation in 26 lines of Python
import math
class Var:
def __init__(self, val: float, local_gradients=()):
self.val = val
self.local_gradients = local_gradients
self.grad = 0
def backward(self, path_value: float = 1):
for child_var, local_gradient in self.local_gradients:
@Qfl3x
Qfl3x / week3.md
Last active May 30, 2023 07:31
MLOps-Zoomcamp: Workflow Orchestration - Prefect

NOTE: commands and UI are deprecated

Content:

  • Negative Engineering
  • What is workflow orchestration?
  • Introduction to Prefect 2.0
  • First Prefect flow and Basics

Workflow Orchestration

@kevin-smets
kevin-smets / _README.md
Last active July 19, 2024 18:34
Google Sheets API on Node.js
@DavidKuennen
DavidKuennen / minimal-analytics-snippet.js
Last active March 17, 2025 06:52
Minimal Analytics Snippet
(function (context, trackingId, options) {
const history = context.history;
const doc = document;
const nav = navigator || {};
const storage = localStorage;
const encode = encodeURIComponent;
const pushState = history.pushState;
const typeException = 'exception';
const generateId = () => Math.random().toString(36);
const getId = () => {

A complete list of books, articles, blog posts, videos and neat pages that support Data Fundamentals (H), organised by Unit.

Formatting

If the resource is available online (legally) I have included a link to it. Each entry has symbols following it.

  • ⨕⨕⨕ indicates difficulty/depth, from ⨕ (easy to pick up intro, no background required) through ⨕⨕⨕⨕⨕ (graduate level textbook, maths heavy, expect equations)
  • ⭐ indicates a particularly recommended resource; 🌟 is a very strongly recommended resource and you should look at it.
@andersevenrud
andersevenrud / TODO.md
Last active January 5, 2019 23:37
OS.js v3 TODO list

OS.js v3 has been released

This is the "general" list of tasks for OS.js v3 development. I'm just adding things here as I go or things pop into mind. So if you see anything interesting and you want to pitch in, that'd be much appreciated.

See the following repositories for issues/todos that are not included here: