wolfecameron

[LLM Training and Fundamentals]
- GPT and GPT-2: https://cameronrwolfe.substack.com/p/language-models-gpt-and-gpt-2
- GPT-3 and LLM Scaling: https://cameronrwolfe.substack.com/p/language-model-scaling-laws-and-gpt
- Modern LLMs: https://cameronrwolfe.substack.com/p/modern-llms-mt-nlg-chinchilla-gopher
- Specialized LLMs: https://cameronrwolfe.substack.com/p/specialized-llms-chatgpt-lamda-galactica

[Open Source LLMs]
- LLaMA: https://cameronrwolfe.substack.com/p/llama-llms-for-everyone
- Beyond LLaMA (Imitation Models): https://cameronrwolfe.substack.com/p/beyond-llama-the-power-of-open-llms
- False Promise of Imitation: https://cameronrwolfe.substack.com/p/imitation-models-and-the-open-source

wolfecameron

Summaries and Overviews:
- GPT and GPT-2: https://cameronrwolfe.substack.com/p/language-models-gpt-and-gpt-2
- Scaling Laws and GPT-3: https://cameronrwolfe.substack.com/p/language-model-scaling-laws-and-gpt
- OPT-175B (Open-Source GPT-3): https://cameronrwolfe.substack.com/p/understanding-the-open-pre-trained-transformers-opt-library-193a29c14a15
- Modern LLMs: https://cameronrwolfe.substack.com/p/modern-llms-mt-nlg-chinchilla-gopher
- Specialized LLMs: https://cameronrwolfe.substack.com/p/specialized-llms-chatgpt-lamda-galactica
- Why does ChatGPT work?: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
- Orca: https://cameronrwolfe.substack.com/p/orca-properly-imitating-proprietary
- LLaMA: https://cameronrwolfe.substack.com/p/llama-llms-for-everyone
- MPT: https://cameronrwolfe.substack.com/p/democratizing-ai-mosaicmls-impact

wolfecameron

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)

wolfecameron

Learning Rate Scheduling:
- Overview of useful LR schedules: https://cameronrwolfe.substack.com/p/the-best-learning-rate-schedules
- REX Paper: https://arxiv.org/abs/2107.04197

Precision Scheduling:
- Overview of Low Precision Training Techniques: https://cameronrwolfe.substack.com/p/quantized-training-with-deep-networks-82ea7f516dc6
- CPT Paper: https://arxiv.org/abs/2101.09868

Video Batch Size Scheduling:
- Overview of Video Deep Learning (Part One): https://cameronrwolfe.substack.com/p/deep-learning-on-video-part-one-the-early-days-8a3632ed47d4

wolfecameron

import torch

class FFNN(torch.nn.Module):
  def __init__(self, input_size, hidden_size, output_size, num_layers):
    super().__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.output_size = output_size
    self.num_layers = num_layers

wolfecameron

import matplotlib.pyplot as plt

def calc_demon_decay(total_iter, curr_iter, min_val, max_val):
    z = float(total_iter - curr_iter) / total_iter
    return min_val + float(max_val - min_val) * (z / (1 - 0.9 + 0.9*z))
  
train_iters = 100
max_mom = 0.9
min_mom = 0.0
plt.title('Demon Decay')