razhangwei

Key Features

• PyTorch implementations of popular LLMs (Llama, Gemma, Mistral, Phi, Qwen)
• Training recipes for full fine-tuning, LoRA, QLoRA, DPO, PPO, QAT, knowledge distillation
• Memory efficiency and performance improvements
• YAML configs for easy recipe configuration
• Support for various dataset formats and prompt templates

Fine-tuning Recipes

razhangwei

Prompt Engineering

Prompt Forge

• Allow you to quickly test system/user prompt along with self-defined examples
• prompt template with placeholder variables.
• Evaluators

Inference

razhangwei

# Create an interactive bar chart with two bars per date
fig = go.Figure()
fig.add_trace(go.Bar(
    x=daily_formula_intake.index,
    y=daily_formula_intake.values,
    name='Formula'
))
fig.add_trace(go.Bar(
    x=daily_expressed_intake.index,
    y=daily_expressed_intake.values,

razhangwei

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class LanguageModel(nn.Module):
  def __init__(self):
      super().__init__()
      # Define transformer layers, embeddings, etc.

razhangwei

# Pseudocode for estimating advantage function in RLHF using PPO

import numpy as np
import torch
import torch.nn.functional as F

def compute_ppo_loss(policy, old_policy, token_sequences, advantages, returns, clip_epsilon=0.2):
  """
  Compute the PPO loss for language model policy update
  

razhangwei

• quite powerful as it simplies the writing the system prompt

• often need an extra step of post processing, which can be done by LLM or traditional programs.

• structured output vs json mode:

  • structured output: 100% gurantee schema; schema doesn't consume tokens

• json: scheme usually goes to system prompts, costing tokens, doesn't guarantee json or scheme correctness.

razhangwei

pipx install llm
llm install llm-gemini 
llm -m gemini-1.5-flash-latest

razhangwei

## sudo visudo -f /etc/sudoers.d/custom_config


# /etc/sudoers.d/custom_config

# Extend the sudo ticket lifetime to 2 hours (7200 seconds)
Defaults timestamp_timeout=7200

# Allow your user to run the script without a password
wei ALL=(ALL) NOPASSWD: /Users/wei/start_nano_transcriber.sh

razhangwei

class FeedbackManager:
    """
    Manage user feedback by printing a message and printing dots after it.

    This class takes care of printing a message and then appending dots to it
    until told to clear. It also handles clearing the feedback by deleting the
    message and dots.

    The printing of dots is done in a separate thread to avoid blocking.
    """

razhangwei

class AudioRecorder:
    def __init__(self):

        self.is_recording = False
        self.recording_queue = queue.Queue()
        self.stop_recording = threading.Event()
        self.recording_thread = None

    def start_recording(self):
        if not self.is_recording: