w32zhong

Understanding SGLang's Radix Cache, the LeetCode Way

Overview

What is Radix Cache?

When an LLM processes a prompt, it computes a Key and Value vector for every token — the KV cache. If many requests share the same system prompt, recomputing its KV cache from scratch each time is wasteful. Radix Cache stores these computed prefixes in a Radix Tree and reuses them across requests, which is one of the main reasons SGLang achieves high throughput.

Why Read mini-sglang Instead of SGLang Directly?

w32zhong

🤗Huggingface Model Downloader

Note

(2025-01-08) Add feature for 🏷️Tag(Revision) Selection, contributed by @Bamboo-D.
(2024-12-17) Add feature for ⚡Quick Startup and ⏭️Fast Resume, enabling skipping of downloaded files, while removing the git clone dependency to accelerate file list retrieval.

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, This command-line tool leverages curl and aria2c for fast and robust downloading of models and datasets.

Features

• ⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.

w32zhong

window.scrollTo(0, 0)
var bodyRect = document.body.getBoundingClientRect();

var items = Array.prototype.slice.call(
  document.querySelectorAll('*')
).map(function(element) {
  var rect=element.getBoundingClientRect();
  return {
    element: element,
    include: (element.tagName === "BUTTON" || element.tagName === "A" || (element.onclick != null) || window.getComputedStyle(element).cursor == "pointer"),

w32zhong

import os
import torch
import psutil
import datasets
import glob

from transformers import (
    AutoTokenizer, LlamaConfig, LlamaForCausalLM, Trainer, TrainingArguments,
    DataCollatorForLanguageModeling
)

w32zhong

# train_grpo.py
#
# See https://github.com/willccbb/verifiers for ongoing developments
#
import re
import torch
from datasets import load_dataset, Dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer

w32zhong

import random

class TreapNode(object):
    def __init__(self, key, data):
        self.key = key
        self.ran = random.random()
        self.size = 1
        self.cnt = 1
        self.data = data
        self.left = None

w32zhong

'''
Non-parametric computation of entropy and mutual-information

Adapted by G Varoquaux for code created by R Brette, itself
from several papers (see in the code).

This code is maintained at https://github.com/mutualinfo/mutual_info 
Please download the latest code there, to have improvements and 
bug fixes.

w32zhong

A minimal HTTP server in python. It sends a JSON Hello World for GET requests, and echoes back JSON for POST requests.

python server.py 8009
Starting httpd on port 8009...

curl http://localhost:8009
{"received": "ok", "hello": "world"}

w32zhong

view icons: https://fontawesomeicons.com/

{"name":"Font Awesome v5.0.1", "icons":[
    "fab fa-500px",
    "fab fa-accessible-icon",
    "fab fa-accusoft",
    "fas fa-address-book", "far fa-address-book",
    "fas fa-address-card", "far fa-address-card",
    "fas fa-adjust",
    "fab fa-adn",

w32zhong

<script type="text/javascript">
  window.MathJax = {
    jax: ["input/TeX", "output/CommonHTML"],
    extensions: ["tex2jax.js"],
    TeX: {
      equationNumbers: { autoNumber: "AMS" },
      extensions: ["AMSmath.js", "AMSsymbols.js", "noErrors.js", "noUndefined.js"]
    },
    /* The MathJax API provides us with the ability to register
    callbacks on signals broadcast during its startup sequence.