Skip to content

Instantly share code, notes, and snippets.

View edwardzjl's full-sized avatar
🦝

Junlin Zhou edwardzjl

🦝
34 followers · 108 following
View GitHub Profile
@edwardzjl
edwardzjl / chatml.md
Created December 1, 2023 03:25
openai's chat markup language

Important

This page is not currently maintained and is intended to provide general insight into the ChatML format, not current up-to-date information. (This document is a preview of the underlying format consumed by GPT models. As a developer, you can use our higher-level API and won't need to interact directly with this format today — but expect to have the option in the future!)

Traditionally, GPT models consumed unstructured text. ChatGPT models instead expect a structured format, called Chat Markup Language

@edwardzjl
edwardzjl / count.sh
Last active December 13, 2023 06:28
Count code lines of a git repo
git log --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }'
@edwardzjl
edwardzjl / stat.sh
Last active January 10, 2024 04:32
code stat by user
git log --author="\($USERNAME1\)\|\($USERNAME2\)" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' -
@edwardzjl
edwardzjl / pretrain.md
Last active May 15, 2024 07:45

Continual Pretrain

目标

  • 小样本单机多卡验证环境配置
  • PipelineParallel or TensorParallel 验证该集群所能训练最大模型规模，以及所需的训练时间
  • 验证灾难性遗忘程度
  • 验证新数据学习程度

训练框架和准备工作