sshleifer

The way I test things quickly with srun:

(1) on devfair:

srun --gres=gpu:8 --partition=devaccel --nodes=1 --cpus-per-task 64 \
    --ntasks-per-node 1 --mem=400G --constraint volta32gb \
    --time="2-00:00:00" --pty /bin/zsh -l

(2) on the resultant shell:

sshleifer

Setup

To use it on the fair cluster gshard branch, you need the following dependencies: (from inside fairseq env, assuming cuda 11.0)

pip install -i https://test.pypi.org/simple/ bitsandbytes-cuda110 -U
pip install -U fairscale

WARNING: if you dont do this step your checkpoints will not be usable!

sshleifer

Results

Params 209,190,912. Fraction Embedding: 19%
Params 265,814,016. Fraction Embedding: 15%
Params 354,418,688. Fraction Embedding: 15%
Params 455,081,984. Fraction Embedding: 12%
Params 1,312,817,152. Fraction Embedding: 8%
Params 1,715,470,336. Fraction Embedding: 6%
Params 2,875,195,392. Fraction Embedding: 5%

sshleifer

Setup

• git clone git@github.com:fairinternal/fairseq-py.git && cd fairseq-py && git checkout stable-emb
• if you don't have the fairseq conda env, follow these instructions
• pip install numpy==1.20. (optional, but some people needed this)
• pip install fairscale (should be > 0.3.7, as of writing)
• on FAIR cluster: pip install -i https://test.pypi.org/simple/ bitsandbytes-cuda110 -U)
• OR on AWS: pip install -i https://test.pypi.org/simple/ bitsandbytes-cuda111 -U)

Common Logic for all commands

Edit this as needed

sshleifer

Constructing a sharded dataset

• cat all your raw text into one huge file in /scratch/
• run your favorite bpe on that file (20mins for 160GB with 20 workers), writing the result to /scratch.

Then we do some filtering of newlines

grep -A1 . /scratch/rc_train_big.bpe | grep -v "^--$" > /scratch/rc.filtered.train.bpe

sshleifer

Plugins:

• Time remaining 1508357010 https://ankiweb.net/shared/info/1508357010
• Customize keyboard shortcuts 24411424 https://ankiweb.net/shared/info/24411424
• I set editor cloze to Ctrl-shift-m (ctrl =cmd on mac) m stands for "mask"

sshleifer

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import time
from tqdm import tqdm
from pathlib import Path
import pandas as pd


models = ['sshleifer/distilbart-cnn-12-3',
          'sshleifer/distilbart-cnn-12-6',
          'sshleifer/distilbart-cnn-6-6',

sshleifer

Avoid:

• [!h] for figures/tables.
• two datasets in one 1plot
• NameError introducing terms that haven't been defined.

sshleifer

from pathlib import Path

import fire
from tqdm import tqdm


DS_TO_KEY = {
    'gigaword': ('document', 'summary'),
    'xsum': ('document', 'summary'),
    'aeslc': ('email_body', 'subject_line'),

sshleifer

from pathlib import Path

import fire
from tqdm import tqdm


DS_TO_KEY = {
    'gigaword': ('document', 'summary'),
    'xsum': ('document', 'summary'),
    'aeslc': ('email_body', 'subject_line'),