Sam Shleifer sshleifer

Problem:

In WMT datasets, there is wide variation in the length of examples. Some are one sentence. Some are 10 sentences.
The max batch size that can fit on a v100 is roughly (4, 512)
you end up with lots of batches of shape (4, 12) or (4, small_int) which don't fully utilize the GPU.

Dynamic Batch Size: try to organize batches to be 4*512=2048 tokens, one batch might be shaped (4,512) another (32, 64).

Details of Fairseq Solution:

for file in ls */*bleu.json
do
   echo "$file:"
   cat "$file" | sed -n '/^\s*$/!{p;q}' 
   echo  "------"
done

enro test bleu (distil-mbart unless otherwise specified, before post processing).

Moved to https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/4

README.md

based on this table: https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/models/released-models.txt and ISO mappings.

metadata.json

{'hf_name': 'zlw-zlw',
 'source_languages': 'zlw',
 'target_languages': 'zlw',
 'opus_readme_url': 'https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/zlw-zlw/README.md',

(first, wget fairseq_wmt_enro.tgz from s3)

During training, fairseq passes mbart dynamically sized batches (up to 128 tokens), in a dict called sample with the following relevant keys:

target (our labels): no bos, ends with [2, tgt_lang_code]
net_input.src_tokens (our input_ids): ends with [2, 250004]
net_input.prev_output_tokens (our decoder_input_ids): startswith 250020, ends with 2 . This is the "shift_tokens_right" version of target.

Here are the logs from my breakpoint:

	export b="s3://models.huggingface.co/bert"
	stas_to_fb () {
	src=$1
	shift
	aws s3 sync $b/stas/$src $b/facebook/$src $@
	}

	stas_to_allenai () {
	src=$1
	shift

	python finetune.py \
	--task summarization \
	--learning_rate=3e-4 \
	--do_train \
	--do_predict \
	--val_check_interval 0.25 --n_val 1000 \
	--data_dir xsum \
	--max_source_length 512 --max_target_length=56 \
	--freeze_embeds \
	--model_name_or_path google/pegasus-large \

	# three letter code -> (group/language name, {constituents...}
	# if this language is on the target side the constituents can be used as target language codes.
	# if the language is on the source side they are supported natively without special codes.
	{'aav': ('Austro-Asiatic languages',
	{'hoc', 'hoc_Latn', 'kha', 'khm', 'khm_Latn', 'mnw', 'vie', 'vie_Hani'}),
	'afa': ('Afro-Asiatic languages',
	{'acm', 'afb', 'amh', 'apc', 'ara', 'arq', 'ary', 'arz', 'hau_Latn', 'heb', 'kab', 'mlt', 'rif_Latn', 'shy_Latn', 'som', 'thv', 'tir'}),
	'afr': ('Afrikaans', {'afr'}),
	'alv': ('Atlantic-Congo languages',
	{'ewe', 'fuc', 'fuv', 'ibo', 'kin', 'lin', 'lug', 'nya', 'run', 'sag', 'sna', 'swh', 'toi_Latn', 'tso', 'umb', 'wol', 'xho', 'yor', 'zul'}),


	tar -czvf wmt16_en_ru.tgz wmt16_en_ru
	# wmt16_en_ru/
	# wmt16_en_ru/train.source
	# wmt16_en_ru/train.target
	# wmt16_en_ru/test.target
	# wmt16_en_ru/test.source
	# wmt16_en_ru/val.source
	# wmt16_en_ru/val.target