bd3lm on windows, perplexity

3090.py

\conda\envs\bd3lm\python.exe -u main.py loader.eval_batch_size=16 model=small algo=bd3lm algo.backbone=hf_dit data=openwebtext-split data.insert_valid_special=False model.length=1024 model.attn_backend=sdpa block_size=4 eval.checkpoint_path=kuleshov-group/bd3lm-owt-block_size4 wandb=null mode=ppl_eval strategy=single_device

bd3lm on windows, perplexity

Validation DataLoader 0: 100%|█████████████████████████████████████████████████| 6891/6891 [1:54:19<00:00,  1.00it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Validate metric      ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│     sampling_eps_max      │            1.0            │
│     sampling_eps_min      │   0.0010000000474974513   │
│          val/bpd          │     4.371912002563477     │
│          val/nll          │     3.030378580093384     │
│          val/ppl          │     20.70507049560547     │
│     valid_var_0.0 - 1     │    22.228891372680664     │
└───────────────────────────┴───────────────────────────┘