memory_stats.csv

T5 Lines with top memory consumption:
=> 376: mem 1956:         weights = F.dropout(weights, p=self.dropout, training=self.training)  # (bs, n_heads, qlen, klen)
=> 359: mem 1728:         scores = torch.einsum("bnqd,bnkd->bnqk", q, k)  # (bs, n_heads, qlen, klen)
=> 375: mem 1728:         weights = F.softmax(scores.float(), dim=-1).type_as(scores)  # (bs, n_heads, qlen, klen)
=> 382: mem 600:         context = torch.matmul(weights, v)  # (bs, n_heads, qlen, dim_per_head)
=> 188: mem 396:         layer_output = hidden_states + self.dropout(y)
=> 172: mem 288:         h = F.relu(h)

python run_benchmark.py --models t5-large facebook/bart-large t5-base \
   --save_to_csv --inference_time_csv_file plots/time.csv  --train_memory_csv_file plots/train_memory.csv \
   --inference_memory_csv_file plots/memory.csv --training  --batch_size 1 --sequence_lengths 128 512 1024 --trace_memory_line_by_line   --save_to_csv | tee -a trace.txt