Skip to content

Instantly share code, notes, and snippets.

@mrdrozdov

mrdrozdov/logs.txt

Last active January 23, 2019 14:38
Show Gist options
  • Save mrdrozdov/8ec616e3208db20f2caf05ba93f5b1c2 to your computer and use it in GitHub Desktop.
Save mrdrozdov/8ec616e3208db20f2caf05ba93f5b1c2 to your computer and use it in GitHub Desktop.
Download ZIP
gpu summary
Raw
logs.txt
less baseline-l_3-lr_002-d_800-dist-1gpu-b4/experiment.log.0
27s for 100 batches of 4 (1x speedup)
2019-01-22 22:22:09,913 [INFO] Epoch/Step/Batch=0/1300/1300 reconstruction_mse_loss=0.382 total_loss=0.382
2019-01-22 22:22:11,924 [INFO] Average-Length=10.29
2019-01-22 22:22:11,924 [INFO] Words-Per-Second=1
2019-01-22 22:22:36,047 [INFO] Epoch/Step/Batch=0/1400/1400 reconstruction_mse_loss=0.380 total_loss=0.380
2019-01-22 22:22:38,032 [INFO] Average-Length=10.410000000000007
2019-01-22 22:22:38,033 [INFO] Words-Per-Second=1
2019-01-22 22:23:02,929 [INFO] Epoch/Step/Batch=0/1500/1500 reconstruction_mse_loss=0.367 total_loss=0.367
2019-01-22 22:23:07,179 [INFO] Average-Length=10.54
2019-01-22 22:23:07,902 [INFO] Words-Per-Second=1
---
less baseline-l_3-lr_002-d_800-dist-4gpu-b16/experiment.log.0
25s for 100 batches of 16 (4x speedup)
2019-01-22 22:21:13,139 [INFO] Epoch/Step/Batch=0/1300/1300 reconstruction_mse_loss=0.347 total_loss=0.347
2019-01-22 22:21:13,148 [INFO] Average-Length=10.150000000000002
2019-01-22 22:21:13,148 [INFO] Words-Per-Second=6
2019-01-22 22:21:38,571 [INFO] Epoch/Step/Batch=0/1400/1400 reconstruction_mse_loss=0.326 total_loss=0.326
2019-01-22 22:21:38,575 [INFO] Average-Length=10.48
2019-01-22 22:21:38,576 [INFO] Words-Per-Second=6
2019-01-22 22:22:03,475 [INFO] Epoch/Step/Batch=0/1500/1500 reconstruction_mse_loss=0.313 total_loss=0.313
2019-01-22 22:22:03,500 [INFO] Average-Length=10.4
2019-01-22 22:22:03,500 [INFO] Words-Per-Second=6
---
less baseline-l_3-lr_002-d_800-dist-8gpu-b32-glove/experiment.log.0
25s for 100 batches of 32 (8x speedup)
2019-01-22 22:21:10,653 [INFO] Epoch/Step/Batch=0/1300/1300 reconstruction_mse_loss=0.182 total_loss=0.182
2019-01-22 22:21:10,657 [INFO] Average-Length=10.880000000000003
2019-01-22 22:21:10,657 [INFO] Words-Per-Second=13
2019-01-22 22:21:35,676 [INFO] Epoch/Step/Batch=0/1400/1400 reconstruction_mse_loss=0.167 total_loss=0.167
2019-01-22 22:21:35,681 [INFO] Average-Length=10.88
2019-01-22 22:21:35,681 [INFO] Words-Per-Second=13
2019-01-22 22:21:59,509 [INFO] Epoch/Step/Batch=0/1500/1500 reconstruction_mse_loss=0.152 total_loss=0.152
2019-01-22 22:21:59,513 [INFO] Average-Length=10.3
2019-01-22 22:21:59,513 [INFO] Words-Per-Second=13
Raw
summary.txt
baseline-l_3-lr_002-d_800-dist-8gpu-b64
(diora) [adrozdov@node105 ~]$ nvidia-smi
Wed Jan 23 09:37:57 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.26 Driver Version: 396.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:1A:00.0 Off | N/A |
| 23% 35C P8 16W / 250W | 6769MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:1B:00.0 Off | N/A |
| 31% 54C P2 75W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... On | 00000000:60:00.0 Off | N/A |
| 32% 55C P2 75W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... On | 00000000:61:00.0 Off | N/A |
| 31% 54C P2 74W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 108... On | 00000000:B1:00.0 Off | N/A |
| 30% 52C P2 74W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 108... On | 00000000:B2:00.0 Off | N/A |
| 31% 54C P2 74W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 108... On | 00000000:DA:00.0 Off | N/A |
| 27% 47C P2 72W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 108... On | 00000000:DB:00.0 Off | N/A |
| 30% 52C P2 73W / 250W | 8549MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 38593 C ...rozdov/miniconda3/envs/diora/bin/python 967MiB |
| 0 38594 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 38595 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 38596 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 38597 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 38598 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 38599 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 1 38593 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 2 38594 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 3 38595 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 4 38596 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 5 38597 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 6 38598 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
| 7 38599 C ...rozdov/miniconda3/envs/diora/bin/python 8539MiB |
+-----------------------------------------------------------------------------+
baseline-l_3-lr_002-d_800-dist-8gpu-b32-glove.sh
(diora) [adrozdov@node155 ~]$ nvidia-smi
Tue Jan 22 22:19:55 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.26 Driver Version: 396.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:1A:00.0 Off | N/A |
| 49% 82C P2 196W / 250W | 4749MiB / 11178MiB | 68% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:1B:00.0 Off | N/A |
| 49% 82C P2 65W / 250W | 5135MiB / 11178MiB | 69% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... On | 00000000:60:00.0 Off | N/A |
| 49% 81C P2 111W / 250W | 5135MiB / 11178MiB | 61% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... On | 00000000:61:00.0 Off | N/A |
| 49% 83C P2 143W / 250W | 5135MiB / 11178MiB | 61% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 108... On | 00000000:B1:00.0 Off | N/A |
| 48% 80C P2 141W / 250W | 5135MiB / 11178MiB | 52% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 108... On | 00000000:B2:00.0 Off | N/A |
| 48% 80C P2 116W / 250W | 5135MiB / 11178MiB | 71% Default |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 108... On | 00000000:DA:00.0 Off | N/A |
| 41% 69C P2 116W / 250W | 5135MiB / 11178MiB | 55% Default |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 108... On | 00000000:DB:00.0 Off | N/A |
| 46% 78C P2 130W / 250W | 5135MiB / 11178MiB | 60% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 107877 C ...rozdov/miniconda3/envs/diora/bin/python 4739MiB |
| 1 107878 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 2 107879 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 3 107880 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 4 107881 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 5 107882 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 6 107883 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
| 7 107884 C ...rozdov/miniconda3/envs/diora/bin/python 5125MiB |
+-----------------------------------------------------------------------------+
baseline-l_3-lr_002-d_800-dist-1gpu-b4
(diora) [adrozdov@node117 ~]$ nvidia-smi
Tue Jan 22 22:24:49 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.26 Driver Version: 396.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:1A:00.0 Off | N/A |
| 25% 37C P8 16W / 250W | 7929MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:1B:00.0 Off | N/A |
| 27% 48C P2 64W / 250W | 8097MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... On | 00000000:60:00.0 Off | N/A |
| 29% 51C P2 77W / 250W | 10669MiB / 11178MiB | 8% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... On | 00000000:61:00.0 Off | N/A |
| 30% 53C P2 59W / 250W | 10669MiB / 11178MiB | 8% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 108... On | 00000000:B1:00.0 Off | N/A |
| 25% 33C P8 16W / 250W | 0MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 108... On | 00000000:B2:00.0 Off | N/A |
| 46% 76C P2 166W / 250W | 5091MiB / 11178MiB | 58% Default |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 108... On | 00000000:DA:00.0 Off | N/A |
| 25% 44C P2 66W / 250W | 10669MiB / 11178MiB | 8% Default |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 108... On | 00000000:DB:00.0 Off | N/A |
| 25% 36C P8 16W / 250W | 0MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 25604 C python 7919MiB |
| 1 26045 C python 8087MiB |
| 2 136143 C python 10659MiB |
| 3 154412 C python 10659MiB |
| 5 162352 C ...rozdov/miniconda3/envs/diora/bin/python 5081MiB |
| 6 160917 C python 10659MiB |
+-----------------------------------------------------------------------------+
baseline-l_3-lr_002-d_800-dist-4gpu-b16
(diora) [adrozdov@node106 ~]$ nvidia-smi
Tue Jan 22 22:27:28 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.26 Driver Version: 396.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:1A:00.0 Off | N/A |
| 44% 73C P2 101W / 250W | 8318MiB / 11178MiB | 48% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:1B:00.0 Off | N/A |
| 43% 71C P2 95W / 250W | 5463MiB / 11178MiB | 31% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... On | 00000000:60:00.0 Off | N/A |
| 29% 51C P2 58W / 250W | 10669MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... On | 00000000:61:00.0 Off | N/A |
| 23% 37C P8 16W / 250W | 10799MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 108... On | 00000000:B1:00.0 Off | N/A |
| 23% 37C P8 16W / 250W | 10853MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 108... On | 00000000:B2:00.0 Off | N/A |
| 30% 52C P2 63W / 250W | 10669MiB / 11178MiB | 14% Default |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 108... On | 00000000:DA:00.0 Off | N/A |
| 37% 62C P2 130W / 250W | 5463MiB / 11178MiB | 37% Default |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 108... On | 00000000:DB:00.0 Off | N/A |
| 42% 69C P2 148W / 250W | 5463MiB / 11178MiB | 31% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 79606 C ...rozdov/miniconda3/envs/diora/bin/python 5413MiB |
| 0 79607 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 79608 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 0 79609 C ...rozdov/miniconda3/envs/diora/bin/python 965MiB |
| 1 79607 C ...rozdov/miniconda3/envs/diora/bin/python 5453MiB |
| 2 69434 C python 10659MiB |
| 3 40638 C ...ared/apps/python/3.6.6-1810/bin/python3 10779MiB |
| 4 40703 C ...ared/apps/python/3.6.6-1810/bin/python3 10779MiB |
| 5 68982 C python 10659MiB |
| 6 79608 C ...rozdov/miniconda3/envs/diora/bin/python 5453MiB |
| 7 79609 C ...rozdov/miniconda3/envs/diora/bin/python 5453MiB |
+-----------------------------------------------------------------------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment