Jacob Kahn jacobkahn
jacobkahn
/ gist:786d5c573bd9fb2a80843fe006d21030
Created
July 24, 2019 05:10
NCCL Tests on 16 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| 8 2 float sum 118.3 0.00 0.00 2e-07 117.2 0.00 0.00 1e-07 | |
| 16 4 float sum 116.2 0.00 0.00 0e+00 117.3 0.00 0.00 1e-07 | |
| 32 8 float sum 118.5 0.00 0.00 1e-07 118.3 0.00 0.00 1e-07 | |
| 64 16 float sum 118.7 0.00 0.00 1e-07 117.1 0.00 0.00 6e-08 | |
| 128 32 float sum 118.5 0.00 0.00 6e-08 118.2 0.00 0.00 6e-08 | |
| 256 64 float sum 118.7 0.00 0.00 6e-08 119.1 0.00 0.00 6e-08 |
jacobkahn
/ gist:43076dbce3922677058dc8094eb726b7
Last active
July 24, 2019 06:49
NCCL Tests on 32 node p3dn.24xlarge + EFA - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| 8 2 float sum 2460.5 0.00 0.00 1e-06 2458.0 0.00 0.00 5e-07 | |
| 16 4 float sum 2457.7 0.00 0.00 5e-07 2467.2 0.00 0.00 5e-07 | |
| 32 8 float sum 2466.2 0.00 0.00 5e-07 2466.3 0.00 0.00 5e-07 | |
| 64 16 float sum 2459.3 0.00 0.00 1e-06 2460.7 0.00 0.00 1e-06 | |
| 128 32 float sum 2457.8 0.00 0.00 1e-06 2456.7 0.00 0.00 1e-06 | |
| 256 64 float sum 2453.9 0.00 0.00 1e-06 2457.4 0.00 0.00 1e-06 |
jacobkahn
/ gist:71a0c1c7ae2f7260471d5bb5da829a67
Last active
July 24, 2019 07:02
NCCL Tests on 32 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| 8 2 float sum 1790.6 0.00 0.00 1e-06 1787.9 0.00 0.00 1e-06 | |
| 16 4 float sum 1844.8 0.00 0.00 5e-07 1798.8 0.00 0.00 5e-07 | |
| 32 8 float sum 1789.6 0.00 0.00 1e-06 1795.4 0.00 0.00 1e-06 | |
| 64 16 float sum 1799.5 0.00 0.00 1e-06 1801.5 0.00 0.00 1e-06 | |
| 128 32 float sum 1816.3 0.00 0.00 1e-06 1814.9 0.00 0.00 1e-06 | |
| 256 64 float sum 1826.6 0.00 0.00 1e-06 1883.2 0.00 0.00 1e-06 |
jacobkahn
/ gist:4e43bfccd79bf864970a98c191cd6c9b
Created
July 24, 2019 06:41
NCCL Tests on 1 node p3dn.24xlarge + efa - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| ip-172-31-39-141:50814:50814 [0] NCCL INFO Launch mode Parallel | |
| 8 2 float sum 13.84 0.00 0.00 1e-07 13.75 0.00 0.00 1e-07 | |
| 16 4 float sum 13.82 0.00 0.00 1e-07 13.96 0.00 0.00 1e-07 | |
| 32 8 float sum 13.79 0.00 0.00 6e-08 13.79 0.00 0.00 6e-08 | |
| 64 16 float sum 13.88 0.00 0.01 6e-08 13.73 0.00 0.01 6e-08 | |
| 128 32 float sum 14.04 0.01 0.02 6e-08 13.94 0.01 0.02 6e-08 |
jacobkahn
/ gist:31fc0bedde5d2dd9d084d56b42dc3208
Created
July 24, 2019 06:42
NCCL Tests on 1 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # out-of-place in-place | |
| # size count type redop time algbw busbw error time algbw busbw error | |
| # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) | |
| ip-172-31-39-141:50926:50926 [0] NCCL INFO Launch mode Parallel | |
| 8 2 float sum 13.88 0.00 0.00 1e-07 13.97 0.00 0.00 1e-07 | |
| 16 4 float sum 14.03 0.00 0.00 1e-07 14.04 0.00 0.00 1e-07 | |
| 32 8 float sum 13.93 0.00 0.00 6e-08 13.86 0.00 0.00 6e-08 | |
| 64 16 float sum 14.11 0.00 0.01 6e-08 13.88 0.00 0.01 6e-08 | |
| 128 32 float sum 14.23 0.01 0.02 6e-08 14.11 0.01 0.02 6e-08 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /home/ec2-user/anaconda3/bin/mpirun -x FI_PROVIDER="efa" -x FI_OFI_RXR_RX_COPY_UNEXP=1 -x FI_OFI_RXR_RX_COPY_OOO=1 -x FI_EFA_MR_CACHE_ENABLE=1 -x FI_OFI_RXR_INLINE_MR_ENABLE=1 -x LD_LIBRARY_PATH=/home/ec2-user/aws-ofi-nccl/install/lib/:/home/ec2-user/nccl/build/lib:/usr/local/cuda-10.0/lib64:/opt/amazon/efa/lib64:$LD_LIBRARY_PATH -x NCCL_DEBUG=INFO -x NCCL_TREE_THRESHOLD=0 -H 172.31.39.141:16,172.31.43.16:16 -n 16 -N 8 --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 --bind-to none /home/ec2-user/nccl-tests/build/all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100 |
jacobkahn
/ gist:153a6b7d4f7b892cb568e3e7bc7a4135
Created
August 15, 2019 05:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Remove libiomp to avoid double linkage | |
| mkdir ~/misc && sudo mv /usr/local/lib/libiomp5.so ~/misc | |
| mkdir -p /data/usr | |
| ### Install libsndfile ### | |
| # Dependencies for libsndfile. The entries in Amazon Linux's RHEL | |
| # package repo are too old, so some of these need to be built from source. | |
| # This is likely not an issue on Ubuntu, and packages from apt-get can be used. | |
| # Build and install flac |
jacobkahn
/ ArrayFire.3.6.4.log
Last active
December 3, 2019 22:32
wav2letter Conv + GLU model with ArrayFire 3.6.4 and master (pre 3.7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| epoch: 1 | lr: 0.800000 | lrcriterion: 0.008000 | runtime: 03:03:23 | bch(ms): 1251.95 | smp(ms): 0.64 | fwd(ms): 663.27 | crit-fwd(ms): 30.43 | bwd(ms): 520.30 | optim(ms): 45.35 | loss: 27.04214 | train-LER: 99.35 | train-WER: 99.79 | dev-clean.lst-loss: 19.76846 | dev-clean.lst-LER: 98.35 | dev-clean.lst-WER: 99.76 | dev-other.lst-loss: 18.46809 | dev-other.lst-LER: 98.13 | dev-other.lst-WER: 99.66 | avg-isz: 1228 | avg-tsz: 206 | max-tsz: 430 | hrs: 959.74 | thrpt(sec/sec): 314.00 | |
| epoch: 2 | lr: 0.800000 | lrcriterion: 0.008000 | runtime: 03:04:39 | bch(ms): 1260.59 | smp(ms): 0.62 | fwd(ms): 672.51 | crit-fwd(ms): 31.58 | bwd(ms): 519.22 | optim(ms): 45.44 | loss: 10.13135 | train-LER: 27.68 | train-WER: 58.20 | dev-clean.lst-loss: 2.91011 | dev-clean.lst-LER: 10.81 | dev-clean.lst-WER: 29.63 | dev-other.lst-loss: 5.84437 | dev-other.lst-LER: 23.03 | dev-other.lst-WER: 50.14 | avg-isz: 1228 | avg-tsz: 206 | max-tsz: 430 | hrs: 959.74 | thrpt(sec/sec): 311.85 | |
| epoch: 3 |
jacobkahn
/ gutenberg.librivox.manual.excluded.titles.lst
Last active
January 16, 2020 23:19
List of Libri-Light books found in Gutenberg with fuzzy matching procedure (word-level Levenshtein, score = 0.3, distancesim = 0.75)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 100 the story of a patriot|100 the story of a patriot|0 | |
| 1914 and other poems|1914 and other poems|0 | |
| 32 caliber|32 caliber|0 | |
| 35 sonnets|35 sonnets|0 | |
| 813|813|0 | |
| aaron's rod|aaron's rod|0 | |
| abaft the funnel|abaft the funnel|0 | |
| abandoned|abandoned|0 | |
| abbotsford and newstead abbey|abbotsford and newstead abbey|0 | |
| abe and mawruss being further adventures of potash and perlmutter|abe and mawruss being further adventures of potash and perlmutter|0 |