Skip to content

Instantly share code, notes, and snippets.

@jacobkahn

jacobkahn/gist:786d5c573bd9fb2a80843fe006d21030

Created July 24, 2019 05:10
Show Gist options
  • Save jacobkahn/786d5c573bd9fb2a80843fe006d21030 to your computer and use it in GitHub Desktop.
Save jacobkahn/786d5c573bd9fb2a80843fe006d21030 to your computer and use it in GitHub Desktop.
Download ZIP
NCCL Tests on 16 node p3dn.24xlarge + ethernet - all_reduce_perf -b 8 -e 1G -f 2 -g 1 -c 1 -n 100
Raw
gistfile1.txt
#
# out-of-place in-place
# size count type redop time algbw busbw error time algbw busbw error
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum 118.3 0.00 0.00 2e-07 117.2 0.00 0.00 1e-07
16 4 float sum 116.2 0.00 0.00 0e+00 117.3 0.00 0.00 1e-07
32 8 float sum 118.5 0.00 0.00 1e-07 118.3 0.00 0.00 1e-07
64 16 float sum 118.7 0.00 0.00 1e-07 117.1 0.00 0.00 6e-08
128 32 float sum 118.5 0.00 0.00 6e-08 118.2 0.00 0.00 6e-08
256 64 float sum 118.7 0.00 0.00 6e-08 119.1 0.00 0.00 6e-08
512 128 float sum 118.8 0.00 0.01 6e-08 117.5 0.00 0.01 6e-08
1024 256 float sum 131.0 0.01 0.01 2e-07 129.5 0.01 0.01 2e-07
2048 512 float sum 169.0 0.01 0.02 2e-07 167.5 0.01 0.02 2e-07
4096 1024 float sum 194.3 0.02 0.04 5e-07 192.3 0.02 0.04 5e-07
8192 2048 float sum 214.7 0.04 0.07 5e-07 215.3 0.04 0.07 5e-07
16384 4096 float sum 225.4 0.07 0.14 5e-07 223.6 0.07 0.14 5e-07
32768 8192 float sum 248.7 0.13 0.25 5e-07 248.6 0.13 0.25 5e-07
65536 16384 float sum 323.4 0.20 0.38 5e-07 324.6 0.20 0.38 5e-07
131072 32768 float sum 383.0 0.34 0.64 5e-07 383.4 0.34 0.64 5e-07
262144 65536 float sum 510.1 0.51 0.96 5e-07 510.8 0.51 0.96 5e-07
524288 131072 float sum 828.3 0.63 1.19 5e-07 829.5 0.63 1.19 5e-07
1048576 262144 float sum 1666.6 0.63 1.18 5e-07 1721.4 0.61 1.14 5e-07
2097152 524288 float sum 1664.6 1.26 2.36 5e-07 1664.1 1.26 2.36 5e-07
4194304 1048576 float sum 3337.1 1.26 2.36 5e-07 3335.6 1.26 2.36 5e-07
8388608 2097152 float sum 6674.1 1.26 2.36 5e-07 6673.9 1.26 2.36 5e-07
16777216 4194304 float sum 13349 1.26 2.36 5e-07 13348 1.26 2.36 5e-07
33554432 8388608 float sum 26700 1.26 2.36 5e-07 26701 1.26 2.36 5e-07
67108864 16777216 float sum 53401 1.26 2.36 5e-07 53402 1.26 2.36 5e-07
134217728 33554432 float sum 106804 1.26 2.36 5e-07 106804 1.26 2.36 5e-07
268435456 67108864 float sum 213655 1.26 2.36 5e-07 213610 1.26 2.36 5e-07
536870912 134217728 float sum 427265 1.26 2.36 5e-07 427221 1.26 2.36 5e-07
1073741824 268435456 float sum 854544 1.26 2.36 5e-07 854486 1.26 2.36 5e-07
# Out of bounds values : 0 OK
# Avg bus bandwidth : 1.01605
#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment