Created
July 20, 2020 19:57
-
-
Save taylanbil/62a1d2eca60ca6b86d906e94e088c347 to your computer and use it in GitHub Desktop.
lining up the raw losses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| taylanbil@dlrm-gpu-8:~/kkissmart-fairseq$ paste <( grep RAWLOS gpu-repro.txt ) <( grep RAWLOS tpulog.txt ) | |
| RAWLOSS @ 100 tensor(1410.1447, device='cuda:0') RAWLOSS @ 100 tensor(1408.6381, device='xla:1') | |
| RAWLOSS @ 200 tensor(1351.2732, device='cuda:0') RAWLOSS @ 200 tensor(1351.1304, device='xla:1') | |
| RAWLOSS @ 300 tensor(2289.2922, device='cuda:0') RAWLOSS @ 300 tensor(2289.3340, device='xla:1') | |
| RAWLOSS @ 400 tensor(1715.7347, device='cuda:0') RAWLOSS @ 400 tensor(1715.6587, device='xla:1') | |
| RAWLOSS @ 500 tensor(2465.9827, device='cuda:0') RAWLOSS @ 500 tensor(2465.9453, device='xla:1') | |
| RAWLOSS @ 600 tensor(2054.0337, device='cuda:0') RAWLOSS @ 600 tensor(2054.0742, device='xla:1') | |
| RAWLOSS @ 700 tensor(1702.4202, device='cuda:0') RAWLOSS @ 700 tensor(1702.3367, device='xla:1') | |
| RAWLOSS @ 800 tensor(1390.9583, device='cuda:0') RAWLOSS @ 800 tensor(1390.9158, device='xla:1') | |
| RAWLOSS @ 900 tensor(1542.2812, device='cuda:0') RAWLOSS @ 900 tensor(1542.2075, device='xla:1') | |
| RAWLOSS @ 1000 tensor(1371.7253, device='cuda:0') RAWLOSS @ 1000 tensor(1371.6144, device='xla:1') | |
| RAWLOSS @ 1100 tensor(1693.6879, device='cuda:0') RAWLOSS @ 1100 tensor(1693.7521, device='xla:1') | |
| RAWLOSS @ 1200 tensor(1228.1172, device='cuda:0') RAWLOSS @ 1200 tensor(1228.0624, device='xla:1') | |
| RAWLOSS @ 1300 tensor(1328.1652, device='cuda:0') RAWLOSS @ 1300 tensor(1328.1467, device='xla:1') | |
| RAWLOSS @ 1400 tensor(1641.3103, device='cuda:0') RAWLOSS @ 1400 tensor(1641.4343, device='xla:1') | |
| RAWLOSS @ 1500 tensor(1422.3258, device='cuda:0') RAWLOSS @ 1500 tensor(1422.3253, device='xla:1') | |
| RAWLOSS @ 1600 tensor(1923.9146, device='cuda:0') RAWLOSS @ 1600 tensor(1923.9608, device='xla:1') | |
| RAWLOSS @ 1700 tensor(1688.3428, device='cuda:0') RAWLOSS @ 1700 tensor(1688.1075, device='xla:1') | |
| RAWLOSS @ 1800 tensor(1570.4878, device='cuda:0') RAWLOSS @ 1800 tensor(1570.7827, device='xla:1') | |
| RAWLOSS @ 1900 tensor(1828.7078, device='cuda:0') RAWLOSS @ 1900 tensor(1828.8079, device='xla:1') | |
| RAWLOSS @ 2000 tensor(1782.3804, device='cuda:0') RAWLOSS @ 2000 tensor(1782.3601, device='xla:1') | |
| RAWLOSS @ 2100 tensor(2344.2183, device='cuda:0') RAWLOSS @ 2100 tensor(2344.3152, device='xla:1') | |
| RAWLOSS @ 2200 tensor(1709.7332, device='cuda:0') RAWLOSS @ 2200 tensor(1709.8677, device='xla:1') | |
| RAWLOSS @ 2300 tensor(1072.0853, device='cuda:0') RAWLOSS @ 2300 tensor(1072.0686, device='xla:1') | |
| RAWLOSS @ 2400 tensor(1800.6949, device='cuda:0') RAWLOSS @ 2400 tensor(1801.0403, device='xla:1') | |
| RAWLOSS @ 2500 tensor(1392.2799, device='cuda:0') RAWLOSS @ 2500 tensor(1392.2241, device='xla:1') | |
| RAWLOSS @ 2600 tensor(1319.8795, device='cuda:0') RAWLOSS @ 2600 tensor(1319.8892, device='xla:1') | |
| RAWLOSS @ 2700 tensor(1346.2390, device='cuda:0') RAWLOSS @ 2700 tensor(1346.2131, device='xla:1') | |
| RAWLOSS @ 2800 tensor(1433.3136, device='cuda:0') RAWLOSS @ 2800 tensor(1433.2075, device='xla:1') | |
| RAWLOSS @ 2900 tensor(1693.1978, device='cuda:0') RAWLOSS @ 2900 tensor(1693.4897, device='xla:1') | |
| RAWLOSS @ 3000 tensor(867.2125, device='cuda:0') RAWLOSS @ 3000 tensor(867.6174, device='xla:1') | |
| RAWLOSS @ 3100 tensor(1706.2174, device='cuda:0') RAWLOSS @ 3100 tensor(1705.9694, device='xla:1') | |
| RAWLOSS @ 3200 tensor(1676.3699, device='cuda:0') RAWLOSS @ 3200 tensor(1676.2924, device='xla:1') | |
| RAWLOSS @ 3300 tensor(1000.9597, device='cuda:0') RAWLOSS @ 3300 tensor(1001.3185, device='xla:1') | |
| RAWLOSS @ 3400 tensor(1666.6620, device='cuda:0') RAWLOSS @ 3400 tensor(1667.1111, device='xla:1') | |
| RAWLOSS @ 3500 tensor(1426.3322, device='cuda:0') RAWLOSS @ 3500 tensor(1426.0515, device='xla:1') | |
| RAWLOSS @ 3600 tensor(1634.4712, device='cuda:0') RAWLOSS @ 3600 tensor(1634.5762, device='xla:1') | |
| RAWLOSS @ 3700 tensor(1551.6877, device='cuda:0') RAWLOSS @ 3700 tensor(1552.2892, device='xla:1') | |
| RAWLOSS @ 3800 tensor(1543.4656, device='cuda:0') RAWLOSS @ 3800 tensor(1543.8973, device='xla:1') | |
| RAWLOSS @ 3900 tensor(1488.4733, device='cuda:0') RAWLOSS @ 3900 tensor(1488.0979, device='xla:1') | |
| RAWLOSS @ 4000 tensor(1679.9448, device='cuda:0') RAWLOSS @ 4000 tensor(1680.7052, device='xla:1') | |
| RAWLOSS @ 4100 tensor(1940.9502, device='cuda:0') RAWLOSS @ 4100 tensor(1941.4369, device='xla:1') | |
| RAWLOSS @ 4200 tensor(1738.5826, device='cuda:0') RAWLOSS @ 4200 tensor(1739.0806, device='xla:1') | |
| RAWLOSS @ 4300 tensor(1848.1174, device='cuda:0') RAWLOSS @ 4300 tensor(1847.4691, device='xla:1') | |
| RAWLOSS @ 4400 tensor(2160.4170, device='cuda:0') RAWLOSS @ 4400 tensor(2160.4172, device='xla:1') | |
| RAWLOSS @ 4500 tensor(1741.8374, device='cuda:0') RAWLOSS @ 4500 tensor(1743.6643, device='xla:1') | |
| RAWLOSS @ 4600 tensor(1013.6898, device='cuda:0') RAWLOSS @ 4600 tensor(1013.4943, device='xla:1') | |
| RAWLOSS @ 4700 tensor(1215.4442, device='cuda:0') RAWLOSS @ 4700 tensor(1215.8892, device='xla:1') | |
| RAWLOSS @ 4800 tensor(1686.5798, device='cuda:0') RAWLOSS @ 4800 tensor(1687.0696, device='xla:1') | |
| RAWLOSS @ 4900 tensor(1270.6083, device='cuda:0') RAWLOSS @ 4900 tensor(1267.5745, device='xla:1') | |
| RAWLOSS @ 5000 tensor(1934.5712, device='cuda:0') RAWLOSS @ 5000 tensor(1933.2743, device='xla:1') | |
| RAWLOSS @ 5100 tensor(1389.7025, device='cuda:0') RAWLOSS @ 5100 tensor(1389.2589, device='xla:1') | |
| RAWLOSS @ 5200 tensor(1521.7289, device='cuda:0') RAWLOSS @ 5200 tensor(1521.3794, device='xla:1') | |
| RAWLOSS @ 5300 tensor(986.0732, device='cuda:0') RAWLOSS @ 5300 tensor(985.2541, device='xla:1') | |
| RAWLOSS @ 5400 tensor(1238.0280, device='cuda:0') RAWLOSS @ 5400 tensor(1239.3900, device='xla:1') | |
| RAWLOSS @ 5500 tensor(1070.5186, device='cuda:0') RAWLOSS @ 5500 tensor(1070.6313, device='xla:1') | |
| RAWLOSS @ 5600 tensor(2137.5652, device='cuda:0') RAWLOSS @ 5600 tensor(2136.5396, device='xla:1') | |
| RAWLOSS @ 5700 tensor(1708.1031, device='cuda:0') RAWLOSS @ 5700 tensor(1710.4930, device='xla:1') | |
| RAWLOSS @ 5800 tensor(1423.8556, device='cuda:0') RAWLOSS @ 5800 tensor(1422.4799, device='xla:1') | |
| RAWLOSS @ 5900 tensor(1369.5192, device='cuda:0') RAWLOSS @ 5900 tensor(1369.4492, device='xla:1') | |
| RAWLOSS @ 6000 tensor(1245.5112, device='cuda:0') RAWLOSS @ 6000 tensor(1244.3560, device='xla:1') | |
| RAWLOSS @ 6100 tensor(2512.7710, device='cuda:0') RAWLOSS @ 6100 tensor(2512.4678, device='xla:1') | |
| RAWLOSS @ 6200 tensor(1891.8496, device='cuda:0') RAWLOSS @ 6200 tensor(1890.8009, device='xla:1') | |
| RAWLOSS @ 6300 tensor(1130.3253, device='cuda:0') RAWLOSS @ 6300 tensor(1129.4983, device='xla:1') | |
| RAWLOSS @ 6400 tensor(961.2222, device='cuda:0') RAWLOSS @ 6400 tensor(961.3102, device='xla:1') | |
| RAWLOSS @ 6500 tensor(1826.8185, device='cuda:0') RAWLOSS @ 6500 tensor(1826.2983, device='xla:1') | |
| RAWLOSS @ 6600 tensor(668.8315, device='cuda:0') RAWLOSS @ 6600 tensor(669.0168, device='xla:1') | |
| RAWLOSS @ 6700 tensor(1368.3977, device='cuda:0') RAWLOSS @ 6700 tensor(1369.3162, device='xla:1') | |
| RAWLOSS @ 6800 tensor(2059.0720, device='cuda:0') RAWLOSS @ 6800 tensor(2058.0190, device='xla:1') | |
| RAWLOSS @ 6900 tensor(1417.4907, device='cuda:0') RAWLOSS @ 6900 tensor(1416.5652, device='xla:1') | |
| RAWLOSS @ 7000 tensor(1296.0875, device='cuda:0') RAWLOSS @ 7000 tensor(1297.1597, device='xla:1') | |
| RAWLOSS @ 7100 tensor(1533.7075, device='cuda:0') RAWLOSS @ 7100 tensor(1532.3556, device='xla:1') | |
| RAWLOSS @ 7200 tensor(1195.6879, device='cuda:0') RAWLOSS @ 7200 tensor(1194.9662, device='xla:1') | |
| RAWLOSS @ 7300 tensor(1629.7560, device='cuda:0') RAWLOSS @ 7300 tensor(1631.6942, device='xla:1') | |
| RAWLOSS @ 7400 tensor(1123.7363, device='cuda:0') RAWLOSS @ 7400 tensor(1123.8306, device='xla:1') | |
| RAWLOSS @ 7500 tensor(1551.4454, device='cuda:0') RAWLOSS @ 7500 tensor(1551.5527, device='xla:1') | |
| RAWLOSS @ 7600 tensor(1700.6863, device='cuda:0') RAWLOSS @ 7600 tensor(1701.3202, device='xla:1') | |
| RAWLOSS @ 7700 tensor(1107.6963, device='cuda:0') RAWLOSS @ 7700 tensor(1108.4896, device='xla:1') | |
| RAWLOSS @ 7800 tensor(1450.6442, device='cuda:0') RAWLOSS @ 7800 tensor(1451.0496, device='xla:1') | |
| RAWLOSS @ 7900 tensor(1845.3767, device='cuda:0') RAWLOSS @ 7900 tensor(1845.2185, device='xla:1') | |
| RAWLOSS @ 8000 tensor(1751.1384, device='cuda:0') RAWLOSS @ 8000 tensor(1752.1614, device='xla:1') | |
| RAWLOSS @ 8100 tensor(1534.0872, device='cuda:0') RAWLOSS @ 8100 tensor(1533.9989, device='xla:1') | |
| RAWLOSS @ 8200 tensor(1527.2074, device='cuda:0') RAWLOSS @ 8200 tensor(1526.4739, device='xla:1') | |
| RAWLOSS @ 8300 tensor(2236.2222, device='cuda:0') RAWLOSS @ 8300 tensor(2234.0649, device='xla:1') | |
| RAWLOSS @ 8400 tensor(1125.7655, device='cuda:0') RAWLOSS @ 8400 tensor(1127.3406, device='xla:1') | |
| RAWLOSS @ 8500 tensor(1681.7218, device='cuda:0') RAWLOSS @ 8500 tensor(1681.1058, device='xla:1') | |
| RAWLOSS @ 8600 tensor(1828.4143, device='cuda:0') RAWLOSS @ 8600 tensor(1827.9142, device='xla:1') | |
| RAWLOSS @ 8700 tensor(1644.8458, device='cuda:0') RAWLOSS @ 8700 tensor(1644.2002, device='xla:1') | |
| RAWLOSS @ 8800 tensor(1691.0027, device='cuda:0') RAWLOSS @ 8800 tensor(1690.3883, device='xla:1') | |
| RAWLOSS @ 8900 tensor(1959.2830, device='cuda:0') RAWLOSS @ 8900 tensor(1959.0852, device='xla:1') | |
| RAWLOSS @ 9000 tensor(1217.1136, device='cuda:0') RAWLOSS @ 9000 tensor(1217.4624, device='xla:1') | |
| RAWLOSS @ 9100 tensor(1144.0026, device='cuda:0') RAWLOSS @ 9100 tensor(1144.5541, device='xla:1') | |
| RAWLOSS @ 9200 tensor(1399.2024, device='cuda:0') RAWLOSS @ 9200 tensor(1397.6129, device='xla:1') | |
| RAWLOSS @ 9300 tensor(1251.1783, device='cuda:0') RAWLOSS @ 9300 tensor(1250.4963, device='xla:1') | |
| RAWLOSS @ 9400 tensor(1406.9491, device='cuda:0') RAWLOSS @ 9400 tensor(1405.1880, device='xla:1') | |
| RAWLOSS @ 9500 tensor(937.4207, device='cuda:0') RAWLOSS @ 9500 tensor(938.4954, device='xla:1') | |
| RAWLOSS @ 9600 tensor(1411.9232, device='cuda:0') RAWLOSS @ 9600 tensor(1413.3824, device='xla:1') | |
| RAWLOSS @ 9700 tensor(1632.3060, device='cuda:0') RAWLOSS @ 9700 tensor(1633.5691, device='xla:1') | |
| RAWLOSS @ 9800 tensor(2037.6841, device='cuda:0') RAWLOSS @ 9800 tensor(2035.8236, device='xla:1') | |
| RAWLOSS @ 9900 tensor(1467.7316, device='cuda:0') RAWLOSS @ 9900 tensor(1467.5835, device='xla:1') | |
| RAWLOSS @ 10000 tensor(1566.2866, device='cuda:0') RAWLOSS @ 10000 tensor(1565.9857, device='xla:1') | |
| RAWLOSS @ 10100 tensor(1275.3107, device='cuda:0') RAWLOSS @ 10100 tensor(1274.9399, device='xla:1') | |
| RAWLOSS @ 10200 tensor(1482.7819, device='cuda:0') RAWLOSS @ 10200 tensor(1482.6823, device='xla:1') | |
| RAWLOSS @ 10300 tensor(1025.0098, device='cuda:0') RAWLOSS @ 10300 tensor(1025.6306, device='xla:1') | |
| RAWLOSS @ 10400 tensor(1641.3466, device='cuda:0') RAWLOSS @ 10400 tensor(1639.2822, device='xla:1') | |
| RAWLOSS @ 10500 tensor(1740.3737, device='cuda:0') RAWLOSS @ 10500 tensor(1739.4558, device='xla:1') | |
| RAWLOSS @ 10600 tensor(1478.7198, device='cuda:0') RAWLOSS @ 10600 tensor(1478.3591, device='xla:1') | |
| RAWLOSS @ 10700 tensor(1440.9719, device='cuda:0') RAWLOSS @ 10700 tensor(1441.7552, device='xla:1') | |
| RAWLOSS @ 10800 tensor(1178.1041, device='cuda:0') RAWLOSS @ 10800 tensor(1178.0557, device='xla:1') | |
| RAWLOSS @ 10900 tensor(1759.4432, device='cuda:0') RAWLOSS @ 10900 tensor(1758.1354, device='xla:1') | |
| RAWLOSS @ 11000 tensor(2279.4111, device='cuda:0') RAWLOSS @ 11000 tensor(2277.9707, device='xla:1') | |
| RAWLOSS @ 11100 tensor(1960.0864, device='cuda:0') RAWLOSS @ 11100 tensor(1958.7487, device='xla:1') | |
| RAWLOSS @ 11200 tensor(1575.8950, device='cuda:0') RAWLOSS @ 11200 tensor(1572.9771, device='xla:1') | |
| RAWLOSS @ 11300 tensor(1673.9479, device='cuda:0') RAWLOSS @ 11300 tensor(1671.0051, device='xla:1') | |
| RAWLOSS @ 11400 tensor(1403.0437, device='cuda:0') RAWLOSS @ 11400 tensor(1405.1251, device='xla:1') | |
| RAWLOSS @ 11500 tensor(2001.8221, device='cuda:0') RAWLOSS @ 11500 tensor(2001.3026, device='xla:1') |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment