Training logs for 8 machines - p3.16xlarge. High learning rate warmup with batchnorm set to 0.

64gpu_93_aspect_ratio.txt

~~epoch	hours	top1Accuracy

Distributed: init_process_group success
Loaded model
Defined loss and optimizer
Created data loaders
Begin training
Changing LR from None to 1.5
~~0	0.015269303611111111	8.536

 * Prec@1 2.754 Prec@5 8.536
Changing LR from 1.7961783439490446 to 1.8
~~1	0.023635661111111112	21.258

 * Prec@1 8.432 Prec@5 21.258
Changing LR from 2.0961783439490445 to 2.1
~~2	0.03173788527777778	37.690

 * Prec@1 17.334 Prec@5 37.690
Changing LR from 2.3961783439490447 to 2.4000000000000004
~~3	0.039703370277777775	33.122

 * Prec@1 14.496 Prec@5 33.122
Changing LR from 2.6961783439490445 to 2.6999999999999997
~~4	0.04787378527777778	50.588

 * Prec@1 26.480 Prec@5 50.588
Changing LR from 2.996178343949045 to 2.0
~~5	0.05568475555555556	59.004

 * Prec@1 33.008 Prec@5 59.004
~~6	0.0639391811111111	53.928

 * Prec@1 29.774 Prec@5 53.928
~~7	0.07236078416666668	51.474

 * Prec@1 27.276 Prec@5 51.474
~~8	0.08073456555555555	60.892

 * Prec@1 35.056 Prec@5 60.892
~~9	0.08893140777777778	50.422

 * Prec@1 26.872 Prec@5 50.422
~~10	0.09690940777777778	65.224

 * Prec@1 38.542 Prec@5 65.224
~~11	0.10507692777777779	52.642

 * Prec@1 29.386 Prec@5 52.642
~~12	0.11343941944444444	67.358

 * Prec@1 41.262 Prec@5 67.358
~~13	0.12170321777777778	61.786

 * Prec@1 35.736 Prec@5 61.786
DataManager changing image size to 244
~~14	0.13926569777777778	71.286

 * Prec@1 45.048 Prec@5 71.286
~~15	0.15423443805555556	72.606

 * Prec@1 45.718 Prec@5 72.606
~~16	0.16906809694444444	72.932

 * Prec@1 46.532 Prec@5 72.932
Changing LR from 2.0 to 0.2
~~17	0.18378840666666668	87.702

 * Prec@1 66.650 Prec@5 87.702
~~18	0.19871511277777779	88.390

 * Prec@1 67.792 Prec@5 88.390
~~19	0.21360281750000001	88.516

 * Prec@1 68.104 Prec@5 88.516
~~20	0.22842486722222222	88.766

 * Prec@1 68.110 Prec@5 88.766
~~21	0.24325293527777778	88.824

 * Prec@1 68.288 Prec@5 88.824
~~22	0.25836048833333336	88.840

 * Prec@1 68.718 Prec@5 88.840
~~23	0.27334701805555556	88.814

 * Prec@1 68.286 Prec@5 88.814
~~24	0.28863768055555555	89.032

 * Prec@1 68.870 Prec@5 89.032
~~25	0.3036785363888889	89.240

 * Prec@1 69.358 Prec@5 89.240
~~26	0.3184671144444445	88.690

 * Prec@1 68.220 Prec@5 88.690
~~27	0.3334224694444444	89.278

 * Prec@1 69.288 Prec@5 89.278
~~28	0.34860874722222224	89.014

 * Prec@1 68.592 Prec@5 89.014
~~29	0.3633554663888889	88.702

 * Prec@1 68.442 Prec@5 88.702
Changing LR from 0.2 to 0.02
~~30	0.37831854333333337	90.902

 * Prec@1 72.466 Prec@5 90.902
~~31	0.39322078027777774	91.084

 * Prec@1 72.672 Prec@5 91.084
~~32	0.4083793536111111	91.112

 * Prec@1 72.900 Prec@5 91.112
~~33	0.42323898694444445	91.008

 * Prec@1 72.914 Prec@5 91.008
~~34	0.4382319338888889	91.164

 * Prec@1 72.864 Prec@5 91.164
DataManager changing image size to 288
~~35	0.47201463000000005	92.836

 * Prec@1 75.322 Prec@5 92.836
~~36	0.4919849927777778	92.966

 * Prec@1 75.368 Prec@5 92.966
~~37	0.512011628888889	93.072

 * Prec@1 75.484 Prec@5 93.072
Changing LR from 0.02 to 0.002
~~38	0.5319756361111111	93.102

 * Prec@1 75.562 Prec@5 93.102
~~39	0.5515782158333333	93.090

 * Prec@1 75.596 Prec@5 93.090