pytorch_imagent_e55_lr1d6.log

Command line run:
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=4 --node_rank=0 --master_addr=192.168.47.71 --master_port=6006 train_imagenet_nv.py ~/data/imagenet --save-dir ~/data/training/nv/2018-07-01T18:29:18-cluster_4_region_c_spot-0-lr1d6-e55 --loss-scale 512 --fp16 -b 192 --sz 224 -j 8 --lr 1.6 --epochs 55 --small --dist-url env:// --dist-backend nccl --distributed



~~epoch	hours	top1Accuracy

Distributed: init_process_group success
Loaded model
Defined loss and optimizer
Created data loaders
Begin training
~~0	0.02115771972222222	3.305

 * Prec@1 0.822 Prec@5 3.305
~~1	0.03566609111111111	17.224

 * Prec@1 6.060 Prec@5 17.224
~~2	0.05005269	28.455

 * Prec@1 11.776 Prec@5 28.455
~~3	0.06456034111111111	34.019

 * Prec@1 15.061 Prec@5 34.019
~~4	0.07905200361111112	32.674

 * Prec@1 14.517 Prec@5 32.674
~~5	0.09351822416666668	54.137

 * Prec@1 28.785 Prec@5 54.137
~~6	0.10799496027777777	54.297

 * Prec@1 29.275 Prec@5 54.297
~~7	0.12250558722222221	53.009

 * Prec@1 28.603 Prec@5 53.009
~~8	0.13693416611111112	56.458

 * Prec@1 31.540 Prec@5 56.458
~~9	0.1514491991666667	61.940

 * Prec@1 35.547 Prec@5 61.940
~~10	0.16591053972222225	65.229

 * Prec@1 38.580 Prec@5 65.229
~~11	0.18033650444444443	61.204

 * Prec@1 35.525 Prec@5 61.204
~~12	0.19471902972222224	65.893

 * Prec@1 39.655 Prec@5 65.893
~~13	0.20916098222222224	63.272

 * Prec@1 37.172 Prec@5 63.272
~~14	0.22353786999999997	61.278

 * Prec@1 35.595 Prec@5 61.278
~~15	0.23779359194444444	64.911

 * Prec@1 38.978 Prec@5 64.911
~~16	0.25222053222222224	55.502

 * Prec@1 31.156 Prec@5 55.502
~~17	0.2666208247222222	62.716

 * Prec@1 36.650 Prec@5 62.716
~~18	0.2809242213888889	62.480

 * Prec@1 36.866 Prec@5 62.480
~~19	0.29524531388888886	68.018

 * Prec@1 41.525 Prec@5 68.018
~~20	0.30958530638888887	68.152

 * Prec@1 41.815 Prec@5 68.152
~~21	0.32389034777777775	63.326

 * Prec@1 37.612 Prec@5 63.326
~~22	0.35748271472222226	68.718

 * Prec@1 41.765 Prec@5 68.718
~~23	0.3867777436111111	75.720

 * Prec@1 50.138 Prec@5 75.720
~~24	0.4160877447222222	66.377

 * Prec@1 40.619 Prec@5 66.377
~~25	0.4453867519444445	66.569

 * Prec@1 40.533 Prec@5 66.569
~~26	0.4747570483333333	88.396

 * Prec@1 67.776 Prec@5 88.396
~~27	0.5043125308333333	88.600

 * Prec@1 68.104 Prec@5 88.600
~~28	0.5336038547222222	88.958

 * Prec@1 68.964 Prec@5 88.958
~~29	0.5630766452777778	89.159

 * Prec@1 68.998 Prec@5 89.159
~~30	0.5923633925	89.559

 * Prec@1 70.034 Prec@5 89.559
~~31	0.6216736391666667	89.421

 * Prec@1 69.464 Prec@5 89.421
~~32	0.6510797169444444	89.551

 * Prec@1 69.716 Prec@5 89.551
~~33	0.6803207261111112	89.115

 * Prec@1 68.926 Prec@5 89.115
~~34	0.70979135	89.669

 * Prec@1 69.840 Prec@5 89.669
~~35	0.7390890566666667	89.061

 * Prec@1 69.040 Prec@5 89.061
~~36	0.7682257322222221	89.321

 * Prec@1 69.336 Prec@5 89.321
~~37	0.7975089430555555	88.822

 * Prec@1 68.508 Prec@5 88.822
~~38	0.8267278697222222	88.308

 * Prec@1 67.426 Prec@5 88.308
~~39	0.8559170680555556	89.333

 * Prec@1 69.470 Prec@5 89.333
~~40	0.8852173652777777	89.339

 * Prec@1 69.446 Prec@5 89.339
~~41	0.9146027002777777	88.908

 * Prec@1 68.612 Prec@5 88.908
~~42	0.9437983355555556	87.722

 * Prec@1 66.587 Prec@5 87.722
~~43	0.9731793472222222	91.557

 * Prec@1 73.299 Prec@5 91.557
~~44	1.0026220183333332	91.599

 * Prec@1 73.570 Prec@5 91.599
~~45	1.0318271805555557	91.697

 * Prec@1 73.752 Prec@5 91.697
~~46	1.0611022966666668	91.727

 * Prec@1 73.860 Prec@5 91.727
~~47	1.0902345866666667	91.823

 * Prec@1 73.958 Prec@5 91.823
~~48	1.119547578888889	91.865

 * Prec@1 73.960 Prec@5 91.865
~~49	1.1486920980555555	91.823

 * Prec@1 73.886 Prec@5 91.823
~~50	1.17787483	91.859

 * Prec@1 74.032 Prec@5 91.859
~~51	1.2623385825	92.922

 * Prec@1 75.342 Prec@5 92.922
~~52	1.3351064161111112	93.040

 * Prec@1 75.600 Prec@5 93.040
~~53	1.4084640425000001	93.035

 * Prec@1 75.662 Prec@5 93.035
~~54	1.4821460480555555	93.074

 * Prec@1 75.629 Prec@5 93.074
~~55	1.5555668911111111	93.080

 * Prec@1 75.737 Prec@5 93.080
~~56	1.6290011797222224	93.040

 * Prec@1 75.631 Prec@5 93.040

🎉