single_machine_benchmark_2hrs.txt

Namespace(arch='resnet50', batch_sched='512,192,128', data='/home/ubuntu/data/imagenet', dist_backend='nccl', dist_url='file:///home/ubuntu/data/file.sync', distributed=True, epochs=35, evaluate=False, fp16=True, init_bn0=True, local_rank=5, logdir='/efs/runs/one_machine_e35_nobnwd.03', loss_scale=1024.0, lr=1.0, lr_linear_scale=True, lr_sched='0.14,0.47,0.78,0.95', momentum=0.9, no_bn_wd=True, pretrained=False, print_freq=10, prof=False, resize_sched='0.4,0.92', resume='', save_dir='/home/ubuntu/data/training/nv/2018-08-01_22-38-one_machine_e35_nobnwd-w8', start_epoch=0, val_ar=True, weight_decay=0.0001, workers=8, world_size=8)
~~epoch	hours	top1Accuracy

Distributed: initializing process group
Distributed: success (5/8)
Loading model
Creating data loaders (this could take 6-12 minutes)
Begin training
Dataset changed. 
Image size: 128 
Batch size: 512 
Train Directory: /home/ubuntu/data/imagenet-sz/160/train
Validation Directory: /home/ubuntu/data/imagenet-sz/160/validation
Changing LR from None to 1.000591645959058
Changing LR from 1.184593539226127 to 1.1851851851851851
~~0	0.0426273775	18.652

 * Prec@1 6.876 Prec@5 18.652
Changing LR from 1.1851851851851851 to 1.1857768311442434
Changing LR from 1.3697787244113122 to 1.3703703703703705
~~1	0.06789368777777778	38.038

 * Prec@1 17.532 Prec@5 38.038
Changing LR from 1.3703703703703705 to 1.3709620163294285
Changing LR from 1.5549639095964976 to 1.5555555555555556
~~2	0.09296256305555556	46.106

 * Prec@1 23.422 Prec@5 46.106
Changing LR from 1.5555555555555556 to 1.5561472015146136
Changing LR from 1.7401490947816827 to 1.740740740740741
~~3	0.11860157055555555	52.318

 * Prec@1 27.874 Prec@5 52.318
Changing LR from 1.740740740740741 to 1.7413323866997987
Changing LR from 1.9253342799668678 to 1.925925925925926
~~4	0.14378222583333333	56.776

 * Prec@1 31.808 Prec@5 56.776
Changing LR from 1.925925925925926 to 1.0
~~5	0.16920713305555554	67.516

 * Prec@1 41.174 Prec@5 67.516
~~6	0.19468587527777778	65.922

 * Prec@1 39.994 Prec@5 65.922
~~7	0.21993714416666668	66.622

 * Prec@1 40.608 Prec@5 66.622
~~8	0.24514031805555556	70.656

 * Prec@1 44.042 Prec@5 70.656
~~9	0.27017809583333335	70.892

 * Prec@1 45.000 Prec@5 70.892
~~10	0.29543351166666665	69.956

 * Prec@1 44.056 Prec@5 69.956
~~11	0.32042904333333333	71.894

 * Prec@1 46.634 Prec@5 71.894
~~12	0.34655562055555555	68.698

 * Prec@1 42.316 Prec@5 68.698
~~13	0.37163148194444445	72.050

 * Prec@1 46.502 Prec@5 72.050
Dataset changed. 
Image size: 224 
Batch size: 192 
Train Directory: /home/ubuntu/data/imagenet/train
Validation Directory: /home/ubuntu/data/imagenet/validation
Changing LR from 1.0 to 0.375
~~14	0.4712265972222222	77.998

 * Prec@1 53.084 Prec@5 77.998
~~15	0.5513257394444445	76.610

 * Prec@1 51.236 Prec@5 76.610
Changing LR from 0.375 to 0.0375
~~16	0.6313707680555556	88.860

 * Prec@1 69.070 Prec@5 88.860
~~17	0.71148009	89.080

 * Prec@1 69.208 Prec@5 89.080
~~18	0.7917240083333333	89.592

 * Prec@1 70.258 Prec@5 89.592
~~19	0.8720450455555556	89.912

 * Prec@1 70.444 Prec@5 89.912
~~20	0.9520147263888888	89.636

 * Prec@1 70.258 Prec@5 89.636
~~21	1.031996066388889	89.904

 * Prec@1 70.436 Prec@5 89.904
~~22	1.1121742494444444	89.906

 * Prec@1 70.308 Prec@5 89.906
~~23	1.1923665575	89.956

 * Prec@1 70.468 Prec@5 89.956
~~24	1.2724976972222224	90.044

 * Prec@1 71.026 Prec@5 90.044
~~25	1.352271226388889	90.116

 * Prec@1 70.996 Prec@5 90.116
~~26	1.432344064722222	89.328

 * Prec@1 69.338 Prec@5 89.328
Changing LR from 0.0375 to 0.00375
~~27	1.5122118486111111	91.336

 * Prec@1 73.270 Prec@5 91.336
~~28	1.5923057180555555	91.426

 * Prec@1 73.440 Prec@5 91.426
~~29	1.672590641111111	91.380

 * Prec@1 73.388 Prec@5 91.380
~~30	1.7527346602777778	91.516

 * Prec@1 73.648 Prec@5 91.516
~~31	1.8328738022222224	91.550

 * Prec@1 73.478 Prec@5 91.550
Dataset changed. 
Image size: 288 
Batch size: 128 
Train Directory: /home/ubuntu/data/imagenet/train
Validation Directory: /home/ubuntu/data/imagenet/validation
Changing LR from 0.00375 to 0.0025
~~32	1.9829085527777779	93.000

 * Prec@1 75.968 Prec@5 93.000
Changing LR from 0.0025 to 0.00025
~~33	2.1030219275	93.154

 * Prec@1 76.080 Prec@5 93.154
~~34	2.2229548641666668	93.084

 * Prec@1 76.126 Prec@5 93.084