-
-
Save AdamStelmaszczyk/efc1813b37dd27a87df7c992578f9996 to your computer and use it in GitHub Desktop.
3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ python run_job.py -n 5 -g 60 -c 12 --simulator_procs 10 --use_sync --name breakout --short | |
| args.offline: False | |
| ('bash command: ', 'srun -A luna -N 5 -n 5 -c 12 -t 55:00 distributed_tensorpack_mkl.sh 61216 30065 Breakout-v0 adam 1 "3nodes 12cores" "breakout_1517329773.87" 0.00015 128 60 0 None 256 10 1 uniform normal False . True 1 False /net/archive/groups/plggluna/intel_2/logs/ 1e-08 0.9 0.999 0 False False False False 120 False') | |
| srun: job 9511741 queued and waiting for resources | |
| srun: job 9511741 has been allocated resources | |
| SLURM_JOB_ID 9511741 ; SLURM_JOB_NAME distributed_tensorpack_mkl.sh ; SLURM_JOB_NODELIST p[0111-0112,0115,0574,0576] ; SLURMD_NODENAME p0112 ; SLURM_JOB_NUM_NODES 5 | |
| SLURM_JOB_ID 9511741 ; SLURM_JOB_NAME distributed_tensorpack_mkl.sh ; SLURM_JOB_NODELIST p[0111-0112,0115,0574,0576] ; SLURMD_NODENAME p0111 ; SLURM_JOB_NUM_NODES 5 | |
| SLURM_JOB_ID 9511741 ; SLURM_JOB_NAME distributed_tensorpack_mkl.sh ; SLURM_JOB_NODELIST p[0111-0112,0115,0574,0576] ; SLURMD_NODENAME p0115 ; SLURM_JOB_NUM_NODES 5 | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87’: File exists | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87’: File exists | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/’: File exists | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/’: File exists | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage/’: File exists | |
| mkdir: cannot create directory ‘/net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage/’: File exists | |
| SLURM_JOB_ID 9511741 ; SLURM_JOB_NAME distributed_tensorpack_mkl.sh ; SLURM_JOB_NODELIST p[0111-0112,0115,0574,0576] ; SLURMD_NODENAME p0574 ; SLURM_JOB_NUM_NODES 5 | |
| SLURM_JOB_ID 9511741 ; SLURM_JOB_NAME distributed_tensorpack_mkl.sh ; SLURM_JOB_NODELIST p[0111-0112,0115,0574,0576] ; SLURMD_NODENAME p0576 ; SLURM_JOB_NUM_NODES 5 | |
| plgrid/libs/qt/5.4.1 loaded. | |
| plgrid/libs/qt/5.4.1 loaded. | |
| plgrid/libs/qt/5.4.1 loaded. | |
| plgrid/libs/mkl/11.3.1 loaded. | |
| plgrid/libs/mkl/11.3.1 loaded. | |
| plgrid/libs/mkl/11.3.1 loaded. | |
| plgrid/tools/gcc/4.9.2 loaded. | |
| plgrid/tools/gcc/4.9.2 loaded. | |
| plgrid/tools/intel/15.0.2 loaded. | |
| plgrid/tools/intel/15.0.2 loaded. | |
| plgrid/tools/gcc/4.9.2 loaded. | |
| plgrid/tools/intel/15.0.2 loaded. | |
| plgrid/tools/tcltk/8.5.19-threads loaded. | |
| plgrid/tools/tcltk/8.5.19-threads loaded. | |
| plgrid/tools/python/2.7.13 loaded. | |
| plgrid/tools/python/2.7.13 loaded. | |
| plgrid/tools/tcltk/8.5.19-threads loaded. | |
| plgrid/tools/python/2.7.13 loaded. | |
| plgrid/libs/qt/5.4.1 loaded. | |
| plgrid/libs/mkl/11.3.1 loaded. | |
| plgrid/tools/gcc/4.9.2 loaded. | |
| plgrid/tools/intel/15.0.2 loaded. | |
| plgrid/tools/tcltk/8.5.19-threads loaded. | |
| plgrid/tools/python/2.7.13 loaded. | |
| plgrid/libs/qt/5.4.1 loaded. | |
| plgrid/libs/mkl/11.3.1 loaded. | |
| plgrid/tools/gcc/4.9.2 loaded. | |
| plgrid/tools/intel/15.0.2 loaded. | |
| plgrid/tools/tcltk/8.5.19-threads loaded. | |
| plgrid/tools/python/2.7.13 loaded. | |
| plgrid/libs/mkl/11.3.1 unloaded. | |
| plgrid/libs/mkl/2017.0.0 loaded. | |
| The following have been reloaded with a version change: | |
| 1) plgrid/libs/mkl/11.3.1 => plgrid/libs/mkl/2017.0.0 | |
| plgrid/libs/mkl/11.3.1 unloaded. | |
| plgrid/libs/mkl/2017.0.0 loaded. | |
| The following have been reloaded with a version change: | |
| 1) plgrid/libs/mkl/11.3.1 => plgrid/libs/mkl/2017.0.0 | |
| plgrid/libs/mkl/11.3.1 unloaded. | |
| plgrid/libs/mkl/2017.0.0 loaded. | |
| The following have been reloaded with a version change: | |
| 1) plgrid/libs/mkl/11.3.1 => plgrid/libs/mkl/2017.0.0 | |
| tools/gcc/6.2.0 loaded. | |
| tools/gcc/6.2.0 loaded. | |
| tools/gcc/6.2.0 loaded. | |
| plgrid/libs/mkl/11.3.1 unloaded. | |
| plgrid/libs/mkl/2017.0.0 loaded. | |
| The following have been reloaded with a version change: | |
| 1) plgrid/libs/mkl/11.3.1 => plgrid/libs/mkl/2017.0.0 | |
| plgrid/libs/mkl/11.3.1 unloaded. | |
| plgrid/libs/mkl/2017.0.0 loaded. | |
| The following have been reloaded with a version change: | |
| 1) plgrid/libs/mkl/11.3.1 => plgrid/libs/mkl/2017.0.0 | |
| tools/gcc/6.2.0 loaded. | |
| tools/gcc/6.2.0 loaded. | |
| PROGRAM_ARGS: --mkl 0 --dummy 0 --sync 0 --cpu 1 --artificial_slowdown 0 --queue_size 1 --my_sim_master_queue 1 --train_log_path /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage//atari_trainlog/ --predict_batch_size 16 --dummy_predictor 0 --do_train 1 --simulator_procs 10 --env Breakout-v0 --nr_towers 1 --nr_predict_towers 3 --steps_per_epoch 1000 --fc_neurons 256 --batch_size 128 --learning_rate 0.00015 --port 61216 --tf_port 30065 --optimizer adam --use_sync_opt 1 --num_grad 60 --early_stopping None --ps 1 --fc_init uniform --conv_init normal --replace_with_conv True --fc_splits 1 --debug_charts False --epsilon 1e-08 --beta1 0.9 --beta2 0.999 --save_every 0 --models_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/ --experiment_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87 --adam_debug False --eval_node False --record_node False --schedule_hyper False | |
| OFFLINE: False | |
| PROGRAM_ARGS: --mkl 0 --dummy 0 --sync 0 --cpu 1 --artificial_slowdown 0 --queue_size 1 --my_sim_master_queue 1 --train_log_path /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage//atari_trainlog/ --predict_batch_size 16 --dummy_predictor 0 --do_train 1 --simulator_procs 10 --env Breakout-v0 --nr_towers 1 --nr_predict_towers 3 --steps_per_epoch 1000 --fc_neurons 256 --batch_size 128 --learning_rate 0.00015 --port 61216 --tf_port 30065 --optimizer adam --use_sync_opt 1 --num_grad 60 --early_stopping None --ps 1 --fc_init uniform --conv_init normal --replace_with_conv True --fc_splits 1 --debug_charts False --epsilon 1e-08 --beta1 0.9 --beta2 0.999 --save_every 0 --models_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/ --experiment_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87 --adam_debug False --eval_node False --record_node False --schedule_hyper False | |
| OFFLINE: False | |
| PROGRAM_ARGS: --mkl 0 --dummy 0 --sync 0 --cpu 1 --artificial_slowdown 0 --queue_size 1 --my_sim_master_queue 1 --train_log_path /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage//atari_trainlog/ --predict_batch_size 16 --dummy_predictor 0 --do_train 1 --simulator_procs 10 --env Breakout-v0 --nr_towers 1 --nr_predict_towers 3 --steps_per_epoch 1000 --fc_neurons 256 --batch_size 128 --learning_rate 0.00015 --port 61216 --tf_port 30065 --optimizer adam --use_sync_opt 1 --num_grad 60 --early_stopping None --ps 1 --fc_init uniform --conv_init normal --replace_with_conv True --fc_splits 1 --debug_charts False --epsilon 1e-08 --beta1 0.9 --beta2 0.999 --save_every 0 --models_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/ --experiment_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87 --adam_debug False --eval_node False --record_node False --schedule_hyper False | |
| OFFLINE: False | |
| 2018-01-30 17:29:48.435736: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.435772: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.435780: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.435788: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.435795: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.445459: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:30065} | |
| 2018-01-30 17:29:48.445499: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> p0112:30066, 1 -> p0115:30066, 2 -> p0574:30066, 3 -> p0576:30066} | |
| 2018-01-30 17:29:48.446966: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:30065 | |
| {'ps': ['p0111:30065'], 'worker': ['p0112:30066', 'p0115:30066', 'p0574:30066', 'p0576:30066']} | |
| [ps:0] Starting the TF server | |
| [0130 17:29:48 @train.py:84] [ps:0] joining the server. | |
| 2018-01-30 17:29:48.492019: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.492076: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.492084: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.492091: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.492097: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.502455: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> p0111:30065} | |
| 2018-01-30 17:29:48.502505: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:30066, 1 -> p0115:30066, 2 -> p0574:30066, 3 -> p0576:30066} | |
| 2018-01-30 17:29:48.504127: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:30066 | |
| [2018-01-30 17:29:48,505] Making new env: Breakout-v0 | |
| {'ps': ['p0111:30065'], 'worker': ['p0112:30066', 'p0115:30066', 'p0574:30066', 'p0576:30066']} | |
| [worker:0] Starting the TF server | |
| args.mkl == 0 | |
| using tensorflow convolution | |
| [2018-01-30 17:29:48,609] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,614] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,620] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,625] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,631] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,637] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,643] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,649] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,655] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:48,661] Making new env: Breakout-v0 | |
| None <type 'NoneType'> | |
| worker host: grpc://localhost:30066 | |
| [0130 17:29:48 @train.py:711] [BA3C] Train on gpu 0 and infer on gpu 0,0,0 | |
| [0130 17:29:48 @train.py:717] using async version | |
| DUMMY PREDICTOR 0 | |
| MultiGPUTrainer __init__ dummy = 0 | |
| [0130 17:29:48 @multigpu.py:57] Training a model of 1 tower | |
| [0130 17:29:48 @multigpu.py:67] Building graph for training tower 0..., /cpu:0 | |
| 12 | |
| [0130 17:29:48 @_common.py:61] conv0 input: [None, 84, 84, 16] | |
| Tensor("tower0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] conv0 output: [None, 80, 80, 32] | |
| [0130 17:29:48 @_common.py:61] pool0 input: [None, 80, 80, 32] | |
| Tensor("tower0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] pool0 output: [None, 40, 40, 32] | |
| [0130 17:29:48 @_common.py:61] conv1 input: [None, 40, 40, 32] | |
| Tensor("tower0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] conv1 output: [None, 36, 36, 32] | |
| [0130 17:29:48 @_common.py:61] pool1 input: [None, 36, 36, 32] | |
| Tensor("tower0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] pool1 output: [None, 18, 18, 32] | |
| [0130 17:29:48 @_common.py:61] conv2 input: [None, 18, 18, 32] | |
| Tensor("tower0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] conv2 output: [None, 14, 14, 64] | |
| [0130 17:29:48 @_common.py:61] pool2 input: [None, 14, 14, 64] | |
| Tensor("tower0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] pool2 output: [None, 7, 7, 64] | |
| [0130 17:29:48 @_common.py:61] conv3 input: [None, 7, 7, 64] | |
| Tensor("tower0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] conv3 output: [None, 5, 5, 64] | |
| [0130 17:29:48 @_common.py:61] fc1_0 input: [None, 5, 5, 64] | |
| Tensor("tower0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] fc1_0 output: [None, 1, 1, 256] | |
| [0130 17:29:48 @_common.py:61] fc-pi input: [None, 256] | |
| Tensor("tower0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:48 @_common.py:70] fc-pi output: [None, 6] | |
| [0130 17:29:48 @_common.py:61] fc-v input: [None, 256] | |
| PROGRAM_ARGS: --mkl 0 --dummy 0 --sync 0 --cpu 1 --artificial_slowdown 0 --queue_size 1 --my_sim_master_queue 1 --train_log_path /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage//atari_trainlog/ --predict_batch_size 16 --dummy_predictor 0 --do_train 1 --simulator_procs 10 --env Breakout-v0 --nr_towers 1 --nr_predict_towers 3 --steps_per_epoch 1000 --fc_neurons 256 --batch_size 128 --learning_rate 0.00015 --port 61216 --tf_port 30065 --optimizer adam --use_sync_opt 1 --num_grad 60 --early_stopping None --ps 1 --fc_init uniform --conv_init normal --replace_with_conv True --fc_splits 1 --debug_charts False --epsilon 1e-08 --beta1 0.9 --beta2 0.999 --save_every 0 --models_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/ --experiment_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87 --adam_debug False --eval_node False --record_node False --schedule_hyper False | |
| OFFLINE: False | |
| PROGRAM_ARGS: --mkl 0 --dummy 0 --sync 0 --cpu 1 --artificial_slowdown 0 --queue_size 1 --my_sim_master_queue 1 --train_log_path /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/storage//atari_trainlog/ --predict_batch_size 16 --dummy_predictor 0 --do_train 1 --simulator_procs 10 --env Breakout-v0 --nr_towers 1 --nr_predict_towers 3 --steps_per_epoch 1000 --fc_neurons 256 --batch_size 128 --learning_rate 0.00015 --port 61216 --tf_port 30065 --optimizer adam --use_sync_opt 1 --num_grad 60 --early_stopping None --ps 1 --fc_init uniform --conv_init normal --replace_with_conv True --fc_splits 1 --debug_charts False --epsilon 1e-08 --beta1 0.9 --beta2 0.999 --save_every 0 --models_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87/models/ --experiment_dir /net/archive/groups/plggluna/adam/experiments/breakout_1517329773.87 --adam_debug False --eval_node False --record_node False --schedule_hyper False | |
| OFFLINE: False | |
| 2018-01-30 17:29:48.970453: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.970511: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.970531: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.970539: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.970546: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:48.981347: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> p0111:30065} | |
| 2018-01-30 17:29:48.981385: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> p0112:30066, 1 -> localhost:30066, 2 -> p0574:30066, 3 -> p0576:30066} | |
| 2018-01-30 17:29:48.983013: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:30066 | |
| [2018-01-30 17:29:48,984] Making new env: Breakout-v0 | |
| {'ps': ['p0111:30065'], 'worker': ['p0112:30066', 'p0115:30066', 'p0574:30066', 'p0576:30066']} | |
| [worker:1] Starting the TF server | |
| args.mkl == 0 | |
| Tensor("tower0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] fc-v output: [None, 1] | |
| using tensorflow convolution | |
| [2018-01-30 17:29:49,097] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,103] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,109] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,114] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,120] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,126] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,132] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,138] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,145] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:49,151] Making new env: Breakout-v0 | |
| None <type 'NoneType'> | |
| worker host: grpc://localhost:30066 | |
| [0130 17:29:49 @train.py:711] [BA3C] Train on gpu 0 and infer on gpu 0,0,0 | |
| [0130 17:29:49 @train.py:717] using async version | |
| DUMMY PREDICTOR 0 | |
| MultiGPUTrainer __init__ dummy = 0 | |
| [0130 17:29:49 @multigpu.py:57] Training a model of 1 tower | |
| [0130 17:29:49 @multigpu.py:67] Building graph for training tower 0..., /cpu:0 | |
| 12 | |
| [0130 17:29:49 @_common.py:61] conv0 input: [None, 84, 84, 16] | |
| Tensor("tower0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] conv0 output: [None, 80, 80, 32] | |
| [0130 17:29:49 @_common.py:61] pool0 input: [None, 80, 80, 32] | |
| Tensor("tower0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] pool0 output: [None, 40, 40, 32] | |
| [0130 17:29:49 @_common.py:61] conv1 input: [None, 40, 40, 32] | |
| Tensor("tower0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] conv1 output: [None, 36, 36, 32] | |
| [0130 17:29:49 @_common.py:61] pool1 input: [None, 36, 36, 32] | |
| Tensor("tower0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] pool1 output: [None, 18, 18, 32] | |
| [0130 17:29:49 @_common.py:61] conv2 input: [None, 18, 18, 32] | |
| Tensor("tower0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] conv2 output: [None, 14, 14, 64] | |
| [0130 17:29:49 @_common.py:61] pool2 input: [None, 14, 14, 64] | |
| Tensor("tower0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] pool2 output: [None, 7, 7, 64] | |
| [0130 17:29:49 @_common.py:61] conv3 input: [None, 7, 7, 64] | |
| Tensor("tower0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] conv3 output: [None, 5, 5, 64] | |
| [0130 17:29:49 @_common.py:61] fc1_0 input: [None, 5, 5, 64] | |
| Tensor("tower0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] fc1_0 output: [None, 1, 1, 256] | |
| [0130 17:29:49 @_common.py:61] fc-pi input: [None, 256] | |
| Tensor("tower0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] fc-pi output: [None, 6] | |
| [0130 17:29:49 @_common.py:61] fc-v input: [None, 256] | |
| Tensor("tower0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:49 @_common.py:70] fc-v output: [None, 1] | |
| MOVING_SUMMARY_VARIABLES | |
| [] | |
| [0130 17:29:49 @modelutils.py:22] Model Parameters: | |
| conv0/W:0: shape=[5, 5, 16, 32], dim=12800 | |
| conv1/W:0: shape=[5, 5, 32, 32], dim=25600 | |
| conv2/W:0: shape=[5, 5, 32, 64], dim=51200 | |
| conv3/W:0: shape=[3, 3, 64, 64], dim=36864 | |
| fc1_0/W:0: shape=[5, 5, 64, 256], dim=409600 | |
| fc-pi/W:0: shape=[256, 6], dim=1536 | |
| fc-pi/b:0: shape=[6], dim=6 | |
| fc-v/W:0: shape=[256, 1], dim=256 | |
| fc-v/b:0: shape=[1], dim=1 | |
| Total param=537863 (2.051785 MB assuming all float32) | |
| MOVING_SUMMARY_VARIABLES | |
| [] | |
| [0130 17:29:50 @modelutils.py:22] Model Parameters: | |
| conv0/W:0: shape=[5, 5, 16, 32], dim=12800 | |
| conv1/W:0: shape=[5, 5, 32, 32], dim=25600 | |
| conv2/W:0: shape=[5, 5, 32, 64], dim=51200 | |
| conv3/W:0: shape=[3, 3, 64, 64], dim=36864 | |
| fc1_0/W:0: shape=[5, 5, 64, 256], dim=409600 | |
| fc-pi/W:0: shape=[256, 6], dim=1536 | |
| fc-pi/b:0: shape=[6], dim=6 | |
| fc-v/W:0: shape=[256, 1], dim=256 | |
| fc-v/b:0: shape=[1], dim=1 | |
| Total param=537863 (2.051785 MB assuming all float32) | |
| [0130 17:29:50 @multigpu.py:228] Setup callbacks ... | |
| Creating Predictorfactor 0 | |
| [0130 17:29:50 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:51 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0_1/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_1/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:51 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0_2/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| Tensor("towerp0_2/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:0/device:CPU:0) | |
| [0130 17:29:51 @base.py:177] =============================================================== | |
| [0130 17:29:51 @base.py:179] CHIEF! | |
| [0130 17:29:51 @base.py:180] [p0112] Creating the session | |
| [0130 17:29:51 @base.py:181] =============================================================== | |
| [0130 17:29:51 @multigpu.py:228] Setup callbacks ... | |
| Creating Predictorfactor 0 | |
| [0130 17:29:51 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:51 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0_1/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_1/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:51 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0_2/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| Tensor("towerp0_2/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:1/device:CPU:0) | |
| [0130 17:29:51 @base.py:177] =============================================================== | |
| [0130 17:29:51 @base.py:180] [p0115] Creating the session | |
| [0130 17:29:51 @base.py:181] =============================================================== | |
| 2018-01-30 17:29:57.665912: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.665949: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.665958: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.665965: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.665973: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.670158: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.670198: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.670207: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.670215: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.670222: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. | |
| 2018-01-30 17:29:57.676919: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> p0111:30065} | |
| 2018-01-30 17:29:57.676959: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> p0112:30066, 1 -> p0115:30066, 2 -> p0574:30066, 3 -> localhost:30066} | |
| 2018-01-30 17:29:57.678659: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:30066 | |
| 2018-01-30 17:29:57.680658: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> p0111:30065} | |
| [2018-01-30 17:29:57,679] Making new env: Breakout-v0 | |
| 2018-01-30 17:29:57.680702: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> p0112:30066, 1 -> p0115:30066, 2 -> localhost:30066, 3 -> p0576:30066} | |
| 2018-01-30 17:29:57.682393: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:30066 | |
| [2018-01-30 17:29:57,683] Making new env: Breakout-v0 | |
| {'ps': ['p0111:30065'], 'worker': ['p0112:30066', 'p0115:30066', 'p0574:30066', 'p0576:30066']} | |
| [worker:3] Starting the TF server | |
| args.mkl == 0 | |
| {'ps': ['p0111:30065'], 'worker': ['p0112:30066', 'p0115:30066', 'p0574:30066', 'p0576:30066']} | |
| [worker:2] Starting the TF server | |
| args.mkl == 0 | |
| using tensorflow convolution | |
| using tensorflow convolution | |
| [2018-01-30 17:29:57,918] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,917] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,922] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,923] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,928] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,929] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,934] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,935] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,939] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,940] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,952] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,956] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,957] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,959] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,961] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,964] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,965] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,971] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,978] Making new env: Breakout-v0 | |
| [2018-01-30 17:29:57,978] Making new env: Breakout-v0 | |
| None <type 'NoneType'> | |
| worker host: grpc://localhost:30066 | |
| [0130 17:29:57 @train.py:711] [BA3C] Train on gpu 0 and infer on gpu 0,0,0 | |
| [0130 17:29:57 @train.py:717] using async version | |
| DUMMY PREDICTOR 0 | |
| MultiGPUTrainer __init__ dummy = 0 | |
| [0130 17:29:57 @multigpu.py:57] Training a model of 1 tower | |
| [0130 17:29:57 @multigpu.py:67] Building graph for training tower 0..., /cpu:0 | |
| None <type 'NoneType'> | |
| worker host: grpc://localhost:30066 | |
| [0130 17:29:58 @train.py:711] [BA3C] Train on gpu 0 and infer on gpu 0,0,0 | |
| [0130 17:29:58 @train.py:717] using async version | |
| 12 | |
| DUMMY PREDICTOR 0 | |
| MultiGPUTrainer __init__ dummy = 0 | |
| [0130 17:29:58 @multigpu.py:57] Training a model of 1 tower | |
| [0130 17:29:58 @multigpu.py:67] Building graph for training tower 0..., /cpu:0 | |
| [0130 17:29:58 @_common.py:61] conv0 input: [None, 84, 84, 16] | |
| 12 | |
| Tensor("tower0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv0 output: [None, 80, 80, 32] | |
| [0130 17:29:58 @_common.py:61] pool0 input: [None, 80, 80, 32] | |
| Tensor("tower0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool0 output: [None, 40, 40, 32] | |
| [0130 17:29:58 @_common.py:61] conv1 input: [None, 40, 40, 32] | |
| [0130 17:29:58 @_common.py:61] conv0 input: [None, 84, 84, 16] | |
| Tensor("tower0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv1 output: [None, 36, 36, 32] | |
| [0130 17:29:58 @_common.py:61] pool1 input: [None, 36, 36, 32] | |
| Tensor("tower0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool1 output: [None, 18, 18, 32] | |
| [0130 17:29:58 @_common.py:61] conv2 input: [None, 18, 18, 32] | |
| Tensor("tower0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv0 output: [None, 80, 80, 32] | |
| [0130 17:29:58 @_common.py:61] pool0 input: [None, 80, 80, 32] | |
| Tensor("tower0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv2 output: [None, 14, 14, 64] | |
| Tensor("tower0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool0 output: [None, 40, 40, 32] | |
| [0130 17:29:58 @_common.py:61] conv1 input: [None, 40, 40, 32] | |
| [0130 17:29:58 @_common.py:61] pool2 input: [None, 14, 14, 64] | |
| Tensor("tower0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool2 output: [None, 7, 7, 64] | |
| [0130 17:29:58 @_common.py:61] conv3 input: [None, 7, 7, 64] | |
| Tensor("tower0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv1 output: [None, 36, 36, 32] | |
| Tensor("tower0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv3 output: [None, 5, 5, 64] | |
| [0130 17:29:58 @_common.py:61] pool1 input: [None, 36, 36, 32] | |
| Tensor("tower0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool1 output: [None, 18, 18, 32] | |
| [0130 17:29:58 @_common.py:61] conv2 input: [None, 18, 18, 32] | |
| [0130 17:29:58 @_common.py:61] fc1_0 input: [None, 5, 5, 64] | |
| Tensor("tower0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv2 output: [None, 14, 14, 64] | |
| Tensor("tower0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc1_0 output: [None, 1, 1, 256] | |
| [0130 17:29:58 @_common.py:61] pool2 input: [None, 14, 14, 64] | |
| Tensor("tower0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] pool2 output: [None, 7, 7, 64] | |
| [0130 17:29:58 @_common.py:61] conv3 input: [None, 7, 7, 64] | |
| [0130 17:29:58 @_common.py:61] fc-pi input: [None, 256] | |
| Tensor("tower0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] conv3 output: [None, 5, 5, 64] | |
| [0130 17:29:58 @_common.py:61] fc1_0 input: [None, 5, 5, 64] | |
| Tensor("tower0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc1_0 output: [None, 1, 1, 256] | |
| Tensor("tower0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc-pi output: [None, 6] | |
| [0130 17:29:58 @_common.py:61] fc-v input: [None, 256] | |
| [0130 17:29:58 @_common.py:61] fc-pi input: [None, 256] | |
| Tensor("tower0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc-pi output: [None, 6] | |
| [0130 17:29:58 @_common.py:61] fc-v input: [None, 256] | |
| Tensor("tower0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc-v output: [None, 1] | |
| Tensor("tower0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:29:58 @_common.py:70] fc-v output: [None, 1] | |
| MOVING_SUMMARY_VARIABLES | |
| [] | |
| [0130 17:29:58 @modelutils.py:22] Model Parameters: | |
| conv0/W:0: shape=[5, 5, 16, 32], dim=12800 | |
| conv1/W:0: shape=[5, 5, 32, 32], dim=25600 | |
| conv2/W:0: shape=[5, 5, 32, 64], dim=51200 | |
| conv3/W:0: shape=[3, 3, 64, 64], dim=36864 | |
| fc1_0/W:0: shape=[5, 5, 64, 256], dim=409600 | |
| fc-pi/W:0: shape=[256, 6], dim=1536 | |
| fc-pi/b:0: shape=[6], dim=6 | |
| fc-v/W:0: shape=[256, 1], dim=256 | |
| fc-v/b:0: shape=[1], dim=1 | |
| Total param=537863 (2.051785 MB assuming all float32) | |
| MOVING_SUMMARY_VARIABLES | |
| [] | |
| [0130 17:29:59 @modelutils.py:22] Model Parameters: | |
| conv0/W:0: shape=[5, 5, 16, 32], dim=12800 | |
| conv1/W:0: shape=[5, 5, 32, 32], dim=25600 | |
| conv2/W:0: shape=[5, 5, 32, 64], dim=51200 | |
| conv3/W:0: shape=[3, 3, 64, 64], dim=36864 | |
| fc1_0/W:0: shape=[5, 5, 64, 256], dim=409600 | |
| fc-pi/W:0: shape=[256, 6], dim=1536 | |
| fc-pi/b:0: shape=[6], dim=6 | |
| fc-v/W:0: shape=[256, 1], dim=256 | |
| fc-v/b:0: shape=[1], dim=1 | |
| Total param=537863 (2.051785 MB assuming all float32) | |
| [0130 17:30:00 @multigpu.py:228] Setup callbacks ... | |
| Creating Predictorfactor 0 | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| 2018-01-30 17:30:00.118372: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session f133859301d86503 with config: | |
| 12 | |
| Tensor("towerp0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:30:00 @multigpu.py:228] Setup callbacks ... | |
| Tensor("towerp0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Creating Predictorfactor 0 | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| Tensor("towerp0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| 12 | |
| Tensor("towerp0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| 12 | |
| Tensor("towerp0_1/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| Tensor("towerp0_1/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| 12 | |
| Tensor("towerp0_1/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_1/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| Tensor("towerp0_1/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_1/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| 12 | |
| Tensor("towerp0_2/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| [0130 17:30:00 @base.py:132] Building graph for predictor tower 0... | |
| Tensor("towerp0_2/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| 12 | |
| Tensor("towerp0_2/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:3/device:CPU:0) | |
| Tensor("towerp0_2/conv0/output:0", shape=(?, 80, 80, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/pool0/MaxPool:0", shape=(?, 40, 40, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/conv1/output:0", shape=(?, 36, 36, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/pool1/MaxPool:0", shape=(?, 18, 18, 32), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/conv2/output:0", shape=(?, 14, 14, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/pool2/MaxPool:0", shape=(?, 7, 7, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/conv3/output:0", shape=(?, 5, 5, 64), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/fc1_0/output:0", shape=(?, 1, 1, 256), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:30:00 @base.py:177] =============================================================== | |
| [0130 17:30:00 @base.py:180] [p0576] Creating the session | |
| [0130 17:30:00 @base.py:181] =============================================================== | |
| Tensor("towerp0_2/fc-pi/output:0", shape=(?, 6), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| Tensor("towerp0_2/fc-v/output:0", shape=(?, 1), dtype=float32, device=/job:worker/task:2/device:CPU:0) | |
| [0130 17:30:00 @base.py:177] =============================================================== | |
| [0130 17:30:00 @base.py:180] [p0574] Creating the session | |
| [0130 17:30:00 @base.py:181] =============================================================== | |
| 2018-01-30 17:30:00.790835: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session ce42849ba70813b5 with config: | |
| [0130 17:30:01 @base.py:189] =============================================================== | |
| [0130 17:30:01 @base.py:190] [p0112] Session created | |
| [0130 17:30:01 @base.py:191] =============================================================== | |
| [0130 17:30:01 @base.py:112] [p0112] Initializing graph variables ... | |
| [0130 17:30:01 @base.py:119] [p0112] Starting concurrency... | |
| [0130 17:30:01 @base.py:198] Starting all threads & procs ... | |
| [0130 17:30:01 @base.py:122] [p0112] Setting default session | |
| [0130 17:30:01 @base.py:125] [p0112] Getting global step | |
| [0130 17:30:01 @base.py:127] [p0112] Start training with global_step=0 | |
| server main loop | |
| before socket bind... tcp://*:61216 | |
| receiving | |
| 2018-01-30 17:30:01.209105: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 2eefc7caccbf1c89 with config: | |
| 2018-01-30 17:30:01.276788: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session e774dcd3d1ac3165 with config: | |
| [0130 17:30:01 @base.py:189] =============================================================== | |
| [0130 17:30:01 @base.py:190] [p0576] Session created | |
| [0130 17:30:01 @base.py:191] =============================================================== | |
| [0130 17:30:01 @base.py:112] [p0576] Initializing graph variables ... | |
| [0130 17:30:01 @base.py:119] [p0576] Starting concurrency... | |
| [0130 17:30:01 @base.py:198] Starting all threads & procs ... | |
| [0130 17:30:01 @base.py:122] [p0576] Setting default session | |
| [0130 17:30:01 @base.py:125] [p0576] Getting global step | |
| [0130 17:30:01 @base.py:127] [p0576] Start training with global_step=0 | |
| [0130 17:30:01 @base.py:189] =============================================================== | |
| [0130 17:30:01 @base.py:190] [p0574] Session created | |
| [0130 17:30:01 @base.py:191] =============================================================== | |
| [0130 17:30:01 @base.py:112] [p0574] Initializing graph variables ... | |
| [0130 17:30:01 @base.py:119] [p0574] Starting concurrency... | |
| [0130 17:30:01 @base.py:198] Starting all threads & procs ... | |
| [0130 17:30:01 @base.py:122] [p0574] Setting default session | |
| [0130 17:30:01 @base.py:125] [p0574] Getting global step | |
| [0130 17:30:01 @base.py:127] [p0574] Start training with global_step=0 | |
| [0130 17:30:03 @multigpu.py:323] [p0576] step: count(1), step_time 4762.69, mean_step_time 4762.69, it/s 0.21 | |
| [0130 17:30:03 @multigpu.py:323] [p0574] step: count(1), step_time 4870.79, mean_step_time 4870.79, it/s 0.21 | |
| [0130 17:30:05 @multigpu.py:323] [p0576] step: count(2), step_time 1437.01, mean_step_time 3099.85, it/s 0.32 | |
| [0130 17:30:05 @multigpu.py:323] [p0574] step: count(2), step_time 1388.9, mean_step_time 3129.85, it/s 0.32 | |
| [0130 17:30:06 @multigpu.py:323] [p0576] step: count(3), step_time 1392.92, mean_step_time 2530.87, it/s 0.4 | |
| [0130 17:30:06 @multigpu.py:323] [p0574] step: count(3), step_time 1417.79, mean_step_time 2559.16, it/s 0.39 | |
| [0130 17:30:07 @multigpu.py:323] [p0576] step: count(4), step_time 1381.3, mean_step_time 2243.48, it/s 0.45 | |
| [0130 17:30:08 @multigpu.py:323] [p0574] step: count(4), step_time 1380.92, mean_step_time 2264.6, it/s 0.44 | |
| [0130 17:30:09 @multigpu.py:323] [p0576] step: count(5), step_time 1413.79, mean_step_time 2077.54, it/s 0.48 | |
| [0130 17:30:09 @multigpu.py:323] [p0574] step: count(5), step_time 1414.17, mean_step_time 2094.51, it/s 0.48 | |
| [0130 17:30:10 @multigpu.py:323] [p0576] step: count(6), step_time 1387.62, mean_step_time 1962.56, it/s 0.51 | |
| [0130 17:30:10 @multigpu.py:323] [p0574] step: count(6), step_time 1367.55, mean_step_time 1973.35, it/s 0.51 | |
| [0130 17:30:12 @multigpu.py:323] [p0576] step: count(7), step_time 1363.13, mean_step_time 1876.92, it/s 0.53 | |
| [0130 17:30:12 @multigpu.py:323] [p0574] step: count(7), step_time 1377.66, mean_step_time 1888.25, it/s 0.53 | |
| [0130 17:30:13 @multigpu.py:323] [p0574] step: count(8), step_time 1369.67, mean_step_time 1823.43, it/s 0.55 | |
| [0130 17:30:13 @multigpu.py:323] [p0576] step: count(8), step_time 1444.03, mean_step_time 1822.81, it/s 0.55 | |
| [0130 17:30:14 @multigpu.py:323] [p0574] step: count(9), step_time 1376.32, mean_step_time 1773.75, it/s 0.56 | |
| [0130 17:30:15 @multigpu.py:323] [p0576] step: count(9), step_time 1402.34, mean_step_time 1776.09, it/s 0.56 | |
| [0130 17:30:16 @multigpu.py:323] [p0574] step: count(10), step_time 1358.5, mean_step_time 1732.23, it/s 0.58 | |
| [0130 17:30:16 @multigpu.py:323] [p0576] step: count(10), step_time 1372.5, mean_step_time 1735.73, it/s 0.58 | |
| [0130 17:30:17 @multigpu.py:323] [p0574] step: count(11), step_time 1402.19, mean_step_time 1702.22, it/s 0.59 | |
| [0130 17:30:17 @multigpu.py:323] [p0576] step: count(11), step_time 1415.18, mean_step_time 1706.59, it/s 0.59 | |
| [0130 17:30:19 @multigpu.py:323] [p0574] step: count(12), step_time 1360.35, mean_step_time 1673.73, it/s 0.6 | |
| [0130 17:30:19 @multigpu.py:323] [p0576] step: count(12), step_time 1386.91, mean_step_time 1679.95, it/s 0.6 | |
| [0130 17:30:20 @multigpu.py:323] [p0574] step: count(13), step_time 1434.23, mean_step_time 1655.31, it/s 0.6 | |
| [0130 17:30:20 @multigpu.py:323] [p0576] step: count(13), step_time 1391.04, mean_step_time 1657.73, it/s 0.6 | |
| [0130 17:30:21 @multigpu.py:323] [p0574] step: count(14), step_time 1350.67, mean_step_time 1633.55, it/s 0.61 | |
| [0130 17:30:22 @multigpu.py:323] [p0576] step: count(14), step_time 1425.22, mean_step_time 1641.12, it/s 0.61 | |
| [0130 17:30:23 @multigpu.py:323] [p0574] step: count(15), step_time 1427.02, mean_step_time 1619.78, it/s 0.62 | |
| [0130 17:30:23 @multigpu.py:323] [p0576] step: count(15), step_time 1376.45, mean_step_time 1623.48, it/s 0.62 | |
| [0130 17:30:24 @multigpu.py:323] [p0574] step: count(16), step_time 1363.51, mean_step_time 1603.76, it/s 0.62 | |
| [0130 17:30:24 @multigpu.py:323] [p0576] step: count(16), step_time 1441.44, mean_step_time 1612.1, it/s 0.62 | |
| [0130 17:30:26 @multigpu.py:323] [p0574] step: count(17), step_time 1414.29, mean_step_time 1592.62, it/s 0.63 | |
| [0130 17:30:26 @multigpu.py:323] [p0576] step: count(17), step_time 1441.84, mean_step_time 1602.08, it/s 0.62 | |
| [0130 17:30:27 @multigpu.py:323] [p0574] step: count(18), step_time 1425.53, mean_step_time 1583.34, it/s 0.63 | |
| [0130 17:30:27 @multigpu.py:323] [p0576] step: count(18), step_time 1347.63, mean_step_time 1587.95, it/s 0.63 | |
| [0130 17:30:28 @multigpu.py:323] [p0574] step: count(19), step_time 1392.45, mean_step_time 1573.29, it/s 0.64 | |
| [0130 17:30:29 @multigpu.py:323] [p0576] step: count(19), step_time 1404.59, mean_step_time 1578.3, it/s 0.63 | |
| [0130 17:30:30 @multigpu.py:323] [p0574] step: count(20), step_time 1396.35, mean_step_time 1564.44, it/s 0.64 | |
| [0130 17:30:30 @multigpu.py:323] [p0576] step: count(20), step_time 1363.71, mean_step_time 1567.57, it/s 0.64 | |
| 2018-01-30 17:30:30.682429: I tensorflow/core/distributed_runtime/master_session.cc:999] Start master session 8454efd2be83293e with config: | |
| [0130 17:30:30 @base.py:189] =============================================================== | |
| [0130 17:30:30 @base.py:190] [p0115] Session created | |
| [0130 17:30:30 @base.py:191] =============================================================== | |
| [0130 17:30:30 @base.py:112] [p0115] Initializing graph variables ... | |
| [0130 17:30:30 @base.py:119] [p0115] Starting concurrency... | |
| [0130 17:30:30 @base.py:198] Starting all threads & procs ... | |
| [0130 17:30:30 @base.py:122] [p0115] Setting default session | |
| [0130 17:30:30 @base.py:125] [p0115] Getting global step | |
| [0130 17:30:30 @base.py:127] [p0115] Start training with global_step=0 | |
| [0130 17:30:31 @multigpu.py:323] [p0574] step: count(21), step_time 1395.6, mean_step_time 1390.68, it/s 0.72 | |
| [0130 17:30:31 @multigpu.py:323] [p0576] step: count(21), step_time 1391.65, mean_step_time 1399.02, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.00865667495463 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:30:33 @multigpu.py:323] [p0574] step: count(22), step_time 1402.65, mean_step_time 1391.37, it/s 0.72 | |
| [0130 17:30:33 @multigpu.py:323] [p0576] step: count(22), step_time 1399.34, mean_step_time 1397.13, it/s 0.72 | |
| [0130 17:30:33 @multigpu.py:323] [p0115] step: count(1), step_time 42890.76, mean_step_time 42890.76, it/s 0.02 | |
| [0130 17:30:34 @multigpu.py:323] [p0574] step: count(23), step_time 1389.17, mean_step_time 1389.94, it/s 0.72 | |
| [0130 17:30:34 @multigpu.py:323] [p0576] step: count(23), step_time 1378.85, mean_step_time 1396.43, it/s 0.72 | |
| [0130 17:30:34 @multigpu.py:323] [p0115] step: count(2), step_time 1448.3, mean_step_time 22169.53, it/s 0.05 | |
| [0130 17:30:35 @multigpu.py:323] [p0574] step: count(24), step_time 1366.48, mean_step_time 1389.22, it/s 0.72 | |
| [0130 17:30:35 @multigpu.py:323] [p0576] step: count(24), step_time 1417.93, mean_step_time 1398.26, it/s 0.72 | |
| [0130 17:30:36 @multigpu.py:323] [p0115] step: count(3), step_time 1356.88, mean_step_time 15231.98, it/s 0.07 | |
| [0130 17:30:37 @multigpu.py:323] [p0574] step: count(25), step_time 1368.03, mean_step_time 1386.91, it/s 0.72 | |
| [0130 17:30:37 @multigpu.py:323] [p0576] step: count(25), step_time 1394.76, mean_step_time 1397.31, it/s 0.72 | |
| [0130 17:30:37 @multigpu.py:323] [p0115] step: count(4), step_time 1482.5, mean_step_time 11794.61, it/s 0.08 | |
| [0130 17:30:38 @multigpu.py:323] [p0574] step: count(26), step_time 1432.12, mean_step_time 1390.14, it/s 0.72 | |
| [0130 17:30:38 @multigpu.py:323] [p0576] step: count(26), step_time 1385.95, mean_step_time 1397.23, it/s 0.72 | |
| [0130 17:30:38 @multigpu.py:323] [p0115] step: count(5), step_time 1416.5, mean_step_time 9718.99, it/s 0.1 | |
| [0130 17:30:40 @multigpu.py:323] [p0574] step: count(27), step_time 1389.58, mean_step_time 1390.73, it/s 0.72 | |
| [0130 17:30:40 @multigpu.py:323] [p0576] step: count(27), step_time 1469.08, mean_step_time 1402.52, it/s 0.71 | |
| [0130 17:30:40 @multigpu.py:323] [p0115] step: count(6), step_time 1373.32, mean_step_time 8328.04, it/s 0.12 | |
| [0130 17:30:41 @multigpu.py:323] [p0574] step: count(28), step_time 1414.32, mean_step_time 1392.97, it/s 0.72 | |
| [0130 17:30:41 @multigpu.py:323] [p0576] step: count(28), step_time 1390.28, mean_step_time 1399.83, it/s 0.71 | |
| [0130 17:30:41 @multigpu.py:323] [p0115] step: count(7), step_time 1393.66, mean_step_time 7337.41, it/s 0.14 | |
| [0130 17:30:42 @multigpu.py:323] [p0574] step: count(29), step_time 1395.04, mean_step_time 1393.9, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0116006455157 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:30:43 @multigpu.py:323] [p0576] step: count(29), step_time 1390.76, mean_step_time 1399.26, it/s 0.71 | |
| [0130 17:30:43 @multigpu.py:323] [p0115] step: count(8), step_time 1377.49, mean_step_time 6592.42, it/s 0.15 | |
| [0130 17:30:44 @multigpu.py:323] [p0574] step: count(30), step_time 1366.0, mean_step_time 1394.28, it/s 0.72 | |
| [0130 17:30:44 @multigpu.py:323] [p0576] step: count(30), step_time 1366.41, mean_step_time 1398.95, it/s 0.71 | |
| [0130 17:30:44 @multigpu.py:323] [p0115] step: count(9), step_time 1394.2, mean_step_time 6014.84, it/s 0.17 | |
| [0130 17:30:45 @multigpu.py:323] [p0574] step: count(31), step_time 1413.87, mean_step_time 1394.86, it/s 0.72 | |
| [0130 17:30:45 @multigpu.py:323] [p0576] step: count(31), step_time 1402.14, mean_step_time 1398.3, it/s 0.72 | |
| [0130 17:30:45 @multigpu.py:323] [p0115] step: count(10), step_time 1394.34, mean_step_time 5552.79, it/s 0.18 | |
| [0130 17:30:47 @multigpu.py:323] [p0574] step: count(32), step_time 1385.71, mean_step_time 1396.13, it/s 0.72 | |
| [0130 17:30:47 @multigpu.py:323] [p0576] step: count(32), step_time 1385.07, mean_step_time 1398.21, it/s 0.72 | |
| [0130 17:30:47 @multigpu.py:323] [p0115] step: count(11), step_time 1371.74, mean_step_time 5172.7, it/s 0.19 | |
| [0130 17:30:48 @multigpu.py:323] [p0574] step: count(33), step_time 1395.79, mean_step_time 1394.21, it/s 0.72 | |
| [0130 17:30:48 @multigpu.py:323] [p0576] step: count(33), step_time 1399.75, mean_step_time 1398.64, it/s 0.71 | |
| [0130 17:30:48 @multigpu.py:323] [p0115] step: count(12), step_time 1378.85, mean_step_time 4856.54, it/s 0.21 | |
| [0130 17:30:49 @multigpu.py:323] [p0574] step: count(34), step_time 1391.1, mean_step_time 1396.23, it/s 0.72 | |
| [0130 17:30:49 @multigpu.py:323] [p0576] step: count(34), step_time 1391.81, mean_step_time 1396.97, it/s 0.72 | |
| [0130 17:30:50 @multigpu.py:323] [p0115] step: count(13), step_time 1400.51, mean_step_time 4590.69, it/s 0.22 | |
| [0130 17:30:51 @multigpu.py:323] [p0574] step: count(35), step_time 1359.27, mean_step_time 1392.84, it/s 0.72 | |
| [0130 17:30:51 @multigpu.py:323] [p0576] step: count(35), step_time 1379.89, mean_step_time 1397.15, it/s 0.72 | |
| [0130 17:30:51 @multigpu.py:323] [p0115] step: count(14), step_time 1362.87, mean_step_time 4360.14, it/s 0.23 | |
| [0130 17:30:52 @multigpu.py:323] [p0574] step: count(36), step_time 1472.95, mean_step_time 1398.32, it/s 0.72 | |
| [0130 17:30:52 @multigpu.py:323] [p0576] step: count(36), step_time 1389.22, mean_step_time 1394.53, it/s 0.72 | |
| [0130 17:30:52 @multigpu.py:323] [p0115] step: count(15), step_time 1390.51, mean_step_time 4162.16, it/s 0.24 | |
| [0130 17:30:54 @multigpu.py:323] [p0574] step: count(37), step_time 1416.91, mean_step_time 1398.45, it/s 0.72 | |
| [0130 17:30:54 @multigpu.py:323] [p0576] step: count(37), step_time 1372.71, mean_step_time 1391.08, it/s 0.72 | |
| [0130 17:30:54 @multigpu.py:323] [p0115] step: count(16), step_time 1361.38, mean_step_time 3987.11, it/s 0.25 | |
| [0130 17:30:55 @multigpu.py:323] [p0576] step: count(38), step_time 1411.73, mean_step_time 1394.28, it/s 0.72 | |
| [0130 17:30:55 @multigpu.py:323] [p0115] step: count(17), step_time 1413.07, mean_step_time 3835.7, it/s 0.26 | |
| [0130 17:30:55 @multigpu.py:323] [p0574] step: count(38), step_time 1520.18, mean_step_time 1403.18, it/s 0.71 | |
| [0130 17:30:56 @multigpu.py:323] [p0115] step: count(18), step_time 1368.26, mean_step_time 3698.62, it/s 0.27 | |
| [0130 17:30:56 @multigpu.py:323] [p0576] step: count(39), step_time 1408.37, mean_step_time 1394.47, it/s 0.72 | |
| [0130 17:30:57 @multigpu.py:323] [p0574] step: count(39), step_time 1564.09, mean_step_time 1411.76, it/s 0.71 | |
| [0130 17:30:58 @multigpu.py:323] [p0115] step: count(19), step_time 1387.09, mean_step_time 3576.96, it/s 0.28 | |
| [0130 17:30:58 @multigpu.py:323] [p0576] step: count(40), step_time 1374.96, mean_step_time 1395.03, it/s 0.72 | |
| [0130 17:30:58 @multigpu.py:323] [p0574] step: count(40), step_time 1608.55, mean_step_time 1422.37, it/s 0.7 | |
| [0130 17:30:59 @multigpu.py:323] [p0576] step: count(41), step_time 1359.14, mean_step_time 1393.41, it/s 0.72 | |
| [0130 17:30:59 @multigpu.py:323] [p0115] step: count(20), step_time 1390.59, mean_step_time 3467.64, it/s 0.29 | |
| [0130 17:31:00 @multigpu.py:323] [p0574] step: count(41), step_time 1480.41, mean_step_time 1426.61, it/s 0.7 | |
| [0130 17:31:01 @multigpu.py:323] [p0576] step: count(42), step_time 1370.21, mean_step_time 1391.95, it/s 0.72 | |
| [0130 17:31:01 @multigpu.py:323] [p0115] step: count(21), step_time 1360.8, mean_step_time 1391.14, it/s 0.72 | |
| [0130 17:31:01 @multigpu.py:323] [p0574] step: count(42), step_time 1462.71, mean_step_time 1429.61, it/s 0.7 | |
| [0130 17:31:02 @multigpu.py:323] [p0576] step: count(43), step_time 1372.3, mean_step_time 1391.62, it/s 0.72 | |
| [0130 17:31:02 @multigpu.py:323] [p0115] step: count(22), step_time 1454.88, mean_step_time 1391.47, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0171053697003 , 1.2 ##### | |
| [u'online', 1.2] | |
| ##### Sending to neptune: active_workers : 0.0171054766575 , 2 ##### | |
| receiving | |
| [0130 17:31:03 @multigpu.py:323] [p0574] step: count(43), step_time 1458.14, mean_step_time 1433.06, it/s 0.7 | |
| [0130 17:31:03 @multigpu.py:323] [p0576] step: count(44), step_time 1351.7, mean_step_time 1388.31, it/s 0.72 | |
| [0130 17:31:03 @multigpu.py:323] [p0115] step: count(23), step_time 1368.19, mean_step_time 1392.04, it/s 0.72 | |
| [0130 17:31:04 @multigpu.py:323] [p0574] step: count(44), step_time 1500.34, mean_step_time 1439.76, it/s 0.69 | |
| [0130 17:31:05 @multigpu.py:323] [p0576] step: count(45), step_time 1414.58, mean_step_time 1389.3, it/s 0.72 | |
| [0130 17:31:05 @multigpu.py:323] [p0115] step: count(24), step_time 1397.49, mean_step_time 1387.79, it/s 0.72 | |
| [0130 17:31:06 @multigpu.py:323] [p0574] step: count(45), step_time 1476.14, mean_step_time 1445.16, it/s 0.69 | |
| [0130 17:31:06 @multigpu.py:323] [p0576] step: count(46), step_time 1353.29, mean_step_time 1387.67, it/s 0.72 | |
| [0130 17:31:06 @multigpu.py:323] [p0115] step: count(25), step_time 1354.62, mean_step_time 1384.69, it/s 0.72 | |
| [0130 17:31:07 @multigpu.py:323] [p0574] step: count(46), step_time 1478.86, mean_step_time 1447.5, it/s 0.69 | |
| [0130 17:31:07 @multigpu.py:323] [p0576] step: count(47), step_time 1360.59, mean_step_time 1382.25, it/s 0.72 | |
| [0130 17:31:08 @multigpu.py:323] [p0115] step: count(26), step_time 1414.51, mean_step_time 1386.75, it/s 0.72 | |
| [0130 17:31:09 @multigpu.py:323] [p0574] step: count(47), step_time 1581.36, mean_step_time 1457.09, it/s 0.69 | |
| [0130 17:31:10 @multigpu.py:323] [p0574] step: count(48), step_time 1608.18, mean_step_time 1466.78, it/s 0.68 | |
| [0130 17:31:10 @multigpu.py:323] [p0576] step: count(48), step_time 2931.45, mean_step_time 1459.3, it/s 0.69 | |
| [0130 17:31:10 @multigpu.py:323] [p0115] step: count(27), step_time 2777.06, mean_step_time 1455.92, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0195416252481 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:31:12 @multigpu.py:323] [p0576] step: count(49), step_time 1407.16, mean_step_time 1460.12, it/s 0.68 | |
| [0130 17:31:12 @multigpu.py:323] [p0115] step: count(28), step_time 1437.32, mean_step_time 1458.91, it/s 0.69 | |
| [0130 17:31:12 @multigpu.py:323] [p0574] step: count(49), step_time 1500.8, mean_step_time 1472.07, it/s 0.68 | |
| [0130 17:31:13 @multigpu.py:323] [p0576] step: count(50), step_time 1344.68, mean_step_time 1459.04, it/s 0.69 | |
| [0130 17:31:13 @multigpu.py:323] [p0115] step: count(29), step_time 1368.68, mean_step_time 1457.64, it/s 0.69 | |
| [0130 17:31:13 @multigpu.py:323] [p0574] step: count(50), step_time 1415.66, mean_step_time 1474.55, it/s 0.68 | |
| [0130 17:31:14 @multigpu.py:323] [p0576] step: count(51), step_time 1377.45, mean_step_time 1457.8, it/s 0.69 | |
| [0130 17:31:14 @multigpu.py:323] [p0115] step: count(30), step_time 1373.92, mean_step_time 1456.62, it/s 0.69 | |
| [0130 17:31:15 @multigpu.py:323] [p0574] step: count(51), step_time 1430.28, mean_step_time 1475.37, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0207704544067 , 2.4 ##### | |
| [u'online', 2.4] | |
| receiving | |
| [0130 17:31:16 @multigpu.py:323] [p0576] step: count(52), step_time 1360.11, mean_step_time 1456.56, it/s 0.69 | |
| [0130 17:31:16 @multigpu.py:323] [p0115] step: count(31), step_time 1392.93, mean_step_time 1457.68, it/s 0.69 | |
| [0130 17:31:16 @multigpu.py:323] [p0574] step: count(52), step_time 1571.52, mean_step_time 1484.66, it/s 0.67 | |
| [0130 17:31:17 @multigpu.py:323] [p0576] step: count(53), step_time 1366.86, mean_step_time 1454.91, it/s 0.69 | |
| [0130 17:31:17 @multigpu.py:323] [p0115] step: count(32), step_time 1387.37, mean_step_time 1458.1, it/s 0.69 | |
| [0130 17:31:18 @multigpu.py:323] [p0574] step: count(53), step_time 1407.6, mean_step_time 1485.25, it/s 0.67 | |
| [0130 17:31:19 @multigpu.py:323] [p0576] step: count(54), step_time 1391.97, mean_step_time 1454.92, it/s 0.69 | |
| [0130 17:31:19 @multigpu.py:323] [p0115] step: count(33), step_time 1400.33, mean_step_time 1458.09, it/s 0.69 | |
| [0130 17:31:19 @multigpu.py:323] [p0574] step: count(54), step_time 1387.15, mean_step_time 1485.06, it/s 0.67 | |
| [0130 17:31:20 @multigpu.py:323] [p0576] step: count(55), step_time 1381.99, mean_step_time 1455.02, it/s 0.69 | |
| [0130 17:31:20 @multigpu.py:323] [p0115] step: count(34), step_time 1369.56, mean_step_time 1458.43, it/s 0.69 | |
| [0130 17:31:20 @multigpu.py:323] [p0574] step: count(55), step_time 1407.24, mean_step_time 1487.45, it/s 0.67 | |
| [0130 17:31:21 @multigpu.py:323] [p0576] step: count(56), step_time 1388.41, mean_step_time 1454.98, it/s 0.69 | |
| [0130 17:31:21 @multigpu.py:323] [p0115] step: count(35), step_time 1419.57, mean_step_time 1459.88, it/s 0.68 | |
| [0130 17:31:22 @multigpu.py:323] [p0574] step: count(56), step_time 1437.67, mean_step_time 1485.69, it/s 0.67 | |
| [0130 17:31:23 @multigpu.py:323] [p0576] step: count(57), step_time 1392.69, mean_step_time 1455.98, it/s 0.69 | |
| [0130 17:31:23 @multigpu.py:323] [p0115] step: count(36), step_time 1377.4, mean_step_time 1460.68, it/s 0.68 | |
| [0130 17:31:23 @multigpu.py:323] [p0574] step: count(57), step_time 1357.55, mean_step_time 1482.72, it/s 0.67 | |
| [0130 17:31:24 @multigpu.py:323] [p0576] step: count(58), step_time 1387.9, mean_step_time 1454.79, it/s 0.69 | |
| [0130 17:31:24 @multigpu.py:323] [p0115] step: count(37), step_time 1381.1, mean_step_time 1459.08, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0233404641019 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:31:25 @multigpu.py:323] [p0574] step: count(58), step_time 1408.59, mean_step_time 1477.14, it/s 0.68 | |
| [0130 17:31:26 @multigpu.py:323] [p0576] step: count(59), step_time 1427.41, mean_step_time 1455.74, it/s 0.69 | |
| [0130 17:31:26 @multigpu.py:323] [p0115] step: count(38), step_time 1400.34, mean_step_time 1460.69, it/s 0.68 | |
| [0130 17:31:26 @multigpu.py:323] [p0574] step: count(59), step_time 1421.86, mean_step_time 1470.03, it/s 0.68 | |
| [0130 17:31:27 @multigpu.py:323] [p0576] step: count(60), step_time 1393.25, mean_step_time 1456.66, it/s 0.69 | |
| [0130 17:31:27 @multigpu.py:323] [p0115] step: count(39), step_time 1392.08, mean_step_time 1460.94, it/s 0.68 | |
| [0130 17:31:27 @multigpu.py:323] [p0574] step: count(60), step_time 1420.09, mean_step_time 1460.61, it/s 0.68 | |
| [0130 17:31:28 @multigpu.py:323] [p0576] step: count(61), step_time 1368.35, mean_step_time 1457.12, it/s 0.69 | |
| [0130 17:31:28 @multigpu.py:323] [p0115] step: count(40), step_time 1355.33, mean_step_time 1459.17, it/s 0.69 | |
| [0130 17:31:29 @multigpu.py:323] [p0574] step: count(61), step_time 1562.18, mean_step_time 1464.7, it/s 0.68 | |
| [0130 17:31:30 @multigpu.py:323] [p0576] step: count(62), step_time 1390.44, mean_step_time 1458.13, it/s 0.69 | |
| [0130 17:31:30 @multigpu.py:323] [p0115] step: count(41), step_time 1372.64, mean_step_time 1459.77, it/s 0.69 | |
| [0130 17:31:31 @multigpu.py:323] [p0574] step: count(62), step_time 1453.26, mean_step_time 1464.22, it/s 0.68 | |
| [0130 17:31:31 @multigpu.py:323] [p0576] step: count(63), step_time 1373.3, mean_step_time 1458.18, it/s 0.69 | |
| [0130 17:31:31 @multigpu.py:323] [p0115] step: count(42), step_time 1391.55, mean_step_time 1456.6, it/s 0.69 | |
| [0130 17:31:32 @multigpu.py:323] [p0574] step: count(63), step_time 1447.01, mean_step_time 1463.67, it/s 0.68 | |
| [0130 17:31:32 @multigpu.py:323] [p0576] step: count(64), step_time 1352.92, mean_step_time 1458.24, it/s 0.69 | |
| [0130 17:31:32 @multigpu.py:323] [p0115] step: count(43), step_time 1351.92, mean_step_time 1455.79, it/s 0.69 | |
| [0130 17:31:34 @multigpu.py:323] [p0574] step: count(64), step_time 1567.27, mean_step_time 1467.01, it/s 0.68 | |
| [0130 17:31:34 @multigpu.py:323] [p0576] step: count(65), step_time 1410.33, mean_step_time 1458.03, it/s 0.69 | |
| [0130 17:31:34 @multigpu.py:323] [p0115] step: count(44), step_time 1417.63, mean_step_time 1456.79, it/s 0.69 | |
| [0130 17:31:35 @multigpu.py:323] [p0574] step: count(65), step_time 1457.16, mean_step_time 1466.06, it/s 0.68 | |
| [0130 17:31:35 @multigpu.py:323] [p0576] step: count(66), step_time 1330.76, mean_step_time 1456.9, it/s 0.69 | |
| [0130 17:31:35 @multigpu.py:323] [p0115] step: count(45), step_time 1425.71, mean_step_time 1460.35, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0265158143971 , 0.4 ##### | |
| [u'online', 0.4] | |
| receiving | |
| [0130 17:31:36 @multigpu.py:323] [p0574] step: count(66), step_time 1510.81, mean_step_time 1467.66, it/s 0.68 | |
| [0130 17:31:37 @multigpu.py:323] [p0576] step: count(67), step_time 1398.99, mean_step_time 1458.82, it/s 0.69 | |
| [0130 17:31:37 @multigpu.py:323] [p0115] step: count(46), step_time 1376.59, mean_step_time 1458.45, it/s 0.69 | |
| [0130 17:31:38 @multigpu.py:323] [p0576] step: count(68), step_time 1353.56, mean_step_time 1379.93, it/s 0.72 | |
| [0130 17:31:39 @multigpu.py:323] [p0574] step: count(67), step_time 2849.34, mean_step_time 1531.06, it/s 0.65 | |
| [0130 17:31:39 @multigpu.py:323] [p0576] step: count(69), step_time 1413.17, mean_step_time 1380.23, it/s 0.72 | |
| [0130 17:31:39 @multigpu.py:323] [p0115] step: count(47), step_time 2621.15, mean_step_time 1450.66, it/s 0.69 | |
| [0130 17:31:41 @multigpu.py:323] [p0576] step: count(70), step_time 1364.24, mean_step_time 1381.21, it/s 0.72 | |
| [0130 17:31:41 @multigpu.py:323] [p0574] step: count(68), step_time 1413.99, mean_step_time 1521.35, it/s 0.66 | |
| [0130 17:31:41 @multigpu.py:323] [p0115] step: count(48), step_time 1445.55, mean_step_time 1451.07, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0280429810948 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:31:42 @multigpu.py:323] [p0576] step: count(71), step_time 1389.69, mean_step_time 1381.82, it/s 0.72 | |
| [0130 17:31:42 @multigpu.py:323] [p0574] step: count(69), step_time 1360.72, mean_step_time 1514.35, it/s 0.66 | |
| [0130 17:31:42 @multigpu.py:323] [p0115] step: count(49), step_time 1398.37, mean_step_time 1452.55, it/s 0.69 | |
| [0130 17:31:43 @multigpu.py:323] [p0576] step: count(72), step_time 1337.14, mean_step_time 1380.67, it/s 0.72 | |
| [0130 17:31:44 @multigpu.py:323] [p0574] step: count(70), step_time 1395.54, mean_step_time 1513.34, it/s 0.66 | |
| [0130 17:31:44 @multigpu.py:323] [p0115] step: count(50), step_time 1376.98, mean_step_time 1452.7, it/s 0.69 | |
| [0130 17:31:45 @multigpu.py:323] [p0576] step: count(73), step_time 1400.4, mean_step_time 1382.35, it/s 0.72 | |
| [0130 17:31:45 @multigpu.py:323] [p0574] step: count(71), step_time 1398.99, mean_step_time 1511.78, it/s 0.66 | |
| [0130 17:31:45 @multigpu.py:323] [p0115] step: count(51), step_time 1356.44, mean_step_time 1450.88, it/s 0.69 | |
| [0130 17:31:46 @multigpu.py:323] [p0576] step: count(74), step_time 1366.03, mean_step_time 1381.05, it/s 0.72 | |
| [0130 17:31:46 @multigpu.py:323] [p0574] step: count(72), step_time 1349.11, mean_step_time 1500.66, it/s 0.67 | |
| [0130 17:31:46 @multigpu.py:323] [p0115] step: count(52), step_time 1407.75, mean_step_time 1451.9, it/s 0.69 | |
| [0130 17:31:48 @multigpu.py:323] [p0574] step: count(73), step_time 1410.12, mean_step_time 1500.78, it/s 0.67 | |
| [0130 17:31:48 @multigpu.py:323] [p0576] step: count(75), step_time 1482.05, mean_step_time 1386.05, it/s 0.72 | |
| [0130 17:31:48 @multigpu.py:323] [p0115] step: count(53), step_time 1391.0, mean_step_time 1451.43, it/s 0.69 | |
| [0130 17:31:49 @multigpu.py:323] [p0115] step: count(54), step_time 1359.58, mean_step_time 1450.93, it/s 0.69 | |
| [0130 17:31:49 @multigpu.py:323] [p0576] step: count(76), step_time 1401.94, mean_step_time 1386.73, it/s 0.72 | |
| [0130 17:31:49 @multigpu.py:323] [p0574] step: count(74), step_time 1418.6, mean_step_time 1502.36, it/s 0.67 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0303178360727 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:31:50 @multigpu.py:323] [p0576] step: count(77), step_time 1349.13, mean_step_time 1384.55, it/s 0.72 | |
| [0130 17:31:50 @multigpu.py:323] [p0574] step: count(75), step_time 1366.17, mean_step_time 1500.3, it/s 0.67 | |
| [0130 17:31:50 @multigpu.py:323] [p0115] step: count(55), step_time 1408.07, mean_step_time 1450.36, it/s 0.69 | |
| [0130 17:31:52 @multigpu.py:323] [p0574] step: count(76), step_time 1361.83, mean_step_time 1496.51, it/s 0.67 | |
| [0130 17:31:52 @multigpu.py:323] [p0576] step: count(78), step_time 1419.72, mean_step_time 1386.14, it/s 0.72 | |
| [0130 17:31:52 @multigpu.py:323] [p0115] step: count(56), step_time 1383.6, mean_step_time 1450.67, it/s 0.69 | |
| [0130 17:31:53 @multigpu.py:323] [p0574] step: count(77), step_time 1408.0, mean_step_time 1499.03, it/s 0.67 | |
| [0130 17:31:53 @multigpu.py:323] [p0576] step: count(79), step_time 1385.7, mean_step_time 1384.06, it/s 0.72 | |
| [0130 17:31:53 @multigpu.py:323] [p0115] step: count(57), step_time 1368.43, mean_step_time 1450.04, it/s 0.69 | |
| [0130 17:31:55 @multigpu.py:323] [p0115] step: count(58), step_time 1342.06, mean_step_time 1447.12, it/s 0.69 | |
| [0130 17:31:55 @multigpu.py:323] [p0574] step: count(78), step_time 1385.83, mean_step_time 1497.89, it/s 0.67 | |
| [0130 17:31:55 @multigpu.py:323] [p0576] step: count(80), step_time 1390.21, mean_step_time 1383.9, it/s 0.72 | |
| [0130 17:31:56 @multigpu.py:323] [p0115] step: count(59), step_time 1394.26, mean_step_time 1447.23, it/s 0.69 | |
| [0130 17:31:56 @multigpu.py:323] [p0576] step: count(81), step_time 1366.46, mean_step_time 1383.81, it/s 0.72 | |
| [0130 17:31:56 @multigpu.py:323] [p0574] step: count(79), step_time 1432.96, mean_step_time 1498.45, it/s 0.67 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0322564772103 , 0.7 ##### | |
| [u'online', 0.7] | |
| receiving | |
| [0130 17:31:57 @multigpu.py:323] [p0576] step: count(82), step_time 1393.91, mean_step_time 1383.98, it/s 0.72 | |
| [0130 17:31:57 @multigpu.py:323] [p0115] step: count(60), step_time 1417.24, mean_step_time 1450.33, it/s 0.69 | |
| [0130 17:31:57 @multigpu.py:323] [p0574] step: count(80), step_time 1353.39, mean_step_time 1495.11, it/s 0.67 | |
| [0130 17:31:59 @multigpu.py:323] [p0576] step: count(83), step_time 1352.62, mean_step_time 1382.95, it/s 0.72 | |
| [0130 17:31:59 @multigpu.py:323] [p0115] step: count(61), step_time 1369.22, mean_step_time 1450.16, it/s 0.69 | |
| [0130 17:31:59 @multigpu.py:323] [p0574] step: count(81), step_time 1419.05, mean_step_time 1487.96, it/s 0.67 | |
| [0130 17:32:00 @multigpu.py:323] [p0576] step: count(84), step_time 1376.94, mean_step_time 1384.15, it/s 0.72 | |
| [0130 17:32:00 @multigpu.py:323] [p0115] step: count(62), step_time 1392.11, mean_step_time 1450.18, it/s 0.69 | |
| [0130 17:32:00 @multigpu.py:323] [p0574] step: count(82), step_time 1567.51, mean_step_time 1493.67, it/s 0.67 | |
| [0130 17:32:01 @multigpu.py:323] [p0576] step: count(85), step_time 1364.35, mean_step_time 1381.85, it/s 0.72 | |
| [0130 17:32:02 @multigpu.py:323] [p0115] step: count(63), step_time 1402.84, mean_step_time 1452.73, it/s 0.69 | |
| [0130 17:32:02 @multigpu.py:323] [p0574] step: count(83), step_time 1388.35, mean_step_time 1490.74, it/s 0.67 | |
| [0130 17:32:03 @multigpu.py:323] [p0576] step: count(86), step_time 1388.53, mean_step_time 1384.74, it/s 0.72 | |
| [0130 17:32:03 @multigpu.py:323] [p0115] step: count(64), step_time 1359.18, mean_step_time 1449.81, it/s 0.69 | |
| [0130 17:32:03 @multigpu.py:323] [p0574] step: count(84), step_time 1357.99, mean_step_time 1480.27, it/s 0.68 | |
| [0130 17:32:04 @multigpu.py:323] [p0576] step: count(87), step_time 1357.1, mean_step_time 1382.64, it/s 0.72 | |
| [0130 17:32:04 @multigpu.py:323] [p0115] step: count(65), step_time 1433.11, mean_step_time 1450.18, it/s 0.69 | |
| [0130 17:32:05 @multigpu.py:323] [p0574] step: count(85), step_time 1411.06, mean_step_time 1477.97, it/s 0.68 | |
| [0130 17:32:06 @multigpu.py:323] [p0576] step: count(88), step_time 1413.38, mean_step_time 1385.63, it/s 0.72 | |
| [0130 17:32:06 @multigpu.py:323] [p0115] step: count(66), step_time 1376.56, mean_step_time 1450.18, it/s 0.69 | |
| [0130 17:32:06 @multigpu.py:323] [p0574] step: count(86), step_time 1371.07, mean_step_time 1470.98, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0349453416136 , 1.0 ##### | |
| [u'online', 1.0] | |
| ##### Sending to neptune: active_workers : 0.034945417775 , 3 ##### | |
| receiving | |
| [0130 17:32:07 @multigpu.py:323] [p0574] step: count(87), step_time 1381.74, mean_step_time 1397.6, it/s 0.72 | |
| [0130 17:32:07 @multigpu.py:323] [p0576] step: count(89), step_time 1651.44, mean_step_time 1397.55, it/s 0.72 | |
| [0130 17:32:07 @multigpu.py:323] [p0115] step: count(67), step_time 1568.35, mean_step_time 1397.54, it/s 0.72 | |
| [0130 17:32:09 @multigpu.py:323] [p0574] step: count(88), step_time 1388.4, mean_step_time 1396.32, it/s 0.72 | |
| [0130 17:32:09 @multigpu.py:323] [p0115] step: count(68), step_time 1399.14, mean_step_time 1395.21, it/s 0.72 | |
| [0130 17:32:09 @multigpu.py:323] [p0576] step: count(90), step_time 1415.9, mean_step_time 1400.13, it/s 0.71 | |
| [0130 17:32:10 @multigpu.py:323] [p0574] step: count(89), step_time 1346.75, mean_step_time 1395.62, it/s 0.72 | |
| [0130 17:32:10 @multigpu.py:323] [p0576] step: count(91), step_time 1336.87, mean_step_time 1397.49, it/s 0.72 | |
| [0130 17:32:10 @multigpu.py:323] [p0115] step: count(69), step_time 1472.22, mean_step_time 1398.91, it/s 0.71 | |
| [0130 17:32:11 @multigpu.py:323] [p0574] step: count(90), step_time 1396.78, mean_step_time 1395.68, it/s 0.72 | |
| [0130 17:32:12 @multigpu.py:323] [p0576] step: count(92), step_time 1474.83, mean_step_time 1404.37, it/s 0.71 | |
| [0130 17:32:12 @multigpu.py:323] [p0115] step: count(70), step_time 1359.43, mean_step_time 1398.03, it/s 0.72 | |
| [0130 17:32:13 @multigpu.py:323] [p0574] step: count(91), step_time 1389.08, mean_step_time 1395.19, it/s 0.72 | |
| [0130 17:32:13 @multigpu.py:323] [p0576] step: count(93), step_time 1346.17, mean_step_time 1401.66, it/s 0.71 | |
| [0130 17:32:13 @multigpu.py:323] [p0115] step: count(71), step_time 1420.7, mean_step_time 1401.24, it/s 0.71 | |
| [0130 17:32:14 @multigpu.py:323] [p0574] step: count(92), step_time 1377.89, mean_step_time 1396.63, it/s 0.72 | |
| [0130 17:32:14 @multigpu.py:323] [p0576] step: count(94), step_time 1409.36, mean_step_time 1403.83, it/s 0.71 | |
| [0130 17:32:14 @multigpu.py:323] [p0115] step: count(72), step_time 1387.85, mean_step_time 1400.25, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0373082438442 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:32:16 @multigpu.py:323] [p0574] step: count(93), step_time 1398.81, mean_step_time 1396.06, it/s 0.72 | |
| [0130 17:32:16 @multigpu.py:323] [p0576] step: count(95), step_time 1378.33, mean_step_time 1398.64, it/s 0.71 | |
| [0130 17:32:16 @multigpu.py:323] [p0115] step: count(73), step_time 1430.54, mean_step_time 1402.23, it/s 0.71 | |
| [0130 17:32:17 @multigpu.py:323] [p0576] step: count(96), step_time 1360.46, mean_step_time 1396.57, it/s 0.72 | |
| [0130 17:32:17 @multigpu.py:323] [p0574] step: count(94), step_time 1466.82, mean_step_time 1398.47, it/s 0.72 | |
| [0130 17:32:17 @multigpu.py:323] [p0115] step: count(74), step_time 1409.39, mean_step_time 1404.72, it/s 0.71 | |
| [0130 17:32:18 @multigpu.py:323] [p0576] step: count(97), step_time 1391.39, mean_step_time 1398.68, it/s 0.71 | |
| [0130 17:32:19 @multigpu.py:323] [p0574] step: count(95), step_time 1440.57, mean_step_time 1402.19, it/s 0.71 | |
| [0130 17:32:19 @multigpu.py:323] [p0115] step: count(75), step_time 1384.89, mean_step_time 1403.56, it/s 0.71 | |
| [0130 17:32:20 @multigpu.py:323] [p0576] step: count(98), step_time 1356.32, mean_step_time 1395.51, it/s 0.72 | |
| [0130 17:32:20 @multigpu.py:323] [p0115] step: count(76), step_time 1322.07, mean_step_time 1400.48, it/s 0.71 | |
| [0130 17:32:20 @multigpu.py:323] [p0574] step: count(96), step_time 1402.93, mean_step_time 1404.25, it/s 0.71 | |
| [0130 17:32:21 @multigpu.py:323] [p0576] step: count(99), step_time 1364.49, mean_step_time 1394.45, it/s 0.72 | |
| [0130 17:32:21 @multigpu.py:323] [p0115] step: count(77), step_time 1393.59, mean_step_time 1401.74, it/s 0.71 | |
| [0130 17:32:21 @multigpu.py:323] [p0574] step: count(97), step_time 1396.28, mean_step_time 1403.66, it/s 0.71 | |
| [0130 17:32:23 @multigpu.py:323] [p0576] step: count(100), step_time 1392.71, mean_step_time 1394.58, it/s 0.72 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.0394246213304 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.0394246213304 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.0394246213304 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.0394251494275 , -0.0056879054755 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.0394251494275 , 0.399972140789 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.0394251494275 , -2.29181861877 ##### | |
| ##### Sending to neptune: value_loss : 0.0394251494275 , 1.16379439831 ##### | |
| ##### Sending to neptune: advantage : 0.0394251494275 , -0.00176788156386 ##### | |
| ##### Sending to neptune: pred_reward : 0.0394251494275 , 0.261878997087 ##### | |
| ##### Sending to neptune: max_logit : 0.0394251494275 , 0.181682661176 ##### | |
| [u'loss', -0.005687905475497246, 0.399972140789032, -2.291818618774414, 1.1637943983078003, -0.0017678815638646483, 0.26187899708747864, 0.18168266117572784] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0394261499908 , 9003946.08 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0394261499908 , 86.5037380059 ##### | |
| [u'other', 9003946.08, 86.50373800587406] | |
| receiving | |
| [0130 17:32:23 @multigpu.py:323] [p0115] step: count(78), step_time 1362.33, mean_step_time 1402.75, it/s 0.71 | |
| [0130 17:32:23 @multigpu.py:323] [p0574] step: count(98), step_time 1373.19, mean_step_time 1403.03, it/s 0.71 | |
| [0130 17:32:24 @multigpu.py:323] [p0576] step: count(101), step_time 1420.15, mean_step_time 1397.26, it/s 0.72 | |
| [0130 17:32:24 @multigpu.py:323] [p0115] step: count(79), step_time 1413.0, mean_step_time 1403.69, it/s 0.71 | |
| [0130 17:32:24 @multigpu.py:323] [p0574] step: count(99), step_time 1382.8, mean_step_time 1400.52, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.040045074953 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 17:32:25 @multigpu.py:323] [p0576] step: count(102), step_time 1406.42, mean_step_time 1397.89, it/s 0.72 | |
| [0130 17:32:25 @multigpu.py:323] [p0115] step: count(80), step_time 1359.02, mean_step_time 1400.78, it/s 0.71 | |
| [0130 17:32:25 @multigpu.py:323] [p0574] step: count(100), step_time 1389.18, mean_step_time 1402.31, it/s 0.71 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.0402365121577 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.0402365121577 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.0402365121577 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.0402370121744 , 0.00216898717918 ##### | |
| ##### Sending to neptune: policy_loss : 0.0402370121744 , 1.23316562176 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.0402370121744 , -2.29180669785 ##### | |
| ##### Sending to neptune: value_loss : 0.0402370121744 , 1.3362711668 ##### | |
| ##### Sending to neptune: advantage : 0.0402370121744 , -0.00537997484207 ##### | |
| ##### Sending to neptune: pred_reward : 0.0402370121744 , 0.260432839394 ##### | |
| ##### Sending to neptune: max_logit : 0.0402370121744 , 0.181708931923 ##### | |
| [u'loss', 0.0021689871791750193, 1.2331656217575073, -2.291806697845459, 1.3362711668014526, -0.005379974842071533, 0.2604328393936157, 0.1817089319229126] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0402375141117 , 8991974.32 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0402375141117 , 84.6942909111 ##### | |
| [u'other', 8991974.32, 84.69429091109998] | |
| receiving | |
| [0130 17:32:27 @multigpu.py:323] [p0576] step: count(103), step_time 1386.27, mean_step_time 1399.57, it/s 0.71 | |
| [0130 17:32:27 @multigpu.py:323] [p0115] step: count(81), step_time 1420.97, mean_step_time 1403.37, it/s 0.71 | |
| [0130 17:32:27 @multigpu.py:323] [p0574] step: count(101), step_time 1417.69, mean_step_time 1402.24, it/s 0.71 | |
| [0130 17:32:28 @multigpu.py:323] [p0576] step: count(104), step_time 1378.61, mean_step_time 1399.65, it/s 0.71 | |
| [0130 17:32:28 @multigpu.py:323] [p0574] step: count(102), step_time 1342.28, mean_step_time 1390.98, it/s 0.72 | |
| [0130 17:32:28 @multigpu.py:323] [p0115] step: count(82), step_time 1416.15, mean_step_time 1404.57, it/s 0.71 | |
| [0130 17:32:30 @multigpu.py:323] [p0576] step: count(105), step_time 1376.33, mean_step_time 1400.25, it/s 0.71 | |
| [0130 17:32:30 @multigpu.py:323] [p0574] step: count(103), step_time 1393.23, mean_step_time 1391.23, it/s 0.72 | |
| [0130 17:32:30 @multigpu.py:323] [p0115] step: count(83), step_time 1367.98, mean_step_time 1402.82, it/s 0.71 | |
| [0130 17:32:31 @multigpu.py:323] [p0576] step: count(106), step_time 1405.28, mean_step_time 1401.09, it/s 0.71 | |
| [0130 17:32:31 @multigpu.py:323] [p0574] step: count(104), step_time 1371.23, mean_step_time 1391.89, it/s 0.72 | |
| [0130 17:32:31 @multigpu.py:323] [p0115] step: count(84), step_time 1419.1, mean_step_time 1405.82, it/s 0.71 | |
| [0130 17:32:32 @multigpu.py:323] [p0576] step: count(107), step_time 1391.55, mean_step_time 1402.81, it/s 0.71 | |
| [0130 17:32:32 @multigpu.py:323] [p0574] step: count(105), step_time 1418.55, mean_step_time 1392.26, it/s 0.72 | |
| [0130 17:32:32 @multigpu.py:323] [p0115] step: count(85), step_time 1394.13, mean_step_time 1403.87, it/s 0.71 | |
| [0130 17:32:34 @multigpu.py:323] [p0576] step: count(108), step_time 1447.96, mean_step_time 1404.54, it/s 0.71 | |
| [0130 17:32:34 @multigpu.py:323] [p0574] step: count(106), step_time 1361.46, mean_step_time 1391.78, it/s 0.72 | |
| [0130 17:32:34 @multigpu.py:323] [p0115] step: count(86), step_time 1395.83, mean_step_time 1404.84, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0427628719144 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:32:35 @multigpu.py:323] [p0574] step: count(107), step_time 1429.53, mean_step_time 1394.17, it/s 0.72 | |
| [0130 17:32:35 @multigpu.py:323] [p0576] step: count(109), step_time 1442.03, mean_step_time 1394.07, it/s 0.72 | |
| [0130 17:32:35 @multigpu.py:323] [p0115] step: count(87), step_time 1353.86, mean_step_time 1394.11, it/s 0.72 | |
| [0130 17:32:37 @multigpu.py:323] [p0576] step: count(110), step_time 1348.97, mean_step_time 1390.72, it/s 0.72 | |
| [0130 17:32:37 @multigpu.py:323] [p0574] step: count(108), step_time 1386.77, mean_step_time 1394.09, it/s 0.72 | |
| [0130 17:32:37 @multigpu.py:323] [p0115] step: count(88), step_time 1412.35, mean_step_time 1394.77, it/s 0.72 | |
| [0130 17:32:38 @multigpu.py:323] [p0576] step: count(111), step_time 1377.92, mean_step_time 1392.78, it/s 0.72 | |
| [0130 17:32:38 @multigpu.py:323] [p0115] step: count(89), step_time 1370.59, mean_step_time 1389.69, it/s 0.72 | |
| [0130 17:32:38 @multigpu.py:323] [p0574] step: count(109), step_time 1411.26, mean_step_time 1397.32, it/s 0.72 | |
| [0130 17:32:39 @multigpu.py:323] [p0576] step: count(112), step_time 1386.66, mean_step_time 1388.37, it/s 0.72 | |
| [0130 17:32:39 @multigpu.py:323] [p0115] step: count(90), step_time 1351.87, mean_step_time 1389.31, it/s 0.72 | |
| [0130 17:32:39 @multigpu.py:323] [p0574] step: count(110), step_time 1452.75, mean_step_time 1400.11, it/s 0.71 | |
| [0130 17:32:41 @multigpu.py:323] [p0115] step: count(91), step_time 1359.65, mean_step_time 1386.26, it/s 0.72 | |
| [0130 17:32:41 @multigpu.py:323] [p0576] step: count(113), step_time 1427.16, mean_step_time 1392.42, it/s 0.72 | |
| [0130 17:32:41 @multigpu.py:323] [p0574] step: count(111), step_time 1353.86, mean_step_time 1398.35, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0445858110984 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:32:42 @multigpu.py:323] [p0576] step: count(114), step_time 1351.34, mean_step_time 1389.52, it/s 0.72 | |
| [0130 17:32:42 @multigpu.py:323] [p0115] step: count(92), step_time 1456.38, mean_step_time 1389.68, it/s 0.72 | |
| [0130 17:32:42 @multigpu.py:323] [p0574] step: count(112), step_time 1408.49, mean_step_time 1399.88, it/s 0.71 | |
| [0130 17:32:44 @multigpu.py:323] [p0115] step: count(93), step_time 1363.97, mean_step_time 1386.36, it/s 0.72 | |
| [0130 17:32:44 @multigpu.py:323] [p0576] step: count(115), step_time 1460.08, mean_step_time 1393.61, it/s 0.72 | |
| [0130 17:32:44 @multigpu.py:323] [p0574] step: count(113), step_time 1432.72, mean_step_time 1401.58, it/s 0.71 | |
| [0130 17:32:45 @multigpu.py:323] [p0115] step: count(94), step_time 1370.03, mean_step_time 1384.39, it/s 0.72 | |
| [0130 17:32:45 @multigpu.py:323] [p0576] step: count(116), step_time 1421.28, mean_step_time 1396.65, it/s 0.72 | |
| [0130 17:32:45 @multigpu.py:323] [p0574] step: count(114), step_time 1433.08, mean_step_time 1399.89, it/s 0.71 | |
| [0130 17:32:46 @multigpu.py:323] [p0115] step: count(95), step_time 1378.92, mean_step_time 1384.09, it/s 0.72 | |
| [0130 17:32:46 @multigpu.py:323] [p0576] step: count(117), step_time 1382.6, mean_step_time 1396.21, it/s 0.72 | |
| [0130 17:32:47 @multigpu.py:323] [p0574] step: count(115), step_time 1430.45, mean_step_time 1399.39, it/s 0.71 | |
| [0130 17:32:48 @multigpu.py:323] [p0115] step: count(96), step_time 1401.86, mean_step_time 1388.08, it/s 0.72 | |
| [0130 17:32:48 @multigpu.py:323] [p0576] step: count(118), step_time 1369.23, mean_step_time 1396.85, it/s 0.72 | |
| [0130 17:32:48 @multigpu.py:323] [p0574] step: count(116), step_time 1421.34, mean_step_time 1400.31, it/s 0.71 | |
| [0130 17:32:49 @multigpu.py:323] [p0115] step: count(97), step_time 1382.13, mean_step_time 1387.51, it/s 0.72 | |
| [0130 17:32:49 @multigpu.py:323] [p0576] step: count(119), step_time 1389.47, mean_step_time 1398.1, it/s 0.72 | |
| [0130 17:32:49 @multigpu.py:323] [p0574] step: count(117), step_time 1401.27, mean_step_time 1400.56, it/s 0.71 | |
| [0130 17:32:50 @multigpu.py:323] [p0115] step: count(98), step_time 1443.72, mean_step_time 1391.58, it/s 0.72 | |
| [0130 17:32:51 @multigpu.py:323] [p0576] step: count(120), step_time 1429.82, mean_step_time 1399.96, it/s 0.71 | |
| [0130 17:32:51 @multigpu.py:323] [p0574] step: count(118), step_time 1454.38, mean_step_time 1404.62, it/s 0.71 | |
| [0130 17:32:52 @multigpu.py:323] [p0115] step: count(99), step_time 1387.8, mean_step_time 1390.31, it/s 0.72 | |
| [0130 17:32:52 @multigpu.py:323] [p0576] step: count(121), step_time 1392.97, mean_step_time 1398.6, it/s 0.72 | |
| [0130 17:32:52 @multigpu.py:323] [p0574] step: count(119), step_time 1527.24, mean_step_time 1411.84, it/s 0.71 | |
| [0130 17:32:53 @multigpu.py:323] [p0115] step: count(100), step_time 1381.26, mean_step_time 1391.43, it/s 0.72 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.047960005535 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.047960005535 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.047960005535 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.0479606341653 , -0.0105593772605 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.0479606341653 , -0.167208120227 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.0479606341653 , -2.29150485992 ##### | |
| ##### Sending to neptune: value_loss : 0.0479606341653 , 1.10711240768 ##### | |
| ##### Sending to neptune: advantage : 0.0479606341653 , 0.000706825871021 ##### | |
| ##### Sending to neptune: pred_reward : 0.0479606341653 , 0.30842140317 ##### | |
| ##### Sending to neptune: max_logit : 0.0479606341653 , 0.18301834166 ##### | |
| [u'loss', -0.010559377260506153, -0.16720812022686005, -2.2915048599243164, 1.1071124076843262, 0.0007068258710205555, 0.30842140316963196, 0.18301834166049957] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0479611127244 , 8962460.69 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0479611127244 , 76.3888871862 ##### | |
| [u'other', 8962460.69, 76.38888718623758] | |
| receiving | |
| [0130 17:32:53 @multigpu.py:323] [p0576] step: count(122), step_time 1392.28, mean_step_time 1397.89, it/s 0.72 | |
| [0130 17:32:54 @multigpu.py:323] [p0574] step: count(120), step_time 1530.71, mean_step_time 1418.91, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0481446746985 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:32:55 @multigpu.py:323] [p0115] step: count(101), step_time 1410.22, mean_step_time 1390.89, it/s 0.72 | |
| [0130 17:32:55 @multigpu.py:323] [p0576] step: count(123), step_time 1348.5, mean_step_time 1396.0, it/s 0.72 | |
| [0130 17:32:55 @multigpu.py:323] [p0574] step: count(121), step_time 1452.44, mean_step_time 1420.65, it/s 0.7 | |
| [0130 17:32:56 @multigpu.py:323] [p0115] step: count(102), step_time 1357.53, mean_step_time 1387.96, it/s 0.72 | |
| [0130 17:32:56 @multigpu.py:323] [p0576] step: count(124), step_time 1363.25, mean_step_time 1395.23, it/s 0.72 | |
| [0130 17:32:57 @multigpu.py:323] [p0574] step: count(122), step_time 1581.99, mean_step_time 1432.64, it/s 0.7 | |
| [0130 17:32:57 @multigpu.py:323] [p0115] step: count(103), step_time 1383.49, mean_step_time 1388.73, it/s 0.72 | |
| [0130 17:32:57 @multigpu.py:323] [p0576] step: count(125), step_time 1443.28, mean_step_time 1398.58, it/s 0.72 | |
| [0130 17:32:58 @multigpu.py:323] [p0574] step: count(123), step_time 1531.42, mean_step_time 1439.55, it/s 0.69 | |
| [0130 17:32:59 @multigpu.py:323] [p0115] step: count(104), step_time 1381.62, mean_step_time 1386.86, it/s 0.72 | |
| [0130 17:32:59 @multigpu.py:323] [p0576] step: count(126), step_time 1381.94, mean_step_time 1397.42, it/s 0.72 | |
| [0130 17:33:00 @multigpu.py:323] [p0574] step: count(124), step_time 1630.48, mean_step_time 1452.51, it/s 0.69 | |
| [0130 17:33:00 @multigpu.py:323] [p0115] step: count(105), step_time 1369.54, mean_step_time 1385.63, it/s 0.72 | |
| [0130 17:33:00 @multigpu.py:323] [p0576] step: count(127), step_time 1395.76, mean_step_time 1397.63, it/s 0.72 | |
| [0130 17:33:02 @multigpu.py:323] [p0115] step: count(106), step_time 1406.69, mean_step_time 1386.17, it/s 0.72 | |
| [0130 17:33:02 @multigpu.py:323] [p0574] step: count(125), step_time 1546.76, mean_step_time 1458.92, it/s 0.69 | |
| [0130 17:33:02 @multigpu.py:323] [p0576] step: count(128), step_time 1399.48, mean_step_time 1395.2, it/s 0.72 | |
| [0130 17:33:03 @multigpu.py:323] [p0576] step: count(129), step_time 1363.1, mean_step_time 1391.25, it/s 0.72 | |
| [0130 17:33:04 @multigpu.py:323] [p0576] step: count(130), step_time 1338.32, mean_step_time 1390.72, it/s 0.72 | |
| [0130 17:33:04 @multigpu.py:323] [p0115] step: count(107), step_time 2796.9, mean_step_time 1458.32, it/s 0.69 | |
| [0130 17:33:04 @multigpu.py:323] [p0574] step: count(126), step_time 2779.41, mean_step_time 1529.82, it/s 0.65 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0510955774784 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:33:06 @multigpu.py:323] [p0115] step: count(108), step_time 1399.23, mean_step_time 1457.67, it/s 0.69 | |
| [0130 17:33:06 @multigpu.py:323] [p0576] step: count(131), step_time 1472.37, mean_step_time 1395.44, it/s 0.72 | |
| [0130 17:33:06 @multigpu.py:323] [p0574] step: count(127), step_time 1573.33, mean_step_time 1537.01, it/s 0.65 | |
| [0130 17:33:07 @multigpu.py:323] [p0115] step: count(109), step_time 1380.55, mean_step_time 1458.17, it/s 0.69 | |
| [0130 17:33:07 @multigpu.py:323] [p0576] step: count(132), step_time 1427.56, mean_step_time 1397.49, it/s 0.72 | |
| [0130 17:33:08 @multigpu.py:323] [p0574] step: count(128), step_time 1619.61, mean_step_time 1548.65, it/s 0.65 | |
| [0130 17:33:09 @multigpu.py:323] [p0115] step: count(110), step_time 1370.05, mean_step_time 1459.08, it/s 0.69 | |
| [0130 17:33:09 @multigpu.py:323] [p0576] step: count(133), step_time 1399.82, mean_step_time 1396.12, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0522849238581 , 1.6 ##### | |
| [u'online', 1.6] | |
| ##### Sending to neptune: active_workers : 0.0522849566407 , 3 ##### | |
| receiving | |
| [0130 17:33:09 @multigpu.py:323] [p0574] step: count(129), step_time 1624.04, mean_step_time 1559.29, it/s 0.64 | |
| [0130 17:33:10 @multigpu.py:323] [p0115] step: count(111), step_time 1376.21, mean_step_time 1459.9, it/s 0.68 | |
| [0130 17:33:10 @multigpu.py:323] [p0576] step: count(134), step_time 1361.15, mean_step_time 1396.61, it/s 0.72 | |
| [0130 17:33:11 @multigpu.py:323] [p0574] step: count(130), step_time 1590.44, mean_step_time 1566.17, it/s 0.64 | |
| [0130 17:33:11 @multigpu.py:323] [p0115] step: count(112), step_time 1365.93, mean_step_time 1455.38, it/s 0.69 | |
| [0130 17:33:11 @multigpu.py:323] [p0576] step: count(135), step_time 1388.74, mean_step_time 1393.05, it/s 0.72 | |
| [0130 17:33:13 @multigpu.py:323] [p0574] step: count(131), step_time 1731.61, mean_step_time 1585.06, it/s 0.63 | |
| [0130 17:33:13 @multigpu.py:323] [p0115] step: count(113), step_time 1383.15, mean_step_time 1456.34, it/s 0.69 | |
| [0130 17:33:13 @multigpu.py:323] [p0576] step: count(136), step_time 1343.67, mean_step_time 1389.17, it/s 0.72 | |
| [0130 17:33:14 @multigpu.py:323] [p0115] step: count(114), step_time 1378.8, mean_step_time 1456.78, it/s 0.69 | |
| [0130 17:33:14 @multigpu.py:323] [p0574] step: count(132), step_time 1601.35, mean_step_time 1594.7, it/s 0.63 | |
| [0130 17:33:14 @multigpu.py:323] [p0576] step: count(137), step_time 1405.46, mean_step_time 1390.31, it/s 0.72 | |
| [0130 17:33:15 @multigpu.py:323] [p0115] step: count(115), step_time 1399.48, mean_step_time 1457.81, it/s 0.69 | |
| [0130 17:33:16 @multigpu.py:323] [p0576] step: count(138), step_time 1388.43, mean_step_time 1391.27, it/s 0.72 | |
| [0130 17:33:16 @multigpu.py:323] [p0574] step: count(133), step_time 1626.81, mean_step_time 1604.41, it/s 0.62 | |
| [0130 17:33:17 @multigpu.py:323] [p0115] step: count(116), step_time 1407.25, mean_step_time 1458.08, it/s 0.69 | |
| [0130 17:33:17 @multigpu.py:323] [p0576] step: count(139), step_time 1346.21, mean_step_time 1389.11, it/s 0.72 | |
| [0130 17:33:17 @multigpu.py:323] [p0574] step: count(134), step_time 1527.52, mean_step_time 1609.13, it/s 0.62 | |
| [0130 17:33:18 @multigpu.py:323] [p0115] step: count(117), step_time 1409.14, mean_step_time 1459.43, it/s 0.69 | |
| [0130 17:33:18 @multigpu.py:323] [p0576] step: count(140), step_time 1372.22, mean_step_time 1386.23, it/s 0.72 | |
| [0130 17:33:19 @multigpu.py:323] [p0574] step: count(135), step_time 1426.88, mean_step_time 1608.95, it/s 0.62 | |
| [0130 17:33:20 @multigpu.py:323] [p0115] step: count(118), step_time 1375.3, mean_step_time 1456.01, it/s 0.69 | |
| [0130 17:33:20 @multigpu.py:323] [p0576] step: count(141), step_time 1366.67, mean_step_time 1384.91, it/s 0.72 | |
| [0130 17:33:20 @multigpu.py:323] [p0574] step: count(136), step_time 1466.03, mean_step_time 1611.19, it/s 0.62 | |
| [0130 17:33:21 @multigpu.py:323] [p0576] step: count(142), step_time 1389.33, mean_step_time 1384.76, it/s 0.72 | |
| [0130 17:33:21 @multigpu.py:323] [p0115] step: count(119), step_time 1424.17, mean_step_time 1457.83, it/s 0.69 | |
| [0130 17:33:22 @multigpu.py:323] [p0574] step: count(137), step_time 1382.97, mean_step_time 1610.27, it/s 0.62 | |
| [0130 17:33:22 @multigpu.py:323] [p0576] step: count(143), step_time 1315.06, mean_step_time 1383.09, it/s 0.72 | |
| [0130 17:33:22 @multigpu.py:323] [p0115] step: count(120), step_time 1378.73, mean_step_time 1457.7, it/s 0.69 | |
| [0130 17:33:23 @multigpu.py:323] [p0574] step: count(138), step_time 1377.13, mean_step_time 1606.41, it/s 0.62 | |
| [0130 17:33:24 @multigpu.py:323] [p0576] step: count(144), step_time 1410.72, mean_step_time 1385.46, it/s 0.72 | |
| [0130 17:33:24 @multigpu.py:323] [p0115] step: count(121), step_time 1428.49, mean_step_time 1458.61, it/s 0.69 | |
| [0130 17:33:24 @multigpu.py:323] [p0574] step: count(139), step_time 1379.38, mean_step_time 1599.02, it/s 0.63 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0566255952252 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0567920777533 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:33:25 @multigpu.py:323] [p0576] step: count(145), step_time 1331.44, mean_step_time 1379.87, it/s 0.72 | |
| [0130 17:33:25 @multigpu.py:323] [p0115] step: count(122), step_time 1338.18, mean_step_time 1457.65, it/s 0.69 | |
| [0130 17:33:26 @multigpu.py:323] [p0574] step: count(140), step_time 1408.26, mean_step_time 1592.89, it/s 0.63 | |
| [0130 17:33:27 @multigpu.py:323] [p0576] step: count(146), step_time 1414.04, mean_step_time 1381.48, it/s 0.72 | |
| [0130 17:33:27 @multigpu.py:323] [p0115] step: count(123), step_time 1422.33, mean_step_time 1459.59, it/s 0.69 | |
| [0130 17:33:27 @multigpu.py:323] [p0574] step: count(141), step_time 1368.26, mean_step_time 1588.68, it/s 0.63 | |
| [0130 17:33:28 @multigpu.py:323] [p0576] step: count(147), step_time 1339.09, mean_step_time 1378.64, it/s 0.73 | |
| [0130 17:33:28 @multigpu.py:323] [p0115] step: count(124), step_time 1376.01, mean_step_time 1459.31, it/s 0.69 | |
| [0130 17:33:29 @multigpu.py:323] [p0574] step: count(142), step_time 1417.7, mean_step_time 1580.47, it/s 0.63 | |
| [0130 17:33:29 @multigpu.py:323] [p0576] step: count(148), step_time 1386.21, mean_step_time 1377.98, it/s 0.73 | |
| [0130 17:33:29 @multigpu.py:323] [p0115] step: count(125), step_time 1357.21, mean_step_time 1458.69, it/s 0.69 | |
| [0130 17:33:30 @multigpu.py:323] [p0574] step: count(143), step_time 1351.42, mean_step_time 1571.47, it/s 0.64 | |
| [0130 17:33:31 @multigpu.py:323] [p0576] step: count(149), step_time 1354.23, mean_step_time 1377.54, it/s 0.73 | |
| [0130 17:33:31 @multigpu.py:323] [p0115] step: count(126), step_time 1383.09, mean_step_time 1457.51, it/s 0.69 | |
| [0130 17:33:31 @multigpu.py:323] [p0574] step: count(144), step_time 1374.31, mean_step_time 1558.66, it/s 0.64 | |
| [0130 17:33:32 @multigpu.py:323] [p0576] step: count(150), step_time 1377.74, mean_step_time 1379.51, it/s 0.72 | |
| [0130 17:33:33 @multigpu.py:323] [p0576] step: count(151), step_time 1383.28, mean_step_time 1375.05, it/s 0.73 | |
| [0130 17:33:33 @multigpu.py:323] [p0574] step: count(145), step_time 2126.63, mean_step_time 1587.65, it/s 0.63 | |
| [0130 17:33:33 @multigpu.py:323] [p0115] step: count(127), step_time 2620.68, mean_step_time 1448.7, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0592391738627 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:33:35 @multigpu.py:323] [p0574] step: count(146), step_time 1375.08, mean_step_time 1517.44, it/s 0.66 | |
| [0130 17:33:35 @multigpu.py:323] [p0576] step: count(152), step_time 1375.58, mean_step_time 1372.45, it/s 0.73 | |
| [0130 17:33:35 @multigpu.py:323] [p0115] step: count(128), step_time 1440.45, mean_step_time 1450.76, it/s 0.69 | |
| [0130 17:33:36 @multigpu.py:323] [p0576] step: count(153), step_time 1407.13, mean_step_time 1372.82, it/s 0.73 | |
| [0130 17:33:36 @multigpu.py:323] [p0115] step: count(129), step_time 1366.56, mean_step_time 1450.06, it/s 0.69 | |
| [0130 17:33:36 @multigpu.py:323] [p0574] step: count(147), step_time 1436.0, mean_step_time 1510.57, it/s 0.66 | |
| [0130 17:33:37 @multigpu.py:323] [p0576] step: count(154), step_time 1322.97, mean_step_time 1370.91, it/s 0.73 | |
| [0130 17:33:38 @multigpu.py:323] [p0574] step: count(148), step_time 1352.32, mean_step_time 1497.21, it/s 0.67 | |
| [0130 17:33:38 @multigpu.py:323] [p0115] step: count(130), step_time 1406.66, mean_step_time 1451.89, it/s 0.69 | |
| [0130 17:33:39 @multigpu.py:323] [p0576] step: count(155), step_time 1419.88, mean_step_time 1372.47, it/s 0.73 | |
| [0130 17:33:39 @multigpu.py:323] [p0574] step: count(149), step_time 1408.69, mean_step_time 1486.44, it/s 0.67 | |
| [0130 17:33:39 @multigpu.py:323] [p0115] step: count(131), step_time 1452.33, mean_step_time 1455.7, it/s 0.69 | |
| [0130 17:33:40 @multigpu.py:323] [p0576] step: count(156), step_time 1347.77, mean_step_time 1372.67, it/s 0.73 | |
| [0130 17:33:40 @multigpu.py:323] [p0574] step: count(150), step_time 1384.36, mean_step_time 1476.13, it/s 0.68 | |
| [0130 17:33:40 @multigpu.py:323] [p0115] step: count(132), step_time 1370.15, mean_step_time 1455.91, it/s 0.69 | |
| [0130 17:33:42 @multigpu.py:323] [p0576] step: count(157), step_time 1357.76, mean_step_time 1370.29, it/s 0.73 | |
| [0130 17:33:42 @multigpu.py:323] [p0574] step: count(151), step_time 1339.38, mean_step_time 1456.52, it/s 0.69 | |
| [0130 17:33:42 @multigpu.py:323] [p0115] step: count(133), step_time 1349.21, mean_step_time 1454.21, it/s 0.69 | |
| [0130 17:33:43 @multigpu.py:323] [p0576] step: count(158), step_time 1393.96, mean_step_time 1370.56, it/s 0.73 | |
| [0130 17:33:43 @multigpu.py:323] [p0574] step: count(152), step_time 1389.87, mean_step_time 1445.95, it/s 0.69 | |
| [0130 17:33:43 @multigpu.py:323] [p0115] step: count(134), step_time 1368.66, mean_step_time 1453.7, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0620354730553 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:33:44 @multigpu.py:323] [p0576] step: count(159), step_time 1365.54, mean_step_time 1371.53, it/s 0.73 | |
| [0130 17:33:44 @multigpu.py:323] [p0574] step: count(153), step_time 1363.79, mean_step_time 1432.8, it/s 0.7 | |
| [0130 17:33:44 @multigpu.py:323] [p0115] step: count(135), step_time 1364.54, mean_step_time 1451.96, it/s 0.69 | |
| [0130 17:33:46 @multigpu.py:323] [p0576] step: count(160), step_time 1351.72, mean_step_time 1370.5, it/s 0.73 | |
| [0130 17:33:46 @multigpu.py:323] [p0574] step: count(154), step_time 1415.48, mean_step_time 1427.2, it/s 0.7 | |
| [0130 17:33:46 @multigpu.py:323] [p0115] step: count(136), step_time 1433.35, mean_step_time 1453.26, it/s 0.69 | |
| [0130 17:33:47 @multigpu.py:323] [p0576] step: count(161), step_time 1397.89, mean_step_time 1372.07, it/s 0.73 | |
| [0130 17:33:47 @multigpu.py:323] [p0574] step: count(155), step_time 1358.27, mean_step_time 1423.77, it/s 0.7 | |
| [0130 17:33:47 @multigpu.py:323] [p0115] step: count(137), step_time 1386.5, mean_step_time 1452.13, it/s 0.69 | |
| [0130 17:33:48 @multigpu.py:323] [p0576] step: count(162), step_time 1338.21, mean_step_time 1369.51, it/s 0.73 | |
| [0130 17:33:49 @multigpu.py:323] [p0574] step: count(156), step_time 1386.11, mean_step_time 1419.77, it/s 0.7 | |
| [0130 17:33:49 @multigpu.py:323] [p0115] step: count(138), step_time 1378.19, mean_step_time 1452.27, it/s 0.69 | |
| [0130 17:33:50 @multigpu.py:323] [p0576] step: count(163), step_time 1400.41, mean_step_time 1373.78, it/s 0.73 | |
| [0130 17:33:50 @multigpu.py:323] [p0574] step: count(157), step_time 1369.1, mean_step_time 1419.08, it/s 0.7 | |
| [0130 17:33:50 @multigpu.py:323] [p0115] step: count(139), step_time 1378.65, mean_step_time 1450.0, it/s 0.69 | |
| [0130 17:33:51 @multigpu.py:323] [p0576] step: count(164), step_time 1331.14, mean_step_time 1369.8, it/s 0.73 | |
| [0130 17:33:51 @multigpu.py:323] [p0574] step: count(158), step_time 1411.4, mean_step_time 1420.79, it/s 0.7 | |
| [0130 17:33:51 @multigpu.py:323] [p0115] step: count(140), step_time 1361.03, mean_step_time 1449.11, it/s 0.69 | |
| [0130 17:33:53 @multigpu.py:323] [p0576] step: count(165), step_time 1387.46, mean_step_time 1372.6, it/s 0.73 | |
| [0130 17:33:53 @multigpu.py:323] [p0115] step: count(141), step_time 1417.65, mean_step_time 1448.57, it/s 0.69 | |
| [0130 17:33:53 @multigpu.py:323] [p0574] step: count(159), step_time 1508.85, mean_step_time 1427.26, it/s 0.7 | |
| [0130 17:33:54 @multigpu.py:323] [p0576] step: count(166), step_time 1382.32, mean_step_time 1371.01, it/s 0.73 | |
| [0130 17:33:54 @multigpu.py:323] [p0115] step: count(142), step_time 1374.83, mean_step_time 1450.4, it/s 0.69 | |
| [0130 17:33:54 @multigpu.py:323] [p0574] step: count(160), step_time 1371.93, mean_step_time 1425.45, it/s 0.7 | |
| [0130 17:33:55 @multigpu.py:323] [p0576] step: count(167), step_time 1386.89, mean_step_time 1373.4, it/s 0.73 | |
| [0130 17:33:56 @multigpu.py:323] [p0115] step: count(143), step_time 1362.51, mean_step_time 1447.41, it/s 0.69 | |
| [0130 17:33:56 @multigpu.py:323] [p0574] step: count(161), step_time 1391.61, mean_step_time 1426.61, it/s 0.7 | |
| [0130 17:33:57 @multigpu.py:323] [p0576] step: count(168), step_time 1337.04, mean_step_time 1370.95, it/s 0.73 | |
| [0130 17:33:57 @multigpu.py:323] [p0574] step: count(162), step_time 1362.97, mean_step_time 1423.88, it/s 0.7 | |
| [0130 17:33:57 @multigpu.py:323] [p0115] step: count(144), step_time 1451.4, mean_step_time 1451.18, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0657542652554 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:33:58 @multigpu.py:323] [p0576] step: count(169), step_time 1383.69, mean_step_time 1372.42, it/s 0.73 | |
| [0130 17:33:58 @multigpu.py:323] [p0574] step: count(163), step_time 1399.8, mean_step_time 1426.3, it/s 0.7 | |
| [0130 17:33:58 @multigpu.py:323] [p0115] step: count(145), step_time 1387.85, mean_step_time 1452.71, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0660777313842 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:33:59 @multigpu.py:323] [p0576] step: count(170), step_time 1392.35, mean_step_time 1373.15, it/s 0.73 | |
| [0130 17:34:00 @multigpu.py:323] [p0115] step: count(146), step_time 1359.71, mean_step_time 1451.55, it/s 0.69 | |
| [0130 17:34:00 @multigpu.py:323] [p0574] step: count(164), step_time 1395.34, mean_step_time 1427.35, it/s 0.7 | |
| [0130 17:34:01 @multigpu.py:323] [p0115] step: count(147), step_time 1373.86, mean_step_time 1389.2, it/s 0.72 | |
| [0130 17:34:01 @multigpu.py:323] [p0574] step: count(165), step_time 1364.25, mean_step_time 1389.23, it/s 0.72 | |
| [0130 17:34:01 @multigpu.py:323] [p0576] step: count(171), step_time 1705.46, mean_step_time 1389.26, it/s 0.72 | |
| [0130 17:34:03 @multigpu.py:323] [p0115] step: count(148), step_time 1351.29, mean_step_time 1384.75, it/s 0.72 | |
| [0130 17:34:03 @multigpu.py:323] [p0576] step: count(172), step_time 1396.38, mean_step_time 1390.3, it/s 0.72 | |
| [0130 17:34:03 @multigpu.py:323] [p0574] step: count(166), step_time 1407.4, mean_step_time 1390.85, it/s 0.72 | |
| [0130 17:34:04 @multigpu.py:323] [p0115] step: count(149), step_time 1391.96, mean_step_time 1386.02, it/s 0.72 | |
| [0130 17:34:04 @multigpu.py:323] [p0574] step: count(167), step_time 1387.59, mean_step_time 1388.43, it/s 0.72 | |
| [0130 17:34:04 @multigpu.py:323] [p0576] step: count(173), step_time 1418.44, mean_step_time 1390.86, it/s 0.72 | |
| [0130 17:34:05 @multigpu.py:323] [p0115] step: count(150), step_time 1357.6, mean_step_time 1383.56, it/s 0.72 | |
| [0130 17:34:05 @multigpu.py:323] [p0574] step: count(168), step_time 1366.03, mean_step_time 1389.11, it/s 0.72 | |
| [0130 17:34:05 @multigpu.py:323] [p0576] step: count(174), step_time 1388.43, mean_step_time 1394.14, it/s 0.72 | |
| [0130 17:34:07 @multigpu.py:323] [p0115] step: count(151), step_time 1400.82, mean_step_time 1380.99, it/s 0.72 | |
| [0130 17:34:07 @multigpu.py:323] [p0574] step: count(169), step_time 1367.17, mean_step_time 1387.04, it/s 0.72 | |
| [0130 17:34:07 @multigpu.py:323] [p0576] step: count(175), step_time 1413.54, mean_step_time 1393.82, it/s 0.72 | |
| [0130 17:34:08 @multigpu.py:323] [p0115] step: count(152), step_time 1361.39, mean_step_time 1380.55, it/s 0.72 | |
| [0130 17:34:08 @multigpu.py:323] [p0574] step: count(170), step_time 1396.26, mean_step_time 1387.63, it/s 0.72 | |
| [0130 17:34:08 @multigpu.py:323] [p0576] step: count(176), step_time 1380.22, mean_step_time 1395.44, it/s 0.72 | |
| [0130 17:34:09 @multigpu.py:323] [p0115] step: count(153), step_time 1385.38, mean_step_time 1382.36, it/s 0.72 | |
| [0130 17:34:09 @multigpu.py:323] [p0574] step: count(171), step_time 1366.67, mean_step_time 1388.99, it/s 0.72 | |
| [0130 17:34:10 @multigpu.py:323] [p0576] step: count(177), step_time 1360.44, mean_step_time 1395.58, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0692923605442 , 0.7 ##### | |
| [u'online', 0.7] | |
| ##### Sending to neptune: active_workers : 0.0692924124665 , 3 ##### | |
| receiving | |
| [0130 17:34:11 @multigpu.py:323] [p0115] step: count(154), step_time 1383.39, mean_step_time 1383.09, it/s 0.72 | |
| [0130 17:34:11 @multigpu.py:323] [p0574] step: count(172), step_time 1391.29, mean_step_time 1389.07, it/s 0.72 | |
| [0130 17:34:11 @multigpu.py:323] [p0576] step: count(178), step_time 1413.78, mean_step_time 1396.57, it/s 0.72 | |
| [0130 17:34:12 @multigpu.py:323] [p0115] step: count(155), step_time 1360.79, mean_step_time 1382.91, it/s 0.72 | |
| [0130 17:34:12 @multigpu.py:323] [p0574] step: count(173), step_time 1400.33, mean_step_time 1390.89, it/s 0.72 | |
| [0130 17:34:12 @multigpu.py:323] [p0576] step: count(179), step_time 1355.16, mean_step_time 1396.05, it/s 0.72 | |
| [0130 17:34:14 @multigpu.py:323] [p0115] step: count(156), step_time 1419.04, mean_step_time 1382.19, it/s 0.72 | |
| [0130 17:34:14 @multigpu.py:323] [p0574] step: count(174), step_time 1371.01, mean_step_time 1388.67, it/s 0.72 | |
| [0130 17:34:14 @multigpu.py:323] [p0576] step: count(180), step_time 1350.09, mean_step_time 1395.97, it/s 0.72 | |
| [0130 17:34:15 @multigpu.py:323] [p0115] step: count(157), step_time 1368.62, mean_step_time 1381.3, it/s 0.72 | |
| [0130 17:34:15 @multigpu.py:323] [p0574] step: count(175), step_time 1396.0, mean_step_time 1390.56, it/s 0.72 | |
| [0130 17:34:15 @multigpu.py:323] [p0576] step: count(181), step_time 1375.06, mean_step_time 1394.83, it/s 0.72 | |
| [0130 17:34:16 @multigpu.py:323] [p0115] step: count(158), step_time 1412.57, mean_step_time 1383.02, it/s 0.72 | |
| [0130 17:34:16 @multigpu.py:323] [p0574] step: count(176), step_time 1350.15, mean_step_time 1388.76, it/s 0.72 | |
| [0130 17:34:16 @multigpu.py:323] [p0576] step: count(182), step_time 1365.4, mean_step_time 1396.19, it/s 0.72 | |
| [0130 17:34:18 @multigpu.py:323] [p0115] step: count(159), step_time 1386.58, mean_step_time 1383.41, it/s 0.72 | |
| [0130 17:34:18 @multigpu.py:323] [p0576] step: count(183), step_time 1362.43, mean_step_time 1394.29, it/s 0.72 | |
| [0130 17:34:18 @multigpu.py:323] [p0574] step: count(177), step_time 1407.9, mean_step_time 1390.7, it/s 0.72 | |
| [0130 17:34:19 @multigpu.py:323] [p0576] step: count(184), step_time 1366.59, mean_step_time 1396.06, it/s 0.72 | |
| [0130 17:34:19 @multigpu.py:323] [p0115] step: count(160), step_time 1374.1, mean_step_time 1384.07, it/s 0.72 | |
| [0130 17:34:19 @multigpu.py:323] [p0574] step: count(178), step_time 1409.13, mean_step_time 1390.58, it/s 0.72 | |
| [0130 17:34:20 @multigpu.py:323] [p0576] step: count(185), step_time 1375.44, mean_step_time 1395.46, it/s 0.72 | |
| [0130 17:34:20 @multigpu.py:323] [p0574] step: count(179), step_time 1321.83, mean_step_time 1381.23, it/s 0.72 | |
| [0130 17:34:21 @multigpu.py:323] [p0115] step: count(161), step_time 1406.45, mean_step_time 1383.51, it/s 0.72 | |
| [0130 17:34:22 @multigpu.py:323] [p0576] step: count(186), step_time 1394.41, mean_step_time 1396.06, it/s 0.72 | |
| [0130 17:34:22 @multigpu.py:323] [p0574] step: count(180), step_time 1405.91, mean_step_time 1382.93, it/s 0.72 | |
| [0130 17:34:22 @multigpu.py:323] [p0115] step: count(162), step_time 1402.49, mean_step_time 1384.89, it/s 0.72 | |
| [0130 17:34:23 @multigpu.py:323] [p0576] step: count(187), step_time 1335.38, mean_step_time 1393.49, it/s 0.72 | |
| [0130 17:34:23 @multigpu.py:323] [p0115] step: count(163), step_time 1373.02, mean_step_time 1385.41, it/s 0.72 | |
| [0130 17:34:23 @multigpu.py:323] [p0574] step: count(181), step_time 1410.48, mean_step_time 1383.88, it/s 0.72 | |
| [0130 17:34:25 @multigpu.py:323] [p0576] step: count(188), step_time 1389.98, mean_step_time 1396.13, it/s 0.72 | |
| [0130 17:34:25 @multigpu.py:323] [p0574] step: count(182), step_time 1353.91, mean_step_time 1383.42, it/s 0.72 | |
| [0130 17:34:25 @multigpu.py:323] [p0115] step: count(164), step_time 1392.64, mean_step_time 1382.48, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0735821829902 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:34:26 @multigpu.py:323] [p0576] step: count(189), step_time 1328.79, mean_step_time 1393.39, it/s 0.72 | |
| [0130 17:34:26 @multigpu.py:323] [p0115] step: count(165), step_time 1361.66, mean_step_time 1381.17, it/s 0.72 | |
| [0130 17:34:26 @multigpu.py:323] [p0574] step: count(183), step_time 1382.06, mean_step_time 1382.54, it/s 0.72 | |
| [0130 17:34:27 @multigpu.py:323] [p0576] step: count(190), step_time 1381.09, mean_step_time 1392.83, it/s 0.72 | |
| [0130 17:34:27 @multigpu.py:323] [p0574] step: count(184), step_time 1377.0, mean_step_time 1381.62, it/s 0.72 | |
| [0130 17:34:27 @multigpu.py:323] [p0115] step: count(166), step_time 1383.95, mean_step_time 1382.38, it/s 0.72 | |
| [0130 17:34:29 @multigpu.py:323] [p0115] step: count(167), step_time 1391.89, mean_step_time 1383.28, it/s 0.72 | |
| [0130 17:34:29 @multigpu.py:323] [p0574] step: count(185), step_time 1398.33, mean_step_time 1383.32, it/s 0.72 | |
| [0130 17:34:29 @multigpu.py:323] [p0576] step: count(191), step_time 1513.07, mean_step_time 1383.21, it/s 0.72 | |
| [0130 17:34:30 @multigpu.py:323] [p0115] step: count(168), step_time 1358.74, mean_step_time 1383.65, it/s 0.72 | |
| [0130 17:34:30 @multigpu.py:323] [p0574] step: count(186), step_time 1370.96, mean_step_time 1381.5, it/s 0.72 | |
| [0130 17:34:30 @multigpu.py:323] [p0576] step: count(192), step_time 1385.18, mean_step_time 1382.65, it/s 0.72 | |
| [0130 17:34:32 @multigpu.py:323] [p0576] step: count(193), step_time 1358.67, mean_step_time 1379.66, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0752706305186 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 17:34:32 @multigpu.py:323] [p0574] step: count(187), step_time 1392.0, mean_step_time 1381.72, it/s 0.72 | |
| [0130 17:34:32 @multigpu.py:323] [p0115] step: count(169), step_time 1417.02, mean_step_time 1384.91, it/s 0.72 | |
| [0130 17:34:33 @multigpu.py:323] [p0115] step: count(170), step_time 1332.3, mean_step_time 1383.64, it/s 0.72 | |
| [0130 17:34:33 @multigpu.py:323] [p0574] step: count(188), step_time 1353.15, mean_step_time 1381.08, it/s 0.72 | |
| [0130 17:34:33 @multigpu.py:323] [p0576] step: count(194), step_time 1381.28, mean_step_time 1379.3, it/s 0.73 | |
| [0130 17:34:34 @multigpu.py:323] [p0576] step: count(195), step_time 1359.54, mean_step_time 1376.6, it/s 0.73 | |
| [0130 17:34:34 @multigpu.py:323] [p0574] step: count(189), step_time 1392.51, mean_step_time 1382.34, it/s 0.72 | |
| [0130 17:34:34 @multigpu.py:323] [p0115] step: count(171), step_time 1418.38, mean_step_time 1384.52, it/s 0.72 | |
| [0130 17:34:36 @multigpu.py:323] [p0576] step: count(196), step_time 1368.99, mean_step_time 1376.04, it/s 0.73 | |
| [0130 17:34:36 @multigpu.py:323] [p0574] step: count(190), step_time 1355.72, mean_step_time 1380.32, it/s 0.72 | |
| [0130 17:34:36 @multigpu.py:323] [p0115] step: count(172), step_time 1368.87, mean_step_time 1384.89, it/s 0.72 | |
| [0130 17:34:37 @multigpu.py:323] [p0576] step: count(197), step_time 1351.55, mean_step_time 1375.59, it/s 0.73 | |
| [0130 17:34:37 @multigpu.py:323] [p0574] step: count(191), step_time 1382.27, mean_step_time 1381.1, it/s 0.72 | |
| [0130 17:34:37 @multigpu.py:323] [p0115] step: count(173), step_time 1384.74, mean_step_time 1384.86, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0769521447023 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:34:38 @multigpu.py:323] [p0576] step: count(198), step_time 1382.29, mean_step_time 1374.02, it/s 0.73 | |
| [0130 17:34:38 @multigpu.py:323] [p0574] step: count(192), step_time 1367.78, mean_step_time 1379.92, it/s 0.72 | |
| [0130 17:34:38 @multigpu.py:323] [p0115] step: count(174), step_time 1381.63, mean_step_time 1384.77, it/s 0.72 | |
| [0130 17:34:40 @multigpu.py:323] [p0576] step: count(199), step_time 1398.43, mean_step_time 1376.18, it/s 0.73 | |
| [0130 17:34:40 @multigpu.py:323] [p0574] step: count(193), step_time 1392.39, mean_step_time 1379.52, it/s 0.72 | |
| [0130 17:34:40 @multigpu.py:323] [p0115] step: count(175), step_time 1386.47, mean_step_time 1386.06, it/s 0.72 | |
| [0130 17:34:41 @multigpu.py:323] [p0576] step: count(200), step_time 1342.74, mean_step_time 1375.81, it/s 0.73 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.0779323197073 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.0779323197073 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.0779323197073 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.0779328760836 , -0.00780312530696 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.0779328760836 , 0.118199490011 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.0779328760836 , -2.29004001617 ##### | |
| ##### Sending to neptune: value_loss : 0.0779328760836 , 1.17304027081 ##### | |
| ##### Sending to neptune: advantage : 0.0779328760836 , -0.000583324115723 ##### | |
| ##### Sending to neptune: pred_reward : 0.0779328760836 , 0.359293758869 ##### | |
| ##### Sending to neptune: max_logit : 0.0779328760836 , 0.189353853464 ##### | |
| [u'loss', -0.007803125306963921, 0.11819949001073837, -2.2900400161743164, 1.1730402708053589, -0.0005833241157233715, 0.35929375886917114, 0.1893538534641266] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0779333224561 , 8862300.92 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0779333224561 , 92.2689357559 ##### | |
| [u'other', 8862300.92, 92.26893575586253] | |
| receiving | |
| [0130 17:34:41 @multigpu.py:323] [p0574] step: count(194), step_time 1399.89, mean_step_time 1380.97, it/s 0.72 | |
| [0130 17:34:41 @multigpu.py:323] [p0115] step: count(176), step_time 1364.66, mean_step_time 1383.34, it/s 0.72 | |
| [0130 17:34:43 @multigpu.py:323] [p0576] step: count(201), step_time 1375.37, mean_step_time 1375.83, it/s 0.73 | |
| [0130 17:34:43 @multigpu.py:323] [p0115] step: count(177), step_time 1421.45, mean_step_time 1385.98, it/s 0.72 | |
| [0130 17:34:43 @multigpu.py:323] [p0574] step: count(195), step_time 1460.79, mean_step_time 1384.21, it/s 0.72 | |
| [0130 17:34:44 @multigpu.py:323] [p0576] step: count(202), step_time 1389.25, mean_step_time 1377.02, it/s 0.73 | |
| [0130 17:34:44 @multigpu.py:323] [p0115] step: count(178), step_time 1326.35, mean_step_time 1381.67, it/s 0.72 | |
| [0130 17:34:44 @multigpu.py:323] [p0574] step: count(196), step_time 1371.53, mean_step_time 1385.28, it/s 0.72 | |
| [0130 17:34:45 @multigpu.py:323] [p0576] step: count(203), step_time 1359.61, mean_step_time 1376.88, it/s 0.73 | |
| [0130 17:34:45 @multigpu.py:323] [p0115] step: count(179), step_time 1378.42, mean_step_time 1381.26, it/s 0.72 | |
| [0130 17:34:46 @multigpu.py:323] [p0574] step: count(197), step_time 1484.98, mean_step_time 1389.13, it/s 0.72 | |
| [0130 17:34:47 @multigpu.py:323] [p0576] step: count(204), step_time 1380.84, mean_step_time 1377.59, it/s 0.73 | |
| [0130 17:34:47 @multigpu.py:323] [p0115] step: count(180), step_time 1379.61, mean_step_time 1381.54, it/s 0.72 | |
| [0130 17:34:47 @multigpu.py:323] [p0574] step: count(198), step_time 1512.61, mean_step_time 1394.3, it/s 0.72 | |
| [0130 17:34:48 @multigpu.py:323] [p0576] step: count(205), step_time 1359.08, mean_step_time 1376.78, it/s 0.73 | |
| [0130 17:34:48 @multigpu.py:323] [p0115] step: count(181), step_time 1355.4, mean_step_time 1378.99, it/s 0.73 | |
| [0130 17:34:48 @multigpu.py:323] [p0574] step: count(199), step_time 1361.12, mean_step_time 1396.27, it/s 0.72 | |
| [0130 17:34:49 @multigpu.py:323] [p0576] step: count(206), step_time 1383.91, mean_step_time 1376.25, it/s 0.73 | |
| [0130 17:34:50 @multigpu.py:323] [p0115] step: count(182), step_time 1399.66, mean_step_time 1378.84, it/s 0.73 | |
| [0130 17:34:50 @multigpu.py:323] [p0574] step: count(200), step_time 1434.78, mean_step_time 1397.71, it/s 0.72 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.0803513783216 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.0803513783216 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.0803513783216 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| sending to address tcp://p0112:61216 | |
| receiving | |
| ##### Sending to neptune: cost : 0.080351874431 , -0.00596463959664 ##### | |
| ##### Sending to neptune: policy_loss : 0.080351874431 , 0.299598842859 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.080351874431 , -2.28987956047 ##### | |
| ##### Sending to neptune: value_loss : 0.080351874431 , 1.22680675983 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: advantage : 0.080351874431 , -0.00136904569808 ##### | |
| ##### Sending to neptune: pred_reward : 0.080351874431 , 0.362240970135 ##### | |
| ##### Sending to neptune: max_logit : 0.080351874431 , 0.190133050084 ##### | |
| [u'loss', -0.005964639596641064, 0.2995988428592682, -2.289879560470581, 1.2268067598342896, -0.0013690456980839372, 0.3622409701347351, 0.19013305008411407] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0803523513344 , 8882302.82 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0803523513344 , 88.9797109993 ##### | |
| [u'other', 8882302.82, 88.97971099925284] | |
| receiving | |
| [0130 17:34:51 @multigpu.py:323] [p0576] step: count(207), step_time 1366.96, mean_step_time 1377.83, it/s 0.73 | |
| [0130 17:34:51 @multigpu.py:323] [p0115] step: count(183), step_time 1397.31, mean_step_time 1380.06, it/s 0.72 | |
| [0130 17:34:51 @multigpu.py:323] [p0574] step: count(201), step_time 1405.29, mean_step_time 1397.45, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0809522196982 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:34:52 @multigpu.py:323] [p0576] step: count(208), step_time 1430.94, mean_step_time 1379.88, it/s 0.72 | |
| [0130 17:34:52 @multigpu.py:323] [p0115] step: count(184), step_time 1378.93, mean_step_time 1379.37, it/s 0.72 | |
| [0130 17:34:53 @multigpu.py:323] [p0574] step: count(202), step_time 1402.51, mean_step_time 1399.88, it/s 0.71 | |
| [0130 17:34:54 @multigpu.py:323] [p0576] step: count(209), step_time 1332.2, mean_step_time 1380.05, it/s 0.72 | |
| [0130 17:34:54 @multigpu.py:323] [p0115] step: count(185), step_time 1397.63, mean_step_time 1381.17, it/s 0.72 | |
| [0130 17:34:54 @multigpu.py:323] [p0574] step: count(203), step_time 1388.81, mean_step_time 1400.22, it/s 0.71 | |
| [0130 17:34:55 @multigpu.py:323] [p0576] step: count(210), step_time 1379.6, mean_step_time 1379.97, it/s 0.72 | |
| [0130 17:34:55 @multigpu.py:323] [p0115] step: count(186), step_time 1369.04, mean_step_time 1380.43, it/s 0.72 | |
| [0130 17:34:55 @multigpu.py:323] [p0574] step: count(204), step_time 1403.95, mean_step_time 1401.57, it/s 0.71 | |
| [0130 17:34:57 @multigpu.py:323] [p0574] step: count(205), step_time 1470.88, mean_step_time 1405.2, it/s 0.71 | |
| [0130 17:34:57 @multigpu.py:323] [p0115] step: count(187), step_time 1890.06, mean_step_time 1405.33, it/s 0.71 | |
| [0130 17:34:57 @multigpu.py:323] [p0576] step: count(211), step_time 2018.29, mean_step_time 1405.24, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0823914271593 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:34:58 @multigpu.py:323] [p0576] step: count(212), step_time 1364.15, mean_step_time 1404.18, it/s 0.71 | |
| [0130 17:34:58 @multigpu.py:323] [p0115] step: count(188), step_time 1385.2, mean_step_time 1406.66, it/s 0.71 | |
| [0130 17:34:58 @multigpu.py:323] [p0574] step: count(206), step_time 1431.93, mean_step_time 1408.24, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0828070782953 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:35:00 @multigpu.py:323] [p0576] step: count(213), step_time 1372.01, mean_step_time 1404.85, it/s 0.71 | |
| [0130 17:35:00 @multigpu.py:323] [p0115] step: count(189), step_time 1385.77, mean_step_time 1405.09, it/s 0.71 | |
| [0130 17:35:00 @multigpu.py:323] [p0574] step: count(207), step_time 1425.89, mean_step_time 1409.94, it/s 0.71 | |
| [0130 17:35:01 @multigpu.py:323] [p0115] step: count(190), step_time 1344.78, mean_step_time 1405.72, it/s 0.71 | |
| [0130 17:35:01 @multigpu.py:323] [p0576] step: count(214), step_time 1387.69, mean_step_time 1405.17, it/s 0.71 | |
| [0130 17:35:01 @multigpu.py:323] [p0574] step: count(208), step_time 1364.84, mean_step_time 1410.52, it/s 0.71 | |
| [0130 17:35:02 @multigpu.py:323] [p0576] step: count(215), step_time 1356.17, mean_step_time 1405.0, it/s 0.71 | |
| [0130 17:35:02 @multigpu.py:323] [p0115] step: count(191), step_time 1402.47, mean_step_time 1404.92, it/s 0.71 | |
| [0130 17:35:03 @multigpu.py:323] [p0574] step: count(209), step_time 1413.0, mean_step_time 1411.55, it/s 0.71 | |
| [0130 17:35:04 @multigpu.py:323] [p0576] step: count(216), step_time 1396.33, mean_step_time 1406.37, it/s 0.71 | |
| [0130 17:35:04 @multigpu.py:323] [p0115] step: count(192), step_time 1363.41, mean_step_time 1404.65, it/s 0.71 | |
| [0130 17:35:04 @multigpu.py:323] [p0574] step: count(210), step_time 1400.8, mean_step_time 1413.8, it/s 0.71 | |
| [0130 17:35:05 @multigpu.py:323] [p0576] step: count(217), step_time 1388.98, mean_step_time 1408.24, it/s 0.71 | |
| [0130 17:35:05 @multigpu.py:323] [p0115] step: count(193), step_time 1392.09, mean_step_time 1405.02, it/s 0.71 | |
| [0130 17:35:05 @multigpu.py:323] [p0574] step: count(211), step_time 1408.62, mean_step_time 1415.12, it/s 0.71 | |
| [0130 17:35:07 @multigpu.py:323] [p0576] step: count(218), step_time 1369.08, mean_step_time 1407.58, it/s 0.71 | |
| [0130 17:35:07 @multigpu.py:323] [p0115] step: count(194), step_time 1404.0, mean_step_time 1406.14, it/s 0.71 | |
| [0130 17:35:07 @multigpu.py:323] [p0574] step: count(212), step_time 1398.96, mean_step_time 1416.68, it/s 0.71 | |
| [0130 17:35:08 @multigpu.py:323] [p0115] step: count(195), step_time 1346.86, mean_step_time 1404.16, it/s 0.71 | |
| [0130 17:35:08 @multigpu.py:323] [p0576] step: count(219), step_time 1433.47, mean_step_time 1409.33, it/s 0.71 | |
| [0130 17:35:08 @multigpu.py:323] [p0574] step: count(213), step_time 1362.13, mean_step_time 1415.17, it/s 0.71 | |
| [0130 17:35:09 @multigpu.py:323] [p0115] step: count(196), step_time 1389.49, mean_step_time 1405.4, it/s 0.71 | |
| [0130 17:35:09 @multigpu.py:323] [p0576] step: count(220), step_time 1391.26, mean_step_time 1411.76, it/s 0.71 | |
| [0130 17:35:10 @multigpu.py:323] [p0574] step: count(214), step_time 1386.5, mean_step_time 1414.5, it/s 0.71 | |
| [0130 17:35:11 @multigpu.py:323] [p0576] step: count(221), step_time 1364.82, mean_step_time 1411.23, it/s 0.71 | |
| [0130 17:35:11 @multigpu.py:323] [p0115] step: count(197), step_time 1424.18, mean_step_time 1405.53, it/s 0.71 | |
| [0130 17:35:11 @multigpu.py:323] [p0574] step: count(215), step_time 1408.09, mean_step_time 1411.86, it/s 0.71 | |
| [0130 17:35:12 @multigpu.py:323] [p0576] step: count(222), step_time 1351.14, mean_step_time 1409.33, it/s 0.71 | |
| [0130 17:35:12 @multigpu.py:323] [p0115] step: count(198), step_time 1369.54, mean_step_time 1407.69, it/s 0.71 | |
| [0130 17:35:12 @multigpu.py:323] [p0574] step: count(216), step_time 1432.83, mean_step_time 1414.93, it/s 0.71 | |
| [0130 17:35:14 @multigpu.py:323] [p0576] step: count(223), step_time 1396.41, mean_step_time 1411.17, it/s 0.71 | |
| [0130 17:35:14 @multigpu.py:323] [p0115] step: count(199), step_time 1409.2, mean_step_time 1409.23, it/s 0.71 | |
| [0130 17:35:14 @multigpu.py:323] [p0574] step: count(217), step_time 1347.35, mean_step_time 1408.04, it/s 0.71 | |
| [0130 17:35:15 @multigpu.py:323] [p0115] step: count(200), step_time 1346.51, mean_step_time 1407.58, it/s 0.71 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.0873081782791 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.0873081782791 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.0873081782791 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.0873083741135 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.0873086949852 , -0.0134393870831 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.0873086949852 , -0.510861754417 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.0873086949852 , -2.28939032555 ##### | |
| ##### Sending to neptune: value_loss : 0.0873086949852 , 1.08001065254 ##### | |
| ##### Sending to neptune: advantage : 0.0873086949852 , 0.00229038670659 ##### | |
| ##### Sending to neptune: pred_reward : 0.0873086949852 , 0.375006139278 ##### | |
| ##### Sending to neptune: max_logit : 0.0873086949852 , 0.191750437021 ##### | |
| [u'loss', -0.013439387083053589, -0.5108617544174194, -2.2893903255462646, 1.0800106525421143, 0.0022903867065906525, 0.37500613927841187, 0.1917504370212555] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.0873091855314 , 8887920.51 ##### | |
| ##### Sending to neptune: dp_per_s : 0.0873091855314 , 90.4954598592 ##### | |
| [u'other', 8887920.51, 90.49545985924865] | |
| receiving | |
| [0130 17:35:15 @multigpu.py:323] [p0576] step: count(224), step_time 1399.52, mean_step_time 1412.1, it/s 0.71 | |
| [0130 17:35:15 @multigpu.py:323] [p0574] step: count(218), step_time 1376.75, mean_step_time 1401.25, it/s 0.71 | |
| [0130 17:35:16 @multigpu.py:323] [p0115] step: count(201), step_time 1388.62, mean_step_time 1409.24, it/s 0.71 | |
| [0130 17:35:16 @multigpu.py:323] [p0576] step: count(225), step_time 1385.96, mean_step_time 1413.44, it/s 0.71 | |
| [0130 17:35:16 @multigpu.py:323] [p0574] step: count(219), step_time 1376.69, mean_step_time 1402.03, it/s 0.71 | |
| [0130 17:35:18 @multigpu.py:323] [p0115] step: count(202), step_time 1362.17, mean_step_time 1407.36, it/s 0.71 | |
| [0130 17:35:18 @multigpu.py:323] [p0576] step: count(226), step_time 1362.54, mean_step_time 1412.38, it/s 0.71 | |
| [0130 17:35:18 @multigpu.py:323] [p0574] step: count(220), step_time 1402.37, mean_step_time 1400.41, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0882514672147 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:35:19 @multigpu.py:323] [p0576] step: count(227), step_time 1344.1, mean_step_time 1411.23, it/s 0.71 | |
| [0130 17:35:19 @multigpu.py:323] [p0115] step: count(203), step_time 1401.89, mean_step_time 1407.59, it/s 0.71 | |
| [0130 17:35:19 @multigpu.py:323] [p0574] step: count(221), step_time 1395.37, mean_step_time 1399.91, it/s 0.71 | |
| [0130 17:35:20 @multigpu.py:323] [p0576] step: count(228), step_time 1371.69, mean_step_time 1408.27, it/s 0.71 | |
| [0130 17:35:20 @multigpu.py:323] [p0115] step: count(204), step_time 1370.23, mean_step_time 1407.16, it/s 0.71 | |
| [0130 17:35:21 @multigpu.py:323] [p0574] step: count(222), step_time 1453.52, mean_step_time 1402.46, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0889305941264 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:35:22 @multigpu.py:323] [p0576] step: count(229), step_time 1400.69, mean_step_time 1411.7, it/s 0.71 | |
| [0130 17:35:22 @multigpu.py:323] [p0115] step: count(205), step_time 1375.32, mean_step_time 1406.04, it/s 0.71 | |
| [0130 17:35:22 @multigpu.py:323] [p0574] step: count(223), step_time 1446.75, mean_step_time 1405.36, it/s 0.71 | |
| [0130 17:35:23 @multigpu.py:323] [p0576] step: count(230), step_time 1384.77, mean_step_time 1411.95, it/s 0.71 | |
| [0130 17:35:23 @multigpu.py:323] [p0115] step: count(206), step_time 1368.15, mean_step_time 1406.0, it/s 0.71 | |
| [0130 17:35:24 @multigpu.py:323] [p0574] step: count(224), step_time 1409.12, mean_step_time 1405.62, it/s 0.71 | |
| [0130 17:35:25 @multigpu.py:323] [p0574] step: count(225), step_time 1367.4, mean_step_time 1400.45, it/s 0.71 | |
| [0130 17:35:25 @multigpu.py:323] [p0115] step: count(207), step_time 1781.69, mean_step_time 1400.58, it/s 0.71 | |
| [0130 17:35:25 @multigpu.py:323] [p0576] step: count(231), step_time 1789.81, mean_step_time 1400.53, it/s 0.71 | |
| [0130 17:35:26 @multigpu.py:323] [p0576] step: count(232), step_time 1419.14, mean_step_time 1403.28, it/s 0.71 | |
| [0130 17:35:26 @multigpu.py:323] [p0574] step: count(226), step_time 1427.12, mean_step_time 1400.2, it/s 0.71 | |
| [0130 17:35:26 @multigpu.py:323] [p0115] step: count(208), step_time 1432.61, mean_step_time 1402.95, it/s 0.71 | |
| [0130 17:35:28 @multigpu.py:323] [p0574] step: count(227), step_time 1343.76, mean_step_time 1396.1, it/s 0.72 | |
| [0130 17:35:28 @multigpu.py:323] [p0576] step: count(233), step_time 1364.21, mean_step_time 1402.89, it/s 0.71 | |
| [0130 17:35:28 @multigpu.py:323] [p0115] step: count(209), step_time 1398.28, mean_step_time 1403.58, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0910588008165 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:35:29 @multigpu.py:323] [p0574] step: count(228), step_time 1385.26, mean_step_time 1397.12, it/s 0.72 | |
| [0130 17:35:29 @multigpu.py:323] [p0576] step: count(234), step_time 1381.8, mean_step_time 1402.59, it/s 0.71 | |
| [0130 17:35:29 @multigpu.py:323] [p0115] step: count(210), step_time 1369.82, mean_step_time 1404.83, it/s 0.71 | |
| [0130 17:35:30 @multigpu.py:323] [p0574] step: count(229), step_time 1344.25, mean_step_time 1393.68, it/s 0.72 | |
| [0130 17:35:31 @multigpu.py:323] [p0576] step: count(235), step_time 1426.86, mean_step_time 1406.13, it/s 0.71 | |
| [0130 17:35:31 @multigpu.py:323] [p0115] step: count(211), step_time 1409.13, mean_step_time 1405.16, it/s 0.71 | |
| [0130 17:35:32 @multigpu.py:323] [p0574] step: count(230), step_time 1422.48, mean_step_time 1394.77, it/s 0.72 | |
| [0130 17:35:32 @multigpu.py:323] [p0115] step: count(212), step_time 1342.63, mean_step_time 1404.12, it/s 0.71 | |
| [0130 17:35:32 @multigpu.py:323] [p0576] step: count(236), step_time 1360.8, mean_step_time 1404.35, it/s 0.71 | |
| [0130 17:35:33 @multigpu.py:323] [p0574] step: count(231), step_time 1366.62, mean_step_time 1392.67, it/s 0.72 | |
| [0130 17:35:33 @multigpu.py:323] [p0576] step: count(237), step_time 1404.87, mean_step_time 1405.15, it/s 0.71 | |
| [0130 17:35:33 @multigpu.py:323] [p0115] step: count(213), step_time 1440.57, mean_step_time 1406.55, it/s 0.71 | |
| [0130 17:35:35 @multigpu.py:323] [p0576] step: count(238), step_time 1372.38, mean_step_time 1405.31, it/s 0.71 | |
| [0130 17:35:35 @multigpu.py:323] [p0574] step: count(232), step_time 1442.58, mean_step_time 1394.85, it/s 0.72 | |
| [0130 17:35:35 @multigpu.py:323] [p0115] step: count(214), step_time 1366.34, mean_step_time 1404.66, it/s 0.71 | |
| [0130 17:35:36 @multigpu.py:323] [p0576] step: count(239), step_time 1396.97, mean_step_time 1403.49, it/s 0.71 | |
| [0130 17:35:36 @multigpu.py:323] [p0115] step: count(215), step_time 1435.04, mean_step_time 1409.07, it/s 0.71 | |
| [0130 17:35:36 @multigpu.py:323] [p0574] step: count(233), step_time 1562.81, mean_step_time 1404.88, it/s 0.71 | |
| [0130 17:35:37 @multigpu.py:323] [p0576] step: count(240), step_time 1401.66, mean_step_time 1404.01, it/s 0.71 | |
| [0130 17:35:38 @multigpu.py:323] [p0115] step: count(216), step_time 1426.44, mean_step_time 1410.92, it/s 0.71 | |
| [0130 17:35:38 @multigpu.py:323] [p0574] step: count(234), step_time 1530.0, mean_step_time 1412.06, it/s 0.71 | |
| [0130 17:35:39 @multigpu.py:323] [p0576] step: count(241), step_time 1389.36, mean_step_time 1405.23, it/s 0.71 | |
| [0130 17:35:39 @multigpu.py:323] [p0115] step: count(217), step_time 1348.05, mean_step_time 1407.11, it/s 0.71 | |
| [0130 17:35:39 @multigpu.py:323] [p0574] step: count(235), step_time 1505.13, mean_step_time 1416.91, it/s 0.71 | |
| [0130 17:35:40 @multigpu.py:323] [p0576] step: count(242), step_time 1439.39, mean_step_time 1409.65, it/s 0.71 | |
| [0130 17:35:40 @multigpu.py:323] [p0115] step: count(218), step_time 1401.83, mean_step_time 1408.73, it/s 0.71 | |
| [0130 17:35:41 @multigpu.py:323] [p0574] step: count(236), step_time 1410.48, mean_step_time 1415.79, it/s 0.71 | |
| [0130 17:35:42 @multigpu.py:323] [p0576] step: count(243), step_time 1347.01, mean_step_time 1407.18, it/s 0.71 | |
| [0130 17:35:42 @multigpu.py:323] [p0115] step: count(219), step_time 1380.97, mean_step_time 1407.31, it/s 0.71 | |
| [0130 17:35:42 @multigpu.py:323] [p0574] step: count(237), step_time 1423.83, mean_step_time 1419.62, it/s 0.7 | |
| [0130 17:35:43 @multigpu.py:323] [p0576] step: count(244), step_time 1387.0, mean_step_time 1406.55, it/s 0.71 | |
| [0130 17:35:43 @multigpu.py:323] [p0115] step: count(220), step_time 1371.53, mean_step_time 1408.57, it/s 0.71 | |
| [0130 17:35:44 @multigpu.py:323] [p0574] step: count(238), step_time 1374.06, mean_step_time 1419.48, it/s 0.7 | |
| [0130 17:35:44 @multigpu.py:323] [p0576] step: count(245), step_time 1384.53, mean_step_time 1406.48, it/s 0.71 | |
| [0130 17:35:44 @multigpu.py:323] [p0115] step: count(221), step_time 1386.56, mean_step_time 1408.46, it/s 0.71 | |
| [0130 17:35:45 @multigpu.py:323] [p0574] step: count(239), step_time 1363.55, mean_step_time 1418.82, it/s 0.7 | |
| [0130 17:35:46 @multigpu.py:323] [p0576] step: count(246), step_time 1337.92, mean_step_time 1405.25, it/s 0.71 | |
| [0130 17:35:46 @multigpu.py:323] [p0115] step: count(222), step_time 1363.66, mean_step_time 1408.54, it/s 0.71 | |
| [0130 17:35:46 @multigpu.py:323] [p0574] step: count(240), step_time 1375.6, mean_step_time 1417.48, it/s 0.71 | |
| [0130 17:35:47 @multigpu.py:323] [p0576] step: count(247), step_time 1425.29, mean_step_time 1409.31, it/s 0.71 | |
| [0130 17:35:47 @multigpu.py:323] [p0115] step: count(223), step_time 1375.84, mean_step_time 1407.23, it/s 0.71 | |
| [0130 17:35:48 @multigpu.py:323] [p0574] step: count(241), step_time 1392.03, mean_step_time 1417.32, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0964728083213 , 0.7 ##### | |
| [u'online', 0.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0965966761112 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 17:35:49 @multigpu.py:323] [p0576] step: count(248), step_time 1370.39, mean_step_time 1409.24, it/s 0.71 | |
| [0130 17:35:49 @multigpu.py:323] [p0115] step: count(224), step_time 1380.98, mean_step_time 1407.77, it/s 0.71 | |
| [0130 17:35:49 @multigpu.py:323] [p0574] step: count(242), step_time 1367.97, mean_step_time 1413.04, it/s 0.71 | |
| [0130 17:35:50 @multigpu.py:323] [p0576] step: count(249), step_time 1371.62, mean_step_time 1407.79, it/s 0.71 | |
| [0130 17:35:50 @multigpu.py:323] [p0115] step: count(225), step_time 1399.34, mean_step_time 1408.97, it/s 0.71 | |
| [0130 17:35:50 @multigpu.py:323] [p0574] step: count(243), step_time 1377.29, mean_step_time 1409.57, it/s 0.71 | |
| [0130 17:35:51 @multigpu.py:323] [p0576] step: count(250), step_time 1401.31, mean_step_time 1408.62, it/s 0.71 | |
| [0130 17:35:51 @multigpu.py:323] [p0115] step: count(226), step_time 1427.54, mean_step_time 1411.94, it/s 0.71 | |
| [0130 17:35:52 @multigpu.py:323] [p0574] step: count(244), step_time 1402.78, mean_step_time 1409.25, it/s 0.71 | |
| [0130 17:35:53 @multigpu.py:323] [p0574] step: count(245), step_time 1371.89, mean_step_time 1409.47, it/s 0.71 | |
| [0130 17:35:53 @multigpu.py:323] [p0576] step: count(251), step_time 1805.83, mean_step_time 1409.42, it/s 0.71 | |
| [0130 17:35:53 @multigpu.py:323] [p0115] step: count(227), step_time 1732.07, mean_step_time 1409.46, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.0980394785934 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:35:55 @multigpu.py:323] [p0115] step: count(228), step_time 1372.59, mean_step_time 1406.46, it/s 0.71 | |
| [0130 17:35:55 @multigpu.py:323] [p0574] step: count(246), step_time 1373.57, mean_step_time 1406.8, it/s 0.71 | |
| [0130 17:35:55 @multigpu.py:323] [p0576] step: count(252), step_time 1411.17, mean_step_time 1409.02, it/s 0.71 | |
| [0130 17:35:56 @multigpu.py:323] [p0115] step: count(229), step_time 1372.77, mean_step_time 1405.18, it/s 0.71 | |
| [0130 17:35:56 @multigpu.py:323] [p0574] step: count(247), step_time 1372.83, mean_step_time 1408.25, it/s 0.71 | |
| [0130 17:35:56 @multigpu.py:323] [p0576] step: count(253), step_time 1386.74, mean_step_time 1410.14, it/s 0.71 | |
| [0130 17:35:57 @multigpu.py:323] [p0115] step: count(230), step_time 1386.84, mean_step_time 1406.04, it/s 0.71 | |
| [0130 17:35:57 @multigpu.py:323] [p0574] step: count(248), step_time 1412.52, mean_step_time 1409.61, it/s 0.71 | |
| [0130 17:35:57 @multigpu.py:323] [p0576] step: count(254), step_time 1386.09, mean_step_time 1410.36, it/s 0.71 | |
| [0130 17:35:59 @multigpu.py:323] [p0574] step: count(249), step_time 1337.01, mean_step_time 1409.25, it/s 0.71 | |
| [0130 17:35:59 @multigpu.py:323] [p0115] step: count(231), step_time 1365.18, mean_step_time 1403.84, it/s 0.71 | |
| [0130 17:35:59 @multigpu.py:323] [p0576] step: count(255), step_time 1402.17, mean_step_time 1409.13, it/s 0.71 | |
| [0130 17:36:00 @multigpu.py:323] [p0115] step: count(232), step_time 1371.18, mean_step_time 1405.27, it/s 0.71 | |
| [0130 17:36:00 @multigpu.py:323] [p0574] step: count(250), step_time 1394.7, mean_step_time 1407.86, it/s 0.71 | |
| [0130 17:36:00 @multigpu.py:323] [p0576] step: count(256), step_time 1370.45, mean_step_time 1409.61, it/s 0.71 | |
| [0130 17:36:01 @multigpu.py:323] [p0115] step: count(233), step_time 1407.6, mean_step_time 1403.62, it/s 0.71 | |
| [0130 17:36:01 @multigpu.py:323] [p0574] step: count(251), step_time 1392.27, mean_step_time 1409.14, it/s 0.71 | |
| [0130 17:36:02 @multigpu.py:323] [p0576] step: count(257), step_time 1404.48, mean_step_time 1409.59, it/s 0.71 | |
| [0130 17:36:03 @multigpu.py:323] [p0115] step: count(234), step_time 1388.9, mean_step_time 1404.75, it/s 0.71 | |
| [0130 17:36:03 @multigpu.py:323] [p0574] step: count(252), step_time 1392.01, mean_step_time 1406.62, it/s 0.71 | |
| [0130 17:36:03 @multigpu.py:323] [p0576] step: count(258), step_time 1382.35, mean_step_time 1410.09, it/s 0.71 | |
| [0130 17:36:04 @multigpu.py:323] [p0574] step: count(253), step_time 1358.53, mean_step_time 1396.4, it/s 0.72 | |
| [0130 17:36:04 @multigpu.py:323] [p0115] step: count(235), step_time 1397.92, mean_step_time 1402.89, it/s 0.71 | |
| [0130 17:36:04 @multigpu.py:323] [p0576] step: count(259), step_time 1351.43, mean_step_time 1407.81, it/s 0.71 | |
| [0130 17:36:06 @multigpu.py:323] [p0115] step: count(236), step_time 1348.7, mean_step_time 1399.0, it/s 0.71 | |
| [0130 17:36:06 @multigpu.py:323] [p0576] step: count(260), step_time 1386.78, mean_step_time 1407.07, it/s 0.71 | |
| [0130 17:36:06 @multigpu.py:323] [p0574] step: count(254), step_time 1552.29, mean_step_time 1397.52, it/s 0.72 | |
| [0130 17:36:07 @multigpu.py:323] [p0115] step: count(237), step_time 1384.47, mean_step_time 1400.82, it/s 0.71 | |
| [0130 17:36:07 @multigpu.py:323] [p0576] step: count(261), step_time 1396.65, mean_step_time 1407.43, it/s 0.71 | |
| [0130 17:36:07 @multigpu.py:323] [p0574] step: count(255), step_time 1380.3, mean_step_time 1391.28, it/s 0.72 | |
| [0130 17:36:08 @multigpu.py:323] [p0115] step: count(238), step_time 1420.84, mean_step_time 1401.77, it/s 0.71 | |
| [0130 17:36:08 @multigpu.py:323] [p0576] step: count(262), step_time 1414.32, mean_step_time 1406.18, it/s 0.71 | |
| [0130 17:36:09 @multigpu.py:323] [p0574] step: count(256), step_time 1380.86, mean_step_time 1389.79, it/s 0.72 | |
| [0130 17:36:10 @multigpu.py:323] [p0115] step: count(239), step_time 1345.4, mean_step_time 1400.0, it/s 0.71 | |
| [0130 17:36:10 @multigpu.py:323] [p0576] step: count(263), step_time 1396.73, mean_step_time 1408.66, it/s 0.71 | |
| [0130 17:36:10 @multigpu.py:323] [p0574] step: count(257), step_time 1388.26, mean_step_time 1388.02, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.102663866348 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:36:11 @multigpu.py:323] [p0115] step: count(240), step_time 1385.78, mean_step_time 1400.71, it/s 0.71 | |
| [0130 17:36:11 @multigpu.py:323] [p0576] step: count(264), step_time 1350.94, mean_step_time 1406.86, it/s 0.71 | |
| [0130 17:36:11 @multigpu.py:323] [p0574] step: count(258), step_time 1439.78, mean_step_time 1391.3, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.103130486078 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:36:12 @multigpu.py:323] [p0115] step: count(241), step_time 1377.61, mean_step_time 1400.26, it/s 0.71 | |
| [0130 17:36:13 @multigpu.py:323] [p0576] step: count(265), step_time 1359.71, mean_step_time 1405.62, it/s 0.71 | |
| [0130 17:36:13 @multigpu.py:323] [p0574] step: count(259), step_time 1362.89, mean_step_time 1391.27, it/s 0.72 | |
| [0130 17:36:14 @multigpu.py:323] [p0115] step: count(242), step_time 1394.91, mean_step_time 1401.82, it/s 0.71 | |
| [0130 17:36:14 @multigpu.py:323] [p0576] step: count(266), step_time 1413.71, mean_step_time 1409.41, it/s 0.71 | |
| [0130 17:36:14 @multigpu.py:323] [p0574] step: count(260), step_time 1389.64, mean_step_time 1391.97, it/s 0.72 | |
| [0130 17:36:15 @multigpu.py:323] [p0115] step: count(243), step_time 1373.49, mean_step_time 1401.7, it/s 0.71 | |
| [0130 17:36:15 @multigpu.py:323] [p0576] step: count(267), step_time 1374.35, mean_step_time 1406.86, it/s 0.71 | |
| [0130 17:36:15 @multigpu.py:323] [p0574] step: count(261), step_time 1348.26, mean_step_time 1389.78, it/s 0.72 | |
| [0130 17:36:17 @multigpu.py:323] [p0115] step: count(244), step_time 1386.47, mean_step_time 1401.98, it/s 0.71 | |
| [0130 17:36:17 @multigpu.py:323] [p0576] step: count(268), step_time 1384.0, mean_step_time 1407.54, it/s 0.71 | |
| [0130 17:36:17 @multigpu.py:323] [p0574] step: count(262), step_time 1349.67, mean_step_time 1388.87, it/s 0.72 | |
| [0130 17:36:18 @multigpu.py:323] [p0115] step: count(245), step_time 1383.07, mean_step_time 1401.17, it/s 0.71 | |
| [0130 17:36:18 @multigpu.py:323] [p0576] step: count(269), step_time 1380.73, mean_step_time 1408.0, it/s 0.71 | |
| [0130 17:36:18 @multigpu.py:323] [p0574] step: count(263), step_time 1389.54, mean_step_time 1389.48, it/s 0.72 | |
| [0130 17:36:19 @multigpu.py:323] [p0115] step: count(246), step_time 1361.13, mean_step_time 1397.85, it/s 0.72 | |
| [0130 17:36:20 @multigpu.py:323] [p0574] step: count(264), step_time 1324.5, mean_step_time 1385.57, it/s 0.72 | |
| [0130 17:36:20 @multigpu.py:323] [p0576] step: count(270), step_time 1399.82, mean_step_time 1407.92, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.10532262471 , 0.9 ##### | |
| [u'online', 0.9] | |
| ##### Sending to neptune: active_workers : 0.105322658552 , 3 ##### | |
| receiving | |
| [0130 17:36:21 @multigpu.py:323] [p0574] step: count(265), step_time 1493.54, mean_step_time 1391.65, it/s 0.72 | |
| [0130 17:36:21 @multigpu.py:323] [p0576] step: count(271), step_time 1479.04, mean_step_time 1391.58, it/s 0.72 | |
| [0130 17:36:21 @multigpu.py:323] [p0115] step: count(247), step_time 1609.22, mean_step_time 1391.7, it/s 0.72 | |
| [0130 17:36:22 @multigpu.py:323] [p0115] step: count(248), step_time 1364.46, mean_step_time 1391.3, it/s 0.72 | |
| [0130 17:36:22 @multigpu.py:323] [p0576] step: count(272), step_time 1373.2, mean_step_time 1389.69, it/s 0.72 | |
| [0130 17:36:23 @multigpu.py:323] [p0574] step: count(266), step_time 1546.98, mean_step_time 1400.32, it/s 0.71 | |
| [0130 17:36:24 @multigpu.py:323] [p0115] step: count(249), step_time 1400.73, mean_step_time 1392.69, it/s 0.72 | |
| [0130 17:36:24 @multigpu.py:323] [p0576] step: count(273), step_time 1402.29, mean_step_time 1390.46, it/s 0.72 | |
| [0130 17:36:24 @multigpu.py:323] [p0574] step: count(267), step_time 1500.75, mean_step_time 1406.72, it/s 0.71 | |
| [0130 17:36:25 @multigpu.py:323] [p0115] step: count(250), step_time 1340.03, mean_step_time 1390.35, it/s 0.72 | |
| [0130 17:36:25 @multigpu.py:323] [p0576] step: count(274), step_time 1363.4, mean_step_time 1389.33, it/s 0.72 | |
| [0130 17:36:26 @multigpu.py:323] [p0574] step: count(268), step_time 1638.14, mean_step_time 1418.0, it/s 0.71 | |
| [0130 17:36:27 @multigpu.py:323] [p0115] step: count(251), step_time 1383.45, mean_step_time 1391.27, it/s 0.72 | |
| [0130 17:36:27 @multigpu.py:323] [p0576] step: count(275), step_time 1434.35, mean_step_time 1390.94, it/s 0.72 | |
| [0130 17:36:27 @multigpu.py:323] [p0574] step: count(269), step_time 1532.01, mean_step_time 1427.75, it/s 0.7 | |
| [0130 17:36:28 @multigpu.py:323] [p0115] step: count(252), step_time 1385.99, mean_step_time 1392.01, it/s 0.72 | |
| [0130 17:36:28 @multigpu.py:323] [p0576] step: count(276), step_time 1401.97, mean_step_time 1392.51, it/s 0.72 | |
| [0130 17:36:29 @multigpu.py:323] [p0574] step: count(270), step_time 1592.14, mean_step_time 1437.62, it/s 0.7 | |
| [0130 17:36:29 @multigpu.py:323] [p0115] step: count(253), step_time 1385.17, mean_step_time 1390.89, it/s 0.72 | |
| [0130 17:36:29 @multigpu.py:323] [p0576] step: count(277), step_time 1378.67, mean_step_time 1391.22, it/s 0.72 | |
| [0130 17:36:30 @multigpu.py:323] [p0574] step: count(271), step_time 1486.68, mean_step_time 1442.34, it/s 0.69 | |
| [0130 17:36:31 @multigpu.py:323] [p0115] step: count(254), step_time 1391.45, mean_step_time 1391.01, it/s 0.72 | |
| [0130 17:36:31 @multigpu.py:323] [p0576] step: count(278), step_time 1465.1, mean_step_time 1395.36, it/s 0.72 | |
| [0130 17:36:32 @multigpu.py:323] [p0574] step: count(272), step_time 1466.54, mean_step_time 1446.07, it/s 0.69 | |
| [0130 17:36:32 @multigpu.py:323] [p0115] step: count(255), step_time 1385.32, mean_step_time 1390.38, it/s 0.72 | |
| [0130 17:36:32 @multigpu.py:323] [p0576] step: count(279), step_time 1373.63, mean_step_time 1396.47, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.108991423051 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:36:33 @multigpu.py:323] [p0574] step: count(273), step_time 1636.7, mean_step_time 1459.97, it/s 0.68 | |
| [0130 17:36:33 @multigpu.py:323] [p0115] step: count(256), step_time 1384.76, mean_step_time 1392.19, it/s 0.72 | |
| [0130 17:36:34 @multigpu.py:323] [p0576] step: count(280), step_time 1397.0, mean_step_time 1396.98, it/s 0.72 | |
| [0130 17:36:35 @multigpu.py:323] [p0115] step: count(257), step_time 1385.74, mean_step_time 1392.25, it/s 0.72 | |
| [0130 17:36:35 @multigpu.py:323] [p0576] step: count(281), step_time 1343.67, mean_step_time 1394.33, it/s 0.72 | |
| [0130 17:36:35 @multigpu.py:323] [p0574] step: count(274), step_time 1662.91, mean_step_time 1465.5, it/s 0.68 | |
| [0130 17:36:36 @multigpu.py:323] [p0115] step: count(258), step_time 1358.49, mean_step_time 1389.13, it/s 0.72 | |
| [0130 17:36:36 @multigpu.py:323] [p0576] step: count(282), step_time 1396.08, mean_step_time 1393.42, it/s 0.72 | |
| [0130 17:36:37 @multigpu.py:323] [p0574] step: count(275), step_time 1619.75, mean_step_time 1477.48, it/s 0.68 | |
| [0130 17:36:38 @multigpu.py:323] [p0115] step: count(259), step_time 1387.44, mean_step_time 1391.24, it/s 0.72 | |
| [0130 17:36:38 @multigpu.py:323] [p0576] step: count(283), step_time 1457.86, mean_step_time 1396.48, it/s 0.72 | |
| [0130 17:36:38 @multigpu.py:323] [p0574] step: count(276), step_time 1383.84, mean_step_time 1477.63, it/s 0.68 | |
| [0130 17:36:39 @multigpu.py:323] [p0115] step: count(260), step_time 1375.11, mean_step_time 1390.7, it/s 0.72 | |
| [0130 17:36:39 @multigpu.py:323] [p0576] step: count(284), step_time 1445.71, mean_step_time 1401.21, it/s 0.71 | |
| [0130 17:36:39 @multigpu.py:323] [p0574] step: count(277), step_time 1414.63, mean_step_time 1478.94, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.110936076641 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:36:40 @multigpu.py:323] [p0115] step: count(261), step_time 1376.47, mean_step_time 1390.64, it/s 0.72 | |
| [0130 17:36:41 @multigpu.py:323] [p0576] step: count(285), step_time 1379.43, mean_step_time 1402.2, it/s 0.71 | |
| [0130 17:36:41 @multigpu.py:323] [p0574] step: count(278), step_time 1411.53, mean_step_time 1477.53, it/s 0.68 | |
| [0130 17:36:42 @multigpu.py:323] [p0115] step: count(262), step_time 1371.64, mean_step_time 1389.48, it/s 0.72 | |
| [0130 17:36:42 @multigpu.py:323] [p0576] step: count(286), step_time 1378.01, mean_step_time 1400.42, it/s 0.71 | |
| [0130 17:36:42 @multigpu.py:323] [p0574] step: count(279), step_time 1407.81, mean_step_time 1479.78, it/s 0.68 | |
| [0130 17:36:43 @multigpu.py:323] [p0115] step: count(263), step_time 1485.85, mean_step_time 1395.1, it/s 0.72 | |
| [0130 17:36:43 @multigpu.py:323] [p0576] step: count(287), step_time 1401.66, mean_step_time 1401.78, it/s 0.71 | |
| [0130 17:36:44 @multigpu.py:323] [p0574] step: count(280), step_time 1467.43, mean_step_time 1483.67, it/s 0.67 | |
| [0130 17:36:45 @multigpu.py:323] [p0115] step: count(264), step_time 1356.22, mean_step_time 1393.59, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.112287428313 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:36:45 @multigpu.py:323] [p0576] step: count(288), step_time 1476.2, mean_step_time 1406.39, it/s 0.71 | |
| [0130 17:36:45 @multigpu.py:323] [p0574] step: count(281), step_time 1594.94, mean_step_time 1496.0, it/s 0.67 | |
| [0130 17:36:46 @multigpu.py:323] [p0115] step: count(265), step_time 1363.91, mean_step_time 1392.63, it/s 0.72 | |
| [0130 17:36:46 @multigpu.py:323] [p0576] step: count(289), step_time 1398.59, mean_step_time 1407.28, it/s 0.71 | |
| [0130 17:36:47 @multigpu.py:323] [p0574] step: count(282), step_time 1580.49, mean_step_time 1507.54, it/s 0.66 | |
| [0130 17:36:47 @multigpu.py:323] [p0115] step: count(266), step_time 1397.36, mean_step_time 1394.44, it/s 0.72 | |
| [0130 17:36:48 @multigpu.py:323] [p0576] step: count(290), step_time 1325.89, mean_step_time 1403.59, it/s 0.71 | |
| [0130 17:36:48 @multigpu.py:323] [p0574] step: count(283), step_time 1467.9, mean_step_time 1511.46, it/s 0.66 | |
| [0130 17:36:49 @multigpu.py:323] [p0115] step: count(267), step_time 1365.89, mean_step_time 1382.27, it/s 0.72 | |
| [0130 17:36:50 @multigpu.py:323] [p0115] step: count(268), step_time 1384.45, mean_step_time 1383.27, it/s 0.72 | |
| [0130 17:36:50 @multigpu.py:323] [p0574] step: count(284), step_time 1619.19, mean_step_time 1526.2, it/s 0.66 | |
| [0130 17:36:50 @multigpu.py:323] [p0576] step: count(291), step_time 2436.1, mean_step_time 1451.44, it/s 0.69 | |
| [0130 17:36:51 @multigpu.py:323] [p0576] step: count(292), step_time 1357.52, mean_step_time 1450.66, it/s 0.69 | |
| [0130 17:36:51 @multigpu.py:323] [p0115] step: count(269), step_time 1398.63, mean_step_time 1383.17, it/s 0.72 | |
| [0130 17:36:52 @multigpu.py:323] [p0574] step: count(285), step_time 1613.86, mean_step_time 1532.21, it/s 0.65 | |
| [0130 17:36:53 @multigpu.py:323] [p0115] step: count(270), step_time 1342.82, mean_step_time 1383.31, it/s 0.72 | |
| [0130 17:36:53 @multigpu.py:323] [p0576] step: count(293), step_time 1395.05, mean_step_time 1450.29, it/s 0.69 | |
| [0130 17:36:53 @multigpu.py:323] [p0574] step: count(286), step_time 1630.58, mean_step_time 1536.39, it/s 0.65 | |
| [0130 17:36:54 @multigpu.py:323] [p0115] step: count(271), step_time 1308.8, mean_step_time 1379.58, it/s 0.72 | |
| [0130 17:36:54 @multigpu.py:323] [p0576] step: count(294), step_time 1346.21, mean_step_time 1449.44, it/s 0.69 | |
| [0130 17:36:55 @multigpu.py:323] [p0574] step: count(287), step_time 1521.75, mean_step_time 1537.44, it/s 0.65 | |
| [0130 17:36:56 @multigpu.py:323] [p0115] step: count(272), step_time 1412.48, mean_step_time 1380.9, it/s 0.72 | |
| [0130 17:36:56 @multigpu.py:323] [p0576] step: count(295), step_time 1369.98, mean_step_time 1446.22, it/s 0.69 | |
| [0130 17:36:56 @multigpu.py:323] [p0574] step: count(288), step_time 1358.74, mean_step_time 1523.47, it/s 0.66 | |
| [0130 17:36:57 @multigpu.py:323] [p0115] step: count(273), step_time 1386.04, mean_step_time 1380.94, it/s 0.72 | |
| [0130 17:36:57 @multigpu.py:323] [p0576] step: count(296), step_time 1424.16, mean_step_time 1447.33, it/s 0.69 | |
| [0130 17:36:58 @multigpu.py:323] [p0574] step: count(289), step_time 1388.73, mean_step_time 1516.31, it/s 0.66 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.115880426632 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:36:58 @multigpu.py:323] [p0115] step: count(274), step_time 1380.34, mean_step_time 1380.39, it/s 0.72 | |
| [0130 17:36:58 @multigpu.py:323] [p0576] step: count(297), step_time 1368.9, mean_step_time 1446.84, it/s 0.69 | |
| [0130 17:36:59 @multigpu.py:323] [p0574] step: count(290), step_time 1394.5, mean_step_time 1506.43, it/s 0.66 | |
| [0130 17:37:00 @multigpu.py:323] [p0115] step: count(275), step_time 1379.69, mean_step_time 1380.11, it/s 0.72 | |
| [0130 17:37:00 @multigpu.py:323] [p0576] step: count(298), step_time 1394.81, mean_step_time 1443.32, it/s 0.69 | |
| [0130 17:37:00 @multigpu.py:323] [p0574] step: count(291), step_time 1387.45, mean_step_time 1501.46, it/s 0.67 | |
| [0130 17:37:01 @multigpu.py:323] [p0115] step: count(276), step_time 1387.1, mean_step_time 1380.22, it/s 0.72 | |
| [0130 17:37:01 @multigpu.py:323] [p0576] step: count(299), step_time 1385.73, mean_step_time 1443.93, it/s 0.69 | |
| [0130 17:37:02 @multigpu.py:323] [p0574] step: count(292), step_time 1369.93, mean_step_time 1496.63, it/s 0.67 | |
| [0130 17:37:02 @multigpu.py:323] [p0115] step: count(277), step_time 1383.91, mean_step_time 1380.13, it/s 0.72 | |
| [0130 17:37:03 @multigpu.py:323] [p0576] step: count(300), step_time 1462.3, mean_step_time 1447.19, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.117210962176 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.117210962176 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.117210962176 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.117211548554 , -0.0115024279803 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.117211548554 , -0.317066699266 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.117211548554 , -2.28820109367 ##### | |
| ##### Sending to neptune: value_loss : 0.117211548554 , 1.13295674324 ##### | |
| ##### Sending to neptune: advantage : 0.117211548554 , 0.00142617744859 ##### | |
| ##### Sending to neptune: pred_reward : 0.117211548554 , 0.396612882614 ##### | |
| ##### Sending to neptune: max_logit : 0.117211548554 , 0.193715110421 ##### | |
| [u'loss', -0.011502427980303764, -0.3170666992664337, -2.288201093673706, 1.1329567432403564, 0.00142617744859308, 0.39661288261413574, 0.19371511042118073] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.117212024993 , 8889058.0 ##### | |
| ##### Sending to neptune: dp_per_s : 0.117212024993 , 91.0211410016 ##### | |
| [u'other', 8889058.0, 91.02114100159582] | |
| receiving | |
| [0130 17:37:03 @multigpu.py:323] [p0574] step: count(293), step_time 1378.74, mean_step_time 1483.74, it/s 0.67 | |
| [0130 17:37:04 @multigpu.py:323] [p0115] step: count(278), step_time 1361.57, mean_step_time 1380.29, it/s 0.72 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.117604462769 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:37:04 @multigpu.py:323] [p0576] step: count(301), step_time 1422.54, mean_step_time 1451.14, it/s 0.69 | |
| [0130 17:37:05 @multigpu.py:323] [p0574] step: count(294), step_time 1524.22, mean_step_time 1476.8, it/s 0.68 | |
| [0130 17:37:05 @multigpu.py:323] [p0115] step: count(279), step_time 1390.55, mean_step_time 1380.44, it/s 0.72 | |
| [0130 17:37:05 @multigpu.py:323] [p0576] step: count(302), step_time 1373.9, mean_step_time 1450.03, it/s 0.69 | |
| [0130 17:37:06 @multigpu.py:323] [p0574] step: count(295), step_time 1396.08, mean_step_time 1465.62, it/s 0.68 | |
| [0130 17:37:07 @multigpu.py:323] [p0115] step: count(280), step_time 1366.44, mean_step_time 1380.01, it/s 0.72 | |
| [0130 17:37:07 @multigpu.py:323] [p0576] step: count(303), step_time 1444.09, mean_step_time 1449.34, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.118510236078 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:37:08 @multigpu.py:323] [p0574] step: count(296), step_time 1630.41, mean_step_time 1477.95, it/s 0.68 | |
| [0130 17:37:08 @multigpu.py:323] [p0115] step: count(281), step_time 1372.06, mean_step_time 1379.79, it/s 0.72 | |
| [0130 17:37:08 @multigpu.py:323] [p0576] step: count(304), step_time 1350.36, mean_step_time 1444.57, it/s 0.69 | |
| [0130 17:37:09 @multigpu.py:323] [p0574] step: count(297), step_time 1582.59, mean_step_time 1486.34, it/s 0.67 | |
| [0130 17:37:09 @multigpu.py:323] [p0115] step: count(282), step_time 1378.39, mean_step_time 1380.13, it/s 0.72 | |
| [0130 17:37:10 @multigpu.py:323] [p0576] step: count(305), step_time 1426.73, mean_step_time 1446.94, it/s 0.69 | |
| [0130 17:37:11 @multigpu.py:323] [p0574] step: count(298), step_time 1373.21, mean_step_time 1484.43, it/s 0.67 | |
| [0130 17:37:11 @multigpu.py:323] [p0115] step: count(283), step_time 1393.71, mean_step_time 1375.52, it/s 0.73 | |
| [0130 17:37:11 @multigpu.py:323] [p0576] step: count(306), step_time 1448.18, mean_step_time 1450.44, it/s 0.69 | |
| [0130 17:37:12 @multigpu.py:323] [p0574] step: count(299), step_time 1393.04, mean_step_time 1483.69, it/s 0.67 | |
| [0130 17:37:12 @multigpu.py:323] [p0115] step: count(284), step_time 1408.19, mean_step_time 1378.12, it/s 0.73 | |
| [0130 17:37:12 @multigpu.py:323] [p0576] step: count(307), step_time 1367.6, mean_step_time 1448.74, it/s 0.69 | |
| [0130 17:37:13 @multigpu.py:323] [p0574] step: count(300), step_time 1382.7, mean_step_time 1479.45, it/s 0.68 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.120217489137 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.120217489137 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.120217489137 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.120218040546 , -0.0103861913085 ##### | |
| ##### Sending to neptune: policy_loss : 0.120218040546 , -0.244617804885 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.120218040546 , -2.28823947906 ##### | |
| ##### Sending to neptune: value_loss : 0.120218040546 , 1.20342481136 ##### | |
| ##### Sending to neptune: advantage : 0.120218040546 , 0.00103269040119 ##### | |
| ##### Sending to neptune: pred_reward : 0.120218040546 , 0.395009964705 ##### | |
| ##### Sending to neptune: max_logit : 0.120218040546 , 0.193438783288 ##### | |
| [u'loss', -0.010386191308498383, -0.24461780488491058, -2.2882394790649414, 1.2034248113632202, 0.0010326904011890292, 0.39500996470451355, 0.19343878328800201] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.120218533013 , 8887224.41 ##### | |
| ##### Sending to neptune: dp_per_s : 0.120218533013 , 89.5399337807 ##### | |
| [u'other', 8887224.41, 89.53993378070659] | |
| receiving | |
| [0130 17:37:13 @multigpu.py:323] [p0115] step: count(285), step_time 1365.26, mean_step_time 1378.19, it/s 0.73 | |
| [0130 17:37:14 @multigpu.py:323] [p0576] step: count(308), step_time 1443.34, mean_step_time 1447.1, it/s 0.69 | |
| [0130 17:37:15 @multigpu.py:323] [p0574] step: count(301), step_time 1380.97, mean_step_time 1468.75, it/s 0.68 | |
| [0130 17:37:15 @multigpu.py:323] [p0115] step: count(286), step_time 1412.84, mean_step_time 1378.96, it/s 0.73 | |
| [0130 17:37:15 @multigpu.py:323] [p0576] step: count(309), step_time 1374.83, mean_step_time 1445.91, it/s 0.69 | |
| [0130 17:37:16 @multigpu.py:323] [p0574] step: count(302), step_time 1371.75, mean_step_time 1458.32, it/s 0.69 | |
| [0130 17:37:16 @multigpu.py:323] [p0115] step: count(287), step_time 1411.56, mean_step_time 1381.24, it/s 0.72 | |
| [0130 17:37:17 @multigpu.py:323] [p0576] step: count(310), step_time 1376.81, mean_step_time 1448.46, it/s 0.69 | |
| [0130 17:37:18 @multigpu.py:323] [p0574] step: count(303), step_time 1409.68, mean_step_time 1455.41, it/s 0.69 | |
| [0130 17:37:19 @multigpu.py:323] [p0574] step: count(304), step_time 1364.59, mean_step_time 1442.68, it/s 0.69 | |
| [0130 17:37:19 @multigpu.py:323] [p0115] step: count(288), step_time 2615.34, mean_step_time 1442.79, it/s 0.69 | |
| [0130 17:37:19 @multigpu.py:323] [p0576] step: count(311), step_time 2319.64, mean_step_time 1442.63, it/s 0.69 | |
| [0130 17:37:20 @multigpu.py:323] [p0115] step: count(289), step_time 1383.5, mean_step_time 1442.03, it/s 0.69 | |
| [0130 17:37:20 @multigpu.py:323] [p0576] step: count(312), step_time 1439.56, mean_step_time 1446.74, it/s 0.69 | |
| [0130 17:37:20 @multigpu.py:323] [p0574] step: count(305), step_time 1487.43, mean_step_time 1436.35, it/s 0.7 | |
| [0130 17:37:22 @multigpu.py:323] [p0115] step: count(290), step_time 1381.82, mean_step_time 1443.98, it/s 0.69 | |
| [0130 17:37:22 @multigpu.py:323] [p0576] step: count(313), step_time 1420.89, mean_step_time 1448.03, it/s 0.69 | |
| [0130 17:37:22 @multigpu.py:323] [p0574] step: count(306), step_time 1430.76, mean_step_time 1426.36, it/s 0.7 | |
| [0130 17:37:23 @multigpu.py:323] [p0115] step: count(291), step_time 1380.89, mean_step_time 1447.58, it/s 0.69 | |
| [0130 17:37:23 @multigpu.py:323] [p0576] step: count(314), step_time 1397.24, mean_step_time 1450.58, it/s 0.69 | |
| [0130 17:37:23 @multigpu.py:323] [p0574] step: count(307), step_time 1395.27, mean_step_time 1420.04, it/s 0.7 | |
| [0130 17:37:24 @multigpu.py:323] [p0115] step: count(292), step_time 1402.06, mean_step_time 1447.06, it/s 0.69 | |
| [0130 17:37:25 @multigpu.py:323] [p0576] step: count(315), step_time 1441.95, mean_step_time 1454.18, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.123340421054 , 1.1 ##### | |
| [u'online', 1.1] | |
| ##### Sending to neptune: active_workers : 0.123340500792 , 3 ##### | |
| receiving | |
| [0130 17:37:25 @multigpu.py:323] [p0574] step: count(308), step_time 1530.98, mean_step_time 1428.65, it/s 0.7 | |
| [0130 17:37:26 @multigpu.py:323] [p0115] step: count(293), step_time 1416.82, mean_step_time 1448.6, it/s 0.69 | |
| [0130 17:37:26 @multigpu.py:323] [p0576] step: count(316), step_time 1384.3, mean_step_time 1452.18, it/s 0.69 | |
| [0130 17:37:26 @multigpu.py:323] [p0574] step: count(309), step_time 1488.91, mean_step_time 1433.66, it/s 0.7 | |
| [0130 17:37:27 @multigpu.py:323] [p0115] step: count(294), step_time 1349.49, mean_step_time 1447.06, it/s 0.69 | |
| [0130 17:37:27 @multigpu.py:323] [p0576] step: count(317), step_time 1357.59, mean_step_time 1451.62, it/s 0.69 | |
| [0130 17:37:28 @multigpu.py:323] [p0574] step: count(310), step_time 1442.14, mean_step_time 1436.04, it/s 0.7 | |
| [0130 17:37:29 @multigpu.py:323] [p0115] step: count(295), step_time 1392.08, mean_step_time 1447.68, it/s 0.69 | |
| [0130 17:37:29 @multigpu.py:323] [p0576] step: count(318), step_time 1462.51, mean_step_time 1455.0, it/s 0.69 | |
| [0130 17:37:29 @multigpu.py:323] [p0574] step: count(311), step_time 1438.97, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:37:30 @multigpu.py:323] [p0115] step: count(296), step_time 1361.63, mean_step_time 1446.41, it/s 0.69 | |
| [0130 17:37:30 @multigpu.py:323] [p0576] step: count(319), step_time 1387.51, mean_step_time 1455.09, it/s 0.69 | |
| [0130 17:37:31 @multigpu.py:323] [p0574] step: count(312), step_time 1377.31, mean_step_time 1438.99, it/s 0.69 | |
| [0130 17:37:31 @multigpu.py:323] [p0115] step: count(297), step_time 1405.26, mean_step_time 1447.47, it/s 0.69 | |
| [0130 17:37:32 @multigpu.py:323] [p0576] step: count(320), step_time 1379.68, mean_step_time 1450.96, it/s 0.69 | |
| [0130 17:37:32 @multigpu.py:323] [p0574] step: count(313), step_time 1417.67, mean_step_time 1440.93, it/s 0.69 | |
| [0130 17:37:33 @multigpu.py:323] [p0115] step: count(298), step_time 1410.03, mean_step_time 1449.9, it/s 0.69 | |
| [0130 17:37:33 @multigpu.py:323] [p0576] step: count(321), step_time 1376.52, mean_step_time 1448.66, it/s 0.69 | |
| [0130 17:37:33 @multigpu.py:323] [p0574] step: count(314), step_time 1339.04, mean_step_time 1431.68, it/s 0.7 | |
| [0130 17:37:34 @multigpu.py:323] [p0115] step: count(299), step_time 1389.03, mean_step_time 1449.82, it/s 0.69 | |
| [0130 17:37:34 @multigpu.py:323] [p0576] step: count(322), step_time 1386.44, mean_step_time 1449.29, it/s 0.69 | |
| [0130 17:37:35 @multigpu.py:323] [p0574] step: count(315), step_time 1398.39, mean_step_time 1431.79, it/s 0.7 | |
| [0130 17:37:36 @multigpu.py:323] [p0115] step: count(300), step_time 1443.03, mean_step_time 1453.65, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.126398886376 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.126398886376 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.126398886376 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.126399410831 , -0.0113957431167 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.126399410831 , -0.226604878902 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.126399410831 , -2.28844261169 ##### | |
| ##### Sending to neptune: value_loss : 0.126399410831 , 1.05639243126 ##### | |
| ##### Sending to neptune: advantage : 0.126399410831 , 0.00103650894016 ##### | |
| ##### Sending to neptune: pred_reward : 0.126399410831 , 0.386131823063 ##### | |
| ##### Sending to neptune: max_logit : 0.126399410831 , 0.19203697145 ##### | |
| [u'loss', -0.011395743116736412, -0.2266048789024353, -2.288442611694336, 1.0563924312591553, 0.0010365089401602745, 0.38613182306289673, 0.192036971449852] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.126399871376 , 8856618.33 ##### | |
| ##### Sending to neptune: dp_per_s : 0.126399871376 , 91.271358836 ##### | |
| [u'other', 8856618.33, 91.27135883601044] | |
| receiving | |
| [0130 17:37:36 @multigpu.py:323] [p0576] step: count(323), step_time 1405.49, mean_step_time 1447.36, it/s 0.69 | |
| [0130 17:37:36 @multigpu.py:323] [p0574] step: count(316), step_time 1407.92, mean_step_time 1420.67, it/s 0.7 | |
| [0130 17:37:37 @multigpu.py:323] [p0115] step: count(301), step_time 1387.53, mean_step_time 1454.42, it/s 0.69 | |
| [0130 17:37:37 @multigpu.py:323] [p0576] step: count(324), step_time 1378.84, mean_step_time 1448.78, it/s 0.69 | |
| [0130 17:37:37 @multigpu.py:323] [p0574] step: count(317), step_time 1393.47, mean_step_time 1411.21, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.12702402691 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.127058637738 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:37:38 @multigpu.py:323] [p0115] step: count(302), step_time 1390.81, mean_step_time 1455.04, it/s 0.69 | |
| [0130 17:37:39 @multigpu.py:323] [p0576] step: count(325), step_time 1451.99, mean_step_time 1450.05, it/s 0.69 | |
| [0130 17:37:39 @multigpu.py:323] [p0574] step: count(318), step_time 1425.59, mean_step_time 1413.83, it/s 0.71 | |
| [0130 17:37:40 @multigpu.py:323] [p0115] step: count(303), step_time 1408.57, mean_step_time 1455.79, it/s 0.69 | |
| [0130 17:37:40 @multigpu.py:323] [p0576] step: count(326), step_time 1362.75, mean_step_time 1445.77, it/s 0.69 | |
| [0130 17:37:40 @multigpu.py:323] [p0574] step: count(319), step_time 1372.05, mean_step_time 1412.78, it/s 0.71 | |
| [0130 17:37:41 @multigpu.py:323] [p0115] step: count(304), step_time 1345.33, mean_step_time 1452.64, it/s 0.69 | |
| [0130 17:37:41 @multigpu.py:323] [p0576] step: count(327), step_time 1445.58, mean_step_time 1449.67, it/s 0.69 | |
| [0130 17:37:42 @multigpu.py:323] [p0574] step: count(320), step_time 1361.26, mean_step_time 1411.71, it/s 0.71 | |
| [0130 17:37:43 @multigpu.py:323] [p0115] step: count(305), step_time 1437.42, mean_step_time 1456.25, it/s 0.69 | |
| [0130 17:37:43 @multigpu.py:323] [p0576] step: count(328), step_time 1383.55, mean_step_time 1446.68, it/s 0.69 | |
| [0130 17:37:43 @multigpu.py:323] [p0574] step: count(321), step_time 1381.44, mean_step_time 1411.73, it/s 0.71 | |
| [0130 17:37:44 @multigpu.py:323] [p0115] step: count(306), step_time 1359.95, mean_step_time 1453.61, it/s 0.69 | |
| [0130 17:37:44 @multigpu.py:323] [p0576] step: count(329), step_time 1418.87, mean_step_time 1448.89, it/s 0.69 | |
| [0130 17:37:44 @multigpu.py:323] [p0574] step: count(322), step_time 1415.14, mean_step_time 1413.9, it/s 0.71 | |
| [0130 17:37:45 @multigpu.py:323] [p0115] step: count(307), step_time 1396.46, mean_step_time 1452.85, it/s 0.69 | |
| [0130 17:37:46 @multigpu.py:323] [p0576] step: count(330), step_time 1455.6, mean_step_time 1452.83, it/s 0.69 | |
| [0130 17:37:46 @multigpu.py:323] [p0574] step: count(323), step_time 1414.92, mean_step_time 1414.16, it/s 0.71 | |
| [0130 17:37:47 @multigpu.py:323] [p0574] step: count(324), step_time 1423.54, mean_step_time 1417.11, it/s 0.71 | |
| [0130 17:37:47 @multigpu.py:323] [p0115] step: count(308), step_time 1900.99, mean_step_time 1417.14, it/s 0.71 | |
| [0130 17:37:47 @multigpu.py:323] [p0576] step: count(331), step_time 1604.34, mean_step_time 1417.06, it/s 0.71 | |
| [0130 17:37:49 @multigpu.py:323] [p0115] step: count(309), step_time 1365.43, mean_step_time 1416.23, it/s 0.71 | |
| [0130 17:37:49 @multigpu.py:323] [p0574] step: count(325), step_time 1368.14, mean_step_time 1411.15, it/s 0.71 | |
| [0130 17:37:49 @multigpu.py:323] [p0576] step: count(332), step_time 1407.4, mean_step_time 1415.45, it/s 0.71 | |
| [0130 17:37:50 @multigpu.py:323] [p0574] step: count(326), step_time 1396.46, mean_step_time 1409.43, it/s 0.71 | |
| [0130 17:37:50 @multigpu.py:323] [p0115] step: count(310), step_time 1424.3, mean_step_time 1418.36, it/s 0.71 | |
| [0130 17:37:50 @multigpu.py:323] [p0576] step: count(333), step_time 1397.06, mean_step_time 1414.26, it/s 0.71 | |
| [0130 17:37:51 @multigpu.py:323] [p0574] step: count(327), step_time 1358.35, mean_step_time 1407.58, it/s 0.71 | |
| [0130 17:37:51 @multigpu.py:323] [p0115] step: count(311), step_time 1356.57, mean_step_time 1417.14, it/s 0.71 | |
| [0130 17:37:51 @multigpu.py:323] [p0576] step: count(334), step_time 1354.64, mean_step_time 1412.13, it/s 0.71 | |
| [0130 17:37:53 @multigpu.py:323] [p0574] step: count(328), step_time 1380.41, mean_step_time 1400.06, it/s 0.71 | |
| [0130 17:37:53 @multigpu.py:323] [p0115] step: count(312), step_time 1374.11, mean_step_time 1415.74, it/s 0.71 | |
| [0130 17:37:53 @multigpu.py:323] [p0576] step: count(335), step_time 1438.54, mean_step_time 1411.96, it/s 0.71 | |
| [0130 17:37:54 @multigpu.py:323] [p0574] step: count(329), step_time 1412.16, mean_step_time 1396.22, it/s 0.72 | |
| [0130 17:37:54 @multigpu.py:323] [p0115] step: count(313), step_time 1416.59, mean_step_time 1415.73, it/s 0.71 | |
| [0130 17:37:54 @multigpu.py:323] [p0576] step: count(336), step_time 1447.87, mean_step_time 1415.14, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.131745723883 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:37:56 @multigpu.py:323] [p0574] step: count(330), step_time 1360.8, mean_step_time 1392.15, it/s 0.72 | |
| [0130 17:37:56 @multigpu.py:323] [p0115] step: count(314), step_time 1370.28, mean_step_time 1416.77, it/s 0.71 | |
| [0130 17:37:56 @multigpu.py:323] [p0576] step: count(337), step_time 1417.26, mean_step_time 1418.12, it/s 0.71 | |
| [0130 17:37:57 @multigpu.py:323] [p0574] step: count(331), step_time 1391.51, mean_step_time 1389.78, it/s 0.72 | |
| [0130 17:37:57 @multigpu.py:323] [p0115] step: count(315), step_time 1387.7, mean_step_time 1416.55, it/s 0.71 | |
| [0130 17:37:57 @multigpu.py:323] [p0576] step: count(338), step_time 1444.41, mean_step_time 1417.22, it/s 0.71 | |
| [0130 17:37:58 @multigpu.py:323] [p0574] step: count(332), step_time 1374.04, mean_step_time 1389.62, it/s 0.72 | |
| [0130 17:37:58 @multigpu.py:323] [p0115] step: count(316), step_time 1388.04, mean_step_time 1417.87, it/s 0.71 | |
| [0130 17:37:59 @multigpu.py:323] [p0576] step: count(339), step_time 1442.65, mean_step_time 1419.97, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.132890012993 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:38:00 @multigpu.py:323] [p0574] step: count(333), step_time 1416.93, mean_step_time 1389.58, it/s 0.72 | |
| [0130 17:38:00 @multigpu.py:323] [p0115] step: count(317), step_time 1383.89, mean_step_time 1416.8, it/s 0.71 | |
| [0130 17:38:00 @multigpu.py:323] [p0576] step: count(340), step_time 1449.19, mean_step_time 1423.45, it/s 0.7 | |
| [0130 17:38:01 @multigpu.py:323] [p0574] step: count(334), step_time 1360.92, mean_step_time 1390.67, it/s 0.72 | |
| [0130 17:38:01 @multigpu.py:323] [p0115] step: count(318), step_time 1384.03, mean_step_time 1415.5, it/s 0.71 | |
| [0130 17:38:01 @multigpu.py:323] [p0576] step: count(341), step_time 1400.62, mean_step_time 1424.65, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.133605820537 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:38:02 @multigpu.py:323] [p0115] step: count(319), step_time 1350.75, mean_step_time 1413.59, it/s 0.71 | |
| [0130 17:38:03 @multigpu.py:323] [p0574] step: count(335), step_time 1407.67, mean_step_time 1391.14, it/s 0.72 | |
| [0130 17:38:03 @multigpu.py:323] [p0576] step: count(342), step_time 1447.8, mean_step_time 1427.72, it/s 0.7 | |
| [0130 17:38:04 @multigpu.py:323] [p0574] step: count(336), step_time 1384.51, mean_step_time 1389.97, it/s 0.72 | |
| [0130 17:38:04 @multigpu.py:323] [p0115] step: count(320), step_time 1447.59, mean_step_time 1413.82, it/s 0.71 | |
| [0130 17:38:04 @multigpu.py:323] [p0576] step: count(343), step_time 1418.6, mean_step_time 1428.38, it/s 0.7 | |
| [0130 17:38:05 @multigpu.py:323] [p0574] step: count(337), step_time 1368.34, mean_step_time 1388.71, it/s 0.72 | |
| [0130 17:38:05 @multigpu.py:323] [p0115] step: count(321), step_time 1407.05, mean_step_time 1414.79, it/s 0.71 | |
| [0130 17:38:06 @multigpu.py:323] [p0576] step: count(344), step_time 1410.33, mean_step_time 1429.95, it/s 0.7 | |
| [0130 17:38:07 @multigpu.py:323] [p0574] step: count(338), step_time 1393.57, mean_step_time 1387.11, it/s 0.72 | |
| [0130 17:38:07 @multigpu.py:323] [p0115] step: count(322), step_time 1391.66, mean_step_time 1414.83, it/s 0.71 | |
| [0130 17:38:07 @multigpu.py:323] [p0576] step: count(345), step_time 1421.3, mean_step_time 1428.42, it/s 0.7 | |
| [0130 17:38:08 @multigpu.py:323] [p0574] step: count(339), step_time 1349.97, mean_step_time 1386.0, it/s 0.72 | |
| [0130 17:38:08 @multigpu.py:323] [p0115] step: count(323), step_time 1379.38, mean_step_time 1413.38, it/s 0.71 | |
| [0130 17:38:09 @multigpu.py:323] [p0576] step: count(346), step_time 1406.67, mean_step_time 1430.61, it/s 0.7 | |
| [0130 17:38:09 @multigpu.py:323] [p0574] step: count(340), step_time 1452.74, mean_step_time 1390.58, it/s 0.72 | |
| [0130 17:38:09 @multigpu.py:323] [p0115] step: count(324), step_time 1370.54, mean_step_time 1414.64, it/s 0.71 | |
| [0130 17:38:10 @multigpu.py:323] [p0576] step: count(347), step_time 1397.1, mean_step_time 1428.19, it/s 0.7 | |
| [0130 17:38:11 @multigpu.py:323] [p0115] step: count(325), step_time 1345.95, mean_step_time 1410.06, it/s 0.71 | |
| [0130 17:38:11 @multigpu.py:323] [p0574] step: count(341), step_time 1485.56, mean_step_time 1395.78, it/s 0.72 | |
| [0130 17:38:11 @multigpu.py:323] [p0576] step: count(348), step_time 1434.95, mean_step_time 1430.76, it/s 0.7 | |
| [0130 17:38:12 @multigpu.py:323] [p0115] step: count(326), step_time 1372.42, mean_step_time 1410.69, it/s 0.71 | |
| [0130 17:38:12 @multigpu.py:323] [p0574] step: count(342), step_time 1347.73, mean_step_time 1392.41, it/s 0.72 | |
| [0130 17:38:13 @multigpu.py:323] [p0576] step: count(349), step_time 1477.29, mean_step_time 1433.68, it/s 0.7 | |
| [0130 17:38:14 @multigpu.py:323] [p0115] step: count(327), step_time 1419.62, mean_step_time 1411.84, it/s 0.71 | |
| [0130 17:38:14 @multigpu.py:323] [p0574] step: count(343), step_time 1402.41, mean_step_time 1391.79, it/s 0.72 | |
| [0130 17:38:14 @multigpu.py:323] [p0576] step: count(350), step_time 1375.61, mean_step_time 1429.68, it/s 0.7 | |
| [0130 17:38:16 @multigpu.py:323] [p0576] step: count(351), step_time 1423.33, mean_step_time 1420.63, it/s 0.7 | |
| [0130 17:38:16 @multigpu.py:323] [p0574] step: count(344), step_time 2003.9, mean_step_time 1420.81, it/s 0.7 | |
| [0130 17:38:16 @multigpu.py:323] [p0115] step: count(328), step_time 2080.96, mean_step_time 1420.84, it/s 0.7 | |
| [0130 17:38:17 @multigpu.py:323] [p0115] step: count(329), step_time 1369.82, mean_step_time 1421.06, it/s 0.7 | |
| [0130 17:38:17 @multigpu.py:323] [p0574] step: count(345), step_time 1407.39, mean_step_time 1422.77, it/s 0.7 | |
| [0130 17:38:17 @multigpu.py:323] [p0576] step: count(352), step_time 1426.08, mean_step_time 1421.56, it/s 0.7 | |
| [0130 17:38:18 @multigpu.py:323] [p0574] step: count(346), step_time 1367.47, mean_step_time 1421.32, it/s 0.7 | |
| [0130 17:38:18 @multigpu.py:323] [p0115] step: count(330), step_time 1413.16, mean_step_time 1420.5, it/s 0.7 | |
| [0130 17:38:19 @multigpu.py:323] [p0576] step: count(353), step_time 1400.77, mean_step_time 1421.75, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.138402113583 , 0.7 ##### | |
| [u'online', 0.7] | |
| receiving | |
| [0130 17:38:20 @multigpu.py:323] [p0574] step: count(347), step_time 1365.51, mean_step_time 1421.68, it/s 0.7 | |
| [0130 17:38:20 @multigpu.py:323] [p0115] step: count(331), step_time 1395.58, mean_step_time 1422.46, it/s 0.7 | |
| [0130 17:38:20 @multigpu.py:323] [p0576] step: count(354), step_time 1377.34, mean_step_time 1422.89, it/s 0.7 | |
| [0130 17:38:21 @multigpu.py:323] [p0574] step: count(348), step_time 1433.04, mean_step_time 1424.31, it/s 0.7 | |
| [0130 17:38:21 @multigpu.py:323] [p0115] step: count(332), step_time 1404.26, mean_step_time 1423.96, it/s 0.7 | |
| [0130 17:38:21 @multigpu.py:323] [p0576] step: count(355), step_time 1390.11, mean_step_time 1420.46, it/s 0.7 | |
| [0130 17:38:23 @multigpu.py:323] [p0115] step: count(333), step_time 1366.41, mean_step_time 1421.45, it/s 0.7 | |
| [0130 17:38:23 @multigpu.py:323] [p0574] step: count(349), step_time 1398.66, mean_step_time 1423.63, it/s 0.7 | |
| [0130 17:38:23 @multigpu.py:323] [p0576] step: count(356), step_time 1410.44, mean_step_time 1418.59, it/s 0.7 | |
| [0130 17:38:24 @multigpu.py:323] [p0574] step: count(350), step_time 1343.51, mean_step_time 1422.77, it/s 0.7 | |
| [0130 17:38:24 @multigpu.py:323] [p0115] step: count(334), step_time 1460.58, mean_step_time 1425.97, it/s 0.7 | |
| [0130 17:38:24 @multigpu.py:323] [p0576] step: count(357), step_time 1432.73, mean_step_time 1419.37, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.139930734701 , 0.2 ##### | |
| [u'online', 0.2] | |
| receiving | |
| [0130 17:38:25 @multigpu.py:323] [p0574] step: count(351), step_time 1363.59, mean_step_time 1421.37, it/s 0.7 | |
| [0130 17:38:25 @multigpu.py:323] [p0115] step: count(335), step_time 1355.52, mean_step_time 1424.36, it/s 0.7 | |
| [0130 17:38:26 @multigpu.py:323] [p0576] step: count(358), step_time 1470.23, mean_step_time 1420.66, it/s 0.7 | |
| [0130 17:38:27 @multigpu.py:323] [p0574] step: count(352), step_time 1403.23, mean_step_time 1422.83, it/s 0.7 | |
| [0130 17:38:27 @multigpu.py:323] [p0115] step: count(336), step_time 1392.58, mean_step_time 1424.59, it/s 0.7 | |
| [0130 17:38:27 @multigpu.py:323] [p0576] step: count(359), step_time 1365.14, mean_step_time 1416.78, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.140695282751 , 1.5 ##### | |
| [u'online', 1.5] | |
| ##### Sending to neptune: active_workers : 0.140695316063 , 3 ##### | |
| receiving | |
| [0130 17:38:28 @multigpu.py:323] [p0574] step: count(353), step_time 1368.95, mean_step_time 1420.43, it/s 0.7 | |
| [0130 17:38:28 @multigpu.py:323] [p0115] step: count(337), step_time 1372.64, mean_step_time 1424.03, it/s 0.7 | |
| [0130 17:38:28 @multigpu.py:323] [p0576] step: count(360), step_time 1449.26, mean_step_time 1416.78, it/s 0.71 | |
| [0130 17:38:30 @multigpu.py:323] [p0574] step: count(354), step_time 1349.47, mean_step_time 1419.86, it/s 0.7 | |
| [0130 17:38:30 @multigpu.py:323] [p0115] step: count(338), step_time 1391.57, mean_step_time 1424.4, it/s 0.7 | |
| [0130 17:38:30 @multigpu.py:323] [p0576] step: count(361), step_time 1403.23, mean_step_time 1416.92, it/s 0.71 | |
| [0130 17:38:31 @multigpu.py:323] [p0574] step: count(355), step_time 1500.65, mean_step_time 1424.51, it/s 0.7 | |
| [0130 17:38:31 @multigpu.py:323] [p0115] step: count(339), step_time 1386.29, mean_step_time 1426.18, it/s 0.7 | |
| [0130 17:38:31 @multigpu.py:323] [p0576] step: count(362), step_time 1423.1, mean_step_time 1415.68, it/s 0.71 | |
| [0130 17:38:32 @multigpu.py:323] [p0574] step: count(356), step_time 1366.82, mean_step_time 1423.63, it/s 0.7 | |
| [0130 17:38:32 @multigpu.py:323] [p0115] step: count(340), step_time 1403.96, mean_step_time 1424.0, it/s 0.7 | |
| [0130 17:38:33 @multigpu.py:323] [p0576] step: count(363), step_time 1424.65, mean_step_time 1415.98, it/s 0.71 | |
| [0130 17:38:34 @multigpu.py:323] [p0115] step: count(341), step_time 1344.14, mean_step_time 1420.85, it/s 0.7 | |
| [0130 17:38:34 @multigpu.py:323] [p0574] step: count(357), step_time 1401.42, mean_step_time 1425.28, it/s 0.7 | |
| [0130 17:38:34 @multigpu.py:323] [p0576] step: count(364), step_time 1440.01, mean_step_time 1417.47, it/s 0.71 | |
| [0130 17:38:35 @multigpu.py:323] [p0115] step: count(342), step_time 1358.81, mean_step_time 1419.21, it/s 0.7 | |
| [0130 17:38:35 @multigpu.py:323] [p0574] step: count(358), step_time 1376.03, mean_step_time 1424.4, it/s 0.7 | |
| [0130 17:38:36 @multigpu.py:323] [p0576] step: count(365), step_time 1424.0, mean_step_time 1417.6, it/s 0.71 | |
| [0130 17:38:37 @multigpu.py:323] [p0574] step: count(359), step_time 1362.57, mean_step_time 1425.03, it/s 0.7 | |
| [0130 17:38:37 @multigpu.py:323] [p0115] step: count(343), step_time 1408.86, mean_step_time 1420.68, it/s 0.7 | |
| [0130 17:38:37 @multigpu.py:323] [p0576] step: count(366), step_time 1437.07, mean_step_time 1419.12, it/s 0.7 | |
| [0130 17:38:38 @multigpu.py:323] [p0574] step: count(360), step_time 1373.87, mean_step_time 1421.09, it/s 0.7 | |
| [0130 17:38:38 @multigpu.py:323] [p0115] step: count(344), step_time 1415.34, mean_step_time 1422.92, it/s 0.7 | |
| [0130 17:38:38 @multigpu.py:323] [p0576] step: count(367), step_time 1466.44, mean_step_time 1422.59, it/s 0.7 | |
| [0130 17:38:39 @multigpu.py:323] [p0574] step: count(361), step_time 1423.31, mean_step_time 1417.98, it/s 0.71 | |
| [0130 17:38:39 @multigpu.py:323] [p0115] step: count(345), step_time 1372.49, mean_step_time 1424.25, it/s 0.7 | |
| [0130 17:38:40 @multigpu.py:323] [p0576] step: count(368), step_time 1395.91, mean_step_time 1420.64, it/s 0.7 | |
| [0130 17:38:41 @multigpu.py:323] [p0115] step: count(346), step_time 1396.56, mean_step_time 1425.46, it/s 0.7 | |
| [0130 17:38:41 @multigpu.py:323] [p0574] step: count(362), step_time 1432.84, mean_step_time 1422.23, it/s 0.7 | |
| [0130 17:38:41 @multigpu.py:323] [p0576] step: count(369), step_time 1425.34, mean_step_time 1418.04, it/s 0.71 | |
| [0130 17:38:42 @multigpu.py:323] [p0574] step: count(363), step_time 1337.44, mean_step_time 1418.98, it/s 0.7 | |
| [0130 17:38:42 @multigpu.py:323] [p0115] step: count(347), step_time 1367.95, mean_step_time 1422.87, it/s 0.7 | |
| [0130 17:38:43 @multigpu.py:323] [p0576] step: count(370), step_time 1406.6, mean_step_time 1419.59, it/s 0.7 | |
| [0130 17:38:44 @multigpu.py:323] [p0576] step: count(371), step_time 1513.52, mean_step_time 1424.1, it/s 0.7 | |
| [0130 17:38:44 @multigpu.py:323] [p0115] step: count(348), step_time 2106.69, mean_step_time 1424.16, it/s 0.7 | |
| [0130 17:38:44 @multigpu.py:323] [p0574] step: count(364), step_time 2107.26, mean_step_time 1424.15, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.145541653037 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:38:46 @multigpu.py:323] [p0574] step: count(365), step_time 1374.01, mean_step_time 1422.48, it/s 0.7 | |
| [0130 17:38:46 @multigpu.py:323] [p0576] step: count(372), step_time 1403.78, mean_step_time 1422.98, it/s 0.7 | |
| [0130 17:38:46 @multigpu.py:323] [p0115] step: count(349), step_time 1428.43, mean_step_time 1427.09, it/s 0.7 | |
| [0130 17:38:47 @multigpu.py:323] [p0115] step: count(350), step_time 1381.07, mean_step_time 1425.49, it/s 0.7 | |
| [0130 17:38:47 @multigpu.py:323] [p0574] step: count(366), step_time 1463.64, mean_step_time 1427.29, it/s 0.7 | |
| [0130 17:38:47 @multigpu.py:323] [p0576] step: count(373), step_time 1461.16, mean_step_time 1426.0, it/s 0.7 | |
| [0130 17:38:48 @multigpu.py:323] [p0115] step: count(351), step_time 1376.54, mean_step_time 1424.53, it/s 0.7 | |
| [0130 17:38:48 @multigpu.py:323] [p0574] step: count(367), step_time 1424.99, mean_step_time 1430.27, it/s 0.7 | |
| [0130 17:38:48 @multigpu.py:323] [p0576] step: count(374), step_time 1417.18, mean_step_time 1427.99, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.146764651073 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.146777135266 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:38:50 @multigpu.py:323] [p0115] step: count(352), step_time 1377.77, mean_step_time 1423.21, it/s 0.7 | |
| [0130 17:38:50 @multigpu.py:323] [p0574] step: count(368), step_time 1360.8, mean_step_time 1426.65, it/s 0.7 | |
| [0130 17:38:50 @multigpu.py:323] [p0576] step: count(375), step_time 1404.25, mean_step_time 1428.7, it/s 0.7 | |
| [0130 17:38:51 @multigpu.py:323] [p0115] step: count(353), step_time 1405.07, mean_step_time 1425.14, it/s 0.7 | |
| [0130 17:38:51 @multigpu.py:323] [p0574] step: count(369), step_time 1383.47, mean_step_time 1425.89, it/s 0.7 | |
| [0130 17:38:51 @multigpu.py:323] [p0576] step: count(376), step_time 1468.44, mean_step_time 1431.6, it/s 0.7 | |
| [0130 17:38:53 @multigpu.py:323] [p0115] step: count(354), step_time 1378.63, mean_step_time 1421.05, it/s 0.7 | |
| [0130 17:38:53 @multigpu.py:323] [p0574] step: count(370), step_time 1375.83, mean_step_time 1427.51, it/s 0.7 | |
| [0130 17:38:53 @multigpu.py:323] [p0576] step: count(377), step_time 1402.5, mean_step_time 1430.09, it/s 0.7 | |
| [0130 17:38:54 @multigpu.py:323] [p0574] step: count(371), step_time 1368.96, mean_step_time 1427.78, it/s 0.7 | |
| [0130 17:38:54 @multigpu.py:323] [p0115] step: count(355), step_time 1424.46, mean_step_time 1424.49, it/s 0.7 | |
| [0130 17:38:54 @multigpu.py:323] [p0576] step: count(378), step_time 1438.33, mean_step_time 1428.5, it/s 0.7 | |
| [0130 17:38:55 @multigpu.py:323] [p0115] step: count(356), step_time 1400.95, mean_step_time 1424.91, it/s 0.7 | |
| [0130 17:38:55 @multigpu.py:323] [p0574] step: count(372), step_time 1506.86, mean_step_time 1432.96, it/s 0.7 | |
| [0130 17:38:56 @multigpu.py:323] [p0576] step: count(379), step_time 1412.34, mean_step_time 1430.86, it/s 0.7 | |
| [0130 17:38:57 @multigpu.py:323] [p0115] step: count(357), step_time 1399.37, mean_step_time 1426.25, it/s 0.7 | |
| [0130 17:38:57 @multigpu.py:323] [p0574] step: count(373), step_time 1384.96, mean_step_time 1433.76, it/s 0.7 | |
| [0130 17:38:57 @multigpu.py:323] [p0576] step: count(380), step_time 1378.29, mean_step_time 1427.31, it/s 0.7 | |
| [0130 17:38:58 @multigpu.py:323] [p0115] step: count(358), step_time 1350.07, mean_step_time 1424.17, it/s 0.7 | |
| [0130 17:38:58 @multigpu.py:323] [p0574] step: count(374), step_time 1464.46, mean_step_time 1439.51, it/s 0.69 | |
| [0130 17:38:58 @multigpu.py:323] [p0576] step: count(381), step_time 1442.18, mean_step_time 1429.25, it/s 0.7 | |
| [0130 17:39:00 @multigpu.py:323] [p0115] step: count(359), step_time 1409.5, mean_step_time 1425.33, it/s 0.7 | |
| [0130 17:39:00 @multigpu.py:323] [p0574] step: count(375), step_time 1375.47, mean_step_time 1433.25, it/s 0.7 | |
| [0130 17:39:00 @multigpu.py:323] [p0576] step: count(382), step_time 1397.03, mean_step_time 1427.95, it/s 0.7 | |
| [0130 17:39:01 @multigpu.py:323] [p0115] step: count(360), step_time 1412.0, mean_step_time 1425.74, it/s 0.7 | |
| [0130 17:39:01 @multigpu.py:323] [p0574] step: count(376), step_time 1413.74, mean_step_time 1435.6, it/s 0.7 | |
| [0130 17:39:01 @multigpu.py:323] [p0576] step: count(383), step_time 1439.07, mean_step_time 1428.67, it/s 0.7 | |
| [0130 17:39:02 @multigpu.py:323] [p0115] step: count(361), step_time 1429.66, mean_step_time 1430.01, it/s 0.7 | |
| [0130 17:39:02 @multigpu.py:323] [p0574] step: count(377), step_time 1388.78, mean_step_time 1434.96, it/s 0.7 | |
| [0130 17:39:03 @multigpu.py:323] [p0576] step: count(384), step_time 1395.49, mean_step_time 1426.45, it/s 0.7 | |
| [0130 17:39:04 @multigpu.py:323] [p0115] step: count(362), step_time 1366.82, mean_step_time 1430.41, it/s 0.7 | |
| [0130 17:39:04 @multigpu.py:323] [p0574] step: count(378), step_time 1380.51, mean_step_time 1435.19, it/s 0.7 | |
| [0130 17:39:04 @multigpu.py:323] [p0576] step: count(385), step_time 1405.33, mean_step_time 1425.51, it/s 0.7 | |
| [0130 17:39:05 @multigpu.py:323] [p0115] step: count(363), step_time 1430.37, mean_step_time 1431.49, it/s 0.7 | |
| [0130 17:39:05 @multigpu.py:323] [p0574] step: count(379), step_time 1398.73, mean_step_time 1437.0, it/s 0.7 | |
| [0130 17:39:06 @multigpu.py:323] [p0576] step: count(386), step_time 1445.15, mean_step_time 1425.92, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.151504749722 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 17:39:07 @multigpu.py:323] [p0115] step: count(364), step_time 1380.69, mean_step_time 1429.75, it/s 0.7 | |
| [0130 17:39:07 @multigpu.py:323] [p0574] step: count(380), step_time 1397.96, mean_step_time 1438.2, it/s 0.7 | |
| [0130 17:39:07 @multigpu.py:323] [p0576] step: count(387), step_time 1396.97, mean_step_time 1422.44, it/s 0.7 | |
| [0130 17:39:08 @multigpu.py:323] [p0115] step: count(365), step_time 1369.8, mean_step_time 1429.62, it/s 0.7 | |
| [0130 17:39:08 @multigpu.py:323] [p0574] step: count(381), step_time 1393.46, mean_step_time 1436.71, it/s 0.7 | |
| [0130 17:39:08 @multigpu.py:323] [p0576] step: count(388), step_time 1398.0, mean_step_time 1422.55, it/s 0.7 | |
| [0130 17:39:09 @multigpu.py:323] [p0115] step: count(366), step_time 1370.75, mean_step_time 1428.33, it/s 0.7 | |
| [0130 17:39:09 @multigpu.py:323] [p0574] step: count(382), step_time 1404.33, mean_step_time 1435.28, it/s 0.7 | |
| [0130 17:39:10 @multigpu.py:323] [p0576] step: count(389), step_time 1382.74, mean_step_time 1420.42, it/s 0.7 | |
| [0130 17:39:11 @multigpu.py:323] [p0115] step: count(367), step_time 1404.09, mean_step_time 1430.14, it/s 0.7 | |
| [0130 17:39:11 @multigpu.py:323] [p0574] step: count(383), step_time 1458.87, mean_step_time 1441.35, it/s 0.69 | |
| [0130 17:39:11 @multigpu.py:323] [p0576] step: count(390), step_time 1433.6, mean_step_time 1421.77, it/s 0.7 | |
| [0130 17:39:13 @multigpu.py:323] [p0574] step: count(384), step_time 1579.52, mean_step_time 1414.97, it/s 0.71 | |
| [0130 17:39:13 @multigpu.py:323] [p0576] step: count(391), step_time 1378.32, mean_step_time 1415.01, it/s 0.71 | |
| [0130 17:39:13 @multigpu.py:323] [p0115] step: count(368), step_time 1803.48, mean_step_time 1414.97, it/s 0.71 | |
| [0130 17:39:14 @multigpu.py:323] [p0115] step: count(369), step_time 1356.61, mean_step_time 1411.38, it/s 0.71 | |
| [0130 17:39:14 @multigpu.py:323] [p0576] step: count(392), step_time 1430.82, mean_step_time 1416.36, it/s 0.71 | |
| [0130 17:39:14 @multigpu.py:323] [p0574] step: count(385), step_time 1488.43, mean_step_time 1420.69, it/s 0.7 | |
| [0130 17:39:15 @multigpu.py:323] [p0115] step: count(370), step_time 1395.35, mean_step_time 1412.1, it/s 0.71 | |
| [0130 17:39:15 @multigpu.py:323] [p0576] step: count(393), step_time 1378.79, mean_step_time 1412.24, it/s 0.71 | |
| [0130 17:39:15 @multigpu.py:323] [p0574] step: count(386), step_time 1410.11, mean_step_time 1418.01, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.154189984997 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.154235092733 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:39:17 @multigpu.py:323] [p0115] step: count(371), step_time 1412.42, mean_step_time 1413.89, it/s 0.71 | |
| [0130 17:39:17 @multigpu.py:323] [p0576] step: count(394), step_time 1456.54, mean_step_time 1414.21, it/s 0.71 | |
| [0130 17:39:17 @multigpu.py:323] [p0574] step: count(387), step_time 1381.87, mean_step_time 1415.86, it/s 0.71 | |
| [0130 17:39:18 @multigpu.py:323] [p0115] step: count(372), step_time 1387.79, mean_step_time 1414.39, it/s 0.71 | |
| [0130 17:39:18 @multigpu.py:323] [p0574] step: count(388), step_time 1394.24, mean_step_time 1417.53, it/s 0.71 | |
| [0130 17:39:18 @multigpu.py:323] [p0576] step: count(395), step_time 1440.95, mean_step_time 1416.04, it/s 0.71 | |
| [0130 17:39:19 @multigpu.py:323] [p0115] step: count(373), step_time 1417.49, mean_step_time 1415.01, it/s 0.71 | |
| [0130 17:39:20 @multigpu.py:323] [p0574] step: count(389), step_time 1441.07, mean_step_time 1420.41, it/s 0.7 | |
| [0130 17:39:20 @multigpu.py:323] [p0576] step: count(396), step_time 1436.47, mean_step_time 1414.45, it/s 0.71 | |
| [0130 17:39:21 @multigpu.py:323] [p0115] step: count(374), step_time 1368.12, mean_step_time 1414.49, it/s 0.71 | |
| [0130 17:39:21 @multigpu.py:323] [p0574] step: count(390), step_time 1398.35, mean_step_time 1421.53, it/s 0.7 | |
| [0130 17:39:21 @multigpu.py:323] [p0576] step: count(397), step_time 1397.34, mean_step_time 1414.19, it/s 0.71 | |
| [0130 17:39:22 @multigpu.py:323] [p0115] step: count(375), step_time 1369.1, mean_step_time 1411.72, it/s 0.71 | |
| [0130 17:39:22 @multigpu.py:323] [p0574] step: count(391), step_time 1369.83, mean_step_time 1421.58, it/s 0.7 | |
| [0130 17:39:23 @multigpu.py:323] [p0576] step: count(398), step_time 1483.03, mean_step_time 1416.42, it/s 0.71 | |
| [0130 17:39:24 @multigpu.py:323] [p0115] step: count(376), step_time 1407.21, mean_step_time 1412.03, it/s 0.71 | |
| [0130 17:39:24 @multigpu.py:323] [p0574] step: count(392), step_time 1344.19, mean_step_time 1413.44, it/s 0.71 | |
| [0130 17:39:24 @multigpu.py:323] [p0576] step: count(399), step_time 1428.4, mean_step_time 1417.23, it/s 0.71 | |
| [0130 17:39:25 @multigpu.py:323] [p0115] step: count(377), step_time 1352.72, mean_step_time 1409.7, it/s 0.71 | |
| [0130 17:39:25 @multigpu.py:323] [p0574] step: count(393), step_time 1421.02, mean_step_time 1415.25, it/s 0.71 | |
| [0130 17:39:25 @multigpu.py:323] [p0576] step: count(400), step_time 1398.51, mean_step_time 1418.24, it/s 0.71 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.156876291103 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.156876291103 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.156876291103 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.156876804696 , -0.00845347996801 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.156876804696 , 0.143131583929 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.156876804696 , -2.290143013 ##### | |
| ##### Sending to neptune: value_loss : 0.156876804696 , 1.06496596336 ##### | |
| ##### Sending to neptune: advantage : 0.156876804696 , -0.000597213045694 ##### | |
| ##### Sending to neptune: pred_reward : 0.156876804696 , 0.332899808884 ##### | |
| ##### Sending to neptune: max_logit : 0.156876804696 , 0.185626909137 ##### | |
| [u'loss', -0.00845347996801138, 0.1431315839290619, -2.2901430130004883, 1.0649659633636475, -0.000597213045693934, 0.332899808883667, 0.18562690913677216] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.15687728107 , 8742840.19 ##### | |
| ##### Sending to neptune: dp_per_s : 0.15687728107 , 89.5178593025 ##### | |
| [u'other', 8742840.19, 89.51785930253774] | |
| receiving | |
| [0130 17:39:26 @multigpu.py:323] [p0115] step: count(378), step_time 1391.41, mean_step_time 1411.77, it/s 0.71 | |
| [0130 17:39:27 @multigpu.py:323] [p0574] step: count(394), step_time 1364.77, mean_step_time 1410.26, it/s 0.71 | |
| [0130 17:39:27 @multigpu.py:323] [p0576] step: count(401), step_time 1528.51, mean_step_time 1422.55, it/s 0.7 | |
| [0130 17:39:28 @multigpu.py:323] [p0115] step: count(379), step_time 1352.05, mean_step_time 1408.9, it/s 0.71 | |
| [0130 17:39:28 @multigpu.py:323] [p0574] step: count(395), step_time 1368.2, mean_step_time 1409.9, it/s 0.71 | |
| [0130 17:39:28 @multigpu.py:323] [p0576] step: count(402), step_time 1385.77, mean_step_time 1421.99, it/s 0.7 | |
| [0130 17:39:29 @multigpu.py:323] [p0115] step: count(380), step_time 1376.06, mean_step_time 1407.1, it/s 0.71 | |
| [0130 17:39:29 @multigpu.py:323] [p0574] step: count(396), step_time 1420.37, mean_step_time 1410.23, it/s 0.71 | |
| [0130 17:39:30 @multigpu.py:323] [p0576] step: count(403), step_time 1407.63, mean_step_time 1420.42, it/s 0.7 | |
| [0130 17:39:30 @multigpu.py:323] [p0115] step: count(381), step_time 1404.98, mean_step_time 1405.86, it/s 0.71 | |
| [0130 17:39:31 @multigpu.py:323] [p0574] step: count(397), step_time 1370.97, mean_step_time 1409.34, it/s 0.71 | |
| [0130 17:39:31 @multigpu.py:323] [p0576] step: count(404), step_time 1454.24, mean_step_time 1423.36, it/s 0.7 | |
| [0130 17:39:32 @multigpu.py:323] [p0115] step: count(382), step_time 1387.83, mean_step_time 1406.92, it/s 0.71 | |
| [0130 17:39:32 @multigpu.py:323] [p0574] step: count(398), step_time 1396.96, mean_step_time 1410.16, it/s 0.71 | |
| [0130 17:39:33 @multigpu.py:323] [p0576] step: count(405), step_time 1429.75, mean_step_time 1424.58, it/s 0.7 | |
| [0130 17:39:33 @multigpu.py:323] [p0115] step: count(383), step_time 1429.27, mean_step_time 1406.86, it/s 0.71 | |
| [0130 17:39:34 @multigpu.py:323] [p0574] step: count(399), step_time 1423.11, mean_step_time 1411.38, it/s 0.71 | |
| [0130 17:39:34 @multigpu.py:323] [p0576] step: count(406), step_time 1410.93, mean_step_time 1422.87, it/s 0.7 | |
| [0130 17:39:35 @multigpu.py:323] [p0115] step: count(384), step_time 1370.8, mean_step_time 1406.37, it/s 0.71 | |
| [0130 17:39:35 @multigpu.py:323] [p0574] step: count(400), step_time 1392.73, mean_step_time 1411.12, it/s 0.71 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.159525871608 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.159525871608 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.159525871608 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.159526103603 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.159526361095 , -0.00946252513677 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.159526361095 , 0.028890747577 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.159526361095 , -2.29028081894 ##### | |
| ##### Sending to neptune: value_loss : 0.159526361095 , 1.05018687248 ##### | |
| ##### Sending to neptune: advantage : 0.159526361095 , -0.000191484694369 ##### | |
| ##### Sending to neptune: pred_reward : 0.159526361095 , 0.328690111637 ##### | |
| ##### Sending to neptune: max_logit : 0.159526361095 , 0.185199126601 ##### | |
| [u'loss', -0.009462525136768818, 0.02889074757695198, -2.290280818939209, 1.0501868724822998, -0.0001914846943691373, 0.3286901116371155, 0.18519912660121918] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.159526825216 , 8762594.07 ##### | |
| ##### Sending to neptune: dp_per_s : 0.159526825216 , 90.186045923 ##### | |
| [u'other', 8762594.07, 90.18604592303011] | |
| receiving | |
| [0130 17:39:35 @multigpu.py:323] [p0576] step: count(407), step_time 1431.07, mean_step_time 1424.57, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.15983586106 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:39:36 @multigpu.py:323] [p0115] step: count(385), step_time 1370.4, mean_step_time 1406.4, it/s 0.71 | |
| [0130 17:39:36 @multigpu.py:323] [p0574] step: count(401), step_time 1408.05, mean_step_time 1411.85, it/s 0.71 | |
| [0130 17:39:37 @multigpu.py:323] [p0576] step: count(408), step_time 1494.54, mean_step_time 1429.4, it/s 0.7 | |
| [0130 17:39:38 @multigpu.py:323] [p0115] step: count(386), step_time 1452.71, mean_step_time 1410.49, it/s 0.71 | |
| [0130 17:39:38 @multigpu.py:323] [p0574] step: count(402), step_time 1369.83, mean_step_time 1410.12, it/s 0.71 | |
| [0130 17:39:38 @multigpu.py:323] [p0576] step: count(409), step_time 1383.21, mean_step_time 1429.42, it/s 0.7 | |
| [0130 17:39:39 @multigpu.py:323] [p0115] step: count(387), step_time 1360.87, mean_step_time 1408.33, it/s 0.71 | |
| [0130 17:39:39 @multigpu.py:323] [p0574] step: count(403), step_time 1416.83, mean_step_time 1408.02, it/s 0.71 | |
| [0130 17:39:40 @multigpu.py:323] [p0576] step: count(410), step_time 1437.49, mean_step_time 1429.62, it/s 0.7 | |
| [0130 17:39:41 @multigpu.py:323] [p0576] step: count(411), step_time 1399.93, mean_step_time 1430.7, it/s 0.7 | |
| [0130 17:39:41 @multigpu.py:323] [p0574] step: count(404), step_time 2033.89, mean_step_time 1430.74, it/s 0.7 | |
| [0130 17:39:41 @multigpu.py:323] [p0115] step: count(388), step_time 2251.8, mean_step_time 1430.75, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.161397429109 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:39:43 @multigpu.py:323] [p0576] step: count(412), step_time 1401.71, mean_step_time 1429.24, it/s 0.7 | |
| [0130 17:39:43 @multigpu.py:323] [p0115] step: count(389), step_time 1446.82, mean_step_time 1435.26, it/s 0.7 | |
| [0130 17:39:43 @multigpu.py:323] [p0574] step: count(405), step_time 1498.12, mean_step_time 1431.22, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.161699656049 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:39:44 @multigpu.py:323] [p0576] step: count(413), step_time 1411.56, mean_step_time 1430.88, it/s 0.7 | |
| [0130 17:39:44 @multigpu.py:323] [p0115] step: count(390), step_time 1435.59, mean_step_time 1437.27, it/s 0.7 | |
| [0130 17:39:44 @multigpu.py:323] [p0574] step: count(406), step_time 1426.94, mean_step_time 1432.07, it/s 0.7 | |
| [0130 17:39:45 @multigpu.py:323] [p0576] step: count(414), step_time 1422.95, mean_step_time 1429.2, it/s 0.7 | |
| [0130 17:39:45 @multigpu.py:323] [p0115] step: count(391), step_time 1368.46, mean_step_time 1435.07, it/s 0.7 | |
| [0130 17:39:45 @multigpu.py:323] [p0574] step: count(407), step_time 1412.92, mean_step_time 1433.62, it/s 0.7 | |
| [0130 17:39:47 @multigpu.py:323] [p0576] step: count(415), step_time 1430.9, mean_step_time 1428.7, it/s 0.7 | |
| [0130 17:39:47 @multigpu.py:323] [p0115] step: count(392), step_time 1425.59, mean_step_time 1436.96, it/s 0.7 | |
| [0130 17:39:47 @multigpu.py:323] [p0574] step: count(408), step_time 1395.29, mean_step_time 1433.67, it/s 0.7 | |
| [0130 17:39:48 @multigpu.py:323] [p0115] step: count(393), step_time 1358.57, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:39:48 @multigpu.py:323] [p0576] step: count(416), step_time 1435.64, mean_step_time 1428.66, it/s 0.7 | |
| [0130 17:39:48 @multigpu.py:323] [p0574] step: count(409), step_time 1427.05, mean_step_time 1432.97, it/s 0.7 | |
| [0130 17:39:50 @multigpu.py:323] [p0115] step: count(394), step_time 1398.43, mean_step_time 1435.53, it/s 0.7 | |
| [0130 17:39:50 @multigpu.py:323] [p0576] step: count(417), step_time 1410.86, mean_step_time 1429.33, it/s 0.7 | |
| [0130 17:39:50 @multigpu.py:323] [p0574] step: count(410), step_time 1422.03, mean_step_time 1434.16, it/s 0.7 | |
| [0130 17:39:51 @multigpu.py:323] [p0115] step: count(395), step_time 1386.76, mean_step_time 1436.42, it/s 0.7 | |
| [0130 17:39:51 @multigpu.py:323] [p0576] step: count(418), step_time 1420.38, mean_step_time 1426.2, it/s 0.7 | |
| [0130 17:39:51 @multigpu.py:323] [p0574] step: count(411), step_time 1371.37, mean_step_time 1434.23, it/s 0.7 | |
| [0130 17:39:52 @multigpu.py:323] [p0115] step: count(396), step_time 1371.74, mean_step_time 1434.64, it/s 0.7 | |
| [0130 17:39:53 @multigpu.py:323] [p0574] step: count(412), step_time 1423.33, mean_step_time 1438.19, it/s 0.7 | |
| [0130 17:39:53 @multigpu.py:323] [p0576] step: count(419), step_time 1508.1, mean_step_time 1430.18, it/s 0.7 | |
| [0130 17:39:54 @multigpu.py:323] [p0115] step: count(397), step_time 1408.93, mean_step_time 1437.45, it/s 0.7 | |
| [0130 17:39:54 @multigpu.py:323] [p0574] step: count(413), step_time 1415.17, mean_step_time 1437.9, it/s 0.7 | |
| [0130 17:39:54 @multigpu.py:323] [p0576] step: count(420), step_time 1399.22, mean_step_time 1430.22, it/s 0.7 | |
| [0130 17:39:55 @multigpu.py:323] [p0115] step: count(398), step_time 1413.22, mean_step_time 1438.54, it/s 0.7 | |
| [0130 17:39:55 @multigpu.py:323] [p0574] step: count(414), step_time 1410.68, mean_step_time 1440.19, it/s 0.69 | |
| [0130 17:39:56 @multigpu.py:323] [p0576] step: count(421), step_time 1536.37, mean_step_time 1430.61, it/s 0.7 | |
| [0130 17:39:57 @multigpu.py:323] [p0115] step: count(399), step_time 1444.12, mean_step_time 1443.15, it/s 0.69 | |
| [0130 17:39:57 @multigpu.py:323] [p0574] step: count(415), step_time 1390.2, mean_step_time 1441.29, it/s 0.69 | |
| [0130 17:39:57 @multigpu.py:323] [p0576] step: count(422), step_time 1412.54, mean_step_time 1431.95, it/s 0.7 | |
| [0130 17:39:58 @multigpu.py:323] [p0115] step: count(400), step_time 1427.37, mean_step_time 1445.71, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.165948813889 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.165948813889 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.165948813889 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.165949346357 , -0.00503255799413 ##### | |
| ##### Sending to neptune: policy_loss : 0.165949346357 , 0.491833776236 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.165949346357 , -2.29054236412 ##### | |
| ##### Sending to neptune: value_loss : 0.165949346357 , 1.15454125404 ##### | |
| ##### Sending to neptune: advantage : 0.165949346357 , -0.00217221933417 ##### | |
| ##### Sending to neptune: pred_reward : 0.165949346357 , 0.325955986977 ##### | |
| ##### Sending to neptune: max_logit : 0.165949346357 , 0.184464037418 ##### | |
| [u'loss', -0.0050325579941272736, 0.49183377623558044, -2.2905423641204834, 1.154541254043579, -0.0021722193341702223, 0.32595598697662354, 0.18446403741836548] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.165949839685 , 8744566.58 ##### | |
| ##### Sending to neptune: dp_per_s : 0.165949839685 , 89.9175458849 ##### | |
| [u'other', 8744566.58, 89.91754588494861] | |
| receiving | |
| [0130 17:39:58 @multigpu.py:323] [p0574] step: count(416), step_time 1385.62, mean_step_time 1439.55, it/s 0.69 | |
| [0130 17:39:58 @multigpu.py:323] [p0576] step: count(423), step_time 1421.15, mean_step_time 1432.63, it/s 0.7 | |
| [0130 17:39:59 @multigpu.py:323] [p0115] step: count(401), step_time 1441.04, mean_step_time 1447.52, it/s 0.69 | |
| [0130 17:39:59 @multigpu.py:323] [p0574] step: count(417), step_time 1377.82, mean_step_time 1439.9, it/s 0.69 | |
| [0130 17:40:00 @multigpu.py:323] [p0576] step: count(424), step_time 1440.47, mean_step_time 1431.94, it/s 0.7 | |
| [0130 17:40:01 @multigpu.py:323] [p0115] step: count(402), step_time 1358.7, mean_step_time 1446.06, it/s 0.69 | |
| [0130 17:40:01 @multigpu.py:323] [p0574] step: count(418), step_time 1439.77, mean_step_time 1442.04, it/s 0.69 | |
| [0130 17:40:01 @multigpu.py:323] [p0576] step: count(425), step_time 1406.44, mean_step_time 1430.77, it/s 0.7 | |
| [0130 17:40:02 @multigpu.py:323] [p0115] step: count(403), step_time 1423.27, mean_step_time 1445.76, it/s 0.69 | |
| [0130 17:40:02 @multigpu.py:323] [p0574] step: count(419), step_time 1406.12, mean_step_time 1441.19, it/s 0.69 | |
| [0130 17:40:03 @multigpu.py:323] [p0576] step: count(426), step_time 1398.42, mean_step_time 1430.15, it/s 0.7 | |
| [0130 17:40:04 @multigpu.py:323] [p0115] step: count(404), step_time 1408.34, mean_step_time 1447.64, it/s 0.69 | |
| [0130 17:40:04 @multigpu.py:323] [p0574] step: count(420), step_time 1404.68, mean_step_time 1441.79, it/s 0.69 | |
| [0130 17:40:04 @multigpu.py:323] [p0576] step: count(427), step_time 1447.22, mean_step_time 1430.95, it/s 0.7 | |
| [0130 17:40:05 @multigpu.py:323] [p0115] step: count(405), step_time 1429.56, mean_step_time 1450.6, it/s 0.69 | |
| [0130 17:40:05 @multigpu.py:323] [p0574] step: count(421), step_time 1381.4, mean_step_time 1440.45, it/s 0.69 | |
| [0130 17:40:05 @multigpu.py:323] [p0576] step: count(428), step_time 1413.83, mean_step_time 1426.92, it/s 0.7 | |
| [0130 17:40:06 @multigpu.py:323] [p0115] step: count(406), step_time 1383.02, mean_step_time 1447.11, it/s 0.69 | |
| [0130 17:40:07 @multigpu.py:323] [p0574] step: count(422), step_time 1432.03, mean_step_time 1443.56, it/s 0.69 | |
| [0130 17:40:07 @multigpu.py:323] [p0576] step: count(429), step_time 1435.3, mean_step_time 1429.52, it/s 0.7 | |
| [0130 17:40:08 @multigpu.py:323] [p0115] step: count(407), step_time 1403.32, mean_step_time 1449.23, it/s 0.69 | |
| [0130 17:40:08 @multigpu.py:323] [p0574] step: count(423), step_time 1403.98, mean_step_time 1442.92, it/s 0.69 | |
| 42%|####2 |423/1000[1[0130 17:40:08 @multigpu.py:323] [p0576] step: count(430), step_time 1396.29, mean_step_time 1427.46, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.168805354701 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.168823744721 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:40:10 @multigpu.py:323] [p0576] step: count(431), step_time 1448.01, mean_step_time 1429.87, it/s 0.7 | |
| 43%|####3 |431/1000[1[0130 17:40:10 @multigpu.py:323] [p0574] step: count(424), step_time 1774.42, mean_step_time 1429.95, it/s 0.7 | |
| [0130 17:40:10 @multigpu.py:323] [p0115] step: count(408), step_time 1866.32, mean_step_time 1429.96, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.169260831343 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:40:11 @multigpu.py:323] [p0576] step: count(432), step_time 1369.59, mean_step_time 1428.26, it/s 0.7 | |
| [0130 17:40:11 @multigpu.py:323] [p0574] step: count(425), step_time 1395.36, mean_step_time 1424.81, it/s 0.7 | |
| [0130 17:40:11 @multigpu.py:323] [p0115] step: count(409), step_time 1451.41, mean_step_time 1430.19, it/s 0.7 | |
| [0130 17:40:13 @multigpu.py:323] [p0574] step: count(426), step_time 1415.79, mean_step_time 1424.25, it/s 0.7 | |
| [0130 17:40:13 @multigpu.py:323] [p0115] step: count(410), step_time 1391.93, mean_step_time 1428.01, it/s 0.7 | |
| [0130 17:40:13 @multigpu.py:323] [p0576] step: count(433), step_time 1563.09, mean_step_time 1435.84, it/s 0.7 | |
| [0130 17:40:14 @multigpu.py:323] [p0115] step: count(411), step_time 1372.63, mean_step_time 1428.21, it/s 0.7 | |
| [0130 17:40:14 @multigpu.py:323] [p0574] step: count(427), step_time 1437.07, mean_step_time 1425.46, it/s 0.7 | |
| [0130 17:40:14 @multigpu.py:323] [p0576] step: count(434), step_time 1453.46, mean_step_time 1437.36, it/s 0.7 | |
| [0130 17:40:15 @multigpu.py:323] [p0115] step: count(412), step_time 1360.42, mean_step_time 1424.96, it/s 0.7 | |
| [0130 17:40:15 @multigpu.py:323] [p0574] step: count(428), step_time 1463.36, mean_step_time 1428.86, it/s 0.7 | |
| [0130 17:40:16 @multigpu.py:323] [p0576] step: count(435), step_time 1442.92, mean_step_time 1437.97, it/s 0.7 | |
| [0130 17:40:17 @multigpu.py:323] [p0115] step: count(413), step_time 1390.98, mean_step_time 1426.58, it/s 0.7 | |
| [0130 17:40:17 @multigpu.py:323] [p0574] step: count(429), step_time 1381.28, mean_step_time 1426.57, it/s 0.7 | |
| [0130 17:40:17 @multigpu.py:323] [p0576] step: count(436), step_time 1397.1, mean_step_time 1436.04, it/s 0.7 | |
| [0130 17:40:18 @multigpu.py:323] [p0115] step: count(414), step_time 1367.71, mean_step_time 1425.04, it/s 0.7 | |
| [0130 17:40:18 @multigpu.py:323] [p0574] step: count(430), step_time 1453.28, mean_step_time 1428.14, it/s 0.7 | |
| [0130 17:40:18 @multigpu.py:323] [p0576] step: count(437), step_time 1508.3, mean_step_time 1440.91, it/s 0.69 | |
| [0130 17:40:20 @multigpu.py:323] [p0115] step: count(415), step_time 1425.78, mean_step_time 1426.99, it/s 0.7 | |
| [0130 17:40:20 @multigpu.py:323] [p0574] step: count(431), step_time 1398.99, mean_step_time 1429.52, it/s 0.7 | |
| [0130 17:40:20 @multigpu.py:323] [p0576] step: count(438), step_time 1407.49, mean_step_time 1440.27, it/s 0.69 | |
| [0130 17:40:21 @multigpu.py:323] [p0115] step: count(416), step_time 1336.99, mean_step_time 1425.25, it/s 0.7 | |
| [0130 17:40:21 @multigpu.py:323] [p0574] step: count(432), step_time 1432.96, mean_step_time 1430.0, it/s 0.7 | |
| [0130 17:40:21 @multigpu.py:323] [p0576] step: count(439), step_time 1407.52, mean_step_time 1435.24, it/s 0.7 | |
| [0130 17:40:22 @multigpu.py:323] [p0115] step: count(417), step_time 1408.77, mean_step_time 1425.25, it/s 0.7 | |
| [0130 17:40:23 @multigpu.py:323] [p0574] step: count(433), step_time 1437.57, mean_step_time 1431.12, it/s 0.7 | |
| [0130 17:40:23 @multigpu.py:323] [p0576] step: count(440), step_time 1469.58, mean_step_time 1438.75, it/s 0.7 | |
| [0130 17:40:24 @multigpu.py:323] [p0115] step: count(418), step_time 1355.93, mean_step_time 1422.38, it/s 0.7 | |
| [0130 17:40:24 @multigpu.py:323] [p0574] step: count(434), step_time 1321.66, mean_step_time 1426.67, it/s 0.7 | |
| [0130 17:40:24 @multigpu.py:323] [p0576] step: count(441), step_time 1463.11, mean_step_time 1435.09, it/s 0.7 | |
| [0130 17:40:25 @multigpu.py:323] [p0115] step: count(419), step_time 1403.54, mean_step_time 1420.35, it/s 0.7 | |
| [0130 17:40:25 @multigpu.py:323] [p0574] step: count(435), step_time 1510.72, mean_step_time 1432.69, it/s 0.7 | |
| [0130 17:40:26 @multigpu.py:323] [p0576] step: count(442), step_time 1466.47, mean_step_time 1437.79, it/s 0.7 | |
| [0130 17:40:26 @multigpu.py:323] [p0115] step: count(420), step_time 1437.86, mean_step_time 1420.88, it/s 0.7 | |
| [0130 17:40:27 @multigpu.py:323] [p0574] step: count(436), step_time 1422.17, mean_step_time 1434.52, it/s 0.7 | |
| [0130 17:40:27 @multigpu.py:323] [p0576] step: count(443), step_time 1403.78, mean_step_time 1436.92, it/s 0.7 | |
| [0130 17:40:28 @multigpu.py:323] [p0115] step: count(421), step_time 1481.53, mean_step_time 1422.9, it/s 0.7 | |
| [0130 17:40:28 @multigpu.py:323] [p0574] step: count(437), step_time 1404.15, mean_step_time 1435.84, it/s 0.7 | |
| [0130 17:40:29 @multigpu.py:323] [p0576] step: count(444), step_time 1448.02, mean_step_time 1437.3, it/s 0.7 | |
| [0130 17:40:29 @multigpu.py:323] [p0115] step: count(422), step_time 1408.9, mean_step_time 1425.41, it/s 0.7 | |
| [0130 17:40:30 @multigpu.py:323] [p0574] step: count(438), step_time 1427.92, mean_step_time 1435.25, it/s 0.7 | |
| [0130 17:40:30 @multigpu.py:323] [p0576] step: count(445), step_time 1463.08, mean_step_time 1440.13, it/s 0.69 | |
| [0130 17:40:31 @multigpu.py:323] [p0115] step: count(423), step_time 1399.13, mean_step_time 1424.2, it/s 0.7 | |
| [0130 17:40:31 @multigpu.py:323] [p0574] step: count(439), step_time 1418.53, mean_step_time 1435.87, it/s 0.7 | |
| [0130 17:40:31 @multigpu.py:323] [p0576] step: count(446), step_time 1432.68, mean_step_time 1441.84, it/s 0.69 | |
| [0130 17:40:32 @multigpu.py:323] [p0115] step: count(424), step_time 1430.48, mean_step_time 1425.31, it/s 0.7 | |
| [0130 17:40:32 @multigpu.py:323] [p0574] step: count(440), step_time 1376.25, mean_step_time 1434.45, it/s 0.7 | |
| [0130 17:40:33 @multigpu.py:323] [p0576] step: count(447), step_time 1395.59, mean_step_time 1439.26, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.175806029704 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 17:40:34 @multigpu.py:323] [p0115] step: count(425), step_time 1367.57, mean_step_time 1422.21, it/s 0.7 | |
| [0130 17:40:34 @multigpu.py:323] [p0574] step: count(441), step_time 1522.2, mean_step_time 1441.49, it/s 0.69 | |
| [0130 17:40:34 @multigpu.py:323] [p0576] step: count(448), step_time 1427.95, mean_step_time 1439.97, it/s 0.69 | |
| [0130 17:40:35 @multigpu.py:323] [p0115] step: count(426), step_time 1404.57, mean_step_time 1423.29, it/s 0.7 | |
| [0130 17:40:35 @multigpu.py:323] [p0574] step: count(442), step_time 1472.0, mean_step_time 1443.48, it/s 0.69 | |
| [0130 17:40:36 @multigpu.py:323] [p0576] step: count(449), step_time 1448.3, mean_step_time 1440.62, it/s 0.69 | |
| [0130 17:40:36 @multigpu.py:323] [p0115] step: count(427), step_time 1345.11, mean_step_time 1420.38, it/s 0.7 | |
| [0130 17:40:37 @multigpu.py:323] [p0574] step: count(443), step_time 1446.05, mean_step_time 1445.59, it/s 0.69 | |
| [0130 17:40:37 @multigpu.py:323] [p0576] step: count(450), step_time 1419.29, mean_step_time 1441.77, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.176930492719 , 1.4 ##### | |
| [u'online', 1.4] | |
| ##### Sending to neptune: active_workers : 0.176930545833 , 3 ##### | |
| receiving | |
| [0130 17:40:39 @multigpu.py:323] [p0576] step: count(451), step_time 1455.66, mean_step_time 1442.15, it/s 0.69 | |
| [0130 17:40:39 @multigpu.py:323] [p0115] step: count(428), step_time 2300.21, mean_step_time 1442.07, it/s 0.69 | |
| [0130 17:40:39 @multigpu.py:323] [p0574] step: count(444), step_time 1705.61, mean_step_time 1442.15, it/s 0.69 | |
| 43%|####2 |428/1000[1[0130 17:40:40 @multigpu.py:323] [p0115] step: count(429), step_time 1387.77, mean_step_time 1438.89, it/s 0.69 | |
| [0130 17:40:40 @multigpu.py:323] [p0574] step: count(445), step_time 1435.97, mean_step_time 1444.18, it/s 0.69 | |
| [0130 17:40:40 @multigpu.py:323] [p0576] step: count(452), step_time 1460.36, mean_step_time 1446.69, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.177810673846 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:40:41 @multigpu.py:323] [p0574] step: count(446), step_time 1355.41, mean_step_time 1441.16, it/s 0.69 | |
| [0130 17:40:41 @multigpu.py:323] [p0115] step: count(430), step_time 1422.15, mean_step_time 1440.4, it/s 0.69 | |
| [0130 17:40:42 @multigpu.py:323] [p0576] step: count(453), step_time 1466.27, mean_step_time 1441.85, it/s 0.69 | |
| [0130 17:40:43 @multigpu.py:323] [p0574] step: count(447), step_time 1443.97, mean_step_time 1441.5, it/s 0.69 | |
| [0130 17:40:43 @multigpu.py:323] [p0115] step: count(431), step_time 1449.53, mean_step_time 1444.25, it/s 0.69 | |
| [0130 17:40:43 @multigpu.py:323] [p0576] step: count(454), step_time 1419.22, mean_step_time 1440.13, it/s 0.69 | |
| [0130 17:40:44 @multigpu.py:323] [p0574] step: count(448), step_time 1394.73, mean_step_time 1438.07, it/s 0.7 | |
| [0130 17:40:44 @multigpu.py:323] [p0115] step: count(432), step_time 1402.93, mean_step_time 1446.37, it/s 0.69 | |
| [0130 17:40:44 @multigpu.py:323] [p0576] step: count(455), step_time 1422.21, mean_step_time 1439.1, it/s 0.69 | |
| [0130 17:40:46 @multigpu.py:323] [p0115] step: count(433), step_time 1391.4, mean_step_time 1446.39, it/s 0.69 | |
| [0130 17:40:46 @multigpu.py:323] [p0574] step: count(449), step_time 1448.14, mean_step_time 1441.42, it/s 0.69 | |
| [0130 17:40:46 @multigpu.py:323] [p0576] step: count(456), step_time 1384.99, mean_step_time 1438.49, it/s 0.7 | |
| [0130 17:40:47 @multigpu.py:323] [p0115] step: count(434), step_time 1393.72, mean_step_time 1447.69, it/s 0.69 | |
| [0130 17:40:47 @multigpu.py:323] [p0574] step: count(450), step_time 1379.57, mean_step_time 1437.73, it/s 0.7 | |
| [0130 17:40:47 @multigpu.py:323] [p0576] step: count(457), step_time 1415.17, mean_step_time 1433.84, it/s 0.7 | |
| [0130 17:40:48 @multigpu.py:323] [p0574] step: count(451), step_time 1423.27, mean_step_time 1438.94, it/s 0.69 | |
| [0130 17:40:48 @multigpu.py:323] [p0115] step: count(435), step_time 1451.37, mean_step_time 1448.97, it/s 0.69 | |
| [0130 17:40:49 @multigpu.py:323] [p0576] step: count(458), step_time 1438.82, mean_step_time 1435.4, it/s 0.7 | |
| [0130 17:40:50 @multigpu.py:323] [p0574] step: count(452), step_time 1390.49, mean_step_time 1436.82, it/s 0.7 | |
| [0130 17:40:50 @multigpu.py:323] [p0115] step: count(436), step_time 1378.98, mean_step_time 1451.07, it/s 0.69 | |
| [0130 17:40:50 @multigpu.py:323] [p0576] step: count(459), step_time 1444.31, mean_step_time 1437.24, it/s 0.7 | |
| [0130 17:40:51 @multigpu.py:323] [p0574] step: count(453), step_time 1400.46, mean_step_time 1434.96, it/s 0.7 | |
| [0130 17:40:51 @multigpu.py:323] [p0115] step: count(437), step_time 1464.35, mean_step_time 1453.85, it/s 0.69 | |
| [0130 17:40:51 @multigpu.py:323] [p0576] step: count(460), step_time 1397.47, mean_step_time 1433.64, it/s 0.7 | |
| [0130 17:40:53 @multigpu.py:323] [p0574] step: count(454), step_time 1389.83, mean_step_time 1438.37, it/s 0.7 | |
| [0130 17:40:53 @multigpu.py:323] [p0115] step: count(438), step_time 1365.71, mean_step_time 1454.34, it/s 0.69 | |
| [0130 17:40:53 @multigpu.py:323] [p0576] step: count(461), step_time 1436.6, mean_step_time 1432.31, it/s 0.7 | |
| [0130 17:40:54 @multigpu.py:323] [p0574] step: count(455), step_time 1378.02, mean_step_time 1431.74, it/s 0.7 | |
| [0130 17:40:54 @multigpu.py:323] [p0115] step: count(439), step_time 1415.1, mean_step_time 1454.92, it/s 0.69 | |
| [0130 17:40:54 @multigpu.py:323] [p0576] step: count(462), step_time 1447.98, mean_step_time 1431.39, it/s 0.7 | |
| [0130 17:40:55 @multigpu.py:323] [p0574] step: count(456), step_time 1394.3, mean_step_time 1430.34, it/s 0.7 | |
| [0130 17:40:56 @multigpu.py:323] [p0115] step: count(440), step_time 1380.95, mean_step_time 1452.07, it/s 0.69 | |
| [0130 17:40:56 @multigpu.py:323] [p0576] step: count(463), step_time 1442.55, mean_step_time 1433.33, it/s 0.7 | |
| [0130 17:40:57 @multigpu.py:323] [p0574] step: count(457), step_time 1398.26, mean_step_time 1430.05, it/s 0.7 | |
| [0130 17:40:57 @multigpu.py:323] [p0115] step: count(441), step_time 1458.1, mean_step_time 1450.9, it/s 0.69 | |
| [0130 17:40:57 @multigpu.py:323] [p0576] step: count(464), step_time 1360.36, mean_step_time 1428.94, it/s 0.7 | |
| [0130 17:40:58 @multigpu.py:323] [p0574] step: count(458), step_time 1472.43, mean_step_time 1432.27, it/s 0.7 | |
| [0130 17:40:58 @multigpu.py:323] [p0115] step: count(442), step_time 1427.88, mean_step_time 1451.85, it/s 0.69 | |
| [0130 17:40:59 @multigpu.py:323] [p0576] step: count(465), step_time 1417.51, mean_step_time 1426.66, it/s 0.7 | |
| [0130 17:41:00 @multigpu.py:323] [p0574] step: count(459), step_time 1415.56, mean_step_time 1432.13, it/s 0.7 | |
| [0130 17:41:00 @multigpu.py:323] [p0115] step: count(443), step_time 1447.21, mean_step_time 1454.26, it/s 0.69 | |
| [0130 17:41:00 @multigpu.py:323] [p0576] step: count(466), step_time 1473.78, mean_step_time 1428.72, it/s 0.7 | |
| [0130 17:41:01 @multigpu.py:323] [p0574] step: count(460), step_time 1463.47, mean_step_time 1436.49, it/s 0.7 | |
| [0130 17:41:01 @multigpu.py:323] [p0115] step: count(444), step_time 1474.85, mean_step_time 1456.47, it/s 0.69 | |
| [0130 17:41:01 @multigpu.py:323] [p0576] step: count(467), step_time 1465.99, mean_step_time 1432.24, it/s 0.7 | |
| [0130 17:41:03 @multigpu.py:323] [p0574] step: count(461), step_time 1431.21, mean_step_time 1431.94, it/s 0.7 | |
| [0130 17:41:03 @multigpu.py:323] [p0115] step: count(445), step_time 1452.57, mean_step_time 1460.72, it/s 0.68 | |
| [0130 17:41:03 @multigpu.py:323] [p0576] step: count(468), step_time 1414.81, mean_step_time 1431.58, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.184176382754 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:41:04 @multigpu.py:323] [p0574] step: count(462), step_time 1385.28, mean_step_time 1427.6, it/s 0.7 | |
| [0130 17:41:04 @multigpu.py:323] [p0115] step: count(446), step_time 1353.37, mean_step_time 1458.16, it/s 0.69 | |
| [0130 17:41:04 @multigpu.py:323] [p0576] step: count(469), step_time 1445.93, mean_step_time 1431.46, it/s 0.7 | |
| [0130 17:41:05 @multigpu.py:323] [p0574] step: count(463), step_time 1400.82, mean_step_time 1425.34, it/s 0.7 | |
| [0130 17:41:06 @multigpu.py:323] [p0115] step: count(447), step_time 1450.28, mean_step_time 1463.42, it/s 0.68 | |
| [0130 17:41:06 @multigpu.py:323] [p0576] step: count(470), step_time 1445.53, mean_step_time 1432.78, it/s 0.7 | |
| [0130 17:41:07 @multigpu.py:323] [p0576] step: count(471), step_time 1439.44, mean_step_time 1431.97, it/s 0.7 | |
| [0130 17:41:07 @multigpu.py:323] [p0115] step: count(448), step_time 1671.13, mean_step_time 1431.97, it/s 0.7 | |
| [0130 17:41:07 @multigpu.py:323] [p0574] step: count(464), step_time 1838.23, mean_step_time 1431.97, it/s 0.7 | |
| [0130 17:41:09 @multigpu.py:323] [p0576] step: count(472), step_time 1414.98, mean_step_time 1429.7, it/s 0.7 | |
| [0130 17:41:09 @multigpu.py:323] [p0574] step: count(465), step_time 1447.22, mean_step_time 1432.53, it/s 0.7 | |
| [0130 17:41:09 @multigpu.py:323] [p0115] step: count(449), step_time 1450.3, mean_step_time 1435.09, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.185665526655 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.185782292485 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:41:10 @multigpu.py:323] [p0115] step: count(450), step_time 1355.29, mean_step_time 1431.75, it/s 0.7 | |
| [0130 17:41:10 @multigpu.py:323] [p0574] step: count(466), step_time 1452.54, mean_step_time 1437.39, it/s 0.7 | |
| [0130 17:41:10 @multigpu.py:323] [p0576] step: count(473), step_time 1488.28, mean_step_time 1430.8, it/s 0.7 | |
| [0130 17:41:12 @multigpu.py:323] [p0115] step: count(451), step_time 1474.42, mean_step_time 1433.0, it/s 0.7 | |
| [0130 17:41:12 @multigpu.py:323] [p0574] step: count(467), step_time 1381.35, mean_step_time 1434.26, it/s 0.7 | |
| [0130 17:41:12 @multigpu.py:323] [p0576] step: count(474), step_time 1435.59, mean_step_time 1431.62, it/s 0.7 | |
| [0130 17:41:13 @multigpu.py:323] [p0574] step: count(468), step_time 1443.75, mean_step_time 1436.71, it/s 0.7 | |
| [0130 17:41:13 @multigpu.py:323] [p0115] step: count(452), step_time 1461.72, mean_step_time 1435.93, it/s 0.7 | |
| [0130 17:41:13 @multigpu.py:323] [p0576] step: count(475), step_time 1461.87, mean_step_time 1433.6, it/s 0.7 | |
| [0130 17:41:14 @multigpu.py:323] [p0574] step: count(469), step_time 1376.0, mean_step_time 1433.1, it/s 0.7 | |
| [0130 17:41:14 @multigpu.py:323] [p0115] step: count(453), step_time 1415.55, mean_step_time 1437.14, it/s 0.7 | |
| [0130 17:41:14 @multigpu.py:323] [p0576] step: count(476), step_time 1399.07, mean_step_time 1434.3, it/s 0.7 | |
| [0130 17:41:16 @multigpu.py:323] [p0574] step: count(470), step_time 1423.37, mean_step_time 1435.29, it/s 0.7 | |
| [0130 17:41:16 @multigpu.py:323] [p0115] step: count(454), step_time 1441.31, mean_step_time 1439.52, it/s 0.69 | |
| [0130 17:41:16 @multigpu.py:323] [p0576] step: count(477), step_time 1432.48, mean_step_time 1435.17, it/s 0.7 | |
| [0130 17:41:17 @multigpu.py:323] [p0574] step: count(471), step_time 1408.02, mean_step_time 1434.53, it/s 0.7 | |
| [0130 17:41:17 @multigpu.py:323] [p0115] step: count(455), step_time 1413.35, mean_step_time 1437.62, it/s 0.7 | |
| [0130 17:41:17 @multigpu.py:323] [p0576] step: count(478), step_time 1450.45, mean_step_time 1435.75, it/s 0.7 | |
| [0130 17:41:19 @multigpu.py:323] [p0574] step: count(472), step_time 1383.97, mean_step_time 1434.2, it/s 0.7 | |
| [0130 17:41:19 @multigpu.py:323] [p0115] step: count(456), step_time 1441.25, mean_step_time 1440.73, it/s 0.69 | |
| [0130 17:41:19 @multigpu.py:323] [p0576] step: count(479), step_time 1452.88, mean_step_time 1436.18, it/s 0.7 | |
| [0130 17:41:20 @multigpu.py:323] [p0574] step: count(473), step_time 1496.55, mean_step_time 1439.01, it/s 0.69 | |
| [0130 17:41:20 @multigpu.py:323] [p0115] step: count(457), step_time 1428.56, mean_step_time 1438.94, it/s 0.69 | |
| [0130 17:41:20 @multigpu.py:323] [p0576] step: count(480), step_time 1485.43, mean_step_time 1440.58, it/s 0.69 | |
| [0130 17:41:22 @multigpu.py:323] [p0574] step: count(474), step_time 1446.31, mean_step_time 1441.83, it/s 0.69 | |
| [0130 17:41:22 @multigpu.py:323] [p0115] step: count(458), step_time 1387.74, mean_step_time 1440.05, it/s 0.69 | |
| [0130 17:41:22 @multigpu.py:323] [p0576] step: count(481), step_time 1456.43, mean_step_time 1441.57, it/s 0.69 | |
| [0130 17:41:23 @multigpu.py:323] [p0574] step: count(475), step_time 1402.51, mean_step_time 1443.06, it/s 0.69 | |
| [0130 17:41:23 @multigpu.py:323] [p0115] step: count(459), step_time 1490.29, mean_step_time 1443.81, it/s 0.69 | |
| [0130 17:41:23 @multigpu.py:323] [p0576] step: count(482), step_time 1437.77, mean_step_time 1441.06, it/s 0.69 | |
| [0130 17:41:24 @multigpu.py:323] [p0574] step: count(476), step_time 1461.35, mean_step_time 1446.41, it/s 0.69 | |
| [0130 17:41:24 @multigpu.py:323] [p0115] step: count(460), step_time 1413.76, mean_step_time 1445.45, it/s 0.69 | |
| [0130 17:41:25 @multigpu.py:323] [p0576] step: count(483), step_time 1371.29, mean_step_time 1437.49, it/s 0.7 | |
| [0130 17:41:26 @multigpu.py:323] [p0574] step: count(477), step_time 1460.21, mean_step_time 1449.51, it/s 0.69 | |
| [0130 17:41:26 @multigpu.py:323] [p0115] step: count(461), step_time 1452.77, mean_step_time 1445.18, it/s 0.69 | |
| [0130 17:41:26 @multigpu.py:323] [p0576] step: count(484), step_time 1469.12, mean_step_time 1442.93, it/s 0.69 | |
| [0130 17:41:27 @multigpu.py:323] [p0574] step: count(478), step_time 1411.56, mean_step_time 1446.46, it/s 0.69 | |
| [0130 17:41:27 @multigpu.py:323] [p0115] step: count(462), step_time 1413.91, mean_step_time 1444.48, it/s 0.69 | |
| [0130 17:41:27 @multigpu.py:323] [p0576] step: count(485), step_time 1386.82, mean_step_time 1441.4, it/s 0.69 | |
| [0130 17:41:29 @multigpu.py:323] [p0574] step: count(479), step_time 1431.48, mean_step_time 1447.26, it/s 0.69 | |
| [0130 17:41:29 @multigpu.py:323] [p0115] step: count(463), step_time 1387.01, mean_step_time 1441.47, it/s 0.69 | |
| [0130 17:41:29 @multigpu.py:323] [p0576] step: count(486), step_time 1379.58, mean_step_time 1436.69, it/s 0.7 | |
| [0130 17:41:30 @multigpu.py:323] [p0115] step: count(464), step_time 1401.1, mean_step_time 1437.78, it/s 0.7 | |
| [0130 17:41:30 @multigpu.py:323] [p0574] step: count(480), step_time 1439.94, mean_step_time 1446.08, it/s 0.69 | |
| [0130 17:41:30 @multigpu.py:323] [p0576] step: count(487), step_time 1441.15, mean_step_time 1435.45, it/s 0.7 | |
| [0130 17:41:31 @multigpu.py:323] [p0115] step: count(465), step_time 1415.79, mean_step_time 1435.95, it/s 0.7 | |
| [0130 17:41:32 @multigpu.py:323] [p0574] step: count(481), step_time 1418.57, mean_step_time 1445.45, it/s 0.69 | |
| [0130 17:41:32 @multigpu.py:323] [p0576] step: count(488), step_time 1418.84, mean_step_time 1435.65, it/s 0.7 | |
| [0130 17:41:33 @multigpu.py:323] [p0115] step: count(466), step_time 1443.08, mean_step_time 1440.43, it/s 0.69 | |
| [0130 17:41:33 @multigpu.py:323] [p0574] step: count(482), step_time 1420.62, mean_step_time 1447.22, it/s 0.69 | |
| [0130 17:41:33 @multigpu.py:323] [p0576] step: count(489), step_time 1418.78, mean_step_time 1434.29, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.192464135223 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:41:34 @multigpu.py:323] [p0574] step: count(483), step_time 1385.07, mean_step_time 1446.43, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.192708853881 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:41:34 @multigpu.py:323] [p0576] step: count(490), step_time 1362.98, mean_step_time 1430.16, it/s 0.7 | |
| [0130 17:41:34 @multigpu.py:323] [p0115] step: count(467), step_time 1487.79, mean_step_time 1442.31, it/s 0.69 | |
| [0130 17:41:36 @multigpu.py:323] [p0115] step: count(468), step_time 1426.91, mean_step_time 1430.1, it/s 0.7 | |
| [0130 17:41:36 @multigpu.py:323] [p0574] step: count(484), step_time 1515.46, mean_step_time 1430.29, it/s 0.7 | |
| [0130 17:41:36 @multigpu.py:323] [p0576] step: count(491), step_time 1442.33, mean_step_time 1430.31, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.193346670535 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:41:37 @multigpu.py:323] [p0115] step: count(469), step_time 1406.09, mean_step_time 1427.89, it/s 0.7 | |
| [0130 17:41:37 @multigpu.py:323] [p0574] step: count(485), step_time 1420.3, mean_step_time 1428.95, it/s 0.7 | |
| [0130 17:41:37 @multigpu.py:323] [p0576] step: count(492), step_time 1433.82, mean_step_time 1431.25, it/s 0.7 | |
| [0130 17:41:39 @multigpu.py:323] [p0115] step: count(470), step_time 1469.25, mean_step_time 1433.58, it/s 0.7 | |
| [0130 17:41:39 @multigpu.py:323] [p0574] step: count(486), step_time 1462.15, mean_step_time 1429.43, it/s 0.7 | |
| [0130 17:41:39 @multigpu.py:323] [p0576] step: count(493), step_time 1465.1, mean_step_time 1430.09, it/s 0.7 | |
| [0130 17:41:40 @multigpu.py:323] [p0574] step: count(487), step_time 1401.71, mean_step_time 1430.44, it/s 0.7 | |
| [0130 17:41:40 @multigpu.py:323] [p0115] step: count(471), step_time 1413.02, mean_step_time 1430.51, it/s 0.7 | |
| [0130 17:41:40 @multigpu.py:323] [p0576] step: count(494), step_time 1428.58, mean_step_time 1429.74, it/s 0.7 | |
| [0130 17:41:42 @multigpu.py:323] [p0574] step: count(488), step_time 1397.83, mean_step_time 1428.15, it/s 0.7 | |
| [0130 17:41:42 @multigpu.py:323] [p0115] step: count(472), step_time 1402.81, mean_step_time 1427.57, it/s 0.7 | |
| [0130 17:41:42 @multigpu.py:323] [p0576] step: count(495), step_time 1393.27, mean_step_time 1426.31, it/s 0.7 | |
| [0130 17:41:43 @multigpu.py:323] [p0574] step: count(489), step_time 1419.44, mean_step_time 1430.32, it/s 0.7 | |
| [0130 17:41:43 @multigpu.py:323] [p0576] step: count(496), step_time 1402.95, mean_step_time 1426.5, it/s 0.7 | |
| [0130 17:41:43 @multigpu.py:323] [p0115] step: count(473), step_time 1439.46, mean_step_time 1428.76, it/s 0.7 | |
| [0130 17:41:44 @multigpu.py:323] [p0574] step: count(490), step_time 1411.47, mean_step_time 1429.73, it/s 0.7 | |
| [0130 17:41:44 @multigpu.py:323] [p0576] step: count(497), step_time 1391.16, mean_step_time 1424.44, it/s 0.7 | |
| [0130 17:41:44 @multigpu.py:323] [p0115] step: count(474), step_time 1432.71, mean_step_time 1428.33, it/s 0.7 | |
| [0130 17:41:46 @multigpu.py:323] [p0576] step: count(498), step_time 1395.99, mean_step_time 1421.71, it/s 0.7 | |
| [0130 17:41:46 @multigpu.py:323] [p0574] step: count(491), step_time 1425.6, mean_step_time 1430.6, it/s 0.7 | |
| [0130 17:41:46 @multigpu.py:323] [p0115] step: count(475), step_time 1449.3, mean_step_time 1430.13, it/s 0.7 | |
| [0130 17:41:47 @multigpu.py:323] [p0574] step: count(492), step_time 1396.91, mean_step_time 1431.25, it/s 0.7 | |
| [0130 17:41:47 @multigpu.py:323] [p0576] step: count(499), step_time 1446.81, mean_step_time 1421.41, it/s 0.7 | |
| [0130 17:41:47 @multigpu.py:323] [p0115] step: count(476), step_time 1454.84, mean_step_time 1430.81, it/s 0.7 | |
| [0130 17:41:49 @multigpu.py:323] [p0574] step: count(493), step_time 1403.47, mean_step_time 1426.6, it/s 0.7 | |
| [0130 17:41:49 @multigpu.py:323] [p0576] step: count(500), step_time 1426.58, mean_step_time 1418.47, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.19667889244 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.19667889244 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.19667889244 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.196679146356 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.196679431332 , -0.00439015356824 ##### | |
| ##### Sending to neptune: policy_loss : 0.196679431332 , 0.453959643841 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.196679431332 , -2.29136276245 ##### | |
| ##### Sending to neptune: value_loss : 0.196679431332 , 1.27546346188 ##### | |
| ##### Sending to neptune: advantage : 0.196679431332 , -0.00191708060447 ##### | |
| ##### Sending to neptune: pred_reward : 0.196679431332 , 0.342677921057 ##### | |
| ##### Sending to neptune: max_logit : 0.196679431332 , 0.181843400002 ##### | |
| [u'loss', -0.00439015356823802, 0.4539596438407898, -2.291362762451172, 1.2754634618759155, -0.0019170806044712663, 0.34267792105674744, 0.18184340000152588] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.19667988552 , 8799167.4 ##### | |
| ##### Sending to neptune: dp_per_s : 0.19667988552 , 89.3428788891 ##### | |
| [u'other', 8799167.4, 89.34287888906876] | |
| receiving | |
| [0130 17:41:49 @multigpu.py:323] [p0115] step: count(477), step_time 1390.74, mean_step_time 1428.92, it/s 0.7 | |
| [0130 17:41:50 @multigpu.py:323] [p0574] step: count(494), step_time 1432.89, mean_step_time 1425.93, it/s 0.7 | |
| [0130 17:41:50 @multigpu.py:323] [p0115] step: count(478), step_time 1369.78, mean_step_time 1428.02, it/s 0.7 | |
| [0130 17:41:50 @multigpu.py:323] [p0576] step: count(501), step_time 1493.99, mean_step_time 1420.34, it/s 0.7 | |
| [0130 17:41:51 @multigpu.py:323] [p0574] step: count(495), step_time 1441.23, mean_step_time 1427.86, it/s 0.7 | |
| [0130 17:41:51 @multigpu.py:323] [p0115] step: count(479), step_time 1400.71, mean_step_time 1423.54, it/s 0.7 | |
| [0130 17:41:52 @multigpu.py:323] [p0576] step: count(502), step_time 1473.53, mean_step_time 1422.13, it/s 0.7 | |
| [0130 17:41:53 @multigpu.py:323] [p0574] step: count(496), step_time 1426.57, mean_step_time 1426.12, it/s 0.7 | |
| [0130 17:41:53 @multigpu.py:323] [p0115] step: count(480), step_time 1464.67, mean_step_time 1426.09, it/s 0.7 | |
| [0130 17:41:53 @multigpu.py:323] [p0576] step: count(503), step_time 1417.0, mean_step_time 1424.42, it/s 0.7 | |
| [0130 17:41:54 @multigpu.py:323] [p0574] step: count(497), step_time 1475.96, mean_step_time 1426.91, it/s 0.7 | |
| [0130 17:41:54 @multigpu.py:323] [p0115] step: count(481), step_time 1456.84, mean_step_time 1426.29, it/s 0.7 | |
| [0130 17:41:54 @multigpu.py:323] [p0576] step: count(504), step_time 1443.88, mean_step_time 1423.16, it/s 0.7 | |
| [0130 17:41:56 @multigpu.py:323] [p0574] step: count(498), step_time 1423.72, mean_step_time 1427.52, it/s 0.7 | |
| [0130 17:41:56 @multigpu.py:323] [p0115] step: count(482), step_time 1441.86, mean_step_time 1427.69, it/s 0.7 | |
| [0130 17:41:56 @multigpu.py:323] [p0576] step: count(505), step_time 1450.04, mean_step_time 1426.32, it/s 0.7 | |
| [0130 17:41:57 @multigpu.py:323] [p0574] step: count(499), step_time 1483.14, mean_step_time 1430.1, it/s 0.7 | |
| [0130 17:41:57 @multigpu.py:323] [p0115] step: count(483), step_time 1466.56, mean_step_time 1431.67, it/s 0.7 | |
| [0130 17:41:57 @multigpu.py:323] [p0576] step: count(506), step_time 1446.31, mean_step_time 1429.65, it/s 0.7 | |
| [0130 17:41:59 @multigpu.py:323] [p0115] step: count(484), step_time 1380.52, mean_step_time 1430.64, it/s 0.7 | |
| [0130 17:41:59 @multigpu.py:323] [p0574] step: count(500), step_time 1416.52, mean_step_time 1428.93, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.199472724663 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.199472724663 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.199472724663 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.199473265807 , -0.00781606975943 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.199473265807 , 0.0337311848998 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.199473265807 , -2.29143595695 ##### | |
| ##### Sending to neptune: value_loss : 0.199473265807 , 1.2572479248 ##### | |
| ##### Sending to neptune: advantage : 0.199473265807 , -3.6175846617e-05 ##### | |
| ##### Sending to neptune: pred_reward : 0.199473265807 , 0.347458809614 ##### | |
| ##### Sending to neptune: max_logit : 0.199473265807 , 0.181484788656 ##### | |
| [u'loss', -0.007816069759428501, 0.033731184899806976, -2.291435956954956, 1.2572479248046875, -3.6175846616970375e-05, 0.3474588096141815, 0.18148478865623474] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.199473697742 , 8790208.01 ##### | |
| ##### Sending to neptune: dp_per_s : 0.199473697742 , 89.2401421731 ##### | |
| [u'other', 8790208.01, 89.24014217307855] | |
| receiving | |
| [0130 17:41:59 @multigpu.py:323] [p0576] step: count(507), step_time 1433.45, mean_step_time 1429.27, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.199656606052 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:42:00 @multigpu.py:323] [p0574] step: count(501), step_time 1400.88, mean_step_time 1428.05, it/s 0.7 | |
| [0130 17:42:00 @multigpu.py:323] [p0115] step: count(485), step_time 1468.38, mean_step_time 1433.27, it/s 0.7 | |
| [0130 17:42:00 @multigpu.py:323] [p0576] step: count(508), step_time 1441.14, mean_step_time 1430.38, it/s 0.7 | |
| [0130 17:42:01 @multigpu.py:323] [p0574] step: count(502), step_time 1372.72, mean_step_time 1425.65, it/s 0.7 | |
| [0130 17:42:02 @multigpu.py:323] [p0115] step: count(486), step_time 1390.88, mean_step_time 1430.66, it/s 0.7 | |
| [0130 17:42:02 @multigpu.py:323] [p0576] step: count(509), step_time 1419.81, mean_step_time 1430.44, it/s 0.7 | |
| [0130 17:42:03 @multigpu.py:323] [p0574] step: count(503), step_time 1453.52, mean_step_time 1429.07, it/s 0.7 | |
| [0130 17:42:03 @multigpu.py:323] [p0115] step: count(487), step_time 1401.45, mean_step_time 1426.34, it/s 0.7 | |
| [0130 17:42:03 @multigpu.py:323] [p0576] step: count(510), step_time 1404.39, mean_step_time 1432.51, it/s 0.7 | |
| [0130 17:42:05 @multigpu.py:323] [p0115] step: count(488), step_time 1536.76, mean_step_time 1431.83, it/s 0.7 | |
| [0130 17:42:05 @multigpu.py:323] [p0574] step: count(504), step_time 1569.74, mean_step_time 1431.79, it/s 0.7 | |
| [0130 17:42:05 @multigpu.py:323] [p0576] step: count(511), step_time 1427.58, mean_step_time 1431.77, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.201240341332 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:42:06 @multigpu.py:323] [p0574] step: count(505), step_time 1407.0, mean_step_time 1431.12, it/s 0.7 | |
| [0130 17:42:06 @multigpu.py:323] [p0115] step: count(489), step_time 1416.23, mean_step_time 1432.34, it/s 0.7 | |
| [0130 17:42:06 @multigpu.py:323] [p0576] step: count(512), step_time 1428.95, mean_step_time 1431.52, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.201575732165 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:42:07 @multigpu.py:323] [p0115] step: count(490), step_time 1366.42, mean_step_time 1427.2, it/s 0.7 | |
| [0130 17:42:07 @multigpu.py:323] [p0576] step: count(513), step_time 1370.6, mean_step_time 1426.8, it/s 0.7 | |
| [0130 17:42:07 @multigpu.py:323] [p0574] step: count(506), step_time 1396.16, mean_step_time 1427.82, it/s 0.7 | |
| [0130 17:42:09 @multigpu.py:323] [p0115] step: count(491), step_time 1405.04, mean_step_time 1426.8, it/s 0.7 | |
| [0130 17:42:09 @multigpu.py:323] [p0574] step: count(507), step_time 1438.12, mean_step_time 1429.64, it/s 0.7 | |
| [0130 17:42:09 @multigpu.py:323] [p0576] step: count(514), step_time 1453.14, mean_step_time 1428.03, it/s 0.7 | |
| [0130 17:42:10 @multigpu.py:323] [p0115] step: count(492), step_time 1436.25, mean_step_time 1428.47, it/s 0.7 | |
| [0130 17:42:10 @multigpu.py:323] [p0576] step: count(515), step_time 1375.54, mean_step_time 1427.14, it/s 0.7 | |
| [0130 17:42:10 @multigpu.py:323] [p0574] step: count(508), step_time 1403.88, mean_step_time 1429.95, it/s 0.7 | |
| [0130 17:42:12 @multigpu.py:323] [p0115] step: count(493), step_time 1390.78, mean_step_time 1426.04, it/s 0.7 | |
| [0130 17:42:12 @multigpu.py:323] [p0576] step: count(516), step_time 1406.27, mean_step_time 1427.31, it/s 0.7 | |
| [0130 17:42:12 @multigpu.py:323] [p0574] step: count(509), step_time 1413.02, mean_step_time 1429.63, it/s 0.7 | |
| [0130 17:42:13 @multigpu.py:323] [p0576] step: count(517), step_time 1391.98, mean_step_time 1427.35, it/s 0.7 | |
| [0130 17:42:13 @multigpu.py:323] [p0115] step: count(494), step_time 1422.91, mean_step_time 1425.55, it/s 0.7 | |
| [0130 17:42:13 @multigpu.py:323] [p0574] step: count(510), step_time 1383.5, mean_step_time 1428.23, it/s 0.7 | |
| [0130 17:42:14 @multigpu.py:323] [p0576] step: count(518), step_time 1410.97, mean_step_time 1428.1, it/s 0.7 | |
| [0130 17:42:14 @multigpu.py:323] [p0574] step: count(511), step_time 1423.99, mean_step_time 1428.15, it/s 0.7 | |
| [0130 17:42:14 @multigpu.py:323] [p0115] step: count(495), step_time 1461.77, mean_step_time 1426.17, it/s 0.7 | |
| [0130 17:42:16 @multigpu.py:323] [p0574] step: count(512), step_time 1455.42, mean_step_time 1431.07, it/s 0.7 | |
| [0130 17:42:16 @multigpu.py:323] [p0576] step: count(519), step_time 1486.84, mean_step_time 1430.1, it/s 0.7 | |
| [0130 17:42:16 @multigpu.py:323] [p0115] step: count(496), step_time 1454.06, mean_step_time 1426.13, it/s 0.7 | |
| [0130 17:42:17 @multigpu.py:323] [p0574] step: count(513), step_time 1404.82, mean_step_time 1431.14, it/s 0.7 | |
| [0130 17:42:17 @multigpu.py:323] [p0576] step: count(520), step_time 1438.83, mean_step_time 1430.71, it/s 0.7 | |
| [0130 17:42:17 @multigpu.py:323] [p0115] step: count(497), step_time 1474.65, mean_step_time 1430.33, it/s 0.7 | |
| [0130 17:42:19 @multigpu.py:323] [p0574] step: count(514), step_time 1431.24, mean_step_time 1431.06, it/s 0.7 | |
| [0130 17:42:19 @multigpu.py:323] [p0576] step: count(521), step_time 1396.57, mean_step_time 1425.84, it/s 0.7 | |
| [0130 17:42:19 @multigpu.py:323] [p0115] step: count(498), step_time 1381.51, mean_step_time 1430.91, it/s 0.7 | |
| [0130 17:42:20 @multigpu.py:323] [p0576] step: count(522), step_time 1417.83, mean_step_time 1423.06, it/s 0.7 | |
| [0130 17:42:20 @multigpu.py:323] [p0574] step: count(515), step_time 1437.48, mean_step_time 1430.87, it/s 0.7 | |
| [0130 17:42:20 @multigpu.py:323] [p0115] step: count(499), step_time 1391.41, mean_step_time 1430.45, it/s 0.7 | |
| [0130 17:42:22 @multigpu.py:323] [p0574] step: count(516), step_time 1395.2, mean_step_time 1429.3, it/s 0.7 | |
| [0130 17:42:22 @multigpu.py:323] [p0576] step: count(523), step_time 1412.15, mean_step_time 1422.81, it/s 0.7 | |
| [0130 17:42:22 @multigpu.py:323] [p0115] step: count(500), step_time 1470.12, mean_step_time 1430.72, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.205827626652 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.205827626652 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.205827626652 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.205828117198 , -0.0121910655871 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.205828117198 , -0.40860247612 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.205828117198 , -2.29158711433 ##### | |
| ##### Sending to neptune: value_loss : 0.205828117198 , 1.1397330761 ##### | |
| ##### Sending to neptune: advantage : 0.205828117198 , 0.00180070789065 ##### | |
| ##### Sending to neptune: pred_reward : 0.205828117198 , 0.358477056026 ##### | |
| ##### Sending to neptune: max_logit : 0.205828117198 , 0.181056678295 ##### | |
| [u'loss', -0.012191065587103367, -0.4086024761199951, -2.2915871143341064, 1.139733076095581, 0.0018007078906521201, 0.35847705602645874, 0.1810566782951355] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.205828589996 , 8779679.08 ##### | |
| ##### Sending to neptune: dp_per_s : 0.205828589996 , 89.1530482324 ##### | |
| [u'other', 8779679.08, 89.15304823242096] | |
| receiving | |
| [0130 17:42:23 @multigpu.py:323] [p0574] step: count(517), step_time 1393.95, mean_step_time 1425.2, it/s 0.7 | |
| [0130 17:42:23 @multigpu.py:323] [p0576] step: count(524), step_time 1418.22, mean_step_time 1421.53, it/s 0.7 | |
| [0130 17:42:23 @multigpu.py:323] [p0115] step: count(501), step_time 1392.3, mean_step_time 1427.49, it/s 0.7 | |
| [0130 17:42:24 @multigpu.py:323] [p0574] step: count(518), step_time 1433.31, mean_step_time 1425.68, it/s 0.7 | |
| [0130 17:42:24 @multigpu.py:323] [p0576] step: count(525), step_time 1478.57, mean_step_time 1422.96, it/s 0.7 | |
| [0130 17:42:24 @multigpu.py:323] [p0115] step: count(502), step_time 1453.34, mean_step_time 1428.07, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.206725753281 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:42:26 @multigpu.py:323] [p0574] step: count(519), step_time 1437.16, mean_step_time 1423.38, it/s 0.7 | |
| [0130 17:42:26 @multigpu.py:323] [p0576] step: count(526), step_time 1381.81, mean_step_time 1419.73, it/s 0.7 | |
| [0130 17:42:26 @multigpu.py:323] [p0115] step: count(503), step_time 1398.28, mean_step_time 1424.65, it/s 0.7 | |
| [0130 17:42:27 @multigpu.py:323] [p0576] step: count(527), step_time 1381.54, mean_step_time 1417.14, it/s 0.71 | |
| [0130 17:42:27 @multigpu.py:323] [p0574] step: count(520), step_time 1449.36, mean_step_time 1425.02, it/s 0.7 | |
| [0130 17:42:27 @multigpu.py:323] [p0115] step: count(504), step_time 1423.18, mean_step_time 1426.79, it/s 0.7 | |
| [0130 17:42:29 @multigpu.py:323] [p0576] step: count(528), step_time 1415.51, mean_step_time 1415.86, it/s 0.71 | |
| [0130 17:42:29 @multigpu.py:323] [p0115] step: count(505), step_time 1373.35, mean_step_time 1422.04, it/s 0.7 | |
| [0130 17:42:29 @multigpu.py:323] [p0574] step: count(521), step_time 1408.41, mean_step_time 1425.4, it/s 0.7 | |
| [0130 17:42:30 @multigpu.py:323] [p0576] step: count(529), step_time 1428.52, mean_step_time 1416.29, it/s 0.71 | |
| [0130 17:42:30 @multigpu.py:323] [p0574] step: count(522), step_time 1431.4, mean_step_time 1428.33, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| [0130 17:42:30 @multigpu.py:323] [p0115] step: count(506), step_time 1448.29, mean_step_time 1424.91, it/s 0.7 | |
| ##### Sending to neptune: online_score : 0.208189289702 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:42:31 @multigpu.py:323] [p0576] step: count(530), step_time 1397.03, mean_step_time 1415.92, it/s 0.71 | |
| [0130 17:42:32 @multigpu.py:323] [p0115] step: count(507), step_time 1437.19, mean_step_time 1426.69, it/s 0.7 | |
| [0130 17:42:32 @multigpu.py:323] [p0574] step: count(523), step_time 1454.59, mean_step_time 1428.39, it/s 0.7 | |
| [0130 17:42:33 @multigpu.py:323] [p0574] step: count(524), step_time 1493.74, mean_step_time 1424.59, it/s 0.7 | |
| [0130 17:42:33 @multigpu.py:323] [p0576] step: count(531), step_time 1604.96, mean_step_time 1424.79, it/s 0.7 | |
| [0130 17:42:33 @multigpu.py:323] [p0115] step: count(508), step_time 1498.7, mean_step_time 1424.79, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.209143604438 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:42:34 @multigpu.py:323] [p0574] step: count(525), step_time 1396.72, mean_step_time 1424.07, it/s 0.7 | |
| [0130 17:42:34 @multigpu.py:323] [p0115] step: count(509), step_time 1457.94, mean_step_time 1426.88, it/s 0.7 | |
| [0130 17:42:34 @multigpu.py:323] [p0576] step: count(532), step_time 1484.47, mean_step_time 1427.57, it/s 0.7 | |
| [0130 17:42:36 @multigpu.py:323] [p0574] step: count(526), step_time 1379.54, mean_step_time 1423.24, it/s 0.7 | |
| [0130 17:42:36 @multigpu.py:323] [p0115] step: count(510), step_time 1384.93, mean_step_time 1427.8, it/s 0.7 | |
| [0130 17:42:36 @multigpu.py:323] [p0576] step: count(533), step_time 1391.0, mean_step_time 1428.59, it/s 0.7 | |
| [0130 17:42:37 @multigpu.py:323] [p0574] step: count(527), step_time 1455.82, mean_step_time 1424.13, it/s 0.7 | |
| [0130 17:42:37 @multigpu.py:323] [p0576] step: count(534), step_time 1399.56, mean_step_time 1425.91, it/s 0.7 | |
| [0130 17:42:37 @multigpu.py:323] [p0115] step: count(511), step_time 1458.11, mean_step_time 1430.45, it/s 0.7 | |
| [0130 17:42:39 @multigpu.py:323] [p0115] step: count(512), step_time 1391.15, mean_step_time 1428.2, it/s 0.7 | |
| [0130 17:42:39 @multigpu.py:323] [p0574] step: count(528), step_time 1511.41, mean_step_time 1429.5, it/s 0.7 | |
| [0130 17:42:39 @multigpu.py:323] [p0576] step: count(535), step_time 1479.66, mean_step_time 1431.11, it/s 0.7 | |
| [0130 17:42:40 @multigpu.py:323] [p0576] step: count(536), step_time 1398.88, mean_step_time 1430.74, it/s 0.7 | |
| [0130 17:42:40 @multigpu.py:323] [p0115] step: count(513), step_time 1464.78, mean_step_time 1431.9, it/s 0.7 | |
| [0130 17:42:40 @multigpu.py:323] [p0574] step: count(529), step_time 1445.78, mean_step_time 1431.14, it/s 0.7 | |
| [0130 17:42:42 @multigpu.py:323] [p0115] step: count(514), step_time 1391.88, mean_step_time 1430.35, it/s 0.7 | |
| [0130 17:42:42 @multigpu.py:323] [p0574] step: count(530), step_time 1430.96, mean_step_time 1433.52, it/s 0.7 | |
| [0130 17:42:42 @multigpu.py:323] [p0576] step: count(537), step_time 1485.03, mean_step_time 1435.4, it/s 0.7 | |
| [0130 17:42:43 @multigpu.py:323] [p0115] step: count(515), step_time 1505.01, mean_step_time 1432.51, it/s 0.7 | |
| [0130 17:42:43 @multigpu.py:323] [p0574] step: count(531), step_time 1441.49, mean_step_time 1434.39, it/s 0.7 | |
| [0130 17:42:43 @multigpu.py:323] [p0576] step: count(538), step_time 1469.73, mean_step_time 1438.34, it/s 0.7 | |
| [0130 17:42:44 @multigpu.py:323] [p0115] step: count(516), step_time 1386.08, mean_step_time 1429.11, it/s 0.7 | |
| [0130 17:42:44 @multigpu.py:323] [p0574] step: count(532), step_time 1381.01, mean_step_time 1430.67, it/s 0.7 | |
| [0130 17:42:44 @multigpu.py:323] [p0576] step: count(539), step_time 1360.79, mean_step_time 1432.03, it/s 0.7 | |
| [0130 17:42:46 @multigpu.py:323] [p0574] step: count(533), step_time 1399.77, mean_step_time 1430.42, it/s 0.7 | |
| [0130 17:42:46 @multigpu.py:323] [p0576] step: count(540), step_time 1413.46, mean_step_time 1430.76, it/s 0.7 | |
| [0130 17:42:46 @multigpu.py:323] [p0115] step: count(517), step_time 1497.86, mean_step_time 1430.27, it/s 0.7 | |
| [0130 17:42:47 @multigpu.py:323] [p0574] step: count(534), step_time 1418.43, mean_step_time 1429.78, it/s 0.7 | |
| [0130 17:42:47 @multigpu.py:323] [p0576] step: count(541), step_time 1464.92, mean_step_time 1434.18, it/s 0.7 | |
| [0130 17:42:47 @multigpu.py:323] [p0115] step: count(518), step_time 1473.97, mean_step_time 1434.89, it/s 0.7 | |
| [0130 17:42:49 @multigpu.py:323] [p0574] step: count(535), step_time 1418.27, mean_step_time 1428.82, it/s 0.7 | |
| [0130 17:42:49 @multigpu.py:323] [p0576] step: count(542), step_time 1427.37, mean_step_time 1434.66, it/s 0.7 | |
| [0130 17:42:49 @multigpu.py:323] [p0115] step: count(519), step_time 1454.96, mean_step_time 1438.07, it/s 0.7 | |
| [0130 17:42:50 @multigpu.py:323] [p0574] step: count(536), step_time 1423.84, mean_step_time 1430.25, it/s 0.7 | |
| [0130 17:42:50 @multigpu.py:323] [p0576] step: count(543), step_time 1444.12, mean_step_time 1436.26, it/s 0.7 | |
| [0130 17:42:50 @multigpu.py:323] [p0115] step: count(520), step_time 1469.49, mean_step_time 1438.04, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.213829125497 , 1.3 ##### | |
| [u'online', 1.3] | |
| ##### Sending to neptune: active_workers : 0.213829183843 , 3 ##### | |
| receiving | |
| [0130 17:42:51 @multigpu.py:323] [p0574] step: count(537), step_time 1378.7, mean_step_time 1429.49, it/s 0.7 | |
| [0130 17:42:52 @multigpu.py:323] [p0576] step: count(544), step_time 1363.11, mean_step_time 1433.5, it/s 0.7 | |
| [0130 17:42:52 @multigpu.py:323] [p0115] step: count(521), step_time 1388.45, mean_step_time 1437.85, it/s 0.7 | |
| [0130 17:42:53 @multigpu.py:323] [p0574] step: count(538), step_time 1451.19, mean_step_time 1430.38, it/s 0.7 | |
| [0130 17:42:53 @multigpu.py:323] [p0576] step: count(545), step_time 1422.98, mean_step_time 1430.72, it/s 0.7 | |
| [0130 17:42:53 @multigpu.py:323] [p0115] step: count(522), step_time 1421.59, mean_step_time 1436.26, it/s 0.7 | |
| [0130 17:42:54 @multigpu.py:323] [p0574] step: count(539), step_time 1439.49, mean_step_time 1430.5, it/s 0.7 | |
| [0130 17:42:54 @multigpu.py:323] [p0576] step: count(546), step_time 1419.38, mean_step_time 1432.6, it/s 0.7 | |
| [0130 17:42:55 @multigpu.py:323] [p0115] step: count(523), step_time 1437.96, mean_step_time 1438.24, it/s 0.7 | |
| [0130 17:42:56 @multigpu.py:323] [p0574] step: count(540), step_time 1451.98, mean_step_time 1430.63, it/s 0.7 | |
| [0130 17:42:56 @multigpu.py:323] [p0576] step: count(547), step_time 1419.85, mean_step_time 1434.52, it/s 0.7 | |
| [0130 17:42:56 @multigpu.py:323] [p0115] step: count(524), step_time 1419.43, mean_step_time 1438.06, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.21545850469 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 17:42:57 @multigpu.py:323] [p0574] step: count(541), step_time 1421.9, mean_step_time 1431.3, it/s 0.7 | |
| [0130 17:42:57 @multigpu.py:323] [p0576] step: count(548), step_time 1434.48, mean_step_time 1435.46, it/s 0.7 | |
| [0130 17:42:57 @multigpu.py:323] [p0115] step: count(525), step_time 1418.32, mean_step_time 1440.3, it/s 0.69 | |
| [0130 17:42:59 @multigpu.py:323] [p0574] step: count(542), step_time 1432.74, mean_step_time 1431.37, it/s 0.7 | |
| [0130 17:42:59 @multigpu.py:323] [p0576] step: count(549), step_time 1452.17, mean_step_time 1436.65, it/s 0.7 | |
| [0130 17:42:59 @multigpu.py:323] [p0115] step: count(526), step_time 1431.79, mean_step_time 1439.48, it/s 0.69 | |
| [0130 17:43:00 @multigpu.py:323] [p0574] step: count(543), step_time 1390.72, mean_step_time 1428.18, it/s 0.7 | |
| [0130 17:43:00 @multigpu.py:323] [p0576] step: count(550), step_time 1431.72, mean_step_time 1438.38, it/s 0.7 | |
| [0130 17:43:00 @multigpu.py:323] [p0115] step: count(527), step_time 1469.17, mean_step_time 1441.08, it/s 0.69 | |
| [0130 17:43:02 @multigpu.py:323] [p0115] step: count(528), step_time 1439.59, mean_step_time 1438.12, it/s 0.7 | |
| [0130 17:43:02 @multigpu.py:323] [p0576] step: count(551), step_time 1598.78, mean_step_time 1438.07, it/s 0.7 | |
| [0130 17:43:02 @multigpu.py:323] [p0574] step: count(544), step_time 1693.96, mean_step_time 1438.19, it/s 0.7 | |
| [0130 17:43:03 @multigpu.py:323] [p0576] step: count(552), step_time 1420.36, mean_step_time 1434.87, it/s 0.7 | |
| [0130 17:43:03 @multigpu.py:323] [p0574] step: count(545), step_time 1434.88, mean_step_time 1440.09, it/s 0.69 | |
| [0130 17:43:03 @multigpu.py:323] [p0115] step: count(529), step_time 1502.04, mean_step_time 1440.33, it/s 0.69 | |
| [0130 17:43:05 @multigpu.py:323] [p0576] step: count(553), step_time 1381.06, mean_step_time 1434.37, it/s 0.7 | |
| [0130 17:43:05 @multigpu.py:323] [p0574] step: count(546), step_time 1432.99, mean_step_time 1442.77, it/s 0.69 | |
| [0130 17:43:05 @multigpu.py:323] [p0115] step: count(530), step_time 1425.52, mean_step_time 1442.36, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.217906641629 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:43:06 @multigpu.py:323] [p0576] step: count(554), step_time 1442.1, mean_step_time 1436.5, it/s 0.7 | |
| [0130 17:43:06 @multigpu.py:323] [p0574] step: count(547), step_time 1473.52, mean_step_time 1443.65, it/s 0.69 | |
| [0130 17:43:06 @multigpu.py:323] [p0115] step: count(531), step_time 1427.41, mean_step_time 1440.82, it/s 0.69 | |
| [0130 17:43:07 @multigpu.py:323] [p0576] step: count(555), step_time 1424.39, mean_step_time 1433.73, it/s 0.7 | |
| [0130 17:43:08 @multigpu.py:323] [p0574] step: count(548), step_time 1439.58, mean_step_time 1440.06, it/s 0.69 | |
| [0130 17:43:08 @multigpu.py:323] [p0115] step: count(532), step_time 1430.38, mean_step_time 1442.78, it/s 0.69 | |
| [0130 17:43:09 @multigpu.py:323] [p0576] step: count(556), step_time 1518.46, mean_step_time 1439.71, it/s 0.69 | |
| [0130 17:43:09 @multigpu.py:323] [p0574] step: count(549), step_time 1427.17, mean_step_time 1439.13, it/s 0.69 | |
| [0130 17:43:09 @multigpu.py:323] [p0115] step: count(533), step_time 1443.05, mean_step_time 1441.7, it/s 0.69 | |
| [0130 17:43:10 @multigpu.py:323] [p0576] step: count(557), step_time 1374.29, mean_step_time 1434.18, it/s 0.7 | |
| [0130 17:43:10 @multigpu.py:323] [p0115] step: count(534), step_time 1428.15, mean_step_time 1443.51, it/s 0.69 | |
| [0130 17:43:10 @multigpu.py:323] [p0574] step: count(550), step_time 1473.98, mean_step_time 1441.28, it/s 0.69 | |
| [0130 17:43:12 @multigpu.py:323] [p0576] step: count(558), step_time 1458.34, mean_step_time 1433.61, it/s 0.7 | |
| [0130 17:43:12 @multigpu.py:323] [p0574] step: count(551), step_time 1400.64, mean_step_time 1439.24, it/s 0.69 | |
| [0130 17:43:12 @multigpu.py:323] [p0115] step: count(535), step_time 1427.15, mean_step_time 1439.62, it/s 0.69 | |
| [0130 17:43:13 @multigpu.py:323] [p0576] step: count(559), step_time 1451.05, mean_step_time 1438.12, it/s 0.7 | |
| [0130 17:43:13 @multigpu.py:323] [p0115] step: count(536), step_time 1428.3, mean_step_time 1441.73, it/s 0.69 | |
| [0130 17:43:13 @multigpu.py:323] [p0574] step: count(552), step_time 1464.41, mean_step_time 1443.41, it/s 0.69 | |
| [0130 17:43:15 @multigpu.py:323] [p0576] step: count(560), step_time 1439.48, mean_step_time 1439.42, it/s 0.69 | |
| [0130 17:43:15 @multigpu.py:323] [p0115] step: count(537), step_time 1441.73, mean_step_time 1438.92, it/s 0.69 | |
| [0130 17:43:15 @multigpu.py:323] [p0574] step: count(553), step_time 1434.15, mean_step_time 1445.13, it/s 0.69 | |
| [0130 17:43:16 @multigpu.py:323] [p0576] step: count(561), step_time 1380.15, mean_step_time 1435.18, it/s 0.7 | |
| [0130 17:43:16 @multigpu.py:323] [p0574] step: count(554), step_time 1402.8, mean_step_time 1444.35, it/s 0.69 | |
| [0130 17:43:16 @multigpu.py:323] [p0115] step: count(538), step_time 1504.04, mean_step_time 1440.43, it/s 0.69 | |
| [0130 17:43:17 @multigpu.py:323] [p0576] step: count(562), step_time 1414.47, mean_step_time 1434.54, it/s 0.7 | |
| [0130 17:43:18 @multigpu.py:323] [p0574] step: count(555), step_time 1400.29, mean_step_time 1443.45, it/s 0.69 | |
| [0130 17:43:18 @multigpu.py:323] [p0115] step: count(539), step_time 1493.64, mean_step_time 1442.36, it/s 0.69 | |
| [0130 17:43:19 @multigpu.py:323] [p0576] step: count(563), step_time 1459.15, mean_step_time 1435.29, it/s 0.7 | |
| [0130 17:43:19 @multigpu.py:323] [p0574] step: count(556), step_time 1398.45, mean_step_time 1442.18, it/s 0.69 | |
| [0130 17:43:19 @multigpu.py:323] [p0115] step: count(540), step_time 1440.72, mean_step_time 1440.92, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.221927292479 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:43:20 @multigpu.py:323] [p0574] step: count(557), step_time 1431.14, mean_step_time 1444.8, it/s 0.69 | |
| [0130 17:43:20 @multigpu.py:323] [p0576] step: count(564), step_time 1454.93, mean_step_time 1439.88, it/s 0.69 | |
| [0130 17:43:21 @multigpu.py:323] [p0115] step: count(541), step_time 1474.27, mean_step_time 1445.21, it/s 0.69 | |
| [0130 17:43:22 @multigpu.py:323] [p0574] step: count(558), step_time 1400.69, mean_step_time 1442.27, it/s 0.69 | |
| [0130 17:43:22 @multigpu.py:323] [p0576] step: count(565), step_time 1448.58, mean_step_time 1441.16, it/s 0.69 | |
| [0130 17:43:22 @multigpu.py:323] [p0115] step: count(542), step_time 1401.76, mean_step_time 1444.22, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.222741466893 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:43:23 @multigpu.py:323] [p0576] step: count(566), step_time 1423.63, mean_step_time 1441.37, it/s 0.69 | |
| [0130 17:43:23 @multigpu.py:323] [p0574] step: count(559), step_time 1486.25, mean_step_time 1444.61, it/s 0.69 | |
| [0130 17:43:23 @multigpu.py:323] [p0115] step: count(543), step_time 1438.65, mean_step_time 1444.26, it/s 0.69 | |
| [0130 17:43:25 @multigpu.py:323] [p0576] step: count(567), step_time 1395.37, mean_step_time 1440.15, it/s 0.69 | |
| [0130 17:43:25 @multigpu.py:323] [p0574] step: count(560), step_time 1489.61, mean_step_time 1446.49, it/s 0.69 | |
| [0130 17:43:25 @multigpu.py:323] [p0115] step: count(544), step_time 1469.23, mean_step_time 1446.75, it/s 0.69 | |
| [0130 17:43:26 @multigpu.py:323] [p0576] step: count(568), step_time 1395.78, mean_step_time 1438.21, it/s 0.7 | |
| [0130 17:43:26 @multigpu.py:323] [p0574] step: count(561), step_time 1387.96, mean_step_time 1444.8, it/s 0.69 | |
| [0130 17:43:26 @multigpu.py:323] [p0115] step: count(545), step_time 1483.14, mean_step_time 1449.99, it/s 0.69 | |
| [0130 17:43:28 @multigpu.py:323] [p0576] step: count(569), step_time 1442.61, mean_step_time 1437.74, it/s 0.7 | |
| [0130 17:43:28 @multigpu.py:323] [p0574] step: count(562), step_time 1438.72, mean_step_time 1445.1, it/s 0.69 | |
| [0130 17:43:28 @multigpu.py:323] [p0115] step: count(546), step_time 1477.18, mean_step_time 1452.26, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.224450801346 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:43:29 @multigpu.py:323] [p0576] step: count(570), step_time 1441.42, mean_step_time 1438.22, it/s 0.7 | |
| [0130 17:43:29 @multigpu.py:323] [p0574] step: count(563), step_time 1451.3, mean_step_time 1448.12, it/s 0.69 | |
| [0130 17:43:29 @multigpu.py:323] [p0115] step: count(547), step_time 1422.33, mean_step_time 1449.91, it/s 0.69 | |
| [0130 17:43:31 @multigpu.py:323] [p0115] step: count(548), step_time 1458.19, mean_step_time 1450.84, it/s 0.69 | |
| [0130 17:43:31 @multigpu.py:323] [p0574] step: count(564), step_time 1747.66, mean_step_time 1450.81, it/s 0.69 | |
| [0130 17:43:31 @multigpu.py:323] [p0576] step: count(571), step_time 1850.98, mean_step_time 1450.83, it/s 0.69 | |
| [0130 17:43:32 @multigpu.py:323] [p0576] step: count(572), step_time 1377.4, mean_step_time 1448.68, it/s 0.69 | |
| [0130 17:43:32 @multigpu.py:323] [p0115] step: count(549), step_time 1430.69, mean_step_time 1447.28, it/s 0.69 | |
| [0130 17:43:32 @multigpu.py:323] [p0574] step: count(565), step_time 1487.1, mean_step_time 1453.42, it/s 0.69 | |
| [0130 17:43:34 @multigpu.py:323] [p0576] step: count(573), step_time 1371.86, mean_step_time 1448.22, it/s 0.69 | |
| [0130 17:43:34 @multigpu.py:323] [p0115] step: count(550), step_time 1427.34, mean_step_time 1447.37, it/s 0.69 | |
| [0130 17:43:34 @multigpu.py:323] [p0574] step: count(566), step_time 1374.94, mean_step_time 1450.52, it/s 0.69 | |
| [0130 17:43:35 @multigpu.py:323] [p0576] step: count(574), step_time 1433.33, mean_step_time 1447.78, it/s 0.69 | |
| [0130 17:43:35 @multigpu.py:323] [p0115] step: count(551), step_time 1411.06, mean_step_time 1446.55, it/s 0.69 | |
| [0130 17:43:35 @multigpu.py:323] [p0574] step: count(567), step_time 1443.21, mean_step_time 1449.0, it/s 0.69 | |
| [0130 17:43:36 @multigpu.py:323] [p0576] step: count(575), step_time 1403.55, mean_step_time 1446.74, it/s 0.69 | |
| [0130 17:43:37 @multigpu.py:323] [p0115] step: count(552), step_time 1421.4, mean_step_time 1446.1, it/s 0.69 | |
| [0130 17:43:37 @multigpu.py:323] [p0574] step: count(568), step_time 1440.96, mean_step_time 1449.07, it/s 0.69 | |
| [0130 17:43:38 @multigpu.py:323] [p0576] step: count(576), step_time 1403.86, mean_step_time 1441.01, it/s 0.69 | |
| [0130 17:43:38 @multigpu.py:323] [p0115] step: count(553), step_time 1453.49, mean_step_time 1446.62, it/s 0.69 | |
| [0130 17:43:38 @multigpu.py:323] [p0574] step: count(569), step_time 1424.56, mean_step_time 1448.94, it/s 0.69 | |
| [0130 17:43:39 @multigpu.py:323] [p0576] step: count(577), step_time 1452.25, mean_step_time 1444.91, it/s 0.69 | |
| [0130 17:43:39 @multigpu.py:323] [p0115] step: count(554), step_time 1380.55, mean_step_time 1444.24, it/s 0.69 | |
| [0130 17:43:39 @multigpu.py:323] [p0574] step: count(570), step_time 1430.86, mean_step_time 1446.78, it/s 0.69 | |
| [0130 17:43:41 @multigpu.py:323] [p0576] step: count(578), step_time 1408.74, mean_step_time 1442.43, it/s 0.69 | |
| [0130 17:43:41 @multigpu.py:323] [p0115] step: count(555), step_time 1435.03, mean_step_time 1444.64, it/s 0.69 | |
| [0130 17:43:41 @multigpu.py:323] [p0574] step: count(571), step_time 1406.29, mean_step_time 1447.07, it/s 0.69 | |
| [0130 17:43:42 @multigpu.py:323] [p0576] step: count(579), step_time 1439.34, mean_step_time 1441.84, it/s 0.69 | |
| [0130 17:43:42 @multigpu.py:323] [p0115] step: count(556), step_time 1421.63, mean_step_time 1444.3, it/s 0.69 | |
| [0130 17:43:42 @multigpu.py:323] [p0574] step: count(572), step_time 1442.9, mean_step_time 1445.99, it/s 0.69 | |
| [0130 17:43:43 @multigpu.py:323] [p0576] step: count(580), step_time 1380.83, mean_step_time 1438.91, it/s 0.69 | |
| [0130 17:43:44 @multigpu.py:323] [p0115] step: count(557), step_time 1395.92, mean_step_time 1442.01, it/s 0.69 | |
| [0130 17:43:44 @multigpu.py:323] [p0574] step: count(573), step_time 1402.75, mean_step_time 1444.42, it/s 0.69 | |
| [0130 17:43:45 @multigpu.py:323] [p0576] step: count(581), step_time 1417.78, mean_step_time 1440.79, it/s 0.69 | |
| [0130 17:43:45 @multigpu.py:323] [p0574] step: count(574), step_time 1430.28, mean_step_time 1445.8, it/s 0.69 | |
| [0130 17:43:45 @multigpu.py:323] [p0115] step: count(558), step_time 1527.13, mean_step_time 1443.17, it/s 0.69 | |
| [0130 17:43:46 @multigpu.py:323] [p0576] step: count(582), step_time 1476.98, mean_step_time 1443.92, it/s 0.69 | |
| [0130 17:43:47 @multigpu.py:323] [p0115] step: count(559), step_time 1452.16, mean_step_time 1441.09, it/s 0.69 | |
| [0130 17:43:47 @multigpu.py:323] [p0574] step: count(575), step_time 1483.65, mean_step_time 1449.96, it/s 0.69 | |
| [0130 17:43:48 @multigpu.py:323] [p0576] step: count(583), step_time 1433.33, mean_step_time 1442.63, it/s 0.69 | |
| [0130 17:43:48 @multigpu.py:323] [p0115] step: count(560), step_time 1500.53, mean_step_time 1444.08, it/s 0.69 | |
| [0130 17:43:48 @multigpu.py:323] [p0574] step: count(576), step_time 1490.74, mean_step_time 1454.58, it/s 0.69 | |
| [0130 17:43:49 @multigpu.py:323] [p0576] step: count(584), step_time 1371.12, mean_step_time 1438.44, it/s 0.7 | |
| [0130 17:43:50 @multigpu.py:323] [p0115] step: count(561), step_time 1460.94, mean_step_time 1443.42, it/s 0.69 | |
| [0130 17:43:50 @multigpu.py:323] [p0574] step: count(577), step_time 1482.66, mean_step_time 1457.15, it/s 0.69 | |
| [0130 17:43:51 @multigpu.py:323] [p0576] step: count(585), step_time 1448.1, mean_step_time 1438.41, it/s 0.7 | |
| [0130 17:43:51 @multigpu.py:323] [p0115] step: count(562), step_time 1387.06, mean_step_time 1442.68, it/s 0.69 | |
| [0130 17:43:51 @multigpu.py:323] [p0574] step: count(578), step_time 1464.13, mean_step_time 1460.33, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.230837176641 , 1.8 ##### | |
| [u'online', 1.8] | |
| ##### Sending to neptune: active_workers : 0.230837286645 , 3 ##### | |
| receiving | |
| [0130 17:43:52 @multigpu.py:323] [p0576] step: count(586), step_time 1444.38, mean_step_time 1439.45, it/s 0.69 | |
| [0130 17:43:52 @multigpu.py:323] [p0115] step: count(563), step_time 1462.85, mean_step_time 1443.89, it/s 0.69 | |
| [0130 17:43:52 @multigpu.py:323] [p0574] step: count(579), step_time 1430.18, mean_step_time 1457.52, it/s 0.69 | |
| [0130 17:43:54 @multigpu.py:323] [p0576] step: count(587), step_time 1440.66, mean_step_time 1441.71, it/s 0.69 | |
| [0130 17:43:54 @multigpu.py:323] [p0115] step: count(564), step_time 1395.56, mean_step_time 1440.21, it/s 0.69 | |
| [0130 17:43:54 @multigpu.py:323] [p0574] step: count(580), step_time 1406.88, mean_step_time 1453.39, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.23157861299 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:43:55 @multigpu.py:323] [p0576] step: count(588), step_time 1370.65, mean_step_time 1440.46, it/s 0.69 | |
| [0130 17:43:55 @multigpu.py:323] [p0115] step: count(565), step_time 1395.37, mean_step_time 1435.82, it/s 0.7 | |
| [0130 17:43:55 @multigpu.py:323] [p0574] step: count(581), step_time 1436.96, mean_step_time 1455.84, it/s 0.69 | |
| [0130 17:43:56 @multigpu.py:323] [p0576] step: count(589), step_time 1421.98, mean_step_time 1439.43, it/s 0.69 | |
| [0130 17:43:57 @multigpu.py:323] [p0115] step: count(566), step_time 1388.97, mean_step_time 1431.41, it/s 0.7 | |
| [0130 17:43:57 @multigpu.py:323] [p0574] step: count(582), step_time 1424.21, mean_step_time 1455.11, it/s 0.69 | |
| [0130 17:43:58 @multigpu.py:323] [p0576] step: count(590), step_time 1406.28, mean_step_time 1437.67, it/s 0.7 | |
| [0130 17:43:58 @multigpu.py:323] [p0115] step: count(567), step_time 1483.43, mean_step_time 1434.47, it/s 0.7 | |
| [0130 17:43:58 @multigpu.py:323] [p0574] step: count(583), step_time 1436.96, mean_step_time 1454.39, it/s 0.69 | |
| [0130 17:44:00 @multigpu.py:323] [p0115] step: count(568), step_time 1544.23, mean_step_time 1438.77, it/s 0.7 | |
| [0130 17:44:00 @multigpu.py:323] [p0574] step: count(584), step_time 1433.22, mean_step_time 1438.67, it/s 0.7 | |
| [0130 17:44:00 @multigpu.py:323] [p0576] step: count(591), step_time 1869.94, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:44:01 @multigpu.py:323] [p0115] step: count(569), step_time 1396.0, mean_step_time 1437.03, it/s 0.7 | |
| [0130 17:44:01 @multigpu.py:323] [p0576] step: count(592), step_time 1410.35, mean_step_time 1440.27, it/s 0.69 | |
| [0130 17:44:01 @multigpu.py:323] [p0574] step: count(585), step_time 1446.87, mean_step_time 1436.66, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.233688505226 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:44:02 @multigpu.py:323] [p0115] step: count(570), step_time 1421.33, mean_step_time 1436.73, it/s 0.7 | |
| [0130 17:44:02 @multigpu.py:323] [p0576] step: count(593), step_time 1417.78, mean_step_time 1442.56, it/s 0.69 | |
| [0130 17:44:02 @multigpu.py:323] [p0574] step: count(586), step_time 1389.72, mean_step_time 1437.4, it/s 0.7 | |
| [0130 17:44:04 @multigpu.py:323] [p0115] step: count(571), step_time 1425.36, mean_step_time 1437.45, it/s 0.7 | |
| [0130 17:44:04 @multigpu.py:323] [p0574] step: count(587), step_time 1438.96, mean_step_time 1437.19, it/s 0.7 | |
| [0130 17:44:04 @multigpu.py:323] [p0576] step: count(594), step_time 1469.7, mean_step_time 1444.38, it/s 0.69 | |
| [0130 17:44:05 @multigpu.py:323] [p0115] step: count(572), step_time 1433.1, mean_step_time 1438.03, it/s 0.7 | |
| [0130 17:44:05 @multigpu.py:323] [p0576] step: count(595), step_time 1457.12, mean_step_time 1447.06, it/s 0.69 | |
| [0130 17:44:05 @multigpu.py:323] [p0574] step: count(588), step_time 1488.52, mean_step_time 1439.57, it/s 0.69 | |
| [0130 17:44:07 @multigpu.py:323] [p0115] step: count(573), step_time 1404.14, mean_step_time 1435.56, it/s 0.7 | |
| [0130 17:44:07 @multigpu.py:323] [p0576] step: count(596), step_time 1434.74, mean_step_time 1448.6, it/s 0.69 | |
| [0130 17:44:07 @multigpu.py:323] [p0574] step: count(589), step_time 1433.57, mean_step_time 1440.02, it/s 0.69 | |
| [0130 17:44:08 @multigpu.py:323] [p0115] step: count(574), step_time 1462.73, mean_step_time 1439.67, it/s 0.69 | |
| [0130 17:44:08 @multigpu.py:323] [p0576] step: count(597), step_time 1438.55, mean_step_time 1447.92, it/s 0.69 | |
| [0130 17:44:08 @multigpu.py:323] [p0574] step: count(590), step_time 1460.88, mean_step_time 1441.52, it/s 0.69 | |
| [0130 17:44:10 @multigpu.py:323] [p0115] step: count(575), step_time 1428.03, mean_step_time 1439.32, it/s 0.69 | |
| [0130 17:44:10 @multigpu.py:323] [p0576] step: count(598), step_time 1411.28, mean_step_time 1448.04, it/s 0.69 | |
| [0130 17:44:10 @multigpu.py:323] [p0574] step: count(591), step_time 1442.06, mean_step_time 1443.3, it/s 0.69 | |
| [0130 17:44:11 @multigpu.py:323] [p0115] step: count(576), step_time 1430.0, mean_step_time 1439.74, it/s 0.69 | |
| [0130 17:44:11 @multigpu.py:323] [p0576] step: count(599), step_time 1431.67, mean_step_time 1447.66, it/s 0.69 | |
| [0130 17:44:11 @multigpu.py:323] [p0574] step: count(592), step_time 1423.58, mean_step_time 1442.34, it/s 0.69 | |
| [0130 17:44:12 @multigpu.py:323] [p0115] step: count(577), step_time 1407.09, mean_step_time 1440.3, it/s 0.69 | |
| [0130 17:44:13 @multigpu.py:323] [p0574] step: count(593), step_time 1393.41, mean_step_time 1441.87, it/s 0.69 | |
| [0130 17:44:13 @multigpu.py:323] [p0576] step: count(600), step_time 1467.88, mean_step_time 1452.01, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.236649152173 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.236649152173 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.236649152173 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.236649644706 , -0.0012414510129 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.236649644706 , 0.698001503944 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.236649644706 , -2.29192209244 ##### | |
| ##### Sending to neptune: value_loss : 0.236649644706 , 1.43501460552 ##### | |
| ##### Sending to neptune: advantage : 0.236649644706 , -0.00304705230519 ##### | |
| ##### Sending to neptune: pred_reward : 0.236649644706 , 0.39679774642 ##### | |
| ##### Sending to neptune: max_logit : 0.236649644706 , 0.181564688683 ##### | |
| [u'loss', -0.0012414510129019618, 0.698001503944397, -2.291922092437744, 1.4350146055221558, -0.0030470523051917553, 0.3967977464199066, 0.18156468868255615] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.236650113596 , 8806054.65 ##### | |
| ##### Sending to neptune: dp_per_s : 0.236650113596 , 89.2224737498 ##### | |
| [u'other', 8806054.65, 89.22247374981353] | |
| receiving | |
| [0130 17:44:14 @multigpu.py:323] [p0115] step: count(578), step_time 1487.14, mean_step_time 1438.3, it/s 0.7 | |
| [0130 17:44:14 @multigpu.py:323] [p0574] step: count(594), step_time 1423.42, mean_step_time 1441.53, it/s 0.69 | |
| [0130 17:44:14 @multigpu.py:323] [p0576] step: count(601), step_time 1441.36, mean_step_time 1453.19, it/s 0.69 | |
| [0130 17:44:15 @multigpu.py:323] [p0115] step: count(579), step_time 1395.8, mean_step_time 1435.48, it/s 0.7 | |
| [0130 17:44:15 @multigpu.py:323] [p0574] step: count(595), step_time 1435.73, mean_step_time 1439.13, it/s 0.69 | |
| [0130 17:44:15 @multigpu.py:323] [p0576] step: count(602), step_time 1481.95, mean_step_time 1453.44, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.237544338836 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:44:17 @multigpu.py:323] [p0115] step: count(580), step_time 1488.18, mean_step_time 1434.87, it/s 0.7 | |
| [0130 17:44:17 @multigpu.py:323] [p0574] step: count(596), step_time 1435.03, mean_step_time 1436.35, it/s 0.7 | |
| [0130 17:44:17 @multigpu.py:323] [p0576] step: count(603), step_time 1463.81, mean_step_time 1454.97, it/s 0.69 | |
| [0130 17:44:18 @multigpu.py:323] [p0115] step: count(581), step_time 1418.59, mean_step_time 1432.75, it/s 0.7 | |
| [0130 17:44:18 @multigpu.py:323] [p0574] step: count(597), step_time 1469.03, mean_step_time 1435.67, it/s 0.7 | |
| [0130 17:44:18 @multigpu.py:323] [p0576] step: count(604), step_time 1420.29, mean_step_time 1457.42, it/s 0.69 | |
| [0130 17:44:20 @multigpu.py:323] [p0115] step: count(582), step_time 1458.28, mean_step_time 1436.31, it/s 0.7 | |
| [0130 17:44:20 @multigpu.py:323] [p0574] step: count(598), step_time 1442.36, mean_step_time 1434.58, it/s 0.7 | |
| [0130 17:44:20 @multigpu.py:323] [p0576] step: count(605), step_time 1400.04, mean_step_time 1455.02, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.238823784126 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:44:21 @multigpu.py:323] [p0574] step: count(599), step_time 1400.31, mean_step_time 1433.08, it/s 0.7 | |
| [0130 17:44:21 @multigpu.py:323] [p0115] step: count(583), step_time 1498.62, mean_step_time 1438.1, it/s 0.7 | |
| [0130 17:44:21 @multigpu.py:323] [p0576] step: count(606), step_time 1421.09, mean_step_time 1453.86, it/s 0.69 | |
| [0130 17:44:23 @multigpu.py:323] [p0115] step: count(584), step_time 1412.91, mean_step_time 1438.97, it/s 0.69 | |
| [0130 17:44:23 @multigpu.py:323] [p0576] step: count(607), step_time 1404.31, mean_step_time 1452.04, it/s 0.69 | |
| [0130 17:44:23 @multigpu.py:323] [p0574] step: count(600), step_time 1464.53, mean_step_time 1435.97, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.239441586071 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.239441586071 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.239441586071 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.239442152182 , -0.0125148463994 ##### | |
| ##### Sending to neptune: policy_loss : 0.239442152182 , -0.49882593751 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.239442152182 , -2.29190254211 ##### | |
| ##### Sending to neptune: value_loss : 0.239442152182 , 1.18882787228 ##### | |
| ##### Sending to neptune: advantage : 0.239442152182 , 0.00211322447285 ##### | |
| ##### Sending to neptune: pred_reward : 0.239442152182 , 0.398575603962 ##### | |
| ##### Sending to neptune: max_logit : 0.239442152182 , 0.181839182973 ##### | |
| [u'loss', -0.01251484639942646, -0.49882593750953674, -2.291902542114258, 1.1888278722763062, 0.002113224472850561, 0.3985756039619446, 0.18183918297290802] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.239442588554 , 8800469.51 ##### | |
| ##### Sending to neptune: dp_per_s : 0.239442588554 , 89.0259561405 ##### | |
| [u'other', 8800469.51, 89.02595614054479] | |
| receiving | |
| [0130 17:44:24 @multigpu.py:323] [p0115] step: count(585), step_time 1430.45, mean_step_time 1440.72, it/s 0.69 | |
| [0130 17:44:24 @multigpu.py:323] [p0576] step: count(608), step_time 1455.01, mean_step_time 1456.26, it/s 0.69 | |
| [0130 17:44:24 @multigpu.py:323] [p0574] step: count(601), step_time 1447.28, mean_step_time 1436.48, it/s 0.7 | |
| [0130 17:44:25 @multigpu.py:323] [p0115] step: count(586), step_time 1403.09, mean_step_time 1441.43, it/s 0.69 | |
| [0130 17:44:25 @multigpu.py:323] [p0574] step: count(602), step_time 1389.81, mean_step_time 1434.76, it/s 0.7 | |
| [0130 17:44:25 @multigpu.py:323] [p0576] step: count(609), step_time 1459.53, mean_step_time 1458.13, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.240421581931 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:44:27 @multigpu.py:323] [p0574] step: count(603), step_time 1412.17, mean_step_time 1433.52, it/s 0.7 | |
| [0130 17:44:27 @multigpu.py:323] [p0115] step: count(587), step_time 1454.29, mean_step_time 1439.97, it/s 0.69 | |
| [0130 17:44:27 @multigpu.py:323] [p0576] step: count(610), step_time 1377.06, mean_step_time 1456.67, it/s 0.69 | |
| [0130 17:44:28 @multigpu.py:323] [p0115] step: count(588), step_time 1469.67, mean_step_time 1436.24, it/s 0.7 | |
| [0130 17:44:28 @multigpu.py:323] [p0574] step: count(604), step_time 1487.58, mean_step_time 1436.24, it/s 0.7 | |
| [0130 17:44:28 @multigpu.py:323] [p0576] step: count(611), step_time 1462.63, mean_step_time 1436.31, it/s 0.7 | |
| [0130 17:44:30 @multigpu.py:323] [p0574] step: count(605), step_time 1389.52, mean_step_time 1433.37, it/s 0.7 | |
| [0130 17:44:30 @multigpu.py:323] [p0115] step: count(589), step_time 1390.94, mean_step_time 1435.99, it/s 0.7 | |
| [0130 17:44:30 @multigpu.py:323] [p0576] step: count(612), step_time 1450.4, mean_step_time 1438.31, it/s 0.7 | |
| [0130 17:44:31 @multigpu.py:323] [p0576] step: count(613), step_time 1381.31, mean_step_time 1436.49, it/s 0.7 | |
| [0130 17:44:31 @multigpu.py:323] [p0574] step: count(606), step_time 1463.87, mean_step_time 1437.08, it/s 0.7 | |
| [0130 17:44:31 @multigpu.py:323] [p0115] step: count(590), step_time 1475.51, mean_step_time 1438.7, it/s 0.7 | |
| [0130 17:44:33 @multigpu.py:323] [p0576] step: count(614), step_time 1400.85, mean_step_time 1433.04, it/s 0.7 | |
| [0130 17:44:33 @multigpu.py:323] [p0115] step: count(591), step_time 1427.56, mean_step_time 1438.81, it/s 0.7 | |
| [0130 17:44:33 @multigpu.py:323] [p0574] step: count(607), step_time 1496.17, mean_step_time 1439.94, it/s 0.69 | |
| [0130 17:44:34 @multigpu.py:323] [p0576] step: count(615), step_time 1415.65, mean_step_time 1430.97, it/s 0.7 | |
| [0130 17:44:34 @multigpu.py:323] [p0115] step: count(592), step_time 1362.75, mean_step_time 1435.29, it/s 0.7 | |
| [0130 17:44:34 @multigpu.py:323] [p0574] step: count(608), step_time 1415.51, mean_step_time 1436.29, it/s 0.7 | |
| [0130 17:44:35 @multigpu.py:323] [p0576] step: count(616), step_time 1422.25, mean_step_time 1430.35, it/s 0.7 | |
| [0130 17:44:35 @multigpu.py:323] [p0115] step: count(593), step_time 1428.81, mean_step_time 1436.52, it/s 0.7 | |
| [0130 17:44:36 @multigpu.py:323] [p0574] step: count(609), step_time 1479.88, mean_step_time 1438.61, it/s 0.7 | |
| [0130 17:44:37 @multigpu.py:323] [p0576] step: count(617), step_time 1405.63, mean_step_time 1428.7, it/s 0.7 | |
| [0130 17:44:37 @multigpu.py:323] [p0115] step: count(594), step_time 1480.72, mean_step_time 1437.42, it/s 0.7 | |
| [0130 17:44:37 @multigpu.py:323] [p0574] step: count(610), step_time 1439.04, mean_step_time 1437.52, it/s 0.7 | |
| [0130 17:44:38 @multigpu.py:323] [p0576] step: count(618), step_time 1453.8, mean_step_time 1430.83, it/s 0.7 | |
| [0130 17:44:38 @multigpu.py:323] [p0115] step: count(595), step_time 1421.83, mean_step_time 1437.11, it/s 0.7 | |
| [0130 17:44:38 @multigpu.py:323] [p0574] step: count(611), step_time 1399.35, mean_step_time 1435.38, it/s 0.7 | |
| [0130 17:44:40 @multigpu.py:323] [p0576] step: count(619), step_time 1407.74, mean_step_time 1429.63, it/s 0.7 | |
| [0130 17:44:40 @multigpu.py:323] [p0115] step: count(596), step_time 1425.51, mean_step_time 1436.89, it/s 0.7 | |
| [0130 17:44:40 @multigpu.py:323] [p0574] step: count(612), step_time 1478.83, mean_step_time 1438.14, it/s 0.7 | |
| [0130 17:44:41 @multigpu.py:323] [p0576] step: count(620), step_time 1445.59, mean_step_time 1428.51, it/s 0.7 | |
| [0130 17:44:41 @multigpu.py:323] [p0115] step: count(597), step_time 1423.78, mean_step_time 1437.72, it/s 0.7 | |
| [0130 17:44:41 @multigpu.py:323] [p0574] step: count(613), step_time 1408.85, mean_step_time 1438.91, it/s 0.69 | |
| [0130 17:44:43 @multigpu.py:323] [p0576] step: count(621), step_time 1422.57, mean_step_time 1427.57, it/s 0.7 | |
| [0130 17:44:43 @multigpu.py:323] [p0115] step: count(598), step_time 1433.0, mean_step_time 1435.02, it/s 0.7 | |
| [0130 17:44:43 @multigpu.py:323] [p0574] step: count(614), step_time 1403.31, mean_step_time 1437.91, it/s 0.7 | |
| [0130 17:44:44 @multigpu.py:323] [p0576] step: count(622), step_time 1411.85, mean_step_time 1424.07, it/s 0.7 | |
| [0130 17:44:44 @multigpu.py:323] [p0115] step: count(599), step_time 1452.46, mean_step_time 1437.85, it/s 0.7 | |
| [0130 17:44:44 @multigpu.py:323] [p0574] step: count(615), step_time 1476.66, mean_step_time 1439.95, it/s 0.69 | |
| [0130 17:44:45 @multigpu.py:323] [p0576] step: count(623), step_time 1407.37, mean_step_time 1421.25, it/s 0.7 | |
| [0130 17:44:45 @multigpu.py:323] [p0115] step: count(600), step_time 1399.95, mean_step_time 1433.44, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.24579519444 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.24579519444 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.24579519444 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.245795726644 , -0.0124126961455 ##### | |
| ##### Sending to neptune: policy_loss : 0.245795726644 , -0.532102823257 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.245795726644 , -2.29180145264 ##### | |
| ##### Sending to neptune: value_loss : 0.245795726644 , 1.23507893085 ##### | |
| ##### Sending to neptune: advantage : 0.245795726644 , 0.00228149211034 ##### | |
| ##### Sending to neptune: pred_reward : 0.245795726644 , 0.404466867447 ##### | |
| ##### Sending to neptune: max_logit : 0.245795726644 , 0.183123975992 ##### | |
| [u'loss', -0.01241269614547491, -0.5321028232574463, -2.2918014526367188, 1.2350789308547974, 0.0022814921103417873, 0.4044668674468994, 0.18312397599220276] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.245796224409 , 8822338.12 ##### | |
| ##### Sending to neptune: dp_per_s : 0.245796224409 , 89.0029782201 ##### | |
| [u'other', 8822338.12, 89.00297822013982] | |
| receiving | |
| [0130 17:44:46 @multigpu.py:323] [p0574] step: count(616), step_time 1425.14, mean_step_time 1439.46, it/s 0.69 | |
| [0130 17:44:47 @multigpu.py:323] [p0576] step: count(624), step_time 1462.41, mean_step_time 1423.35, it/s 0.7 | |
| [0130 17:44:47 @multigpu.py:323] [p0115] step: count(601), step_time 1463.34, mean_step_time 1435.68, it/s 0.7 | |
| [0130 17:44:47 @multigpu.py:323] [p0574] step: count(617), step_time 1418.75, mean_step_time 1436.95, it/s 0.7 | |
| [0130 17:44:48 @multigpu.py:323] [p0576] step: count(625), step_time 1426.03, mean_step_time 1424.65, it/s 0.7 | |
| [0130 17:44:48 @multigpu.py:323] [p0115] step: count(602), step_time 1403.93, mean_step_time 1432.96, it/s 0.7 | |
| [0130 17:44:48 @multigpu.py:323] [p0574] step: count(618), step_time 1393.44, mean_step_time 1434.5, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.246657715506 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.246721496052 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:44:50 @multigpu.py:323] [p0576] step: count(626), step_time 1402.71, mean_step_time 1423.73, it/s 0.7 | |
| [0130 17:44:50 @multigpu.py:323] [p0115] step: count(603), step_time 1429.02, mean_step_time 1429.48, it/s 0.7 | |
| [0130 17:44:50 @multigpu.py:323] [p0574] step: count(619), step_time 1492.01, mean_step_time 1439.09, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.247186529703 , 0.6 ##### | |
| [u'online', 0.6] | |
| receiving | |
| [0130 17:44:51 @multigpu.py:323] [p0576] step: count(627), step_time 1407.93, mean_step_time 1423.92, it/s 0.7 | |
| [0130 17:44:51 @multigpu.py:323] [p0115] step: count(604), step_time 1412.04, mean_step_time 1429.43, it/s 0.7 | |
| [0130 17:44:51 @multigpu.py:323] [p0574] step: count(620), step_time 1405.33, mean_step_time 1436.13, it/s 0.7 | |
| [0130 17:44:52 @multigpu.py:323] [p0576] step: count(628), step_time 1390.03, mean_step_time 1420.67, it/s 0.7 | |
| [0130 17:44:53 @multigpu.py:323] [p0115] step: count(605), step_time 1426.99, mean_step_time 1429.26, it/s 0.7 | |
| [0130 17:44:53 @multigpu.py:323] [p0574] step: count(621), step_time 1503.47, mean_step_time 1438.93, it/s 0.69 | |
| [0130 17:44:54 @multigpu.py:323] [p0576] step: count(629), step_time 1378.84, mean_step_time 1416.63, it/s 0.71 | |
| [0130 17:44:54 @multigpu.py:323] [p0115] step: count(606), step_time 1481.85, mean_step_time 1433.2, it/s 0.7 | |
| [0130 17:44:54 @multigpu.py:323] [p0574] step: count(622), step_time 1415.95, mean_step_time 1440.24, it/s 0.69 | |
| [0130 17:44:55 @multigpu.py:323] [p0576] step: count(630), step_time 1494.85, mean_step_time 1422.52, it/s 0.7 | |
| [0130 17:44:55 @multigpu.py:323] [p0115] step: count(607), step_time 1399.97, mean_step_time 1430.48, it/s 0.7 | |
| [0130 17:44:56 @multigpu.py:323] [p0574] step: count(623), step_time 1419.95, mean_step_time 1440.63, it/s 0.69 | |
| [0130 17:44:57 @multigpu.py:323] [p0115] step: count(608), step_time 1606.73, mean_step_time 1437.34, it/s 0.7 | |
| [0130 17:44:57 @multigpu.py:323] [p0574] step: count(624), step_time 1424.07, mean_step_time 1437.46, it/s 0.7 | |
| [0130 17:44:57 @multigpu.py:323] [p0576] step: count(631), step_time 1761.94, mean_step_time 1437.49, it/s 0.7 | |
| [0130 17:44:59 @multigpu.py:323] [p0576] step: count(632), step_time 1410.12, mean_step_time 1435.47, it/s 0.7 | |
| [0130 17:44:59 @multigpu.py:323] [p0115] step: count(609), step_time 1436.63, mean_step_time 1439.62, it/s 0.69 | |
| [0130 17:44:59 @multigpu.py:323] [p0574] step: count(625), step_time 1486.65, mean_step_time 1442.31, it/s 0.69 | |
| [0130 17:45:00 @multigpu.py:323] [p0576] step: count(633), step_time 1481.32, mean_step_time 1440.47, it/s 0.69 | |
| [0130 17:45:00 @multigpu.py:323] [p0115] step: count(610), step_time 1477.83, mean_step_time 1439.74, it/s 0.69 | |
| [0130 17:45:00 @multigpu.py:323] [p0574] step: count(626), step_time 1486.27, mean_step_time 1443.43, it/s 0.69 | |
| [0130 17:45:01 @multigpu.py:323] [p0115] step: count(611), step_time 1405.17, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:45:01 @multigpu.py:323] [p0576] step: count(634), step_time 1428.84, mean_step_time 1441.87, it/s 0.69 | |
| [0130 17:45:01 @multigpu.py:323] [p0574] step: count(627), step_time 1387.93, mean_step_time 1438.02, it/s 0.7 | |
| [0130 17:45:03 @multigpu.py:323] [p0115] step: count(612), step_time 1414.92, mean_step_time 1441.22, it/s 0.69 | |
| [0130 17:45:03 @multigpu.py:323] [p0574] step: count(628), step_time 1438.6, mean_step_time 1439.17, it/s 0.69 | |
| [0130 17:45:03 @multigpu.py:323] [p0576] step: count(635), step_time 1515.42, mean_step_time 1446.86, it/s 0.69 | |
| [0130 17:45:04 @multigpu.py:323] [p0115] step: count(613), step_time 1465.02, mean_step_time 1443.03, it/s 0.69 | |
| [0130 17:45:04 @multigpu.py:323] [p0576] step: count(636), step_time 1393.03, mean_step_time 1445.4, it/s 0.69 | |
| [0130 17:45:04 @multigpu.py:323] [p0574] step: count(629), step_time 1468.57, mean_step_time 1438.61, it/s 0.7 | |
| [0130 17:45:06 @multigpu.py:323] [p0115] step: count(614), step_time 1453.47, mean_step_time 1441.67, it/s 0.69 | |
| [0130 17:45:06 @multigpu.py:323] [p0576] step: count(637), step_time 1430.3, mean_step_time 1446.63, it/s 0.69 | |
| [0130 17:45:06 @multigpu.py:323] [p0574] step: count(630), step_time 1414.37, mean_step_time 1437.38, it/s 0.7 | |
| [0130 17:45:07 @multigpu.py:323] [p0574] step: count(631), step_time 1374.32, mean_step_time 1436.12, it/s 0.7 | |
| [0130 17:45:07 @multigpu.py:323] [p0576] step: count(638), step_time 1400.2, mean_step_time 1443.95, it/s 0.69 | |
| [0130 17:45:07 @multigpu.py:323] [p0115] step: count(615), step_time 1432.96, mean_step_time 1442.23, it/s 0.69 | |
| [0130 17:45:09 @multigpu.py:323] [p0576] step: count(639), step_time 1464.58, mean_step_time 1446.8, it/s 0.69 | |
| [0130 17:45:09 @multigpu.py:323] [p0574] step: count(632), step_time 1482.66, mean_step_time 1436.32, it/s 0.7 | |
| [0130 17:45:09 @multigpu.py:323] [p0115] step: count(616), step_time 1473.6, mean_step_time 1444.63, it/s 0.69 | |
| [0130 17:45:10 @multigpu.py:323] [p0574] step: count(633), step_time 1401.24, mean_step_time 1435.94, it/s 0.7 | |
| [0130 17:45:10 @multigpu.py:323] [p0115] step: count(617), step_time 1406.4, mean_step_time 1443.76, it/s 0.69 | |
| [0130 17:45:10 @multigpu.py:323] [p0576] step: count(640), step_time 1458.48, mean_step_time 1447.44, it/s 0.69 | |
| [0130 17:45:11 @multigpu.py:323] [p0115] step: count(618), step_time 1410.03, mean_step_time 1442.61, it/s 0.69 | |
| [0130 17:45:11 @multigpu.py:323] [p0574] step: count(634), step_time 1454.65, mean_step_time 1438.5, it/s 0.7 | |
| [0130 17:45:12 @multigpu.py:323] [p0576] step: count(641), step_time 1431.78, mean_step_time 1447.9, it/s 0.69 | |
| [0130 17:45:13 @multigpu.py:323] [p0115] step: count(619), step_time 1410.05, mean_step_time 1440.49, it/s 0.69 | |
| [0130 17:45:13 @multigpu.py:323] [p0576] step: count(642), step_time 1456.59, mean_step_time 1450.14, it/s 0.69 | |
| [0130 17:45:13 @multigpu.py:323] [p0574] step: count(635), step_time 1477.97, mean_step_time 1438.57, it/s 0.7 | |
| [0130 17:45:14 @multigpu.py:323] [p0115] step: count(620), step_time 1455.96, mean_step_time 1443.29, it/s 0.69 | |
| [0130 17:45:14 @multigpu.py:323] [p0576] step: count(643), step_time 1389.1, mean_step_time 1449.22, it/s 0.69 | |
| [0130 17:45:14 @multigpu.py:323] [p0574] step: count(636), step_time 1431.19, mean_step_time 1438.87, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.25393662552 , 1.5 ##### | |
| [u'online', 1.5] | |
| ##### Sending to neptune: active_workers : 0.253936661614 , 3 ##### | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.253976651099 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:45:16 @multigpu.py:323] [p0115] step: count(621), step_time 1374.5, mean_step_time 1438.85, it/s 0.69 | |
| [0130 17:45:16 @multigpu.py:323] [p0576] step: count(644), step_time 1412.07, mean_step_time 1446.71, it/s 0.69 | |
| [0130 17:45:16 @multigpu.py:323] [p0574] step: count(637), step_time 1401.14, mean_step_time 1437.99, it/s 0.7 | |
| [0130 17:45:17 @multigpu.py:323] [p0115] step: count(622), step_time 1455.7, mean_step_time 1441.44, it/s 0.69 | |
| [0130 17:45:17 @multigpu.py:323] [p0574] step: count(638), step_time 1388.99, mean_step_time 1437.77, it/s 0.7 | |
| [0130 17:45:17 @multigpu.py:323] [p0576] step: count(645), step_time 1450.71, mean_step_time 1447.94, it/s 0.69 | |
| [0130 17:45:19 @multigpu.py:323] [p0576] step: count(646), step_time 1429.79, mean_step_time 1449.3, it/s 0.69 | |
| [0130 17:45:19 @multigpu.py:323] [p0115] step: count(623), step_time 1483.14, mean_step_time 1444.15, it/s 0.69 | |
| [0130 17:45:19 @multigpu.py:323] [p0574] step: count(639), step_time 1500.31, mean_step_time 1438.18, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.255228105783 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:45:20 @multigpu.py:323] [p0115] step: count(624), step_time 1400.48, mean_step_time 1443.57, it/s 0.69 | |
| [0130 17:45:20 @multigpu.py:323] [p0576] step: count(647), step_time 1429.17, mean_step_time 1450.36, it/s 0.69 | |
| [0130 17:45:20 @multigpu.py:323] [p0574] step: count(640), step_time 1431.48, mean_step_time 1439.49, it/s 0.69 | |
| [0130 17:45:21 @multigpu.py:323] [p0115] step: count(625), step_time 1438.56, mean_step_time 1444.15, it/s 0.69 | |
| [0130 17:45:22 @multigpu.py:323] [p0574] step: count(641), step_time 1377.16, mean_step_time 1433.17, it/s 0.7 | |
| [0130 17:45:22 @multigpu.py:323] [p0576] step: count(648), step_time 1458.03, mean_step_time 1453.76, it/s 0.69 | |
| [0130 17:45:23 @multigpu.py:323] [p0574] step: count(642), step_time 1409.73, mean_step_time 1432.86, it/s 0.7 | |
| [0130 17:45:23 @multigpu.py:323] [p0115] step: count(626), step_time 1456.32, mean_step_time 1442.87, it/s 0.69 | |
| [0130 17:45:23 @multigpu.py:323] [p0576] step: count(649), step_time 1492.76, mean_step_time 1459.45, it/s 0.69 | |
| [0130 17:45:24 @multigpu.py:323] [p0574] step: count(643), step_time 1467.67, mean_step_time 1435.25, it/s 0.7 | |
| [0130 17:45:24 @multigpu.py:323] [p0115] step: count(627), step_time 1441.65, mean_step_time 1444.96, it/s 0.69 | |
| [0130 17:45:24 @multigpu.py:323] [p0576] step: count(650), step_time 1411.33, mean_step_time 1455.28, it/s 0.69 | |
| [0130 17:45:26 @multigpu.py:323] [p0115] step: count(628), step_time 1539.64, mean_step_time 1441.6, it/s 0.69 | |
| [0130 17:45:26 @multigpu.py:323] [p0576] step: count(651), step_time 1488.15, mean_step_time 1441.59, it/s 0.69 | |
| [0130 17:45:26 @multigpu.py:323] [p0574] step: count(644), step_time 1551.55, mean_step_time 1441.62, it/s 0.69 | |
| [0130 17:45:27 @multigpu.py:323] [p0576] step: count(652), step_time 1392.0, mean_step_time 1440.68, it/s 0.69 | |
| [0130 17:45:27 @multigpu.py:323] [p0115] step: count(629), step_time 1405.88, mean_step_time 1440.06, it/s 0.69 | |
| [0130 17:45:27 @multigpu.py:323] [p0574] step: count(645), step_time 1471.01, mean_step_time 1440.84, it/s 0.69 | |
| [0130 17:45:29 @multigpu.py:323] [p0576] step: count(653), step_time 1416.31, mean_step_time 1437.43, it/s 0.7 | |
| [0130 17:45:29 @multigpu.py:323] [p0115] step: count(630), step_time 1429.28, mean_step_time 1437.64, it/s 0.7 | |
| [0130 17:45:29 @multigpu.py:323] [p0574] step: count(646), step_time 1418.72, mean_step_time 1437.46, it/s 0.7 | |
| [0130 17:45:30 @multigpu.py:323] [p0576] step: count(654), step_time 1443.74, mean_step_time 1438.18, it/s 0.7 | |
| [0130 17:45:30 @multigpu.py:323] [p0115] step: count(631), step_time 1452.88, mean_step_time 1440.02, it/s 0.69 | |
| [0130 17:45:30 @multigpu.py:323] [p0574] step: count(647), step_time 1431.32, mean_step_time 1439.63, it/s 0.69 | |
| [0130 17:45:32 @multigpu.py:323] [p0576] step: count(655), step_time 1400.57, mean_step_time 1432.44, it/s 0.7 | |
| [0130 17:45:32 @multigpu.py:323] [p0115] step: count(632), step_time 1430.96, mean_step_time 1440.82, it/s 0.69 | |
| [0130 17:45:32 @multigpu.py:323] [p0574] step: count(648), step_time 1474.59, mean_step_time 1441.43, it/s 0.69 | |
| [0130 17:45:33 @multigpu.py:323] [p0576] step: count(656), step_time 1432.96, mean_step_time 1434.43, it/s 0.7 | |
| [0130 17:45:33 @multigpu.py:323] [p0115] step: count(633), step_time 1431.99, mean_step_time 1439.17, it/s 0.69 | |
| [0130 17:45:33 @multigpu.py:323] [p0574] step: count(649), step_time 1383.33, mean_step_time 1437.17, it/s 0.7 | |
| [0130 17:45:34 @multigpu.py:323] [p0576] step: count(657), step_time 1400.38, mean_step_time 1432.94, it/s 0.7 | |
| [0130 17:45:35 @multigpu.py:323] [p0574] step: count(650), step_time 1413.86, mean_step_time 1437.14, it/s 0.7 | |
| [0130 17:45:35 @multigpu.py:323] [p0115] step: count(634), step_time 1454.74, mean_step_time 1439.24, it/s 0.69 | |
| [0130 17:45:36 @multigpu.py:323] [p0576] step: count(658), step_time 1378.2, mean_step_time 1431.84, it/s 0.7 | |
| [0130 17:45:36 @multigpu.py:323] [p0574] step: count(651), step_time 1426.33, mean_step_time 1439.74, it/s 0.69 | |
| [0130 17:45:36 @multigpu.py:323] [p0115] step: count(635), step_time 1506.58, mean_step_time 1442.92, it/s 0.69 | |
| [0130 17:45:37 @multigpu.py:323] [p0576] step: count(659), step_time 1411.7, mean_step_time 1429.19, it/s 0.7 | |
| [0130 17:45:37 @multigpu.py:323] [p0574] step: count(652), step_time 1434.51, mean_step_time 1437.34, it/s 0.7 | |
| [0130 17:45:38 @multigpu.py:323] [p0115] step: count(636), step_time 1487.17, mean_step_time 1443.59, it/s 0.69 | |
| [0130 17:45:39 @multigpu.py:323] [p0576] step: count(660), step_time 1500.11, mean_step_time 1431.27, it/s 0.7 | |
| [0130 17:45:39 @multigpu.py:323] [p0574] step: count(653), step_time 1401.87, mean_step_time 1437.37, it/s 0.7 | |
| [0130 17:45:39 @multigpu.py:323] [p0115] step: count(637), step_time 1403.42, mean_step_time 1443.45, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.260792402493 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:45:40 @multigpu.py:323] [p0576] step: count(661), step_time 1437.46, mean_step_time 1431.56, it/s 0.7 | |
| [0130 17:45:40 @multigpu.py:323] [p0574] step: count(654), step_time 1417.83, mean_step_time 1435.53, it/s 0.7 | |
| [0130 17:45:40 @multigpu.py:323] [p0115] step: count(638), step_time 1405.06, mean_step_time 1443.2, it/s 0.69 | |
| [0130 17:45:42 @multigpu.py:323] [p0576] step: count(662), step_time 1407.57, mean_step_time 1429.11, it/s 0.7 | |
| [0130 17:45:42 @multigpu.py:323] [p0574] step: count(655), step_time 1442.15, mean_step_time 1433.74, it/s 0.7 | |
| [0130 17:45:42 @multigpu.py:323] [p0115] step: count(639), step_time 1439.66, mean_step_time 1444.68, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.261552384694 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:45:43 @multigpu.py:323] [p0576] step: count(663), step_time 1436.11, mean_step_time 1431.46, it/s 0.7 | |
| [0130 17:45:43 @multigpu.py:323] [p0574] step: count(656), step_time 1442.82, mean_step_time 1434.32, it/s 0.7 | |
| [0130 17:45:43 @multigpu.py:323] [p0115] step: count(640), step_time 1440.04, mean_step_time 1443.88, it/s 0.69 | |
| [0130 17:45:44 @multigpu.py:323] [p0576] step: count(664), step_time 1388.96, mean_step_time 1430.3, it/s 0.7 | |
| [0130 17:45:45 @multigpu.py:323] [p0574] step: count(657), step_time 1426.29, mean_step_time 1435.58, it/s 0.7 | |
| [0130 17:45:45 @multigpu.py:323] [p0115] step: count(641), step_time 1479.51, mean_step_time 1449.13, it/s 0.69 | |
| [0130 17:45:46 @multigpu.py:323] [p0576] step: count(665), step_time 1508.77, mean_step_time 1433.2, it/s 0.7 | |
| [0130 17:45:46 @multigpu.py:323] [p0574] step: count(658), step_time 1395.82, mean_step_time 1435.92, it/s 0.7 | |
| [0130 17:45:46 @multigpu.py:323] [p0115] step: count(642), step_time 1406.01, mean_step_time 1446.65, it/s 0.69 | |
| [0130 17:45:47 @multigpu.py:323] [p0576] step: count(666), step_time 1405.08, mean_step_time 1431.97, it/s 0.7 | |
| [0130 17:45:47 @multigpu.py:323] [p0574] step: count(659), step_time 1418.44, mean_step_time 1431.82, it/s 0.7 | |
| [0130 17:45:48 @multigpu.py:323] [p0115] step: count(643), step_time 1422.24, mean_step_time 1443.6, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.263147509164 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:45:49 @multigpu.py:323] [p0576] step: count(667), step_time 1399.86, mean_step_time 1430.5, it/s 0.7 | |
| [0130 17:45:49 @multigpu.py:323] [p0574] step: count(660), step_time 1423.0, mean_step_time 1431.4, it/s 0.7 | |
| [0130 17:45:49 @multigpu.py:323] [p0115] step: count(644), step_time 1418.9, mean_step_time 1444.52, it/s 0.69 | |
| [0130 17:45:50 @multigpu.py:323] [p0576] step: count(668), step_time 1365.71, mean_step_time 1425.89, it/s 0.7 | |
| [0130 17:45:50 @multigpu.py:323] [p0574] step: count(661), step_time 1455.74, mean_step_time 1435.33, it/s 0.7 | |
| [0130 17:45:50 @multigpu.py:323] [p0115] step: count(645), step_time 1434.67, mean_step_time 1444.33, it/s 0.69 | |
| [0130 17:45:52 @multigpu.py:323] [p0576] step: count(669), step_time 1467.99, mean_step_time 1424.65, it/s 0.7 | |
| [0130 17:45:52 @multigpu.py:323] [p0574] step: count(662), step_time 1431.18, mean_step_time 1436.4, it/s 0.7 | |
| [0130 17:45:52 @multigpu.py:323] [p0115] step: count(646), step_time 1457.55, mean_step_time 1444.39, it/s 0.69 | |
| [0130 17:45:53 @multigpu.py:323] [p0576] step: count(670), step_time 1433.2, mean_step_time 1425.74, it/s 0.7 | |
| [0130 17:45:53 @multigpu.py:323] [p0574] step: count(663), step_time 1384.25, mean_step_time 1432.23, it/s 0.7 | |
| [0130 17:45:53 @multigpu.py:323] [p0115] step: count(647), step_time 1400.16, mean_step_time 1442.32, it/s 0.69 | |
| [0130 17:45:55 @multigpu.py:323] [p0115] step: count(648), step_time 1497.96, mean_step_time 1440.23, it/s 0.69 | |
| [0130 17:45:55 @multigpu.py:323] [p0574] step: count(664), step_time 1712.16, mean_step_time 1440.26, it/s 0.69 | |
| [0130 17:45:55 @multigpu.py:323] [p0576] step: count(671), step_time 1778.23, mean_step_time 1440.25, it/s 0.69 | |
| [0130 17:45:56 @multigpu.py:323] [p0574] step: count(665), step_time 1379.79, mean_step_time 1435.7, it/s 0.7 | |
| [0130 17:45:56 @multigpu.py:323] [p0576] step: count(672), step_time 1433.5, mean_step_time 1442.32, it/s 0.69 | |
| [0130 17:45:56 @multigpu.py:323] [p0115] step: count(649), step_time 1457.17, mean_step_time 1442.8, it/s 0.69 | |
| [0130 17:45:58 @multigpu.py:323] [p0576] step: count(673), step_time 1417.96, mean_step_time 1442.4, it/s 0.69 | |
| [0130 17:45:58 @multigpu.py:323] [p0574] step: count(666), step_time 1481.13, mean_step_time 1438.82, it/s 0.7 | |
| [0130 17:45:58 @multigpu.py:323] [p0115] step: count(650), step_time 1437.6, mean_step_time 1443.21, it/s 0.69 | |
| [0130 17:45:59 @multigpu.py:323] [p0574] step: count(667), step_time 1362.51, mean_step_time 1435.38, it/s 0.7 | |
| [0130 17:45:59 @multigpu.py:323] [p0576] step: count(674), step_time 1432.19, mean_step_time 1441.83, it/s 0.69 | |
| [0130 17:45:59 @multigpu.py:323] [p0115] step: count(651), step_time 1409.93, mean_step_time 1441.07, it/s 0.69 | |
| [0130 17:46:00 @multigpu.py:323] [p0574] step: count(668), step_time 1451.17, mean_step_time 1434.21, it/s 0.7 | |
| [0130 17:46:00 @multigpu.py:323] [p0115] step: count(652), step_time 1423.18, mean_step_time 1440.68, it/s 0.69 | |
| [0130 17:46:01 @multigpu.py:323] [p0576] step: count(675), step_time 1480.5, mean_step_time 1445.82, it/s 0.69 | |
| [0130 17:46:02 @multigpu.py:323] [p0574] step: count(669), step_time 1377.37, mean_step_time 1433.91, it/s 0.7 | |
| [0130 17:46:02 @multigpu.py:323] [p0576] step: count(676), step_time 1375.98, mean_step_time 1442.97, it/s 0.69 | |
| [0130 17:46:02 @multigpu.py:323] [p0115] step: count(653), step_time 1451.27, mean_step_time 1441.64, it/s 0.69 | |
| [0130 17:46:03 @multigpu.py:323] [p0574] step: count(670), step_time 1407.84, mean_step_time 1433.61, it/s 0.7 | |
| [0130 17:46:03 @multigpu.py:323] [p0576] step: count(677), step_time 1418.6, mean_step_time 1443.88, it/s 0.69 | |
| [0130 17:46:03 @multigpu.py:323] [p0115] step: count(654), step_time 1465.6, mean_step_time 1442.18, it/s 0.69 | |
| [0130 17:46:05 @multigpu.py:323] [p0574] step: count(671), step_time 1388.14, mean_step_time 1431.7, it/s 0.7 | |
| [0130 17:46:05 @multigpu.py:323] [p0576] step: count(678), step_time 1384.74, mean_step_time 1444.21, it/s 0.69 | |
| [0130 17:46:05 @multigpu.py:323] [p0115] step: count(655), step_time 1431.25, mean_step_time 1438.42, it/s 0.7 | |
| [0130 17:46:06 @multigpu.py:323] [p0574] step: count(672), step_time 1407.39, mean_step_time 1430.34, it/s 0.7 | |
| [0130 17:46:06 @multigpu.py:323] [p0576] step: count(679), step_time 1490.79, mean_step_time 1448.17, it/s 0.69 | |
| [0130 17:46:06 @multigpu.py:323] [p0115] step: count(656), step_time 1417.72, mean_step_time 1434.94, it/s 0.7 | |
| [0130 17:46:07 @multigpu.py:323] [p0574] step: count(673), step_time 1434.03, mean_step_time 1431.95, it/s 0.7 | |
| [0130 17:46:08 @multigpu.py:323] [p0576] step: count(680), step_time 1383.0, mean_step_time 1442.31, it/s 0.69 | |
| [0130 17:46:08 @multigpu.py:323] [p0115] step: count(657), step_time 1430.33, mean_step_time 1436.29, it/s 0.7 | |
| [0130 17:46:09 @multigpu.py:323] [p0574] step: count(674), step_time 1434.18, mean_step_time 1432.77, it/s 0.7 | |
| [0130 17:46:09 @multigpu.py:323] [p0576] step: count(681), step_time 1426.53, mean_step_time 1441.76, it/s 0.69 | |
| [0130 17:46:09 @multigpu.py:323] [p0115] step: count(658), step_time 1511.28, mean_step_time 1441.6, it/s 0.69 | |
| [0130 17:46:10 @multigpu.py:323] [p0574] step: count(675), step_time 1409.21, mean_step_time 1431.12, it/s 0.7 | |
| [0130 17:46:10 @multigpu.py:323] [p0576] step: count(682), step_time 1378.39, mean_step_time 1440.31, it/s 0.69 | |
| [0130 17:46:11 @multigpu.py:323] [p0115] step: count(659), step_time 1437.91, mean_step_time 1441.51, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.269616084165 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:46:12 @multigpu.py:323] [p0574] step: count(676), step_time 1388.14, mean_step_time 1428.39, it/s 0.7 | |
| [0130 17:46:12 @multigpu.py:323] [p0576] step: count(683), step_time 1416.55, mean_step_time 1439.33, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.269859721661 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:46:12 @multigpu.py:323] [p0115] step: count(660), step_time 1474.67, mean_step_time 1443.25, it/s 0.69 | |
| [0130 17:46:13 @multigpu.py:323] [p0574] step: count(677), step_time 1442.56, mean_step_time 1429.2, it/s 0.7 | |
| [0130 17:46:13 @multigpu.py:323] [p0576] step: count(684), step_time 1408.24, mean_step_time 1440.29, it/s 0.69 | |
| [0130 17:46:14 @multigpu.py:323] [p0115] step: count(661), step_time 1416.46, mean_step_time 1440.09, it/s 0.69 | |
| [0130 17:46:15 @multigpu.py:323] [p0574] step: count(678), step_time 1410.63, mean_step_time 1429.94, it/s 0.7 | |
| [0130 17:46:15 @multigpu.py:323] [p0576] step: count(685), step_time 1397.51, mean_step_time 1434.73, it/s 0.7 | |
| [0130 17:46:15 @multigpu.py:323] [p0115] step: count(662), step_time 1455.37, mean_step_time 1442.56, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.270699454414 , 1.3 ##### | |
| [u'online', 1.3] | |
| ##### Sending to neptune: active_workers : 0.270699488322 , 3 ##### | |
| receiving | |
| [0130 17:46:16 @multigpu.py:323] [p0574] step: count(679), step_time 1416.51, mean_step_time 1429.85, it/s 0.7 | |
| [0130 17:46:16 @multigpu.py:323] [p0576] step: count(686), step_time 1431.12, mean_step_time 1436.03, it/s 0.7 | |
| [0130 17:46:16 @multigpu.py:323] [p0115] step: count(663), step_time 1399.97, mean_step_time 1441.45, it/s 0.69 | |
| [0130 17:46:17 @multigpu.py:323] [p0574] step: count(680), step_time 1418.6, mean_step_time 1429.63, it/s 0.7 | |
| [0130 17:46:18 @multigpu.py:323] [p0576] step: count(687), step_time 1467.33, mean_step_time 1439.4, it/s 0.69 | |
| [0130 17:46:18 @multigpu.py:323] [p0115] step: count(664), step_time 1441.72, mean_step_time 1442.59, it/s 0.69 | |
| [0130 17:46:19 @multigpu.py:323] [p0574] step: count(681), step_time 1434.2, mean_step_time 1428.55, it/s 0.7 | |
| [0130 17:46:19 @multigpu.py:323] [p0576] step: count(688), step_time 1412.21, mean_step_time 1441.73, it/s 0.69 | |
| [0130 17:46:19 @multigpu.py:323] [p0115] step: count(665), step_time 1450.61, mean_step_time 1443.39, it/s 0.69 | |
| [0130 17:46:20 @multigpu.py:323] [p0574] step: count(682), step_time 1371.35, mean_step_time 1425.56, it/s 0.7 | |
| [0130 17:46:20 @multigpu.py:323] [p0576] step: count(689), step_time 1424.95, mean_step_time 1439.58, it/s 0.69 | |
| [0130 17:46:21 @multigpu.py:323] [p0115] step: count(666), step_time 1402.55, mean_step_time 1440.64, it/s 0.69 | |
| [0130 17:46:22 @multigpu.py:323] [p0574] step: count(683), step_time 1467.57, mean_step_time 1429.72, it/s 0.7 | |
| [0130 17:46:22 @multigpu.py:323] [p0576] step: count(690), step_time 1417.75, mean_step_time 1438.8, it/s 0.7 | |
| [0130 17:46:22 @multigpu.py:323] [p0115] step: count(667), step_time 1430.56, mean_step_time 1442.16, it/s 0.69 | |
| [0130 17:46:24 @multigpu.py:323] [p0576] step: count(691), step_time 1760.88, mean_step_time 1437.94, it/s 0.7 | |
| [0130 17:46:24 @multigpu.py:323] [p0574] step: count(684), step_time 1876.98, mean_step_time 1437.97, it/s 0.7 | |
| [0130 17:46:24 @multigpu.py:323] [p0115] step: count(668), step_time 1414.87, mean_step_time 1438.0, it/s 0.7 | |
| [0130 17:46:25 @multigpu.py:323] [p0576] step: count(692), step_time 1412.65, mean_step_time 1436.89, it/s 0.7 | |
| [0130 17:46:25 @multigpu.py:323] [p0574] step: count(685), step_time 1433.75, mean_step_time 1440.66, it/s 0.69 | |
| [0130 17:46:25 @multigpu.py:323] [p0115] step: count(669), step_time 1442.29, mean_step_time 1437.26, it/s 0.7 | |
| [0130 17:46:26 @multigpu.py:323] [p0576] step: count(693), step_time 1410.89, mean_step_time 1436.54, it/s 0.7 | |
| [0130 17:46:26 @multigpu.py:323] [p0115] step: count(670), step_time 1392.77, mean_step_time 1435.02, it/s 0.7 | |
| [0130 17:46:26 @multigpu.py:323] [p0574] step: count(686), step_time 1417.46, mean_step_time 1437.48, it/s 0.7 | |
| [0130 17:46:28 @multigpu.py:323] [p0115] step: count(671), step_time 1422.11, mean_step_time 1435.62, it/s 0.7 | |
| [0130 17:46:28 @multigpu.py:323] [p0574] step: count(687), step_time 1428.69, mean_step_time 1440.79, it/s 0.69 | |
| [0130 17:46:28 @multigpu.py:323] [p0576] step: count(694), step_time 1488.4, mean_step_time 1439.35, it/s 0.69 | |
| [0130 17:46:29 @multigpu.py:323] [p0576] step: count(695), step_time 1400.05, mean_step_time 1435.33, it/s 0.7 | |
| [0130 17:46:29 @multigpu.py:323] [p0115] step: count(672), step_time 1488.86, mean_step_time 1438.91, it/s 0.69 | |
| [0130 17:46:29 @multigpu.py:323] [p0574] step: count(688), step_time 1470.6, mean_step_time 1441.76, it/s 0.69 | |
| [0130 17:46:31 @multigpu.py:323] [p0574] step: count(689), step_time 1413.16, mean_step_time 1443.55, it/s 0.69 | |
| [0130 17:46:31 @multigpu.py:323] [p0115] step: count(673), step_time 1431.14, mean_step_time 1437.9, it/s 0.7 | |
| [0130 17:46:31 @multigpu.py:323] [p0576] step: count(696), step_time 1501.96, mean_step_time 1441.63, it/s 0.69 | |
| [0130 17:46:32 @multigpu.py:323] [p0576] step: count(697), step_time 1399.25, mean_step_time 1440.66, it/s 0.69 | |
| [0130 17:46:32 @multigpu.py:323] [p0115] step: count(674), step_time 1447.63, mean_step_time 1437.0, it/s 0.7 | |
| [0130 17:46:32 @multigpu.py:323] [p0574] step: count(690), step_time 1477.92, mean_step_time 1447.05, it/s 0.69 | |
| [0130 17:46:34 @multigpu.py:323] [p0574] step: count(691), step_time 1414.94, mean_step_time 1448.39, it/s 0.69 | |
| [0130 17:46:34 @multigpu.py:323] [p0115] step: count(675), step_time 1448.86, mean_step_time 1437.88, it/s 0.7 | |
| [0130 17:46:34 @multigpu.py:323] [p0576] step: count(698), step_time 1515.76, mean_step_time 1447.21, it/s 0.69 | |
| [0130 17:46:35 @multigpu.py:323] [p0115] step: count(676), step_time 1418.69, mean_step_time 1437.93, it/s 0.7 | |
| [0130 17:46:35 @multigpu.py:323] [p0574] step: count(692), step_time 1453.11, mean_step_time 1450.68, it/s 0.69 | |
| [0130 17:46:35 @multigpu.py:323] [p0576] step: count(699), step_time 1455.22, mean_step_time 1445.43, it/s 0.69 | |
| [0130 17:46:36 @multigpu.py:323] [p0115] step: count(677), step_time 1402.6, mean_step_time 1436.55, it/s 0.7 | |
| [0130 17:46:36 @multigpu.py:323] [p0574] step: count(693), step_time 1420.45, mean_step_time 1450.0, it/s 0.69 | |
| [0130 17:46:37 @multigpu.py:323] [p0576] step: count(700), step_time 1405.56, mean_step_time 1446.56, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.276642559436 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.276642559436 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.276642559436 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.276643062168 , -0.00782114826143 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.276643062168 , -0.00080709339818 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.276643062168 , -2.29106926918 ##### | |
| ##### Sending to neptune: value_loss : 0.276643062168 , 1.2907692194 ##### | |
| ##### Sending to neptune: advantage : 0.276643062168 , -9.11761017051e-05 ##### | |
| ##### Sending to neptune: pred_reward : 0.276643062168 , 0.41138869524 ##### | |
| ##### Sending to neptune: max_logit : 0.276643062168 , 0.188500359654 ##### | |
| [u'loss', -0.00782114826142788, -0.0008070933981798589, -2.291069269180298, 1.2907692193984985, -9.117610170505941e-05, 0.41138869524002075, 0.18850035965442657] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.276643491056 , 8843301.94 ##### | |
| ##### Sending to neptune: dp_per_s : 0.276643491056 , 88.940578809 ##### | |
| [u'other', 8843301.94, 88.94057880896713] | |
| receiving | |
| [0130 17:46:38 @multigpu.py:323] [p0574] step: count(694), step_time 1396.48, mean_step_time 1448.12, it/s 0.69 | |
| [0130 17:46:38 @multigpu.py:323] [p0576] step: count(701), step_time 1402.01, mean_step_time 1445.33, it/s 0.69 | |
| [0130 17:46:38 @multigpu.py:323] [p0115] step: count(678), step_time 1503.46, mean_step_time 1436.15, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.277044685218 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:46:39 @multigpu.py:323] [p0574] step: count(695), step_time 1431.64, mean_step_time 1449.24, it/s 0.69 | |
| [0130 17:46:39 @multigpu.py:323] [p0115] step: count(679), step_time 1387.38, mean_step_time 1433.63, it/s 0.7 | |
| [0130 17:46:39 @multigpu.py:323] [p0576] step: count(702), step_time 1469.67, mean_step_time 1449.9, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.277471321887 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:46:41 @multigpu.py:323] [p0574] step: count(696), step_time 1467.27, mean_step_time 1453.2, it/s 0.69 | |
| [0130 17:46:41 @multigpu.py:323] [p0115] step: count(680), step_time 1459.1, mean_step_time 1432.85, it/s 0.7 | |
| [0130 17:46:41 @multigpu.py:323] [p0576] step: count(703), step_time 1390.85, mean_step_time 1448.61, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.277992236084 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:46:42 @multigpu.py:323] [p0574] step: count(697), step_time 1407.87, mean_step_time 1451.46, it/s 0.69 | |
| [0130 17:46:42 @multigpu.py:323] [p0115] step: count(681), step_time 1395.23, mean_step_time 1431.79, it/s 0.7 | |
| [0130 17:46:42 @multigpu.py:323] [p0576] step: count(704), step_time 1424.77, mean_step_time 1449.44, it/s 0.69 | |
| [0130 17:46:44 @multigpu.py:323] [p0115] step: count(682), step_time 1418.87, mean_step_time 1429.96, it/s 0.7 | |
| [0130 17:46:44 @multigpu.py:323] [p0576] step: count(705), step_time 1390.71, mean_step_time 1449.1, it/s 0.69 | |
| [0130 17:46:44 @multigpu.py:323] [p0574] step: count(698), step_time 1461.76, mean_step_time 1454.02, it/s 0.69 | |
| [0130 17:46:45 @multigpu.py:323] [p0115] step: count(683), step_time 1398.62, mean_step_time 1429.9, it/s 0.7 | |
| [0130 17:46:45 @multigpu.py:323] [p0576] step: count(706), step_time 1449.88, mean_step_time 1450.04, it/s 0.69 | |
| [0130 17:46:45 @multigpu.py:323] [p0574] step: count(699), step_time 1457.84, mean_step_time 1456.08, it/s 0.69 | |
| [0130 17:46:46 @multigpu.py:323] [p0576] step: count(707), step_time 1370.72, mean_step_time 1445.21, it/s 0.69 | |
| [0130 17:46:46 @multigpu.py:323] [p0115] step: count(684), step_time 1500.55, mean_step_time 1432.84, it/s 0.7 | |
| [0130 17:46:47 @multigpu.py:323] [p0574] step: count(700), step_time 1450.23, mean_step_time 1457.66, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.279424770806 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.279424770806 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.279424770806 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.279425232477 , -0.00309272995219 ##### | |
| ##### Sending to neptune: policy_loss : 0.279425232477 , 0.459123522043 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.279425232477 , -2.29101920128 ##### | |
| ##### Sending to neptune: value_loss : 0.279425232477 , 1.43602633476 ##### | |
| ##### Sending to neptune: advantage : 0.279425232477 , -0.00201497459784 ##### | |
| ##### Sending to neptune: pred_reward : 0.279425232477 , 0.410510092974 ##### | |
| ##### Sending to neptune: max_logit : 0.279425232477 , 0.18886230886 ##### | |
| [u'loss', -0.003092729952186346, 0.45912352204322815, -2.2910192012786865, 1.4360263347625732, -0.0020149745978415012, 0.4105100929737091, 0.18886230885982513] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.279425686664 , 8863210.75 ##### | |
| ##### Sending to neptune: dp_per_s : 0.279425686664 , 89.0163982094 ##### | |
| [u'other', 8863210.75, 89.01639820936614] | |
| receiving | |
| [0130 17:46:48 @multigpu.py:323] [p0576] step: count(708), step_time 1449.44, mean_step_time 1447.07, it/s 0.69 | |
| [0130 17:46:48 @multigpu.py:323] [p0115] step: count(685), step_time 1449.69, mean_step_time 1432.79, it/s 0.7 | |
| [0130 17:46:48 @multigpu.py:323] [p0574] step: count(701), step_time 1451.4, mean_step_time 1458.52, it/s 0.69 | |
| [0130 17:46:49 @multigpu.py:323] [p0115] step: count(686), step_time 1420.24, mean_step_time 1433.68, it/s 0.7 | |
| [0130 17:46:49 @multigpu.py:323] [p0576] step: count(709), step_time 1505.57, mean_step_time 1451.1, it/s 0.69 | |
| [0130 17:46:49 @multigpu.py:323] [p0574] step: count(702), step_time 1430.73, mean_step_time 1461.49, it/s 0.68 | |
| [0130 17:46:51 @multigpu.py:323] [p0576] step: count(710), step_time 1397.3, mean_step_time 1450.08, it/s 0.69 | |
| [0130 17:46:51 @multigpu.py:323] [p0115] step: count(687), step_time 1450.65, mean_step_time 1434.68, it/s 0.7 | |
| [0130 17:46:51 @multigpu.py:323] [p0574] step: count(703), step_time 1462.6, mean_step_time 1461.24, it/s 0.68 | |
| [0130 17:46:52 @multigpu.py:323] [p0115] step: count(688), step_time 1520.92, mean_step_time 1439.98, it/s 0.69 | |
| [0130 17:46:52 @multigpu.py:323] [p0576] step: count(711), step_time 1559.96, mean_step_time 1440.03, it/s 0.69 | |
| [0130 17:46:52 @multigpu.py:323] [p0574] step: count(704), step_time 1453.72, mean_step_time 1440.08, it/s 0.69 | |
| [0130 17:46:54 @multigpu.py:323] [p0574] step: count(705), step_time 1416.9, mean_step_time 1439.24, it/s 0.69 | |
| [0130 17:46:54 @multigpu.py:323] [p0576] step: count(712), step_time 1449.48, mean_step_time 1441.87, it/s 0.69 | |
| [0130 17:46:54 @multigpu.py:323] [p0115] step: count(689), step_time 1481.8, mean_step_time 1441.96, it/s 0.69 | |
| [0130 17:46:55 @multigpu.py:323] [p0574] step: count(706), step_time 1416.82, mean_step_time 1439.21, it/s 0.69 | |
| [0130 17:46:55 @multigpu.py:323] [p0576] step: count(713), step_time 1436.16, mean_step_time 1443.14, it/s 0.69 | |
| [0130 17:46:55 @multigpu.py:323] [p0115] step: count(690), step_time 1442.39, mean_step_time 1444.44, it/s 0.69 | |
| [0130 17:46:57 @multigpu.py:323] [p0574] step: count(707), step_time 1404.84, mean_step_time 1438.01, it/s 0.7 | |
| [0130 17:46:57 @multigpu.py:323] [p0576] step: count(714), step_time 1425.54, mean_step_time 1439.99, it/s 0.69 | |
| [0130 17:46:57 @multigpu.py:323] [p0115] step: count(691), step_time 1429.53, mean_step_time 1444.81, it/s 0.69 | |
| [0130 17:46:58 @multigpu.py:323] [p0576] step: count(715), step_time 1397.21, mean_step_time 1439.85, it/s 0.69 | |
| [0130 17:46:58 @multigpu.py:323] [p0574] step: count(708), step_time 1485.55, mean_step_time 1438.76, it/s 0.7 | |
| [0130 17:46:58 @multigpu.py:323] [p0115] step: count(692), step_time 1433.3, mean_step_time 1442.03, it/s 0.69 | |
| [0130 17:46:59 @multigpu.py:323] [p0576] step: count(716), step_time 1443.1, mean_step_time 1436.91, it/s 0.7 | |
| [0130 17:47:00 @multigpu.py:323] [p0574] step: count(709), step_time 1439.46, mean_step_time 1440.08, it/s 0.69 | |
| [0130 17:47:00 @multigpu.py:323] [p0115] step: count(693), step_time 1417.74, mean_step_time 1441.36, it/s 0.69 | |
| [0130 17:47:01 @multigpu.py:323] [p0574] step: count(710), step_time 1380.25, mean_step_time 1435.19, it/s 0.7 | |
| [0130 17:47:01 @multigpu.py:323] [p0576] step: count(717), step_time 1399.54, mean_step_time 1436.92, it/s 0.7 | |
| [0130 17:47:01 @multigpu.py:323] [p0115] step: count(694), step_time 1413.25, mean_step_time 1439.64, it/s 0.69 | |
| [0130 17:47:02 @multigpu.py:323] [p0576] step: count(718), step_time 1416.18, mean_step_time 1431.94, it/s 0.7 | |
| [0130 17:47:02 @multigpu.py:323] [p0115] step: count(695), step_time 1381.65, mean_step_time 1436.28, it/s 0.7 | |
| [0130 17:47:02 @multigpu.py:323] [p0574] step: count(711), step_time 1458.54, mean_step_time 1437.37, it/s 0.7 | |
| [0130 17:47:04 @multigpu.py:323] [p0576] step: count(719), step_time 1403.66, mean_step_time 1429.37, it/s 0.7 | |
| [0130 17:47:04 @multigpu.py:323] [p0115] step: count(696), step_time 1425.07, mean_step_time 1436.6, it/s 0.7 | |
| [0130 17:47:04 @multigpu.py:323] [p0574] step: count(712), step_time 1508.01, mean_step_time 1440.12, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.284432896111 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 17:47:05 @multigpu.py:323] [p0576] step: count(720), step_time 1396.07, mean_step_time 1428.89, it/s 0.7 | |
| [0130 17:47:05 @multigpu.py:323] [p0115] step: count(697), step_time 1415.77, mean_step_time 1437.26, it/s 0.7 | |
| [0130 17:47:05 @multigpu.py:323] [p0574] step: count(713), step_time 1400.91, mean_step_time 1439.14, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.284738871919 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:47:07 @multigpu.py:323] [p0576] step: count(721), step_time 1397.28, mean_step_time 1428.65, it/s 0.7 | |
| [0130 17:47:07 @multigpu.py:323] [p0115] step: count(698), step_time 1417.27, mean_step_time 1432.95, it/s 0.7 | |
| [0130 17:47:07 @multigpu.py:323] [p0574] step: count(714), step_time 1433.64, mean_step_time 1441.0, it/s 0.69 | |
| [0130 17:47:08 @multigpu.py:323] [p0576] step: count(722), step_time 1401.82, mean_step_time 1425.26, it/s 0.7 | |
| [0130 17:47:08 @multigpu.py:323] [p0115] step: count(699), step_time 1443.38, mean_step_time 1435.75, it/s 0.7 | |
| [0130 17:47:08 @multigpu.py:323] [p0574] step: count(715), step_time 1549.43, mean_step_time 1446.89, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.285529997481 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:47:09 @multigpu.py:323] [p0576] step: count(723), step_time 1446.5, mean_step_time 1428.04, it/s 0.7 | |
| [0130 17:47:09 @multigpu.py:323] [p0115] step: count(700), step_time 1448.74, mean_step_time 1435.23, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.285801506374 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.285801506374 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.285801506374 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.285802074141 , -0.0108789335936 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.285802074141 , -0.324730724096 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.285802074141 , -2.29079914093 ##### | |
| ##### Sending to neptune: value_loss : 0.285802074141 , 1.22302639484 ##### | |
| ##### Sending to neptune: advantage : 0.285802074141 , 0.0014331714483 ##### | |
| ##### Sending to neptune: pred_reward : 0.285802074141 , 0.409147292376 ##### | |
| ##### Sending to neptune: max_logit : 0.285802074141 , 0.190351292491 ##### | |
| [u'loss', -0.01087893359363079, -0.3247307240962982, -2.290799140930176, 1.2230263948440552, 0.0014331714482977986, 0.4091472923755646, 0.19035129249095917] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.285802551111 , 8855540.19 ##### | |
| ##### Sending to neptune: dp_per_s : 0.285802551111 , 88.9247276744 ##### | |
| [u'other', 8855540.19, 88.92472767440891] | |
| receiving | |
| [0130 17:47:10 @multigpu.py:323] [p0574] step: count(716), step_time 1499.9, mean_step_time 1448.52, it/s 0.69 | |
| [0130 17:47:11 @multigpu.py:323] [p0576] step: count(724), step_time 1457.24, mean_step_time 1429.67, it/s 0.7 | |
| [0130 17:47:11 @multigpu.py:323] [p0115] step: count(701), step_time 1416.31, mean_step_time 1436.29, it/s 0.7 | |
| [0130 17:47:11 @multigpu.py:323] [p0574] step: count(717), step_time 1476.77, mean_step_time 1451.97, it/s 0.69 | |
| [0130 17:47:12 @multigpu.py:323] [p0576] step: count(725), step_time 1449.38, mean_step_time 1432.6, it/s 0.7 | |
| [0130 17:47:12 @multigpu.py:323] [p0115] step: count(702), step_time 1464.9, mean_step_time 1438.59, it/s 0.7 | |
| [0130 17:47:13 @multigpu.py:323] [p0574] step: count(718), step_time 1487.39, mean_step_time 1453.25, it/s 0.69 | |
| [0130 17:47:14 @multigpu.py:323] [p0576] step: count(726), step_time 1405.97, mean_step_time 1430.41, it/s 0.7 | |
| [0130 17:47:14 @multigpu.py:323] [p0115] step: count(703), step_time 1397.31, mean_step_time 1438.52, it/s 0.7 | |
| [0130 17:47:14 @multigpu.py:323] [p0574] step: count(719), step_time 1456.28, mean_step_time 1453.17, it/s 0.69 | |
| [0130 17:47:15 @multigpu.py:323] [p0576] step: count(727), step_time 1413.06, mean_step_time 1432.52, it/s 0.7 | |
| [0130 17:47:15 @multigpu.py:323] [p0115] step: count(704), step_time 1476.83, mean_step_time 1437.34, it/s 0.7 | |
| [0130 17:47:16 @multigpu.py:323] [p0574] step: count(720), step_time 1470.73, mean_step_time 1454.19, it/s 0.69 | |
| [0130 17:47:17 @multigpu.py:323] [p0576] step: count(728), step_time 1434.16, mean_step_time 1431.76, it/s 0.7 | |
| [0130 17:47:17 @multigpu.py:323] [p0115] step: count(705), step_time 1408.52, mean_step_time 1435.28, it/s 0.7 | |
| [0130 17:47:17 @multigpu.py:323] [p0574] step: count(721), step_time 1504.53, mean_step_time 1456.85, it/s 0.69 | |
| [0130 17:47:18 @multigpu.py:323] [p0576] step: count(729), step_time 1437.14, mean_step_time 1428.34, it/s 0.7 | |
| [0130 17:47:18 @multigpu.py:323] [p0115] step: count(706), step_time 1473.58, mean_step_time 1437.95, it/s 0.7 | |
| [0130 17:47:19 @multigpu.py:323] [p0574] step: count(722), step_time 1579.61, mean_step_time 1464.29, it/s 0.68 | |
| [0130 17:47:19 @multigpu.py:323] [p0576] step: count(730), step_time 1430.95, mean_step_time 1430.02, it/s 0.7 | |
| [0130 17:47:20 @multigpu.py:323] [p0115] step: count(707), step_time 1453.48, mean_step_time 1438.09, it/s 0.7 | |
| [0130 17:47:20 @multigpu.py:323] [p0574] step: count(723), step_time 1468.18, mean_step_time 1464.57, it/s 0.68 | |
| [0130 17:47:22 @multigpu.py:323] [p0574] step: count(724), step_time 1484.95, mean_step_time 1466.14, it/s 0.68 | |
| [0130 17:47:22 @multigpu.py:323] [p0115] step: count(708), step_time 2084.26, mean_step_time 1466.25, it/s 0.68 | |
| [0130 17:47:22 @multigpu.py:323] [p0576] step: count(731), step_time 2284.87, mean_step_time 1466.27, it/s 0.68 | |
| [0130 17:47:23 @multigpu.py:323] [p0576] step: count(732), step_time 1417.49, mean_step_time 1464.67, it/s 0.68 | |
| [0130 17:47:23 @multigpu.py:323] [p0115] step: count(709), step_time 1423.44, mean_step_time 1463.34, it/s 0.68 | |
| [0130 17:47:23 @multigpu.py:323] [p0574] step: count(725), step_time 1466.45, mean_step_time 1468.61, it/s 0.68 | |
| [0130 17:47:25 @multigpu.py:323] [p0576] step: count(733), step_time 1450.55, mean_step_time 1465.39, it/s 0.68 | |
| [0130 17:47:25 @multigpu.py:323] [p0115] step: count(710), step_time 1444.98, mean_step_time 1463.47, it/s 0.68 | |
| [0130 17:47:25 @multigpu.py:323] [p0574] step: count(726), step_time 1466.99, mean_step_time 1471.12, it/s 0.68 | |
| [0130 17:47:26 @multigpu.py:323] [p0576] step: count(734), step_time 1418.3, mean_step_time 1465.02, it/s 0.68 | |
| [0130 17:47:26 @multigpu.py:323] [p0115] step: count(711), step_time 1459.8, mean_step_time 1464.98, it/s 0.68 | |
| [0130 17:47:26 @multigpu.py:323] [p0574] step: count(727), step_time 1479.33, mean_step_time 1474.85, it/s 0.68 | |
| [0130 17:47:27 @multigpu.py:323] [p0576] step: count(735), step_time 1407.67, mean_step_time 1465.55, it/s 0.68 | |
| [0130 17:47:27 @multigpu.py:323] [p0115] step: count(712), step_time 1419.85, mean_step_time 1464.31, it/s 0.68 | |
| [0130 17:47:28 @multigpu.py:323] [p0574] step: count(728), step_time 1647.55, mean_step_time 1482.95, it/s 0.67 | |
| [0130 17:47:29 @multigpu.py:323] [p0576] step: count(736), step_time 1399.97, mean_step_time 1463.39, it/s 0.68 | |
| [0130 17:47:29 @multigpu.py:323] [p0115] step: count(713), step_time 1468.94, mean_step_time 1466.87, it/s 0.68 | |
| [0130 17:47:29 @multigpu.py:323] [p0574] step: count(729), step_time 1496.2, mean_step_time 1485.78, it/s 0.67 | |
| [0130 17:47:30 @multigpu.py:323] [p0576] step: count(737), step_time 1421.17, mean_step_time 1464.47, it/s 0.68 | |
| [0130 17:47:30 @multigpu.py:323] [p0115] step: count(714), step_time 1443.28, mean_step_time 1468.37, it/s 0.68 | |
| [0130 17:47:31 @multigpu.py:323] [p0574] step: count(730), step_time 1426.75, mean_step_time 1488.11, it/s 0.67 | |
| [0130 17:47:32 @multigpu.py:323] [p0576] step: count(738), step_time 1426.87, mean_step_time 1465.01, it/s 0.68 | |
| [0130 17:47:32 @multigpu.py:323] [p0115] step: count(715), step_time 1442.87, mean_step_time 1471.43, it/s 0.68 | |
| [0130 17:47:32 @multigpu.py:323] [p0574] step: count(731), step_time 1566.06, mean_step_time 1493.48, it/s 0.67 | |
| [0130 17:47:33 @multigpu.py:323] [p0576] step: count(739), step_time 1378.06, mean_step_time 1463.73, it/s 0.68 | |
| [0130 17:47:33 @multigpu.py:323] [p0115] step: count(716), step_time 1397.81, mean_step_time 1470.07, it/s 0.68 | |
| [0130 17:47:34 @multigpu.py:323] [p0574] step: count(732), step_time 1510.79, mean_step_time 1493.62, it/s 0.67 | |
| [0130 17:47:34 @multigpu.py:323] [p0576] step: count(740), step_time 1453.38, mean_step_time 1466.59, it/s 0.68 | |
| [0130 17:47:35 @multigpu.py:323] [p0115] step: count(717), step_time 1452.81, mean_step_time 1471.92, it/s 0.68 | |
| [0130 17:47:35 @multigpu.py:323] [p0574] step: count(733), step_time 1517.2, mean_step_time 1499.44, it/s 0.67 | |
| [0130 17:47:36 @multigpu.py:323] [p0576] step: count(741), step_time 1380.13, mean_step_time 1465.73, it/s 0.68 | |
| [0130 17:47:36 @multigpu.py:323] [p0115] step: count(718), step_time 1455.88, mean_step_time 1473.85, it/s 0.68 | |
| [0130 17:47:37 @multigpu.py:323] [p0574] step: count(734), step_time 1566.86, mean_step_time 1506.1, it/s 0.66 | |
| [0130 17:47:37 @multigpu.py:323] [p0576] step: count(742), step_time 1463.09, mean_step_time 1468.8, it/s 0.68 | |
| [0130 17:47:38 @multigpu.py:323] [p0115] step: count(719), step_time 1447.73, mean_step_time 1474.07, it/s 0.68 | |
| [0130 17:47:38 @multigpu.py:323] [p0574] step: count(735), step_time 1595.75, mean_step_time 1508.41, it/s 0.66 | |
| [0130 17:47:39 @multigpu.py:323] [p0576] step: count(743), step_time 1368.8, mean_step_time 1464.91, it/s 0.68 | |
| [0130 17:47:39 @multigpu.py:323] [p0115] step: count(720), step_time 1455.55, mean_step_time 1474.41, it/s 0.68 | |
| [0130 17:47:40 @multigpu.py:323] [p0574] step: count(736), step_time 1489.2, mean_step_time 1507.88, it/s 0.66 | |
| [0130 17:47:40 @multigpu.py:323] [p0576] step: count(744), step_time 1383.44, mean_step_time 1461.22, it/s 0.68 | |
| [0130 17:47:40 @multigpu.py:323] [p0115] step: count(721), step_time 1451.48, mean_step_time 1476.16, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.294545846913 , 1.8 ##### | |
| [u'online', 1.8] | |
| ##### Sending to neptune: active_workers : 0.294546027448 , 3 ##### | |
| receiving | |
| [0130 17:47:41 @multigpu.py:323] [p0576] step: count(745), step_time 1408.92, mean_step_time 1459.2, it/s 0.69 | |
| [0130 17:47:42 @multigpu.py:323] [p0574] step: count(737), step_time 1684.3, mean_step_time 1518.26, it/s 0.66 | |
| [0130 17:47:42 @multigpu.py:323] [p0115] step: count(722), step_time 1436.37, mean_step_time 1474.74, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.294824038612 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.294907082717 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:47:43 @multigpu.py:323] [p0576] step: count(746), step_time 1430.48, mean_step_time 1460.42, it/s 0.68 | |
| [0130 17:47:43 @multigpu.py:323] [p0574] step: count(738), step_time 1636.61, mean_step_time 1525.72, it/s 0.66 | |
| [0130 17:47:43 @multigpu.py:323] [p0115] step: count(723), step_time 1376.08, mean_step_time 1473.68, it/s 0.68 | |
| [0130 17:47:44 @multigpu.py:323] [p0576] step: count(747), step_time 1489.13, mean_step_time 1464.23, it/s 0.68 | |
| [0130 17:47:45 @multigpu.py:323] [p0115] step: count(724), step_time 1441.64, mean_step_time 1471.92, it/s 0.68 | |
| [0130 17:47:45 @multigpu.py:323] [p0574] step: count(739), step_time 1543.1, mean_step_time 1530.06, it/s 0.65 | |
| [0130 17:47:46 @multigpu.py:323] [p0576] step: count(748), step_time 1405.09, mean_step_time 1462.77, it/s 0.68 | |
| [0130 17:47:46 @multigpu.py:323] [p0115] step: count(725), step_time 1436.45, mean_step_time 1473.31, it/s 0.68 | |
| [0130 17:47:46 @multigpu.py:323] [p0574] step: count(740), step_time 1569.09, mean_step_time 1534.98, it/s 0.65 | |
| [0130 17:47:47 @multigpu.py:323] [p0576] step: count(749), step_time 1407.46, mean_step_time 1461.29, it/s 0.68 | |
| [0130 17:47:48 @multigpu.py:323] [p0115] step: count(726), step_time 1458.22, mean_step_time 1472.55, it/s 0.68 | |
| [0130 17:47:48 @multigpu.py:323] [p0574] step: count(741), step_time 1437.3, mean_step_time 1531.61, it/s 0.65 | |
| [0130 17:47:49 @multigpu.py:323] [p0576] step: count(750), step_time 1427.61, mean_step_time 1461.12, it/s 0.68 | |
| [0130 17:47:49 @multigpu.py:323] [p0115] step: count(727), step_time 1435.61, mean_step_time 1471.65, it/s 0.68 | |
| [0130 17:47:49 @multigpu.py:323] [p0574] step: count(742), step_time 1468.07, mean_step_time 1526.04, it/s 0.66 | |
| [0130 17:47:50 @multigpu.py:323] [p0576] step: count(751), step_time 1362.66, mean_step_time 1415.01, it/s 0.71 | |
| [0130 17:47:51 @multigpu.py:323] [p0115] step: count(728), step_time 2405.93, mean_step_time 1487.74, it/s 0.67 | |
| [0130 17:47:51 @multigpu.py:323] [p0574] step: count(743), step_time 2188.64, mean_step_time 1562.06, it/s 0.64 | |
| [0130 17:47:51 @multigpu.py:323] [p0576] step: count(752), step_time 1456.0, mean_step_time 1416.94, it/s 0.71 | |
| [0130 17:47:53 @multigpu.py:323] [p0576] step: count(753), step_time 1413.45, mean_step_time 1415.08, it/s 0.71 | |
| [0130 17:47:53 @multigpu.py:323] [p0115] step: count(729), step_time 1422.24, mean_step_time 1487.68, it/s 0.67 | |
| [0130 17:47:53 @multigpu.py:323] [p0574] step: count(744), step_time 1479.45, mean_step_time 1561.78, it/s 0.64 | |
| [0130 17:47:54 @multigpu.py:323] [p0576] step: count(754), step_time 1401.71, mean_step_time 1414.25, it/s 0.71 | |
| [0130 17:47:54 @multigpu.py:323] [p0115] step: count(730), step_time 1433.0, mean_step_time 1487.08, it/s 0.67 | |
| [0130 17:47:54 @multigpu.py:323] [p0574] step: count(745), step_time 1522.8, mean_step_time 1564.6, it/s 0.64 | |
| [0130 17:47:56 @multigpu.py:323] [p0576] step: count(755), step_time 1378.85, mean_step_time 1412.81, it/s 0.71 | |
| [0130 17:47:56 @multigpu.py:323] [p0115] step: count(731), step_time 1391.26, mean_step_time 1483.65, it/s 0.67 | |
| [0130 17:47:56 @multigpu.py:323] [p0574] step: count(746), step_time 1454.22, mean_step_time 1563.96, it/s 0.64 | |
| [0130 17:47:57 @multigpu.py:323] [p0576] step: count(756), step_time 1429.84, mean_step_time 1414.31, it/s 0.71 | |
| [0130 17:47:57 @multigpu.py:323] [p0115] step: count(732), step_time 1433.66, mean_step_time 1484.34, it/s 0.67 | |
| [0130 17:47:57 @multigpu.py:323] [p0574] step: count(747), step_time 1442.47, mean_step_time 1562.12, it/s 0.64 | |
| [0130 17:47:58 @multigpu.py:323] [p0576] step: count(757), step_time 1381.52, mean_step_time 1412.32, it/s 0.71 | |
| [0130 17:47:59 @multigpu.py:323] [p0115] step: count(733), step_time 1428.08, mean_step_time 1482.3, it/s 0.67 | |
| [0130 17:47:59 @multigpu.py:323] [p0574] step: count(748), step_time 1438.44, mean_step_time 1551.67, it/s 0.64 | |
| [0130 17:48:00 @multigpu.py:323] [p0576] step: count(758), step_time 1397.49, mean_step_time 1410.85, it/s 0.71 | |
| [0130 17:48:00 @multigpu.py:323] [p0115] step: count(734), step_time 1458.07, mean_step_time 1483.04, it/s 0.67 | |
| [0130 17:48:00 @multigpu.py:323] [p0574] step: count(749), step_time 1499.59, mean_step_time 1551.84, it/s 0.64 | |
| [0130 17:48:01 @multigpu.py:323] [p0576] step: count(759), step_time 1388.58, mean_step_time 1411.38, it/s 0.71 | |
| [0130 17:48:01 @multigpu.py:323] [p0115] step: count(735), step_time 1409.96, mean_step_time 1481.39, it/s 0.68 | |
| [0130 17:48:02 @multigpu.py:323] [p0574] step: count(750), step_time 1420.14, mean_step_time 1551.5, it/s 0.64 | |
| [0130 17:48:03 @multigpu.py:323] [p0576] step: count(760), step_time 1410.87, mean_step_time 1409.26, it/s 0.71 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.300571438273 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:48:03 @multigpu.py:323] [p0115] step: count(736), step_time 1424.39, mean_step_time 1482.72, it/s 0.67 | |
| [0130 17:48:03 @multigpu.py:323] [p0574] step: count(751), step_time 1604.72, mean_step_time 1553.44, it/s 0.64 | |
| [0130 17:48:04 @multigpu.py:323] [p0576] step: count(761), step_time 1423.25, mean_step_time 1411.41, it/s 0.71 | |
| [0130 17:48:04 @multigpu.py:323] [p0115] step: count(737), step_time 1482.03, mean_step_time 1484.18, it/s 0.67 | |
| [0130 17:48:05 @multigpu.py:323] [p0574] step: count(752), step_time 1496.24, mean_step_time 1552.71, it/s 0.64 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.301201084124 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:48:06 @multigpu.py:323] [p0576] step: count(762), step_time 1436.6, mean_step_time 1410.09, it/s 0.71 | |
| [0130 17:48:06 @multigpu.py:323] [p0115] step: count(738), step_time 1408.71, mean_step_time 1481.82, it/s 0.67 | |
| [0130 17:48:06 @multigpu.py:323] [p0574] step: count(753), step_time 1493.95, mean_step_time 1551.55, it/s 0.64 | |
| [0130 17:48:07 @multigpu.py:323] [p0576] step: count(763), step_time 1437.35, mean_step_time 1413.51, it/s 0.71 | |
| [0130 17:48:07 @multigpu.py:323] [p0115] step: count(739), step_time 1434.51, mean_step_time 1481.16, it/s 0.68 | |
| [0130 17:48:08 @multigpu.py:323] [p0574] step: count(754), step_time 1429.3, mean_step_time 1544.67, it/s 0.65 | |
| [0130 17:48:08 @multigpu.py:323] [p0576] step: count(764), step_time 1373.29, mean_step_time 1413.01, it/s 0.71 | |
| [0130 17:48:09 @multigpu.py:323] [p0115] step: count(740), step_time 1423.09, mean_step_time 1479.54, it/s 0.68 | |
| [0130 17:48:09 @multigpu.py:323] [p0574] step: count(755), step_time 1581.44, mean_step_time 1543.95, it/s 0.65 | |
| [0130 17:48:10 @multigpu.py:323] [p0576] step: count(765), step_time 1404.71, mean_step_time 1412.8, it/s 0.71 | |
| [0130 17:48:10 @multigpu.py:323] [p0115] step: count(741), step_time 1420.29, mean_step_time 1477.98, it/s 0.68 | |
| [0130 17:48:11 @multigpu.py:323] [p0574] step: count(756), step_time 1490.79, mean_step_time 1544.03, it/s 0.65 | |
| [0130 17:48:11 @multigpu.py:323] [p0576] step: count(766), step_time 1401.43, mean_step_time 1411.34, it/s 0.71 | |
| [0130 17:48:12 @multigpu.py:323] [p0115] step: count(742), step_time 1500.94, mean_step_time 1481.21, it/s 0.68 | |
| [0130 17:48:12 @multigpu.py:323] [p0574] step: count(757), step_time 1599.73, mean_step_time 1539.8, it/s 0.65 | |
| [0130 17:48:13 @multigpu.py:323] [p0576] step: count(767), step_time 1473.04, mean_step_time 1410.54, it/s 0.71 | |
| [0130 17:48:13 @multigpu.py:323] [p0115] step: count(743), step_time 1456.1, mean_step_time 1485.21, it/s 0.67 | |
| [0130 17:48:14 @multigpu.py:323] [p0574] step: count(758), step_time 1563.87, mean_step_time 1536.17, it/s 0.65 | |
| [0130 17:48:14 @multigpu.py:323] [p0576] step: count(768), step_time 1401.15, mean_step_time 1410.34, it/s 0.71 | |
| [0130 17:48:14 @multigpu.py:323] [p0115] step: count(744), step_time 1457.8, mean_step_time 1486.02, it/s 0.67 | |
| [0130 17:48:15 @multigpu.py:323] [p0574] step: count(759), step_time 1448.84, mean_step_time 1531.46, it/s 0.65 | |
| [0130 17:48:15 @multigpu.py:323] [p0576] step: count(769), step_time 1433.32, mean_step_time 1411.64, it/s 0.71 | |
| [0130 17:48:16 @multigpu.py:323] [p0115] step: count(745), step_time 1430.88, mean_step_time 1485.74, it/s 0.67 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.304254507754 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:48:17 @multigpu.py:323] [p0574] step: count(760), step_time 1419.68, mean_step_time 1523.98, it/s 0.66 | |
| [0130 17:48:17 @multigpu.py:323] [p0576] step: count(770), step_time 1421.96, mean_step_time 1411.35, it/s 0.71 | |
| [0130 17:48:17 @multigpu.py:323] [p0115] step: count(746), step_time 1416.76, mean_step_time 1483.67, it/s 0.67 | |
| [0130 17:48:18 @multigpu.py:323] [p0574] step: count(761), step_time 1396.02, mean_step_time 1521.92, it/s 0.66 | |
| [0130 17:48:18 @multigpu.py:323] [p0576] step: count(771), step_time 1417.25, mean_step_time 1414.08, it/s 0.71 | |
| [0130 17:48:19 @multigpu.py:323] [p0115] step: count(747), step_time 1478.87, mean_step_time 1485.83, it/s 0.67 | |
| [0130 17:48:20 @multigpu.py:323] [p0574] step: count(762), step_time 1405.99, mean_step_time 1518.82, it/s 0.66 | |
| [0130 17:48:21 @multigpu.py:323] [p0574] step: count(763), step_time 1430.9, mean_step_time 1480.93, it/s 0.68 | |
| [0130 17:48:21 @multigpu.py:323] [p0115] step: count(748), step_time 2311.05, mean_step_time 1481.08, it/s 0.68 | |
| [0130 17:48:21 @multigpu.py:323] [p0576] step: count(772), step_time 2796.65, mean_step_time 1481.11, it/s 0.68 | |
| [0130 17:48:22 @multigpu.py:323] [p0115] step: count(749), step_time 1422.93, mean_step_time 1481.12, it/s 0.68 | |
| [0130 17:48:23 @multigpu.py:323] [p0574] step: count(764), step_time 1436.66, mean_step_time 1478.79, it/s 0.68 | |
| [0130 17:48:23 @multigpu.py:323] [p0576] step: count(773), step_time 1463.31, mean_step_time 1483.61, it/s 0.67 | |
| [0130 17:48:24 @multigpu.py:323] [p0115] step: count(750), step_time 1418.68, mean_step_time 1480.4, it/s 0.68 | |
| [0130 17:48:24 @multigpu.py:323] [p0576] step: count(774), step_time 1410.31, mean_step_time 1484.04, it/s 0.67 | |
| [0130 17:48:24 @multigpu.py:323] [p0574] step: count(765), step_time 1467.2, mean_step_time 1476.01, it/s 0.68 | |
| [0130 17:48:25 @multigpu.py:323] [p0115] step: count(751), step_time 1426.06, mean_step_time 1482.14, it/s 0.67 | |
| [0130 17:48:25 @multigpu.py:323] [p0576] step: count(775), step_time 1424.35, mean_step_time 1486.31, it/s 0.67 | |
| [0130 17:48:25 @multigpu.py:323] [p0574] step: count(766), step_time 1451.39, mean_step_time 1475.87, it/s 0.68 | |
| [0130 17:48:27 @multigpu.py:323] [p0115] step: count(752), step_time 1385.07, mean_step_time 1479.71, it/s 0.68 | |
| [0130 17:48:27 @multigpu.py:323] [p0576] step: count(776), step_time 1461.88, mean_step_time 1487.92, it/s 0.67 | |
| [0130 17:48:27 @multigpu.py:323] [p0574] step: count(767), step_time 1466.48, mean_step_time 1477.07, it/s 0.68 | |
| [0130 17:48:28 @multigpu.py:323] [p0115] step: count(753), step_time 1429.92, mean_step_time 1479.81, it/s 0.68 | |
| [0130 17:48:28 @multigpu.py:323] [p0576] step: count(777), step_time 1398.9, mean_step_time 1488.78, it/s 0.67 | |
| [0130 17:48:28 @multigpu.py:323] [p0574] step: count(768), step_time 1492.78, mean_step_time 1479.79, it/s 0.68 | |
| [0130 17:48:30 @multigpu.py:323] [p0115] step: count(754), step_time 1413.36, mean_step_time 1477.57, it/s 0.68 | |
| [0130 17:48:30 @multigpu.py:323] [p0576] step: count(778), step_time 1383.07, mean_step_time 1488.06, it/s 0.67 | |
| [0130 17:48:30 @multigpu.py:323] [p0574] step: count(769), step_time 1375.72, mean_step_time 1473.59, it/s 0.68 | |
| [0130 17:48:31 @multigpu.py:323] [p0115] step: count(755), step_time 1414.28, mean_step_time 1477.79, it/s 0.68 | |
| [0130 17:48:31 @multigpu.py:323] [p0576] step: count(779), step_time 1457.36, mean_step_time 1491.5, it/s 0.67 | |
| [0130 17:48:31 @multigpu.py:323] [p0574] step: count(770), step_time 1419.43, mean_step_time 1473.56, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.308666802446 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:48:32 @multigpu.py:323] [p0115] step: count(756), step_time 1425.67, mean_step_time 1477.85, it/s 0.68 | |
| [0130 17:48:32 @multigpu.py:323] [p0576] step: count(780), step_time 1396.16, mean_step_time 1490.77, it/s 0.67 | |
| [0130 17:48:33 @multigpu.py:323] [p0574] step: count(771), step_time 1385.33, mean_step_time 1462.59, it/s 0.68 | |
| [0130 17:48:34 @multigpu.py:323] [p0115] step: count(757), step_time 1374.16, mean_step_time 1472.46, it/s 0.68 | |
| [0130 17:48:34 @multigpu.py:323] [p0576] step: count(781), step_time 1418.66, mean_step_time 1490.54, it/s 0.67 | |
| [0130 17:48:34 @multigpu.py:323] [p0574] step: count(772), step_time 1427.58, mean_step_time 1459.16, it/s 0.69 | |
| [0130 17:48:35 @multigpu.py:323] [p0115] step: count(758), step_time 1386.69, mean_step_time 1471.36, it/s 0.68 | |
| [0130 17:48:35 @multigpu.py:323] [p0576] step: count(782), step_time 1421.39, mean_step_time 1489.78, it/s 0.67 | |
| [0130 17:48:35 @multigpu.py:323] [p0574] step: count(773), step_time 1451.68, mean_step_time 1457.04, it/s 0.69 | |
| [0130 17:48:37 @multigpu.py:323] [p0115] step: count(759), step_time 1447.37, mean_step_time 1472.0, it/s 0.68 | |
| [0130 17:48:37 @multigpu.py:323] [p0576] step: count(783), step_time 1463.02, mean_step_time 1491.06, it/s 0.67 | |
| [0130 17:48:37 @multigpu.py:323] [p0574] step: count(774), step_time 1477.53, mean_step_time 1459.45, it/s 0.69 | |
| [0130 17:48:38 @multigpu.py:323] [p0115] step: count(760), step_time 1403.82, mean_step_time 1471.04, it/s 0.68 | |
| [0130 17:48:38 @multigpu.py:323] [p0576] step: count(784), step_time 1419.34, mean_step_time 1493.36, it/s 0.67 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.310479466054 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:48:38 @multigpu.py:323] [p0574] step: count(775), step_time 1417.12, mean_step_time 1451.24, it/s 0.69 | |
| [0130 17:48:39 @multigpu.py:323] [p0115] step: count(761), step_time 1476.73, mean_step_time 1473.86, it/s 0.68 | |
| [0130 17:48:40 @multigpu.py:323] [p0576] step: count(785), step_time 1408.38, mean_step_time 1493.55, it/s 0.67 | |
| [0130 17:48:40 @multigpu.py:323] [p0574] step: count(776), step_time 1408.75, mean_step_time 1447.14, it/s 0.69 | |
| [0130 17:48:41 @multigpu.py:323] [p0115] step: count(762), step_time 1445.59, mean_step_time 1471.09, it/s 0.68 | |
| [0130 17:48:41 @multigpu.py:323] [p0576] step: count(786), step_time 1509.02, mean_step_time 1498.93, it/s 0.67 | |
| [0130 17:48:41 @multigpu.py:323] [p0574] step: count(777), step_time 1415.0, mean_step_time 1437.9, it/s 0.7 | |
| [0130 17:48:42 @multigpu.py:323] [p0115] step: count(763), step_time 1418.96, mean_step_time 1469.23, it/s 0.68 | |
| [0130 17:48:43 @multigpu.py:323] [p0576] step: count(787), step_time 1405.37, mean_step_time 1495.54, it/s 0.67 | |
| [0130 17:48:43 @multigpu.py:323] [p0574] step: count(778), step_time 1384.63, mean_step_time 1428.94, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.311744225224 , 2.4 ##### | |
| [u'online', 2.4] | |
| ##### Sending to neptune: active_workers : 0.311744298604 , 3 ##### | |
| receiving | |
| [0130 17:48:44 @multigpu.py:323] [p0115] step: count(764), step_time 1405.48, mean_step_time 1466.62, it/s 0.68 | |
| [0130 17:48:44 @multigpu.py:323] [p0576] step: count(788), step_time 1391.93, mean_step_time 1495.08, it/s 0.67 | |
| [0130 17:48:44 @multigpu.py:323] [p0574] step: count(779), step_time 1459.28, mean_step_time 1429.46, it/s 0.7 | |
| [0130 17:48:45 @multigpu.py:323] [p0115] step: count(765), step_time 1454.4, mean_step_time 1467.79, it/s 0.68 | |
| [0130 17:48:45 @multigpu.py:323] [p0576] step: count(789), step_time 1437.04, mean_step_time 1495.27, it/s 0.67 | |
| [0130 17:48:45 @multigpu.py:323] [p0574] step: count(780), step_time 1420.63, mean_step_time 1429.51, it/s 0.7 | |
| [0130 17:48:47 @multigpu.py:323] [p0115] step: count(766), step_time 1402.97, mean_step_time 1467.1, it/s 0.68 | |
| [0130 17:48:47 @multigpu.py:323] [p0576] step: count(790), step_time 1389.53, mean_step_time 1493.65, it/s 0.67 | |
| [0130 17:48:47 @multigpu.py:323] [p0574] step: count(781), step_time 1421.5, mean_step_time 1430.78, it/s 0.7 | |
| [0130 17:48:48 @multigpu.py:323] [p0115] step: count(767), step_time 1469.79, mean_step_time 1466.65, it/s 0.68 | |
| [0130 17:48:48 @multigpu.py:323] [p0576] step: count(791), step_time 1432.2, mean_step_time 1494.39, it/s 0.67 | |
| [0130 17:48:48 @multigpu.py:323] [p0574] step: count(782), step_time 1412.91, mean_step_time 1431.13, it/s 0.7 | |
| [0130 17:48:50 @multigpu.py:323] [p0115] step: count(768), step_time 1627.26, mean_step_time 1432.46, it/s 0.7 | |
| [0130 17:48:50 @multigpu.py:323] [p0576] step: count(792), step_time 1555.32, mean_step_time 1432.33, it/s 0.7 | |
| [0130 17:48:50 @multigpu.py:323] [p0574] step: count(783), step_time 1456.03, mean_step_time 1432.38, it/s 0.7 | |
| [0130 17:48:51 @multigpu.py:323] [p0574] step: count(784), step_time 1414.5, mean_step_time 1431.27, it/s 0.7 | |
| [0130 17:48:51 @multigpu.py:323] [p0115] step: count(769), step_time 1418.66, mean_step_time 1432.25, it/s 0.7 | |
| [0130 17:48:51 @multigpu.py:323] [p0576] step: count(793), step_time 1477.83, mean_step_time 1433.05, it/s 0.7 | |
| [0130 17:48:53 @multigpu.py:323] [p0574] step: count(785), step_time 1395.25, mean_step_time 1427.68, it/s 0.7 | |
| [0130 17:48:53 @multigpu.py:323] [p0576] step: count(794), step_time 1397.31, mean_step_time 1432.4, it/s 0.7 | |
| [0130 17:48:53 @multigpu.py:323] [p0115] step: count(770), step_time 1479.11, mean_step_time 1435.27, it/s 0.7 | |
| [0130 17:48:54 @multigpu.py:323] [p0574] step: count(786), step_time 1441.36, mean_step_time 1427.18, it/s 0.7 | |
| [0130 17:48:54 @multigpu.py:323] [p0576] step: count(795), step_time 1391.2, mean_step_time 1430.75, it/s 0.7 | |
| [0130 17:48:54 @multigpu.py:323] [p0115] step: count(771), step_time 1410.43, mean_step_time 1434.49, it/s 0.7 | |
| [0130 17:48:55 @multigpu.py:323] [p0576] step: count(796), step_time 1407.22, mean_step_time 1428.01, it/s 0.7 | |
| [0130 17:48:55 @multigpu.py:323] [p0115] step: count(772), step_time 1371.4, mean_step_time 1433.8, it/s 0.7 | |
| [0130 17:48:55 @multigpu.py:323] [p0574] step: count(787), step_time 1435.45, mean_step_time 1425.62, it/s 0.7 | |
| [0130 17:48:57 @multigpu.py:323] [p0576] step: count(797), step_time 1437.44, mean_step_time 1429.94, it/s 0.7 | |
| [0130 17:48:57 @multigpu.py:323] [p0115] step: count(773), step_time 1437.88, mean_step_time 1434.2, it/s 0.7 | |
| [0130 17:48:57 @multigpu.py:323] [p0574] step: count(788), step_time 1518.88, mean_step_time 1426.93, it/s 0.7 | |
| [0130 17:48:58 @multigpu.py:323] [p0576] step: count(798), step_time 1373.41, mean_step_time 1429.46, it/s 0.7 | |
| [0130 17:48:58 @multigpu.py:323] [p0115] step: count(774), step_time 1420.67, mean_step_time 1434.57, it/s 0.7 | |
| [0130 17:48:58 @multigpu.py:323] [p0574] step: count(789), step_time 1434.05, mean_step_time 1429.85, it/s 0.7 | |
| [0130 17:49:00 @multigpu.py:323] [p0576] step: count(799), step_time 1400.6, mean_step_time 1426.62, it/s 0.7 | |
| [0130 17:49:00 @multigpu.py:323] [p0115] step: count(775), step_time 1451.44, mean_step_time 1436.42, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.316423896088 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:49:00 @multigpu.py:323] [p0574] step: count(790), step_time 1374.9, mean_step_time 1427.62, it/s 0.7 | |
| [0130 17:49:01 @multigpu.py:323] [p0576] step: count(800), step_time 1407.41, mean_step_time 1427.18, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.316783569985 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.316783569985 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.316783569985 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.316784079936 , -0.00433429796249 ##### | |
| ##### Sending to neptune: policy_loss : 0.316784079936 , 0.223585680127 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.316784079936 , -2.28890061378 ##### | |
| ##### Sending to neptune: value_loss : 0.316784079936 , 1.51052486897 ##### | |
| ##### Sending to neptune: advantage : 0.316784079936 , -0.00105175154749 ##### | |
| ##### Sending to neptune: pred_reward : 0.316784079936 , 0.432729452848 ##### | |
| ##### Sending to neptune: max_logit : 0.316784079936 , 0.198514163494 ##### | |
| [u'loss', -0.004334297962486744, 0.22358568012714386, -2.28890061378479, 1.510524868965149, -0.0010517515474930406, 0.43272945284843445, 0.1985141634941101] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.316784537435 , 9006400.55 ##### | |
| ##### Sending to neptune: dp_per_s : 0.316784537435 , 88.4437987855 ##### | |
| [u'other', 9006400.55, 88.44379878546097] | |
| receiving | |
| [0130 17:49:01 @multigpu.py:323] [p0115] step: count(776), step_time 1429.33, mean_step_time 1436.61, it/s 0.7 | |
| [0130 17:49:01 @multigpu.py:323] [p0574] step: count(791), step_time 1407.01, mean_step_time 1428.7, it/s 0.7 | |
| [0130 17:49:02 @multigpu.py:323] [p0576] step: count(801), step_time 1415.41, mean_step_time 1427.02, it/s 0.7 | |
| [0130 17:49:03 @multigpu.py:323] [p0574] step: count(792), step_time 1369.7, mean_step_time 1425.81, it/s 0.7 | |
| [0130 17:49:03 @multigpu.py:323] [p0115] step: count(777), step_time 1390.76, mean_step_time 1437.44, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.317268399994 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:49:04 @multigpu.py:323] [p0576] step: count(802), step_time 1496.01, mean_step_time 1430.75, it/s 0.7 | |
| [0130 17:49:04 @multigpu.py:323] [p0574] step: count(793), step_time 1414.25, mean_step_time 1423.94, it/s 0.7 | |
| [0130 17:49:04 @multigpu.py:323] [p0115] step: count(778), step_time 1460.37, mean_step_time 1441.12, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.317744605806 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:49:05 @multigpu.py:323] [p0576] step: count(803), step_time 1389.31, mean_step_time 1427.06, it/s 0.7 | |
| [0130 17:49:05 @multigpu.py:323] [p0574] step: count(794), step_time 1399.28, mean_step_time 1420.02, it/s 0.7 | |
| [0130 17:49:05 @multigpu.py:323] [p0115] step: count(779), step_time 1458.67, mean_step_time 1441.69, it/s 0.69 | |
| [0130 17:49:07 @multigpu.py:323] [p0576] step: count(804), step_time 1407.45, mean_step_time 1426.47, it/s 0.7 | |
| [0130 17:49:07 @multigpu.py:323] [p0574] step: count(795), step_time 1521.25, mean_step_time 1425.23, it/s 0.7 | |
| [0130 17:49:07 @multigpu.py:323] [p0115] step: count(780), step_time 1413.54, mean_step_time 1442.17, it/s 0.69 | |
| [0130 17:49:08 @multigpu.py:323] [p0576] step: count(805), step_time 1423.43, mean_step_time 1427.22, it/s 0.7 | |
| [0130 17:49:08 @multigpu.py:323] [p0574] step: count(796), step_time 1395.91, mean_step_time 1424.59, it/s 0.7 | |
| [0130 17:49:08 @multigpu.py:323] [p0115] step: count(781), step_time 1415.45, mean_step_time 1439.11, it/s 0.69 | |
| [0130 17:49:10 @multigpu.py:323] [p0576] step: count(806), step_time 1426.55, mean_step_time 1423.1, it/s 0.7 | |
| [0130 17:49:10 @multigpu.py:323] [p0574] step: count(797), step_time 1403.43, mean_step_time 1424.01, it/s 0.7 | |
| [0130 17:49:10 @multigpu.py:323] [p0115] step: count(782), step_time 1402.94, mean_step_time 1436.98, it/s 0.7 | |
| [0130 17:49:11 @multigpu.py:323] [p0576] step: count(807), step_time 1405.85, mean_step_time 1423.12, it/s 0.7 | |
| [0130 17:49:11 @multigpu.py:323] [p0574] step: count(798), step_time 1417.62, mean_step_time 1425.66, it/s 0.7 | |
| [0130 17:49:11 @multigpu.py:323] [p0115] step: count(783), step_time 1446.93, mean_step_time 1438.37, it/s 0.7 | |
| [0130 17:49:12 @multigpu.py:323] [p0576] step: count(808), step_time 1450.79, mean_step_time 1426.07, it/s 0.7 | |
| [0130 17:49:12 @multigpu.py:323] [p0574] step: count(799), step_time 1408.44, mean_step_time 1423.12, it/s 0.7 | |
| [0130 17:49:13 @multigpu.py:323] [p0115] step: count(784), step_time 1434.27, mean_step_time 1439.81, it/s 0.69 | |
| [0130 17:49:14 @multigpu.py:323] [p0574] step: count(800), step_time 1419.21, mean_step_time 1423.05, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.32036268466 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.32036268466 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.32036268466 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.320363213552 , -0.00875152647495 ##### | |
| ##### Sending to neptune: policy_loss : 0.320363213552 , -0.275862365961 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.320363213552 , -2.28863835335 ##### | |
| ##### Sending to neptune: value_loss : 0.320363213552 , 1.44430506229 ##### | |
| ##### Sending to neptune: advantage : 0.320363213552 , 0.00120054150466 ##### | |
| ##### Sending to neptune: pred_reward : 0.320363213552 , 0.438394933939 ##### | |
| ##### Sending to neptune: max_logit : 0.320363213552 , 0.199205473065 ##### | |
| [u'loss', -0.008751526474952698, -0.27586236596107483, -2.2886383533477783, 1.4443050622940063, 0.0012005415046587586, 0.4383949339389801, 0.19920547306537628] | |
| [0130 17:49:14 @multigpu.py:323] [p0576] step: count(809), step_time 1471.75, mean_step_time 1427.8, it/s 0.7 | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.320363672442 , 9008079.44 ##### | |
| ##### Sending to neptune: dp_per_s : 0.320363672442 , 86.8237239399 ##### | |
| [u'other', 9008079.44, 86.82372393990129] | |
| receiving | |
| [0130 17:49:14 @multigpu.py:323] [p0115] step: count(785), step_time 1388.29, mean_step_time 1436.51, it/s 0.7 | |
| [0130 17:49:15 @multigpu.py:323] [p0576] step: count(810), step_time 1411.18, mean_step_time 1428.88, it/s 0.7 | |
| [0130 17:49:15 @multigpu.py:323] [p0574] step: count(801), step_time 1438.67, mean_step_time 1423.91, it/s 0.7 | |
| [0130 17:49:15 @multigpu.py:323] [p0115] step: count(786), step_time 1420.86, mean_step_time 1437.4, it/s 0.7 | |
| [0130 17:49:17 @multigpu.py:323] [p0576] step: count(811), step_time 1454.29, mean_step_time 1429.99, it/s 0.7 | |
| [0130 17:49:17 @multigpu.py:323] [p0115] step: count(787), step_time 1397.73, mean_step_time 1433.8, it/s 0.7 | |
| [0130 17:49:17 @multigpu.py:323] [p0574] step: count(802), step_time 1481.59, mean_step_time 1427.34, it/s 0.7 | |
| [0130 17:49:18 @multigpu.py:323] [p0574] step: count(803), step_time 1418.38, mean_step_time 1425.46, it/s 0.7 | |
| [0130 17:49:18 @multigpu.py:323] [p0576] step: count(812), step_time 1467.82, mean_step_time 1425.61, it/s 0.7 | |
| [0130 17:49:18 @multigpu.py:323] [p0115] step: count(788), step_time 1465.4, mean_step_time 1425.71, it/s 0.7 | |
| [0130 17:49:20 @multigpu.py:323] [p0574] step: count(804), step_time 1392.49, mean_step_time 1424.36, it/s 0.7 | |
| [0130 17:49:20 @multigpu.py:323] [p0115] step: count(789), step_time 1419.76, mean_step_time 1425.76, it/s 0.7 | |
| [0130 17:49:20 @multigpu.py:323] [p0576] step: count(813), step_time 1429.68, mean_step_time 1423.21, it/s 0.7 | |
| [0130 17:49:21 @multigpu.py:323] [p0576] step: count(814), step_time 1372.1, mean_step_time 1421.95, it/s 0.7 | |
| [0130 17:49:21 @multigpu.py:323] [p0574] step: count(805), step_time 1411.0, mean_step_time 1425.14, it/s 0.7 | |
| [0130 17:49:21 @multigpu.py:323] [p0115] step: count(790), step_time 1485.53, mean_step_time 1426.08, it/s 0.7 | |
| [0130 17:49:22 @multigpu.py:323] [p0576] step: count(815), step_time 1413.62, mean_step_time 1423.07, it/s 0.7 | |
| [0130 17:49:23 @multigpu.py:323] [p0574] step: count(806), step_time 1455.09, mean_step_time 1425.83, it/s 0.7 | |
| [0130 17:49:23 @multigpu.py:323] [p0115] step: count(791), step_time 1448.14, mean_step_time 1427.97, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.322972231905 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:49:24 @multigpu.py:323] [p0576] step: count(816), step_time 1431.39, mean_step_time 1424.28, it/s 0.7 | |
| [0130 17:49:24 @multigpu.py:323] [p0574] step: count(807), step_time 1466.36, mean_step_time 1427.37, it/s 0.7 | |
| [0130 17:49:24 @multigpu.py:323] [p0115] step: count(792), step_time 1426.63, mean_step_time 1430.73, it/s 0.7 | |
| [0130 17:49:25 @multigpu.py:323] [p0576] step: count(817), step_time 1429.91, mean_step_time 1423.9, it/s 0.7 | |
| [0130 17:49:25 @multigpu.py:323] [p0574] step: count(808), step_time 1424.38, mean_step_time 1422.65, it/s 0.7 | |
| [0130 17:49:26 @multigpu.py:323] [p0115] step: count(793), step_time 1473.78, mean_step_time 1432.52, it/s 0.7 | |
| [0130 17:49:27 @multigpu.py:323] [p0576] step: count(818), step_time 1418.34, mean_step_time 1426.14, it/s 0.7 | |
| [0130 17:49:27 @multigpu.py:323] [p0574] step: count(809), step_time 1412.76, mean_step_time 1421.59, it/s 0.7 | |
| [0130 17:49:27 @multigpu.py:323] [p0115] step: count(794), step_time 1480.85, mean_step_time 1435.53, it/s 0.7 | |
| [0130 17:49:28 @multigpu.py:323] [p0576] step: count(819), step_time 1430.83, mean_step_time 1427.66, it/s 0.7 | |
| [0130 17:49:28 @multigpu.py:323] [p0574] step: count(810), step_time 1391.33, mean_step_time 1422.41, it/s 0.7 | |
| [0130 17:49:28 @multigpu.py:323] [p0115] step: count(795), step_time 1477.14, mean_step_time 1436.82, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.324464672738 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:49:30 @multigpu.py:323] [p0576] step: count(820), step_time 1396.28, mean_step_time 1427.1, it/s 0.7 | |
| [0130 17:49:30 @multigpu.py:323] [p0574] step: count(811), step_time 1441.45, mean_step_time 1424.13, it/s 0.7 | |
| [0130 17:49:30 @multigpu.py:323] [p0115] step: count(796), step_time 1473.22, mean_step_time 1439.01, it/s 0.69 | |
| [0130 17:49:31 @multigpu.py:323] [p0576] step: count(821), step_time 1408.66, mean_step_time 1426.76, it/s 0.7 | |
| [0130 17:49:31 @multigpu.py:323] [p0574] step: count(812), step_time 1404.4, mean_step_time 1425.86, it/s 0.7 | |
| [0130 17:49:31 @multigpu.py:323] [p0115] step: count(797), step_time 1453.41, mean_step_time 1442.15, it/s 0.69 | |
| 80%|#######9 |799/1000[19:[0130 17:49:32 @multigpu.py:323] [p0576] step: count(822), step_time 1435.22, mean_step_time 1423.72, it/s 0.7 | |
| [0130 17:49:32 @multigpu.py:323] [p0574] step: count(813), step_time 1444.78, mean_step_time 1427.39, it/s 0.7 | |
| [0130 17:49:33 @multigpu.py:323] [p0115] step: count(798), step_time 1475.66, mean_step_time 1442.91, it/s 0.69 | |
| [0130 17:49:34 @multigpu.py:323] [p0576] step: count(823), step_time 1458.8, mean_step_time 1427.2, it/s 0.7 | |
| [0130 17:49:34 @multigpu.py:323] [p0574] step: count(814), step_time 1431.78, mean_step_time 1429.02, it/s 0.7 | |
| [0130 17:49:34 @multigpu.py:323] [p0115] step: count(799), step_time 1477.14, mean_step_time 1443.83, it/s 0.69 | |
| [0130 17:49:35 @multigpu.py:323] [p0576] step: count(824), step_time 1397.17, mean_step_time 1426.68, it/s 0.7 | |
| [0130 17:49:35 @multigpu.py:323] [p0574] step: count(815), step_time 1377.52, mean_step_time 1421.83, it/s 0.7 | |
| [0130 17:49:36 @multigpu.py:323] [p0115] step: count(800), step_time 1403.05, mean_step_time 1443.31, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.326430359946 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| sending to address tcp://p0112:61216 | |
| [0130 17:49:37 @multigpu.py:323] [p0576] step: count(825), step_time 1424.09, mean_step_time 1426.72, it/s 0.7 | |
| [0130 17:49:37 @multigpu.py:323] [p0574] step: count(816), step_time 1485.52, mean_step_time 1426.31, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| [0130 17:49:37 @multigpu.py:323] [p0115] step: count(801), step_time 1467.26, mean_step_time 1445.9, it/s 0.69 | |
| [0130 17:49:38 @multigpu.py:323] [p0576] step: count(826), step_time 1476.38, mean_step_time 1429.21, it/s 0.7 | |
| [0130 17:49:38 @multigpu.py:323] [p0574] step: count(817), step_time 1391.94, mean_step_time 1425.74, it/s 0.7 | |
| [0130 17:49:39 @multigpu.py:323] [p0115] step: count(802), step_time 1377.75, mean_step_time 1444.64, it/s 0.69 | |
| [0130 17:49:40 @multigpu.py:323] [p0576] step: count(827), step_time 1429.75, mean_step_time 1430.4, it/s 0.7 | |
| [0130 17:49:40 @multigpu.py:323] [p0574] step: count(818), step_time 1457.73, mean_step_time 1427.74, it/s 0.7 | |
| [0130 17:49:40 @multigpu.py:323] [p0115] step: count(803), step_time 1449.73, mean_step_time 1444.78, it/s 0.69 | |
| ##### Sending to neptune: max_delay : 0.326430359946 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.326430359946 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.327900048296 , -0.0139869665727 ##### | |
| ##### Sending to neptune: policy_loss : 0.327900048296 , -0.741453886032 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.327900048296 , -2.2882630825 ##### | |
| ##### Sending to neptune: value_loss : 0.327900048296 , 1.23938500881 ##### | |
| ##### Sending to neptune: advantage : 0.327900048296 , 0.003333655186 ##### | |
| ##### Sending to neptune: pred_reward : 0.327900048296 , 0.43941155076 ##### | |
| ##### Sending to neptune: max_logit : 0.327900048296 , 0.20052549243 ##### | |
| [u'loss', -0.013986966572701931, -0.7414538860321045, -2.2882630825042725, 1.2393850088119507, 0.003333655185997486, 0.43941155076026917, 0.20052549242973328] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.327902424998 , 9061856.05 ##### | |
| ##### Sending to neptune: dp_per_s : 0.327902424998 , 87.6794406125 ##### | |
| [u'other', 9061856.05, 87.6794406124564] | |
| receiving | |
| ##### Sending to neptune: online_score : 0.327903013031 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 17:49:41 @multigpu.py:323] [p0576] step: count(828), step_time 1446.81, mean_step_time 1430.2, it/s 0.7 | |
| [0130 17:49:41 @multigpu.py:323] [p0574] step: count(819), step_time 1411.12, mean_step_time 1427.88, it/s 0.7 | |
| [0130 17:49:41 @multigpu.py:323] [p0115] step: count(804), step_time 1437.33, mean_step_time 1444.93, it/s 0.69 | |
| [0130 17:49:42 @multigpu.py:323] [p0574] step: count(820), step_time 1378.63, mean_step_time 1425.85, it/s 0.7 | |
| [0130 17:49:43 @multigpu.py:323] [p0576] step: count(829), step_time 1459.28, mean_step_time 1429.58, it/s 0.7 | |
| [0130 17:49:43 @multigpu.py:323] [p0115] step: count(805), step_time 1406.53, mean_step_time 1445.84, it/s 0.69 | |
| [0130 17:49:44 @multigpu.py:323] [p0574] step: count(821), step_time 1396.77, mean_step_time 1423.75, it/s 0.7 | |
| [0130 17:49:44 @multigpu.py:323] [p0576] step: count(830), step_time 1395.62, mean_step_time 1428.8, it/s 0.7 | |
| [0130 17:49:44 @multigpu.py:323] [p0115] step: count(806), step_time 1423.04, mean_step_time 1445.95, it/s 0.69 | |
| [0130 17:49:45 @multigpu.py:323] [p0574] step: count(822), step_time 1447.25, mean_step_time 1422.03, it/s 0.7 | |
| [0130 17:49:45 @multigpu.py:323] [p0576] step: count(831), step_time 1470.52, mean_step_time 1429.61, it/s 0.7 | |
| [0130 17:49:46 @multigpu.py:323] [p0115] step: count(807), step_time 1488.77, mean_step_time 1450.51, it/s 0.69 | |
| [0130 17:49:47 @multigpu.py:323] [p0115] step: count(808), step_time 1465.87, mean_step_time 1450.53, it/s 0.69 | |
| [0130 17:49:47 @multigpu.py:323] [p0576] step: count(832), step_time 1888.2, mean_step_time 1450.63, it/s 0.69 | |
| [0130 17:49:47 @multigpu.py:323] [p0574] step: count(823), step_time 1990.13, mean_step_time 1450.62, it/s 0.69 | |
| [0130 17:49:49 @multigpu.py:323] [p0115] step: count(809), step_time 1385.91, mean_step_time 1448.84, it/s 0.69 | |
| [0130 17:49:49 @multigpu.py:323] [p0576] step: count(833), step_time 1420.86, mean_step_time 1450.19, it/s 0.69 | |
| [0130 17:49:49 @multigpu.py:323] [p0574] step: count(824), step_time 1462.15, mean_step_time 1454.1, it/s 0.69 | |
| [0130 17:49:50 @multigpu.py:323] [p0115] step: count(810), step_time 1418.66, mean_step_time 1445.49, it/s 0.69 | |
| [0130 17:49:50 @multigpu.py:323] [p0576] step: count(834), step_time 1409.05, mean_step_time 1452.04, it/s 0.69 | |
| [0130 17:49:50 @multigpu.py:323] [p0574] step: count(825), step_time 1411.84, mean_step_time 1454.15, it/s 0.69 | |
| [0130 17:49:51 @multigpu.py:323] [p0115] step: count(811), step_time 1412.64, mean_step_time 1443.72, it/s 0.69 | |
| [0130 17:49:52 @multigpu.py:323] [p0576] step: count(835), step_time 1400.96, mean_step_time 1451.41, it/s 0.69 | |
| [0130 17:49:52 @multigpu.py:323] [p0574] step: count(826), step_time 1382.74, mean_step_time 1450.53, it/s 0.69 | |
| [0130 17:49:53 @multigpu.py:323] [p0115] step: count(812), step_time 1422.38, mean_step_time 1443.5, it/s 0.69 | |
| [0130 17:49:53 @multigpu.py:323] [p0574] step: count(827), step_time 1434.28, mean_step_time 1448.92, it/s 0.69 | |
| [0130 17:49:53 @multigpu.py:323] [p0576] step: count(836), step_time 1509.3, mean_step_time 1455.3, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.33134215388 , 0.8 ##### | |
| [u'online', 0.8] | |
| ##### Sending to neptune: active_workers : 0.331342185802 , 3 ##### | |
| receiving | |
| [0130 17:49:54 @multigpu.py:323] [p0115] step: count(813), step_time 1456.38, mean_step_time 1442.63, it/s 0.69 | |
| [0130 17:49:54 @multigpu.py:323] [p0574] step: count(828), step_time 1431.32, mean_step_time 1449.27, it/s 0.69 | |
| [0130 17:49:54 @multigpu.py:323] [p0576] step: count(837), step_time 1418.01, mean_step_time 1454.71, it/s 0.69 | |
| [0130 17:49:56 @multigpu.py:323] [p0115] step: count(814), step_time 1406.15, mean_step_time 1438.9, it/s 0.69 | |
| [0130 17:49:56 @multigpu.py:323] [p0574] step: count(829), step_time 1429.44, mean_step_time 1450.11, it/s 0.69 | |
| [0130 17:49:56 @multigpu.py:323] [p0576] step: count(838), step_time 1479.45, mean_step_time 1457.76, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.332217746642 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:49:57 @multigpu.py:323] [p0115] step: count(815), step_time 1472.49, mean_step_time 1438.67, it/s 0.7 | |
| [0130 17:49:57 @multigpu.py:323] [p0574] step: count(830), step_time 1459.66, mean_step_time 1453.52, it/s 0.69 | |
| [0130 17:49:57 @multigpu.py:323] [p0576] step: count(839), step_time 1454.89, mean_step_time 1458.96, it/s 0.69 | |
| [0130 17:49:59 @multigpu.py:323] [p0115] step: count(816), step_time 1392.18, mean_step_time 1434.61, it/s 0.7 | |
| [0130 17:49:59 @multigpu.py:323] [p0574] step: count(831), step_time 1366.21, mean_step_time 1449.76, it/s 0.69 | |
| [0130 17:49:59 @multigpu.py:323] [p0576] step: count(840), step_time 1417.42, mean_step_time 1460.02, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.332876266903 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:50:00 @multigpu.py:323] [p0574] step: count(832), step_time 1406.13, mean_step_time 1449.85, it/s 0.69 | |
| [0130 17:50:00 @multigpu.py:323] [p0115] step: count(817), step_time 1455.63, mean_step_time 1434.73, it/s 0.7 | |
| [0130 17:50:00 @multigpu.py:323] [p0576] step: count(841), step_time 1411.56, mean_step_time 1460.17, it/s 0.68 | |
| 81%|######## |806/1000[19:[0130 17:50:02 @multigpu.py:323] [p0574] step: count(833), step_time 1442.25, mean_step_time 1449.72, it/s 0.69 | |
| [0130 17:50:02 @multigpu.py:323] [p0115] step: count(818), step_time 1425.83, mean_step_time 1432.23, it/s 0.7 | |
| [0130 17:50:02 @multigpu.py:323] [p0576] step: count(842), step_time 1418.03, mean_step_time 1459.31, it/s 0.69 | |
| [0130 17:50:03 @multigpu.py:323] [p0574] step: count(834), step_time 1427.28, mean_step_time 1449.5, it/s 0.69 | |
| [0130 17:50:03 @multigpu.py:323] [p0115] step: count(819), step_time 1511.54, mean_step_time 1433.95, it/s 0.7 | |
| [0130 17:50:03 @multigpu.py:323] [p0576] step: count(843), step_time 1472.28, mean_step_time 1459.98, it/s 0.68 | |
| [0130 17:50:04 @multigpu.py:323] [p0574] step: count(835), step_time 1462.29, mean_step_time 1453.73, it/s 0.69 | |
| [0130 17:50:04 @multigpu.py:323] [p0115] step: count(820), step_time 1412.58, mean_step_time 1434.43, it/s 0.7 | |
| [0130 17:50:04 @multigpu.py:323] [p0576] step: count(844), step_time 1389.04, mean_step_time 1459.57, it/s 0.69 | |
| [0130 17:50:06 @multigpu.py:323] [p0574] step: count(836), step_time 1450.97, mean_step_time 1452.01, it/s 0.69 | |
| [0130 17:50:06 @multigpu.py:323] [p0115] step: count(821), step_time 1431.96, mean_step_time 1432.67, it/s 0.7 | |
| [0130 17:50:06 @multigpu.py:323] [p0576] step: count(845), step_time 1426.12, mean_step_time 1459.68, it/s 0.69 | |
| [0130 17:50:07 @multigpu.py:323] [p0574] step: count(837), step_time 1483.71, mean_step_time 1456.59, it/s 0.69 | |
| [0130 17:50:07 @multigpu.py:323] [p0576] step: count(846), step_time 1441.06, mean_step_time 1457.91, it/s 0.69 | |
| [0130 17:50:07 @multigpu.py:323] [p0115] step: count(822), step_time 1465.82, mean_step_time 1437.07, it/s 0.7 | |
| [0130 17:50:09 @multigpu.py:323] [p0574] step: count(838), step_time 1427.26, mean_step_time 1455.07, it/s 0.69 | |
| [0130 17:50:09 @multigpu.py:323] [p0115] step: count(823), step_time 1444.49, mean_step_time 1436.81, it/s 0.7 | |
| [0130 17:50:09 @multigpu.py:323] [p0576] step: count(847), step_time 1471.8, mean_step_time 1460.01, it/s 0.68 | |
| [0130 17:50:10 @multigpu.py:323] [p0574] step: count(839), step_time 1411.78, mean_step_time 1455.1, it/s 0.69 | |
| [0130 17:50:10 @multigpu.py:323] [p0576] step: count(848), step_time 1398.02, mean_step_time 1457.57, it/s 0.69 | |
| [0130 17:50:10 @multigpu.py:323] [p0115] step: count(824), step_time 1426.87, mean_step_time 1436.29, it/s 0.7 | |
| [0130 17:50:12 @multigpu.py:323] [p0574] step: count(840), step_time 1421.72, mean_step_time 1457.26, it/s 0.69 | |
| [0130 17:50:12 @multigpu.py:323] [p0576] step: count(849), step_time 1433.66, mean_step_time 1456.29, it/s 0.69 | |
| [0130 17:50:12 @multigpu.py:323] [p0115] step: count(825), step_time 1437.46, mean_step_time 1437.83, it/s 0.7 | |
| [0130 17:50:13 @multigpu.py:323] [p0574] step: count(841), step_time 1459.84, mean_step_time 1460.41, it/s 0.68 | |
| [0130 17:50:13 @multigpu.py:323] [p0115] step: count(826), step_time 1400.72, mean_step_time 1436.72, it/s 0.7 | |
| [0130 17:50:13 @multigpu.py:323] [p0576] step: count(850), step_time 1425.38, mean_step_time 1457.78, it/s 0.69 | |
| [0130 17:50:14 @multigpu.py:323] [p0115] step: count(827), step_time 1374.98, mean_step_time 1431.03, it/s 0.7 | |
| [0130 17:50:14 @multigpu.py:323] [p0574] step: count(842), step_time 1388.76, mean_step_time 1457.49, it/s 0.69 | |
| [0130 17:50:15 @multigpu.py:323] [p0576] step: count(851), step_time 1446.51, mean_step_time 1456.58, it/s 0.69 | |
| [0130 17:50:16 @multigpu.py:323] [p0115] step: count(828), step_time 1508.09, mean_step_time 1433.14, it/s 0.7 | |
| [0130 17:50:16 @multigpu.py:323] [p0576] step: count(852), step_time 1418.63, mean_step_time 1433.1, it/s 0.7 | |
| [0130 17:50:16 @multigpu.py:323] [p0574] step: count(843), step_time 1501.38, mean_step_time 1433.05, it/s 0.7 | |
| [0130 17:50:17 @multigpu.py:323] [p0576] step: count(853), step_time 1382.78, mean_step_time 1431.2, it/s 0.7 | |
| [0130 17:50:17 @multigpu.py:323] [p0115] step: count(829), step_time 1417.21, mean_step_time 1434.7, it/s 0.7 | |
| [0130 17:50:17 @multigpu.py:323] [p0574] step: count(844), step_time 1492.5, mean_step_time 1434.57, it/s 0.7 | |
| [0130 17:50:19 @multigpu.py:323] [p0576] step: count(854), step_time 1431.19, mean_step_time 1432.3, it/s 0.7 | |
| [0130 17:50:19 @multigpu.py:323] [p0115] step: count(830), step_time 1462.53, mean_step_time 1436.9, it/s 0.7 | |
| [0130 17:50:19 @multigpu.py:323] [p0574] step: count(845), step_time 1454.74, mean_step_time 1436.71, it/s 0.7 | |
| [0130 17:50:20 @multigpu.py:323] [p0576] step: count(855), step_time 1445.91, mean_step_time 1434.55, it/s 0.7 | |
| [0130 17:50:20 @multigpu.py:323] [p0115] step: count(831), step_time 1409.05, mean_step_time 1436.72, it/s 0.7 | |
| [0130 17:50:20 @multigpu.py:323] [p0574] step: count(846), step_time 1417.08, mean_step_time 1438.43, it/s 0.7 | |
| [0130 17:50:22 @multigpu.py:323] [p0576] step: count(856), step_time 1438.58, mean_step_time 1431.02, it/s 0.7 | |
| [0130 17:50:22 @multigpu.py:323] [p0115] step: count(832), step_time 1501.15, mean_step_time 1440.66, it/s 0.69 | |
| [0130 17:50:22 @multigpu.py:323] [p0574] step: count(847), step_time 1478.96, mean_step_time 1440.66, it/s 0.69 | |
| [0130 17:50:23 @multigpu.py:323] [p0576] step: count(857), step_time 1419.0, mean_step_time 1431.07, it/s 0.7 | |
| [0130 17:50:23 @multigpu.py:323] [p0115] step: count(833), step_time 1378.58, mean_step_time 1436.77, it/s 0.7 | |
| [0130 17:50:23 @multigpu.py:323] [p0574] step: count(848), step_time 1425.69, mean_step_time 1440.38, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.339696066909 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.339726577732 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.339784356621 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:50:24 @multigpu.py:323] [p0576] step: count(858), step_time 1418.17, mean_step_time 1428.0, it/s 0.7 | |
| [0130 17:50:25 @multigpu.py:323] [p0115] step: count(834), step_time 1484.33, mean_step_time 1440.68, it/s 0.69 | |
| [0130 17:50:25 @multigpu.py:323] [p0574] step: count(849), step_time 1422.01, mean_step_time 1440.01, it/s 0.69 | |
| [0130 17:50:26 @multigpu.py:323] [p0576] step: count(859), step_time 1401.0, mean_step_time 1425.31, it/s 0.7 | |
| [0130 17:50:26 @multigpu.py:323] [p0115] step: count(835), step_time 1418.18, mean_step_time 1437.96, it/s 0.7 | |
| [0130 17:50:26 @multigpu.py:323] [p0574] step: count(850), step_time 1462.0, mean_step_time 1440.13, it/s 0.69 | |
| [0130 17:50:27 @multigpu.py:323] [p0576] step: count(860), step_time 1458.91, mean_step_time 1427.38, it/s 0.7 | |
| [0130 17:50:27 @multigpu.py:323] [p0115] step: count(836), step_time 1469.8, mean_step_time 1441.84, it/s 0.69 | |
| [0130 17:50:28 @multigpu.py:323] [p0574] step: count(851), step_time 1412.3, mean_step_time 1442.43, it/s 0.69 | |
| [0130 17:50:29 @multigpu.py:323] [p0576] step: count(861), step_time 1416.72, mean_step_time 1427.64, it/s 0.7 | |
| [0130 17:50:29 @multigpu.py:323] [p0115] step: count(837), step_time 1418.69, mean_step_time 1439.99, it/s 0.69 | |
| [0130 17:50:29 @multigpu.py:323] [p0574] step: count(852), step_time 1398.8, mean_step_time 1442.07, it/s 0.69 | |
| [0130 17:50:30 @multigpu.py:323] [p0576] step: count(862), step_time 1441.13, mean_step_time 1428.79, it/s 0.7 | |
| [0130 17:50:30 @multigpu.py:323] [p0115] step: count(838), step_time 1361.38, mean_step_time 1436.77, it/s 0.7 | |
| [0130 17:50:30 @multigpu.py:323] [p0574] step: count(853), step_time 1438.35, mean_step_time 1441.87, it/s 0.69 | |
| [0130 17:50:32 @multigpu.py:323] [p0576] step: count(863), step_time 1437.12, mean_step_time 1427.04, it/s 0.7 | |
| [0130 17:50:32 @multigpu.py:323] [p0115] step: count(839), step_time 1439.47, mean_step_time 1433.17, it/s 0.7 | |
| [0130 17:50:32 @multigpu.py:323] [p0574] step: count(854), step_time 1416.04, mean_step_time 1441.31, it/s 0.69 | |
| [0130 17:50:33 @multigpu.py:323] [p0576] step: count(864), step_time 1436.49, mean_step_time 1429.41, it/s 0.7 | |
| [0130 17:50:33 @multigpu.py:323] [p0115] step: count(840), step_time 1375.77, mean_step_time 1431.33, it/s 0.7 | |
| [0130 17:50:33 @multigpu.py:323] [p0574] step: count(855), step_time 1452.58, mean_step_time 1440.82, it/s 0.69 | |
| [0130 17:50:35 @multigpu.py:323] [p0115] step: count(841), step_time 1487.83, mean_step_time 1434.12, it/s 0.7 | |
| [0130 17:50:35 @multigpu.py:323] [p0576] step: count(865), step_time 1508.24, mean_step_time 1433.52, it/s 0.7 | |
| [0130 17:50:35 @multigpu.py:323] [p0574] step: count(856), step_time 1409.41, mean_step_time 1438.75, it/s 0.7 | |
| [0130 17:50:36 @multigpu.py:323] [p0576] step: count(866), step_time 1412.2, mean_step_time 1432.07, it/s 0.7 | |
| [0130 17:50:36 @multigpu.py:323] [p0115] step: count(842), step_time 1436.41, mean_step_time 1432.65, it/s 0.7 | |
| [0130 17:50:36 @multigpu.py:323] [p0574] step: count(857), step_time 1446.84, mean_step_time 1436.9, it/s 0.7 | |
| [0130 17:50:37 @multigpu.py:323] [p0576] step: count(867), step_time 1435.4, mean_step_time 1430.25, it/s 0.7 | |
| [0130 17:50:37 @multigpu.py:323] [p0574] step: count(858), step_time 1403.72, mean_step_time 1435.72, it/s 0.7 | |
| [0130 17:50:38 @multigpu.py:323] [p0115] step: count(843), step_time 1490.01, mean_step_time 1434.93, it/s 0.7 | |
| [0130 17:50:39 @multigpu.py:323] [p0115] step: count(844), step_time 1376.99, mean_step_time 1432.43, it/s 0.7 | |
| [0130 17:50:39 @multigpu.py:323] [p0576] step: count(868), step_time 1444.38, mean_step_time 1432.57, it/s 0.7 | |
| [0130 17:50:39 @multigpu.py:323] [p0574] step: count(859), step_time 1435.49, mean_step_time 1436.91, it/s 0.7 | |
| [0130 17:50:40 @multigpu.py:323] [p0115] step: count(845), step_time 1388.07, mean_step_time 1429.96, it/s 0.7 | |
| [0130 17:50:40 @multigpu.py:323] [p0576] step: count(869), step_time 1400.91, mean_step_time 1430.93, it/s 0.7 | |
| [0130 17:50:40 @multigpu.py:323] [p0574] step: count(860), step_time 1422.81, mean_step_time 1436.96, it/s 0.7 | |
| [0130 17:50:42 @multigpu.py:323] [p0115] step: count(846), step_time 1405.59, mean_step_time 1430.21, it/s 0.7 | |
| [0130 17:50:42 @multigpu.py:323] [p0576] step: count(870), step_time 1414.07, mean_step_time 1430.37, it/s 0.7 | |
| [0130 17:50:42 @multigpu.py:323] [p0574] step: count(861), step_time 1389.45, mean_step_time 1433.45, it/s 0.7 | |
| [0130 17:50:43 @multigpu.py:323] [p0576] step: count(871), step_time 1439.59, mean_step_time 1430.02, it/s 0.7 | |
| [0130 17:50:43 @multigpu.py:323] [p0115] step: count(847), step_time 1472.43, mean_step_time 1435.08, it/s 0.7 | |
| [0130 17:50:43 @multigpu.py:323] [p0574] step: count(862), step_time 1486.02, mean_step_time 1438.31, it/s 0.7 | |
| [0130 17:50:45 @multigpu.py:323] [p0574] step: count(863), step_time 1380.9, mean_step_time 1432.28, it/s 0.7 | |
| [0130 17:50:45 @multigpu.py:323] [p0115] step: count(848), step_time 1453.38, mean_step_time 1432.34, it/s 0.7 | |
| [0130 17:50:45 @multigpu.py:323] [p0576] step: count(872), step_time 1464.71, mean_step_time 1432.33, it/s 0.7 | |
| [0130 17:50:46 @multigpu.py:323] [p0115] step: count(849), step_time 1356.6, mean_step_time 1429.31, it/s 0.7 | |
| [0130 17:50:46 @multigpu.py:323] [p0576] step: count(873), step_time 1428.41, mean_step_time 1434.61, it/s 0.7 | |
| [0130 17:50:46 @multigpu.py:323] [p0574] step: count(864), step_time 1449.39, mean_step_time 1430.13, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.346096348829 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:50:47 @multigpu.py:323] [p0115] step: count(850), step_time 1441.86, mean_step_time 1428.28, it/s 0.7 | |
| [0130 17:50:47 @multigpu.py:323] [p0576] step: count(874), step_time 1386.22, mean_step_time 1432.36, it/s 0.7 | |
| [0130 17:50:48 @multigpu.py:323] [p0574] step: count(865), step_time 1461.56, mean_step_time 1430.47, it/s 0.7 | |
| [0130 17:50:49 @multigpu.py:323] [p0115] step: count(851), step_time 1407.82, mean_step_time 1428.22, it/s 0.7 | |
| [0130 17:50:49 @multigpu.py:323] [p0576] step: count(875), step_time 1401.26, mean_step_time 1430.13, it/s 0.7 | |
| [0130 17:50:49 @multigpu.py:323] [p0574] step: count(866), step_time 1420.17, mean_step_time 1430.62, it/s 0.7 | |
| [0130 17:50:50 @multigpu.py:323] [p0115] step: count(852), step_time 1406.47, mean_step_time 1423.48, it/s 0.7 | |
| [0130 17:50:50 @multigpu.py:323] [p0576] step: count(876), step_time 1468.13, mean_step_time 1431.6, it/s 0.7 | |
| [0130 17:50:50 @multigpu.py:323] [p0574] step: count(867), step_time 1370.78, mean_step_time 1425.22, it/s 0.7 | |
| [0130 17:50:52 @multigpu.py:323] [p0115] step: count(853), step_time 1394.32, mean_step_time 1424.27, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.347506924139 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:50:52 @multigpu.py:323] [p0576] step: count(877), step_time 1458.1, mean_step_time 1433.56, it/s 0.7 | |
| [0130 17:50:52 @multigpu.py:323] [p0574] step: count(868), step_time 1443.72, mean_step_time 1426.12, it/s 0.7 | |
| [0130 17:50:53 @multigpu.py:323] [p0115] step: count(854), step_time 1444.81, mean_step_time 1422.29, it/s 0.7 | |
| [0130 17:50:53 @multigpu.py:323] [p0576] step: count(878), step_time 1429.86, mean_step_time 1434.14, it/s 0.7 | |
| [0130 17:50:53 @multigpu.py:323] [p0574] step: count(869), step_time 1440.8, mean_step_time 1427.06, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.348158127997 , 1.8 ##### | |
| [u'online', 1.8] | |
| ##### Sending to neptune: active_workers : 0.348158168859 , 3 ##### | |
| receiving | |
| [0130 17:50:55 @multigpu.py:323] [p0115] step: count(855), step_time 1461.99, mean_step_time 1424.48, it/s 0.7 | |
| [0130 17:50:55 @multigpu.py:323] [p0576] step: count(879), step_time 1429.79, mean_step_time 1435.58, it/s 0.7 | |
| [0130 17:50:55 @multigpu.py:323] [p0574] step: count(870), step_time 1415.5, mean_step_time 1424.73, it/s 0.7 | |
| [0130 17:50:56 @multigpu.py:323] [p0115] step: count(856), step_time 1440.17, mean_step_time 1423.0, it/s 0.7 | |
| [0130 17:50:56 @multigpu.py:323] [p0574] step: count(871), step_time 1401.46, mean_step_time 1424.19, it/s 0.7 | |
| [0130 17:50:56 @multigpu.py:323] [p0576] step: count(880), step_time 1490.08, mean_step_time 1437.14, it/s 0.7 | |
| [0130 17:50:57 @multigpu.py:323] [p0115] step: count(857), step_time 1438.35, mean_step_time 1423.98, it/s 0.7 | |
| [0130 17:50:57 @multigpu.py:323] [p0574] step: count(872), step_time 1437.87, mean_step_time 1426.14, it/s 0.7 | |
| [0130 17:50:58 @multigpu.py:323] [p0576] step: count(881), step_time 1417.36, mean_step_time 1437.17, it/s 0.7 | |
| [0130 17:50:59 @multigpu.py:323] [p0115] step: count(858), step_time 1433.35, mean_step_time 1427.58, it/s 0.7 | |
| [0130 17:50:59 @multigpu.py:323] [p0574] step: count(873), step_time 1393.58, mean_step_time 1423.9, it/s 0.7 | |
| [0130 17:50:59 @multigpu.py:323] [p0576] step: count(882), step_time 1393.8, mean_step_time 1434.81, it/s 0.7 | |
| [0130 17:51:00 @multigpu.py:323] [p0115] step: count(859), step_time 1424.23, mean_step_time 1426.82, it/s 0.7 | |
| [0130 17:51:00 @multigpu.py:323] [p0576] step: count(883), step_time 1381.47, mean_step_time 1432.02, it/s 0.7 | |
| [0130 17:51:00 @multigpu.py:323] [p0574] step: count(874), step_time 1502.83, mean_step_time 1428.24, it/s 0.7 | |
| [0130 17:51:02 @multigpu.py:323] [p0115] step: count(860), step_time 1398.0, mean_step_time 1427.93, it/s 0.7 | |
| [0130 17:51:02 @multigpu.py:323] [p0576] step: count(884), step_time 1422.88, mean_step_time 1431.34, it/s 0.7 | |
| [0130 17:51:02 @multigpu.py:323] [p0574] step: count(875), step_time 1428.62, mean_step_time 1427.05, it/s 0.7 | |
| [0130 17:51:03 @multigpu.py:323] [p0115] step: count(861), step_time 1420.77, mean_step_time 1424.58, it/s 0.7 | |
| [0130 17:51:03 @multigpu.py:323] [p0576] step: count(885), step_time 1435.11, mean_step_time 1427.69, it/s 0.7 | |
| [0130 17:51:03 @multigpu.py:323] [p0574] step: count(876), step_time 1461.93, mean_step_time 1429.67, it/s 0.7 | |
| [0130 17:51:05 @multigpu.py:323] [p0115] step: count(862), step_time 1455.64, mean_step_time 1425.54, it/s 0.7 | |
| [0130 17:51:05 @multigpu.py:323] [p0576] step: count(886), step_time 1386.82, mean_step_time 1426.42, it/s 0.7 | |
| [0130 17:51:05 @multigpu.py:323] [p0574] step: count(877), step_time 1405.52, mean_step_time 1427.61, it/s 0.7 | |
| [0130 17:51:06 @multigpu.py:323] [p0115] step: count(863), step_time 1414.31, mean_step_time 1421.76, it/s 0.7 | |
| [0130 17:51:06 @multigpu.py:323] [p0576] step: count(887), step_time 1438.14, mean_step_time 1426.56, it/s 0.7 | |
| [0130 17:51:06 @multigpu.py:323] [p0574] step: count(878), step_time 1390.07, mean_step_time 1426.92, it/s 0.7 | |
| [0130 17:51:07 @multigpu.py:323] [p0115] step: count(864), step_time 1426.81, mean_step_time 1424.25, it/s 0.7 | |
| [0130 17:51:07 @multigpu.py:323] [p0576] step: count(888), step_time 1423.35, mean_step_time 1425.5, it/s 0.7 | |
| [0130 17:51:08 @multigpu.py:323] [p0574] step: count(879), step_time 1486.03, mean_step_time 1429.45, it/s 0.7 | |
| [0130 17:51:09 @multigpu.py:323] [p0115] step: count(865), step_time 1462.49, mean_step_time 1427.97, it/s 0.7 | |
| [0130 17:51:09 @multigpu.py:323] [p0576] step: count(889), step_time 1489.7, mean_step_time 1429.94, it/s 0.7 | |
| [0130 17:51:09 @multigpu.py:323] [p0574] step: count(880), step_time 1528.76, mean_step_time 1434.75, it/s 0.7 | |
| [0130 17:51:10 @multigpu.py:323] [p0115] step: count(866), step_time 1439.06, mean_step_time 1429.64, it/s 0.7 | |
| [0130 17:51:10 @multigpu.py:323] [p0576] step: count(890), step_time 1421.23, mean_step_time 1430.3, it/s 0.7 | |
| [0130 17:51:11 @multigpu.py:323] [p0574] step: count(881), step_time 1589.16, mean_step_time 1444.73, it/s 0.69 | |
| [0130 17:51:12 @multigpu.py:323] [p0115] step: count(867), step_time 1414.91, mean_step_time 1426.77, it/s 0.7 | |
| [0130 17:51:12 @multigpu.py:323] [p0576] step: count(891), step_time 1479.83, mean_step_time 1432.31, it/s 0.7 | |
| [0130 17:51:12 @multigpu.py:323] [p0574] step: count(882), step_time 1398.45, mean_step_time 1440.35, it/s 0.69 | |
| [0130 17:51:14 @multigpu.py:323] [p0574] step: count(883), step_time 1463.84, mean_step_time 1444.5, it/s 0.69 | |
| [0130 17:51:14 @multigpu.py:323] [p0115] step: count(868), step_time 1808.96, mean_step_time 1444.55, it/s 0.69 | |
| [0130 17:51:14 @multigpu.py:323] [p0576] step: count(892), step_time 1699.42, mean_step_time 1444.05, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.353776509696 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:51:15 @multigpu.py:323] [p0115] step: count(869), step_time 1415.04, mean_step_time 1447.47, it/s 0.69 | |
| [0130 17:51:15 @multigpu.py:323] [p0576] step: count(893), step_time 1428.94, mean_step_time 1444.08, it/s 0.69 | |
| [0130 17:51:15 @multigpu.py:323] [p0574] step: count(884), step_time 1437.96, mean_step_time 1443.93, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.354189967182 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:51:16 @multigpu.py:323] [p0576] step: count(894), step_time 1421.53, mean_step_time 1445.84, it/s 0.69 | |
| [0130 17:51:16 @multigpu.py:323] [p0115] step: count(870), step_time 1473.59, mean_step_time 1449.05, it/s 0.69 | |
| [0130 17:51:16 @multigpu.py:323] [p0574] step: count(885), step_time 1468.89, mean_step_time 1444.3, it/s 0.69 | |
| [0130 17:51:18 @multigpu.py:323] [p0576] step: count(895), step_time 1458.08, mean_step_time 1448.68, it/s 0.69 | |
| [0130 17:51:18 @multigpu.py:323] [p0574] step: count(886), step_time 1462.1, mean_step_time 1446.39, it/s 0.69 | |
| [0130 17:51:18 @multigpu.py:323] [p0115] step: count(871), step_time 1488.1, mean_step_time 1453.07, it/s 0.69 | |
| [0130 17:51:19 @multigpu.py:323] [p0576] step: count(896), step_time 1421.15, mean_step_time 1446.33, it/s 0.69 | |
| [0130 17:51:19 @multigpu.py:323] [p0574] step: count(887), step_time 1424.94, mean_step_time 1449.1, it/s 0.69 | |
| [0130 17:51:19 @multigpu.py:323] [p0115] step: count(872), step_time 1470.23, mean_step_time 1456.26, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.35528920273 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:51:21 @multigpu.py:323] [p0576] step: count(897), step_time 1400.25, mean_step_time 1443.44, it/s 0.69 | |
| [0130 17:51:21 @multigpu.py:323] [p0574] step: count(888), step_time 1429.18, mean_step_time 1448.37, it/s 0.69 | |
| [0130 17:51:21 @multigpu.py:323] [p0115] step: count(873), step_time 1425.93, mean_step_time 1457.84, it/s 0.69 | |
| [0130 17:51:22 @multigpu.py:323] [p0576] step: count(898), step_time 1511.99, mean_step_time 1447.55, it/s 0.69 | |
| [0130 17:51:22 @multigpu.py:323] [p0574] step: count(889), step_time 1438.55, mean_step_time 1448.26, it/s 0.69 | |
| [0130 17:51:22 @multigpu.py:323] [p0115] step: count(874), step_time 1399.34, mean_step_time 1455.56, it/s 0.69 | |
| [0130 17:51:24 @multigpu.py:323] [p0115] step: count(875), step_time 1390.43, mean_step_time 1451.99, it/s 0.69 | |
| [0130 17:51:24 @multigpu.py:323] [p0574] step: count(890), step_time 1411.99, mean_step_time 1448.09, it/s 0.69 | |
| [0130 17:51:24 @multigpu.py:323] [p0576] step: count(899), step_time 1448.49, mean_step_time 1448.48, it/s 0.69 | |
| [0130 17:51:25 @multigpu.py:323] [p0576] step: count(900), step_time 1443.06, mean_step_time 1446.13, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| [0130 17:51:25 @multigpu.py:323] [p0115] step: count(876), step_time 1472.49, mean_step_time 1453.6, it/s 0.69 | |
| ##### Sending to neptune: mean_delay : 0.356788568298 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.356788568298 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.356788568298 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.35678909911 , -0.0122179845348 ##### | |
| ##### Sending to neptune: policy_loss : 0.35678909911 , -0.376462846994 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.35678909911 , -2.28860378265 ##### | |
| ##### Sending to neptune: value_loss : 0.35678909911 , 1.10116481781 ##### | |
| ##### Sending to neptune: advantage : 0.35678909911 , 0.00160939083435 ##### | |
| ##### Sending to neptune: pred_reward : 0.35678909911 , 0.382754832506 ##### | |
| ##### Sending to neptune: max_logit : 0.35678909911 , 0.197250828147 ##### | |
| [u'loss', -0.012217984534800053, -0.3764628469944, -2.2886037826538086, 1.1011648178100586, 0.0016093908343464136, 0.3827548325061798, 0.1972508281469345] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.356789570782 , 9117830.05 ##### | |
| ##### Sending to neptune: dp_per_s : 0.356789570782 , 89.1289708126 ##### | |
| [u'other', 9117830.05, 89.128970812551] | |
| receiving | |
| [0130 17:51:25 @multigpu.py:323] [p0574] step: count(891), step_time 1477.12, mean_step_time 1451.87, it/s 0.69 | |
| [0130 17:51:26 @multigpu.py:323] [p0576] step: count(901), step_time 1402.28, mean_step_time 1445.38, it/s 0.69 | |
| [0130 17:51:26 @multigpu.py:323] [p0574] step: count(892), step_time 1390.05, mean_step_time 1449.48, it/s 0.69 | |
| [0130 17:51:26 @multigpu.py:323] [p0115] step: count(877), step_time 1425.55, mean_step_time 1452.96, it/s 0.69 | |
| [0130 17:51:28 @multigpu.py:323] [p0576] step: count(902), step_time 1435.78, mean_step_time 1447.48, it/s 0.69 | |
| [0130 17:51:28 @multigpu.py:323] [p0115] step: count(878), step_time 1411.88, mean_step_time 1451.89, it/s 0.69 | |
| [0130 17:51:28 @multigpu.py:323] [p0574] step: count(893), step_time 1516.56, mean_step_time 1455.63, it/s 0.69 | |
| [0130 17:51:29 @multigpu.py:323] [p0576] step: count(903), step_time 1406.34, mean_step_time 1448.72, it/s 0.69 | |
| [0130 17:51:29 @multigpu.py:323] [p0115] step: count(879), step_time 1462.48, mean_step_time 1453.8, it/s 0.69 | |
| [0130 17:51:29 @multigpu.py:323] [p0574] step: count(894), step_time 1401.91, mean_step_time 1450.58, it/s 0.69 | |
| [0130 17:51:31 @multigpu.py:323] [p0576] step: count(904), step_time 1378.58, mean_step_time 1446.5, it/s 0.69 | |
| [0130 17:51:31 @multigpu.py:323] [p0115] step: count(880), step_time 1418.22, mean_step_time 1454.81, it/s 0.69 | |
| [0130 17:51:31 @multigpu.py:323] [p0574] step: count(895), step_time 1464.02, mean_step_time 1452.35, it/s 0.69 | |
| [0130 17:51:32 @multigpu.py:323] [p0576] step: count(905), step_time 1438.57, mean_step_time 1446.68, it/s 0.69 | |
| [0130 17:51:32 @multigpu.py:323] [p0115] step: count(881), step_time 1422.05, mean_step_time 1454.88, it/s 0.69 | |
| [0130 17:51:32 @multigpu.py:323] [p0574] step: count(896), step_time 1423.22, mean_step_time 1450.42, it/s 0.69 | |
| [0130 17:51:34 @multigpu.py:323] [p0576] step: count(906), step_time 1412.81, mean_step_time 1447.98, it/s 0.69 | |
| [0130 17:51:34 @multigpu.py:323] [p0115] step: count(882), step_time 1423.91, mean_step_time 1453.29, it/s 0.69 | |
| [0130 17:51:34 @multigpu.py:323] [p0574] step: count(897), step_time 1433.91, mean_step_time 1451.84, it/s 0.69 | |
| [0130 17:51:35 @multigpu.py:323] [p0576] step: count(907), step_time 1400.85, mean_step_time 1446.11, it/s 0.69 | |
| [0130 17:51:35 @multigpu.py:323] [p0115] step: count(883), step_time 1509.1, mean_step_time 1458.03, it/s 0.69 | |
| [0130 17:51:35 @multigpu.py:323] [p0574] step: count(898), step_time 1437.74, mean_step_time 1454.22, it/s 0.69 | |
| [0130 17:51:36 @multigpu.py:323] [p0576] step: count(908), step_time 1459.62, mean_step_time 1447.92, it/s 0.69 | |
| [0130 17:51:37 @multigpu.py:323] [p0574] step: count(899), step_time 1388.74, mean_step_time 1449.35, it/s 0.69 | |
| [0130 17:51:37 @multigpu.py:323] [p0115] step: count(884), step_time 1415.69, mean_step_time 1457.47, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.360088469982 , 0.7 ##### | |
| [u'online', 0.7] | |
| receiving | |
| [0130 17:51:38 @multigpu.py:323] [p0576] step: count(909), step_time 1351.63, mean_step_time 1441.02, it/s 0.69 | |
| [0130 17:51:38 @multigpu.py:323] [p0574] step: count(900), step_time 1442.32, mean_step_time 1445.03, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.360377523568 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.360377523568 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.360377523568 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.360378028022 , -0.00242157001048 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.360378028022 , 0.664090633392 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.360378028022 , -2.28888463974 ##### | |
| ##### Sending to neptune: value_loss : 0.360378028022 , 1.3148329258 ##### | |
| ##### Sending to neptune: advantage : 0.360378028022 , -0.00289527932182 ##### | |
| ##### Sending to neptune: pred_reward : 0.360378028022 , 0.36841109395 ##### | |
| ##### Sending to neptune: max_logit : 0.360378028022 , 0.195855751634 ##### | |
| [u'loss', -0.002421570010483265, 0.664090633392334, -2.2888846397399902, 1.3148329257965088, -0.002895279321819544, 0.3684110939502716, 0.1958557516336441] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.360378457175 , 9146761.02 ##### | |
| ##### Sending to neptune: dp_per_s : 0.360378457175 , 88.9845936388 ##### | |
| [u'other', 9146761.02, 88.98459363878979] | |
| receiving | |
| [0130 17:51:38 @multigpu.py:323] [p0115] step: count(885), step_time 1454.51, mean_step_time 1457.07, it/s 0.69 | |
| [0130 17:51:39 @multigpu.py:323] [p0576] step: count(910), step_time 1534.17, mean_step_time 1446.67, it/s 0.69 | |
| [0130 17:51:39 @multigpu.py:323] [p0574] step: count(901), step_time 1410.25, mean_step_time 1436.09, it/s 0.7 | |
| [0130 17:51:39 @multigpu.py:323] [p0115] step: count(886), step_time 1448.48, mean_step_time 1457.54, it/s 0.69 | |
| [0130 17:51:41 @multigpu.py:323] [p0576] step: count(911), step_time 1396.51, mean_step_time 1442.5, it/s 0.69 | |
| [0130 17:51:41 @multigpu.py:323] [p0574] step: count(902), step_time 1444.8, mean_step_time 1438.4, it/s 0.7 | |
| [0130 17:51:41 @multigpu.py:323] [p0115] step: count(887), step_time 1455.82, mean_step_time 1459.59, it/s 0.69 | |
| [0130 17:51:42 @multigpu.py:323] [p0115] step: count(888), step_time 1403.94, mean_step_time 1439.34, it/s 0.69 | |
| [0130 17:51:42 @multigpu.py:323] [p0574] step: count(903), step_time 1482.73, mean_step_time 1439.35, it/s 0.69 | |
| [0130 17:51:42 @multigpu.py:323] [p0576] step: count(912), step_time 1636.16, mean_step_time 1439.34, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.361710860266 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 17:51:44 @multigpu.py:323] [p0574] step: count(904), step_time 1389.31, mean_step_time 1436.92, it/s 0.7 | |
| [0130 17:51:44 @multigpu.py:323] [p0576] step: count(913), step_time 1450.55, mean_step_time 1440.42, it/s 0.69 | |
| [0130 17:51:44 @multigpu.py:323] [p0115] step: count(889), step_time 1481.5, mean_step_time 1442.66, it/s 0.69 | |
| [0130 17:51:45 @multigpu.py:323] [p0574] step: count(905), step_time 1451.54, mean_step_time 1436.05, it/s 0.7 | |
| [0130 17:51:45 @multigpu.py:323] [p0576] step: count(914), step_time 1420.29, mean_step_time 1440.36, it/s 0.69 | |
| [0130 17:51:45 @multigpu.py:323] [p0115] step: count(890), step_time 1405.77, mean_step_time 1439.27, it/s 0.69 | |
| [0130 17:51:47 @multigpu.py:323] [p0574] step: count(906), step_time 1417.06, mean_step_time 1433.8, it/s 0.7 | |
| [0130 17:51:47 @multigpu.py:323] [p0115] step: count(891), step_time 1381.88, mean_step_time 1433.96, it/s 0.7 | |
| [0130 17:51:47 @multigpu.py:323] [p0576] step: count(915), step_time 1453.4, mean_step_time 1440.12, it/s 0.69 | |
| [0130 17:51:48 @multigpu.py:323] [p0115] step: count(892), step_time 1433.03, mean_step_time 1432.1, it/s 0.7 | |
| [0130 17:51:48 @multigpu.py:323] [p0574] step: count(907), step_time 1453.54, mean_step_time 1435.23, it/s 0.7 | |
| [0130 17:51:48 @multigpu.py:323] [p0576] step: count(916), step_time 1415.91, mean_step_time 1439.86, it/s 0.69 | |
| [0130 17:51:49 @multigpu.py:323] [p0574] step: count(908), step_time 1397.95, mean_step_time 1433.67, it/s 0.7 | |
| [0130 17:51:49 @multigpu.py:323] [p0115] step: count(893), step_time 1417.88, mean_step_time 1431.7, it/s 0.7 | |
| [0130 17:51:49 @multigpu.py:323] [p0576] step: count(917), step_time 1407.75, mean_step_time 1440.24, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.363701251083 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:51:51 @multigpu.py:323] [p0574] step: count(909), step_time 1447.39, mean_step_time 1434.11, it/s 0.7 | |
| [0130 17:51:51 @multigpu.py:323] [p0115] step: count(894), step_time 1468.22, mean_step_time 1435.14, it/s 0.7 | |
| [0130 17:51:51 @multigpu.py:323] [p0576] step: count(918), step_time 1449.44, mean_step_time 1437.11, it/s 0.7 | |
| [0130 17:51:52 @multigpu.py:323] [p0574] step: count(910), step_time 1378.61, mean_step_time 1432.44, it/s 0.7 | |
| [0130 17:51:52 @multigpu.py:323] [p0576] step: count(919), step_time 1437.36, mean_step_time 1436.55, it/s 0.7 | |
| [0130 17:51:52 @multigpu.py:323] [p0115] step: count(895), step_time 1470.5, mean_step_time 1439.14, it/s 0.69 | |
| [0130 17:51:54 @multigpu.py:323] [p0574] step: count(911), step_time 1399.76, mean_step_time 1428.57, it/s 0.7 | |
| [0130 17:51:54 @multigpu.py:323] [p0576] step: count(920), step_time 1376.39, mean_step_time 1433.22, it/s 0.7 | |
| [0130 17:51:54 @multigpu.py:323] [p0115] step: count(896), step_time 1374.8, mean_step_time 1434.26, it/s 0.7 | |
| [0130 17:51:55 @multigpu.py:323] [p0574] step: count(912), step_time 1422.39, mean_step_time 1430.19, it/s 0.7 | |
| [0130 17:51:55 @multigpu.py:323] [p0115] step: count(897), step_time 1424.27, mean_step_time 1434.2, it/s 0.7 | |
| [0130 17:51:55 @multigpu.py:323] [p0576] step: count(921), step_time 1496.51, mean_step_time 1437.93, it/s 0.7 | |
| [0130 17:51:56 @multigpu.py:323] [p0574] step: count(913), step_time 1422.53, mean_step_time 1425.49, it/s 0.7 | |
| [0130 17:51:57 @multigpu.py:323] [p0115] step: count(898), step_time 1408.28, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:51:57 @multigpu.py:323] [p0576] step: count(922), step_time 1424.22, mean_step_time 1437.35, it/s 0.7 | |
| [0130 17:51:58 @multigpu.py:323] [p0574] step: count(914), step_time 1438.2, mean_step_time 1427.3, it/s 0.7 | |
| [0130 17:51:58 @multigpu.py:323] [p0115] step: count(899), step_time 1420.96, mean_step_time 1431.94, it/s 0.7 | |
| [0130 17:51:58 @multigpu.py:323] [p0576] step: count(923), step_time 1398.27, mean_step_time 1436.95, it/s 0.7 | |
| [0130 17:51:59 @multigpu.py:323] [p0574] step: count(915), step_time 1396.61, mean_step_time 1423.93, it/s 0.7 | |
| [0130 17:51:59 @multigpu.py:323] [p0115] step: count(900), step_time 1392.29, mean_step_time 1430.64, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.366329143312 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.366329143312 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.366329143312 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.366329351332 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.36632966472 , -0.00497338781133 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.36632966472 , 0.409150511026 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.36632966472 , -2.28915762901 ##### | |
| ##### Sending to neptune: value_loss : 0.36632966472 , 1.24341356754 ##### | |
| ##### Sending to neptune: advantage : 0.36632966472 , -0.00184266630094 ##### | |
| ##### Sending to neptune: pred_reward : 0.36632966472 , 0.362062007189 ##### | |
| ##### Sending to neptune: max_logit : 0.36632966472 , 0.194479733706 ##### | |
| [u'loss', -0.004973387811332941, 0.40915051102638245, -2.2891576290130615, 1.2434135675430298, -0.001842666300944984, 0.362062007188797, 0.19447973370552063] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.366330153611 , 9098058.43 ##### | |
| ##### Sending to neptune: dp_per_s : 0.366330153611 , 88.9767374844 ##### | |
| [u'other', 9098058.43, 88.97673748436563] | |
| receiving | |
| [0130 17:51:59 @multigpu.py:323] [p0576] step: count(924), step_time 1418.48, mean_step_time 1438.94, it/s 0.69 | |
| [0130 17:52:01 @multigpu.py:323] [p0574] step: count(916), step_time 1421.35, mean_step_time 1423.84, it/s 0.7 | |
| [0130 17:52:01 @multigpu.py:323] [p0115] step: count(901), step_time 1459.75, mean_step_time 1432.53, it/s 0.7 | |
| [0130 17:52:01 @multigpu.py:323] [p0576] step: count(925), step_time 1415.02, mean_step_time 1437.77, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.366962843868 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:52:02 @multigpu.py:323] [p0574] step: count(917), step_time 1491.58, mean_step_time 1426.72, it/s 0.7 | |
| [0130 17:52:02 @multigpu.py:323] [p0115] step: count(902), step_time 1418.01, mean_step_time 1432.23, it/s 0.7 | |
| [0130 17:52:02 @multigpu.py:323] [p0576] step: count(926), step_time 1406.49, mean_step_time 1437.45, it/s 0.7 | |
| [0130 17:52:04 @multigpu.py:323] [p0574] step: count(918), step_time 1416.31, mean_step_time 1425.65, it/s 0.7 | |
| [0130 17:52:04 @multigpu.py:323] [p0115] step: count(903), step_time 1426.32, mean_step_time 1428.1, it/s 0.7 | |
| [0130 17:52:04 @multigpu.py:323] [p0576] step: count(927), step_time 1441.29, mean_step_time 1439.47, it/s 0.69 | |
| [0130 17:52:05 @multigpu.py:323] [p0574] step: count(919), step_time 1489.08, mean_step_time 1430.67, it/s 0.7 | |
| [0130 17:52:05 @multigpu.py:323] [p0115] step: count(904), step_time 1470.43, mean_step_time 1430.83, it/s 0.7 | |
| [0130 17:52:05 @multigpu.py:323] [p0576] step: count(928), step_time 1460.7, mean_step_time 1439.53, it/s 0.69 | |
| [0130 17:52:07 @multigpu.py:323] [p0574] step: count(920), step_time 1434.11, mean_step_time 1430.25, it/s 0.7 | |
| [0130 17:52:07 @multigpu.py:323] [p0115] step: count(905), step_time 1428.51, mean_step_time 1429.53, it/s 0.7 | |
| [0130 17:52:07 @multigpu.py:323] [p0576] step: count(929), step_time 1417.13, mean_step_time 1442.8, it/s 0.69 | |
| [0130 17:52:08 @multigpu.py:323] [p0115] step: count(906), step_time 1419.18, mean_step_time 1428.07, it/s 0.7 | |
| [0130 17:52:08 @multigpu.py:323] [p0574] step: count(921), step_time 1438.38, mean_step_time 1431.66, it/s 0.7 | |
| [0130 17:52:08 @multigpu.py:323] [p0576] step: count(930), step_time 1424.73, mean_step_time 1437.33, it/s 0.7 | |
| [0130 17:52:09 @multigpu.py:323] [p0115] step: count(907), step_time 1414.07, mean_step_time 1425.98, it/s 0.7 | |
| [0130 17:52:09 @multigpu.py:323] [p0574] step: count(922), step_time 1411.44, mean_step_time 1429.99, it/s 0.7 | |
| [0130 17:52:09 @multigpu.py:323] [p0576] step: count(931), step_time 1422.66, mean_step_time 1438.64, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.369363233844 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:52:11 @multigpu.py:323] [p0115] step: count(908), step_time 1445.59, mean_step_time 1428.06, it/s 0.7 | |
| [0130 17:52:11 @multigpu.py:323] [p0574] step: count(923), step_time 1445.31, mean_step_time 1428.12, it/s 0.7 | |
| [0130 17:52:11 @multigpu.py:323] [p0576] step: count(932), step_time 1426.28, mean_step_time 1428.14, it/s 0.7 | |
| [0130 17:52:12 @multigpu.py:323] [p0115] step: count(909), step_time 1423.35, mean_step_time 1425.16, it/s 0.7 | |
| [0130 17:52:12 @multigpu.py:323] [p0576] step: count(933), step_time 1424.03, mean_step_time 1426.82, it/s 0.7 | |
| [0130 17:52:12 @multigpu.py:323] [p0574] step: count(924), step_time 1471.94, mean_step_time 1432.25, it/s 0.7 | |
| [0130 17:52:14 @multigpu.py:323] [p0115] step: count(910), step_time 1438.34, mean_step_time 1426.78, it/s 0.7 | |
| [0130 17:52:14 @multigpu.py:323] [p0576] step: count(934), step_time 1455.56, mean_step_time 1428.58, it/s 0.7 | |
| [0130 17:52:14 @multigpu.py:323] [p0574] step: count(925), step_time 1450.05, mean_step_time 1432.18, it/s 0.7 | |
| [0130 17:52:15 @multigpu.py:323] [p0115] step: count(911), step_time 1494.43, mean_step_time 1432.41, it/s 0.7 | |
| [0130 17:52:15 @multigpu.py:323] [p0574] step: count(926), step_time 1443.2, mean_step_time 1433.49, it/s 0.7 | |
| [0130 17:52:15 @multigpu.py:323] [p0576] step: count(935), step_time 1490.3, mean_step_time 1430.43, it/s 0.7 | |
| [0130 17:52:17 @multigpu.py:323] [p0576] step: count(936), step_time 1407.64, mean_step_time 1430.01, it/s 0.7 | |
| [0130 17:52:17 @multigpu.py:323] [p0115] step: count(912), step_time 1449.7, mean_step_time 1433.24, it/s 0.7 | |
| [0130 17:52:17 @multigpu.py:323] [p0574] step: count(927), step_time 1445.49, mean_step_time 1433.08, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.371138255795 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:52:18 @multigpu.py:323] [p0115] step: count(913), step_time 1401.75, mean_step_time 1432.44, it/s 0.7 | |
| [0130 17:52:18 @multigpu.py:323] [p0576] step: count(937), step_time 1431.2, mean_step_time 1431.18, it/s 0.7 | |
| [0130 17:52:18 @multigpu.py:323] [p0574] step: count(928), step_time 1416.73, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:52:19 @multigpu.py:323] [p0576] step: count(938), step_time 1408.21, mean_step_time 1429.12, it/s 0.7 | |
| [0130 17:52:20 @multigpu.py:323] [p0574] step: count(929), step_time 1421.65, mean_step_time 1432.73, it/s 0.7 | |
| [0130 17:52:20 @multigpu.py:323] [p0115] step: count(914), step_time 1503.67, mean_step_time 1434.21, it/s 0.7 | |
| [0130 17:52:21 @multigpu.py:323] [p0576] step: count(939), step_time 1432.68, mean_step_time 1428.89, it/s 0.7 | |
| [0130 17:52:21 @multigpu.py:323] [p0115] step: count(915), step_time 1405.72, mean_step_time 1430.97, it/s 0.7 | |
| [0130 17:52:21 @multigpu.py:323] [p0574] step: count(930), step_time 1506.38, mean_step_time 1439.12, it/s 0.69 | |
| [0130 17:52:22 @multigpu.py:323] [p0576] step: count(940), step_time 1390.55, mean_step_time 1429.6, it/s 0.7 | |
| [0130 17:52:22 @multigpu.py:323] [p0574] step: count(931), step_time 1398.75, mean_step_time 1439.07, it/s 0.69 | |
| [0130 17:52:22 @multigpu.py:323] [p0115] step: count(916), step_time 1443.18, mean_step_time 1434.39, it/s 0.7 | |
| [0130 17:52:24 @multigpu.py:323] [p0576] step: count(941), step_time 1413.95, mean_step_time 1425.47, it/s 0.7 | |
| [0130 17:52:24 @multigpu.py:323] [p0115] step: count(917), step_time 1429.44, mean_step_time 1434.65, it/s 0.7 | |
| [0130 17:52:24 @multigpu.py:323] [p0574] step: count(932), step_time 1435.86, mean_step_time 1439.75, it/s 0.69 | |
| [0130 17:52:25 @multigpu.py:323] [p0576] step: count(942), step_time 1419.92, mean_step_time 1425.25, it/s 0.7 | |
| [0130 17:52:25 @multigpu.py:323] [p0574] step: count(933), step_time 1378.39, mean_step_time 1437.54, it/s 0.7 | |
| [0130 17:52:25 @multigpu.py:323] [p0115] step: count(918), step_time 1477.73, mean_step_time 1438.12, it/s 0.7 | |
| [0130 17:52:27 @multigpu.py:323] [p0576] step: count(943), step_time 1432.5, mean_step_time 1426.97, it/s 0.7 | |
| [0130 17:52:27 @multigpu.py:323] [p0574] step: count(934), step_time 1421.4, mean_step_time 1436.7, it/s 0.7 | |
| [0130 17:52:27 @multigpu.py:323] [p0115] step: count(919), step_time 1469.72, mean_step_time 1440.56, it/s 0.69 | |
| [0130 17:52:28 @multigpu.py:323] [p0576] step: count(944), step_time 1463.06, mean_step_time 1429.19, it/s 0.7 | |
| [0130 17:52:28 @multigpu.py:323] [p0574] step: count(935), step_time 1433.35, mean_step_time 1438.54, it/s 0.7 | |
| [0130 17:52:28 @multigpu.py:323] [p0115] step: count(920), step_time 1392.17, mean_step_time 1440.55, it/s 0.69 | |
| [0130 17:52:30 @multigpu.py:323] [p0576] step: count(945), step_time 1464.76, mean_step_time 1431.68, it/s 0.7 | |
| [0130 17:52:30 @multigpu.py:323] [p0574] step: count(936), step_time 1420.2, mean_step_time 1438.48, it/s 0.7 | |
| [0130 17:52:30 @multigpu.py:323] [p0115] step: count(921), step_time 1468.92, mean_step_time 1441.01, it/s 0.69 | |
| [0130 17:52:31 @multigpu.py:323] [p0576] step: count(946), step_time 1380.21, mean_step_time 1430.37, it/s 0.7 | |
| [0130 17:52:31 @multigpu.py:323] [p0574] step: count(937), step_time 1443.1, mean_step_time 1436.05, it/s 0.7 | |
| [0130 17:52:31 @multigpu.py:323] [p0115] step: count(922), step_time 1389.22, mean_step_time 1439.57, it/s 0.69 | |
| [0130 17:52:32 @multigpu.py:323] [p0576] step: count(947), step_time 1413.24, mean_step_time 1428.97, it/s 0.7 | |
| [0130 17:52:32 @multigpu.py:323] [p0574] step: count(938), step_time 1386.52, mean_step_time 1434.56, it/s 0.7 | |
| [0130 17:52:33 @multigpu.py:323] [p0115] step: count(923), step_time 1465.42, mean_step_time 1441.53, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.375655125512 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:52:34 @multigpu.py:323] [p0574] step: count(939), step_time 1403.78, mean_step_time 1430.3, it/s 0.7 | |
| [0130 17:52:34 @multigpu.py:323] [p0576] step: count(948), step_time 1467.53, mean_step_time 1429.31, it/s 0.7 | |
| [0130 17:52:34 @multigpu.py:323] [p0115] step: count(924), step_time 1459.0, mean_step_time 1440.96, it/s 0.69 | |
| [0130 17:52:35 @multigpu.py:323] [p0576] step: count(949), step_time 1397.18, mean_step_time 1428.31, it/s 0.7 | |
| [0130 17:52:35 @multigpu.py:323] [p0574] step: count(940), step_time 1420.18, mean_step_time 1429.6, it/s 0.7 | |
| [0130 17:52:35 @multigpu.py:323] [p0115] step: count(925), step_time 1417.88, mean_step_time 1440.42, it/s 0.69 | |
| [0130 17:52:37 @multigpu.py:323] [p0574] step: count(941), step_time 1432.8, mean_step_time 1429.32, it/s 0.7 | |
| [0130 17:52:37 @multigpu.py:323] [p0576] step: count(950), step_time 1466.85, mean_step_time 1430.42, it/s 0.7 | |
| [0130 17:52:37 @multigpu.py:323] [p0115] step: count(926), step_time 1421.81, mean_step_time 1440.56, it/s 0.69 | |
| [0130 17:52:38 @multigpu.py:323] [p0574] step: count(942), step_time 1401.24, mean_step_time 1428.81, it/s 0.7 | |
| [0130 17:52:38 @multigpu.py:323] [p0576] step: count(951), step_time 1443.41, mean_step_time 1431.45, it/s 0.7 | |
| [0130 17:52:38 @multigpu.py:323] [p0115] step: count(927), step_time 1461.71, mean_step_time 1442.94, it/s 0.69 | |
| [0130 17:52:40 @multigpu.py:323] [p0115] step: count(928), step_time 1489.4, mean_step_time 1445.13, it/s 0.69 | |
| [0130 17:52:40 @multigpu.py:323] [p0574] step: count(943), step_time 1767.06, mean_step_time 1444.9, it/s 0.69 | |
| [0130 17:52:40 @multigpu.py:323] [p0576] step: count(952), step_time 1698.21, mean_step_time 1445.05, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.377704124981 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 17:52:41 @multigpu.py:323] [p0115] step: count(929), step_time 1411.02, mean_step_time 1444.51, it/s 0.69 | |
| [0130 17:52:41 @multigpu.py:323] [p0576] step: count(953), step_time 1473.86, mean_step_time 1447.54, it/s 0.69 | |
| [0130 17:52:41 @multigpu.py:323] [p0574] step: count(944), step_time 1499.27, mean_step_time 1446.27, it/s 0.69 | |
| [0130 17:52:43 @multigpu.py:323] [p0115] step: count(930), step_time 1465.97, mean_step_time 1445.89, it/s 0.69 | |
| [0130 17:52:43 @multigpu.py:323] [p0576] step: count(954), step_time 1407.2, mean_step_time 1445.12, it/s 0.69 | |
| [0130 17:52:43 @multigpu.py:323] [p0574] step: count(945), step_time 1412.17, mean_step_time 1444.37, it/s 0.69 | |
| [0130 17:52:44 @multigpu.py:323] [p0576] step: count(955), step_time 1382.05, mean_step_time 1439.71, it/s 0.69 | |
| [0130 17:52:44 @multigpu.py:323] [p0115] step: count(931), step_time 1397.76, mean_step_time 1441.06, it/s 0.69 | |
| [0130 17:52:44 @multigpu.py:323] [p0574] step: count(946), step_time 1457.16, mean_step_time 1445.07, it/s 0.69 | |
| [0130 17:52:46 @multigpu.py:323] [p0115] step: count(932), step_time 1439.31, mean_step_time 1440.54, it/s 0.69 | |
| [0130 17:52:46 @multigpu.py:323] [p0576] step: count(956), step_time 1478.99, mean_step_time 1443.28, it/s 0.69 | |
| [0130 17:52:46 @multigpu.py:323] [p0574] step: count(947), step_time 1416.09, mean_step_time 1443.6, it/s 0.69 | |
| [0130 17:52:47 @multigpu.py:323] [p0115] step: count(933), step_time 1444.71, mean_step_time 1442.69, it/s 0.69 | |
| [0130 17:52:47 @multigpu.py:323] [p0576] step: count(957), step_time 1423.76, mean_step_time 1442.91, it/s 0.69 | |
| [0130 17:52:47 @multigpu.py:323] [p0574] step: count(948), step_time 1437.24, mean_step_time 1444.63, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.379729216364 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:52:48 @multigpu.py:323] [p0576] step: count(958), step_time 1396.68, mean_step_time 1442.33, it/s 0.69 | |
| [0130 17:52:48 @multigpu.py:323] [p0115] step: count(934), step_time 1444.85, mean_step_time 1439.75, it/s 0.69 | |
| [0130 17:52:48 @multigpu.py:323] [p0574] step: count(949), step_time 1407.62, mean_step_time 1443.93, it/s 0.69 | |
| [0130 17:52:50 @multigpu.py:323] [p0576] step: count(959), step_time 1417.47, mean_step_time 1441.57, it/s 0.69 | |
| [0130 17:52:50 @multigpu.py:323] [p0574] step: count(950), step_time 1404.91, mean_step_time 1438.85, it/s 0.69 | |
| [0130 17:52:50 @multigpu.py:323] [p0115] step: count(935), step_time 1462.43, mean_step_time 1442.58, it/s 0.69 | |
| [0130 17:52:51 @multigpu.py:323] [p0576] step: count(960), step_time 1401.59, mean_step_time 1442.12, it/s 0.69 | |
| [0130 17:52:51 @multigpu.py:323] [p0574] step: count(951), step_time 1445.19, mean_step_time 1441.18, it/s 0.69 | |
| [0130 17:52:51 @multigpu.py:323] [p0115] step: count(936), step_time 1465.44, mean_step_time 1443.7, it/s 0.69 | |
| [0130 17:52:53 @multigpu.py:323] [p0576] step: count(961), step_time 1413.59, mean_step_time 1442.1, it/s 0.69 | |
| [0130 17:52:53 @multigpu.py:323] [p0574] step: count(952), step_time 1432.08, mean_step_time 1440.99, it/s 0.69 | |
| [0130 17:52:53 @multigpu.py:323] [p0115] step: count(937), step_time 1403.75, mean_step_time 1442.41, it/s 0.69 | |
| [0130 17:52:54 @multigpu.py:323] [p0576] step: count(962), step_time 1436.9, mean_step_time 1442.95, it/s 0.69 | |
| [0130 17:52:54 @multigpu.py:323] [p0574] step: count(953), step_time 1420.77, mean_step_time 1443.11, it/s 0.69 | |
| [0130 17:52:54 @multigpu.py:323] [p0115] step: count(938), step_time 1450.15, mean_step_time 1441.03, it/s 0.69 | |
| [0130 17:52:55 @multigpu.py:323] [p0576] step: count(963), step_time 1391.48, mean_step_time 1440.9, it/s 0.69 | |
| [0130 17:52:56 @multigpu.py:323] [p0115] step: count(939), step_time 1399.14, mean_step_time 1437.5, it/s 0.7 | |
| [0130 17:52:56 @multigpu.py:323] [p0574] step: count(954), step_time 1484.84, mean_step_time 1446.28, it/s 0.69 | |
| [0130 17:52:57 @multigpu.py:323] [p0576] step: count(964), step_time 1456.94, mean_step_time 1440.59, it/s 0.69 | |
| [0130 17:52:57 @multigpu.py:323] [p0115] step: count(940), step_time 1430.27, mean_step_time 1439.41, it/s 0.69 | |
| [0130 17:52:57 @multigpu.py:323] [p0574] step: count(955), step_time 1496.64, mean_step_time 1449.44, it/s 0.69 | |
| [0130 17:52:58 @multigpu.py:323] [p0576] step: count(965), step_time 1422.67, mean_step_time 1438.49, it/s 0.7 | |
| [0130 17:52:58 @multigpu.py:323] [p0115] step: count(941), step_time 1436.0, mean_step_time 1437.76, it/s 0.7 | |
| [0130 17:52:59 @multigpu.py:323] [p0574] step: count(956), step_time 1402.66, mean_step_time 1448.57, it/s 0.69 | |
| [0130 17:53:00 @multigpu.py:323] [p0576] step: count(966), step_time 1394.0, mean_step_time 1439.18, it/s 0.69 | |
| [0130 17:53:00 @multigpu.py:323] [p0115] step: count(942), step_time 1460.6, mean_step_time 1441.33, it/s 0.69 | |
| [0130 17:53:00 @multigpu.py:323] [p0574] step: count(957), step_time 1448.67, mean_step_time 1448.84, it/s 0.69 | |
| [0130 17:53:01 @multigpu.py:323] [p0576] step: count(967), step_time 1405.76, mean_step_time 1438.81, it/s 0.7 | |
| [0130 17:53:01 @multigpu.py:323] [p0115] step: count(943), step_time 1406.58, mean_step_time 1438.39, it/s 0.7 | |
| [0130 17:53:01 @multigpu.py:323] [p0574] step: count(958), step_time 1448.4, mean_step_time 1451.94, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.383631636103 , 2.6 ##### | |
| [u'online', 2.6] | |
| ##### Sending to neptune: active_workers : 0.383631717165 , 3 ##### | |
| receiving | |
| [0130 17:53:02 @multigpu.py:323] [p0576] step: count(968), step_time 1398.77, mean_step_time 1435.37, it/s 0.7 | |
| [0130 17:53:03 @multigpu.py:323] [p0115] step: count(944), step_time 1453.51, mean_step_time 1438.12, it/s 0.7 | |
| [0130 17:53:03 @multigpu.py:323] [p0574] step: count(959), step_time 1431.84, mean_step_time 1453.34, it/s 0.69 | |
| [0130 17:53:04 @multigpu.py:323] [p0576] step: count(969), step_time 1474.77, mean_step_time 1439.25, it/s 0.69 | |
| [0130 17:53:04 @multigpu.py:323] [p0115] step: count(945), step_time 1394.46, mean_step_time 1436.94, it/s 0.7 | |
| [0130 17:53:04 @multigpu.py:323] [p0574] step: count(960), step_time 1483.19, mean_step_time 1456.49, it/s 0.69 | |
| [0130 17:53:05 @multigpu.py:323] [p0576] step: count(970), step_time 1470.58, mean_step_time 1439.43, it/s 0.69 | |
| [0130 17:53:06 @multigpu.py:323] [p0115] step: count(946), step_time 1519.65, mean_step_time 1441.84, it/s 0.69 | |
| [0130 17:53:06 @multigpu.py:323] [p0574] step: count(961), step_time 1434.17, mean_step_time 1456.56, it/s 0.69 | |
| [0130 17:53:07 @multigpu.py:323] [p0576] step: count(971), step_time 1422.48, mean_step_time 1438.39, it/s 0.7 | |
| [0130 17:53:07 @multigpu.py:323] [p0115] step: count(947), step_time 1403.57, mean_step_time 1438.93, it/s 0.69 | |
| [0130 17:53:07 @multigpu.py:323] [p0574] step: count(962), step_time 1431.35, mean_step_time 1458.07, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.385261432992 , 2.5 ##### | |
| [u'online', 2.5] | |
| receiving | |
| [0130 17:53:09 @multigpu.py:323] [p0574] step: count(963), step_time 1488.1, mean_step_time 1444.12, it/s 0.69 | |
| [0130 17:53:09 @multigpu.py:323] [p0576] step: count(972), step_time 1815.85, mean_step_time 1444.27, it/s 0.69 | |
| [0130 17:53:09 @multigpu.py:323] [p0115] step: count(948), step_time 1599.35, mean_step_time 1444.43, it/s 0.69 | |
| [0130 17:53:10 @multigpu.py:323] [p0115] step: count(949), step_time 1377.1, mean_step_time 1442.73, it/s 0.69 | |
| [0130 17:53:10 @multigpu.py:323] [p0574] step: count(964), step_time 1450.41, mean_step_time 1441.67, it/s 0.69 | |
| [0130 17:53:10 @multigpu.py:323] [p0576] step: count(973), step_time 1457.84, mean_step_time 1443.47, it/s 0.69 | |
| [0130 17:53:11 @multigpu.py:323] [p0115] step: count(950), step_time 1417.53, mean_step_time 1440.31, it/s 0.69 | |
| [0130 17:53:12 @multigpu.py:323] [p0574] step: count(965), step_time 1431.62, mean_step_time 1442.65, it/s 0.69 | |
| [0130 17:53:12 @multigpu.py:323] [p0576] step: count(974), step_time 1460.11, mean_step_time 1446.11, it/s 0.69 | |
| [0130 17:53:13 @multigpu.py:323] [p0115] step: count(951), step_time 1498.04, mean_step_time 1445.32, it/s 0.69 | |
| [0130 17:53:13 @multigpu.py:323] [p0576] step: count(975), step_time 1427.4, mean_step_time 1448.38, it/s 0.69 | |
| [0130 17:53:13 @multigpu.py:323] [p0574] step: count(966), step_time 1468.06, mean_step_time 1443.19, it/s 0.69 | |
| [0130 17:53:14 @multigpu.py:323] [p0115] step: count(952), step_time 1413.3, mean_step_time 1444.02, it/s 0.69 | |
| [0130 17:53:14 @multigpu.py:323] [p0574] step: count(967), step_time 1408.96, mean_step_time 1442.84, it/s 0.69 | |
| [0130 17:53:14 @multigpu.py:323] [p0576] step: count(976), step_time 1427.37, mean_step_time 1445.8, it/s 0.69 | |
| [0130 17:53:16 @multigpu.py:323] [p0115] step: count(953), step_time 1412.05, mean_step_time 1442.39, it/s 0.69 | |
| [0130 17:53:16 @multigpu.py:323] [p0576] step: count(977), step_time 1417.17, mean_step_time 1445.47, it/s 0.69 | |
| [0130 17:53:16 @multigpu.py:323] [p0574] step: count(968), step_time 1450.61, mean_step_time 1443.5, it/s 0.69 | |
| [0130 17:53:17 @multigpu.py:323] [p0574] step: count(969), step_time 1392.42, mean_step_time 1442.74, it/s 0.69 | |
| [0130 17:53:17 @multigpu.py:323] [p0115] step: count(954), step_time 1479.82, mean_step_time 1444.14, it/s 0.69 | |
| [0130 17:53:17 @multigpu.py:323] [p0576] step: count(978), step_time 1437.37, mean_step_time 1447.51, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.388035714693 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:53:19 @multigpu.py:323] [p0115] step: count(955), step_time 1388.68, mean_step_time 1440.45, it/s 0.69 | |
| [0130 17:53:19 @multigpu.py:323] [p0574] step: count(970), step_time 1410.08, mean_step_time 1443.0, it/s 0.69 | |
| [0130 17:53:19 @multigpu.py:323] [p0576] step: count(979), step_time 1452.04, mean_step_time 1449.23, it/s 0.69 | |
| [0130 17:53:20 @multigpu.py:323] [p0115] step: count(956), step_time 1440.68, mean_step_time 1439.21, it/s 0.69 | |
| [0130 17:53:20 @multigpu.py:323] [p0574] step: count(971), step_time 1435.06, mean_step_time 1442.5, it/s 0.69 | |
| [0130 17:53:20 @multigpu.py:323] [p0576] step: count(980), step_time 1443.09, mean_step_time 1451.31, it/s 0.69 | |
| [0130 17:53:22 @multigpu.py:323] [p0574] step: count(972), step_time 1401.32, mean_step_time 1440.96, it/s 0.69 | |
| [0130 17:53:22 @multigpu.py:323] [p0115] step: count(957), step_time 1474.39, mean_step_time 1442.74, it/s 0.69 | |
| [0130 17:53:22 @multigpu.py:323] [p0576] step: count(981), step_time 1491.56, mean_step_time 1455.21, it/s 0.69 | |
| [0130 17:53:23 @multigpu.py:323] [p0574] step: count(973), step_time 1395.79, mean_step_time 1439.71, it/s 0.69 | |
| [0130 17:53:23 @multigpu.py:323] [p0115] step: count(958), step_time 1392.72, mean_step_time 1439.87, it/s 0.69 | |
| [0130 17:53:23 @multigpu.py:323] [p0576] step: count(982), step_time 1429.28, mean_step_time 1454.83, it/s 0.69 | |
| [0130 17:53:24 @multigpu.py:323] [p0574] step: count(974), step_time 1448.35, mean_step_time 1437.89, it/s 0.7 | |
| [0130 17:53:24 @multigpu.py:323] [p0115] step: count(959), step_time 1500.38, mean_step_time 1444.93, it/s 0.69 | |
| [0130 17:53:25 @multigpu.py:323] [p0576] step: count(983), step_time 1436.56, mean_step_time 1457.08, it/s 0.69 | |
| [0130 17:53:26 @multigpu.py:323] [p0574] step: count(975), step_time 1406.09, mean_step_time 1433.36, it/s 0.7 | |
| [0130 17:53:26 @multigpu.py:323] [p0115] step: count(960), step_time 1421.83, mean_step_time 1444.51, it/s 0.69 | |
| [0130 17:53:26 @multigpu.py:323] [p0576] step: count(984), step_time 1435.02, mean_step_time 1455.99, it/s 0.69 | |
| [0130 17:53:27 @multigpu.py:323] [p0574] step: count(976), step_time 1450.68, mean_step_time 1435.76, it/s 0.7 | |
| [0130 17:53:27 @multigpu.py:323] [p0115] step: count(961), step_time 1472.84, mean_step_time 1446.35, it/s 0.69 | |
| [0130 17:53:27 @multigpu.py:323] [p0576] step: count(985), step_time 1397.26, mean_step_time 1454.72, it/s 0.69 | |
| [0130 17:53:29 @multigpu.py:323] [p0574] step: count(977), step_time 1470.44, mean_step_time 1436.85, it/s 0.7 | |
| [0130 17:53:29 @multigpu.py:323] [p0576] step: count(986), step_time 1413.1, mean_step_time 1455.67, it/s 0.69 | |
| [0130 17:53:29 @multigpu.py:323] [p0115] step: count(962), step_time 1433.59, mean_step_time 1445.0, it/s 0.69 | |
| [0130 17:53:30 @multigpu.py:323] [p0574] step: count(978), step_time 1425.64, mean_step_time 1435.71, it/s 0.7 | |
| [0130 17:53:30 @multigpu.py:323] [p0576] step: count(987), step_time 1473.67, mean_step_time 1459.07, it/s 0.69 | |
| [0130 17:53:30 @multigpu.py:323] [p0115] step: count(963), step_time 1473.48, mean_step_time 1448.35, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.391809822453 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:53:32 @multigpu.py:323] [p0574] step: count(979), step_time 1390.61, mean_step_time 1433.65, it/s 0.7 | |
| [0130 17:53:32 @multigpu.py:323] [p0115] step: count(964), step_time 1384.83, mean_step_time 1444.91, it/s 0.69 | |
| [0130 17:53:32 @multigpu.py:323] [p0576] step: count(988), step_time 1442.05, mean_step_time 1461.23, it/s 0.68 | |
| [0130 17:53:33 @multigpu.py:323] [p0574] step: count(980), step_time 1450.56, mean_step_time 1432.02, it/s 0.7 | |
| [0130 17:53:33 @multigpu.py:323] [p0576] step: count(989), step_time 1396.6, mean_step_time 1457.32, it/s 0.69 | |
| [0130 17:53:33 @multigpu.py:323] [p0115] step: count(965), step_time 1480.84, mean_step_time 1449.23, it/s 0.69 | |
| [0130 17:53:34 @multigpu.py:323] [p0574] step: count(981), step_time 1427.71, mean_step_time 1431.69, it/s 0.7 | |
| [0130 17:53:35 @multigpu.py:323] [p0576] step: count(990), step_time 1409.66, mean_step_time 1454.27, it/s 0.69 | |
| [0130 17:53:35 @multigpu.py:323] [p0115] step: count(966), step_time 1397.85, mean_step_time 1443.14, it/s 0.69 | |
| [0130 17:53:36 @multigpu.py:323] [p0574] step: count(982), step_time 1412.82, mean_step_time 1430.77, it/s 0.7 | |
| [0130 17:53:36 @multigpu.py:323] [p0576] step: count(991), step_time 1453.47, mean_step_time 1455.82, it/s 0.69 | |
| [0130 17:53:36 @multigpu.py:323] [p0115] step: count(967), step_time 1452.68, mean_step_time 1445.6, it/s 0.69 | |
| [0130 17:53:37 @multigpu.py:323] [p0115] step: count(968), step_time 1397.95, mean_step_time 1435.53, it/s 0.7 | |
| [0130 17:53:37 @multigpu.py:323] [p0574] step: count(983), step_time 1587.46, mean_step_time 1435.74, it/s 0.7 | |
| [0130 17:53:37 @multigpu.py:323] [p0576] step: count(992), step_time 1414.94, mean_step_time 1435.78, it/s 0.7 | |
| [0130 17:53:39 @multigpu.py:323] [p0115] step: count(969), step_time 1432.59, mean_step_time 1438.3, it/s 0.7 | |
| [0130 17:53:39 @multigpu.py:323] [p0574] step: count(984), step_time 1438.65, mean_step_time 1435.15, it/s 0.7 | |
| [0130 17:53:39 @multigpu.py:323] [p0576] step: count(993), step_time 1525.21, mean_step_time 1439.15, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.394126037492 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:53:40 @multigpu.py:323] [p0115] step: count(970), step_time 1408.3, mean_step_time 1437.84, it/s 0.7 | |
| [0130 17:53:40 @multigpu.py:323] [p0574] step: count(985), step_time 1457.66, mean_step_time 1436.45, it/s 0.7 | |
| [0130 17:53:40 @multigpu.py:323] [p0576] step: count(994), step_time 1441.68, mean_step_time 1438.23, it/s 0.7 | |
| [0130 17:53:42 @multigpu.py:323] [p0115] step: count(971), step_time 1414.53, mean_step_time 1433.67, it/s 0.7 | |
| [0130 17:53:42 @multigpu.py:323] [p0574] step: count(986), step_time 1401.51, mean_step_time 1433.12, it/s 0.7 | |
| [0130 17:53:42 @multigpu.py:323] [p0576] step: count(995), step_time 1420.92, mean_step_time 1437.9, it/s 0.7 | |
| [0130 17:53:43 @multigpu.py:323] [p0115] step: count(972), step_time 1420.67, mean_step_time 1434.03, it/s 0.7 | |
| [0130 17:53:43 @multigpu.py:323] [p0574] step: count(987), step_time 1406.8, mean_step_time 1433.01, it/s 0.7 | |
| [0130 17:53:43 @multigpu.py:323] [p0576] step: count(996), step_time 1412.42, mean_step_time 1437.15, it/s 0.7 | |
| [0130 17:53:45 @multigpu.py:323] [p0115] step: count(973), step_time 1475.81, mean_step_time 1437.22, it/s 0.7 | |
| [0130 17:53:45 @multigpu.py:323] [p0574] step: count(988), step_time 1505.52, mean_step_time 1435.76, it/s 0.7 | |
| [0130 17:53:45 @multigpu.py:323] [p0576] step: count(997), step_time 1430.64, mean_step_time 1437.83, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.395659864677 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 17:53:46 @multigpu.py:323] [p0115] step: count(974), step_time 1447.97, mean_step_time 1435.63, it/s 0.7 | |
| [0130 17:53:46 @multigpu.py:323] [p0574] step: count(989), step_time 1450.31, mean_step_time 1438.65, it/s 0.7 | |
| [0130 17:53:46 @multigpu.py:323] [p0576] step: count(998), step_time 1459.58, mean_step_time 1438.94, it/s 0.69 | |
| [0130 17:53:47 @multigpu.py:323] [p0115] step: count(975), step_time 1414.3, mean_step_time 1436.91, it/s 0.7 | |
| [0130 17:53:47 @multigpu.py:323] [p0574] step: count(990), step_time 1403.42, mean_step_time 1438.32, it/s 0.7 | |
| [0130 17:53:48 @multigpu.py:323] [p0576] step: count(999), step_time 1423.1, mean_step_time 1437.49, it/s 0.7 | |
| [0130 17:53:49 @multigpu.py:323] [p0115] step: count(976), step_time 1433.69, mean_step_time 1436.56, it/s 0.7 | |
| [0130 17:53:49 @multigpu.py:323] [p0576] step: count(1000), step_time 1402.87, mean_step_time 1435.48, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.396757584148 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.396757584148 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.396757584148 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.396758102775 , -0.00669238669798 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.396758102775 , 0.0405781567097 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.396758102775 , -2.28896999359 ##### | |
| ##### Sending to neptune: value_loss : 0.396758102775 , 1.39176607132 ##### | |
| ##### Sending to neptune: advantage : 0.396758102775 , -0.000199115689611 ##### | |
| ##### Sending to neptune: pred_reward : 0.396758102775 , 0.394293904305 ##### | |
| ##### Sending to neptune: max_logit : 0.396758102775 , 0.192680105567 ##### | |
| [u'loss', -0.006692386697977781, 0.04057815670967102, -2.2889699935913086, 1.39176607131958, -0.0001991156896110624, 0.3942939043045044, 0.19268010556697845] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.396758571664 , 9202785.36 ##### | |
| ##### Sending to neptune: dp_per_s : 0.396758571664 , 88.8521250407 ##### | |
| [u'other', 9202785.36, 88.8521250406519] | |
| receiving | |
| 100%|##########|1000/1000[23:48<00:00, 0.70it/s] | |
| [0130 17:53:49 @multigpu.py:323] [p0574] step: count(991), step_time 1493.14, mean_step_time 1441.22, it/s 0.69 | |
| [0130 17:53:49 @stat.py:81] async_global_step: 1000 | |
| [0130 17:53:49 @stat.py:81] learning_rate_1: 0.00015 | |
| EPOCH ENDS HERE | |
| [0130 17:53:49 @timer.py:52] Epoch 1, global_step=1000 finished, time=1428.18sec. | |
| [0130 17:53:50 @multigpu.py:323] [p0115] step: count(977), step_time 1437.93, mean_step_time 1434.74, it/s 0.7 | |
| [0130 17:53:50 @multigpu.py:323] [p0574] step: count(992), step_time 1422.59, mean_step_time 1442.29, it/s 0.69 | |
| [0130 17:53:50 @multigpu.py:323] [p0576] step: count(1001), step_time 1473.44, mean_step_time 1434.58, it/s 0.7 | |
| [0130 17:53:52 @multigpu.py:323] [p0115] step: count(978), step_time 1459.16, mean_step_time 1438.06, it/s 0.7 | |
| [0130 17:53:52 @multigpu.py:323] [p0576] step: count(1002), step_time 1379.25, mean_step_time 1432.07, it/s 0.7 | |
| [0130 17:53:52 @multigpu.py:323] [p0574] step: count(993), step_time 1417.62, mean_step_time 1443.38, it/s 0.69 | |
| [0130 17:53:53 @multigpu.py:323] [p0115] step: count(979), step_time 1421.27, mean_step_time 1434.11, it/s 0.7 | |
| [0130 17:53:53 @multigpu.py:323] [p0576] step: count(1003), step_time 1409.38, mean_step_time 1430.71, it/s 0.7 | |
| [0130 17:53:53 @multigpu.py:323] [p0574] step: count(994), step_time 1445.48, mean_step_time 1443.24, it/s 0.69 | |
| [0130 17:53:55 @multigpu.py:323] [p0576] step: count(1004), step_time 1420.49, mean_step_time 1429.99, it/s 0.7 | |
| [0130 17:53:55 @multigpu.py:323] [p0115] step: count(980), step_time 1438.41, mean_step_time 1434.93, it/s 0.7 | |
| [0130 17:53:55 @multigpu.py:323] [p0574] step: count(995), step_time 1407.12, mean_step_time 1443.29, it/s 0.69 | |
| [0130 17:53:56 @multigpu.py:323] [p0576] step: count(1005), step_time 1421.42, mean_step_time 1431.2, it/s 0.7 | |
| [0130 17:53:56 @multigpu.py:323] [p0115] step: count(981), step_time 1457.56, mean_step_time 1434.17, it/s 0.7 | |
| [0130 17:53:56 @multigpu.py:323] [p0574] step: count(996), step_time 1433.5, mean_step_time 1442.43, it/s 0.69 | |
| [0130 17:53:58 @multigpu.py:323] [p0576] step: count(1006), step_time 1467.35, mean_step_time 1433.91, it/s 0.7 | |
| [0130 17:53:58 @multigpu.py:323] [p0574] step: count(997), step_time 1414.08, mean_step_time 1439.61, it/s 0.69 | |
| [0130 17:53:58 @multigpu.py:323] [p0115] step: count(982), step_time 1437.83, mean_step_time 1434.38, it/s 0.7 | |
| [0130 17:53:59 @multigpu.py:323] [p0576] step: count(1007), step_time 1384.39, mean_step_time 1429.44, it/s 0.7 | |
| [0130 17:53:59 @multigpu.py:323] [p0115] step: count(983), step_time 1396.47, mean_step_time 1430.53, it/s 0.7 | |
| [0130 17:53:59 @multigpu.py:323] [p0574] step: count(998), step_time 1408.16, mean_step_time 1438.74, it/s 0.7 | |
| [0130 17:54:00 @multigpu.py:323] [p0576] step: count(1008), step_time 1379.76, mean_step_time 1426.33, it/s 0.7 | |
| [0130 17:54:00 @multigpu.py:323] [p0115] step: count(984), step_time 1402.52, mean_step_time 1431.42, it/s 0.7 | |
| [0130 17:54:00 @multigpu.py:323] [p0574] step: count(999), step_time 1406.89, mean_step_time 1439.55, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.400034431352 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 17:54:02 @multigpu.py:323] [p0576] step: count(1009), step_time 1431.42, mean_step_time 1428.07, it/s 0.7 | |
| [0130 17:54:02 @multigpu.py:323] [p0574] step: count(1000), step_time 1397.69, mean_step_time 1436.91, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.400312784976 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.400312784976 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.400312784976 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.400312963592 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.400313278304 , -0.014850821346 ##### | |
| ##### Sending to neptune: policy_loss : 0.400313278304 , -0.792145967484 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.400313278304 , -2.28885960579 ##### | |
| ##### Sending to neptune: value_loss : 0.400313278304 , 1.18010020256 ##### | |
| ##### Sending to neptune: advantage : 0.400313278304 , 0.00343013252132 ##### | |
| ##### Sending to neptune: pred_reward : 0.400313278304 , 0.402852445841 ##### | |
| ##### Sending to neptune: max_logit : 0.400313278304 , 0.192205965519 ##### | |
| [u'loss', -0.01485082134604454, -0.7921459674835205, -2.2888596057891846, 1.1801002025604248, 0.003430132521316409, 0.40285244584083557, 0.19220596551895142] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.400313773884 , 9198748.82 ##### | |
| ##### Sending to neptune: dp_per_s : 0.400313773884 , 89.0196620523 ##### | |
| [u'other', 9198748.82, 89.0196620522503] | |
| receiving | |
| 100%|##########|1000/1000[24:00<00:00, 0.69it/s] | |
| [0130 17:54:02 @stat.py:81] async_global_step: 1000 | |
| [0130 17:54:02 @stat.py:81] learning_rate_1: 0.00015 | |
| EPOCH ENDS HERE | |
| [0130 17:54:02 @timer.py:52] Epoch 1, global_step=1000 finished, time=1440.88sec. | |
| [0130 17:54:02 @multigpu.py:323] [p0115] step: count(985), step_time 1469.01, mean_step_time 1430.82, it/s 0.7 | |
| [0130 17:54:03 @multigpu.py:323] [p0576] step: count(1010), step_time 1390.46, mean_step_time 1427.11, it/s 0.7 | |
| [0130 17:54:03 @multigpu.py:323] [p0574] step: count(1001), step_time 1500.74, mean_step_time 1440.56, it/s 0.69 | |
| [0130 17:54:03 @multigpu.py:323] [p0115] step: count(986), step_time 1467.34, mean_step_time 1434.3, it/s 0.7 | |
| [0130 17:54:04 @multigpu.py:323] [p0576] step: count(1011), step_time 1380.93, mean_step_time 1423.48, it/s 0.7 | |
| [0130 17:54:05 @multigpu.py:323] [p0574] step: count(1002), step_time 1482.45, mean_step_time 1444.04, it/s 0.69 | |
| [0130 17:54:05 @multigpu.py:323] [p0115] step: count(987), step_time 1484.52, mean_step_time 1435.89, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.40136244052 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:54:06 @multigpu.py:323] [p0115] step: count(988), step_time 1423.16, mean_step_time 1437.15, it/s 0.7 | |
| [0130 17:54:06 @multigpu.py:323] [p0574] step: count(1003), step_time 1449.53, mean_step_time 1437.14, it/s 0.7 | |
| [0130 17:54:06 @multigpu.py:323] [p0576] step: count(1012), step_time 1688.2, mean_step_time 1437.15, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.40154918472 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:54:08 @multigpu.py:323] [p0574] step: count(1004), step_time 1386.11, mean_step_time 1434.52, it/s 0.7 | |
| [0130 17:54:08 @multigpu.py:323] [p0115] step: count(989), step_time 1424.58, mean_step_time 1436.75, it/s 0.7 | |
| [0130 17:54:08 @multigpu.py:323] [p0576] step: count(1013), step_time 1437.21, mean_step_time 1432.75, it/s 0.7 | |
| [0130 17:54:09 @multigpu.py:323] [p0576] step: count(1014), step_time 1386.41, mean_step_time 1429.98, it/s 0.7 | |
| [0130 17:54:09 @multigpu.py:323] [p0574] step: count(1005), step_time 1472.24, mean_step_time 1435.25, it/s 0.7 | |
| [0130 17:54:09 @multigpu.py:323] [p0115] step: count(990), step_time 1449.78, mean_step_time 1438.83, it/s 0.7 | |
| [0130 17:54:10 @multigpu.py:323] [p0574] step: count(1006), step_time 1416.46, mean_step_time 1435.99, it/s 0.7 | |
| [0130 17:54:10 @multigpu.py:323] [p0576] step: count(1015), step_time 1453.32, mean_step_time 1431.6, it/s 0.7 | |
| [0130 17:54:10 @multigpu.py:323] [p0115] step: count(991), step_time 1406.55, mean_step_time 1438.43, it/s 0.7 | |
| [0130 17:54:12 @multigpu.py:323] [p0576] step: count(1016), step_time 1376.92, mean_step_time 1429.83, it/s 0.7 | |
| [0130 17:54:12 @multigpu.py:323] [p0574] step: count(1007), step_time 1404.55, mean_step_time 1435.88, it/s 0.7 | |
| [0130 17:54:12 @multigpu.py:323] [p0115] step: count(992), step_time 1434.62, mean_step_time 1439.12, it/s 0.69 | |
| [0130 17:54:13 @multigpu.py:323] [p0574] step: count(1008), step_time 1392.2, mean_step_time 1430.21, it/s 0.7 | |
| [0130 17:54:13 @multigpu.py:323] [p0115] step: count(993), step_time 1443.97, mean_step_time 1437.53, it/s 0.7 | |
| [0130 17:54:13 @multigpu.py:323] [p0576] step: count(1017), step_time 1511.07, mean_step_time 1433.85, it/s 0.7 | |
| [0130 17:54:15 @multigpu.py:323] [p0574] step: count(1009), step_time 1434.25, mean_step_time 1429.41, it/s 0.7 | |
| [0130 17:54:15 @multigpu.py:323] [p0115] step: count(994), step_time 1373.23, mean_step_time 1433.8, it/s 0.7 | |
| [0130 17:54:15 @multigpu.py:323] [p0576] step: count(1018), step_time 1411.65, mean_step_time 1431.45, it/s 0.7 | |
| [0130 17:54:16 @multigpu.py:323] [p0574] step: count(1010), step_time 1381.34, mean_step_time 1428.31, it/s 0.7 | |
| [0130 17:54:16 @multigpu.py:323] [p0115] step: count(995), step_time 1433.91, mean_step_time 1434.78, it/s 0.7 | |
| [0130 17:54:16 @multigpu.py:323] [p0576] step: count(1019), step_time 1424.85, mean_step_time 1431.54, it/s 0.7 | |
| [0130 17:54:17 @multigpu.py:323] [p0574] step: count(1011), step_time 1408.96, mean_step_time 1424.1, it/s 0.7 | |
| [0130 17:54:18 @multigpu.py:323] [p0115] step: count(996), step_time 1402.86, mean_step_time 1433.23, it/s 0.7 | |
| [0130 17:54:18 @multigpu.py:323] [p0576] step: count(1020), step_time 1480.01, mean_step_time 1435.4, it/s 0.7 | |
| [0130 17:54:19 @multigpu.py:323] [p0574] step: count(1012), step_time 1432.06, mean_step_time 1424.57, it/s 0.7 | |
| [0130 17:54:19 @multigpu.py:323] [p0115] step: count(997), step_time 1412.77, mean_step_time 1431.98, it/s 0.7 | |
| [0130 17:54:19 @multigpu.py:323] [p0576] step: count(1021), step_time 1444.97, mean_step_time 1433.97, it/s 0.7 | |
| [0130 17:54:20 @multigpu.py:323] [p0574] step: count(1013), step_time 1448.9, mean_step_time 1426.13, it/s 0.7 | |
| [0130 17:54:20 @multigpu.py:323] [p0115] step: count(998), step_time 1395.76, mean_step_time 1428.81, it/s 0.7 | |
| [0130 17:54:21 @multigpu.py:323] [p0576] step: count(1022), step_time 1410.01, mean_step_time 1435.51, it/s 0.7 | |
| [0130 17:54:22 @multigpu.py:323] [p0574] step: count(1014), step_time 1413.5, mean_step_time 1424.54, it/s 0.7 | |
| [0130 17:54:22 @multigpu.py:323] [p0115] step: count(999), step_time 1438.66, mean_step_time 1429.68, it/s 0.7 | |
| [0130 17:54:22 @multigpu.py:323] [p0576] step: count(1023), step_time 1382.26, mean_step_time 1434.16, it/s 0.7 | |
| [0130 17:54:23 @multigpu.py:323] [p0574] step: count(1015), step_time 1425.44, mean_step_time 1425.45, it/s 0.7 | |
| [0130 17:54:23 @multigpu.py:323] [p0115] step: count(1000), step_time 1496.72, mean_step_time 1432.59, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.406299769415 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.406299769415 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.406299769415 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.406300278306 , -0.00499816797674 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.406300278306 , 0.282140910625 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.406300278306 , -2.2886068821 ##### | |
| ##### Sending to neptune: value_loss : 0.406300278306 , 1.36670041084 ##### | |
| ##### Sending to neptune: advantage : 0.406300278306 , -0.00127228489146 ##### | |
| ##### Sending to neptune: pred_reward : 0.406300278306 , 0.407911151648 ##### | |
| ##### Sending to neptune: max_logit : 0.406300278306 , 0.192866250873 ##### | |
| [u'loss', -0.004998167976737022, 0.28214091062545776, -2.288606882095337, 1.3667004108428955, -0.0012722848914563656, 0.40791115164756775, 0.192866250872612] | |
| 100%|##########|1000/1000[23:53<00:00, 0.70it/s] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.406300744679 , 9248289.22 ##### | |
| ##### Sending to neptune: dp_per_s : 0.406300744679 , 89.0294158633 ##### | |
| [u'other', 9248289.22, 89.02941586332658] | |
| receiving | |
| [0130 17:54:23 @stat.py:81] async_global_step: 1000 | |
| [0130 17:54:23 @stat.py:81] learning_rate_1: 0.00015 | |
| EPOCH ENDS HERE | |
| [0130 17:54:23 @timer.py:52] Epoch 1, global_step=1000 finished, time=1433.06sec. | |
| [0130 17:54:23 @multigpu.py:323] [p0576] step: count(1024), step_time 1482.16, mean_step_time 1437.24, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.406547241608 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:54:25 @multigpu.py:323] [p0574] step: count(1016), step_time 1396.65, mean_step_time 1423.61, it/s 0.7 | |
| [0130 17:54:25 @multigpu.py:323] [p0115] step: count(1001), step_time 1433.23, mean_step_time 1431.37, it/s 0.7 | |
| [0130 17:54:25 @multigpu.py:323] [p0576] step: count(1025), step_time 1436.77, mean_step_time 1438.01, it/s 0.7 | |
| [0130 17:54:26 @multigpu.py:323] [p0574] step: count(1017), step_time 1386.37, mean_step_time 1422.22, it/s 0.7 | |
| [0130 17:54:26 @multigpu.py:323] [p0115] step: count(1002), step_time 1394.99, mean_step_time 1429.23, it/s 0.7 | |
| [0130 17:54:26 @multigpu.py:323] [p0576] step: count(1026), step_time 1390.23, mean_step_time 1434.15, it/s 0.7 | |
| [0130 17:54:27 @multigpu.py:323] [p0574] step: count(1018), step_time 1411.35, mean_step_time 1422.38, it/s 0.7 | |
| [0130 17:54:28 @multigpu.py:323] [p0115] step: count(1003), step_time 1436.11, mean_step_time 1431.21, it/s 0.7 | |
| [0130 17:54:28 @multigpu.py:323] [p0576] step: count(1027), step_time 1417.47, mean_step_time 1435.8, it/s 0.7 | |
| [0130 17:54:29 @multigpu.py:323] [p0574] step: count(1019), step_time 1413.31, mean_step_time 1422.7, it/s 0.7 | |
| [0130 17:54:29 @multigpu.py:323] [p0115] step: count(1004), step_time 1418.69, mean_step_time 1432.02, it/s 0.7 | |
| [0130 17:54:29 @multigpu.py:323] [p0576] step: count(1028), step_time 1393.45, mean_step_time 1436.49, it/s 0.7 | |
| [0130 17:54:30 @multigpu.py:323] [p0574] step: count(1020), step_time 1430.03, mean_step_time 1424.32, it/s 0.7 | |
| [0130 17:54:30 @multigpu.py:323] [p0115] step: count(1005), step_time 1461.79, mean_step_time 1431.66, it/s 0.7 | |
| [0130 17:54:30 @multigpu.py:323] [p0576] step: count(1029), step_time 1430.71, mean_step_time 1436.45, it/s 0.7 | |
| [0130 17:54:32 @multigpu.py:323] [p0574] step: count(1021), step_time 1470.66, mean_step_time 1422.82, it/s 0.7 | |
| [0130 17:54:32 @multigpu.py:323] [p0115] step: count(1006), step_time 1423.65, mean_step_time 1429.48, it/s 0.7 | |
| [0130 17:54:32 @multigpu.py:323] [p0576] step: count(1030), step_time 1439.9, mean_step_time 1438.93, it/s 0.69 | |
| [0130 17:54:33 @multigpu.py:323] [p0574] step: count(1022), step_time 1408.78, mean_step_time 1419.13, it/s 0.7 | |
| [0130 17:54:33 @multigpu.py:323] [p0115] step: count(1007), step_time 1421.95, mean_step_time 1426.35, it/s 0.7 | |
| [0130 17:54:33 @multigpu.py:323] [p0576] step: count(1031), step_time 1445.41, mean_step_time 1442.15, it/s 0.69 | |
| [0130 17:54:35 @multigpu.py:323] [p0576] step: count(1032), step_time 1395.33, mean_step_time 1427.51, it/s 0.7 | |
| [0130 17:54:35 @multigpu.py:323] [p0574] step: count(1023), step_time 1614.89, mean_step_time 1427.4, it/s 0.7 | |
| [0130 17:54:35 @multigpu.py:323] [p0115] step: count(1008), step_time 1446.89, mean_step_time 1427.54, it/s 0.7 | |
| [0130 17:54:36 @multigpu.py:323] [p0574] step: count(1024), step_time 1390.41, mean_step_time 1427.62, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.409867774116 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:54:36 @multigpu.py:323] [p0576] step: count(1033), step_time 1416.02, mean_step_time 1426.45, it/s 0.7 | |
| [0130 17:54:36 @multigpu.py:323] [p0115] step: count(1009), step_time 1435.11, mean_step_time 1428.06, it/s 0.7 | |
| [0130 17:54:38 @multigpu.py:323] [p0115] step: count(1010), step_time 1396.53, mean_step_time 1425.4, it/s 0.7 | |
| [0130 17:54:38 @multigpu.py:323] [p0574] step: count(1025), step_time 1454.39, mean_step_time 1426.72, it/s 0.7 | |
| [0130 17:54:38 @multigpu.py:323] [p0576] step: count(1034), step_time 1454.06, mean_step_time 1429.83, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.410494857497 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:54:39 @multigpu.py:323] [p0115] step: count(1011), step_time 1457.56, mean_step_time 1427.95, it/s 0.7 | |
| [0130 17:54:39 @multigpu.py:323] [p0574] step: count(1026), step_time 1464.19, mean_step_time 1429.11, it/s 0.7 | |
| [0130 17:54:39 @multigpu.py:323] [p0576] step: count(1035), step_time 1483.78, mean_step_time 1431.35, it/s 0.7 | |
| [0130 17:54:40 @multigpu.py:323] [p0115] step: count(1012), step_time 1456.61, mean_step_time 1429.05, it/s 0.7 | |
| [0130 17:54:40 @multigpu.py:323] [p0574] step: count(1027), step_time 1441.26, mean_step_time 1430.95, it/s 0.7 | |
| [0130 17:54:41 @multigpu.py:323] [p0576] step: count(1036), step_time 1423.45, mean_step_time 1433.68, it/s 0.7 | |
| [0130 17:54:42 @multigpu.py:323] [p0574] step: count(1028), step_time 1432.02, mean_step_time 1432.94, it/s 0.7 | |
| [0130 17:54:42 @multigpu.py:323] [p0115] step: count(1013), step_time 1439.95, mean_step_time 1428.85, it/s 0.7 | |
| [0130 17:54:42 @multigpu.py:323] [p0576] step: count(1037), step_time 1412.17, mean_step_time 1428.73, it/s 0.7 | |
| [0130 17:54:43 @multigpu.py:323] [p0115] step: count(1014), step_time 1422.85, mean_step_time 1431.33, it/s 0.7 | |
| [0130 17:54:43 @multigpu.py:323] [p0576] step: count(1038), step_time 1440.15, mean_step_time 1430.16, it/s 0.7 | |
| [0130 17:54:43 @multigpu.py:323] [p0574] step: count(1029), step_time 1458.63, mean_step_time 1434.16, it/s 0.7 | |
| [0130 17:54:45 @multigpu.py:323] [p0576] step: count(1039), step_time 1383.62, mean_step_time 1428.1, it/s 0.7 | |
| [0130 17:54:45 @multigpu.py:323] [p0115] step: count(1015), step_time 1440.15, mean_step_time 1431.64, it/s 0.7 | |
| [0130 17:54:45 @multigpu.py:323] [p0574] step: count(1030), step_time 1436.54, mean_step_time 1436.92, it/s 0.7 | |
| [0130 17:54:46 @multigpu.py:323] [p0576] step: count(1040), step_time 1419.95, mean_step_time 1425.09, it/s 0.7 | |
| [0130 17:54:46 @multigpu.py:323] [p0574] step: count(1031), step_time 1453.02, mean_step_time 1439.12, it/s 0.69 | |
| [0130 17:54:46 @multigpu.py:323] [p0115] step: count(1016), step_time 1500.55, mean_step_time 1436.53, it/s 0.7 | |
| [0130 17:54:48 @multigpu.py:323] [p0576] step: count(1041), step_time 1415.08, mean_step_time 1423.6, it/s 0.7 | |
| [0130 17:54:48 @multigpu.py:323] [p0574] step: count(1032), step_time 1400.81, mean_step_time 1437.56, it/s 0.7 | |
| [0130 17:54:48 @multigpu.py:323] [p0115] step: count(1017), step_time 1412.98, mean_step_time 1436.54, it/s 0.7 | |
| [0130 17:54:49 @multigpu.py:323] [p0576] step: count(1042), step_time 1366.53, mean_step_time 1421.42, it/s 0.7 | |
| [0130 17:54:49 @multigpu.py:323] [p0574] step: count(1033), step_time 1453.49, mean_step_time 1437.79, it/s 0.7 | |
| [0130 17:54:49 @multigpu.py:323] [p0115] step: count(1018), step_time 1435.46, mean_step_time 1438.52, it/s 0.7 | |
| [0130 17:54:50 @multigpu.py:323] [p0576] step: count(1043), step_time 1474.24, mean_step_time 1426.02, it/s 0.7 | |
| [0130 17:54:51 @multigpu.py:323] [p0574] step: count(1034), step_time 1398.87, mean_step_time 1437.06, it/s 0.7 | |
| [0130 17:54:51 @multigpu.py:323] [p0115] step: count(1019), step_time 1426.7, mean_step_time 1437.92, it/s 0.7 | |
| [0130 17:54:52 @multigpu.py:323] [p0576] step: count(1044), step_time 1480.67, mean_step_time 1425.95, it/s 0.7 | |
| [0130 17:54:52 @multigpu.py:323] [p0574] step: count(1035), step_time 1418.28, mean_step_time 1436.7, it/s 0.7 | |
| [0130 17:54:52 @multigpu.py:323] [p0115] step: count(1020), step_time 1414.59, mean_step_time 1433.82, it/s 0.7 | |
| [0130 17:54:53 @multigpu.py:323] [p0574] step: count(1036), step_time 1373.41, mean_step_time 1435.54, it/s 0.7 | |
| [0130 17:54:53 @multigpu.py:323] [p0576] step: count(1045), step_time 1500.63, mean_step_time 1429.14, it/s 0.7 | |
| [0130 17:54:53 @multigpu.py:323] [p0115] step: count(1021), step_time 1447.12, mean_step_time 1434.51, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.414848094715 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:54:55 @multigpu.py:323] [p0574] step: count(1037), step_time 1446.01, mean_step_time 1438.52, it/s 0.7 | |
| [0130 17:54:55 @multigpu.py:323] [p0576] step: count(1046), step_time 1383.97, mean_step_time 1428.83, it/s 0.7 | |
| [0130 17:54:55 @multigpu.py:323] [p0115] step: count(1022), step_time 1451.57, mean_step_time 1437.34, it/s 0.7 | |
| [0130 17:54:56 @multigpu.py:323] [p0574] step: count(1038), step_time 1386.57, mean_step_time 1437.28, it/s 0.7 | |
| [0130 17:54:56 @multigpu.py:323] [p0576] step: count(1047), step_time 1478.74, mean_step_time 1431.89, it/s 0.7 | |
| [0130 17:54:56 @multigpu.py:323] [p0115] step: count(1023), step_time 1435.33, mean_step_time 1437.3, it/s 0.7 | |
| [0130 17:54:58 @multigpu.py:323] [p0574] step: count(1039), step_time 1427.69, mean_step_time 1438.0, it/s 0.7 | |
| [0130 17:54:58 @multigpu.py:323] [p0576] step: count(1048), step_time 1418.64, mean_step_time 1433.15, it/s 0.7 | |
| [0130 17:54:58 @multigpu.py:323] [p0115] step: count(1024), step_time 1436.89, mean_step_time 1438.21, it/s 0.7 | |
| [0130 17:54:59 @multigpu.py:323] [p0574] step: count(1040), step_time 1395.55, mean_step_time 1436.27, it/s 0.7 | |
| [0130 17:54:59 @multigpu.py:323] [p0115] step: count(1025), step_time 1415.06, mean_step_time 1435.87, it/s 0.7 | |
| [0130 17:54:59 @multigpu.py:323] [p0576] step: count(1049), step_time 1498.11, mean_step_time 1436.52, it/s 0.7 | |
| [0130 17:55:00 @multigpu.py:323] [p0574] step: count(1041), step_time 1429.55, mean_step_time 1434.22, it/s 0.7 | |
| [0130 17:55:01 @multigpu.py:323] [p0576] step: count(1050), step_time 1387.85, mean_step_time 1433.92, it/s 0.7 | |
| [0130 17:55:01 @multigpu.py:323] [p0115] step: count(1026), step_time 1455.92, mean_step_time 1437.49, it/s 0.7 | |
| [0130 17:55:02 @multigpu.py:323] [p0574] step: count(1042), step_time 1404.44, mean_step_time 1434.0, it/s 0.7 | |
| [0130 17:55:02 @multigpu.py:323] [p0576] step: count(1051), step_time 1454.33, mean_step_time 1434.37, it/s 0.7 | |
| [0130 17:55:02 @multigpu.py:323] [p0115] step: count(1027), step_time 1447.54, mean_step_time 1438.77, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.417075758311 , 1.6 ##### | |
| [u'online', 1.6] | |
| ##### Sending to neptune: active_workers : 0.41707579467 , 3 ##### | |
| receiving | |
| [0130 17:55:04 @multigpu.py:323] [p0115] step: count(1028), step_time 1484.34, mean_step_time 1440.64, it/s 0.69 | |
| [0130 17:55:04 @multigpu.py:323] [p0576] step: count(1052), step_time 1519.52, mean_step_time 1440.58, it/s 0.69 | |
| [0130 17:55:04 @multigpu.py:323] [p0574] step: count(1043), step_time 1747.43, mean_step_time 1440.63, it/s 0.69 | |
| [0130 17:55:05 @multigpu.py:323] [p0576] step: count(1053), step_time 1418.04, mean_step_time 1440.68, it/s 0.69 | |
| [0130 17:55:05 @multigpu.py:323] [p0574] step: count(1044), step_time 1417.8, mean_step_time 1442.0, it/s 0.69 | |
| [0130 17:55:05 @multigpu.py:323] [p0115] step: count(1029), step_time 1472.79, mean_step_time 1442.52, it/s 0.69 | |
| [0130 17:55:06 @multigpu.py:323] [p0574] step: count(1045), step_time 1394.07, mean_step_time 1438.98, it/s 0.69 | |
| [0130 17:55:06 @multigpu.py:323] [p0576] step: count(1054), step_time 1410.82, mean_step_time 1438.52, it/s 0.7 | |
| [0130 17:55:06 @multigpu.py:323] [p0115] step: count(1030), step_time 1420.81, mean_step_time 1443.74, it/s 0.69 | |
| [0130 17:55:08 @multigpu.py:323] [p0574] step: count(1046), step_time 1393.73, mean_step_time 1435.46, it/s 0.7 | |
| [0130 17:55:08 @multigpu.py:323] [p0576] step: count(1055), step_time 1458.42, mean_step_time 1437.25, it/s 0.7 | |
| [0130 17:55:08 @multigpu.py:323] [p0115] step: count(1031), step_time 1485.93, mean_step_time 1445.16, it/s 0.69 | |
| [0130 17:55:09 @multigpu.py:323] [p0574] step: count(1047), step_time 1478.91, mean_step_time 1437.34, it/s 0.7 | |
| [0130 17:55:09 @multigpu.py:323] [p0576] step: count(1056), step_time 1488.08, mean_step_time 1440.48, it/s 0.69 | |
| [0130 17:55:09 @multigpu.py:323] [p0115] step: count(1032), step_time 1414.59, mean_step_time 1443.06, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.419183745517 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:55:11 @multigpu.py:323] [p0574] step: count(1048), step_time 1408.46, mean_step_time 1436.16, it/s 0.7 | |
| [0130 17:55:11 @multigpu.py:323] [p0576] step: count(1057), step_time 1376.61, mean_step_time 1438.7, it/s 0.7 | |
| [0130 17:55:11 @multigpu.py:323] [p0115] step: count(1033), step_time 1422.66, mean_step_time 1442.19, it/s 0.69 | |
| [0130 17:55:12 @multigpu.py:323] [p0574] step: count(1049), step_time 1398.56, mean_step_time 1433.16, it/s 0.7 | |
| [0130 17:55:12 @multigpu.py:323] [p0576] step: count(1058), step_time 1436.46, mean_step_time 1438.52, it/s 0.7 | |
| [0130 17:55:12 @multigpu.py:323] [p0115] step: count(1034), step_time 1444.23, mean_step_time 1443.26, it/s 0.69 | |
| [0130 17:55:13 @multigpu.py:323] [p0574] step: count(1050), step_time 1419.26, mean_step_time 1432.29, it/s 0.7 | |
| [0130 17:55:14 @multigpu.py:323] [p0576] step: count(1059), step_time 1464.07, mean_step_time 1442.54, it/s 0.69 | |
| [0130 17:55:14 @multigpu.py:323] [p0115] step: count(1035), step_time 1432.57, mean_step_time 1442.88, it/s 0.69 | |
| [0130 17:55:15 @multigpu.py:323] [p0574] step: count(1051), step_time 1429.73, mean_step_time 1431.13, it/s 0.7 | |
| [0130 17:55:15 @multigpu.py:323] [p0576] step: count(1060), step_time 1391.39, mean_step_time 1441.11, it/s 0.69 | |
| [0130 17:55:15 @multigpu.py:323] [p0115] step: count(1036), step_time 1421.45, mean_step_time 1438.93, it/s 0.69 | |
| [0130 17:55:16 @multigpu.py:323] [p0574] step: count(1052), step_time 1391.14, mean_step_time 1430.65, it/s 0.7 | |
| [0130 17:55:16 @multigpu.py:323] [p0576] step: count(1061), step_time 1390.34, mean_step_time 1439.87, it/s 0.69 | |
| [0130 17:55:16 @multigpu.py:323] [p0115] step: count(1037), step_time 1424.11, mean_step_time 1439.48, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.421175178554 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:55:18 @multigpu.py:323] [p0574] step: count(1053), step_time 1493.1, mean_step_time 1432.63, it/s 0.7 | |
| [0130 17:55:18 @multigpu.py:323] [p0576] step: count(1062), step_time 1423.26, mean_step_time 1442.71, it/s 0.69 | |
| [0130 17:55:18 @multigpu.py:323] [p0115] step: count(1038), step_time 1529.8, mean_step_time 1444.2, it/s 0.69 | |
| [0130 17:55:19 @multigpu.py:323] [p0574] step: count(1054), step_time 1428.93, mean_step_time 1434.13, it/s 0.7 | |
| [0130 17:55:19 @multigpu.py:323] [p0576] step: count(1063), step_time 1395.72, mean_step_time 1438.78, it/s 0.7 | |
| [0130 17:55:19 @multigpu.py:323] [p0115] step: count(1039), step_time 1404.79, mean_step_time 1443.11, it/s 0.69 | |
| [0130 17:55:21 @multigpu.py:323] [p0576] step: count(1064), step_time 1376.76, mean_step_time 1433.59, it/s 0.7 | |
| [0130 17:55:21 @multigpu.py:323] [p0574] step: count(1055), step_time 1439.54, mean_step_time 1435.19, it/s 0.7 | |
| [0130 17:55:21 @multigpu.py:323] [p0115] step: count(1040), step_time 1448.15, mean_step_time 1444.78, it/s 0.69 | |
| [0130 17:55:22 @multigpu.py:323] [p0576] step: count(1065), step_time 1459.01, mean_step_time 1431.51, it/s 0.7 | |
| [0130 17:55:22 @multigpu.py:323] [p0574] step: count(1056), step_time 1463.82, mean_step_time 1439.71, it/s 0.69 | |
| [0130 17:55:22 @multigpu.py:323] [p0115] step: count(1041), step_time 1461.35, mean_step_time 1445.5, it/s 0.69 | |
| [0130 17:55:23 @multigpu.py:323] [p0576] step: count(1066), step_time 1372.05, mean_step_time 1430.91, it/s 0.7 | |
| [0130 17:55:24 @multigpu.py:323] [p0574] step: count(1057), step_time 1457.14, mean_step_time 1440.27, it/s 0.69 | |
| [0130 17:55:24 @multigpu.py:323] [p0115] step: count(1042), step_time 1423.25, mean_step_time 1444.08, it/s 0.69 | |
| [0130 17:55:25 @multigpu.py:323] [p0576] step: count(1067), step_time 1508.45, mean_step_time 1432.4, it/s 0.7 | |
| [0130 17:55:25 @multigpu.py:323] [p0574] step: count(1058), step_time 1400.48, mean_step_time 1440.97, it/s 0.69 | |
| [0130 17:55:25 @multigpu.py:323] [p0115] step: count(1043), step_time 1475.45, mean_step_time 1446.09, it/s 0.69 | |
| [0130 17:55:26 @multigpu.py:323] [p0576] step: count(1068), step_time 1395.59, mean_step_time 1431.24, it/s 0.7 | |
| [0130 17:55:26 @multigpu.py:323] [p0574] step: count(1059), step_time 1448.04, mean_step_time 1441.98, it/s 0.69 | |
| [0130 17:55:27 @multigpu.py:323] [p0115] step: count(1044), step_time 1390.38, mean_step_time 1443.76, it/s 0.69 | |
| [0130 17:55:28 @multigpu.py:323] [p0576] step: count(1069), step_time 1399.62, mean_step_time 1426.32, it/s 0.7 | |
| [0130 17:55:28 @multigpu.py:323] [p0574] step: count(1060), step_time 1422.05, mean_step_time 1443.31, it/s 0.69 | |
| [0130 17:55:28 @multigpu.py:323] [p0115] step: count(1045), step_time 1472.17, mean_step_time 1446.62, it/s 0.69 | |
| [0130 17:55:29 @multigpu.py:323] [p0576] step: count(1070), step_time 1414.84, mean_step_time 1427.67, it/s 0.7 | |
| [0130 17:55:29 @multigpu.py:323] [p0574] step: count(1061), step_time 1457.26, mean_step_time 1444.69, it/s 0.69 | |
| [0130 17:55:30 @multigpu.py:323] [p0115] step: count(1046), step_time 1464.66, mean_step_time 1447.05, it/s 0.69 | |
| [0130 17:55:31 @multigpu.py:323] [p0576] step: count(1071), step_time 1431.19, mean_step_time 1426.51, it/s 0.7 | |
| [0130 17:55:31 @multigpu.py:323] [p0574] step: count(1062), step_time 1417.34, mean_step_time 1445.34, it/s 0.69 | |
| [0130 17:55:31 @multigpu.py:323] [p0115] step: count(1047), step_time 1442.89, mean_step_time 1446.82, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.425126151906 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.42519397027 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:55:32 @multigpu.py:323] [p0574] step: count(1063), step_time 1743.95, mean_step_time 1445.16, it/s 0.69 | |
| [0130 17:55:32 @multigpu.py:323] [p0576] step: count(1072), step_time 1891.85, mean_step_time 1445.13, it/s 0.69 | |
| [0130 17:55:32 @multigpu.py:323] [p0115] step: count(1048), step_time 1451.52, mean_step_time 1445.18, it/s 0.69 | |
| [0130 17:55:34 @multigpu.py:323] [p0576] step: count(1073), step_time 1409.99, mean_step_time 1444.73, it/s 0.69 | |
| [0130 17:55:34 @multigpu.py:323] [p0574] step: count(1064), step_time 1415.11, mean_step_time 1445.03, it/s 0.69 | |
| [0130 17:55:34 @multigpu.py:323] [p0115] step: count(1049), step_time 1455.97, mean_step_time 1444.34, it/s 0.69 | |
| [0130 17:55:35 @multigpu.py:323] [p0576] step: count(1074), step_time 1400.35, mean_step_time 1444.2, it/s 0.69 | |
| [0130 17:55:35 @multigpu.py:323] [p0574] step: count(1065), step_time 1401.07, mean_step_time 1445.38, it/s 0.69 | |
| [0130 17:55:35 @multigpu.py:323] [p0115] step: count(1050), step_time 1371.89, mean_step_time 1441.89, it/s 0.69 | |
| [0130 17:55:37 @multigpu.py:323] [p0574] step: count(1066), step_time 1459.04, mean_step_time 1448.65, it/s 0.69 | |
| [0130 17:55:37 @multigpu.py:323] [p0115] step: count(1051), step_time 1458.72, mean_step_time 1440.53, it/s 0.69 | |
| [0130 17:55:37 @multigpu.py:323] [p0576] step: count(1075), step_time 1482.75, mean_step_time 1445.42, it/s 0.69 | |
| [0130 17:55:38 @multigpu.py:323] [p0574] step: count(1067), step_time 1401.78, mean_step_time 1444.79, it/s 0.69 | |
| [0130 17:55:38 @multigpu.py:323] [p0115] step: count(1052), step_time 1408.94, mean_step_time 1440.25, it/s 0.69 | |
| [0130 17:55:38 @multigpu.py:323] [p0576] step: count(1076), step_time 1422.88, mean_step_time 1442.16, it/s 0.69 | |
| [0130 17:55:40 @multigpu.py:323] [p0576] step: count(1077), step_time 1375.78, mean_step_time 1442.12, it/s 0.69 | |
| [0130 17:55:40 @multigpu.py:323] [p0574] step: count(1068), step_time 1455.3, mean_step_time 1447.13, it/s 0.69 | |
| [0130 17:55:40 @multigpu.py:323] [p0115] step: count(1053), step_time 1468.53, mean_step_time 1442.54, it/s 0.69 | |
| [0130 17:55:41 @multigpu.py:323] [p0574] step: count(1069), step_time 1394.51, mean_step_time 1446.93, it/s 0.69 | |
| [0130 17:55:41 @multigpu.py:323] [p0576] step: count(1078), step_time 1477.76, mean_step_time 1444.18, it/s 0.69 | |
| [0130 17:55:41 @multigpu.py:323] [p0115] step: count(1054), step_time 1483.98, mean_step_time 1444.53, it/s 0.69 | |
| [0130 17:55:42 @multigpu.py:323] [p0574] step: count(1070), step_time 1435.69, mean_step_time 1447.75, it/s 0.69 | |
| [0130 17:55:42 @multigpu.py:323] [p0576] step: count(1079), step_time 1457.34, mean_step_time 1443.85, it/s 0.69 | |
| [0130 17:55:43 @multigpu.py:323] [p0115] step: count(1055), step_time 1441.54, mean_step_time 1444.98, it/s 0.69 | |
| [0130 17:55:44 @multigpu.py:323] [p0574] step: count(1071), step_time 1401.82, mean_step_time 1446.35, it/s 0.69 | |
| [0130 17:55:44 @multigpu.py:323] [p0576] step: count(1080), step_time 1395.95, mean_step_time 1444.07, it/s 0.69 | |
| [0130 17:55:44 @multigpu.py:323] [p0115] step: count(1056), step_time 1444.04, mean_step_time 1446.11, it/s 0.69 | |
| [0130 17:55:45 @multigpu.py:323] [p0574] step: count(1072), step_time 1388.22, mean_step_time 1446.21, it/s 0.69 | |
| [0130 17:55:45 @multigpu.py:323] [p0576] step: count(1081), step_time 1439.41, mean_step_time 1446.53, it/s 0.69 | |
| [0130 17:55:45 @multigpu.py:323] [p0115] step: count(1057), step_time 1424.31, mean_step_time 1446.12, it/s 0.69 | |
| [0130 17:55:47 @multigpu.py:323] [p0574] step: count(1073), step_time 1442.95, mean_step_time 1443.7, it/s 0.69 | |
| [0130 17:55:47 @multigpu.py:323] [p0576] step: count(1082), step_time 1515.43, mean_step_time 1451.14, it/s 0.69 | |
| [0130 17:55:47 @multigpu.py:323] [p0115] step: count(1058), step_time 1443.79, mean_step_time 1441.82, it/s 0.69 | |
| [0130 17:55:48 @multigpu.py:323] [p0574] step: count(1074), step_time 1441.68, mean_step_time 1444.34, it/s 0.69 | |
| [0130 17:55:48 @multigpu.py:323] [p0576] step: count(1083), step_time 1382.39, mean_step_time 1450.47, it/s 0.69 | |
| [0130 17:55:48 @multigpu.py:323] [p0115] step: count(1059), step_time 1439.56, mean_step_time 1443.56, it/s 0.69 | |
| [0130 17:55:49 @multigpu.py:323] [p0574] step: count(1075), step_time 1374.67, mean_step_time 1441.1, it/s 0.69 | |
| [0130 17:55:50 @multigpu.py:323] [p0576] step: count(1084), step_time 1449.83, mean_step_time 1454.12, it/s 0.69 | |
| [0130 17:55:50 @multigpu.py:323] [p0115] step: count(1060), step_time 1461.11, mean_step_time 1444.2, it/s 0.69 | |
| [0130 17:55:51 @multigpu.py:323] [p0574] step: count(1076), step_time 1471.64, mean_step_time 1441.49, it/s 0.69 | |
| [0130 17:55:51 @multigpu.py:323] [p0576] step: count(1085), step_time 1367.67, mean_step_time 1449.56, it/s 0.69 | |
| [0130 17:55:51 @multigpu.py:323] [p0115] step: count(1061), step_time 1464.32, mean_step_time 1444.35, it/s 0.69 | |
| [0130 17:55:52 @multigpu.py:323] [p0574] step: count(1077), step_time 1408.18, mean_step_time 1439.04, it/s 0.69 | |
| [0130 17:55:53 @multigpu.py:323] [p0576] step: count(1086), step_time 1499.73, mean_step_time 1455.94, it/s 0.69 | |
| [0130 17:55:53 @multigpu.py:323] [p0115] step: count(1062), step_time 1418.45, mean_step_time 1444.11, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.431302230491 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:55:54 @multigpu.py:323] [p0574] step: count(1078), step_time 1357.14, mean_step_time 1436.87, it/s 0.7 | |
| [0130 17:55:54 @multigpu.py:323] [p0576] step: count(1087), step_time 1445.39, mean_step_time 1452.79, it/s 0.69 | |
| [0130 17:55:54 @multigpu.py:323] [p0115] step: count(1063), step_time 1366.45, mean_step_time 1438.66, it/s 0.7 | |
| [0130 17:55:55 @multigpu.py:323] [p0574] step: count(1079), step_time 1467.04, mean_step_time 1437.82, it/s 0.7 | |
| [0130 17:55:55 @multigpu.py:323] [p0576] step: count(1088), step_time 1401.09, mean_step_time 1453.06, it/s 0.69 | |
| [0130 17:55:55 @multigpu.py:323] [p0115] step: count(1064), step_time 1437.42, mean_step_time 1441.01, it/s 0.69 | |
| [0130 17:55:57 @multigpu.py:323] [p0574] step: count(1080), step_time 1421.0, mean_step_time 1437.77, it/s 0.7 | |
| [0130 17:55:57 @multigpu.py:323] [p0576] step: count(1089), step_time 1454.31, mean_step_time 1455.8, it/s 0.69 | |
| [0130 17:55:57 @multigpu.py:323] [p0115] step: count(1065), step_time 1415.52, mean_step_time 1438.18, it/s 0.7 | |
| [0130 17:55:58 @multigpu.py:323] [p0574] step: count(1081), step_time 1413.1, mean_step_time 1435.56, it/s 0.7 | |
| [0130 17:55:58 @multigpu.py:323] [p0576] step: count(1090), step_time 1415.9, mean_step_time 1455.85, it/s 0.69 | |
| [0130 17:55:58 @multigpu.py:323] [p0115] step: count(1066), step_time 1426.7, mean_step_time 1436.28, it/s 0.7 | |
| [0130 17:55:59 @multigpu.py:323] [p0574] step: count(1082), step_time 1393.46, mean_step_time 1434.37, it/s 0.7 | |
| [0130 17:56:00 @multigpu.py:323] [p0576] step: count(1091), step_time 1428.45, mean_step_time 1455.71, it/s 0.69 | |
| [0130 17:56:00 @multigpu.py:323] [p0115] step: count(1067), step_time 1429.34, mean_step_time 1435.61, it/s 0.7 | |
| [0130 17:56:01 @multigpu.py:323] [p0115] step: count(1068), step_time 1448.33, mean_step_time 1435.45, it/s 0.7 | |
| [0130 17:56:01 @multigpu.py:323] [p0576] step: count(1092), step_time 1486.72, mean_step_time 1435.46, it/s 0.7 | |
| [0130 17:56:01 @multigpu.py:323] [p0574] step: count(1083), step_time 1768.75, mean_step_time 1435.61, it/s 0.7 | |
| [0130 17:56:03 @multigpu.py:323] [p0576] step: count(1093), step_time 1380.93, mean_step_time 1434.0, it/s 0.7 | |
| [0130 17:56:03 @multigpu.py:323] [p0115] step: count(1069), step_time 1439.15, mean_step_time 1434.6, it/s 0.7 | |
| [0130 17:56:03 @multigpu.py:323] [p0574] step: count(1084), step_time 1507.57, mean_step_time 1440.23, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.433990958863 , 1.7 ##### | |
| [u'online', 1.7] | |
| ##### Sending to neptune: active_workers : 0.433991316888 , 3 ##### | |
| receiving | |
| [0130 17:56:04 @multigpu.py:323] [p0576] step: count(1094), step_time 1471.82, mean_step_time 1437.58, it/s 0.7 | |
| [0130 17:56:04 @multigpu.py:323] [p0115] step: count(1070), step_time 1424.13, mean_step_time 1437.22, it/s 0.7 | |
| [0130 17:56:04 @multigpu.py:323] [p0574] step: count(1085), step_time 1368.88, mean_step_time 1438.62, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.434331390262 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:56:05 @multigpu.py:323] [p0576] step: count(1095), step_time 1405.61, mean_step_time 1433.72, it/s 0.7 | |
| [0130 17:56:05 @multigpu.py:323] [p0115] step: count(1071), step_time 1422.46, mean_step_time 1435.4, it/s 0.7 | |
| [0130 17:56:05 @multigpu.py:323] [p0574] step: count(1086), step_time 1438.94, mean_step_time 1437.62, it/s 0.7 | |
| [0130 17:56:07 @multigpu.py:323] [p0115] step: count(1072), step_time 1412.86, mean_step_time 1435.6, it/s 0.7 | |
| [0130 17:56:07 @multigpu.py:323] [p0576] step: count(1096), step_time 1456.46, mean_step_time 1435.4, it/s 0.7 | |
| [0130 17:56:07 @multigpu.py:323] [p0574] step: count(1087), step_time 1446.37, mean_step_time 1439.85, it/s 0.69 | |
| [0130 17:56:08 @multigpu.py:323] [p0576] step: count(1097), step_time 1388.13, mean_step_time 1436.02, it/s 0.7 | |
| [0130 17:56:08 @multigpu.py:323] [p0115] step: count(1073), step_time 1441.22, mean_step_time 1434.23, it/s 0.7 | |
| [0130 17:56:08 @multigpu.py:323] [p0574] step: count(1088), step_time 1420.67, mean_step_time 1438.11, it/s 0.7 | |
| [0130 17:56:10 @multigpu.py:323] [p0576] step: count(1098), step_time 1390.21, mean_step_time 1431.64, it/s 0.7 | |
| [0130 17:56:10 @multigpu.py:323] [p0115] step: count(1074), step_time 1403.55, mean_step_time 1430.21, it/s 0.7 | |
| [0130 17:56:10 @multigpu.py:323] [p0574] step: count(1089), step_time 1454.05, mean_step_time 1441.09, it/s 0.69 | |
| [0130 17:56:11 @multigpu.py:323] [p0576] step: count(1099), step_time 1401.55, mean_step_time 1428.85, it/s 0.7 | |
| [0130 17:56:11 @multigpu.py:323] [p0115] step: count(1075), step_time 1449.39, mean_step_time 1430.6, it/s 0.7 | |
| [0130 17:56:11 @multigpu.py:323] [p0574] step: count(1090), step_time 1442.23, mean_step_time 1441.42, it/s 0.69 | |
| [0130 17:56:13 @multigpu.py:323] [p0576] step: count(1100), step_time 1444.12, mean_step_time 1431.26, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.436645072765 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.436645072765 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.436645072765 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.436646228035 , -0.0118187069893 ##### | |
| ##### Sending to neptune: policy_loss : 0.436646228035 , -0.450571477413 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.436646228035 , -2.28830933571 ##### | |
| ##### Sending to neptune: value_loss : 0.436646228035 , 1.22608649731 ##### | |
| ##### Sending to neptune: advantage : 0.436646228035 , 0.00196923920885 ##### | |
| ##### Sending to neptune: pred_reward : 0.436646228035 , 0.406696081161 ##### | |
| ##### Sending to neptune: max_logit : 0.436646228035 , 0.194596156478 ##### | |
| [u'loss', -0.01181870698928833, -0.4505714774131775, -2.288309335708618, 1.2260864973068237, 0.0019692392088472843, 0.406696081161499, 0.19459615647792816] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.436649463574 , 9426596.31 ##### | |
| ##### Sending to neptune: dp_per_s : 0.436649463574 , 89.1332720899 ##### | |
| [u'other', 9426596.31, 89.13327208992234] | |
| receiving | |
| [0130 17:56:13 @multigpu.py:323] [p0115] step: count(1076), step_time 1434.04, mean_step_time 1430.1, it/s 0.7 | |
| [0130 17:56:13 @multigpu.py:323] [p0574] step: count(1091), step_time 1448.36, mean_step_time 1443.75, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.436772664388 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:56:14 @multigpu.py:323] [p0576] step: count(1101), step_time 1425.92, mean_step_time 1430.58, it/s 0.7 | |
| [0130 17:56:14 @multigpu.py:323] [p0115] step: count(1077), step_time 1443.31, mean_step_time 1431.05, it/s 0.7 | |
| [0130 17:56:14 @multigpu.py:323] [p0574] step: count(1092), step_time 1399.08, mean_step_time 1444.29, it/s 0.69 | |
| [0130 17:56:15 @multigpu.py:323] [p0576] step: count(1102), step_time 1399.16, mean_step_time 1424.77, it/s 0.7 | |
| [0130 17:56:15 @multigpu.py:323] [p0115] step: count(1078), step_time 1415.94, mean_step_time 1429.66, it/s 0.7 | |
| [0130 17:56:16 @multigpu.py:323] [p0574] step: count(1093), step_time 1449.26, mean_step_time 1444.6, it/s 0.69 | |
| [0130 17:56:17 @multigpu.py:323] [p0576] step: count(1103), step_time 1461.29, mean_step_time 1428.71, it/s 0.7 | |
| [0130 17:56:17 @multigpu.py:323] [p0115] step: count(1079), step_time 1432.04, mean_step_time 1429.29, it/s 0.7 | |
| [0130 17:56:17 @multigpu.py:323] [p0574] step: count(1094), step_time 1396.09, mean_step_time 1442.32, it/s 0.69 | |
| [0130 17:56:18 @multigpu.py:323] [p0576] step: count(1104), step_time 1415.22, mean_step_time 1426.98, it/s 0.7 | |
| [0130 17:56:18 @multigpu.py:323] [p0115] step: count(1080), step_time 1432.88, mean_step_time 1427.87, it/s 0.7 | |
| [0130 17:56:18 @multigpu.py:323] [p0574] step: count(1095), step_time 1397.45, mean_step_time 1443.46, it/s 0.69 | |
| [0130 17:56:20 @multigpu.py:323] [p0576] step: count(1105), step_time 1417.93, mean_step_time 1429.5, it/s 0.7 | |
| [0130 17:56:20 @multigpu.py:323] [p0115] step: count(1081), step_time 1441.1, mean_step_time 1426.71, it/s 0.7 | |
| [0130 17:56:20 @multigpu.py:323] [p0574] step: count(1096), step_time 1459.25, mean_step_time 1442.84, it/s 0.69 | |
| [0130 17:56:21 @multigpu.py:323] [p0576] step: count(1106), step_time 1440.26, mean_step_time 1426.52, it/s 0.7 | |
| [0130 17:56:21 @multigpu.py:323] [p0115] step: count(1082), step_time 1419.84, mean_step_time 1426.78, it/s 0.7 | |
| [0130 17:56:21 @multigpu.py:323] [p0574] step: count(1097), step_time 1449.18, mean_step_time 1444.89, it/s 0.69 | |
| [0130 17:56:23 @multigpu.py:323] [p0576] step: count(1107), step_time 1465.27, mean_step_time 1427.52, it/s 0.7 | |
| [0130 17:56:23 @multigpu.py:323] [p0115] step: count(1083), step_time 1423.24, mean_step_time 1429.62, it/s 0.7 | |
| [0130 17:56:23 @multigpu.py:323] [p0574] step: count(1098), step_time 1523.52, mean_step_time 1453.21, it/s 0.69 | |
| [0130 17:56:24 @multigpu.py:323] [p0576] step: count(1108), step_time 1403.19, mean_step_time 1427.62, it/s 0.7 | |
| [0130 17:56:24 @multigpu.py:323] [p0115] step: count(1084), step_time 1477.21, mean_step_time 1431.61, it/s 0.7 | |
| [0130 17:56:24 @multigpu.py:323] [p0574] step: count(1099), step_time 1472.36, mean_step_time 1453.48, it/s 0.69 | |
| [0130 17:56:25 @multigpu.py:323] [p0576] step: count(1109), step_time 1415.13, mean_step_time 1425.66, it/s 0.7 | |
| [0130 17:56:25 @multigpu.py:323] [p0115] step: count(1085), step_time 1391.89, mean_step_time 1430.43, it/s 0.7 | |
| [0130 17:56:26 @multigpu.py:323] [p0574] step: count(1100), step_time 1408.9, mean_step_time 1452.87, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.440297738579 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.440297738579 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.440297738579 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.440298264954 , -0.00412045605481 ##### | |
| ##### Sending to neptune: policy_loss : 0.440298264954 , 0.338887602091 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.440298264954 , -2.28836631775 ##### | |
| ##### Sending to neptune: value_loss : 0.440298264954 , 1.42206013203 ##### | |
| ##### Sending to neptune: advantage : 0.440298264954 , -0.00142946292181 ##### | |
| ##### Sending to neptune: pred_reward : 0.440298264954 , 0.410076230764 ##### | |
| ##### Sending to neptune: max_logit : 0.440298264954 , 0.194619998336 ##### | |
| [u'loss', -0.004120456054806709, 0.33888760209083557, -2.2883663177490234, 1.4220601320266724, -0.0014294629218056798, 0.41007623076438904, 0.19461999833583832] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.440298698876 , 9435936.15 ##### | |
| ##### Sending to neptune: dp_per_s : 0.440298698876 , 89.0701042424 ##### | |
| [u'other', 9435936.15, 89.07010424237725] | |
| receiving | |
| [0130 17:56:27 @multigpu.py:323] [p0576] step: count(1110), step_time 1438.01, mean_step_time 1426.77, it/s 0.7 | |
| [0130 17:56:27 @multigpu.py:323] [p0115] step: count(1086), step_time 1432.05, mean_step_time 1430.7, it/s 0.7 | |
| [0130 17:56:27 @multigpu.py:323] [p0574] step: count(1101), step_time 1443.19, mean_step_time 1454.38, it/s 0.69 | |
| [0130 17:56:28 @multigpu.py:323] [p0576] step: count(1111), step_time 1431.44, mean_step_time 1426.92, it/s 0.7 | |
| [0130 17:56:28 @multigpu.py:323] [p0115] step: count(1087), step_time 1435.4, mean_step_time 1431.0, it/s 0.7 | |
| [0130 17:56:29 @multigpu.py:323] [p0574] step: count(1102), step_time 1433.72, mean_step_time 1456.39, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.441290416651 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 17:56:30 @multigpu.py:323] [p0115] step: count(1088), step_time 1701.67, mean_step_time 1443.67, it/s 0.69 | |
| [0130 17:56:30 @multigpu.py:323] [p0574] step: count(1103), step_time 1512.89, mean_step_time 1443.6, it/s 0.69 | |
| [0130 17:56:30 @multigpu.py:323] [p0576] step: count(1112), step_time 1822.69, mean_step_time 1443.72, it/s 0.69 | |
| [0130 17:56:32 @multigpu.py:323] [p0115] step: count(1089), step_time 1452.35, mean_step_time 1444.33, it/s 0.69 | |
| [0130 17:56:32 @multigpu.py:323] [p0576] step: count(1113), step_time 1466.74, mean_step_time 1448.01, it/s 0.69 | |
| [0130 17:56:32 @multigpu.py:323] [p0574] step: count(1104), step_time 1532.74, mean_step_time 1444.86, it/s 0.69 | |
| [0130 17:56:33 @multigpu.py:323] [p0115] step: count(1090), step_time 1384.42, mean_step_time 1442.34, it/s 0.69 | |
| [0130 17:56:33 @multigpu.py:323] [p0576] step: count(1114), step_time 1431.12, mean_step_time 1445.97, it/s 0.69 | |
| [0130 17:56:33 @multigpu.py:323] [p0574] step: count(1105), step_time 1461.45, mean_step_time 1449.49, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.442541402446 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 17:56:34 @multigpu.py:323] [p0115] step: count(1091), step_time 1429.84, mean_step_time 1442.71, it/s 0.69 | |
| [0130 17:56:34 @multigpu.py:323] [p0576] step: count(1115), step_time 1403.25, mean_step_time 1445.86, it/s 0.69 | |
| [0130 17:56:35 @multigpu.py:323] [p0574] step: count(1106), step_time 1487.84, mean_step_time 1451.93, it/s 0.69 | |
| [0130 17:56:36 @multigpu.py:323] [p0115] step: count(1092), step_time 1448.2, mean_step_time 1444.48, it/s 0.69 | |
| [0130 17:56:36 @multigpu.py:323] [p0576] step: count(1116), step_time 1437.0, mean_step_time 1444.88, it/s 0.69 | |
| [0130 17:56:36 @multigpu.py:323] [p0574] step: count(1107), step_time 1386.29, mean_step_time 1448.93, it/s 0.69 | |
| [0130 17:56:37 @multigpu.py:323] [p0115] step: count(1093), step_time 1401.62, mean_step_time 1442.5, it/s 0.69 | |
| [0130 17:56:37 @multigpu.py:323] [p0576] step: count(1117), step_time 1425.33, mean_step_time 1446.74, it/s 0.69 | |
| [0130 17:56:37 @multigpu.py:323] [p0574] step: count(1108), step_time 1515.78, mean_step_time 1453.68, it/s 0.69 | |
| [0130 17:56:39 @multigpu.py:323] [p0115] step: count(1094), step_time 1457.24, mean_step_time 1445.18, it/s 0.69 | |
| [0130 17:56:39 @multigpu.py:323] [p0576] step: count(1118), step_time 1448.3, mean_step_time 1449.65, it/s 0.69 | |
| [0130 17:56:39 @multigpu.py:323] [p0574] step: count(1109), step_time 1421.13, mean_step_time 1452.04, it/s 0.69 | |
| [0130 17:56:40 @multigpu.py:323] [p0115] step: count(1095), step_time 1438.18, mean_step_time 1444.62, it/s 0.69 | |
| [0130 17:56:40 @multigpu.py:323] [p0576] step: count(1119), step_time 1476.4, mean_step_time 1453.39, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.444324848586 , 0.9 ##### | |
| [u'online', 0.9] | |
| receiving | |
| [0130 17:56:40 @multigpu.py:323] [p0574] step: count(1110), step_time 1461.83, mean_step_time 1453.02, it/s 0.69 | |
| [0130 17:56:42 @multigpu.py:323] [p0115] step: count(1096), step_time 1436.98, mean_step_time 1444.77, it/s 0.69 | |
| [0130 17:56:42 @multigpu.py:323] [p0576] step: count(1120), step_time 1495.99, mean_step_time 1455.98, it/s 0.69 | |
| [0130 17:56:42 @multigpu.py:323] [p0574] step: count(1111), step_time 1451.65, mean_step_time 1453.18, it/s 0.69 | |
| [0130 17:56:43 @multigpu.py:323] [p0115] step: count(1097), step_time 1431.01, mean_step_time 1444.16, it/s 0.69 | |
| [0130 17:56:43 @multigpu.py:323] [p0576] step: count(1121), step_time 1425.27, mean_step_time 1455.95, it/s 0.69 | |
| [0130 17:56:43 @multigpu.py:323] [p0574] step: count(1112), step_time 1480.7, mean_step_time 1457.26, it/s 0.69 | |
| [0130 17:56:44 @multigpu.py:323] [p0115] step: count(1098), step_time 1418.93, mean_step_time 1444.31, it/s 0.69 | |
| [0130 17:56:45 @multigpu.py:323] [p0576] step: count(1122), step_time 1429.95, mean_step_time 1457.49, it/s 0.69 | |
| [0130 17:56:45 @multigpu.py:323] [p0574] step: count(1113), step_time 1467.23, mean_step_time 1458.16, it/s 0.69 | |
| [0130 17:56:46 @multigpu.py:323] [p0115] step: count(1099), step_time 1460.5, mean_step_time 1445.73, it/s 0.69 | |
| [0130 17:56:46 @multigpu.py:323] [p0576] step: count(1123), step_time 1399.13, mean_step_time 1454.38, it/s 0.69 | |
| [0130 17:56:46 @multigpu.py:323] [p0574] step: count(1114), step_time 1568.11, mean_step_time 1466.76, it/s 0.68 | |
| [0130 17:56:47 @multigpu.py:323] [p0115] step: count(1100), step_time 1442.36, mean_step_time 1446.2, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.446297637489 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.446297637489 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.446297637489 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.446298163599 , -0.00521783763543 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.446298163599 , 0.202429369092 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.446298163599 , -2.28854560852 ##### | |
| ##### Sending to neptune: value_loss : 0.446298163599 , 1.4182331562 ##### | |
| ##### Sending to neptune: advantage : 0.446298163599 , -0.000883870001417 ##### | |
| ##### Sending to neptune: pred_reward : 0.446298163599 , 0.415664792061 ##### | |
| ##### Sending to neptune: max_logit : 0.446298163599 , 0.193621426821 ##### | |
| [u'loss', -0.005217837635427713, 0.2024293690919876, -2.288545608520508, 1.4182331562042236, -0.0008838700014166534, 0.41566479206085205, 0.193621426820755] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.446298656927 , 9437430.47 ##### | |
| ##### Sending to neptune: dp_per_s : 0.446298656927 , 89.0110297975 ##### | |
| [u'other', 9437430.47, 89.01102979748308] | |
| receiving | |
| [0130 17:56:47 @multigpu.py:323] [p0576] step: count(1124), step_time 1413.07, mean_step_time 1454.27, it/s 0.69 | |
| [0130 17:56:48 @multigpu.py:323] [p0574] step: count(1115), step_time 1509.03, mean_step_time 1472.34, it/s 0.68 | |
| [0130 17:56:49 @multigpu.py:323] [p0115] step: count(1101), step_time 1461.1, mean_step_time 1447.2, it/s 0.69 | |
| [0130 17:56:49 @multigpu.py:323] [p0576] step: count(1125), step_time 1438.87, mean_step_time 1455.32, it/s 0.69 | |
| [0130 17:56:49 @multigpu.py:323] [p0574] step: count(1116), step_time 1484.95, mean_step_time 1473.62, it/s 0.68 | |
| [0130 17:56:50 @multigpu.py:323] [p0576] step: count(1126), step_time 1414.05, mean_step_time 1454.01, it/s 0.69 | |
| [0130 17:56:50 @multigpu.py:323] [p0115] step: count(1102), step_time 1447.27, mean_step_time 1448.57, it/s 0.69 | |
| [0130 17:56:51 @multigpu.py:323] [p0574] step: count(1117), step_time 1576.75, mean_step_time 1480.0, it/s 0.68 | |
| [0130 17:56:52 @multigpu.py:323] [p0115] step: count(1103), step_time 1403.75, mean_step_time 1447.6, it/s 0.69 | |
| [0130 17:56:52 @multigpu.py:323] [p0576] step: count(1127), step_time 1415.42, mean_step_time 1451.52, it/s 0.69 | |
| [0130 17:56:52 @multigpu.py:323] [p0574] step: count(1118), step_time 1448.47, mean_step_time 1476.25, it/s 0.68 | |
| [0130 17:56:53 @multigpu.py:323] [p0115] step: count(1104), step_time 1416.13, mean_step_time 1444.54, it/s 0.69 | |
| [0130 17:56:53 @multigpu.py:323] [p0576] step: count(1128), step_time 1455.43, mean_step_time 1454.13, it/s 0.69 | |
| [0130 17:56:54 @multigpu.py:323] [p0574] step: count(1119), step_time 1405.29, mean_step_time 1472.9, it/s 0.68 | |
| [0130 17:56:54 @multigpu.py:323] [p0115] step: count(1105), step_time 1434.33, mean_step_time 1446.67, it/s 0.69 | |
| [0130 17:56:54 @multigpu.py:323] [p0576] step: count(1129), step_time 1427.13, mean_step_time 1454.73, it/s 0.69 | |
| [0130 17:56:55 @multigpu.py:323] [p0574] step: count(1120), step_time 1456.45, mean_step_time 1475.27, it/s 0.68 | |
| [0130 17:56:56 @multigpu.py:323] [p0115] step: count(1106), step_time 1438.43, mean_step_time 1446.99, it/s 0.69 | |
| [0130 17:56:56 @multigpu.py:323] [p0576] step: count(1130), step_time 1414.84, mean_step_time 1453.57, it/s 0.69 | |
| [0130 17:56:57 @multigpu.py:323] [p0574] step: count(1121), step_time 1412.17, mean_step_time 1473.72, it/s 0.68 | |
| [0130 17:56:57 @multigpu.py:323] [p0115] step: count(1107), step_time 1409.18, mean_step_time 1445.67, it/s 0.69 | |
| [0130 17:56:57 @multigpu.py:323] [p0576] step: count(1131), step_time 1397.04, mean_step_time 1451.85, it/s 0.69 | |
| [0130 17:56:58 @multigpu.py:323] [p0574] step: count(1122), step_time 1509.22, mean_step_time 1477.5, it/s 0.68 | |
| [0130 17:57:00 @multigpu.py:323] [p0574] step: count(1123), step_time 1393.09, mean_step_time 1471.51, it/s 0.68 | |
| [0130 17:57:00 @multigpu.py:323] [p0115] step: count(1108), step_time 2220.94, mean_step_time 1471.64, it/s 0.68 | |
| [0130 17:57:00 @multigpu.py:323] [p0576] step: count(1132), step_time 2216.0, mean_step_time 1471.52, it/s 0.68 | |
| [0130 17:57:01 @multigpu.py:323] [p0115] step: count(1109), step_time 1432.01, mean_step_time 1470.62, it/s 0.68 | |
| [0130 17:57:01 @multigpu.py:323] [p0576] step: count(1133), step_time 1447.93, mean_step_time 1470.58, it/s 0.68 | |
| [0130 17:57:01 @multigpu.py:323] [p0574] step: count(1124), step_time 1464.41, mean_step_time 1468.09, it/s 0.68 | |
| [0130 17:57:02 @multigpu.py:323] [p0574] step: count(1125), step_time 1372.86, mean_step_time 1463.66, it/s 0.68 | |
| [0130 17:57:02 @multigpu.py:323] [p0115] step: count(1110), step_time 1438.77, mean_step_time 1473.34, it/s 0.68 | |
| [0130 17:57:02 @multigpu.py:323] [p0576] step: count(1134), step_time 1478.64, mean_step_time 1472.95, it/s 0.68 | |
| [0130 17:57:04 @multigpu.py:323] [p0115] step: count(1111), step_time 1376.28, mean_step_time 1470.66, it/s 0.68 | |
| [0130 17:57:04 @multigpu.py:323] [p0574] step: count(1126), step_time 1469.52, mean_step_time 1462.75, it/s 0.68 | |
| [0130 17:57:04 @multigpu.py:323] [p0576] step: count(1135), step_time 1400.41, mean_step_time 1472.81, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.451079662442 , 2.5 ##### | |
| [u'online', 2.5] | |
| ##### Sending to neptune: active_workers : 0.451079749399 , 3 ##### | |
| receiving | |
| [0130 17:57:05 @multigpu.py:323] [p0574] step: count(1127), step_time 1377.8, mean_step_time 1462.32, it/s 0.68 | |
| [0130 17:57:05 @multigpu.py:323] [p0115] step: count(1112), step_time 1473.36, mean_step_time 1471.92, it/s 0.68 | |
| [0130 17:57:05 @multigpu.py:323] [p0576] step: count(1136), step_time 1422.99, mean_step_time 1472.11, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.451481641663 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 17:57:07 @multigpu.py:323] [p0574] step: count(1128), step_time 1433.09, mean_step_time 1458.19, it/s 0.69 | |
| [0130 17:57:07 @multigpu.py:323] [p0115] step: count(1113), step_time 1415.25, mean_step_time 1472.6, it/s 0.68 | |
| [0130 17:57:07 @multigpu.py:323] [p0576] step: count(1137), step_time 1427.53, mean_step_time 1472.22, it/s 0.68 | |
| [0130 17:57:08 @multigpu.py:323] [p0574] step: count(1129), step_time 1434.12, mean_step_time 1458.84, it/s 0.69 | |
| [0130 17:57:08 @multigpu.py:323] [p0576] step: count(1138), step_time 1416.3, mean_step_time 1470.62, it/s 0.68 | |
| [0130 17:57:08 @multigpu.py:323] [p0115] step: count(1114), step_time 1460.82, mean_step_time 1472.78, it/s 0.68 | |
| [0130 17:57:09 @multigpu.py:323] [p0574] step: count(1130), step_time 1433.63, mean_step_time 1457.43, it/s 0.69 | |
| [0130 17:57:10 @multigpu.py:323] [p0576] step: count(1139), step_time 1407.76, mean_step_time 1467.19, it/s 0.68 | |
| [0130 17:57:10 @multigpu.py:323] [p0115] step: count(1115), step_time 1432.65, mean_step_time 1472.5, it/s 0.68 | |
| [0130 17:57:11 @multigpu.py:323] [p0574] step: count(1131), step_time 1405.0, mean_step_time 1455.09, it/s 0.69 | |
| [0130 17:57:11 @multigpu.py:323] [p0576] step: count(1140), step_time 1421.67, mean_step_time 1463.47, it/s 0.68 | |
| [0130 17:57:11 @multigpu.py:323] [p0115] step: count(1116), step_time 1431.55, mean_step_time 1472.23, it/s 0.68 | |
| [0130 17:57:12 @multigpu.py:323] [p0574] step: count(1132), step_time 1475.83, mean_step_time 1454.85, it/s 0.69 | |
| [0130 17:57:12 @multigpu.py:323] [p0576] step: count(1141), step_time 1445.96, mean_step_time 1464.51, it/s 0.68 | |
| [0130 17:57:12 @multigpu.py:323] [p0115] step: count(1117), step_time 1491.65, mean_step_time 1475.26, it/s 0.68 | |
| [0130 17:57:14 @multigpu.py:323] [p0574] step: count(1133), step_time 1395.66, mean_step_time 1451.27, it/s 0.69 | |
| [0130 17:57:14 @multigpu.py:323] [p0576] step: count(1142), step_time 1430.69, mean_step_time 1464.54, it/s 0.68 | |
| [0130 17:57:14 @multigpu.py:323] [p0115] step: count(1118), step_time 1422.91, mean_step_time 1475.46, it/s 0.68 | |
| [0130 17:57:15 @multigpu.py:323] [p0576] step: count(1143), step_time 1401.77, mean_step_time 1464.68, it/s 0.68 | |
| [0130 17:57:15 @multigpu.py:323] [p0574] step: count(1134), step_time 1496.56, mean_step_time 1447.69, it/s 0.69 | |
| [0130 17:57:15 @multigpu.py:323] [p0115] step: count(1119), step_time 1423.35, mean_step_time 1473.6, it/s 0.68 | |
| [0130 17:57:17 @multigpu.py:323] [p0576] step: count(1144), step_time 1392.55, mean_step_time 1463.65, it/s 0.68 | |
| [0130 17:57:17 @multigpu.py:323] [p0574] step: count(1135), step_time 1422.67, mean_step_time 1443.38, it/s 0.69 | |
| [0130 17:57:17 @multigpu.py:323] [p0115] step: count(1120), step_time 1431.39, mean_step_time 1473.06, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.454599559704 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:57:18 @multigpu.py:323] [p0576] step: count(1145), step_time 1391.93, mean_step_time 1461.3, it/s 0.68 | |
| [0130 17:57:18 @multigpu.py:323] [p0115] step: count(1121), step_time 1430.98, mean_step_time 1471.55, it/s 0.68 | |
| [0130 17:57:18 @multigpu.py:323] [p0574] step: count(1136), step_time 1487.91, mean_step_time 1443.52, it/s 0.69 | |
| [0130 17:57:20 @multigpu.py:323] [p0576] step: count(1146), step_time 1514.51, mean_step_time 1466.33, it/s 0.68 | |
| [0130 17:57:20 @multigpu.py:323] [p0574] step: count(1137), step_time 1420.15, mean_step_time 1435.69, it/s 0.7 | |
| [0130 17:57:20 @multigpu.py:323] [p0115] step: count(1122), step_time 1458.52, mean_step_time 1472.11, it/s 0.68 | |
| [0130 17:57:21 @multigpu.py:323] [p0576] step: count(1147), step_time 1380.34, mean_step_time 1464.57, it/s 0.68 | |
| [0130 17:57:21 @multigpu.py:323] [p0574] step: count(1138), step_time 1355.25, mean_step_time 1431.03, it/s 0.7 | |
| [0130 17:57:21 @multigpu.py:323] [p0115] step: count(1123), step_time 1428.35, mean_step_time 1473.34, it/s 0.68 | |
| [0130 17:57:22 @multigpu.py:323] [p0576] step: count(1148), step_time 1437.57, mean_step_time 1463.68, it/s 0.68 | |
| [0130 17:57:22 @multigpu.py:323] [p0574] step: count(1139), step_time 1437.97, mean_step_time 1432.67, it/s 0.7 | |
| [0130 17:57:22 @multigpu.py:323] [p0115] step: count(1124), step_time 1431.56, mean_step_time 1474.11, it/s 0.68 | |
| [0130 17:57:24 @multigpu.py:323] [p0576] step: count(1149), step_time 1432.13, mean_step_time 1463.93, it/s 0.68 | |
| [0130 17:57:24 @multigpu.py:323] [p0574] step: count(1140), step_time 1433.31, mean_step_time 1431.51, it/s 0.7 | |
| [0130 17:57:24 @multigpu.py:323] [p0115] step: count(1125), step_time 1474.25, mean_step_time 1476.11, it/s 0.68 | |
| [0130 17:57:25 @multigpu.py:323] [p0576] step: count(1150), step_time 1435.1, mean_step_time 1464.94, it/s 0.68 | |
| [0130 17:57:25 @multigpu.py:323] [p0574] step: count(1141), step_time 1379.84, mean_step_time 1429.89, it/s 0.7 | |
| [0130 17:57:25 @multigpu.py:323] [p0115] step: count(1126), step_time 1453.67, mean_step_time 1476.87, it/s 0.68 | |
| [0130 17:57:27 @multigpu.py:323] [p0576] step: count(1151), step_time 1389.31, mean_step_time 1464.55, it/s 0.68 | |
| [0130 17:57:27 @multigpu.py:323] [p0574] step: count(1142), step_time 1416.3, mean_step_time 1425.25, it/s 0.7 | |
| [0130 17:57:27 @multigpu.py:323] [p0115] step: count(1127), step_time 1405.31, mean_step_time 1476.68, it/s 0.68 | |
| [0130 17:57:28 @multigpu.py:323] [p0115] step: count(1128), step_time 1483.4, mean_step_time 1439.8, it/s 0.69 | |
| [0130 17:57:28 @multigpu.py:323] [p0574] step: count(1143), step_time 1683.21, mean_step_time 1439.75, it/s 0.69 | |
| [0130 17:57:28 @multigpu.py:323] [p0576] step: count(1152), step_time 1719.88, mean_step_time 1439.75, it/s 0.69 | |
| [0130 17:57:30 @multigpu.py:323] [p0115] step: count(1129), step_time 1378.47, mean_step_time 1437.12, it/s 0.7 | |
| [0130 17:57:30 @multigpu.py:323] [p0576] step: count(1153), step_time 1401.52, mean_step_time 1437.43, it/s 0.7 | |
| [0130 17:57:30 @multigpu.py:323] [p0574] step: count(1144), step_time 1404.0, mean_step_time 1436.73, it/s 0.7 | |
| [0130 17:57:31 @multigpu.py:323] [p0115] step: count(1130), step_time 1440.34, mean_step_time 1437.2, it/s 0.7 | |
| [0130 17:57:31 @multigpu.py:323] [p0574] step: count(1145), step_time 1415.34, mean_step_time 1438.86, it/s 0.69 | |
| [0130 17:57:31 @multigpu.py:323] [p0576] step: count(1154), step_time 1429.49, mean_step_time 1434.97, it/s 0.7 | |
| [0130 17:57:33 @multigpu.py:323] [p0576] step: count(1155), step_time 1406.06, mean_step_time 1435.25, it/s 0.7 | |
| [0130 17:57:33 @multigpu.py:323] [p0574] step: count(1146), step_time 1434.59, mean_step_time 1437.11, it/s 0.7 | |
| [0130 17:57:33 @multigpu.py:323] [p0115] step: count(1131), step_time 1466.16, mean_step_time 1441.7, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.459099557201 , 2.4 ##### | |
| [u'online', 2.4] | |
| receiving | |
| [0130 17:57:34 @multigpu.py:323] [p0115] step: count(1132), step_time 1373.58, mean_step_time 1436.71, it/s 0.7 | |
| [0130 17:57:34 @multigpu.py:323] [p0576] step: count(1156), step_time 1426.12, mean_step_time 1435.41, it/s 0.7 | |
| [0130 17:57:34 @multigpu.py:323] [p0574] step: count(1147), step_time 1435.5, mean_step_time 1440.0, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.459456433588 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:57:35 @multigpu.py:323] [p0574] step: count(1148), step_time 1371.35, mean_step_time 1436.91, it/s 0.7 | |
| [0130 17:57:35 @multigpu.py:323] [p0576] step: count(1157), step_time 1404.18, mean_step_time 1434.24, it/s 0.7 | |
| [0130 17:57:35 @multigpu.py:323] [p0115] step: count(1133), step_time 1432.68, mean_step_time 1437.58, it/s 0.7 | |
| [0130 17:57:37 @multigpu.py:323] [p0574] step: count(1149), step_time 1419.67, mean_step_time 1436.19, it/s 0.7 | |
| [0130 17:57:37 @multigpu.py:323] [p0115] step: count(1134), step_time 1433.33, mean_step_time 1436.21, it/s 0.7 | |
| [0130 17:57:37 @multigpu.py:323] [p0576] step: count(1158), step_time 1456.34, mean_step_time 1436.24, it/s 0.7 | |
| [0130 17:57:38 @multigpu.py:323] [p0574] step: count(1150), step_time 1417.75, mean_step_time 1435.39, it/s 0.7 | |
| [0130 17:57:38 @multigpu.py:323] [p0115] step: count(1135), step_time 1424.22, mean_step_time 1435.78, it/s 0.7 | |
| [0130 17:57:38 @multigpu.py:323] [p0576] step: count(1159), step_time 1440.73, mean_step_time 1437.89, it/s 0.7 | |
| [0130 17:57:40 @multigpu.py:323] [p0574] step: count(1151), step_time 1422.46, mean_step_time 1436.27, it/s 0.7 | |
| [0130 17:57:40 @multigpu.py:323] [p0115] step: count(1136), step_time 1428.24, mean_step_time 1435.62, it/s 0.7 | |
| [0130 17:57:40 @multigpu.py:323] [p0576] step: count(1160), step_time 1450.56, mean_step_time 1439.34, it/s 0.69 | |
| [0130 17:57:41 @multigpu.py:323] [p0576] step: count(1161), step_time 1392.22, mean_step_time 1436.65, it/s 0.7 | |
| [0130 17:57:41 @multigpu.py:323] [p0574] step: count(1152), step_time 1491.29, mean_step_time 1437.04, it/s 0.7 | |
| [0130 17:57:41 @multigpu.py:323] [p0115] step: count(1137), step_time 1469.56, mean_step_time 1434.51, it/s 0.7 | |
| [0130 17:57:43 @multigpu.py:323] [p0574] step: count(1153), step_time 1377.57, mean_step_time 1436.13, it/s 0.7 | |
| [0130 17:57:43 @multigpu.py:323] [p0576] step: count(1162), step_time 1424.6, mean_step_time 1436.35, it/s 0.7 | |
| [0130 17:57:43 @multigpu.py:323] [p0115] step: count(1138), step_time 1468.64, mean_step_time 1436.8, it/s 0.7 | |
| [0130 17:57:44 @multigpu.py:323] [p0574] step: count(1154), step_time 1434.21, mean_step_time 1433.02, it/s 0.7 | |
| [0130 17:57:44 @multigpu.py:323] [p0576] step: count(1163), step_time 1407.8, mean_step_time 1436.65, it/s 0.7 | |
| [0130 17:57:44 @multigpu.py:323] [p0115] step: count(1139), step_time 1388.54, mean_step_time 1435.06, it/s 0.7 | |
| [0130 17:57:45 @multigpu.py:323] [p0574] step: count(1155), step_time 1405.23, mean_step_time 1432.14, it/s 0.7 | |
| [0130 17:57:45 @multigpu.py:323] [p0576] step: count(1164), step_time 1430.82, mean_step_time 1438.56, it/s 0.7 | |
| [0130 17:57:46 @multigpu.py:323] [p0115] step: count(1140), step_time 1482.05, mean_step_time 1437.59, it/s 0.7 | |
| [0130 17:57:47 @multigpu.py:323] [p0574] step: count(1156), step_time 1441.58, mean_step_time 1429.83, it/s 0.7 | |
| [0130 17:57:47 @multigpu.py:323] [p0576] step: count(1165), step_time 1415.85, mean_step_time 1439.76, it/s 0.69 | |
| [0130 17:57:47 @multigpu.py:323] [p0115] step: count(1141), step_time 1376.95, mean_step_time 1434.89, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.462921938035 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:57:48 @multigpu.py:323] [p0576] step: count(1166), step_time 1386.7, mean_step_time 1433.36, it/s 0.7 | |
| [0130 17:57:48 @multigpu.py:323] [p0574] step: count(1157), step_time 1514.69, mean_step_time 1434.56, it/s 0.7 | |
| [0130 17:57:48 @multigpu.py:323] [p0115] step: count(1142), step_time 1424.27, mean_step_time 1433.18, it/s 0.7 | |
| [0130 17:57:50 @multigpu.py:323] [p0576] step: count(1167), step_time 1497.67, mean_step_time 1439.23, it/s 0.69 | |
| [0130 17:57:50 @multigpu.py:323] [p0574] step: count(1158), step_time 1395.61, mean_step_time 1436.57, it/s 0.7 | |
| [0130 17:57:50 @multigpu.py:323] [p0115] step: count(1143), step_time 1457.88, mean_step_time 1434.66, it/s 0.7 | |
| [0130 17:57:51 @multigpu.py:323] [p0576] step: count(1168), step_time 1398.13, mean_step_time 1437.26, it/s 0.7 | |
| [0130 17:57:51 @multigpu.py:323] [p0574] step: count(1159), step_time 1416.04, mean_step_time 1435.48, it/s 0.7 | |
| [0130 17:57:51 @multigpu.py:323] [p0115] step: count(1144), step_time 1414.34, mean_step_time 1433.79, it/s 0.7 | |
| [0130 17:57:53 @multigpu.py:323] [p0576] step: count(1169), step_time 1466.74, mean_step_time 1438.99, it/s 0.69 | |
| [0130 17:57:53 @multigpu.py:323] [p0574] step: count(1160), step_time 1441.19, mean_step_time 1435.87, it/s 0.7 | |
| [0130 17:57:53 @multigpu.py:323] [p0115] step: count(1145), step_time 1470.76, mean_step_time 1433.62, it/s 0.7 | |
| [0130 17:57:54 @multigpu.py:323] [p0574] step: count(1161), step_time 1391.38, mean_step_time 1436.45, it/s 0.7 | |
| [0130 17:57:54 @multigpu.py:323] [p0576] step: count(1170), step_time 1394.21, mean_step_time 1436.95, it/s 0.7 | |
| [0130 17:57:54 @multigpu.py:323] [p0115] step: count(1146), step_time 1397.14, mean_step_time 1430.79, it/s 0.7 | |
| [0130 17:57:55 @multigpu.py:323] [p0574] step: count(1162), step_time 1442.57, mean_step_time 1437.76, it/s 0.7 | |
| [0130 17:57:55 @multigpu.py:323] [p0576] step: count(1171), step_time 1501.38, mean_step_time 1442.55, it/s 0.69 | |
| [0130 17:57:55 @multigpu.py:323] [p0115] step: count(1147), step_time 1421.85, mean_step_time 1431.62, it/s 0.7 | |
| [0130 17:57:57 @multigpu.py:323] [p0115] step: count(1148), step_time 1466.54, mean_step_time 1430.78, it/s 0.7 | |
| [0130 17:57:57 @multigpu.py:323] [p0576] step: count(1172), step_time 1483.5, mean_step_time 1430.73, it/s 0.7 | |
| [0130 17:57:57 @multigpu.py:323] [p0574] step: count(1163), step_time 1542.55, mean_step_time 1430.73, it/s 0.7 | |
| [0130 17:57:58 @multigpu.py:323] [p0115] step: count(1149), step_time 1364.25, mean_step_time 1430.07, it/s 0.7 | |
| [0130 17:57:58 @multigpu.py:323] [p0574] step: count(1164), step_time 1429.55, mean_step_time 1432.01, it/s 0.7 | |
| [0130 17:57:58 @multigpu.py:323] [p0576] step: count(1173), step_time 1446.76, mean_step_time 1432.99, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.466300260226 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:58:00 @multigpu.py:323] [p0115] step: count(1150), step_time 1485.05, mean_step_time 1432.3, it/s 0.7 | |
| [0130 17:58:00 @multigpu.py:323] [p0576] step: count(1174), step_time 1423.04, mean_step_time 1432.67, it/s 0.7 | |
| [0130 17:58:00 @multigpu.py:323] [p0574] step: count(1165), step_time 1455.67, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:58:01 @multigpu.py:323] [p0574] step: count(1166), step_time 1392.81, mean_step_time 1431.93, it/s 0.7 | |
| [0130 17:58:01 @multigpu.py:323] [p0115] step: count(1151), step_time 1451.4, mean_step_time 1431.56, it/s 0.7 | |
| [0130 17:58:01 @multigpu.py:323] [p0576] step: count(1175), step_time 1432.38, mean_step_time 1433.99, it/s 0.7 | |
| [0130 17:58:03 @multigpu.py:323] [p0574] step: count(1167), step_time 1418.52, mean_step_time 1431.09, it/s 0.7 | |
| [0130 17:58:03 @multigpu.py:323] [p0115] step: count(1152), step_time 1432.02, mean_step_time 1434.49, it/s 0.7 | |
| [0130 17:58:03 @multigpu.py:323] [p0576] step: count(1176), step_time 1450.0, mean_step_time 1435.18, it/s 0.7 | |
| [0130 17:58:04 @multigpu.py:323] [p0574] step: count(1168), step_time 1430.0, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:58:04 @multigpu.py:323] [p0115] step: count(1153), step_time 1400.37, mean_step_time 1432.87, it/s 0.7 | |
| [0130 17:58:04 @multigpu.py:323] [p0576] step: count(1177), step_time 1410.86, mean_step_time 1435.51, it/s 0.7 | |
| [0130 17:58:06 @multigpu.py:323] [p0574] step: count(1169), step_time 1430.97, mean_step_time 1434.58, it/s 0.7 | |
| [0130 17:58:06 @multigpu.py:323] [p0576] step: count(1178), step_time 1409.64, mean_step_time 1433.18, it/s 0.7 | |
| [0130 17:58:06 @multigpu.py:323] [p0115] step: count(1154), step_time 1457.51, mean_step_time 1434.08, it/s 0.7 | |
| [0130 17:58:07 @multigpu.py:323] [p0574] step: count(1170), step_time 1406.46, mean_step_time 1434.02, it/s 0.7 | |
| [0130 17:58:07 @multigpu.py:323] [p0576] step: count(1179), step_time 1408.16, mean_step_time 1431.55, it/s 0.7 | |
| [0130 17:58:07 @multigpu.py:323] [p0115] step: count(1155), step_time 1438.62, mean_step_time 1434.8, it/s 0.7 | |
| [0130 17:58:08 @multigpu.py:323] [p0576] step: count(1180), step_time 1363.05, mean_step_time 1427.18, it/s 0.7 | |
| [0130 17:58:08 @multigpu.py:323] [p0574] step: count(1171), step_time 1409.52, mean_step_time 1433.37, it/s 0.7 | |
| [0130 17:58:08 @multigpu.py:323] [p0115] step: count(1156), step_time 1448.35, mean_step_time 1435.81, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.468956205514 , 1.7 ##### | |
| [u'online', 1.7] | |
| ##### Sending to neptune: active_workers : 0.468956336644 , 3 ##### | |
| receiving | |
| [0130 17:58:10 @multigpu.py:323] [p0574] step: count(1172), step_time 1419.6, mean_step_time 1429.79, it/s 0.7 | |
| [0130 17:58:10 @multigpu.py:323] [p0576] step: count(1181), step_time 1521.66, mean_step_time 1433.65, it/s 0.7 | |
| [0130 17:58:10 @multigpu.py:323] [p0115] step: count(1157), step_time 1429.55, mean_step_time 1433.81, it/s 0.7 | |
| [0130 17:58:11 @multigpu.py:323] [p0574] step: count(1173), step_time 1444.48, mean_step_time 1433.13, it/s 0.7 | |
| [0130 17:58:11 @multigpu.py:323] [p0115] step: count(1158), step_time 1404.46, mean_step_time 1430.6, it/s 0.7 | |
| [0130 17:58:11 @multigpu.py:323] [p0576] step: count(1182), step_time 1444.9, mean_step_time 1434.66, it/s 0.7 | |
| [0130 17:58:13 @multigpu.py:323] [p0574] step: count(1174), step_time 1458.27, mean_step_time 1434.33, it/s 0.7 | |
| [0130 17:58:13 @multigpu.py:323] [p0576] step: count(1183), step_time 1409.56, mean_step_time 1434.75, it/s 0.7 | |
| [0130 17:58:13 @multigpu.py:323] [p0115] step: count(1159), step_time 1438.83, mean_step_time 1433.11, it/s 0.7 | |
| [0130 17:58:14 @multigpu.py:323] [p0574] step: count(1175), step_time 1425.11, mean_step_time 1435.33, it/s 0.7 | |
| [0130 17:58:14 @multigpu.py:323] [p0576] step: count(1184), step_time 1405.42, mean_step_time 1433.48, it/s 0.7 | |
| [0130 17:58:14 @multigpu.py:323] [p0115] step: count(1160), step_time 1442.89, mean_step_time 1431.15, it/s 0.7 | |
| [0130 17:58:15 @multigpu.py:323] [p0576] step: count(1185), step_time 1369.21, mean_step_time 1431.15, it/s 0.7 | |
| [0130 17:58:16 @multigpu.py:323] [p0574] step: count(1176), step_time 1445.12, mean_step_time 1435.51, it/s 0.7 | |
| [0130 17:58:16 @multigpu.py:323] [p0115] step: count(1161), step_time 1429.14, mean_step_time 1433.76, it/s 0.7 | |
| [0130 17:58:17 @multigpu.py:323] [p0576] step: count(1186), step_time 1430.49, mean_step_time 1433.34, it/s 0.7 | |
| [0130 17:58:17 @multigpu.py:323] [p0574] step: count(1177), step_time 1413.46, mean_step_time 1430.44, it/s 0.7 | |
| [0130 17:58:17 @multigpu.py:323] [p0115] step: count(1162), step_time 1433.96, mean_step_time 1434.25, it/s 0.7 | |
| [0130 17:58:18 @multigpu.py:323] [p0576] step: count(1187), step_time 1365.08, mean_step_time 1426.71, it/s 0.7 | |
| [0130 17:58:18 @multigpu.py:323] [p0574] step: count(1178), step_time 1520.99, mean_step_time 1436.71, it/s 0.7 | |
| [0130 17:58:18 @multigpu.py:323] [p0115] step: count(1163), step_time 1451.15, mean_step_time 1433.91, it/s 0.7 | |
| [0130 17:58:20 @multigpu.py:323] [p0576] step: count(1188), step_time 1456.22, mean_step_time 1429.61, it/s 0.7 | |
| [0130 17:58:20 @multigpu.py:323] [p0574] step: count(1179), step_time 1405.58, mean_step_time 1436.19, it/s 0.7 | |
| [0130 17:58:20 @multigpu.py:323] [p0115] step: count(1164), step_time 1411.96, mean_step_time 1433.79, it/s 0.7 | |
| [0130 17:58:21 @multigpu.py:323] [p0576] step: count(1189), step_time 1377.94, mean_step_time 1425.17, it/s 0.7 | |
| [0130 17:58:21 @multigpu.py:323] [p0115] step: count(1165), step_time 1390.93, mean_step_time 1429.8, it/s 0.7 | |
| [0130 17:58:21 @multigpu.py:323] [p0574] step: count(1180), step_time 1425.96, mean_step_time 1435.43, it/s 0.7 | |
| [0130 17:58:22 @multigpu.py:323] [p0576] step: count(1190), step_time 1423.47, mean_step_time 1426.64, it/s 0.7 | |
| [0130 17:58:23 @multigpu.py:323] [p0115] step: count(1166), step_time 1432.86, mean_step_time 1431.59, it/s 0.7 | |
| [0130 17:58:23 @multigpu.py:323] [p0574] step: count(1181), step_time 1417.27, mean_step_time 1436.72, it/s 0.7 | |
| [0130 17:58:24 @multigpu.py:323] [p0576] step: count(1191), step_time 1413.47, mean_step_time 1422.24, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.473143153853 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:58:24 @multigpu.py:323] [p0115] step: count(1167), step_time 1428.48, mean_step_time 1431.92, it/s 0.7 | |
| [0130 17:58:24 @multigpu.py:323] [p0574] step: count(1182), step_time 1495.99, mean_step_time 1439.39, it/s 0.69 | |
| [0130 17:58:26 @multigpu.py:323] [p0115] step: count(1168), step_time 1556.94, mean_step_time 1436.44, it/s 0.7 | |
| [0130 17:58:26 @multigpu.py:323] [p0574] step: count(1183), step_time 1481.8, mean_step_time 1436.36, it/s 0.7 | |
| [0130 17:58:26 @multigpu.py:323] [p0576] step: count(1192), step_time 1767.82, mean_step_time 1436.46, it/s 0.7 | |
| [0130 17:58:27 @multigpu.py:323] [p0574] step: count(1184), step_time 1398.17, mean_step_time 1434.79, it/s 0.7 | |
| [0130 17:58:27 @multigpu.py:323] [p0115] step: count(1169), step_time 1428.32, mean_step_time 1439.64, it/s 0.69 | |
| [0130 17:58:27 @multigpu.py:323] [p0576] step: count(1193), step_time 1467.1, mean_step_time 1437.47, it/s 0.7 | |
| [0130 17:58:29 @multigpu.py:323] [p0574] step: count(1185), step_time 1440.13, mean_step_time 1434.01, it/s 0.7 | |
| [0130 17:58:29 @multigpu.py:323] [p0576] step: count(1194), step_time 1377.48, mean_step_time 1435.2, it/s 0.7 | |
| [0130 17:58:29 @multigpu.py:323] [p0115] step: count(1170), step_time 1434.16, mean_step_time 1437.1, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.474509788288 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 17:58:30 @multigpu.py:323] [p0574] step: count(1186), step_time 1390.93, mean_step_time 1433.92, it/s 0.7 | |
| [0130 17:58:30 @multigpu.py:323] [p0576] step: count(1195), step_time 1464.69, mean_step_time 1436.81, it/s 0.7 | |
| [0130 17:58:30 @multigpu.py:323] [p0115] step: count(1171), step_time 1453.03, mean_step_time 1437.18, it/s 0.7 | |
| [0130 17:58:31 @multigpu.py:323] [p0574] step: count(1187), step_time 1418.75, mean_step_time 1433.93, it/s 0.7 | |
| [0130 17:58:31 @multigpu.py:323] [p0115] step: count(1172), step_time 1395.88, mean_step_time 1435.37, it/s 0.7 | |
| [0130 17:58:31 @multigpu.py:323] [p0576] step: count(1196), step_time 1400.67, mean_step_time 1434.35, it/s 0.7 | |
| [0130 17:58:33 @multigpu.py:323] [p0574] step: count(1188), step_time 1443.55, mean_step_time 1434.61, it/s 0.7 | |
| [0130 17:58:33 @multigpu.py:323] [p0576] step: count(1197), step_time 1418.83, mean_step_time 1434.74, it/s 0.7 | |
| [0130 17:58:33 @multigpu.py:323] [p0115] step: count(1173), step_time 1432.91, mean_step_time 1437.0, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.475631449156 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 17:58:34 @multigpu.py:323] [p0574] step: count(1189), step_time 1408.25, mean_step_time 1433.47, it/s 0.7 | |
| [0130 17:58:34 @multigpu.py:323] [p0115] step: count(1174), step_time 1381.5, mean_step_time 1433.2, it/s 0.7 | |
| [0130 17:58:34 @multigpu.py:323] [p0576] step: count(1198), step_time 1453.78, mean_step_time 1436.95, it/s 0.7 | |
| [0130 17:58:36 @multigpu.py:323] [p0574] step: count(1190), step_time 1429.62, mean_step_time 1434.63, it/s 0.7 | |
| [0130 17:58:36 @multigpu.py:323] [p0115] step: count(1175), step_time 1427.4, mean_step_time 1432.64, it/s 0.7 | |
| [0130 17:58:36 @multigpu.py:323] [p0576] step: count(1199), step_time 1442.31, mean_step_time 1438.66, it/s 0.7 | |
| [0130 17:58:37 @multigpu.py:323] [p0115] step: count(1176), step_time 1407.48, mean_step_time 1430.59, it/s 0.7 | |
| [0130 17:58:37 @multigpu.py:323] [p0576] step: count(1200), step_time 1383.31, mean_step_time 1439.67, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.476802181337 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.476802181337 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.476802181337 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.476802654399 , -0.00253472337499 ##### | |
| ##### Sending to neptune: policy_loss : 0.476802654399 , 0.375536680222 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.476802654399 , -2.28917598724 ##### | |
| ##### Sending to neptune: value_loss : 0.476802654399 , 1.58919465542 ##### | |
| ##### Sending to neptune: advantage : 0.476802654399 , -0.00181525712833 ##### | |
| ##### Sending to neptune: pred_reward : 0.476802654399 , 0.453048050404 ##### | |
| ##### Sending to neptune: max_logit : 0.476802654399 , 0.191985800862 ##### | |
| [u'loss', -0.002534723374992609, 0.3755366802215576, -2.2891759872436523, 1.589194655418396, -0.001815257128328085, 0.45304805040359497, 0.19198580086231232] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.476803139382 , 9613262.34 ##### | |
| ##### Sending to neptune: dp_per_s : 0.476803139382 , 88.6668697671 ##### | |
| [u'other', 9613262.34, 88.66686976706502] | |
| receiving | |
| [0130 17:58:37 @multigpu.py:323] [p0574] step: count(1191), step_time 1487.49, mean_step_time 1438.53, it/s 0.7 | |
| [0130 17:58:38 @multigpu.py:323] [p0115] step: count(1177), step_time 1448.55, mean_step_time 1431.54, it/s 0.7 | |
| [0130 17:58:39 @multigpu.py:323] [p0576] step: count(1201), step_time 1473.19, mean_step_time 1437.25, it/s 0.7 | |
| [0130 17:58:39 @multigpu.py:323] [p0574] step: count(1192), step_time 1479.83, mean_step_time 1441.54, it/s 0.69 | |
| [0130 17:58:40 @multigpu.py:323] [p0115] step: count(1178), step_time 1453.8, mean_step_time 1434.01, it/s 0.7 | |
| [0130 17:58:40 @multigpu.py:323] [p0574] step: count(1193), step_time 1379.85, mean_step_time 1438.31, it/s 0.7 | |
| [0130 17:58:40 @multigpu.py:323] [p0576] step: count(1202), step_time 1401.42, mean_step_time 1435.07, it/s 0.7 | |
| [0130 17:58:41 @multigpu.py:323] [p0115] step: count(1179), step_time 1398.57, mean_step_time 1432.0, it/s 0.7 | |
| [0130 17:58:41 @multigpu.py:323] [p0574] step: count(1194), step_time 1400.66, mean_step_time 1435.43, it/s 0.7 | |
| [0130 17:58:41 @multigpu.py:323] [p0576] step: count(1203), step_time 1400.03, mean_step_time 1434.6, it/s 0.7 | |
| [0130 17:58:43 @multigpu.py:323] [p0576] step: count(1204), step_time 1419.88, mean_step_time 1435.32, it/s 0.7 | |
| [0130 17:58:43 @multigpu.py:323] [p0574] step: count(1195), step_time 1470.39, mean_step_time 1437.69, it/s 0.7 | |
| [0130 17:58:43 @multigpu.py:323] [p0115] step: count(1180), step_time 1514.03, mean_step_time 1435.55, it/s 0.7 | |
| [0130 17:58:44 @multigpu.py:323] [p0574] step: count(1196), step_time 1378.91, mean_step_time 1434.38, it/s 0.7 | |
| [0130 17:58:44 @multigpu.py:323] [p0576] step: count(1205), step_time 1442.53, mean_step_time 1438.99, it/s 0.69 | |
| [0130 17:58:44 @multigpu.py:323] [p0115] step: count(1181), step_time 1418.57, mean_step_time 1435.02, it/s 0.7 | |
| [0130 17:58:46 @multigpu.py:323] [p0574] step: count(1197), step_time 1424.19, mean_step_time 1434.92, it/s 0.7 | |
| [0130 17:58:46 @multigpu.py:323] [p0576] step: count(1206), step_time 1445.85, mean_step_time 1439.75, it/s 0.69 | |
| [0130 17:58:46 @multigpu.py:323] [p0115] step: count(1182), step_time 1449.86, mean_step_time 1435.82, it/s 0.7 | |
| [0130 17:58:47 @multigpu.py:323] [p0576] step: count(1207), step_time 1386.01, mean_step_time 1440.8, it/s 0.69 | |
| [0130 17:58:47 @multigpu.py:323] [p0574] step: count(1198), step_time 1435.24, mean_step_time 1430.63, it/s 0.7 | |
| [0130 17:58:47 @multigpu.py:323] [p0115] step: count(1183), step_time 1491.59, mean_step_time 1437.84, it/s 0.7 | |
| [0130 17:58:49 @multigpu.py:323] [p0576] step: count(1208), step_time 1456.2, mean_step_time 1440.8, it/s 0.69 | |
| [0130 17:58:49 @multigpu.py:323] [p0574] step: count(1199), step_time 1449.49, mean_step_time 1432.82, it/s 0.7 | |
| [0130 17:58:49 @multigpu.py:323] [p0115] step: count(1184), step_time 1419.96, mean_step_time 1438.24, it/s 0.7 | |
| [0130 17:58:50 @multigpu.py:323] [p0576] step: count(1209), step_time 1403.29, mean_step_time 1442.07, it/s 0.69 | |
| [0130 17:58:50 @multigpu.py:323] [p0574] step: count(1200), step_time 1416.98, mean_step_time 1432.38, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.480370954143 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.480370954143 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.480370954143 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.480371489988 , -0.00687376549467 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.480371489988 , -0.0515709109604 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.480371489988 , -2.28922891617 ##### | |
| ##### Sending to neptune: value_loss : 0.480371489988 , 1.46095776558 ##### | |
| ##### Sending to neptune: advantage : 0.480371489988 , 0.000297629856504 ##### | |
| ##### Sending to neptune: pred_reward : 0.480371489988 , 0.45008713007 ##### | |
| ##### Sending to neptune: max_logit : 0.480371489988 , 0.190380632877 ##### | |
| [u'loss', -0.006873765494674444, -0.05157091096043587, -2.289228916168213, 1.4609577655792236, 0.0002976298565044999, 0.45008713006973267, 0.19038063287734985] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.480371931063 , 9601806.97 ##### | |
| ##### Sending to neptune: dp_per_s : 0.480371931063 , 88.6732990532 ##### | |
| [u'other', 9601806.97, 88.6732990532418] | |
| receiving | |
| [0130 17:58:50 @multigpu.py:323] [p0115] step: count(1185), step_time 1475.1, mean_step_time 1442.45, it/s 0.69 | |
| [0130 17:58:51 @multigpu.py:323] [p0574] step: count(1201), step_time 1401.01, mean_step_time 1431.56, it/s 0.7 | |
| [0130 17:58:51 @multigpu.py:323] [p0576] step: count(1210), step_time 1431.24, mean_step_time 1442.46, it/s 0.69 | |
| [0130 17:58:52 @multigpu.py:323] [p0115] step: count(1186), step_time 1473.77, mean_step_time 1444.5, it/s 0.69 | |
| [0130 17:58:53 @multigpu.py:323] [p0574] step: count(1202), step_time 1393.14, mean_step_time 1426.42, it/s 0.7 | |
| [0130 17:58:53 @multigpu.py:323] [p0576] step: count(1211), step_time 1452.95, mean_step_time 1444.43, it/s 0.69 | |
| [0130 17:58:53 @multigpu.py:323] [p0115] step: count(1187), step_time 1401.6, mean_step_time 1443.15, it/s 0.69 | |
| [0130 17:58:54 @multigpu.py:323] [p0115] step: count(1188), step_time 1399.15, mean_step_time 1435.26, it/s 0.7 | |
| [0130 17:58:54 @multigpu.py:323] [p0574] step: count(1203), step_time 1659.28, mean_step_time 1435.29, it/s 0.7 | |
| [0130 17:58:54 @multigpu.py:323] [p0576] step: count(1212), step_time 1584.36, mean_step_time 1435.26, it/s 0.7 | |
| [0130 17:58:56 @multigpu.py:323] [p0576] step: count(1213), step_time 1403.86, mean_step_time 1432.09, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.482001303567 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 17:58:56 @multigpu.py:323] [p0115] step: count(1189), step_time 1445.36, mean_step_time 1436.11, it/s 0.7 | |
| [0130 17:58:56 @multigpu.py:323] [p0574] step: count(1204), step_time 1458.93, mean_step_time 1438.33, it/s 0.7 | |
| [0130 17:58:57 @multigpu.py:323] [p0574] step: count(1205), step_time 1395.93, mean_step_time 1436.12, it/s 0.7 | |
| [0130 17:58:57 @multigpu.py:323] [p0576] step: count(1214), step_time 1477.0, mean_step_time 1437.07, it/s 0.7 | |
| [0130 17:58:57 @multigpu.py:323] [p0115] step: count(1190), step_time 1476.45, mean_step_time 1438.23, it/s 0.7 | |
| [0130 17:58:59 @multigpu.py:323] [p0574] step: count(1206), step_time 1423.39, mean_step_time 1437.74, it/s 0.7 | |
| [0130 17:58:59 @multigpu.py:323] [p0576] step: count(1215), step_time 1412.4, mean_step_time 1434.46, it/s 0.7 | |
| [0130 17:58:59 @multigpu.py:323] [p0115] step: count(1191), step_time 1499.11, mean_step_time 1440.53, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.482869036661 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 17:59:00 @multigpu.py:323] [p0576] step: count(1216), step_time 1419.01, mean_step_time 1435.37, it/s 0.7 | |
| [0130 17:59:00 @multigpu.py:323] [p0574] step: count(1207), step_time 1436.29, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:59:00 @multigpu.py:323] [p0115] step: count(1192), step_time 1446.43, mean_step_time 1443.06, it/s 0.69 | |
| [0130 17:59:02 @multigpu.py:323] [p0576] step: count(1217), step_time 1414.54, mean_step_time 1435.16, it/s 0.7 | |
| [0130 17:59:02 @multigpu.py:323] [p0574] step: count(1208), step_time 1452.89, mean_step_time 1439.09, it/s 0.69 | |
| [0130 17:59:02 @multigpu.py:323] [p0115] step: count(1193), step_time 1457.53, mean_step_time 1444.29, it/s 0.69 | |
| [0130 17:59:03 @multigpu.py:323] [p0576] step: count(1218), step_time 1407.46, mean_step_time 1432.84, it/s 0.7 | |
| [0130 17:59:03 @multigpu.py:323] [p0574] step: count(1209), step_time 1390.8, mean_step_time 1438.22, it/s 0.7 | |
| [0130 17:59:03 @multigpu.py:323] [p0115] step: count(1194), step_time 1429.94, mean_step_time 1446.71, it/s 0.69 | |
| [0130 17:59:04 @multigpu.py:323] [p0576] step: count(1219), step_time 1411.38, mean_step_time 1431.3, it/s 0.7 | |
| [0130 17:59:04 @multigpu.py:323] [p0574] step: count(1210), step_time 1437.77, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:59:05 @multigpu.py:323] [p0115] step: count(1195), step_time 1420.12, mean_step_time 1446.35, it/s 0.69 | |
| [0130 17:59:06 @multigpu.py:323] [p0576] step: count(1220), step_time 1431.94, mean_step_time 1433.73, it/s 0.7 | |
| [0130 17:59:06 @multigpu.py:323] [p0574] step: count(1211), step_time 1405.34, mean_step_time 1434.52, it/s 0.7 | |
| [0130 17:59:06 @multigpu.py:323] [p0115] step: count(1196), step_time 1424.1, mean_step_time 1447.18, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.484892646074 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:59:07 @multigpu.py:323] [p0576] step: count(1221), step_time 1406.25, mean_step_time 1430.38, it/s 0.7 | |
| [0130 17:59:07 @multigpu.py:323] [p0574] step: count(1212), step_time 1464.63, mean_step_time 1433.76, it/s 0.7 | |
| [0130 17:59:07 @multigpu.py:323] [p0115] step: count(1197), step_time 1451.57, mean_step_time 1447.33, it/s 0.69 | |
| [0130 17:59:09 @multigpu.py:323] [p0576] step: count(1222), step_time 1469.92, mean_step_time 1433.8, it/s 0.7 | |
| [0130 17:59:09 @multigpu.py:323] [p0574] step: count(1213), step_time 1403.7, mean_step_time 1434.95, it/s 0.7 | |
| [0130 17:59:09 @multigpu.py:323] [p0115] step: count(1198), step_time 1456.32, mean_step_time 1447.46, it/s 0.69 | |
| [0130 17:59:10 @multigpu.py:323] [p0576] step: count(1223), step_time 1411.45, mean_step_time 1434.38, it/s 0.7 | |
| [0130 17:59:10 @multigpu.py:323] [p0574] step: count(1214), step_time 1427.55, mean_step_time 1436.29, it/s 0.7 | |
| [0130 17:59:10 @multigpu.py:323] [p0115] step: count(1199), step_time 1462.95, mean_step_time 1450.67, it/s 0.69 | |
| [0130 17:59:11 @multigpu.py:323] [p0576] step: count(1224), step_time 1425.05, mean_step_time 1434.63, it/s 0.7 | |
| [0130 17:59:12 @multigpu.py:323] [p0574] step: count(1215), step_time 1438.4, mean_step_time 1434.69, it/s 0.7 | |
| [0130 17:59:12 @multigpu.py:323] [p0115] step: count(1200), step_time 1445.15, mean_step_time 1447.23, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.486447773841 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.486447773841 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.486447773841 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| ##### Sending to neptune: active_workers : 0.486448003054 , 3 ##### | |
| receiving | |
| ##### Sending to neptune: cost : 0.486448276374 , -0.00475755473599 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.486448276374 , 0.0145185422152 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.486448276374 , -2.28904032707 ##### | |
| ##### Sending to neptune: value_loss : 0.486448276374 , 1.66555476189 ##### | |
| ##### Sending to neptune: advantage : 0.486448276374 , -9.42744300119e-05 ##### | |
| ##### Sending to neptune: pred_reward : 0.486448276374 , 0.464911580086 ##### | |
| ##### Sending to neptune: max_logit : 0.486448276374 , 0.191080719233 ##### | |
| [u'loss', -0.0047575547359883785, 0.01451854221522808, -2.2890403270721436, 1.6655547618865967, -9.427443001186475e-05, 0.4649115800857544, 0.1910807192325592] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.486448774669 , 9607864.04 ##### | |
| ##### Sending to neptune: dp_per_s : 0.486448774669 , 88.614272994 ##### | |
| [u'other', 9607864.04, 88.61427299401984] | |
| receiving | |
| [0130 17:59:13 @multigpu.py:323] [p0576] step: count(1225), step_time 1419.94, mean_step_time 1433.5, it/s 0.7 | |
| [0130 17:59:13 @multigpu.py:323] [p0574] step: count(1216), step_time 1427.25, mean_step_time 1437.11, it/s 0.7 | |
| [0130 17:59:13 @multigpu.py:323] [p0115] step: count(1201), step_time 1423.0, mean_step_time 1447.45, it/s 0.69 | |
| [0130 17:59:14 @multigpu.py:323] [p0574] step: count(1217), step_time 1435.25, mean_step_time 1437.66, it/s 0.7 | |
| [0130 17:59:14 @multigpu.py:323] [p0576] step: count(1226), step_time 1497.93, mean_step_time 1436.11, it/s 0.7 | |
| [0130 17:59:15 @multigpu.py:323] [p0115] step: count(1202), step_time 1460.86, mean_step_time 1448.0, it/s 0.69 | |
| [0130 17:59:16 @multigpu.py:323] [p0576] step: count(1227), step_time 1387.57, mean_step_time 1436.19, it/s 0.7 | |
| [0130 17:59:16 @multigpu.py:323] [p0574] step: count(1218), step_time 1428.02, mean_step_time 1437.3, it/s 0.7 | |
| [0130 17:59:16 @multigpu.py:323] [p0115] step: count(1203), step_time 1384.71, mean_step_time 1442.66, it/s 0.69 | |
| [0130 17:59:17 @multigpu.py:323] [p0574] step: count(1219), step_time 1419.66, mean_step_time 1435.81, it/s 0.7 | |
| [0130 17:59:17 @multigpu.py:323] [p0576] step: count(1228), step_time 1456.31, mean_step_time 1436.19, it/s 0.7 | |
| [0130 17:59:18 @multigpu.py:323] [p0115] step: count(1204), step_time 1445.15, mean_step_time 1443.92, it/s 0.69 | |
| [0130 17:59:19 @multigpu.py:323] [p0576] step: count(1229), step_time 1376.86, mean_step_time 1434.87, it/s 0.7 | |
| [0130 17:59:19 @multigpu.py:323] [p0574] step: count(1220), step_time 1418.2, mean_step_time 1435.87, it/s 0.7 | |
| [0130 17:59:19 @multigpu.py:323] [p0115] step: count(1205), step_time 1404.72, mean_step_time 1440.4, it/s 0.69 | |
| [0130 17:59:20 @multigpu.py:323] [p0576] step: count(1230), step_time 1425.41, mean_step_time 1434.58, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.488742454118 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 17:59:20 @multigpu.py:323] [p0574] step: count(1221), step_time 1416.16, mean_step_time 1436.63, it/s 0.7 | |
| [0130 17:59:20 @multigpu.py:323] [p0115] step: count(1206), step_time 1460.48, mean_step_time 1439.73, it/s 0.69 | |
| [0130 17:59:22 @multigpu.py:323] [p0576] step: count(1231), step_time 1449.42, mean_step_time 1434.4, it/s 0.7 | |
| [0130 17:59:22 @multigpu.py:323] [p0574] step: count(1222), step_time 1442.99, mean_step_time 1439.12, it/s 0.69 | |
| [0130 17:59:22 @multigpu.py:323] [p0115] step: count(1207), step_time 1462.84, mean_step_time 1442.8, it/s 0.69 | |
| [0130 17:59:23 @multigpu.py:323] [p0115] step: count(1208), step_time 1410.18, mean_step_time 1443.35, it/s 0.69 | |
| [0130 17:59:23 @multigpu.py:323] [p0576] step: count(1232), step_time 1763.18, mean_step_time 1443.34, it/s 0.69 | |
| [0130 17:59:23 @multigpu.py:323] [p0574] step: count(1223), step_time 1745.65, mean_step_time 1443.44, it/s 0.69 | |
| [0130 17:59:25 @multigpu.py:323] [p0576] step: count(1233), step_time 1403.5, mean_step_time 1443.32, it/s 0.69 | |
| [0130 17:59:25 @multigpu.py:323] [p0115] step: count(1209), step_time 1411.91, mean_step_time 1441.68, it/s 0.69 | |
| [0130 17:59:25 @multigpu.py:323] [p0574] step: count(1224), step_time 1449.48, mean_step_time 1442.97, it/s 0.69 | |
| [0130 17:59:26 @multigpu.py:323] [p0576] step: count(1234), step_time 1413.02, mean_step_time 1440.13, it/s 0.69 | |
| [0130 17:59:26 @multigpu.py:323] [p0115] step: count(1210), step_time 1415.36, mean_step_time 1438.62, it/s 0.7 | |
| [0130 17:59:26 @multigpu.py:323] [p0574] step: count(1225), step_time 1408.73, mean_step_time 1443.61, it/s 0.69 | |
| [0130 17:59:28 @multigpu.py:323] [p0576] step: count(1235), step_time 1410.4, mean_step_time 1440.03, it/s 0.69 | |
| [0130 17:59:28 @multigpu.py:323] [p0574] step: count(1226), step_time 1409.94, mean_step_time 1442.94, it/s 0.69 | |
| [0130 17:59:28 @multigpu.py:323] [p0115] step: count(1211), step_time 1486.26, mean_step_time 1437.98, it/s 0.7 | |
| [0130 17:59:29 @multigpu.py:323] [p0576] step: count(1236), step_time 1377.55, mean_step_time 1437.95, it/s 0.7 | |
| [0130 17:59:29 @multigpu.py:323] [p0574] step: count(1227), step_time 1451.71, mean_step_time 1443.71, it/s 0.69 | |
| [0130 17:59:29 @multigpu.py:323] [p0115] step: count(1212), step_time 1450.74, mean_step_time 1438.19, it/s 0.7 | |
| [0130 17:59:30 @multigpu.py:323] [p0576] step: count(1237), step_time 1406.86, mean_step_time 1437.57, it/s 0.7 | |
| [0130 17:59:30 @multigpu.py:323] [p0574] step: count(1228), step_time 1483.1, mean_step_time 1445.22, it/s 0.69 | |
| [0130 17:59:30 @multigpu.py:323] [p0115] step: count(1213), step_time 1449.25, mean_step_time 1437.78, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.491776681079 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 17:59:32 @multigpu.py:323] [p0576] step: count(1238), step_time 1422.62, mean_step_time 1438.33, it/s 0.7 | |
| [0130 17:59:32 @multigpu.py:323] [p0115] step: count(1214), step_time 1383.4, mean_step_time 1435.45, it/s 0.7 | |
| [0130 17:59:32 @multigpu.py:323] [p0574] step: count(1229), step_time 1452.75, mean_step_time 1448.31, it/s 0.69 | |
| [0130 17:59:33 @multigpu.py:323] [p0576] step: count(1239), step_time 1400.1, mean_step_time 1437.76, it/s 0.7 | |
| [0130 17:59:33 @multigpu.py:323] [p0115] step: count(1215), step_time 1418.15, mean_step_time 1435.36, it/s 0.7 | |
| [0130 17:59:33 @multigpu.py:323] [p0574] step: count(1230), step_time 1462.29, mean_step_time 1449.54, it/s 0.69 | |
| [0130 17:59:35 @multigpu.py:323] [p0576] step: count(1240), step_time 1443.06, mean_step_time 1438.32, it/s 0.7 | |
| [0130 17:59:35 @multigpu.py:323] [p0115] step: count(1216), step_time 1425.73, mean_step_time 1435.44, it/s 0.7 | |
| [0130 17:59:35 @multigpu.py:323] [p0574] step: count(1231), step_time 1409.05, mean_step_time 1449.73, it/s 0.69 | |
| [0130 17:59:36 @multigpu.py:323] [p0576] step: count(1241), step_time 1385.11, mean_step_time 1437.26, it/s 0.7 | |
| [0130 17:59:36 @multigpu.py:323] [p0115] step: count(1217), step_time 1434.85, mean_step_time 1434.6, it/s 0.7 | |
| [0130 17:59:36 @multigpu.py:323] [p0574] step: count(1232), step_time 1401.94, mean_step_time 1446.59, it/s 0.69 | |
| [0130 17:59:37 @multigpu.py:323] [p0576] step: count(1242), step_time 1437.12, mean_step_time 1435.62, it/s 0.7 | |
| [0130 17:59:38 @multigpu.py:323] [p0115] step: count(1218), step_time 1427.21, mean_step_time 1433.15, it/s 0.7 | |
| [0130 17:59:38 @multigpu.py:323] [p0574] step: count(1233), step_time 1444.78, mean_step_time 1448.65, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.493770758841 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 17:59:39 @multigpu.py:323] [p0576] step: count(1243), step_time 1458.66, mean_step_time 1437.98, it/s 0.7 | |
| [0130 17:59:39 @multigpu.py:323] [p0115] step: count(1219), step_time 1430.12, mean_step_time 1431.5, it/s 0.7 | |
| [0130 17:59:39 @multigpu.py:323] [p0574] step: count(1234), step_time 1417.29, mean_step_time 1448.13, it/s 0.69 | |
| [0130 17:59:40 @multigpu.py:323] [p0576] step: count(1244), step_time 1432.94, mean_step_time 1438.38, it/s 0.7 | |
| [0130 17:59:40 @multigpu.py:323] [p0115] step: count(1220), step_time 1428.74, mean_step_time 1430.68, it/s 0.7 | |
| [0130 17:59:41 @multigpu.py:323] [p0574] step: count(1235), step_time 1480.59, mean_step_time 1450.24, it/s 0.69 | |
| [0130 17:59:42 @multigpu.py:323] [p0576] step: count(1245), step_time 1429.8, mean_step_time 1438.87, it/s 0.69 | |
| [0130 17:59:42 @multigpu.py:323] [p0115] step: count(1221), step_time 1496.29, mean_step_time 1434.35, it/s 0.7 | |
| [0130 17:59:42 @multigpu.py:323] [p0574] step: count(1236), step_time 1433.37, mean_step_time 1450.55, it/s 0.69 | |
| [0130 17:59:43 @multigpu.py:323] [p0576] step: count(1246), step_time 1430.66, mean_step_time 1435.51, it/s 0.7 | |
| [0130 17:59:43 @multigpu.py:323] [p0115] step: count(1222), step_time 1429.09, mean_step_time 1432.76, it/s 0.7 | |
| [0130 17:59:43 @multigpu.py:323] [p0574] step: count(1237), step_time 1417.41, mean_step_time 1449.66, it/s 0.69 | |
| [0130 17:59:45 @multigpu.py:323] [p0576] step: count(1247), step_time 1401.68, mean_step_time 1436.21, it/s 0.7 | |
| [0130 17:59:45 @multigpu.py:323] [p0115] step: count(1223), step_time 1466.01, mean_step_time 1436.82, it/s 0.7 | |
| [0130 17:59:45 @multigpu.py:323] [p0574] step: count(1238), step_time 1439.89, mean_step_time 1450.25, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.495688630276 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 17:59:46 @multigpu.py:323] [p0576] step: count(1248), step_time 1470.4, mean_step_time 1436.92, it/s 0.7 | |
| [0130 17:59:46 @multigpu.py:323] [p0574] step: count(1239), step_time 1392.69, mean_step_time 1448.9, it/s 0.69 | |
| [0130 17:59:46 @multigpu.py:323] [p0115] step: count(1224), step_time 1429.94, mean_step_time 1436.06, it/s 0.7 | |
| [0130 17:59:47 @multigpu.py:323] [p0576] step: count(1249), step_time 1418.38, mean_step_time 1438.99, it/s 0.69 | |
| [0130 17:59:48 @multigpu.py:323] [p0574] step: count(1240), step_time 1410.31, mean_step_time 1448.51, it/s 0.69 | |
| [0130 17:59:48 @multigpu.py:323] [p0115] step: count(1225), step_time 1454.57, mean_step_time 1438.56, it/s 0.7 | |
| [0130 17:59:49 @multigpu.py:323] [p0576] step: count(1250), step_time 1469.11, mean_step_time 1441.18, it/s 0.69 | |
| [0130 17:59:49 @multigpu.py:323] [p0574] step: count(1241), step_time 1455.97, mean_step_time 1450.5, it/s 0.69 | |
| [0130 17:59:49 @multigpu.py:323] [p0115] step: count(1226), step_time 1404.38, mean_step_time 1435.75, it/s 0.7 | |
| [0130 17:59:50 @multigpu.py:323] [p0576] step: count(1251), step_time 1474.91, mean_step_time 1442.45, it/s 0.69 | |
| [0130 17:59:50 @multigpu.py:323] [p0574] step: count(1242), step_time 1392.37, mean_step_time 1447.97, it/s 0.69 | |
| [0130 17:59:51 @multigpu.py:323] [p0115] step: count(1227), step_time 1441.44, mean_step_time 1434.68, it/s 0.7 | |
| [0130 17:59:52 @multigpu.py:323] [p0115] step: count(1228), step_time 1428.7, mean_step_time 1435.61, it/s 0.7 | |
| [0130 17:59:52 @multigpu.py:323] [p0574] step: count(1243), step_time 1497.43, mean_step_time 1435.55, it/s 0.7 | |
| [0130 17:59:52 @multigpu.py:323] [p0576] step: count(1252), step_time 1625.91, mean_step_time 1435.59, it/s 0.7 | |
| [0130 17:59:53 @multigpu.py:323] [p0576] step: count(1253), step_time 1398.41, mean_step_time 1435.33, it/s 0.7 | |
| [0130 17:59:53 @multigpu.py:323] [p0115] step: count(1229), step_time 1408.68, mean_step_time 1435.45, it/s 0.7 | |
| [0130 17:59:53 @multigpu.py:323] [p0574] step: count(1244), step_time 1411.42, mean_step_time 1433.65, it/s 0.7 | |
| [0130 17:59:55 @multigpu.py:323] [p0576] step: count(1254), step_time 1420.87, mean_step_time 1435.73, it/s 0.7 | |
| [0130 17:59:55 @multigpu.py:323] [p0574] step: count(1245), step_time 1447.2, mean_step_time 1435.57, it/s 0.7 | |
| [0130 17:59:55 @multigpu.py:323] [p0115] step: count(1230), step_time 1484.63, mean_step_time 1438.91, it/s 0.69 | |
| [0130 17:59:56 @multigpu.py:323] [p0576] step: count(1255), step_time 1397.69, mean_step_time 1435.09, it/s 0.7 | |
| [0130 17:59:56 @multigpu.py:323] [p0115] step: count(1231), step_time 1391.17, mean_step_time 1434.15, it/s 0.7 | |
| [0130 17:59:56 @multigpu.py:323] [p0574] step: count(1246), step_time 1455.19, mean_step_time 1437.84, it/s 0.7 | |
| [0130 17:59:58 @multigpu.py:323] [p0576] step: count(1256), step_time 1380.83, mean_step_time 1435.26, it/s 0.7 | |
| [0130 17:59:58 @multigpu.py:323] [p0115] step: count(1232), step_time 1437.02, mean_step_time 1433.47, it/s 0.7 | |
| [0130 17:59:58 @multigpu.py:323] [p0574] step: count(1247), step_time 1427.1, mean_step_time 1436.61, it/s 0.7 | |
| [0130 17:59:59 @multigpu.py:323] [p0576] step: count(1257), step_time 1488.55, mean_step_time 1439.34, it/s 0.69 | |
| [0130 17:59:59 @multigpu.py:323] [p0574] step: count(1248), step_time 1399.27, mean_step_time 1432.42, it/s 0.7 | |
| [0130 17:59:59 @multigpu.py:323] [p0115] step: count(1233), step_time 1424.49, mean_step_time 1432.23, it/s 0.7 | |
| [0130 18:00:01 @multigpu.py:323] [p0576] step: count(1258), step_time 1418.93, mean_step_time 1439.15, it/s 0.69 | |
| [0130 18:00:01 @multigpu.py:323] [p0574] step: count(1249), step_time 1382.59, mean_step_time 1428.91, it/s 0.7 | |
| [0130 18:00:01 @multigpu.py:323] [p0115] step: count(1234), step_time 1432.11, mean_step_time 1434.67, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.500219986373 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:00:02 @multigpu.py:323] [p0576] step: count(1259), step_time 1450.22, mean_step_time 1441.66, it/s 0.69 | |
| [0130 18:00:02 @multigpu.py:323] [p0115] step: count(1235), step_time 1417.64, mean_step_time 1434.64, it/s 0.7 | |
| [0130 18:00:02 @multigpu.py:323] [p0574] step: count(1250), step_time 1493.7, mean_step_time 1430.48, it/s 0.7 | |
| [0130 18:00:03 @multigpu.py:323] [p0576] step: count(1260), step_time 1436.07, mean_step_time 1441.31, it/s 0.69 | |
| [0130 18:00:03 @multigpu.py:323] [p0115] step: count(1236), step_time 1446.74, mean_step_time 1435.69, it/s 0.7 | |
| [0130 18:00:03 @multigpu.py:323] [p0574] step: count(1251), step_time 1452.27, mean_step_time 1432.64, it/s 0.7 | |
| [0130 18:00:05 @multigpu.py:323] [p0576] step: count(1261), step_time 1421.8, mean_step_time 1443.15, it/s 0.69 | |
| [0130 18:00:05 @multigpu.py:323] [p0574] step: count(1252), step_time 1412.02, mean_step_time 1433.14, it/s 0.7 | |
| [0130 18:00:05 @multigpu.py:323] [p0115] step: count(1237), step_time 1443.62, mean_step_time 1436.13, it/s 0.7 | |
| [0130 18:00:06 @multigpu.py:323] [p0115] step: count(1238), step_time 1427.58, mean_step_time 1436.15, it/s 0.7 | |
| [0130 18:00:06 @multigpu.py:323] [p0576] step: count(1262), step_time 1502.99, mean_step_time 1446.44, it/s 0.69 | |
| [0130 18:00:06 @multigpu.py:323] [p0574] step: count(1253), step_time 1450.51, mean_step_time 1433.43, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.50163765053 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:00:08 @multigpu.py:323] [p0576] step: count(1263), step_time 1376.02, mean_step_time 1442.31, it/s 0.69 | |
| [0130 18:00:08 @multigpu.py:323] [p0574] step: count(1254), step_time 1377.62, mean_step_time 1431.45, it/s 0.7 | |
| [0130 18:00:08 @multigpu.py:323] [p0115] step: count(1239), step_time 1444.37, mean_step_time 1436.86, it/s 0.7 | |
| [0130 18:00:09 @multigpu.py:323] [p0115] step: count(1240), step_time 1384.45, mean_step_time 1434.65, it/s 0.7 | |
| [0130 18:00:09 @multigpu.py:323] [p0576] step: count(1264), step_time 1465.98, mean_step_time 1443.96, it/s 0.69 | |
| [0130 18:00:09 @multigpu.py:323] [p0574] step: count(1255), step_time 1491.25, mean_step_time 1431.98, it/s 0.7 | |
| [0130 18:00:11 @multigpu.py:323] [p0576] step: count(1265), step_time 1377.57, mean_step_time 1441.35, it/s 0.69 | |
| [0130 18:00:11 @multigpu.py:323] [p0115] step: count(1241), step_time 1427.5, mean_step_time 1431.21, it/s 0.7 | |
| [0130 18:00:11 @multigpu.py:323] [p0574] step: count(1256), step_time 1425.77, mean_step_time 1431.6, it/s 0.7 | |
| [0130 18:00:12 @multigpu.py:323] [p0115] step: count(1242), step_time 1413.5, mean_step_time 1430.43, it/s 0.7 | |
| [0130 18:00:12 @multigpu.py:323] [p0576] step: count(1266), step_time 1485.86, mean_step_time 1444.11, it/s 0.69 | |
| [0130 18:00:12 @multigpu.py:323] [p0574] step: count(1257), step_time 1398.03, mean_step_time 1430.63, it/s 0.7 | |
| [0130 18:00:13 @multigpu.py:323] [p0576] step: count(1267), step_time 1384.89, mean_step_time 1443.27, it/s 0.69 | |
| [0130 18:00:13 @multigpu.py:323] [p0574] step: count(1258), step_time 1404.02, mean_step_time 1428.84, it/s 0.7 | |
| [0130 18:00:13 @multigpu.py:323] [p0115] step: count(1243), step_time 1449.9, mean_step_time 1429.62, it/s 0.7 | |
| [0130 18:00:15 @multigpu.py:323] [p0115] step: count(1244), step_time 1399.48, mean_step_time 1428.1, it/s 0.7 | |
| [0130 18:00:15 @multigpu.py:323] [p0574] step: count(1259), step_time 1415.75, mean_step_time 1429.99, it/s 0.7 | |
| [0130 18:00:15 @multigpu.py:323] [p0576] step: count(1268), step_time 1455.89, mean_step_time 1442.54, it/s 0.69 | |
| [0130 18:00:16 @multigpu.py:323] [p0115] step: count(1245), step_time 1399.33, mean_step_time 1425.34, it/s 0.7 | |
| [0130 18:00:16 @multigpu.py:323] [p0574] step: count(1260), step_time 1419.75, mean_step_time 1430.46, it/s 0.7 | |
| [0130 18:00:16 @multigpu.py:323] [p0576] step: count(1269), step_time 1447.95, mean_step_time 1444.02, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.504519854983 , 1.2 ##### | |
| [u'online', 1.2] | |
| ##### Sending to neptune: active_workers : 0.504519919422 , 3 ##### | |
| receiving | |
| [0130 18:00:18 @multigpu.py:323] [p0115] step: count(1246), step_time 1424.38, mean_step_time 1426.34, it/s 0.7 | |
| [0130 18:00:18 @multigpu.py:323] [p0574] step: count(1261), step_time 1404.1, mean_step_time 1427.87, it/s 0.7 | |
| [0130 18:00:18 @multigpu.py:323] [p0576] step: count(1270), step_time 1404.17, mean_step_time 1440.78, it/s 0.69 | |
| [0130 18:00:19 @multigpu.py:323] [p0574] step: count(1262), step_time 1389.37, mean_step_time 1427.72, it/s 0.7 | |
| [0130 18:00:19 @multigpu.py:323] [p0115] step: count(1247), step_time 1435.36, mean_step_time 1426.03, it/s 0.7 | |
| [0130 18:00:19 @multigpu.py:323] [p0576] step: count(1271), step_time 1398.44, mean_step_time 1436.95, it/s 0.7 | |
| [0130 18:00:21 @multigpu.py:323] [p0115] step: count(1248), step_time 1413.05, mean_step_time 1425.25, it/s 0.7 | |
| [0130 18:00:21 @multigpu.py:323] [p0576] step: count(1272), step_time 1389.99, mean_step_time 1425.16, it/s 0.7 | |
| [0130 18:00:21 @multigpu.py:323] [p0574] step: count(1263), step_time 1449.4, mean_step_time 1425.32, it/s 0.7 | |
| [0130 18:00:22 @multigpu.py:323] [p0576] step: count(1273), step_time 1400.21, mean_step_time 1425.25, it/s 0.7 | |
| [0130 18:00:22 @multigpu.py:323] [p0574] step: count(1264), step_time 1407.21, mean_step_time 1425.11, it/s 0.7 | |
| [0130 18:00:22 @multigpu.py:323] [p0115] step: count(1249), step_time 1461.25, mean_step_time 1427.88, it/s 0.7 | |
| [0130 18:00:23 @multigpu.py:323] [p0576] step: count(1274), step_time 1490.45, mean_step_time 1428.73, it/s 0.7 | |
| [0130 18:00:23 @multigpu.py:323] [p0115] step: count(1250), step_time 1429.73, mean_step_time 1425.13, it/s 0.7 | |
| [0130 18:00:23 @multigpu.py:323] [p0574] step: count(1265), step_time 1537.12, mean_step_time 1429.6, it/s 0.7 | |
| [0130 18:00:25 @multigpu.py:323] [p0576] step: count(1275), step_time 1365.34, mean_step_time 1427.11, it/s 0.7 | |
| [0130 18:00:25 @multigpu.py:323] [p0115] step: count(1251), step_time 1407.17, mean_step_time 1425.93, it/s 0.7 | |
| [0130 18:00:25 @multigpu.py:323] [p0574] step: count(1266), step_time 1356.73, mean_step_time 1424.68, it/s 0.7 | |
| [0130 18:00:26 @multigpu.py:323] [p0576] step: count(1276), step_time 1412.74, mean_step_time 1428.7, it/s 0.7 | |
| [0130 18:00:26 @multigpu.py:323] [p0115] step: count(1252), step_time 1420.38, mean_step_time 1425.1, it/s 0.7 | |
| [0130 18:00:26 @multigpu.py:323] [p0574] step: count(1267), step_time 1491.01, mean_step_time 1427.87, it/s 0.7 | |
| [0130 18:00:28 @multigpu.py:323] [p0576] step: count(1277), step_time 1476.89, mean_step_time 1428.12, it/s 0.7 | |
| [0130 18:00:28 @multigpu.py:323] [p0115] step: count(1253), step_time 1436.83, mean_step_time 1425.72, it/s 0.7 | |
| [0130 18:00:28 @multigpu.py:323] [p0574] step: count(1268), step_time 1375.61, mean_step_time 1426.69, it/s 0.7 | |
| [0130 18:00:29 @multigpu.py:323] [p0576] step: count(1278), step_time 1414.23, mean_step_time 1427.89, it/s 0.7 | |
| [0130 18:00:29 @multigpu.py:323] [p0115] step: count(1254), step_time 1456.16, mean_step_time 1426.92, it/s 0.7 | |
| [0130 18:00:29 @multigpu.py:323] [p0574] step: count(1269), step_time 1470.79, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:00:31 @multigpu.py:323] [p0576] step: count(1279), step_time 1439.79, mean_step_time 1427.36, it/s 0.7 | |
| [0130 18:00:31 @multigpu.py:323] [p0115] step: count(1255), step_time 1390.55, mean_step_time 1425.57, it/s 0.7 | |
| [0130 18:00:31 @multigpu.py:323] [p0574] step: count(1270), step_time 1458.49, mean_step_time 1429.34, it/s 0.7 | |
| [0130 18:00:32 @multigpu.py:323] [p0576] step: count(1280), step_time 1430.71, mean_step_time 1427.1, it/s 0.7 | |
| [0130 18:00:32 @multigpu.py:323] [p0115] step: count(1256), step_time 1428.18, mean_step_time 1424.64, it/s 0.7 | |
| [0130 18:00:32 @multigpu.py:323] [p0574] step: count(1271), step_time 1435.24, mean_step_time 1428.49, it/s 0.7 | |
| [0130 18:00:33 @multigpu.py:323] [p0115] step: count(1257), step_time 1449.35, mean_step_time 1424.92, it/s 0.7 | |
| [0130 18:00:33 @multigpu.py:323] [p0576] step: count(1281), step_time 1465.3, mean_step_time 1429.27, it/s 0.7 | |
| [0130 18:00:33 @multigpu.py:323] [p0574] step: count(1272), step_time 1448.46, mean_step_time 1430.31, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.509209632476 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.509221385519 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:00:35 @multigpu.py:323] [p0576] step: count(1282), step_time 1407.69, mean_step_time 1424.51, it/s 0.7 | |
| [0130 18:00:35 @multigpu.py:323] [p0115] step: count(1258), step_time 1431.65, mean_step_time 1425.13, it/s 0.7 | |
| [0130 18:00:35 @multigpu.py:323] [p0574] step: count(1273), step_time 1424.04, mean_step_time 1428.99, it/s 0.7 | |
| [0130 18:00:36 @multigpu.py:323] [p0115] step: count(1259), step_time 1441.86, mean_step_time 1425.0, it/s 0.7 | |
| [0130 18:00:36 @multigpu.py:323] [p0576] step: count(1283), step_time 1467.47, mean_step_time 1429.08, it/s 0.7 | |
| [0130 18:00:36 @multigpu.py:323] [p0574] step: count(1274), step_time 1414.32, mean_step_time 1430.82, it/s 0.7 | |
| [0130 18:00:38 @multigpu.py:323] [p0115] step: count(1260), step_time 1417.55, mean_step_time 1426.66, it/s 0.7 | |
| [0130 18:00:38 @multigpu.py:323] [p0576] step: count(1284), step_time 1448.14, mean_step_time 1428.19, it/s 0.7 | |
| [0130 18:00:38 @multigpu.py:323] [p0574] step: count(1275), step_time 1415.54, mean_step_time 1427.04, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.510377633837 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 18:00:39 @multigpu.py:323] [p0115] step: count(1261), step_time 1374.91, mean_step_time 1424.03, it/s 0.7 | |
| [0130 18:00:39 @multigpu.py:323] [p0576] step: count(1285), step_time 1424.08, mean_step_time 1430.51, it/s 0.7 | |
| [0130 18:00:39 @multigpu.py:323] [p0574] step: count(1276), step_time 1443.27, mean_step_time 1427.91, it/s 0.7 | |
| [0130 18:00:40 @multigpu.py:323] [p0115] step: count(1262), step_time 1429.56, mean_step_time 1424.83, it/s 0.7 | |
| [0130 18:00:41 @multigpu.py:323] [p0576] step: count(1286), step_time 1391.27, mean_step_time 1425.78, it/s 0.7 | |
| [0130 18:00:41 @multigpu.py:323] [p0574] step: count(1277), step_time 1455.26, mean_step_time 1430.77, it/s 0.7 | |
| [0130 18:00:42 @multigpu.py:323] [p0115] step: count(1263), step_time 1416.86, mean_step_time 1423.18, it/s 0.7 | |
| [0130 18:00:42 @multigpu.py:323] [p0576] step: count(1287), step_time 1418.46, mean_step_time 1427.46, it/s 0.7 | |
| [0130 18:00:42 @multigpu.py:323] [p0574] step: count(1278), step_time 1389.05, mean_step_time 1430.03, it/s 0.7 | |
| [0130 18:00:43 @multigpu.py:323] [p0115] step: count(1264), step_time 1434.03, mean_step_time 1424.91, it/s 0.7 | |
| [0130 18:00:43 @multigpu.py:323] [p0576] step: count(1288), step_time 1383.01, mean_step_time 1423.82, it/s 0.7 | |
| [0130 18:00:44 @multigpu.py:323] [p0574] step: count(1279), step_time 1482.24, mean_step_time 1433.35, it/s 0.7 | |
| [0130 18:00:45 @multigpu.py:323] [p0576] step: count(1289), step_time 1412.39, mean_step_time 1422.04, it/s 0.7 | |
| [0130 18:00:45 @multigpu.py:323] [p0115] step: count(1265), step_time 1457.3, mean_step_time 1427.8, it/s 0.7 | |
| [0130 18:00:45 @multigpu.py:323] [p0574] step: count(1280), step_time 1369.54, mean_step_time 1430.84, it/s 0.7 | |
| [0130 18:00:46 @multigpu.py:323] [p0576] step: count(1290), step_time 1454.01, mean_step_time 1424.53, it/s 0.7 | |
| [0130 18:00:46 @multigpu.py:323] [p0115] step: count(1266), step_time 1466.19, mean_step_time 1429.9, it/s 0.7 | |
| [0130 18:00:46 @multigpu.py:323] [p0574] step: count(1281), step_time 1429.17, mean_step_time 1432.09, it/s 0.7 | |
| [0130 18:00:48 @multigpu.py:323] [p0576] step: count(1291), step_time 1417.98, mean_step_time 1425.51, it/s 0.7 | |
| [0130 18:00:48 @multigpu.py:323] [p0115] step: count(1267), step_time 1467.76, mean_step_time 1431.52, it/s 0.7 | |
| [0130 18:00:48 @multigpu.py:323] [p0574] step: count(1282), step_time 1453.63, mean_step_time 1435.31, it/s 0.7 | |
| [0130 18:00:49 @multigpu.py:323] [p0115] step: count(1268), step_time 1453.14, mean_step_time 1433.52, it/s 0.7 | |
| [0130 18:00:49 @multigpu.py:323] [p0574] step: count(1283), step_time 1408.98, mean_step_time 1433.28, it/s 0.7 | |
| [0130 18:00:49 @multigpu.py:323] [p0576] step: count(1292), step_time 1550.01, mean_step_time 1433.51, it/s 0.7 | |
| [0130 18:00:51 @multigpu.py:323] [p0576] step: count(1293), step_time 1398.16, mean_step_time 1433.41, it/s 0.7 | |
| [0130 18:00:51 @multigpu.py:323] [p0115] step: count(1269), step_time 1414.4, mean_step_time 1431.18, it/s 0.7 | |
| [0130 18:00:51 @multigpu.py:323] [p0574] step: count(1284), step_time 1436.26, mean_step_time 1434.74, it/s 0.7 | |
| [0130 18:00:52 @multigpu.py:323] [p0115] step: count(1270), step_time 1387.23, mean_step_time 1429.05, it/s 0.7 | |
| [0130 18:00:52 @multigpu.py:323] [p0576] step: count(1294), step_time 1448.65, mean_step_time 1431.32, it/s 0.7 | |
| [0130 18:00:52 @multigpu.py:323] [p0574] step: count(1285), step_time 1440.68, mean_step_time 1429.91, it/s 0.7 | |
| [0130 18:00:53 @multigpu.py:323] [p0115] step: count(1271), step_time 1413.52, mean_step_time 1429.37, it/s 0.7 | |
| [0130 18:00:53 @multigpu.py:323] [p0574] step: count(1286), step_time 1391.43, mean_step_time 1431.65, it/s 0.7 | |
| [0130 18:00:53 @multigpu.py:323] [p0576] step: count(1295), step_time 1450.11, mean_step_time 1435.56, it/s 0.7 | |
| [0130 18:00:55 @multigpu.py:323] [p0115] step: count(1272), step_time 1425.58, mean_step_time 1429.63, it/s 0.7 | |
| [0130 18:00:55 @multigpu.py:323] [p0574] step: count(1287), step_time 1399.37, mean_step_time 1427.07, it/s 0.7 | |
| [0130 18:00:55 @multigpu.py:323] [p0576] step: count(1296), step_time 1421.49, mean_step_time 1435.99, it/s 0.7 | |
| [0130 18:00:56 @multigpu.py:323] [p0115] step: count(1273), step_time 1414.14, mean_step_time 1428.5, it/s 0.7 | |
| [0130 18:00:56 @multigpu.py:323] [p0574] step: count(1288), step_time 1441.05, mean_step_time 1430.34, it/s 0.7 | |
| [0130 18:00:56 @multigpu.py:323] [p0576] step: count(1297), step_time 1401.6, mean_step_time 1432.23, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.515617745519 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 18:00:58 @multigpu.py:323] [p0115] step: count(1274), step_time 1429.4, mean_step_time 1427.16, it/s 0.7 | |
| [0130 18:00:58 @multigpu.py:323] [p0576] step: count(1298), step_time 1386.47, mean_step_time 1430.84, it/s 0.7 | |
| [0130 18:00:58 @multigpu.py:323] [p0574] step: count(1289), step_time 1408.34, mean_step_time 1427.22, it/s 0.7 | |
| [0130 18:00:59 @multigpu.py:323] [p0576] step: count(1299), step_time 1403.87, mean_step_time 1429.04, it/s 0.7 | |
| [0130 18:00:59 @multigpu.py:323] [p0115] step: count(1275), step_time 1463.58, mean_step_time 1430.81, it/s 0.7 | |
| [0130 18:00:59 @multigpu.py:323] [p0574] step: count(1290), step_time 1480.56, mean_step_time 1428.32, it/s 0.7 | |
| [0130 18:01:01 @multigpu.py:323] [p0576] step: count(1300), step_time 1430.43, mean_step_time 1429.03, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.516647246612 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.516647246612 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.516647246612 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.51664776716 , -0.00946542713791 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.51664776716 , -0.313773274422 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.51664776716 , -2.28776264191 ##### | |
| ##### Sending to neptune: value_loss : 0.51664776716 , 1.38996088505 ##### | |
| ##### Sending to neptune: advantage : 0.51664776716 , 0.00140806438867 ##### | |
| ##### Sending to neptune: pred_reward : 0.51664776716 , 0.47912633419 ##### | |
| ##### Sending to neptune: max_logit : 0.51664776716 , 0.194705918431 ##### | |
| [u'loss', -0.00946542713791132, -0.3137732744216919, -2.2877626419067383, 1.3899608850479126, 0.0014080643886700273, 0.47912633419036865, 0.19470591843128204] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.516648219956 , 9676168.09 ##### | |
| ##### Sending to neptune: dp_per_s : 0.516648219956 , 89.1823527596 ##### | |
| [u'other', 9676168.09, 89.18235275962654] | |
| receiving | |
| [0130 18:01:01 @multigpu.py:323] [p0574] step: count(1291), step_time 1398.52, mean_step_time 1426.49, it/s 0.7 | |
| [0130 18:01:01 @multigpu.py:323] [p0115] step: count(1276), step_time 1465.75, mean_step_time 1432.69, it/s 0.7 | |
| [0130 18:01:02 @multigpu.py:323] [p0576] step: count(1301), step_time 1438.25, mean_step_time 1427.68, it/s 0.7 | |
| [0130 18:01:02 @multigpu.py:323] [p0574] step: count(1292), step_time 1438.86, mean_step_time 1426.0, it/s 0.7 | |
| [0130 18:01:02 @multigpu.py:323] [p0115] step: count(1277), step_time 1457.61, mean_step_time 1433.1, it/s 0.7 | |
| [0130 18:01:03 @multigpu.py:323] [p0576] step: count(1302), step_time 1384.81, mean_step_time 1426.53, it/s 0.7 | |
| [0130 18:01:03 @multigpu.py:323] [p0115] step: count(1278), step_time 1413.94, mean_step_time 1432.22, it/s 0.7 | |
| [0130 18:01:03 @multigpu.py:323] [p0574] step: count(1293), step_time 1452.43, mean_step_time 1427.42, it/s 0.7 | |
| [0130 18:01:05 @multigpu.py:323] [p0576] step: count(1303), step_time 1399.88, mean_step_time 1423.15, it/s 0.7 | |
| [0130 18:01:05 @multigpu.py:323] [p0574] step: count(1294), step_time 1376.51, mean_step_time 1425.53, it/s 0.7 | |
| [0130 18:01:05 @multigpu.py:323] [p0115] step: count(1279), step_time 1481.14, mean_step_time 1434.18, it/s 0.7 | |
| [0130 18:01:06 @multigpu.py:323] [p0576] step: count(1304), step_time 1466.55, mean_step_time 1424.07, it/s 0.7 | |
| [0130 18:01:06 @multigpu.py:323] [p0574] step: count(1295), step_time 1445.89, mean_step_time 1427.05, it/s 0.7 | |
| [0130 18:01:06 @multigpu.py:323] [p0115] step: count(1280), step_time 1414.5, mean_step_time 1434.03, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.518410489957 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 18:01:08 @multigpu.py:323] [p0576] step: count(1305), step_time 1397.55, mean_step_time 1422.75, it/s 0.7 | |
| [0130 18:01:08 @multigpu.py:323] [p0574] step: count(1296), step_time 1387.43, mean_step_time 1424.26, it/s 0.7 | |
| [0130 18:01:08 @multigpu.py:323] [p0115] step: count(1281), step_time 1412.38, mean_step_time 1435.9, it/s 0.7 | |
| [0130 18:01:09 @multigpu.py:323] [p0576] step: count(1306), step_time 1436.32, mean_step_time 1425.0, it/s 0.7 | |
| [0130 18:01:09 @multigpu.py:323] [p0574] step: count(1297), step_time 1474.04, mean_step_time 1425.2, it/s 0.7 | |
| [0130 18:01:09 @multigpu.py:323] [p0115] step: count(1282), step_time 1413.31, mean_step_time 1435.09, it/s 0.7 | |
| [0130 18:01:10 @multigpu.py:323] [p0576] step: count(1307), step_time 1420.68, mean_step_time 1425.11, it/s 0.7 | |
| [0130 18:01:11 @multigpu.py:323] [p0115] step: count(1283), step_time 1401.14, mean_step_time 1434.3, it/s 0.7 | |
| [0130 18:01:11 @multigpu.py:323] [p0574] step: count(1298), step_time 1436.18, mean_step_time 1427.56, it/s 0.7 | |
| [0130 18:01:12 @multigpu.py:323] [p0576] step: count(1308), step_time 1451.23, mean_step_time 1428.52, it/s 0.7 | |
| [0130 18:01:12 @multigpu.py:323] [p0574] step: count(1299), step_time 1454.11, mean_step_time 1426.15, it/s 0.7 | |
| [0130 18:01:12 @multigpu.py:323] [p0115] step: count(1284), step_time 1456.81, mean_step_time 1435.44, it/s 0.7 | |
| [0130 18:01:13 @multigpu.py:323] [p0576] step: count(1309), step_time 1384.38, mean_step_time 1427.12, it/s 0.7 | |
| [0130 18:01:13 @multigpu.py:323] [p0115] step: count(1285), step_time 1412.14, mean_step_time 1433.18, it/s 0.7 | |
| [0130 18:01:14 @multigpu.py:323] [p0574] step: count(1300), step_time 1449.74, mean_step_time 1430.16, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.520251732734 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.520251732734 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.520251732734 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.520252199107 , -0.00133042002562 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.520252199107 , 0.530481159687 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.520252199107 , -2.28754925728 ##### | |
| ##### Sending to neptune: value_loss : 0.520252199107 , 1.58677446842 ##### | |
| ##### Sending to neptune: advantage : 0.520252199107 , -0.0023368424736 ##### | |
| ##### Sending to neptune: pred_reward : 0.520252199107 , 0.480544269085 ##### | |
| ##### Sending to neptune: max_logit : 0.520252199107 , 0.196291729808 ##### | |
| [u'loss', -0.0013304200256243348, 0.5304811596870422, -2.2875492572784424, 1.586774468421936, -0.0023368424735963345, 0.4805442690849304, 0.1962917298078537] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.52025263634 , 9689908.7 ##### | |
| ##### Sending to neptune: dp_per_s : 0.52025263634 , 89.2147275754 ##### | |
| [u'other', 9689908.7, 89.21472757536068] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.520315144393 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 18:01:15 @multigpu.py:323] [p0576] step: count(1310), step_time 1453.13, mean_step_time 1427.08, it/s 0.7 | |
| [0130 18:01:15 @multigpu.py:323] [p0115] step: count(1286), step_time 1374.66, mean_step_time 1428.61, it/s 0.7 | |
| [0130 18:01:15 @multigpu.py:323] [p0574] step: count(1301), step_time 1403.41, mean_step_time 1428.87, it/s 0.7 | |
| [0130 18:01:16 @multigpu.py:323] [p0576] step: count(1311), step_time 1410.4, mean_step_time 1426.7, it/s 0.7 | |
| [0130 18:01:16 @multigpu.py:323] [p0574] step: count(1302), step_time 1379.33, mean_step_time 1425.16, it/s 0.7 | |
| [0130 18:01:16 @multigpu.py:323] [p0115] step: count(1287), step_time 1459.62, mean_step_time 1428.2, it/s 0.7 | |
| [0130 18:01:18 @multigpu.py:323] [p0115] step: count(1288), step_time 1434.8, mean_step_time 1427.28, it/s 0.7 | |
| [0130 18:01:18 @multigpu.py:323] [p0574] step: count(1303), step_time 1451.91, mean_step_time 1427.3, it/s 0.7 | |
| [0130 18:01:18 @multigpu.py:323] [p0576] step: count(1312), step_time 1563.34, mean_step_time 1427.37, it/s 0.7 | |
| [0130 18:01:19 @multigpu.py:323] [p0115] step: count(1289), step_time 1395.34, mean_step_time 1426.33, it/s 0.7 | |
| [0130 18:01:19 @multigpu.py:323] [p0576] step: count(1313), step_time 1409.53, mean_step_time 1427.93, it/s 0.7 | |
| [0130 18:01:19 @multigpu.py:323] [p0574] step: count(1304), step_time 1418.34, mean_step_time 1426.41, it/s 0.7 | |
| [0130 18:01:21 @multigpu.py:323] [p0574] step: count(1305), step_time 1398.31, mean_step_time 1424.29, it/s 0.7 | |
| [0130 18:01:21 @multigpu.py:323] [p0576] step: count(1314), step_time 1409.09, mean_step_time 1425.96, it/s 0.7 | |
| [0130 18:01:21 @multigpu.py:323] [p0115] step: count(1290), step_time 1429.8, mean_step_time 1428.46, it/s 0.7 | |
| [0130 18:01:22 @multigpu.py:323] [p0576] step: count(1315), step_time 1402.83, mean_step_time 1423.59, it/s 0.7 | |
| [0130 18:01:22 @multigpu.py:323] [p0115] step: count(1291), step_time 1396.94, mean_step_time 1427.63, it/s 0.7 | |
| [0130 18:01:22 @multigpu.py:323] [p0574] step: count(1306), step_time 1445.26, mean_step_time 1426.98, it/s 0.7 | |
| [0130 18:01:23 @multigpu.py:323] [p0576] step: count(1316), step_time 1414.48, mean_step_time 1423.24, it/s 0.7 | |
| [0130 18:01:23 @multigpu.py:323] [p0115] step: count(1292), step_time 1442.41, mean_step_time 1428.47, it/s 0.7 | |
| [0130 18:01:23 @multigpu.py:323] [p0574] step: count(1307), step_time 1436.64, mean_step_time 1428.84, it/s 0.7 | |
| [0130 18:01:25 @multigpu.py:323] [p0115] step: count(1293), step_time 1459.15, mean_step_time 1430.72, it/s 0.7 | |
| [0130 18:01:25 @multigpu.py:323] [p0574] step: count(1308), step_time 1428.54, mean_step_time 1428.22, it/s 0.7 | |
| [0130 18:01:25 @multigpu.py:323] [p0576] step: count(1317), step_time 1506.03, mean_step_time 1428.46, it/s 0.7 | |
| [0130 18:01:26 @multigpu.py:323] [p0115] step: count(1294), step_time 1388.66, mean_step_time 1428.68, it/s 0.7 | |
| [0130 18:01:26 @multigpu.py:323] [p0574] step: count(1309), step_time 1408.21, mean_step_time 1428.21, it/s 0.7 | |
| [0130 18:01:26 @multigpu.py:323] [p0576] step: count(1318), step_time 1477.43, mean_step_time 1433.01, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.523968138827 , 0.8 ##### | |
| [u'online', 0.8] | |
| ##### Sending to neptune: active_workers : 0.523968183862 , 3 ##### | |
| receiving | |
| [0130 18:01:28 @multigpu.py:323] [p0574] step: count(1310), step_time 1415.15, mean_step_time 1424.94, it/s 0.7 | |
| [0130 18:01:28 @multigpu.py:323] [p0115] step: count(1295), step_time 1442.54, mean_step_time 1427.63, it/s 0.7 | |
| [0130 18:01:28 @multigpu.py:323] [p0576] step: count(1319), step_time 1366.87, mean_step_time 1431.16, it/s 0.7 | |
| [0130 18:01:29 @multigpu.py:323] [p0115] step: count(1296), step_time 1352.47, mean_step_time 1421.97, it/s 0.7 | |
| [0130 18:01:29 @multigpu.py:323] [p0576] step: count(1320), step_time 1396.82, mean_step_time 1429.48, it/s 0.7 | |
| [0130 18:01:29 @multigpu.py:323] [p0574] step: count(1311), step_time 1455.72, mean_step_time 1427.8, it/s 0.7 | |
| [0130 18:01:31 @multigpu.py:323] [p0115] step: count(1297), step_time 1460.09, mean_step_time 1422.09, it/s 0.7 | |
| [0130 18:01:31 @multigpu.py:323] [p0576] step: count(1321), step_time 1402.61, mean_step_time 1427.7, it/s 0.7 | |
| [0130 18:01:31 @multigpu.py:323] [p0574] step: count(1312), step_time 1426.71, mean_step_time 1427.19, it/s 0.7 | |
| [0130 18:01:32 @multigpu.py:323] [p0576] step: count(1322), step_time 1393.86, mean_step_time 1428.15, it/s 0.7 | |
| [0130 18:01:32 @multigpu.py:323] [p0115] step: count(1298), step_time 1479.78, mean_step_time 1425.38, it/s 0.7 | |
| [0130 18:01:32 @multigpu.py:323] [p0574] step: count(1313), step_time 1424.25, mean_step_time 1425.79, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.525484263566 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:01:33 @multigpu.py:323] [p0576] step: count(1323), step_time 1420.64, mean_step_time 1429.19, it/s 0.7 | |
| [0130 18:01:33 @multigpu.py:323] [p0115] step: count(1299), step_time 1479.2, mean_step_time 1425.29, it/s 0.7 | |
| [0130 18:01:34 @multigpu.py:323] [p0574] step: count(1314), step_time 1495.65, mean_step_time 1431.74, it/s 0.7 | |
| [0130 18:01:35 @multigpu.py:323] [p0576] step: count(1324), step_time 1431.46, mean_step_time 1427.43, it/s 0.7 | |
| [0130 18:01:35 @multigpu.py:323] [p0574] step: count(1315), step_time 1379.58, mean_step_time 1428.43, it/s 0.7 | |
| [0130 18:01:35 @multigpu.py:323] [p0115] step: count(1300), step_time 1504.77, mean_step_time 1429.8, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.526216461658 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.526216461658 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.526216461658 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.526216974987 , -0.0132103003561 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.526216974987 , -0.767839074135 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.526216974987 , -2.2872877121 ##### | |
| ##### Sending to neptune: value_loss : 0.526216974987 , 1.36420881748 ##### | |
| ##### Sending to neptune: advantage : 0.526216974987 , 0.00328848976642 ##### | |
| ##### Sending to neptune: pred_reward : 0.526216974987 , 0.47394412756 ##### | |
| ##### Sending to neptune: max_logit : 0.526216974987 , 0.197944834828 ##### | |
| [u'loss', -0.013210300356149673, -0.7678390741348267, -2.287287712097168, 1.3642088174819946, 0.003288489766418934, 0.47394412755966187, 0.19794483482837677] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.526217454142 , 9719758.83 ##### | |
| ##### Sending to neptune: dp_per_s : 0.526217454142 , 89.3935747297 ##### | |
| [u'other', 9719758.83, 89.39357472974253] | |
| receiving | |
| [0130 18:01:36 @multigpu.py:323] [p0576] step: count(1325), step_time 1412.64, mean_step_time 1428.19, it/s 0.7 | |
| [0130 18:01:36 @multigpu.py:323] [p0574] step: count(1316), step_time 1420.66, mean_step_time 1430.09, it/s 0.7 | |
| [0130 18:01:36 @multigpu.py:323] [p0115] step: count(1301), step_time 1411.45, mean_step_time 1429.75, it/s 0.7 | |
| [0130 18:01:38 @multigpu.py:323] [p0576] step: count(1326), step_time 1386.73, mean_step_time 1425.71, it/s 0.7 | |
| [0130 18:01:38 @multigpu.py:323] [p0574] step: count(1317), step_time 1391.52, mean_step_time 1425.96, it/s 0.7 | |
| [0130 18:01:38 @multigpu.py:323] [p0115] step: count(1302), step_time 1399.21, mean_step_time 1429.05, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.527047368553 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 18:01:39 @multigpu.py:323] [p0576] step: count(1327), step_time 1457.68, mean_step_time 1427.56, it/s 0.7 | |
| [0130 18:01:39 @multigpu.py:323] [p0574] step: count(1318), step_time 1424.61, mean_step_time 1425.38, it/s 0.7 | |
| [0130 18:01:39 @multigpu.py:323] [p0115] step: count(1303), step_time 1447.28, mean_step_time 1431.36, it/s 0.7 | |
| [0130 18:01:41 @multigpu.py:323] [p0576] step: count(1328), step_time 1488.34, mean_step_time 1429.41, it/s 0.7 | |
| [0130 18:01:41 @multigpu.py:323] [p0574] step: count(1319), step_time 1429.17, mean_step_time 1424.14, it/s 0.7 | |
| [0130 18:01:41 @multigpu.py:323] [p0115] step: count(1304), step_time 1399.1, mean_step_time 1428.47, it/s 0.7 | |
| [0130 18:01:42 @multigpu.py:323] [p0576] step: count(1329), step_time 1403.8, mean_step_time 1430.39, it/s 0.7 | |
| [0130 18:01:42 @multigpu.py:323] [p0574] step: count(1320), step_time 1445.3, mean_step_time 1423.91, it/s 0.7 | |
| [0130 18:01:42 @multigpu.py:323] [p0115] step: count(1305), step_time 1497.48, mean_step_time 1432.74, it/s 0.7 | |
| [0130 18:01:43 @multigpu.py:323] [p0576] step: count(1330), step_time 1451.89, mean_step_time 1430.32, it/s 0.7 | |
| [0130 18:01:43 @multigpu.py:323] [p0574] step: count(1321), step_time 1398.76, mean_step_time 1423.68, it/s 0.7 | |
| [0130 18:01:44 @multigpu.py:323] [p0115] step: count(1306), step_time 1401.08, mean_step_time 1434.06, it/s 0.7 | |
| [0130 18:01:45 @multigpu.py:323] [p0576] step: count(1331), step_time 1440.01, mean_step_time 1431.8, it/s 0.7 | |
| [0130 18:01:45 @multigpu.py:323] [p0574] step: count(1322), step_time 1454.99, mean_step_time 1427.46, it/s 0.7 | |
| [0130 18:01:45 @multigpu.py:323] [p0115] step: count(1307), step_time 1458.27, mean_step_time 1433.99, it/s 0.7 | |
| [0130 18:01:46 @multigpu.py:323] [p0574] step: count(1323), step_time 1604.47, mean_step_time 1435.09, it/s 0.7 | |
| [0130 18:01:46 @multigpu.py:323] [p0576] step: count(1332), step_time 1627.95, mean_step_time 1435.03, it/s 0.7 | |
| [0130 18:01:46 @multigpu.py:323] [p0115] step: count(1308), step_time 1458.09, mean_step_time 1435.16, it/s 0.7 | |
| [0130 18:01:48 @multigpu.py:323] [p0576] step: count(1333), step_time 1431.92, mean_step_time 1436.15, it/s 0.7 | |
| [0130 18:01:48 @multigpu.py:323] [p0574] step: count(1324), step_time 1443.86, mean_step_time 1436.37, it/s 0.7 | |
| [0130 18:01:48 @multigpu.py:323] [p0115] step: count(1309), step_time 1452.55, mean_step_time 1438.02, it/s 0.7 | |
| [0130 18:01:49 @multigpu.py:323] [p0115] step: count(1310), step_time 1416.31, mean_step_time 1437.34, it/s 0.7 | |
| [0130 18:01:49 @multigpu.py:323] [p0576] step: count(1334), step_time 1437.23, mean_step_time 1437.56, it/s 0.7 | |
| [0130 18:01:49 @multigpu.py:323] [p0574] step: count(1325), step_time 1444.67, mean_step_time 1438.69, it/s 0.7 | |
| [0130 18:01:51 @multigpu.py:323] [p0574] step: count(1326), step_time 1445.52, mean_step_time 1438.7, it/s 0.7 | |
| [0130 18:01:51 @multigpu.py:323] [p0576] step: count(1335), step_time 1479.94, mean_step_time 1441.42, it/s 0.69 | |
| [0130 18:01:51 @multigpu.py:323] [p0115] step: count(1311), step_time 1513.3, mean_step_time 1443.16, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.530835908585 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:01:52 @multigpu.py:323] [p0574] step: count(1327), step_time 1439.03, mean_step_time 1438.82, it/s 0.7 | |
| [0130 18:01:52 @multigpu.py:323] [p0115] step: count(1312), step_time 1434.38, mean_step_time 1442.76, it/s 0.69 | |
| [0130 18:01:52 @multigpu.py:323] [p0576] step: count(1336), step_time 1471.0, mean_step_time 1444.24, it/s 0.69 | |
| [0130 18:01:54 @multigpu.py:323] [p0574] step: count(1328), step_time 1425.5, mean_step_time 1438.67, it/s 0.7 | |
| [0130 18:01:54 @multigpu.py:323] [p0115] step: count(1313), step_time 1428.68, mean_step_time 1441.23, it/s 0.69 | |
| [0130 18:01:54 @multigpu.py:323] [p0576] step: count(1337), step_time 1450.34, mean_step_time 1441.46, it/s 0.69 | |
| [0130 18:01:55 @multigpu.py:323] [p0574] step: count(1329), step_time 1386.17, mean_step_time 1437.56, it/s 0.7 | |
| [0130 18:01:55 @multigpu.py:323] [p0115] step: count(1314), step_time 1403.13, mean_step_time 1441.96, it/s 0.69 | |
| [0130 18:01:55 @multigpu.py:323] [p0576] step: count(1338), step_time 1405.76, mean_step_time 1437.87, it/s 0.7 | |
| [0130 18:01:57 @multigpu.py:323] [p0574] step: count(1330), step_time 1465.31, mean_step_time 1440.07, it/s 0.69 | |
| [0130 18:01:57 @multigpu.py:323] [p0576] step: count(1339), step_time 1444.95, mean_step_time 1441.78, it/s 0.69 | |
| [0130 18:01:57 @multigpu.py:323] [p0115] step: count(1315), step_time 1514.84, mean_step_time 1445.57, it/s 0.69 | |
| [0130 18:01:58 @multigpu.py:323] [p0574] step: count(1331), step_time 1451.43, mean_step_time 1439.86, it/s 0.69 | |
| [0130 18:01:58 @multigpu.py:323] [p0576] step: count(1340), step_time 1451.64, mean_step_time 1444.52, it/s 0.69 | |
| [0130 18:01:58 @multigpu.py:323] [p0115] step: count(1316), step_time 1430.77, mean_step_time 1449.49, it/s 0.69 | |
| [0130 18:01:59 @multigpu.py:323] [p0574] step: count(1332), step_time 1405.66, mean_step_time 1438.8, it/s 0.7 | |
| [0130 18:01:59 @multigpu.py:323] [p0576] step: count(1341), step_time 1411.03, mean_step_time 1444.94, it/s 0.69 | |
| [0130 18:02:00 @multigpu.py:323] [p0115] step: count(1317), step_time 1455.08, mean_step_time 1449.24, it/s 0.69 | |
| [0130 18:02:01 @multigpu.py:323] [p0574] step: count(1333), step_time 1401.36, mean_step_time 1437.66, it/s 0.7 | |
| [0130 18:02:01 @multigpu.py:323] [p0576] step: count(1342), step_time 1452.52, mean_step_time 1447.87, it/s 0.69 | |
| [0130 18:02:01 @multigpu.py:323] [p0115] step: count(1318), step_time 1443.73, mean_step_time 1447.43, it/s 0.69 | |
| [0130 18:02:02 @multigpu.py:323] [p0574] step: count(1334), step_time 1450.79, mean_step_time 1435.42, it/s 0.7 | |
| [0130 18:02:02 @multigpu.py:323] [p0576] step: count(1343), step_time 1401.61, mean_step_time 1446.92, it/s 0.69 | |
| [0130 18:02:02 @multigpu.py:323] [p0115] step: count(1319), step_time 1411.88, mean_step_time 1444.07, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.533943954706 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.533990806341 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 18:02:04 @multigpu.py:323] [p0574] step: count(1335), step_time 1388.52, mean_step_time 1435.86, it/s 0.7 | |
| [0130 18:02:04 @multigpu.py:323] [p0115] step: count(1320), step_time 1406.04, mean_step_time 1439.13, it/s 0.69 | |
| [0130 18:02:04 @multigpu.py:323] [p0576] step: count(1344), step_time 1489.33, mean_step_time 1449.82, it/s 0.69 | |
| [0130 18:02:05 @multigpu.py:323] [p0574] step: count(1336), step_time 1413.78, mean_step_time 1435.52, it/s 0.7 | |
| [0130 18:02:05 @multigpu.py:323] [p0115] step: count(1321), step_time 1403.22, mean_step_time 1438.72, it/s 0.7 | |
| [0130 18:02:05 @multigpu.py:323] [p0576] step: count(1345), step_time 1422.62, mean_step_time 1450.32, it/s 0.69 | |
| [0130 18:02:06 @multigpu.py:323] [p0574] step: count(1337), step_time 1472.75, mean_step_time 1439.58, it/s 0.69 | |
| [0130 18:02:07 @multigpu.py:323] [p0115] step: count(1322), step_time 1451.03, mean_step_time 1441.31, it/s 0.69 | |
| [0130 18:02:07 @multigpu.py:323] [p0576] step: count(1346), step_time 1488.87, mean_step_time 1455.42, it/s 0.69 | |
| [0130 18:02:08 @multigpu.py:323] [p0574] step: count(1338), step_time 1503.68, mean_step_time 1443.54, it/s 0.69 | |
| [0130 18:02:08 @multigpu.py:323] [p0115] step: count(1323), step_time 1434.57, mean_step_time 1440.68, it/s 0.69 | |
| [0130 18:02:08 @multigpu.py:323] [p0576] step: count(1347), step_time 1404.22, mean_step_time 1452.75, it/s 0.69 | |
| [0130 18:02:09 @multigpu.py:323] [p0574] step: count(1339), step_time 1424.48, mean_step_time 1443.3, it/s 0.69 | |
| [0130 18:02:09 @multigpu.py:323] [p0115] step: count(1324), step_time 1374.08, mean_step_time 1439.42, it/s 0.69 | |
| [0130 18:02:10 @multigpu.py:323] [p0576] step: count(1348), step_time 1401.08, mean_step_time 1448.39, it/s 0.69 | |
| [0130 18:02:11 @multigpu.py:323] [p0574] step: count(1340), step_time 1442.77, mean_step_time 1443.18, it/s 0.69 | |
| [0130 18:02:11 @multigpu.py:323] [p0115] step: count(1325), step_time 1459.87, mean_step_time 1437.54, it/s 0.7 | |
| [0130 18:02:11 @multigpu.py:323] [p0576] step: count(1349), step_time 1440.85, mean_step_time 1450.24, it/s 0.69 | |
| [0130 18:02:12 @multigpu.py:323] [p0574] step: count(1341), step_time 1431.65, mean_step_time 1444.82, it/s 0.69 | |
| [0130 18:02:12 @multigpu.py:323] [p0115] step: count(1326), step_time 1427.27, mean_step_time 1438.85, it/s 0.69 | |
| [0130 18:02:12 @multigpu.py:323] [p0576] step: count(1350), step_time 1395.23, mean_step_time 1447.41, it/s 0.69 | |
| [0130 18:02:14 @multigpu.py:323] [p0115] step: count(1327), step_time 1408.36, mean_step_time 1436.36, it/s 0.7 | |
| [0130 18:02:14 @multigpu.py:323] [p0574] step: count(1342), step_time 1472.65, mean_step_time 1445.7, it/s 0.69 | |
| [0130 18:02:14 @multigpu.py:323] [p0576] step: count(1351), step_time 1438.57, mean_step_time 1447.33, it/s 0.69 | |
| [0130 18:02:15 @multigpu.py:323] [p0115] step: count(1328), step_time 1482.76, mean_step_time 1437.59, it/s 0.7 | |
| [0130 18:02:15 @multigpu.py:323] [p0576] step: count(1352), step_time 1433.1, mean_step_time 1437.59, it/s 0.7 | |
| [0130 18:02:15 @multigpu.py:323] [p0574] step: count(1343), step_time 1443.32, mean_step_time 1437.65, it/s 0.7 | |
| [0130 18:02:17 @multigpu.py:323] [p0574] step: count(1344), step_time 1387.03, mean_step_time 1434.8, it/s 0.7 | |
| [0130 18:02:17 @multigpu.py:323] [p0576] step: count(1353), step_time 1433.47, mean_step_time 1437.67, it/s 0.7 | |
| [0130 18:02:17 @multigpu.py:323] [p0115] step: count(1329), step_time 1459.3, mean_step_time 1437.93, it/s 0.7 | |
| [0130 18:02:18 @multigpu.py:323] [p0576] step: count(1354), step_time 1409.46, mean_step_time 1436.28, it/s 0.7 | |
| [0130 18:02:18 @multigpu.py:323] [p0574] step: count(1345), step_time 1457.71, mean_step_time 1435.46, it/s 0.7 | |
| [0130 18:02:18 @multigpu.py:323] [p0115] step: count(1330), step_time 1426.61, mean_step_time 1438.44, it/s 0.7 | |
| [0130 18:02:19 @multigpu.py:323] [p0574] step: count(1346), step_time 1415.38, mean_step_time 1433.95, it/s 0.7 | |
| [0130 18:02:20 @multigpu.py:323] [p0576] step: count(1355), step_time 1492.05, mean_step_time 1436.89, it/s 0.7 | |
| [0130 18:02:20 @multigpu.py:323] [p0115] step: count(1331), step_time 1460.89, mean_step_time 1435.82, it/s 0.7 | |
| [0130 18:02:21 @multigpu.py:323] [p0574] step: count(1347), step_time 1426.43, mean_step_time 1433.32, it/s 0.7 | |
| [0130 18:02:21 @multigpu.py:323] [p0115] step: count(1332), step_time 1375.86, mean_step_time 1432.9, it/s 0.7 | |
| [0130 18:02:21 @multigpu.py:323] [p0576] step: count(1356), step_time 1408.51, mean_step_time 1433.76, it/s 0.7 | |
| [0130 18:02:22 @multigpu.py:323] [p0574] step: count(1348), step_time 1386.07, mean_step_time 1431.35, it/s 0.7 | |
| [0130 18:02:22 @multigpu.py:323] [p0115] step: count(1333), step_time 1430.46, mean_step_time 1432.99, it/s 0.7 | |
| [0130 18:02:22 @multigpu.py:323] [p0576] step: count(1357), step_time 1415.37, mean_step_time 1432.01, it/s 0.7 | |
| [0130 18:02:24 @multigpu.py:323] [p0574] step: count(1349), step_time 1406.08, mean_step_time 1432.34, it/s 0.7 | |
| [0130 18:02:24 @multigpu.py:323] [p0576] step: count(1358), step_time 1453.69, mean_step_time 1434.41, it/s 0.7 | |
| [0130 18:02:24 @multigpu.py:323] [p0115] step: count(1334), step_time 1488.36, mean_step_time 1437.25, it/s 0.7 | |
| [0130 18:02:25 @multigpu.py:323] [p0574] step: count(1350), step_time 1441.79, mean_step_time 1431.17, it/s 0.7 | |
| [0130 18:02:25 @multigpu.py:323] [p0576] step: count(1359), step_time 1387.2, mean_step_time 1431.52, it/s 0.7 | |
| [0130 18:02:25 @multigpu.py:323] [p0115] step: count(1335), step_time 1428.06, mean_step_time 1432.91, it/s 0.7 | |
| [0130 18:02:27 @multigpu.py:323] [p0574] step: count(1351), step_time 1414.6, mean_step_time 1429.33, it/s 0.7 | |
| [0130 18:02:27 @multigpu.py:323] [p0115] step: count(1336), step_time 1423.21, mean_step_time 1432.53, it/s 0.7 | |
| [0130 18:02:27 @multigpu.py:323] [p0576] step: count(1360), step_time 1503.81, mean_step_time 1434.13, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.540751250254 , 1.5 ##### | |
| [u'online', 1.5] | |
| ##### Sending to neptune: active_workers : 0.540751353833 , 3 ##### | |
| receiving | |
| [0130 18:02:28 @multigpu.py:323] [p0574] step: count(1352), step_time 1405.06, mean_step_time 1429.3, it/s 0.7 | |
| [0130 18:02:28 @multigpu.py:323] [p0576] step: count(1361), step_time 1404.19, mean_step_time 1433.79, it/s 0.7 | |
| [0130 18:02:28 @multigpu.py:323] [p0115] step: count(1337), step_time 1450.23, mean_step_time 1432.29, it/s 0.7 | |
| [0130 18:02:29 @multigpu.py:323] [p0574] step: count(1353), step_time 1415.19, mean_step_time 1429.99, it/s 0.7 | |
| [0130 18:02:30 @multigpu.py:323] [p0576] step: count(1362), step_time 1451.61, mean_step_time 1433.74, it/s 0.7 | |
| [0130 18:02:30 @multigpu.py:323] [p0115] step: count(1338), step_time 1441.16, mean_step_time 1432.16, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.541542358332 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:02:31 @multigpu.py:323] [p0574] step: count(1354), step_time 1390.2, mean_step_time 1426.96, it/s 0.7 | |
| [0130 18:02:31 @multigpu.py:323] [p0576] step: count(1363), step_time 1408.87, mean_step_time 1434.11, it/s 0.7 | |
| [0130 18:02:31 @multigpu.py:323] [p0115] step: count(1339), step_time 1413.46, mean_step_time 1432.24, it/s 0.7 | |
| [0130 18:02:32 @multigpu.py:323] [p0574] step: count(1355), step_time 1440.46, mean_step_time 1429.55, it/s 0.7 | |
| [0130 18:02:32 @multigpu.py:323] [p0576] step: count(1364), step_time 1434.43, mean_step_time 1431.36, it/s 0.7 | |
| [0130 18:02:33 @multigpu.py:323] [p0115] step: count(1340), step_time 1494.65, mean_step_time 1436.67, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.542352651358 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:02:34 @multigpu.py:323] [p0574] step: count(1356), step_time 1404.31, mean_step_time 1429.08, it/s 0.7 | |
| [0130 18:02:34 @multigpu.py:323] [p0576] step: count(1365), step_time 1415.21, mean_step_time 1430.99, it/s 0.7 | |
| [0130 18:02:34 @multigpu.py:323] [p0115] step: count(1341), step_time 1412.43, mean_step_time 1437.13, it/s 0.7 | |
| [0130 18:02:35 @multigpu.py:323] [p0574] step: count(1357), step_time 1387.35, mean_step_time 1424.81, it/s 0.7 | |
| [0130 18:02:35 @multigpu.py:323] [p0576] step: count(1366), step_time 1408.33, mean_step_time 1426.96, it/s 0.7 | |
| [0130 18:02:35 @multigpu.py:323] [p0115] step: count(1342), step_time 1432.7, mean_step_time 1436.21, it/s 0.7 | |
| [0130 18:02:36 @multigpu.py:323] [p0574] step: count(1358), step_time 1415.84, mean_step_time 1420.42, it/s 0.7 | |
| [0130 18:02:37 @multigpu.py:323] [p0576] step: count(1367), step_time 1433.29, mean_step_time 1428.42, it/s 0.7 | |
| [0130 18:02:37 @multigpu.py:323] [p0115] step: count(1343), step_time 1448.73, mean_step_time 1436.92, it/s 0.7 | |
| [0130 18:02:38 @multigpu.py:323] [p0574] step: count(1359), step_time 1483.98, mean_step_time 1423.39, it/s 0.7 | |
| [0130 18:02:38 @multigpu.py:323] [p0576] step: count(1368), step_time 1436.63, mean_step_time 1430.19, it/s 0.7 | |
| [0130 18:02:38 @multigpu.py:323] [p0115] step: count(1344), step_time 1448.18, mean_step_time 1440.63, it/s 0.69 | |
| [0130 18:02:39 @multigpu.py:323] [p0574] step: count(1360), step_time 1426.99, mean_step_time 1422.6, it/s 0.7 | |
| [0130 18:02:40 @multigpu.py:323] [p0576] step: count(1369), step_time 1404.52, mean_step_time 1428.38, it/s 0.7 | |
| [0130 18:02:40 @multigpu.py:323] [p0115] step: count(1345), step_time 1413.7, mean_step_time 1438.32, it/s 0.7 | |
| [0130 18:02:41 @multigpu.py:323] [p0574] step: count(1361), step_time 1372.54, mean_step_time 1419.65, it/s 0.7 | |
| [0130 18:02:41 @multigpu.py:323] [p0576] step: count(1370), step_time 1417.31, mean_step_time 1429.48, it/s 0.7 | |
| [0130 18:02:41 @multigpu.py:323] [p0115] step: count(1346), step_time 1437.96, mean_step_time 1438.85, it/s 0.69 | |
| [0130 18:02:42 @multigpu.py:323] [p0574] step: count(1362), step_time 1486.76, mean_step_time 1420.35, it/s 0.7 | |
| [0130 18:02:42 @multigpu.py:323] [p0576] step: count(1371), step_time 1376.86, mean_step_time 1426.4, it/s 0.7 | |
| [0130 18:02:43 @multigpu.py:323] [p0115] step: count(1347), step_time 1406.57, mean_step_time 1438.76, it/s 0.7 | |
| [0130 18:02:44 @multigpu.py:323] [p0115] step: count(1348), step_time 1442.11, mean_step_time 1436.73, it/s 0.7 | |
| [0130 18:02:44 @multigpu.py:323] [p0576] step: count(1372), step_time 1638.05, mean_step_time 1436.64, it/s 0.7 | |
| [0130 18:02:44 @multigpu.py:323] [p0574] step: count(1363), step_time 1770.49, mean_step_time 1436.71, it/s 0.7 | |
| [0130 18:02:45 @multigpu.py:323] [p0576] step: count(1373), step_time 1429.14, mean_step_time 1436.43, it/s 0.7 | |
| [0130 18:02:45 @multigpu.py:323] [p0574] step: count(1364), step_time 1429.9, mean_step_time 1438.86, it/s 0.69 | |
| [0130 18:02:45 @multigpu.py:323] [p0115] step: count(1349), step_time 1432.33, mean_step_time 1435.38, it/s 0.7 | |
| [0130 18:02:47 @multigpu.py:323] [p0576] step: count(1374), step_time 1433.5, mean_step_time 1437.63, it/s 0.7 | |
| [0130 18:02:47 @multigpu.py:323] [p0574] step: count(1365), step_time 1432.46, mean_step_time 1437.59, it/s 0.7 | |
| [0130 18:02:47 @multigpu.py:323] [p0115] step: count(1350), step_time 1484.87, mean_step_time 1438.3, it/s 0.7 | |
| [0130 18:02:48 @multigpu.py:323] [p0574] step: count(1366), step_time 1406.44, mean_step_time 1437.15, it/s 0.7 | |
| [0130 18:02:48 @multigpu.py:323] [p0576] step: count(1375), step_time 1415.47, mean_step_time 1433.8, it/s 0.7 | |
| [0130 18:02:48 @multigpu.py:323] [p0115] step: count(1351), step_time 1411.37, mean_step_time 1435.82, it/s 0.7 | |
| [0130 18:02:50 @multigpu.py:323] [p0574] step: count(1367), step_time 1408.95, mean_step_time 1436.27, it/s 0.7 | |
| [0130 18:02:50 @multigpu.py:323] [p0576] step: count(1376), step_time 1485.84, mean_step_time 1437.67, it/s 0.7 | |
| [0130 18:02:50 @multigpu.py:323] [p0115] step: count(1352), step_time 1461.74, mean_step_time 1440.11, it/s 0.69 | |
| [0130 18:02:51 @multigpu.py:323] [p0574] step: count(1368), step_time 1430.87, mean_step_time 1438.51, it/s 0.7 | |
| [0130 18:02:51 @multigpu.py:323] [p0115] step: count(1353), step_time 1385.82, mean_step_time 1437.88, it/s 0.7 | |
| [0130 18:02:51 @multigpu.py:323] [p0576] step: count(1377), step_time 1434.55, mean_step_time 1438.63, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.547551592191 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:02:52 @multigpu.py:323] [p0574] step: count(1369), step_time 1421.69, mean_step_time 1439.29, it/s 0.69 | |
| [0130 18:02:53 @multigpu.py:323] [p0576] step: count(1378), step_time 1407.45, mean_step_time 1436.31, it/s 0.7 | |
| [0130 18:02:53 @multigpu.py:323] [p0115] step: count(1354), step_time 1433.65, mean_step_time 1435.15, it/s 0.7 | |
| [0130 18:02:54 @multigpu.py:323] [p0574] step: count(1370), step_time 1417.29, mean_step_time 1438.07, it/s 0.7 | |
| [0130 18:02:54 @multigpu.py:323] [p0576] step: count(1379), step_time 1367.25, mean_step_time 1435.32, it/s 0.7 | |
| [0130 18:02:54 @multigpu.py:323] [p0115] step: count(1355), step_time 1423.47, mean_step_time 1434.92, it/s 0.7 | |
| [0130 18:02:55 @multigpu.py:323] [p0574] step: count(1371), step_time 1411.08, mean_step_time 1437.89, it/s 0.7 | |
| [0130 18:02:55 @multigpu.py:323] [p0576] step: count(1380), step_time 1443.27, mean_step_time 1432.29, it/s 0.7 | |
| [0130 18:02:55 @multigpu.py:323] [p0115] step: count(1356), step_time 1474.0, mean_step_time 1437.46, it/s 0.7 | |
| [0130 18:02:57 @multigpu.py:323] [p0576] step: count(1381), step_time 1368.81, mean_step_time 1430.52, it/s 0.7 | |
| [0130 18:02:57 @multigpu.py:323] [p0574] step: count(1372), step_time 1491.64, mean_step_time 1442.22, it/s 0.69 | |
| [0130 18:02:57 @multigpu.py:323] [p0115] step: count(1357), step_time 1414.76, mean_step_time 1435.68, it/s 0.7 | |
| [0130 18:02:58 @multigpu.py:323] [p0576] step: count(1382), step_time 1484.48, mean_step_time 1432.16, it/s 0.7 | |
| [0130 18:02:58 @multigpu.py:323] [p0574] step: count(1373), step_time 1429.34, mean_step_time 1442.93, it/s 0.69 | |
| [0130 18:02:58 @multigpu.py:323] [p0115] step: count(1358), step_time 1513.46, mean_step_time 1439.3, it/s 0.69 | |
| [0130 18:03:00 @multigpu.py:323] [p0576] step: count(1383), step_time 1430.51, mean_step_time 1433.25, it/s 0.7 | |
| [0130 18:03:00 @multigpu.py:323] [p0574] step: count(1374), step_time 1436.92, mean_step_time 1445.26, it/s 0.69 | |
| [0130 18:03:00 @multigpu.py:323] [p0115] step: count(1359), step_time 1418.1, mean_step_time 1439.53, it/s 0.69 | |
| [0130 18:03:01 @multigpu.py:323] [p0576] step: count(1384), step_time 1404.53, mean_step_time 1431.75, it/s 0.7 | |
| [0130 18:03:01 @multigpu.py:323] [p0574] step: count(1375), step_time 1408.26, mean_step_time 1443.65, it/s 0.69 | |
| [0130 18:03:01 @multigpu.py:323] [p0115] step: count(1360), step_time 1415.47, mean_step_time 1435.57, it/s 0.7 | |
| [0130 18:03:03 @multigpu.py:323] [p0576] step: count(1385), step_time 1442.41, mean_step_time 1433.11, it/s 0.7 | |
| [0130 18:03:03 @multigpu.py:323] [p0574] step: count(1376), step_time 1428.69, mean_step_time 1444.87, it/s 0.69 | |
| [0130 18:03:03 @multigpu.py:323] [p0115] step: count(1361), step_time 1467.4, mean_step_time 1438.32, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.550689121617 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.550788611637 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 18:03:04 @multigpu.py:323] [p0576] step: count(1386), step_time 1435.36, mean_step_time 1434.46, it/s 0.7 | |
| [0130 18:03:04 @multigpu.py:323] [p0574] step: count(1377), step_time 1426.97, mean_step_time 1446.86, it/s 0.69 | |
| [0130 18:03:04 @multigpu.py:323] [p0115] step: count(1362), step_time 1378.13, mean_step_time 1435.59, it/s 0.7 | |
| [0130 18:03:05 @multigpu.py:323] [p0574] step: count(1378), step_time 1406.52, mean_step_time 1446.39, it/s 0.69 | |
| [0130 18:03:05 @multigpu.py:323] [p0576] step: count(1387), step_time 1427.56, mean_step_time 1434.18, it/s 0.7 | |
| [0130 18:03:05 @multigpu.py:323] [p0115] step: count(1363), step_time 1406.02, mean_step_time 1433.46, it/s 0.7 | |
| [0130 18:03:07 @multigpu.py:323] [p0576] step: count(1388), step_time 1447.5, mean_step_time 1434.72, it/s 0.7 | |
| [0130 18:03:07 @multigpu.py:323] [p0574] step: count(1379), step_time 1476.22, mean_step_time 1446.0, it/s 0.69 | |
| [0130 18:03:07 @multigpu.py:323] [p0115] step: count(1364), step_time 1421.08, mean_step_time 1432.1, it/s 0.7 | |
| [0130 18:03:08 @multigpu.py:323] [p0576] step: count(1389), step_time 1389.55, mean_step_time 1433.97, it/s 0.7 | |
| [0130 18:03:08 @multigpu.py:323] [p0574] step: count(1380), step_time 1435.9, mean_step_time 1446.45, it/s 0.69 | |
| [0130 18:03:08 @multigpu.py:323] [p0115] step: count(1365), step_time 1455.47, mean_step_time 1434.19, it/s 0.7 | |
| [0130 18:03:10 @multigpu.py:323] [p0576] step: count(1390), step_time 1399.11, mean_step_time 1433.06, it/s 0.7 | |
| [0130 18:03:10 @multigpu.py:323] [p0115] step: count(1366), step_time 1419.1, mean_step_time 1433.25, it/s 0.7 | |
| [0130 18:03:10 @multigpu.py:323] [p0574] step: count(1381), step_time 1518.95, mean_step_time 1453.77, it/s 0.69 | |
| [0130 18:03:11 @multigpu.py:323] [p0576] step: count(1391), step_time 1394.68, mean_step_time 1433.95, it/s 0.7 | |
| [0130 18:03:11 @multigpu.py:323] [p0574] step: count(1382), step_time 1389.37, mean_step_time 1448.9, it/s 0.69 | |
| [0130 18:03:11 @multigpu.py:323] [p0115] step: count(1367), step_time 1411.2, mean_step_time 1433.48, it/s 0.7 | |
| [0130 18:03:13 @multigpu.py:323] [p0115] step: count(1368), step_time 1451.68, mean_step_time 1433.96, it/s 0.7 | |
| [0130 18:03:13 @multigpu.py:323] [p0576] step: count(1392), step_time 1637.72, mean_step_time 1433.93, it/s 0.7 | |
| [0130 18:03:13 @multigpu.py:323] [p0574] step: count(1383), step_time 1469.12, mean_step_time 1433.83, it/s 0.7 | |
| [0130 18:03:14 @multigpu.py:323] [p0576] step: count(1393), step_time 1392.96, mean_step_time 1432.13, it/s 0.7 | |
| [0130 18:03:14 @multigpu.py:323] [p0574] step: count(1384), step_time 1429.08, mean_step_time 1433.79, it/s 0.7 | |
| [0130 18:03:14 @multigpu.py:323] [p0115] step: count(1369), step_time 1442.2, mean_step_time 1434.45, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.553857938581 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 18:03:15 @multigpu.py:323] [p0576] step: count(1394), step_time 1431.74, mean_step_time 1432.04, it/s 0.7 | |
| [0130 18:03:15 @multigpu.py:323] [p0115] step: count(1370), step_time 1393.41, mean_step_time 1429.88, it/s 0.7 | |
| [0130 18:03:16 @multigpu.py:323] [p0574] step: count(1385), step_time 1439.65, mean_step_time 1434.15, it/s 0.7 | |
| [0130 18:03:17 @multigpu.py:323] [p0576] step: count(1395), step_time 1411.33, mean_step_time 1431.83, it/s 0.7 | |
| [0130 18:03:17 @multigpu.py:323] [p0115] step: count(1371), step_time 1414.81, mean_step_time 1430.05, it/s 0.7 | |
| [0130 18:03:17 @multigpu.py:323] [p0574] step: count(1386), step_time 1430.7, mean_step_time 1435.36, it/s 0.7 | |
| [0130 18:03:18 @multigpu.py:323] [p0576] step: count(1396), step_time 1421.42, mean_step_time 1428.61, it/s 0.7 | |
| [0130 18:03:18 @multigpu.py:323] [p0115] step: count(1372), step_time 1413.61, mean_step_time 1427.64, it/s 0.7 | |
| [0130 18:03:18 @multigpu.py:323] [p0574] step: count(1387), step_time 1418.14, mean_step_time 1435.82, it/s 0.7 | |
| [0130 18:03:20 @multigpu.py:323] [p0115] step: count(1373), step_time 1417.49, mean_step_time 1429.23, it/s 0.7 | |
| [0130 18:03:20 @multigpu.py:323] [p0576] step: count(1397), step_time 1470.89, mean_step_time 1430.43, it/s 0.7 | |
| [0130 18:03:20 @multigpu.py:323] [p0574] step: count(1388), step_time 1456.8, mean_step_time 1437.12, it/s 0.7 | |
| [0130 18:03:21 @multigpu.py:323] [p0115] step: count(1374), step_time 1447.07, mean_step_time 1429.9, it/s 0.7 | |
| [0130 18:03:21 @multigpu.py:323] [p0574] step: count(1389), step_time 1370.25, mean_step_time 1434.54, it/s 0.7 | |
| [0130 18:03:21 @multigpu.py:323] [p0576] step: count(1398), step_time 1519.05, mean_step_time 1436.01, it/s 0.7 | |
| [0130 18:03:23 @multigpu.py:323] [p0115] step: count(1375), step_time 1392.37, mean_step_time 1428.34, it/s 0.7 | |
| [0130 18:03:23 @multigpu.py:323] [p0574] step: count(1390), step_time 1445.91, mean_step_time 1435.98, it/s 0.7 | |
| [0130 18:03:23 @multigpu.py:323] [p0576] step: count(1399), step_time 1382.11, mean_step_time 1436.75, it/s 0.7 | |
| [0130 18:03:24 @multigpu.py:323] [p0115] step: count(1376), step_time 1428.37, mean_step_time 1426.06, it/s 0.7 | |
| [0130 18:03:24 @multigpu.py:323] [p0574] step: count(1391), step_time 1393.81, mean_step_time 1435.11, it/s 0.7 | |
| [0130 18:03:24 @multigpu.py:323] [p0576] step: count(1400), step_time 1472.93, mean_step_time 1438.23, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.556543689966 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.556543689966 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.556543689966 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.556544211904 , -0.00979332625866 ##### | |
| ##### Sending to neptune: policy_loss : 0.556544211904 , -0.407843053341 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.556544211904 , -2.28586602211 ##### | |
| ##### Sending to neptune: value_loss : 0.556544211904 , 1.44016361237 ##### | |
| ##### Sending to neptune: advantage : 0.556544211904 , 0.00166430650279 ##### | |
| ##### Sending to neptune: pred_reward : 0.556544211904 , 0.450704455376 ##### | |
| ##### Sending to neptune: max_logit : 0.556544211904 , 0.210775285959 ##### | |
| [u'loss', -0.009793326258659363, -0.40784305334091187, -2.2858660221099854, 1.4401636123657227, 0.001664306502789259, 0.4507044553756714, 0.21077528595924377] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.556544706888 , 9774847.18 ##### | |
| ##### Sending to neptune: dp_per_s : 0.556544706888 , 89.2697802076 ##### | |
| [u'other', 9774847.18, 89.26978020759367] | |
| receiving | |
| [0130 18:03:25 @multigpu.py:323] [p0115] step: count(1377), step_time 1460.58, mean_step_time 1428.35, it/s 0.7 | |
| [0130 18:03:25 @multigpu.py:323] [p0574] step: count(1392), step_time 1452.28, mean_step_time 1433.14, it/s 0.7 | |
| [0130 18:03:26 @multigpu.py:323] [p0576] step: count(1401), step_time 1385.76, mean_step_time 1439.08, it/s 0.69 | |
| [0130 18:03:27 @multigpu.py:323] [p0574] step: count(1393), step_time 1385.9, mean_step_time 1430.97, it/s 0.7 | |
| [0130 18:03:27 @multigpu.py:323] [p0115] step: count(1378), step_time 1452.95, mean_step_time 1425.33, it/s 0.7 | |
| [0130 18:03:27 @multigpu.py:323] [p0576] step: count(1402), step_time 1407.95, mean_step_time 1435.25, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.557487060494 , 1.7 ##### | |
| [u'online', 1.7] | |
| ##### Sending to neptune: active_workers : 0.557487142483 , 3 ##### | |
| receiving | |
| [0130 18:03:28 @multigpu.py:323] [p0574] step: count(1394), step_time 1447.19, mean_step_time 1431.49, it/s 0.7 | |
| [0130 18:03:28 @multigpu.py:323] [p0115] step: count(1379), step_time 1414.04, mean_step_time 1425.12, it/s 0.7 | |
| [0130 18:03:28 @multigpu.py:323] [p0576] step: count(1403), step_time 1396.03, mean_step_time 1433.53, it/s 0.7 | |
| [0130 18:03:30 @multigpu.py:323] [p0574] step: count(1395), step_time 1396.59, mean_step_time 1430.9, it/s 0.7 | |
| [0130 18:03:30 @multigpu.py:323] [p0576] step: count(1404), step_time 1405.56, mean_step_time 1433.58, it/s 0.7 | |
| [0130 18:03:30 @multigpu.py:323] [p0115] step: count(1380), step_time 1432.02, mean_step_time 1425.95, it/s 0.7 | |
| [0130 18:03:31 @multigpu.py:323] [p0574] step: count(1396), step_time 1405.79, mean_step_time 1429.76, it/s 0.7 | |
| [0130 18:03:31 @multigpu.py:323] [p0576] step: count(1405), step_time 1407.6, mean_step_time 1431.84, it/s 0.7 | |
| [0130 18:03:31 @multigpu.py:323] [p0115] step: count(1381), step_time 1451.89, mean_step_time 1425.17, it/s 0.7 | |
| [0130 18:03:33 @multigpu.py:323] [p0576] step: count(1406), step_time 1429.22, mean_step_time 1431.53, it/s 0.7 | |
| [0130 18:03:33 @multigpu.py:323] [p0574] step: count(1397), step_time 1462.66, mean_step_time 1431.54, it/s 0.7 | |
| [0130 18:03:33 @multigpu.py:323] [p0115] step: count(1382), step_time 1407.0, mean_step_time 1426.62, it/s 0.7 | |
| [0130 18:03:34 @multigpu.py:323] [p0574] step: count(1398), step_time 1379.74, mean_step_time 1430.2, it/s 0.7 | |
| [0130 18:03:34 @multigpu.py:323] [p0115] step: count(1383), step_time 1406.98, mean_step_time 1426.67, it/s 0.7 | |
| [0130 18:03:34 @multigpu.py:323] [p0576] step: count(1407), step_time 1441.96, mean_step_time 1432.25, it/s 0.7 | |
| [0130 18:03:35 @multigpu.py:323] [p0574] step: count(1399), step_time 1403.81, mean_step_time 1426.58, it/s 0.7 | |
| [0130 18:03:35 @multigpu.py:323] [p0576] step: count(1408), step_time 1380.08, mean_step_time 1428.88, it/s 0.7 | |
| [0130 18:03:35 @multigpu.py:323] [p0115] step: count(1384), step_time 1433.42, mean_step_time 1427.28, it/s 0.7 | |
| [0130 18:03:37 @multigpu.py:323] [p0576] step: count(1409), step_time 1406.49, mean_step_time 1429.73, it/s 0.7 | |
| [0130 18:03:37 @multigpu.py:323] [p0574] step: count(1400), step_time 1479.25, mean_step_time 1428.75, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.560071513851 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.560071513851 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.560071513851 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.560071992477 , -0.00401578377932 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.560071992477 , 0.167572632432 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.560071992477 , -2.28561639786 ##### | |
| ##### Sending to neptune: value_loss : 0.560071992477 , 1.60402357578 ##### | |
| ##### Sending to neptune: advantage : 0.560071992477 , -0.000748563557863 ##### | |
| ##### Sending to neptune: pred_reward : 0.560071992477 , 0.458303689957 ##### | |
| ##### Sending to neptune: max_logit : 0.560071992477 , 0.2134732306 ##### | |
| [u'loss', -0.004015783779323101, 0.16757263243198395, -2.285616397857666, 1.6040235757827759, -0.0007485635578632355, 0.45830368995666504, 0.21347323060035706] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.560072435273 , 9771696.53 ##### | |
| ##### Sending to neptune: dp_per_s : 0.560072435273 , 89.2691596223 ##### | |
| [u'other', 9771696.53, 89.26915962233971] | |
| receiving | |
| [0130 18:03:37 @multigpu.py:323] [p0115] step: count(1385), step_time 1439.52, mean_step_time 1426.49, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.560282720261 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:03:38 @multigpu.py:323] [p0576] step: count(1410), step_time 1412.38, mean_step_time 1430.39, it/s 0.7 | |
| [0130 18:03:38 @multigpu.py:323] [p0574] step: count(1401), step_time 1418.35, mean_step_time 1423.72, it/s 0.7 | |
| [0130 18:03:38 @multigpu.py:323] [p0115] step: count(1386), step_time 1427.55, mean_step_time 1426.91, it/s 0.7 | |
| [0130 18:03:40 @multigpu.py:323] [p0576] step: count(1411), step_time 1410.81, mean_step_time 1431.2, it/s 0.7 | |
| [0130 18:03:40 @multigpu.py:323] [p0574] step: count(1402), step_time 1430.52, mean_step_time 1425.78, it/s 0.7 | |
| [0130 18:03:40 @multigpu.py:323] [p0115] step: count(1387), step_time 1387.21, mean_step_time 1425.71, it/s 0.7 | |
| [0130 18:03:41 @multigpu.py:323] [p0115] step: count(1388), step_time 1451.92, mean_step_time 1425.72, it/s 0.7 | |
| [0130 18:03:41 @multigpu.py:323] [p0574] step: count(1403), step_time 1468.66, mean_step_time 1425.75, it/s 0.7 | |
| [0130 18:03:41 @multigpu.py:323] [p0576] step: count(1412), step_time 1529.78, mean_step_time 1425.8, it/s 0.7 | |
| [0130 18:03:43 @multigpu.py:323] [p0574] step: count(1404), step_time 1376.73, mean_step_time 1423.14, it/s 0.7 | |
| [0130 18:03:43 @multigpu.py:323] [p0576] step: count(1413), step_time 1411.29, mean_step_time 1426.72, it/s 0.7 | |
| [0130 18:03:43 @multigpu.py:323] [p0115] step: count(1389), step_time 1436.06, mean_step_time 1425.41, it/s 0.7 | |
| [0130 18:03:44 @multigpu.py:323] [p0574] step: count(1405), step_time 1419.54, mean_step_time 1422.13, it/s 0.7 | |
| [0130 18:03:44 @multigpu.py:323] [p0115] step: count(1390), step_time 1406.9, mean_step_time 1426.09, it/s 0.7 | |
| [0130 18:03:44 @multigpu.py:323] [p0576] step: count(1414), step_time 1453.25, mean_step_time 1427.79, it/s 0.7 | |
| [0130 18:03:45 @multigpu.py:323] [p0574] step: count(1406), step_time 1442.56, mean_step_time 1422.72, it/s 0.7 | |
| [0130 18:03:45 @multigpu.py:323] [p0115] step: count(1391), step_time 1404.4, mean_step_time 1425.57, it/s 0.7 | |
| [0130 18:03:45 @multigpu.py:323] [p0576] step: count(1415), step_time 1411.3, mean_step_time 1427.79, it/s 0.7 | |
| [0130 18:03:47 @multigpu.py:323] [p0574] step: count(1407), step_time 1372.41, mean_step_time 1420.44, it/s 0.7 | |
| [0130 18:03:47 @multigpu.py:323] [p0115] step: count(1392), step_time 1419.39, mean_step_time 1425.86, it/s 0.7 | |
| [0130 18:03:47 @multigpu.py:323] [p0576] step: count(1416), step_time 1451.4, mean_step_time 1429.29, it/s 0.7 | |
| [0130 18:03:48 @multigpu.py:323] [p0574] step: count(1408), step_time 1446.35, mean_step_time 1419.91, it/s 0.7 | |
| [0130 18:03:48 @multigpu.py:323] [p0115] step: count(1393), step_time 1410.03, mean_step_time 1425.48, it/s 0.7 | |
| [0130 18:03:48 @multigpu.py:323] [p0576] step: count(1417), step_time 1358.56, mean_step_time 1423.68, it/s 0.7 | |
| [0130 18:03:50 @multigpu.py:323] [p0576] step: count(1418), step_time 1427.34, mean_step_time 1419.09, it/s 0.7 | |
| [0130 18:03:50 @multigpu.py:323] [p0115] step: count(1394), step_time 1481.06, mean_step_time 1427.18, it/s 0.7 | |
| [0130 18:03:50 @multigpu.py:323] [p0574] step: count(1409), step_time 1503.11, mean_step_time 1426.56, it/s 0.7 | |
| [0130 18:03:51 @multigpu.py:323] [p0115] step: count(1395), step_time 1411.36, mean_step_time 1428.13, it/s 0.7 | |
| [0130 18:03:51 @multigpu.py:323] [p0576] step: count(1419), step_time 1463.2, mean_step_time 1423.14, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.564045781096 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:03:51 @multigpu.py:323] [p0574] step: count(1410), step_time 1455.92, mean_step_time 1427.06, it/s 0.7 | |
| [0130 18:03:53 @multigpu.py:323] [p0576] step: count(1420), step_time 1406.7, mean_step_time 1419.83, it/s 0.7 | |
| [0130 18:03:53 @multigpu.py:323] [p0115] step: count(1396), step_time 1418.38, mean_step_time 1427.63, it/s 0.7 | |
| [0130 18:03:53 @multigpu.py:323] [p0574] step: count(1411), step_time 1499.37, mean_step_time 1432.34, it/s 0.7 | |
| [0130 18:03:54 @multigpu.py:323] [p0115] step: count(1397), step_time 1440.01, mean_step_time 1426.6, it/s 0.7 | |
| [0130 18:03:54 @multigpu.py:323] [p0576] step: count(1421), step_time 1460.03, mean_step_time 1423.55, it/s 0.7 | |
| 42%|####2 |421/1000[1[0130 18:03:54 @multigpu.py:323] [p0574] step: count(1412), step_time 1433.92, mean_step_time 1431.42, it/s 0.7 | |
| [0130 18:03:55 @multigpu.py:323] [p0576] step: count(1422), step_time 1431.75, mean_step_time 1424.74, it/s 0.7 | |
| [0130 18:03:56 @multigpu.py:323] [p0115] step: count(1398), step_time 1491.19, mean_step_time 1428.52, it/s 0.7 | |
| [0130 18:03:56 @multigpu.py:323] [p0574] step: count(1413), step_time 1429.86, mean_step_time 1433.62, it/s 0.7 | |
| [0130 18:03:57 @multigpu.py:323] [p0576] step: count(1423), step_time 1400.64, mean_step_time 1424.97, it/s 0.7 | |
| [0130 18:03:57 @multigpu.py:323] [p0115] step: count(1399), step_time 1451.76, mean_step_time 1430.4, it/s 0.7 | |
| [0130 18:03:57 @multigpu.py:323] [p0574] step: count(1414), step_time 1432.31, mean_step_time 1432.87, it/s 0.7 | |
| [0130 18:03:58 @multigpu.py:323] [p0576] step: count(1424), step_time 1441.02, mean_step_time 1426.74, it/s 0.7 | |
| [0130 18:03:58 @multigpu.py:323] [p0115] step: count(1400), step_time 1432.93, mean_step_time 1430.45, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.566052081651 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.566052081651 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.566052081651 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.566052584383 , -0.00693348702043 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.566052584383 , 0.0658853203058 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.566052584383 , -2.28544616699 ##### | |
| ##### Sending to neptune: value_loss : 0.566052584383 , 1.33207464218 ##### | |
| ##### Sending to neptune: advantage : 0.566052584383 , -0.000360454781912 ##### | |
| ##### Sending to neptune: pred_reward : 0.566052584383 , 0.440528899431 ##### | |
| ##### Sending to neptune: max_logit : 0.566052584383 , 0.215451806784 ##### | |
| [u'loss', -0.006933487020432949, 0.06588532030582428, -2.2854461669921875, 1.3320746421813965, -0.0003604547819122672, 0.44052889943122864, 0.21545180678367615] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.566053047445 , 9768145.36 ##### | |
| ##### Sending to neptune: dp_per_s : 0.566053047445 , 89.2624328751 ##### | |
| [u'other', 9768145.36, 89.26243287508888] | |
| receiving | |
| [0130 18:03:58 @multigpu.py:323] [p0574] step: count(1415), step_time 1433.64, mean_step_time 1434.72, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.566195656061 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:04:00 @multigpu.py:323] [p0576] step: count(1425), step_time 1418.71, mean_step_time 1427.29, it/s 0.7 | |
| [0130 18:04:00 @multigpu.py:323] [p0115] step: count(1401), step_time 1408.59, mean_step_time 1428.28, it/s 0.7 | |
| [0130 18:04:00 @multigpu.py:323] [p0574] step: count(1416), step_time 1428.78, mean_step_time 1435.87, it/s 0.7 | |
| [0130 18:04:01 @multigpu.py:323] [p0576] step: count(1426), step_time 1384.2, mean_step_time 1425.04, it/s 0.7 | |
| [0130 18:04:01 @multigpu.py:323] [p0115] step: count(1402), step_time 1472.46, mean_step_time 1431.56, it/s 0.7 | |
| [0130 18:04:01 @multigpu.py:323] [p0574] step: count(1417), step_time 1427.13, mean_step_time 1434.1, it/s 0.7 | |
| [0130 18:04:03 @multigpu.py:323] [p0576] step: count(1427), step_time 1432.02, mean_step_time 1424.55, it/s 0.7 | |
| [0130 18:04:03 @multigpu.py:323] [p0115] step: count(1403), step_time 1397.03, mean_step_time 1431.06, it/s 0.7 | |
| [0130 18:04:03 @multigpu.py:323] [p0574] step: count(1418), step_time 1401.49, mean_step_time 1435.18, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.567442069716 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 18:04:04 @multigpu.py:323] [p0576] step: count(1428), step_time 1388.66, mean_step_time 1424.98, it/s 0.7 | |
| [0130 18:04:04 @multigpu.py:323] [p0115] step: count(1404), step_time 1416.48, mean_step_time 1430.21, it/s 0.7 | |
| [0130 18:04:04 @multigpu.py:323] [p0574] step: count(1419), step_time 1474.8, mean_step_time 1438.73, it/s 0.7 | |
| [0130 18:04:05 @multigpu.py:323] [p0576] step: count(1429), step_time 1433.08, mean_step_time 1426.31, it/s 0.7 | |
| [0130 18:04:06 @multigpu.py:323] [p0115] step: count(1405), step_time 1485.3, mean_step_time 1432.5, it/s 0.7 | |
| [0130 18:04:06 @multigpu.py:323] [p0574] step: count(1420), step_time 1429.36, mean_step_time 1436.24, it/s 0.7 | |
| [0130 18:04:07 @multigpu.py:323] [p0576] step: count(1430), step_time 1402.01, mean_step_time 1425.79, it/s 0.7 | |
| [0130 18:04:07 @multigpu.py:323] [p0115] step: count(1406), step_time 1430.27, mean_step_time 1432.64, it/s 0.7 | |
| [0130 18:04:07 @multigpu.py:323] [p0574] step: count(1421), step_time 1446.44, mean_step_time 1437.64, it/s 0.7 | |
| 42%|####2 |421/1000[1[0130 18:04:08 @multigpu.py:323] [p0576] step: count(1431), step_time 1437.82, mean_step_time 1427.14, it/s 0.7 | |
| [0130 18:04:08 @multigpu.py:323] [p0574] step: count(1422), step_time 1387.96, mean_step_time 1435.52, it/s 0.7 | |
| [0130 18:04:08 @multigpu.py:323] [p0115] step: count(1407), step_time 1463.99, mean_step_time 1436.48, it/s 0.7 | |
| [0130 18:04:10 @multigpu.py:323] [p0115] step: count(1408), step_time 1417.2, mean_step_time 1434.74, it/s 0.7 | |
| [0130 18:04:10 @multigpu.py:323] [p0576] step: count(1432), step_time 1683.99, mean_step_time 1434.85, it/s 0.7 | |
| [0130 18:04:10 @multigpu.py:323] [p0574] step: count(1423), step_time 1453.76, mean_step_time 1434.77, it/s 0.7 | |
| [0130 18:04:11 @multigpu.py:323] [p0574] step: count(1424), step_time 1397.23, mean_step_time 1435.8, it/s 0.7 | |
| [0130 18:04:11 @multigpu.py:323] [p0576] step: count(1433), step_time 1445.1, mean_step_time 1436.54, it/s 0.7 | |
| [0130 18:04:11 @multigpu.py:323] [p0115] step: count(1409), step_time 1479.96, mean_step_time 1436.93, it/s 0.7 | |
| [0130 18:04:13 @multigpu.py:323] [p0574] step: count(1425), step_time 1455.64, mean_step_time 1437.6, it/s 0.7 | |
| [0130 18:04:13 @multigpu.py:323] [p0576] step: count(1434), step_time 1429.56, mean_step_time 1435.35, it/s 0.7 | |
| [0130 18:04:13 @multigpu.py:323] [p0115] step: count(1410), step_time 1414.51, mean_step_time 1437.31, it/s 0.7 | |
| [0130 18:04:14 @multigpu.py:323] [p0574] step: count(1426), step_time 1404.97, mean_step_time 1435.72, it/s 0.7 | |
| [0130 18:04:14 @multigpu.py:323] [p0576] step: count(1435), step_time 1443.08, mean_step_time 1436.94, it/s 0.7 | |
| [0130 18:04:14 @multigpu.py:323] [p0115] step: count(1411), step_time 1460.86, mean_step_time 1440.14, it/s 0.69 | |
| [0130 18:04:16 @multigpu.py:323] [p0574] step: count(1427), step_time 1421.89, mean_step_time 1438.2, it/s 0.7 | |
| [0130 18:04:16 @multigpu.py:323] [p0115] step: count(1412), step_time 1443.55, mean_step_time 1441.35, it/s 0.69 | |
| [0130 18:04:16 @multigpu.py:323] [p0576] step: count(1436), step_time 1507.11, mean_step_time 1439.73, it/s 0.69 | |
| [0130 18:04:17 @multigpu.py:323] [p0574] step: count(1428), step_time 1436.85, mean_step_time 1437.72, it/s 0.7 | |
| [0130 18:04:17 @multigpu.py:323] [p0115] step: count(1413), step_time 1404.07, mean_step_time 1441.05, it/s 0.69 | |
| [0130 18:04:17 @multigpu.py:323] [p0576] step: count(1437), step_time 1402.3, mean_step_time 1441.92, it/s 0.69 | |
| [0130 18:04:18 @multigpu.py:323] [p0574] step: count(1429), step_time 1403.43, mean_step_time 1432.74, it/s 0.7 | |
| [0130 18:04:19 @multigpu.py:323] [p0115] step: count(1414), step_time 1450.1, mean_step_time 1439.5, it/s 0.69 | |
| [0130 18:04:19 @multigpu.py:323] [p0576] step: count(1438), step_time 1429.5, mean_step_time 1442.02, it/s 0.69 | |
| [0130 18:04:20 @multigpu.py:323] [p0574] step: count(1430), step_time 1454.1, mean_step_time 1432.65, it/s 0.7 | |
| [0130 18:04:20 @multigpu.py:323] [p0576] step: count(1439), step_time 1378.6, mean_step_time 1437.79, it/s 0.7 | |
| [0130 18:04:20 @multigpu.py:323] [p0115] step: count(1415), step_time 1405.54, mean_step_time 1439.21, it/s 0.69 | |
| [0130 18:04:21 @multigpu.py:323] [p0574] step: count(1431), step_time 1439.17, mean_step_time 1429.64, it/s 0.7 | |
| [0130 18:04:21 @multigpu.py:323] [p0115] step: count(1416), step_time 1441.28, mean_step_time 1440.35, it/s 0.69 | |
| [0130 18:04:21 @multigpu.py:323] [p0576] step: count(1440), step_time 1499.94, mean_step_time 1442.46, it/s 0.69 | |
| [0130 18:04:23 @multigpu.py:323] [p0574] step: count(1432), step_time 1418.43, mean_step_time 1428.86, it/s 0.7 | |
| [0130 18:04:23 @multigpu.py:323] [p0115] step: count(1417), step_time 1441.13, mean_step_time 1440.41, it/s 0.69 | |
| [0130 18:04:23 @multigpu.py:323] [p0576] step: count(1441), step_time 1418.78, mean_step_time 1440.39, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.57297924108 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:04:24 @multigpu.py:323] [p0574] step: count(1433), step_time 1421.44, mean_step_time 1428.44, it/s 0.7 | |
| [0130 18:04:24 @multigpu.py:323] [p0115] step: count(1418), step_time 1364.18, mean_step_time 1434.06, it/s 0.7 | |
| [0130 18:04:24 @multigpu.py:323] [p0576] step: count(1442), step_time 1402.34, mean_step_time 1438.92, it/s 0.69 | |
| [0130 18:04:26 @multigpu.py:323] [p0574] step: count(1434), step_time 1469.75, mean_step_time 1430.31, it/s 0.7 | |
| [0130 18:04:26 @multigpu.py:323] [p0115] step: count(1419), step_time 1464.25, mean_step_time 1434.68, it/s 0.7 | |
| [0130 18:04:26 @multigpu.py:323] [p0576] step: count(1443), step_time 1439.18, mean_step_time 1440.85, it/s 0.69 | |
| [0130 18:04:27 @multigpu.py:323] [p0574] step: count(1435), step_time 1390.89, mean_step_time 1428.18, it/s 0.7 | |
| [0130 18:04:27 @multigpu.py:323] [p0576] step: count(1444), step_time 1393.86, mean_step_time 1438.49, it/s 0.7 | |
| [0130 18:04:27 @multigpu.py:323] [p0115] step: count(1420), step_time 1461.02, mean_step_time 1436.09, it/s 0.7 | |
| 42%|####2 |420/1000[1[0130 18:04:28 @multigpu.py:323] [p0574] step: count(1436), step_time 1398.75, mean_step_time 1426.67, it/s 0.7 | |
| [0130 18:04:29 @multigpu.py:323] [p0576] step: count(1445), step_time 1473.53, mean_step_time 1441.23, it/s 0.69 | |
| [0130 18:04:29 @multigpu.py:323] [p0115] step: count(1421), step_time 1498.68, mean_step_time 1440.59, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.574587674141 , 1.3 ##### | |
| [u'online', 1.3] | |
| ##### Sending to neptune: active_workers : 0.57458770944 , 3 ##### | |
| receiving | |
| [0130 18:04:30 @multigpu.py:323] [p0574] step: count(1437), step_time 1419.64, mean_step_time 1426.3, it/s 0.7 | |
| [0130 18:04:30 @multigpu.py:323] [p0115] step: count(1422), step_time 1420.56, mean_step_time 1438.0, it/s 0.7 | |
| [0130 18:04:30 @multigpu.py:323] [p0576] step: count(1446), step_time 1496.92, mean_step_time 1446.87, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.574955003858 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 18:04:31 @multigpu.py:323] [p0574] step: count(1438), step_time 1439.34, mean_step_time 1428.19, it/s 0.7 | |
| [0130 18:04:31 @multigpu.py:323] [p0115] step: count(1423), step_time 1413.13, mean_step_time 1438.8, it/s 0.7 | |
| [0130 18:04:31 @multigpu.py:323] [p0576] step: count(1447), step_time 1426.12, mean_step_time 1446.57, it/s 0.69 | |
| [0130 18:04:33 @multigpu.py:323] [p0574] step: count(1439), step_time 1413.37, mean_step_time 1425.12, it/s 0.7 | |
| [0130 18:04:33 @multigpu.py:323] [p0115] step: count(1424), step_time 1406.32, mean_step_time 1438.29, it/s 0.7 | |
| [0130 18:04:33 @multigpu.py:323] [p0576] step: count(1448), step_time 1443.4, mean_step_time 1449.31, it/s 0.69 | |
| [0130 18:04:34 @multigpu.py:323] [p0574] step: count(1440), step_time 1384.12, mean_step_time 1422.86, it/s 0.7 | |
| [0130 18:04:34 @multigpu.py:323] [p0115] step: count(1425), step_time 1440.45, mean_step_time 1436.05, it/s 0.7 | |
| [0130 18:04:34 @multigpu.py:323] [p0576] step: count(1449), step_time 1399.53, mean_step_time 1447.63, it/s 0.69 | |
| [0130 18:04:35 @multigpu.py:323] [p0574] step: count(1441), step_time 1419.58, mean_step_time 1421.51, it/s 0.7 | |
| [0130 18:04:36 @multigpu.py:323] [p0115] step: count(1426), step_time 1411.7, mean_step_time 1435.12, it/s 0.7 | |
| [0130 18:04:36 @multigpu.py:323] [p0576] step: count(1450), step_time 1423.99, mean_step_time 1448.73, it/s 0.69 | |
| [0130 18:04:37 @multigpu.py:323] [p0574] step: count(1442), step_time 1436.64, mean_step_time 1423.95, it/s 0.7 | |
| [0130 18:04:37 @multigpu.py:323] [p0576] step: count(1451), step_time 1416.57, mean_step_time 1447.67, it/s 0.69 | |
| [0130 18:04:37 @multigpu.py:323] [p0115] step: count(1427), step_time 1510.37, mean_step_time 1437.44, it/s 0.7 | |
| [0130 18:04:39 @multigpu.py:323] [p0115] step: count(1428), step_time 1389.38, mean_step_time 1436.05, it/s 0.7 | |
| [0130 18:04:39 @multigpu.py:323] [p0574] step: count(1443), step_time 1693.65, mean_step_time 1435.94, it/s 0.7 | |
| [0130 18:04:39 @multigpu.py:323] [p0576] step: count(1452), step_time 1449.86, mean_step_time 1435.96, it/s 0.7 | |
| [0130 18:04:40 @multigpu.py:323] [p0574] step: count(1444), step_time 1416.81, mean_step_time 1436.92, it/s 0.7 | |
| [0130 18:04:40 @multigpu.py:323] [p0115] step: count(1429), step_time 1443.53, mean_step_time 1434.23, it/s 0.7 | |
| [0130 18:04:40 @multigpu.py:323] [p0576] step: count(1453), step_time 1449.07, mean_step_time 1436.16, it/s 0.7 | |
| [0130 18:04:41 @multigpu.py:323] [p0574] step: count(1445), step_time 1428.49, mean_step_time 1435.56, it/s 0.7 | |
| [0130 18:04:42 @multigpu.py:323] [p0576] step: count(1454), step_time 1435.61, mean_step_time 1436.46, it/s 0.7 | |
| [0130 18:04:42 @multigpu.py:323] [p0115] step: count(1430), step_time 1450.77, mean_step_time 1436.04, it/s 0.7 | |
| [0130 18:04:43 @multigpu.py:323] [p0574] step: count(1446), step_time 1397.11, mean_step_time 1435.17, it/s 0.7 | |
| [0130 18:04:43 @multigpu.py:323] [p0115] step: count(1431), step_time 1429.27, mean_step_time 1434.46, it/s 0.7 | |
| [0130 18:04:43 @multigpu.py:323] [p0576] step: count(1455), step_time 1445.67, mean_step_time 1436.59, it/s 0.7 | |
| [0130 18:04:44 @multigpu.py:323] [p0574] step: count(1447), step_time 1409.48, mean_step_time 1434.55, it/s 0.7 | |
| [0130 18:04:44 @multigpu.py:323] [p0576] step: count(1456), step_time 1421.41, mean_step_time 1432.31, it/s 0.7 | |
| [0130 18:04:44 @multigpu.py:323] [p0115] step: count(1432), step_time 1447.53, mean_step_time 1434.66, it/s 0.7 | |
| [0130 18:04:46 @multigpu.py:323] [p0574] step: count(1448), step_time 1405.16, mean_step_time 1432.97, it/s 0.7 | |
| [0130 18:04:46 @multigpu.py:323] [p0576] step: count(1457), step_time 1417.23, mean_step_time 1433.06, it/s 0.7 | |
| [0130 18:04:46 @multigpu.py:323] [p0115] step: count(1433), step_time 1425.03, mean_step_time 1435.71, it/s 0.7 | |
| [0130 18:04:47 @multigpu.py:323] [p0574] step: count(1449), step_time 1415.3, mean_step_time 1433.56, it/s 0.7 | |
| [0130 18:04:47 @multigpu.py:323] [p0115] step: count(1434), step_time 1399.97, mean_step_time 1433.2, it/s 0.7 | |
| [0130 18:04:47 @multigpu.py:323] [p0576] step: count(1458), step_time 1449.31, mean_step_time 1434.05, it/s 0.7 | |
| [0130 18:04:49 @multigpu.py:323] [p0574] step: count(1450), step_time 1483.76, mean_step_time 1435.04, it/s 0.7 | |
| [0130 18:04:49 @multigpu.py:323] [p0115] step: count(1435), step_time 1417.79, mean_step_time 1433.82, it/s 0.7 | |
| [0130 18:04:49 @multigpu.py:323] [p0576] step: count(1459), step_time 1436.14, mean_step_time 1436.92, it/s 0.7 | |
| [0130 18:04:50 @multigpu.py:323] [p0574] step: count(1451), step_time 1404.69, mean_step_time 1433.32, it/s 0.7 | |
| [0130 18:04:50 @multigpu.py:323] [p0115] step: count(1436), step_time 1410.73, mean_step_time 1432.29, it/s 0.7 | |
| [0130 18:04:50 @multigpu.py:323] [p0576] step: count(1460), step_time 1416.0, mean_step_time 1432.73, it/s 0.7 | |
| [0130 18:04:51 @multigpu.py:323] [p0574] step: count(1452), step_time 1442.82, mean_step_time 1434.54, it/s 0.7 | |
| [0130 18:04:51 @multigpu.py:323] [p0576] step: count(1461), step_time 1396.68, mean_step_time 1431.62, it/s 0.7 | |
| [0130 18:04:52 @multigpu.py:323] [p0115] step: count(1437), step_time 1458.91, mean_step_time 1433.18, it/s 0.7 | |
| [0130 18:04:53 @multigpu.py:323] [p0576] step: count(1462), step_time 1413.84, mean_step_time 1432.2, it/s 0.7 | |
| [0130 18:04:53 @multigpu.py:323] [p0115] step: count(1438), step_time 1434.71, mean_step_time 1436.7, it/s 0.7 | |
| [0130 18:04:53 @multigpu.py:323] [p0574] step: count(1453), step_time 1518.88, mean_step_time 1439.41, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.581396578881 , 2.4 ##### | |
| [u'online', 2.4] | |
| receiving | |
| [0130 18:04:54 @multigpu.py:323] [p0576] step: count(1463), step_time 1382.78, mean_step_time 1429.38, it/s 0.7 | |
| [0130 18:04:54 @multigpu.py:323] [p0574] step: count(1454), step_time 1381.92, mean_step_time 1435.02, it/s 0.7 | |
| [0130 18:04:54 @multigpu.py:323] [p0115] step: count(1439), step_time 1474.45, mean_step_time 1437.22, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.58178515527 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:04:56 @multigpu.py:323] [p0576] step: count(1464), step_time 1446.41, mean_step_time 1432.0, it/s 0.7 | |
| [0130 18:04:56 @multigpu.py:323] [p0574] step: count(1455), step_time 1437.95, mean_step_time 1437.37, it/s 0.7 | |
| [0130 18:04:56 @multigpu.py:323] [p0115] step: count(1440), step_time 1431.34, mean_step_time 1435.73, it/s 0.7 | |
| [0130 18:04:57 @multigpu.py:323] [p0576] step: count(1465), step_time 1396.41, mean_step_time 1428.15, it/s 0.7 | |
| [0130 18:04:57 @multigpu.py:323] [p0574] step: count(1456), step_time 1366.36, mean_step_time 1435.75, it/s 0.7 | |
| [0130 18:04:57 @multigpu.py:323] [p0115] step: count(1441), step_time 1456.16, mean_step_time 1433.61, it/s 0.7 | |
| [0130 18:04:59 @multigpu.py:323] [p0576] step: count(1466), step_time 1406.44, mean_step_time 1423.62, it/s 0.7 | |
| [0130 18:04:59 @multigpu.py:323] [p0574] step: count(1457), step_time 1428.66, mean_step_time 1436.21, it/s 0.7 | |
| [0130 18:04:59 @multigpu.py:323] [p0115] step: count(1442), step_time 1452.52, mean_step_time 1435.2, it/s 0.7 | |
| [0130 18:05:00 @multigpu.py:323] [p0574] step: count(1458), step_time 1429.06, mean_step_time 1435.69, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.583168856634 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:05:00 @multigpu.py:323] [p0576] step: count(1467), step_time 1486.09, mean_step_time 1426.62, it/s 0.7 | |
| [0130 18:05:00 @multigpu.py:323] [p0115] step: count(1443), step_time 1417.34, mean_step_time 1435.41, it/s 0.7 | |
| [0130 18:05:01 @multigpu.py:323] [p0574] step: count(1459), step_time 1460.65, mean_step_time 1438.06, it/s 0.7 | |
| [0130 18:05:01 @multigpu.py:323] [p0576] step: count(1468), step_time 1442.72, mean_step_time 1426.59, it/s 0.7 | |
| [0130 18:05:02 @multigpu.py:323] [p0115] step: count(1444), step_time 1443.77, mean_step_time 1437.29, it/s 0.7 | |
| [0130 18:05:03 @multigpu.py:323] [p0574] step: count(1460), step_time 1357.34, mean_step_time 1436.72, it/s 0.7 | |
| [0130 18:05:03 @multigpu.py:323] [p0576] step: count(1469), step_time 1421.67, mean_step_time 1427.69, it/s 0.7 | |
| [0130 18:05:03 @multigpu.py:323] [p0115] step: count(1445), step_time 1390.52, mean_step_time 1434.79, it/s 0.7 | |
| [0130 18:05:04 @multigpu.py:323] [p0574] step: count(1461), step_time 1439.14, mean_step_time 1437.69, it/s 0.7 | |
| [0130 18:05:04 @multigpu.py:323] [p0576] step: count(1470), step_time 1397.91, mean_step_time 1426.39, it/s 0.7 | |
| [0130 18:05:04 @multigpu.py:323] [p0115] step: count(1446), step_time 1459.09, mean_step_time 1437.16, it/s 0.7 | |
| [0130 18:05:06 @multigpu.py:323] [p0574] step: count(1462), step_time 1395.75, mean_step_time 1435.65, it/s 0.7 | |
| [0130 18:05:06 @multigpu.py:323] [p0576] step: count(1471), step_time 1491.13, mean_step_time 1430.12, it/s 0.7 | |
| [0130 18:05:06 @multigpu.py:323] [p0115] step: count(1447), step_time 1441.38, mean_step_time 1433.71, it/s 0.7 | |
| [0130 18:05:07 @multigpu.py:323] [p0115] step: count(1448), step_time 1416.11, mean_step_time 1435.05, it/s 0.7 | |
| [0130 18:05:07 @multigpu.py:323] [p0576] step: count(1472), step_time 1545.41, mean_step_time 1434.9, it/s 0.7 | |
| [0130 18:05:07 @multigpu.py:323] [p0574] step: count(1463), step_time 1681.88, mean_step_time 1435.06, it/s 0.7 | |
| [0130 18:05:09 @multigpu.py:323] [p0574] step: count(1464), step_time 1426.25, mean_step_time 1435.53, it/s 0.7 | |
| [0130 18:05:09 @multigpu.py:323] [p0115] step: count(1449), step_time 1460.03, mean_step_time 1435.87, it/s 0.7 | |
| [0130 18:05:09 @multigpu.py:323] [p0576] step: count(1473), step_time 1517.29, mean_step_time 1438.31, it/s 0.7 | |
| [0130 18:05:10 @multigpu.py:323] [p0574] step: count(1465), step_time 1445.47, mean_step_time 1436.38, it/s 0.7 | |
| [0130 18:05:10 @multigpu.py:323] [p0115] step: count(1450), step_time 1445.33, mean_step_time 1435.6, it/s 0.7 | |
| [0130 18:05:10 @multigpu.py:323] [p0576] step: count(1474), step_time 1437.59, mean_step_time 1438.41, it/s 0.7 | |
| [0130 18:05:12 @multigpu.py:323] [p0574] step: count(1466), step_time 1384.36, mean_step_time 1435.74, it/s 0.7 | |
| [0130 18:05:12 @multigpu.py:323] [p0115] step: count(1451), step_time 1414.79, mean_step_time 1434.87, it/s 0.7 | |
| [0130 18:05:12 @multigpu.py:323] [p0576] step: count(1475), step_time 1501.42, mean_step_time 1441.19, it/s 0.69 | |
| [0130 18:05:13 @multigpu.py:323] [p0574] step: count(1467), step_time 1430.6, mean_step_time 1436.8, it/s 0.7 | |
| [0130 18:05:13 @multigpu.py:323] [p0115] step: count(1452), step_time 1469.14, mean_step_time 1435.96, it/s 0.7 | |
| [0130 18:05:13 @multigpu.py:323] [p0576] step: count(1476), step_time 1433.88, mean_step_time 1441.82, it/s 0.69 | |
| [0130 18:05:14 @multigpu.py:323] [p0574] step: count(1468), step_time 1419.41, mean_step_time 1437.51, it/s 0.7 | |
| [0130 18:05:15 @multigpu.py:323] [p0115] step: count(1453), step_time 1404.76, mean_step_time 1434.94, it/s 0.7 | |
| [0130 18:05:15 @multigpu.py:323] [p0576] step: count(1477), step_time 1375.04, mean_step_time 1439.71, it/s 0.69 | |
| [0130 18:05:16 @multigpu.py:323] [p0574] step: count(1469), step_time 1436.21, mean_step_time 1438.56, it/s 0.7 | |
| [0130 18:05:16 @multigpu.py:323] [p0115] step: count(1454), step_time 1409.09, mean_step_time 1435.4, it/s 0.7 | |
| [0130 18:05:16 @multigpu.py:323] [p0576] step: count(1478), step_time 1437.64, mean_step_time 1439.12, it/s 0.69 | |
| [0130 18:05:17 @multigpu.py:323] [p0574] step: count(1470), step_time 1467.17, mean_step_time 1437.73, it/s 0.7 | |
| [0130 18:05:17 @multigpu.py:323] [p0115] step: count(1455), step_time 1412.25, mean_step_time 1435.12, it/s 0.7 | |
| [0130 18:05:17 @multigpu.py:323] [p0576] step: count(1479), step_time 1377.61, mean_step_time 1436.2, it/s 0.7 | |
| [0130 18:05:19 @multigpu.py:323] [p0574] step: count(1471), step_time 1400.23, mean_step_time 1437.51, it/s 0.7 | |
| [0130 18:05:19 @multigpu.py:323] [p0115] step: count(1456), step_time 1438.36, mean_step_time 1436.5, it/s 0.7 | |
| [0130 18:05:19 @multigpu.py:323] [p0576] step: count(1480), step_time 1420.01, mean_step_time 1436.4, it/s 0.7 | |
| [0130 18:05:20 @multigpu.py:323] [p0115] step: count(1457), step_time 1414.06, mean_step_time 1434.26, it/s 0.7 | |
| [0130 18:05:20 @multigpu.py:323] [p0574] step: count(1472), step_time 1526.73, mean_step_time 1441.7, it/s 0.69 | |
| [0130 18:05:20 @multigpu.py:323] [p0576] step: count(1481), step_time 1460.71, mean_step_time 1439.6, it/s 0.69 | |
| [0130 18:05:22 @multigpu.py:323] [p0115] step: count(1458), step_time 1444.06, mean_step_time 1434.73, it/s 0.7 | |
| [0130 18:05:22 @multigpu.py:323] [p0576] step: count(1482), step_time 1396.01, mean_step_time 1438.71, it/s 0.7 | |
| [0130 18:05:22 @multigpu.py:323] [p0574] step: count(1473), step_time 1476.44, mean_step_time 1439.58, it/s 0.69 | |
| [0130 18:05:23 @multigpu.py:323] [p0115] step: count(1459), step_time 1415.03, mean_step_time 1431.76, it/s 0.7 | |
| [0130 18:05:23 @multigpu.py:323] [p0576] step: count(1483), step_time 1455.79, mean_step_time 1442.36, it/s 0.69 | |
| [0130 18:05:23 @multigpu.py:323] [p0574] step: count(1474), step_time 1441.3, mean_step_time 1442.55, it/s 0.69 | |
| [0130 18:05:24 @multigpu.py:323] [p0115] step: count(1460), step_time 1423.93, mean_step_time 1431.39, it/s 0.7 | |
| [0130 18:05:25 @multigpu.py:323] [p0576] step: count(1484), step_time 1390.01, mean_step_time 1439.54, it/s 0.69 | |
| [0130 18:05:25 @multigpu.py:323] [p0574] step: count(1475), step_time 1404.97, mean_step_time 1440.9, it/s 0.69 | |
| [0130 18:05:26 @multigpu.py:323] [p0115] step: count(1461), step_time 1440.58, mean_step_time 1430.61, it/s 0.7 | |
| [0130 18:05:26 @multigpu.py:323] [p0576] step: count(1485), step_time 1408.88, mean_step_time 1440.16, it/s 0.69 | |
| [0130 18:05:26 @multigpu.py:323] [p0574] step: count(1476), step_time 1362.87, mean_step_time 1440.72, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.590532777773 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:05:27 @multigpu.py:323] [p0115] step: count(1462), step_time 1486.69, mean_step_time 1432.32, it/s 0.7 | |
| [0130 18:05:27 @multigpu.py:323] [p0576] step: count(1486), step_time 1470.78, mean_step_time 1443.38, it/s 0.69 | |
| [0130 18:05:27 @multigpu.py:323] [p0574] step: count(1477), step_time 1481.01, mean_step_time 1443.34, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.59100319412 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 18:05:29 @multigpu.py:323] [p0576] step: count(1487), step_time 1376.85, mean_step_time 1437.92, it/s 0.7 | |
| [0130 18:05:29 @multigpu.py:323] [p0115] step: count(1463), step_time 1385.96, mean_step_time 1430.75, it/s 0.7 | |
| [0130 18:05:29 @multigpu.py:323] [p0574] step: count(1478), step_time 1408.74, mean_step_time 1442.33, it/s 0.69 | |
| [0130 18:05:30 @multigpu.py:323] [p0576] step: count(1488), step_time 1440.22, mean_step_time 1437.79, it/s 0.7 | |
| [0130 18:05:30 @multigpu.py:323] [p0574] step: count(1479), step_time 1449.09, mean_step_time 1441.75, it/s 0.69 | |
| [0130 18:05:30 @multigpu.py:323] [p0115] step: count(1464), step_time 1520.66, mean_step_time 1434.59, it/s 0.7 | |
| [0130 18:05:32 @multigpu.py:323] [p0574] step: count(1480), step_time 1359.54, mean_step_time 1441.86, it/s 0.69 | |
| [0130 18:05:32 @multigpu.py:323] [p0576] step: count(1489), step_time 1426.31, mean_step_time 1438.02, it/s 0.7 | |
| [0130 18:05:32 @multigpu.py:323] [p0115] step: count(1465), step_time 1408.79, mean_step_time 1435.5, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.592168379691 , 1.3 ##### | |
| [u'online', 1.3] | |
| ##### Sending to neptune: active_workers : 0.592168413864 , 3 ##### | |
| receiving | |
| [0130 18:05:33 @multigpu.py:323] [p0576] step: count(1490), step_time 1440.04, mean_step_time 1440.13, it/s 0.69 | |
| [0130 18:05:33 @multigpu.py:323] [p0574] step: count(1481), step_time 1445.72, mean_step_time 1442.19, it/s 0.69 | |
| [0130 18:05:33 @multigpu.py:323] [p0115] step: count(1466), step_time 1403.25, mean_step_time 1432.71, it/s 0.7 | |
| [0130 18:05:35 @multigpu.py:323] [p0574] step: count(1482), step_time 1407.35, mean_step_time 1442.77, it/s 0.69 | |
| [0130 18:05:35 @multigpu.py:323] [p0576] step: count(1491), step_time 1422.11, mean_step_time 1436.68, it/s 0.7 | |
| [0130 18:05:35 @multigpu.py:323] [p0115] step: count(1467), step_time 1445.25, mean_step_time 1432.91, it/s 0.7 | |
| [0130 18:05:36 @multigpu.py:323] [p0115] step: count(1468), step_time 1456.31, mean_step_time 1434.92, it/s 0.7 | |
| [0130 18:05:36 @multigpu.py:323] [p0574] step: count(1483), step_time 1523.84, mean_step_time 1434.87, it/s 0.7 | |
| [0130 18:05:36 @multigpu.py:323] [p0576] step: count(1492), step_time 1508.82, mean_step_time 1434.85, it/s 0.7 | |
| [0130 18:05:37 @multigpu.py:323] [p0574] step: count(1484), step_time 1402.04, mean_step_time 1433.66, it/s 0.7 | |
| [0130 18:05:37 @multigpu.py:323] [p0115] step: count(1469), step_time 1415.48, mean_step_time 1432.69, it/s 0.7 | |
| [0130 18:05:37 @multigpu.py:323] [p0576] step: count(1493), step_time 1444.5, mean_step_time 1431.21, it/s 0.7 | |
| [0130 18:05:39 @multigpu.py:323] [p0576] step: count(1494), step_time 1410.45, mean_step_time 1429.85, it/s 0.7 | |
| [0130 18:05:39 @multigpu.py:323] [p0115] step: count(1470), step_time 1446.5, mean_step_time 1432.75, it/s 0.7 | |
| [0130 18:05:39 @multigpu.py:323] [p0574] step: count(1485), step_time 1516.5, mean_step_time 1437.21, it/s 0.7 | |
| [0130 18:05:40 @multigpu.py:323] [p0115] step: count(1471), step_time 1382.08, mean_step_time 1431.11, it/s 0.7 | |
| [0130 18:05:40 @multigpu.py:323] [p0576] step: count(1495), step_time 1417.69, mean_step_time 1425.67, it/s 0.7 | |
| [0130 18:05:40 @multigpu.py:323] [p0574] step: count(1486), step_time 1456.4, mean_step_time 1440.81, it/s 0.69 | |
| [0130 18:05:42 @multigpu.py:323] [p0115] step: count(1472), step_time 1440.42, mean_step_time 1429.68, it/s 0.7 | |
| [0130 18:05:42 @multigpu.py:323] [p0576] step: count(1496), step_time 1468.62, mean_step_time 1427.4, it/s 0.7 | |
| [0130 18:05:42 @multigpu.py:323] [p0574] step: count(1487), step_time 1463.15, mean_step_time 1442.44, it/s 0.69 | |
| [0130 18:05:43 @multigpu.py:323] [p0115] step: count(1473), step_time 1427.59, mean_step_time 1430.82, it/s 0.7 | |
| [0130 18:05:43 @multigpu.py:323] [p0576] step: count(1497), step_time 1416.67, mean_step_time 1429.49, it/s 0.7 | |
| [0130 18:05:43 @multigpu.py:323] [p0574] step: count(1488), step_time 1377.04, mean_step_time 1440.32, it/s 0.69 | |
| [0130 18:05:45 @multigpu.py:323] [p0115] step: count(1474), step_time 1389.29, mean_step_time 1429.83, it/s 0.7 | |
| [0130 18:05:45 @multigpu.py:323] [p0576] step: count(1498), step_time 1364.5, mean_step_time 1425.83, it/s 0.7 | |
| [0130 18:05:45 @multigpu.py:323] [p0574] step: count(1489), step_time 1432.0, mean_step_time 1440.11, it/s 0.69 | |
| [0130 18:05:46 @multigpu.py:323] [p0115] step: count(1475), step_time 1449.54, mean_step_time 1431.69, it/s 0.7 | |
| [0130 18:05:46 @multigpu.py:323] [p0576] step: count(1499), step_time 1459.3, mean_step_time 1429.91, it/s 0.7 | |
| [0130 18:05:46 @multigpu.py:323] [p0574] step: count(1490), step_time 1426.07, mean_step_time 1438.05, it/s 0.7 | |
| [0130 18:05:47 @multigpu.py:323] [p0115] step: count(1476), step_time 1472.42, mean_step_time 1433.4, it/s 0.7 | |
| [0130 18:05:47 @multigpu.py:323] [p0576] step: count(1500), step_time 1443.76, mean_step_time 1431.1, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.596350740261 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.596350740261 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.596350740261 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.596351259947 , -0.00888612959534 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.596351259947 , -0.356634289026 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.596351259947 , -2.28363466263 ##### | |
| ##### Sending to neptune: value_loss : 0.596351259947 , 1.50284433365 ##### | |
| ##### Sending to neptune: advantage : 0.596351259947 , 0.00149029435124 ##### | |
| ##### Sending to neptune: pred_reward : 0.596351259947 , 0.461806833744 ##### | |
| ##### Sending to neptune: max_logit : 0.596351259947 , 0.228477463126 ##### | |
| [u'loss', -0.008886129595339298, -0.3566342890262604, -2.283634662628174, 1.5028443336486816, 0.0014902943512424827, 0.4618068337440491, 0.22847746312618256] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.596351694663 , 9717197.02 ##### | |
| ##### Sending to neptune: dp_per_s : 0.596351694663 , 89.2814797357 ##### | |
| [u'other', 9717197.02, 89.28147973567819] | |
| receiving | |
| [0130 18:05:48 @multigpu.py:323] [p0574] step: count(1491), step_time 1440.54, mean_step_time 1440.07, it/s 0.69 | |
| [0130 18:05:49 @multigpu.py:323] [p0115] step: count(1477), step_time 1435.04, mean_step_time 1434.44, it/s 0.7 | |
| [0130 18:05:49 @multigpu.py:323] [p0576] step: count(1501), step_time 1432.08, mean_step_time 1429.67, it/s 0.7 | |
| [0130 18:05:49 @multigpu.py:323] [p0574] step: count(1492), step_time 1429.17, mean_step_time 1435.19, it/s 0.7 | |
| [0130 18:05:50 @multigpu.py:323] [p0576] step: count(1502), step_time 1436.44, mean_step_time 1431.69, it/s 0.7 | |
| [0130 18:05:50 @multigpu.py:323] [p0115] step: count(1478), step_time 1457.75, mean_step_time 1435.13, it/s 0.7 | |
| [0130 18:05:50 @multigpu.py:323] [p0574] step: count(1493), step_time 1425.53, mean_step_time 1432.64, it/s 0.7 | |
| [0130 18:05:52 @multigpu.py:323] [p0115] step: count(1479), step_time 1425.16, mean_step_time 1435.64, it/s 0.7 | |
| [0130 18:05:52 @multigpu.py:323] [p0576] step: count(1503), step_time 1450.49, mean_step_time 1431.43, it/s 0.7 | |
| [0130 18:05:52 @multigpu.py:323] [p0574] step: count(1494), step_time 1463.16, mean_step_time 1433.74, it/s 0.7 | |
| [0130 18:05:53 @multigpu.py:323] [p0576] step: count(1504), step_time 1405.55, mean_step_time 1432.2, it/s 0.7 | |
| [0130 18:05:53 @multigpu.py:323] [p0115] step: count(1480), step_time 1427.44, mean_step_time 1435.81, it/s 0.7 | |
| [0130 18:05:53 @multigpu.py:323] [p0574] step: count(1495), step_time 1400.87, mean_step_time 1433.53, it/s 0.7 | |
| [0130 18:05:55 @multigpu.py:323] [p0576] step: count(1505), step_time 1428.03, mean_step_time 1433.16, it/s 0.7 | |
| [0130 18:05:55 @multigpu.py:323] [p0115] step: count(1481), step_time 1469.23, mean_step_time 1437.24, it/s 0.7 | |
| [0130 18:05:55 @multigpu.py:323] [p0574] step: count(1496), step_time 1409.19, mean_step_time 1435.85, it/s 0.7 | |
| [0130 18:05:56 @multigpu.py:323] [p0576] step: count(1506), step_time 1411.85, mean_step_time 1430.21, it/s 0.7 | |
| [0130 18:05:56 @multigpu.py:323] [p0574] step: count(1497), step_time 1441.86, mean_step_time 1433.89, it/s 0.7 | |
| [0130 18:05:56 @multigpu.py:323] [p0115] step: count(1482), step_time 1455.18, mean_step_time 1435.67, it/s 0.7 | |
| [0130 18:05:57 @multigpu.py:323] [p0576] step: count(1507), step_time 1409.49, mean_step_time 1431.85, it/s 0.7 | |
| [0130 18:05:58 @multigpu.py:323] [p0574] step: count(1498), step_time 1387.26, mean_step_time 1432.82, it/s 0.7 | |
| [0130 18:05:58 @multigpu.py:323] [p0115] step: count(1483), step_time 1448.08, mean_step_time 1438.77, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.599309429129 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:05:59 @multigpu.py:323] [p0576] step: count(1508), step_time 1413.71, mean_step_time 1430.52, it/s 0.7 | |
| [0130 18:05:59 @multigpu.py:323] [p0574] step: count(1499), step_time 1474.19, mean_step_time 1434.07, it/s 0.7 | |
| [0130 18:05:59 @multigpu.py:323] [p0115] step: count(1484), step_time 1407.54, mean_step_time 1433.12, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.599584539665 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.599748105274 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:06:00 @multigpu.py:323] [p0576] step: count(1509), step_time 1433.24, mean_step_time 1430.87, it/s 0.7 | |
| [0130 18:06:00 @multigpu.py:323] [p0574] step: count(1500), step_time 1422.44, mean_step_time 1437.22, it/s 0.7 | |
| [0130 18:06:00 @multigpu.py:323] [p0115] step: count(1485), step_time 1418.1, mean_step_time 1433.58, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.599946719408 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.599946719408 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.599946719408 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.599947186046 , -0.00632883608341 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.599947186046 , -0.0338481925428 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.599947186046 , -2.28367400169 ##### | |
| ##### Sending to neptune: value_loss : 0.599947186046 , 1.50743150711 ##### | |
| ##### Sending to neptune: advantage : 0.599947186046 , 5.52561978111e-05 ##### | |
| ##### Sending to neptune: pred_reward : 0.599947186046 , 0.468498200178 ##### | |
| ##### Sending to neptune: max_logit : 0.599947186046 , 0.22827039659 ##### | |
| [u'loss', -0.00632883608341217, -0.03384819254279137, -2.2836740016937256, 1.5074315071105957, 5.525619781110436e-05, 0.46849820017814636, 0.22827039659023285] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.599947664407 , 9739097.15 ##### | |
| ##### Sending to neptune: dp_per_s : 0.599947664407 , 89.2479513768 ##### | |
| [u'other', 9739097.15, 89.24795137675618] | |
| receiving | |
| [0130 18:06:02 @multigpu.py:323] [p0576] step: count(1510), step_time 1447.74, mean_step_time 1431.25, it/s 0.7 | |
| [0130 18:06:02 @multigpu.py:323] [p0115] step: count(1486), step_time 1410.68, mean_step_time 1433.95, it/s 0.7 | |
| [0130 18:06:02 @multigpu.py:323] [p0574] step: count(1501), step_time 1438.43, mean_step_time 1436.85, it/s 0.7 | |
| [0130 18:06:03 @multigpu.py:323] [p0576] step: count(1511), step_time 1375.44, mean_step_time 1428.92, it/s 0.7 | |
| [0130 18:06:03 @multigpu.py:323] [p0115] step: count(1487), step_time 1439.88, mean_step_time 1433.69, it/s 0.7 | |
| [0130 18:06:03 @multigpu.py:323] [p0574] step: count(1502), step_time 1416.34, mean_step_time 1437.3, it/s 0.7 | |
| [0130 18:06:05 @multigpu.py:323] [p0115] step: count(1488), step_time 1430.65, mean_step_time 1432.4, it/s 0.7 | |
| [0130 18:06:05 @multigpu.py:323] [p0574] step: count(1503), step_time 1426.86, mean_step_time 1432.45, it/s 0.7 | |
| [0130 18:06:05 @multigpu.py:323] [p0576] step: count(1512), step_time 1578.32, mean_step_time 1432.39, it/s 0.7 | |
| [0130 18:06:06 @multigpu.py:323] [p0574] step: count(1504), step_time 1401.97, mean_step_time 1432.45, it/s 0.7 | |
| [0130 18:06:06 @multigpu.py:323] [p0115] step: count(1489), step_time 1416.3, mean_step_time 1432.44, it/s 0.7 | |
| [0130 18:06:06 @multigpu.py:323] [p0576] step: count(1513), step_time 1443.97, mean_step_time 1432.37, it/s 0.7 | |
| [0130 18:06:08 @multigpu.py:323] [p0576] step: count(1514), step_time 1383.8, mean_step_time 1431.03, it/s 0.7 | |
| [0130 18:06:08 @multigpu.py:323] [p0574] step: count(1505), step_time 1493.87, mean_step_time 1431.32, it/s 0.7 | |
| [0130 18:06:08 @multigpu.py:323] [p0115] step: count(1490), step_time 1512.18, mean_step_time 1435.73, it/s 0.7 | |
| [0130 18:06:09 @multigpu.py:323] [p0576] step: count(1515), step_time 1453.69, mean_step_time 1432.83, it/s 0.7 | |
| [0130 18:06:09 @multigpu.py:323] [p0115] step: count(1491), step_time 1375.9, mean_step_time 1435.42, it/s 0.7 | |
| [0130 18:06:09 @multigpu.py:323] [p0574] step: count(1506), step_time 1456.25, mean_step_time 1431.31, it/s 0.7 | |
| [0130 18:06:10 @multigpu.py:323] [p0576] step: count(1516), step_time 1394.5, mean_step_time 1429.13, it/s 0.7 | |
| [0130 18:06:10 @multigpu.py:323] [p0574] step: count(1507), step_time 1372.31, mean_step_time 1426.77, it/s 0.7 | |
| [0130 18:06:10 @multigpu.py:323] [p0115] step: count(1492), step_time 1453.23, mean_step_time 1436.06, it/s 0.7 | |
| [0130 18:06:12 @multigpu.py:323] [p0574] step: count(1508), step_time 1397.08, mean_step_time 1427.77, it/s 0.7 | |
| [0130 18:06:12 @multigpu.py:323] [p0576] step: count(1517), step_time 1456.13, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:06:12 @multigpu.py:323] [p0115] step: count(1493), step_time 1446.72, mean_step_time 1437.02, it/s 0.7 | |
| [0130 18:06:13 @multigpu.py:323] [p0576] step: count(1518), step_time 1364.42, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:06:13 @multigpu.py:323] [p0574] step: count(1509), step_time 1491.4, mean_step_time 1430.74, it/s 0.7 | |
| [0130 18:06:13 @multigpu.py:323] [p0115] step: count(1494), step_time 1462.95, mean_step_time 1440.7, it/s 0.69 | |
| [0130 18:06:15 @multigpu.py:323] [p0576] step: count(1519), step_time 1408.41, mean_step_time 1428.55, it/s 0.7 | |
| [0130 18:06:15 @multigpu.py:323] [p0574] step: count(1510), step_time 1446.65, mean_step_time 1431.77, it/s 0.7 | |
| [0130 18:06:15 @multigpu.py:323] [p0115] step: count(1495), step_time 1475.53, mean_step_time 1442.0, it/s 0.69 | |
| [0130 18:06:16 @multigpu.py:323] [p0576] step: count(1520), step_time 1400.12, mean_step_time 1426.37, it/s 0.7 | |
| [0130 18:06:16 @multigpu.py:323] [p0574] step: count(1511), step_time 1419.34, mean_step_time 1430.71, it/s 0.7 | |
| [0130 18:06:16 @multigpu.py:323] [p0115] step: count(1496), step_time 1459.53, mean_step_time 1441.35, it/s 0.69 | |
| [0130 18:06:17 @multigpu.py:323] [p0576] step: count(1521), step_time 1461.32, mean_step_time 1427.83, it/s 0.7 | |
| [0130 18:06:18 @multigpu.py:323] [p0574] step: count(1512), step_time 1409.34, mean_step_time 1429.72, it/s 0.7 | |
| [0130 18:06:18 @multigpu.py:323] [p0115] step: count(1497), step_time 1461.02, mean_step_time 1442.65, it/s 0.69 | |
| [0130 18:06:19 @multigpu.py:323] [p0576] step: count(1522), step_time 1395.16, mean_step_time 1425.77, it/s 0.7 | |
| [0130 18:06:19 @multigpu.py:323] [p0574] step: count(1513), step_time 1454.11, mean_step_time 1431.14, it/s 0.7 | |
| [0130 18:06:19 @multigpu.py:323] [p0115] step: count(1498), step_time 1485.19, mean_step_time 1444.02, it/s 0.69 | |
| [0130 18:06:20 @multigpu.py:323] [p0576] step: count(1523), step_time 1449.23, mean_step_time 1425.71, it/s 0.7 | |
| [0130 18:06:20 @multigpu.py:323] [p0574] step: count(1514), step_time 1400.58, mean_step_time 1428.02, it/s 0.7 | |
| [0130 18:06:21 @multigpu.py:323] [p0115] step: count(1499), step_time 1429.95, mean_step_time 1444.26, it/s 0.69 | |
| [0130 18:06:22 @multigpu.py:323] [p0576] step: count(1524), step_time 1418.53, mean_step_time 1426.35, it/s 0.7 | |
| [0130 18:06:22 @multigpu.py:323] [p0574] step: count(1515), step_time 1416.47, mean_step_time 1428.8, it/s 0.7 | |
| [0130 18:06:22 @multigpu.py:323] [p0115] step: count(1500), step_time 1381.96, mean_step_time 1441.99, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.605960784952 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.605960784952 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.605960784952 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.605961283313 , -0.00497105671093 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.605961283313 , 0.0196081455797 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.605961283313 , -2.28358197212 ##### | |
| ##### Sending to neptune: value_loss : 0.605961283313 , 1.62767851353 ##### | |
| ##### Sending to neptune: advantage : 0.605961283313 , 0.000106235434941 ##### | |
| ##### Sending to neptune: pred_reward : 0.605961283313 , 0.468377023935 ##### | |
| ##### Sending to neptune: max_logit : 0.605961283313 , 0.224669262767 ##### | |
| [u'loss', -0.0049710567109286785, 0.0196081455796957, -2.2835819721221924, 1.6276785135269165, 0.0001062354349414818, 0.468377023935318, 0.22466926276683807] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.605961754984 , 9702806.99 ##### | |
| ##### Sending to neptune: dp_per_s : 0.605961754984 , 89.1716589292 ##### | |
| [u'other', 9702806.99, 89.17165892922594] | |
| receiving | |
| [0130 18:06:23 @multigpu.py:323] [p0576] step: count(1525), step_time 1436.64, mean_step_time 1426.78, it/s 0.7 | |
| [0130 18:06:23 @multigpu.py:323] [p0574] step: count(1516), step_time 1442.65, mean_step_time 1430.47, it/s 0.7 | |
| [0130 18:06:23 @multigpu.py:323] [p0115] step: count(1501), step_time 1436.4, mean_step_time 1440.35, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.606569805278 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 18:06:25 @multigpu.py:323] [p0576] step: count(1526), step_time 1410.51, mean_step_time 1426.72, it/s 0.7 | |
| [0130 18:06:25 @multigpu.py:323] [p0574] step: count(1517), step_time 1453.84, mean_step_time 1431.07, it/s 0.7 | |
| [0130 18:06:25 @multigpu.py:323] [p0115] step: count(1502), step_time 1457.57, mean_step_time 1440.47, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.606775739657 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.606916066607 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 18:06:26 @multigpu.py:323] [p0576] step: count(1527), step_time 1413.27, mean_step_time 1426.91, it/s 0.7 | |
| [0130 18:06:26 @multigpu.py:323] [p0574] step: count(1518), step_time 1464.12, mean_step_time 1434.91, it/s 0.7 | |
| [0130 18:06:26 @multigpu.py:323] [p0115] step: count(1503), step_time 1385.83, mean_step_time 1437.36, it/s 0.7 | |
| [0130 18:06:27 @multigpu.py:323] [p0576] step: count(1528), step_time 1397.73, mean_step_time 1426.11, it/s 0.7 | |
| [0130 18:06:28 @multigpu.py:323] [p0574] step: count(1519), step_time 1409.67, mean_step_time 1431.69, it/s 0.7 | |
| [0130 18:06:28 @multigpu.py:323] [p0115] step: count(1504), step_time 1424.17, mean_step_time 1438.19, it/s 0.7 | |
| [0130 18:06:29 @multigpu.py:323] [p0576] step: count(1529), step_time 1454.14, mean_step_time 1427.15, it/s 0.7 | |
| [0130 18:06:29 @multigpu.py:323] [p0574] step: count(1520), step_time 1500.47, mean_step_time 1435.59, it/s 0.7 | |
| [0130 18:06:29 @multigpu.py:323] [p0115] step: count(1505), step_time 1486.96, mean_step_time 1441.63, it/s 0.69 | |
| [0130 18:06:30 @multigpu.py:323] [p0576] step: count(1530), step_time 1396.72, mean_step_time 1424.6, it/s 0.7 | |
| [0130 18:06:31 @multigpu.py:323] [p0574] step: count(1521), step_time 1421.62, mean_step_time 1434.75, it/s 0.7 | |
| [0130 18:06:31 @multigpu.py:323] [p0115] step: count(1506), step_time 1443.23, mean_step_time 1443.26, it/s 0.69 | |
| [0130 18:06:32 @multigpu.py:323] [p0576] step: count(1531), step_time 1410.17, mean_step_time 1426.34, it/s 0.7 | |
| [0130 18:06:32 @multigpu.py:323] [p0574] step: count(1522), step_time 1412.52, mean_step_time 1434.56, it/s 0.7 | |
| [0130 18:06:32 @multigpu.py:323] [p0115] step: count(1507), step_time 1414.24, mean_step_time 1441.98, it/s 0.69 | |
| [0130 18:06:34 @multigpu.py:323] [p0115] step: count(1508), step_time 1425.03, mean_step_time 1441.69, it/s 0.69 | |
| [0130 18:06:34 @multigpu.py:323] [p0574] step: count(1523), step_time 1528.19, mean_step_time 1439.62, it/s 0.69 | |
| [0130 18:06:34 @multigpu.py:323] [p0576] step: count(1532), step_time 1884.68, mean_step_time 1441.66, it/s 0.69 | |
| [0130 18:06:35 @multigpu.py:323] [p0576] step: count(1533), step_time 1429.69, mean_step_time 1440.94, it/s 0.69 | |
| [0130 18:06:35 @multigpu.py:323] [p0115] step: count(1509), step_time 1459.08, mean_step_time 1443.83, it/s 0.69 | |
| [0130 18:06:35 @multigpu.py:323] [p0574] step: count(1524), step_time 1493.5, mean_step_time 1444.2, it/s 0.69 | |
| [0130 18:06:36 @multigpu.py:323] [p0576] step: count(1534), step_time 1403.82, mean_step_time 1441.94, it/s 0.69 | |
| [0130 18:06:36 @multigpu.py:323] [p0115] step: count(1510), step_time 1406.96, mean_step_time 1438.57, it/s 0.7 | |
| [0130 18:06:36 @multigpu.py:323] [p0574] step: count(1525), step_time 1449.17, mean_step_time 1441.96, it/s 0.69 | |
| [0130 18:06:38 @multigpu.py:323] [p0576] step: count(1535), step_time 1401.52, mean_step_time 1439.34, it/s 0.69 | |
| [0130 18:06:38 @multigpu.py:323] [p0115] step: count(1511), step_time 1409.91, mean_step_time 1440.27, it/s 0.69 | |
| [0130 18:06:38 @multigpu.py:323] [p0574] step: count(1526), step_time 1434.71, mean_step_time 1440.89, it/s 0.69 | |
| [0130 18:06:39 @multigpu.py:323] [p0115] step: count(1512), step_time 1424.66, mean_step_time 1438.84, it/s 0.7 | |
| [0130 18:06:39 @multigpu.py:323] [p0576] step: count(1536), step_time 1467.83, mean_step_time 1443.0, it/s 0.69 | |
| [0130 18:06:39 @multigpu.py:323] [p0574] step: count(1527), step_time 1399.48, mean_step_time 1442.25, it/s 0.69 | |
| [0130 18:06:41 @multigpu.py:323] [p0115] step: count(1513), step_time 1423.8, mean_step_time 1437.7, it/s 0.7 | |
| [0130 18:06:41 @multigpu.py:323] [p0576] step: count(1537), step_time 1449.03, mean_step_time 1442.65, it/s 0.69 | |
| [0130 18:06:41 @multigpu.py:323] [p0574] step: count(1528), step_time 1401.56, mean_step_time 1442.47, it/s 0.69 | |
| [0130 18:06:42 @multigpu.py:323] [p0115] step: count(1514), step_time 1410.1, mean_step_time 1435.06, it/s 0.7 | |
| [0130 18:06:42 @multigpu.py:323] [p0576] step: count(1538), step_time 1480.84, mean_step_time 1448.47, it/s 0.69 | |
| [0130 18:06:42 @multigpu.py:323] [p0574] step: count(1529), step_time 1472.88, mean_step_time 1441.54, it/s 0.69 | |
| [0130 18:06:43 @multigpu.py:323] [p0115] step: count(1515), step_time 1391.68, mean_step_time 1430.86, it/s 0.7 | |
| [0130 18:06:44 @multigpu.py:323] [p0574] step: count(1530), step_time 1419.66, mean_step_time 1440.19, it/s 0.69 | |
| [0130 18:06:44 @multigpu.py:323] [p0576] step: count(1539), step_time 1470.23, mean_step_time 1451.56, it/s 0.69 | |
| [0130 18:06:45 @multigpu.py:323] [p0115] step: count(1516), step_time 1472.46, mean_step_time 1431.51, it/s 0.7 | |
| [0130 18:06:45 @multigpu.py:323] [p0576] step: count(1540), step_time 1379.21, mean_step_time 1450.51, it/s 0.69 | |
| [0130 18:06:45 @multigpu.py:323] [p0574] step: count(1531), step_time 1432.77, mean_step_time 1440.87, it/s 0.69 | |
| [0130 18:06:46 @multigpu.py:323] [p0115] step: count(1517), step_time 1476.83, mean_step_time 1432.3, it/s 0.7 | |
| [0130 18:06:46 @multigpu.py:323] [p0574] step: count(1532), step_time 1408.65, mean_step_time 1440.83, it/s 0.69 | |
| [0130 18:06:46 @multigpu.py:323] [p0576] step: count(1541), step_time 1439.1, mean_step_time 1449.4, it/s 0.69 | |
| [0130 18:06:48 @multigpu.py:323] [p0115] step: count(1518), step_time 1387.91, mean_step_time 1427.44, it/s 0.7 | |
| [0130 18:06:48 @multigpu.py:323] [p0576] step: count(1542), step_time 1395.19, mean_step_time 1449.4, it/s 0.69 | |
| [0130 18:06:48 @multigpu.py:323] [p0574] step: count(1533), step_time 1452.84, mean_step_time 1440.77, it/s 0.69 | |
| [0130 18:06:49 @multigpu.py:323] [p0576] step: count(1543), step_time 1407.96, mean_step_time 1447.34, it/s 0.69 | |
| [0130 18:06:49 @multigpu.py:323] [p0115] step: count(1519), step_time 1479.98, mean_step_time 1429.94, it/s 0.7 | |
| [0130 18:06:49 @multigpu.py:323] [p0574] step: count(1534), step_time 1448.62, mean_step_time 1443.17, it/s 0.69 | |
| [0130 18:06:51 @multigpu.py:323] [p0576] step: count(1544), step_time 1377.59, mean_step_time 1445.29, it/s 0.69 | |
| [0130 18:06:51 @multigpu.py:323] [p0115] step: count(1520), step_time 1407.82, mean_step_time 1431.23, it/s 0.7 | |
| [0130 18:06:51 @multigpu.py:323] [p0574] step: count(1535), step_time 1427.36, mean_step_time 1443.72, it/s 0.69 | |
| [0130 18:06:52 @multigpu.py:323] [p0576] step: count(1545), step_time 1436.21, mean_step_time 1445.27, it/s 0.69 | |
| [0130 18:06:52 @multigpu.py:323] [p0115] step: count(1521), step_time 1458.2, mean_step_time 1432.32, it/s 0.7 | |
| [0130 18:06:52 @multigpu.py:323] [p0574] step: count(1536), step_time 1439.56, mean_step_time 1443.56, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.614415989121 , 0.8 ##### | |
| [u'online', 0.8] | |
| ##### Sending to neptune: active_workers : 0.614416100515 , 3 ##### | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.614469838606 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:06:53 @multigpu.py:323] [p0576] step: count(1546), step_time 1365.79, mean_step_time 1443.04, it/s 0.69 | |
| [0130 18:06:54 @multigpu.py:323] [p0115] step: count(1522), step_time 1442.97, mean_step_time 1431.59, it/s 0.7 | |
| [0130 18:06:54 @multigpu.py:323] [p0574] step: count(1537), step_time 1399.53, mean_step_time 1440.85, it/s 0.69 | |
| [0130 18:06:55 @multigpu.py:323] [p0576] step: count(1547), step_time 1463.65, mean_step_time 1445.56, it/s 0.69 | |
| [0130 18:06:55 @multigpu.py:323] [p0115] step: count(1523), step_time 1383.89, mean_step_time 1431.5, it/s 0.7 | |
| [0130 18:06:55 @multigpu.py:323] [p0574] step: count(1538), step_time 1405.73, mean_step_time 1437.93, it/s 0.7 | |
| [0130 18:06:56 @multigpu.py:323] [p0576] step: count(1548), step_time 1425.38, mean_step_time 1446.94, it/s 0.69 | |
| [0130 18:06:56 @multigpu.py:323] [p0115] step: count(1524), step_time 1458.14, mean_step_time 1433.19, it/s 0.7 | |
| [0130 18:06:57 @multigpu.py:323] [p0574] step: count(1539), step_time 1469.78, mean_step_time 1440.93, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.615593735774 , 0.8 ##### | |
| [u'online', 0.8] | |
| receiving | |
| [0130 18:06:58 @multigpu.py:323] [p0576] step: count(1549), step_time 1398.88, mean_step_time 1444.17, it/s 0.69 | |
| [0130 18:06:58 @multigpu.py:323] [p0115] step: count(1525), step_time 1391.89, mean_step_time 1428.44, it/s 0.7 | |
| [0130 18:06:58 @multigpu.py:323] [p0574] step: count(1540), step_time 1427.82, mean_step_time 1437.3, it/s 0.7 | |
| [0130 18:06:59 @multigpu.py:323] [p0576] step: count(1550), step_time 1448.98, mean_step_time 1446.79, it/s 0.69 | |
| [0130 18:06:59 @multigpu.py:323] [p0115] step: count(1526), step_time 1389.32, mean_step_time 1425.74, it/s 0.7 | |
| [0130 18:06:59 @multigpu.py:323] [p0574] step: count(1541), step_time 1421.4, mean_step_time 1437.29, it/s 0.7 | |
| [0130 18:07:01 @multigpu.py:323] [p0115] step: count(1527), step_time 1414.52, mean_step_time 1425.76, it/s 0.7 | |
| [0130 18:07:01 @multigpu.py:323] [p0576] step: count(1551), step_time 1488.68, mean_step_time 1450.71, it/s 0.69 | |
| [0130 18:07:01 @multigpu.py:323] [p0574] step: count(1542), step_time 1428.02, mean_step_time 1438.06, it/s 0.7 | |
| [0130 18:07:02 @multigpu.py:323] [p0574] step: count(1543), step_time 1433.98, mean_step_time 1433.35, it/s 0.7 | |
| [0130 18:07:02 @multigpu.py:323] [p0115] step: count(1528), step_time 1574.99, mean_step_time 1433.26, it/s 0.7 | |
| [0130 18:07:02 @multigpu.py:323] [p0576] step: count(1552), step_time 1535.72, mean_step_time 1433.27, it/s 0.7 | |
| [0130 18:07:04 @multigpu.py:323] [p0115] step: count(1529), step_time 1408.31, mean_step_time 1430.72, it/s 0.7 | |
| [0130 18:07:04 @multigpu.py:323] [p0576] step: count(1553), step_time 1426.85, mean_step_time 1433.12, it/s 0.7 | |
| [0130 18:07:04 @multigpu.py:323] [p0574] step: count(1544), step_time 1444.66, mean_step_time 1430.91, it/s 0.7 | |
| [0130 18:07:05 @multigpu.py:323] [p0576] step: count(1554), step_time 1411.79, mean_step_time 1433.52, it/s 0.7 | |
| [0130 18:07:05 @multigpu.py:323] [p0115] step: count(1530), step_time 1433.44, mean_step_time 1432.04, it/s 0.7 | |
| [0130 18:07:05 @multigpu.py:323] [p0574] step: count(1545), step_time 1416.43, mean_step_time 1429.27, it/s 0.7 | |
| [0130 18:07:06 @multigpu.py:323] [p0576] step: count(1555), step_time 1407.51, mean_step_time 1433.82, it/s 0.7 | |
| [0130 18:07:06 @multigpu.py:323] [p0115] step: count(1531), step_time 1427.83, mean_step_time 1432.94, it/s 0.7 | |
| [0130 18:07:07 @multigpu.py:323] [p0574] step: count(1546), step_time 1441.64, mean_step_time 1429.62, it/s 0.7 | |
| [0130 18:07:08 @multigpu.py:323] [p0576] step: count(1556), step_time 1427.75, mean_step_time 1431.82, it/s 0.7 | |
| [0130 18:07:08 @multigpu.py:323] [p0115] step: count(1532), step_time 1468.38, mean_step_time 1435.12, it/s 0.7 | |
| [0130 18:07:08 @multigpu.py:323] [p0574] step: count(1547), step_time 1443.35, mean_step_time 1431.81, it/s 0.7 | |
| [0130 18:07:09 @multigpu.py:323] [p0576] step: count(1557), step_time 1440.46, mean_step_time 1431.39, it/s 0.7 | |
| [0130 18:07:09 @multigpu.py:323] [p0115] step: count(1533), step_time 1405.02, mean_step_time 1434.19, it/s 0.7 | |
| [0130 18:07:09 @multigpu.py:323] [p0574] step: count(1548), step_time 1479.32, mean_step_time 1435.7, it/s 0.7 | |
| [0130 18:07:11 @multigpu.py:323] [p0576] step: count(1558), step_time 1434.45, mean_step_time 1429.07, it/s 0.7 | |
| [0130 18:07:11 @multigpu.py:323] [p0115] step: count(1534), step_time 1436.15, mean_step_time 1435.49, it/s 0.7 | |
| [0130 18:07:11 @multigpu.py:323] [p0574] step: count(1549), step_time 1454.5, mean_step_time 1434.78, it/s 0.7 | |
| [0130 18:07:12 @multigpu.py:323] [p0576] step: count(1559), step_time 1427.64, mean_step_time 1426.94, it/s 0.7 | |
| [0130 18:07:12 @multigpu.py:323] [p0115] step: count(1535), step_time 1461.25, mean_step_time 1438.97, it/s 0.69 | |
| [0130 18:07:12 @multigpu.py:323] [p0574] step: count(1550), step_time 1408.36, mean_step_time 1434.22, it/s 0.7 | |
| [0130 18:07:14 @multigpu.py:323] [p0576] step: count(1560), step_time 1428.16, mean_step_time 1429.39, it/s 0.7 | |
| [0130 18:07:14 @multigpu.py:323] [p0115] step: count(1536), step_time 1409.0, mean_step_time 1435.79, it/s 0.7 | |
| [0130 18:07:14 @multigpu.py:323] [p0574] step: count(1551), step_time 1414.37, mean_step_time 1433.3, it/s 0.7 | |
| [0130 18:07:15 @multigpu.py:323] [p0576] step: count(1561), step_time 1409.87, mean_step_time 1427.93, it/s 0.7 | |
| [0130 18:07:15 @multigpu.py:323] [p0115] step: count(1537), step_time 1457.82, mean_step_time 1434.84, it/s 0.7 | |
| [0130 18:07:15 @multigpu.py:323] [p0574] step: count(1552), step_time 1425.49, mean_step_time 1434.14, it/s 0.7 | |
| [0130 18:07:16 @multigpu.py:323] [p0576] step: count(1562), step_time 1441.37, mean_step_time 1430.23, it/s 0.7 | |
| [0130 18:07:17 @multigpu.py:323] [p0115] step: count(1538), step_time 1399.1, mean_step_time 1435.4, it/s 0.7 | |
| [0130 18:07:17 @multigpu.py:323] [p0574] step: count(1553), step_time 1392.19, mean_step_time 1431.11, it/s 0.7 | |
| [0130 18:07:18 @multigpu.py:323] [p0576] step: count(1563), step_time 1375.9, mean_step_time 1428.63, it/s 0.7 | |
| [0130 18:07:18 @multigpu.py:323] [p0115] step: count(1539), step_time 1396.81, mean_step_time 1431.24, it/s 0.7 | |
| [0130 18:07:18 @multigpu.py:323] [p0574] step: count(1554), step_time 1443.34, mean_step_time 1430.84, it/s 0.7 | |
| [0130 18:07:19 @multigpu.py:323] [p0576] step: count(1564), step_time 1417.54, mean_step_time 1430.63, it/s 0.7 | |
| [0130 18:07:19 @multigpu.py:323] [p0115] step: count(1540), step_time 1436.86, mean_step_time 1432.7, it/s 0.7 | |
| [0130 18:07:19 @multigpu.py:323] [p0574] step: count(1555), step_time 1447.6, mean_step_time 1431.85, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.622098054952 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 18:07:21 @multigpu.py:323] [p0576] step: count(1565), step_time 1399.06, mean_step_time 1428.77, it/s 0.7 | |
| [0130 18:07:21 @multigpu.py:323] [p0115] step: count(1541), step_time 1431.0, mean_step_time 1431.34, it/s 0.7 | |
| [0130 18:07:21 @multigpu.py:323] [p0574] step: count(1556), step_time 1404.22, mean_step_time 1430.09, it/s 0.7 | |
| [0130 18:07:22 @multigpu.py:323] [p0576] step: count(1566), step_time 1383.49, mean_step_time 1429.66, it/s 0.7 | |
| [0130 18:07:22 @multigpu.py:323] [p0574] step: count(1557), step_time 1425.37, mean_step_time 1431.38, it/s 0.7 | |
| [0130 18:07:22 @multigpu.py:323] [p0115] step: count(1542), step_time 1472.93, mean_step_time 1432.83, it/s 0.7 | |
| [0130 18:07:24 @multigpu.py:323] [p0576] step: count(1567), step_time 1476.75, mean_step_time 1430.31, it/s 0.7 | |
| [0130 18:07:24 @multigpu.py:323] [p0115] step: count(1543), step_time 1435.89, mean_step_time 1435.43, it/s 0.7 | |
| [0130 18:07:24 @multigpu.py:323] [p0574] step: count(1558), step_time 1442.92, mean_step_time 1433.24, it/s 0.7 | |
| [0130 18:07:25 @multigpu.py:323] [p0576] step: count(1568), step_time 1391.28, mean_step_time 1428.61, it/s 0.7 | |
| [0130 18:07:25 @multigpu.py:323] [p0574] step: count(1559), step_time 1423.02, mean_step_time 1430.9, it/s 0.7 | |
| [0130 18:07:25 @multigpu.py:323] [p0115] step: count(1544), step_time 1434.17, mean_step_time 1434.23, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.623693410556 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 18:07:26 @multigpu.py:323] [p0576] step: count(1569), step_time 1386.62, mean_step_time 1427.99, it/s 0.7 | |
| [0130 18:07:27 @multigpu.py:323] [p0574] step: count(1560), step_time 1398.7, mean_step_time 1429.44, it/s 0.7 | |
| [0130 18:07:27 @multigpu.py:323] [p0115] step: count(1545), step_time 1414.45, mean_step_time 1435.36, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.623916657435 , 1.0 ##### | |
| [u'online', 1.0] | |
| receiving | |
| [0130 18:07:28 @multigpu.py:323] [p0576] step: count(1570), step_time 1444.13, mean_step_time 1427.75, it/s 0.7 | |
| [0130 18:07:28 @multigpu.py:323] [p0574] step: count(1561), step_time 1390.77, mean_step_time 1427.91, it/s 0.7 | |
| [0130 18:07:28 @multigpu.py:323] [p0115] step: count(1546), step_time 1464.95, mean_step_time 1439.14, it/s 0.69 | |
| [0130 18:07:29 @multigpu.py:323] [p0576] step: count(1571), step_time 1393.22, mean_step_time 1422.98, it/s 0.7 | |
| [0130 18:07:29 @multigpu.py:323] [p0574] step: count(1562), step_time 1461.98, mean_step_time 1429.61, it/s 0.7 | |
| [0130 18:07:29 @multigpu.py:323] [p0115] step: count(1547), step_time 1375.92, mean_step_time 1437.21, it/s 0.7 | |
| [0130 18:07:31 @multigpu.py:323] [p0574] step: count(1563), step_time 1461.96, mean_step_time 1431.01, it/s 0.7 | |
| [0130 18:07:31 @multigpu.py:323] [p0115] step: count(1548), step_time 1447.15, mean_step_time 1430.82, it/s 0.7 | |
| [0130 18:07:31 @multigpu.py:323] [p0576] step: count(1572), step_time 1695.82, mean_step_time 1430.98, it/s 0.7 | |
| [0130 18:07:32 @multigpu.py:323] [p0574] step: count(1564), step_time 1392.05, mean_step_time 1428.38, it/s 0.7 | |
| [0130 18:07:32 @multigpu.py:323] [p0115] step: count(1549), step_time 1419.71, mean_step_time 1431.39, it/s 0.7 | |
| [0130 18:07:32 @multigpu.py:323] [p0576] step: count(1573), step_time 1422.03, mean_step_time 1430.74, it/s 0.7 | |
| [0130 18:07:34 @multigpu.py:323] [p0576] step: count(1574), step_time 1374.04, mean_step_time 1428.85, it/s 0.7 | |
| [0130 18:07:34 @multigpu.py:323] [p0574] step: count(1565), step_time 1451.87, mean_step_time 1430.15, it/s 0.7 | |
| [0130 18:07:34 @multigpu.py:323] [p0115] step: count(1550), step_time 1432.63, mean_step_time 1431.35, it/s 0.7 | |
| [0130 18:07:35 @multigpu.py:323] [p0576] step: count(1575), step_time 1461.69, mean_step_time 1431.56, it/s 0.7 | |
| [0130 18:07:35 @multigpu.py:323] [p0574] step: count(1566), step_time 1442.13, mean_step_time 1430.18, it/s 0.7 | |
| [0130 18:07:35 @multigpu.py:323] [p0115] step: count(1551), step_time 1473.36, mean_step_time 1433.63, it/s 0.7 | |
| [0130 18:07:37 @multigpu.py:323] [p0576] step: count(1576), step_time 1428.61, mean_step_time 1431.61, it/s 0.7 | |
| [0130 18:07:37 @multigpu.py:323] [p0574] step: count(1567), step_time 1442.95, mean_step_time 1430.16, it/s 0.7 | |
| [0130 18:07:37 @multigpu.py:323] [p0115] step: count(1552), step_time 1424.53, mean_step_time 1431.44, it/s 0.7 | |
| [0130 18:07:38 @multigpu.py:323] [p0576] step: count(1577), step_time 1451.4, mean_step_time 1432.15, it/s 0.7 | |
| [0130 18:07:38 @multigpu.py:323] [p0115] step: count(1553), step_time 1432.63, mean_step_time 1432.82, it/s 0.7 | |
| [0130 18:07:38 @multigpu.py:323] [p0574] step: count(1568), step_time 1483.12, mean_step_time 1430.35, it/s 0.7 | |
| [0130 18:07:39 @multigpu.py:323] [p0576] step: count(1578), step_time 1415.32, mean_step_time 1431.2, it/s 0.7 | |
| [0130 18:07:39 @multigpu.py:323] [p0115] step: count(1554), step_time 1393.06, mean_step_time 1430.66, it/s 0.7 | |
| [0130 18:07:40 @multigpu.py:323] [p0574] step: count(1569), step_time 1439.62, mean_step_time 1429.6, it/s 0.7 | |
| [0130 18:07:41 @multigpu.py:323] [p0115] step: count(1555), step_time 1408.59, mean_step_time 1428.03, it/s 0.7 | |
| [0130 18:07:41 @multigpu.py:323] [p0576] step: count(1579), step_time 1419.95, mean_step_time 1430.81, it/s 0.7 | |
| [0130 18:07:41 @multigpu.py:323] [p0574] step: count(1570), step_time 1429.09, mean_step_time 1430.64, it/s 0.7 | |
| [0130 18:07:42 @multigpu.py:323] [p0576] step: count(1580), step_time 1417.08, mean_step_time 1430.26, it/s 0.7 | |
| [0130 18:07:42 @multigpu.py:323] [p0115] step: count(1556), step_time 1468.9, mean_step_time 1431.02, it/s 0.7 | |
| [0130 18:07:42 @multigpu.py:323] [p0574] step: count(1571), step_time 1420.37, mean_step_time 1430.94, it/s 0.7 | |
| [0130 18:07:44 @multigpu.py:323] [p0576] step: count(1581), step_time 1426.62, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:07:44 @multigpu.py:323] [p0115] step: count(1557), step_time 1384.78, mean_step_time 1427.37, it/s 0.7 | |
| [0130 18:07:44 @multigpu.py:323] [p0574] step: count(1572), step_time 1442.39, mean_step_time 1431.78, it/s 0.7 | |
| [0130 18:07:45 @multigpu.py:323] [p0115] step: count(1558), step_time 1406.15, mean_step_time 1427.72, it/s 0.7 | |
| [0130 18:07:45 @multigpu.py:323] [p0576] step: count(1582), step_time 1482.99, mean_step_time 1433.18, it/s 0.7 | |
| [0130 18:07:45 @multigpu.py:323] [p0574] step: count(1573), step_time 1434.9, mean_step_time 1433.92, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.629087399708 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:07:47 @multigpu.py:323] [p0115] step: count(1559), step_time 1429.28, mean_step_time 1429.35, it/s 0.7 | |
| [0130 18:07:47 @multigpu.py:323] [p0576] step: count(1583), step_time 1467.46, mean_step_time 1437.75, it/s 0.7 | |
| [0130 18:07:47 @multigpu.py:323] [p0574] step: count(1574), step_time 1448.98, mean_step_time 1434.2, it/s 0.7 | |
| [0130 18:07:48 @multigpu.py:323] [p0115] step: count(1560), step_time 1374.07, mean_step_time 1426.21, it/s 0.7 | |
| [0130 18:07:48 @multigpu.py:323] [p0574] step: count(1575), step_time 1368.49, mean_step_time 1430.24, it/s 0.7 | |
| [0130 18:07:48 @multigpu.py:323] [p0576] step: count(1584), step_time 1464.55, mean_step_time 1440.1, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.630022522211 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:07:49 @multigpu.py:323] [p0115] step: count(1561), step_time 1453.9, mean_step_time 1427.35, it/s 0.7 | |
| [0130 18:07:50 @multigpu.py:323] [p0574] step: count(1576), step_time 1467.22, mean_step_time 1433.39, it/s 0.7 | |
| [0130 18:07:50 @multigpu.py:323] [p0576] step: count(1585), step_time 1512.55, mean_step_time 1445.78, it/s 0.69 | |
| [0130 18:07:51 @multigpu.py:323] [p0115] step: count(1562), step_time 1488.79, mean_step_time 1428.15, it/s 0.7 | |
| [0130 18:07:51 @multigpu.py:323] [p0574] step: count(1577), step_time 1436.96, mean_step_time 1433.97, it/s 0.7 | |
| [0130 18:07:51 @multigpu.py:323] [p0576] step: count(1586), step_time 1385.99, mean_step_time 1445.9, it/s 0.69 | |
| [0130 18:07:52 @multigpu.py:323] [p0115] step: count(1563), step_time 1464.72, mean_step_time 1429.59, it/s 0.7 | |
| [0130 18:07:52 @multigpu.py:323] [p0574] step: count(1578), step_time 1443.19, mean_step_time 1433.99, it/s 0.7 | |
| [0130 18:07:52 @multigpu.py:323] [p0576] step: count(1587), step_time 1410.31, mean_step_time 1442.58, it/s 0.69 | |
| [0130 18:07:54 @multigpu.py:323] [p0115] step: count(1564), step_time 1425.23, mean_step_time 1429.14, it/s 0.7 | |
| [0130 18:07:54 @multigpu.py:323] [p0576] step: count(1588), step_time 1406.86, mean_step_time 1443.36, it/s 0.69 | |
| [0130 18:07:54 @multigpu.py:323] [p0574] step: count(1579), step_time 1450.85, mean_step_time 1435.38, it/s 0.7 | |
| [0130 18:07:55 @multigpu.py:323] [p0576] step: count(1589), step_time 1412.81, mean_step_time 1444.67, it/s 0.69 | |
| [0130 18:07:55 @multigpu.py:323] [p0115] step: count(1565), step_time 1504.99, mean_step_time 1433.67, it/s 0.7 | |
| [0130 18:07:55 @multigpu.py:323] [p0574] step: count(1580), step_time 1418.11, mean_step_time 1436.35, it/s 0.7 | |
| [0130 18:07:57 @multigpu.py:323] [p0115] step: count(1566), step_time 1413.67, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:07:57 @multigpu.py:323] [p0576] step: count(1590), step_time 1489.01, mean_step_time 1446.91, it/s 0.69 | |
| [0130 18:07:57 @multigpu.py:323] [p0574] step: count(1581), step_time 1467.07, mean_step_time 1440.17, it/s 0.69 | |
| [0130 18:07:58 @multigpu.py:323] [p0115] step: count(1567), step_time 1436.95, mean_step_time 1434.15, it/s 0.7 | |
| [0130 18:07:58 @multigpu.py:323] [p0576] step: count(1591), step_time 1394.18, mean_step_time 1446.96, it/s 0.69 | |
| [0130 18:07:58 @multigpu.py:323] [p0574] step: count(1582), step_time 1429.64, mean_step_time 1438.55, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.632678529951 , 2.6 ##### | |
| [u'online', 2.6] | |
| ##### Sending to neptune: active_workers : 0.632678614722 , 3 ##### | |
| receiving | |
| [0130 18:08:00 @multigpu.py:323] [p0115] step: count(1568), step_time 1510.66, mean_step_time 1437.33, it/s 0.7 | |
| [0130 18:08:00 @multigpu.py:323] [p0576] step: count(1592), step_time 1487.64, mean_step_time 1436.55, it/s 0.7 | |
| [0130 18:08:00 @multigpu.py:323] [p0574] step: count(1583), step_time 1436.04, mean_step_time 1437.25, it/s 0.7 | |
| [0130 18:08:01 @multigpu.py:323] [p0576] step: count(1593), step_time 1392.4, mean_step_time 1435.07, it/s 0.7 | |
| [0130 18:08:01 @multigpu.py:323] [p0574] step: count(1584), step_time 1434.4, mean_step_time 1439.37, it/s 0.69 | |
| [0130 18:08:01 @multigpu.py:323] [p0115] step: count(1569), step_time 1487.45, mean_step_time 1440.72, it/s 0.69 | |
| [0130 18:08:02 @multigpu.py:323] [p0576] step: count(1594), step_time 1403.35, mean_step_time 1436.54, it/s 0.7 | |
| [0130 18:08:02 @multigpu.py:323] [p0574] step: count(1585), step_time 1435.68, mean_step_time 1438.56, it/s 0.7 | |
| [0130 18:08:03 @multigpu.py:323] [p0115] step: count(1570), step_time 1410.24, mean_step_time 1439.6, it/s 0.69 | |
| [0130 18:08:04 @multigpu.py:323] [p0576] step: count(1595), step_time 1445.1, mean_step_time 1435.71, it/s 0.7 | |
| [0130 18:08:04 @multigpu.py:323] [p0574] step: count(1586), step_time 1411.55, mean_step_time 1437.03, it/s 0.7 | |
| [0130 18:08:04 @multigpu.py:323] [p0115] step: count(1571), step_time 1419.73, mean_step_time 1436.92, it/s 0.7 | |
| [0130 18:08:05 @multigpu.py:323] [p0576] step: count(1596), step_time 1388.89, mean_step_time 1433.72, it/s 0.7 | |
| [0130 18:08:05 @multigpu.py:323] [p0574] step: count(1587), step_time 1495.72, mean_step_time 1439.67, it/s 0.69 | |
| [0130 18:08:05 @multigpu.py:323] [p0115] step: count(1572), step_time 1478.4, mean_step_time 1439.61, it/s 0.69 | |
| [0130 18:08:07 @multigpu.py:323] [p0576] step: count(1597), step_time 1416.04, mean_step_time 1431.95, it/s 0.7 | |
| [0130 18:08:07 @multigpu.py:323] [p0574] step: count(1588), step_time 1388.06, mean_step_time 1434.92, it/s 0.7 | |
| [0130 18:08:07 @multigpu.py:323] [p0115] step: count(1573), step_time 1472.38, mean_step_time 1441.6, it/s 0.69 | |
| [0130 18:08:08 @multigpu.py:323] [p0576] step: count(1598), step_time 1402.14, mean_step_time 1431.3, it/s 0.7 | |
| [0130 18:08:08 @multigpu.py:323] [p0574] step: count(1589), step_time 1437.85, mean_step_time 1434.83, it/s 0.7 | |
| [0130 18:08:08 @multigpu.py:323] [p0115] step: count(1574), step_time 1455.63, mean_step_time 1444.73, it/s 0.69 | |
| [0130 18:08:10 @multigpu.py:323] [p0576] step: count(1599), step_time 1450.04, mean_step_time 1432.8, it/s 0.7 | |
| [0130 18:08:10 @multigpu.py:323] [p0574] step: count(1590), step_time 1438.73, mean_step_time 1435.31, it/s 0.7 | |
| [0130 18:08:10 @multigpu.py:323] [p0115] step: count(1575), step_time 1401.08, mean_step_time 1444.35, it/s 0.69 | |
| [0130 18:08:11 @multigpu.py:323] [p0576] step: count(1600), step_time 1404.48, mean_step_time 1432.17, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.636195923289 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.636195923289 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.636195923289 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.636196420259 , -0.00467890035361 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.636196420259 , 0.0171289928257 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.636196420259 , -2.28448009491 ##### | |
| ##### Sending to neptune: value_loss : 0.636196420259 , 1.66845166683 ##### | |
| ##### Sending to neptune: advantage : 0.636196420259 , -0.000201532704523 ##### | |
| ##### Sending to neptune: pred_reward : 0.636196420259 , 0.438157171011 ##### | |
| ##### Sending to neptune: max_logit : 0.636196420259 , 0.21130207181 ##### | |
| [u'loss', -0.004678900353610516, 0.017128992825746536, -2.284480094909668, 1.6684516668319702, -0.00020153270452283323, 0.43815717101097107, 0.21130207180976868] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.636196913587 , 9650551.52 ##### | |
| ##### Sending to neptune: dp_per_s : 0.636196913587 , 89.2264312438 ##### | |
| [u'other', 9650551.52, 89.22643124376607] | |
| receiving | |
| [0130 18:08:11 @multigpu.py:323] [p0574] step: count(1591), step_time 1391.27, mean_step_time 1433.85, it/s 0.7 | |
| [0130 18:08:11 @multigpu.py:323] [p0115] step: count(1576), step_time 1421.31, mean_step_time 1441.97, it/s 0.69 | |
| [0130 18:08:12 @multigpu.py:323] [p0576] step: count(1601), step_time 1404.77, mean_step_time 1431.08, it/s 0.7 | |
| [0130 18:08:12 @multigpu.py:323] [p0574] step: count(1592), step_time 1449.92, mean_step_time 1434.23, it/s 0.7 | |
| [0130 18:08:13 @multigpu.py:323] [p0115] step: count(1577), step_time 1391.85, mean_step_time 1442.32, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.636856070823 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 18:08:14 @multigpu.py:323] [p0576] step: count(1602), step_time 1425.91, mean_step_time 1428.22, it/s 0.7 | |
| [0130 18:08:14 @multigpu.py:323] [p0574] step: count(1593), step_time 1384.73, mean_step_time 1431.72, it/s 0.7 | |
| [0130 18:08:14 @multigpu.py:323] [p0115] step: count(1578), step_time 1428.74, mean_step_time 1443.45, it/s 0.69 | |
| [0130 18:08:15 @multigpu.py:323] [p0576] step: count(1603), step_time 1414.6, mean_step_time 1425.58, it/s 0.7 | |
| [0130 18:08:15 @multigpu.py:323] [p0574] step: count(1594), step_time 1450.8, mean_step_time 1431.81, it/s 0.7 | |
| [0130 18:08:15 @multigpu.py:323] [p0115] step: count(1579), step_time 1377.69, mean_step_time 1440.87, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.637642319401 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 18:08:17 @multigpu.py:323] [p0576] step: count(1604), step_time 1430.78, mean_step_time 1423.89, it/s 0.7 | |
| [0130 18:08:17 @multigpu.py:323] [p0574] step: count(1595), step_time 1372.67, mean_step_time 1432.02, it/s 0.7 | |
| [0130 18:08:17 @multigpu.py:323] [p0115] step: count(1580), step_time 1411.3, mean_step_time 1442.74, it/s 0.69 | |
| [0130 18:08:18 @multigpu.py:323] [p0576] step: count(1605), step_time 1378.87, mean_step_time 1417.21, it/s 0.71 | |
| [0130 18:08:18 @multigpu.py:323] [p0574] step: count(1596), step_time 1486.99, mean_step_time 1433.01, it/s 0.7 | |
| [0130 18:08:18 @multigpu.py:323] [p0115] step: count(1581), step_time 1444.54, mean_step_time 1442.27, it/s 0.69 | |
| [0130 18:08:19 @multigpu.py:323] [p0576] step: count(1606), step_time 1449.49, mean_step_time 1420.38, it/s 0.7 | |
| [0130 18:08:20 @multigpu.py:323] [p0574] step: count(1597), step_time 1386.55, mean_step_time 1430.49, it/s 0.7 | |
| [0130 18:08:20 @multigpu.py:323] [p0115] step: count(1582), step_time 1479.48, mean_step_time 1441.8, it/s 0.69 | |
| [0130 18:08:21 @multigpu.py:323] [p0576] step: count(1607), step_time 1398.84, mean_step_time 1419.81, it/s 0.7 | |
| [0130 18:08:21 @multigpu.py:323] [p0574] step: count(1598), step_time 1411.25, mean_step_time 1428.89, it/s 0.7 | |
| [0130 18:08:21 @multigpu.py:323] [p0115] step: count(1583), step_time 1442.11, mean_step_time 1440.67, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.639158487452 , 1.4 ##### | |
| [u'online', 1.4] | |
| receiving | |
| [0130 18:08:22 @multigpu.py:323] [p0576] step: count(1608), step_time 1409.88, mean_step_time 1419.96, it/s 0.7 | |
| [0130 18:08:22 @multigpu.py:323] [p0574] step: count(1599), step_time 1403.72, mean_step_time 1426.54, it/s 0.7 | |
| [0130 18:08:23 @multigpu.py:323] [p0115] step: count(1584), step_time 1416.71, mean_step_time 1440.25, it/s 0.69 | |
| [0130 18:08:24 @multigpu.py:323] [p0576] step: count(1609), step_time 1456.66, mean_step_time 1422.15, it/s 0.7 | |
| [0130 18:08:24 @multigpu.py:323] [p0574] step: count(1600), step_time 1426.08, mean_step_time 1426.94, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.63978229026 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.63978229026 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.63978229026 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.639782739414 , -0.00757606513798 ##### | |
| ##### Sending to neptune: policy_loss : 0.639782739414 , -0.300617456436 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.639782739414 , -2.28436470032 ##### | |
| ##### Sending to neptune: value_loss : 0.639782739414 , 1.61524569988 ##### | |
| ##### Sending to neptune: advantage : 0.639782739414 , 0.00124924443662 ##### | |
| ##### Sending to neptune: pred_reward : 0.639782739414 , 0.441886663437 ##### | |
| ##### Sending to neptune: max_logit : 0.639782739414 , 0.210331350565 ##### | |
| [u'loss', -0.0075760651379823685, -0.3006174564361572, -2.284364700317383, 1.6152456998825073, 0.001249244436621666, 0.44188666343688965, 0.21033135056495667] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.63978320082 , 9663250.99 ##### | |
| ##### Sending to neptune: dp_per_s : 0.63978320082 , 89.2543382974 ##### | |
| [u'other', 9663250.99, 89.25433829743697] | |
| receiving | |
| [0130 18:08:24 @multigpu.py:323] [p0115] step: count(1585), step_time 1464.29, mean_step_time 1438.21, it/s 0.7 | |
| [0130 18:08:25 @multigpu.py:323] [p0576] step: count(1610), step_time 1392.29, mean_step_time 1417.32, it/s 0.71 | |
| [0130 18:08:25 @multigpu.py:323] [p0574] step: count(1601), step_time 1506.01, mean_step_time 1428.88, it/s 0.7 | |
| [0130 18:08:25 @multigpu.py:323] [p0115] step: count(1586), step_time 1439.58, mean_step_time 1439.51, it/s 0.69 | |
| [0130 18:08:27 @multigpu.py:323] [p0576] step: count(1611), step_time 1439.59, mean_step_time 1419.59, it/s 0.7 | |
| [0130 18:08:27 @multigpu.py:323] [p0574] step: count(1602), step_time 1467.12, mean_step_time 1430.76, it/s 0.7 | |
| [0130 18:08:27 @multigpu.py:323] [p0115] step: count(1587), step_time 1485.33, mean_step_time 1441.92, it/s 0.69 | |
| [0130 18:08:28 @multigpu.py:323] [p0574] step: count(1603), step_time 1593.5, mean_step_time 1438.63, it/s 0.7 | |
| [0130 18:08:28 @multigpu.py:323] [p0115] step: count(1588), step_time 1445.44, mean_step_time 1438.66, it/s 0.7 | |
| [0130 18:08:28 @multigpu.py:323] [p0576] step: count(1612), step_time 1871.95, mean_step_time 1438.8, it/s 0.7 | |
| [0130 18:08:30 @multigpu.py:323] [p0576] step: count(1613), step_time 1405.8, mean_step_time 1439.47, it/s 0.69 | |
| [0130 18:08:30 @multigpu.py:323] [p0574] step: count(1604), step_time 1428.88, mean_step_time 1438.35, it/s 0.7 | |
| [0130 18:08:30 @multigpu.py:323] [p0115] step: count(1589), step_time 1458.02, mean_step_time 1437.19, it/s 0.7 | |
| [0130 18:08:31 @multigpu.py:323] [p0115] step: count(1590), step_time 1395.07, mean_step_time 1436.43, it/s 0.7 | |
| [0130 18:08:31 @multigpu.py:323] [p0576] step: count(1614), step_time 1448.84, mean_step_time 1441.75, it/s 0.69 | |
| [0130 18:08:31 @multigpu.py:323] [p0574] step: count(1605), step_time 1439.13, mean_step_time 1438.53, it/s 0.7 | |
| [0130 18:08:33 @multigpu.py:323] [p0576] step: count(1615), step_time 1392.18, mean_step_time 1439.1, it/s 0.69 | |
| [0130 18:08:33 @multigpu.py:323] [p0574] step: count(1606), step_time 1439.27, mean_step_time 1439.91, it/s 0.69 | |
| [0130 18:08:33 @multigpu.py:323] [p0115] step: count(1591), step_time 1477.38, mean_step_time 1439.32, it/s 0.69 | |
| [0130 18:08:34 @multigpu.py:323] [p0576] step: count(1616), step_time 1441.12, mean_step_time 1441.71, it/s 0.69 | |
| [0130 18:08:34 @multigpu.py:323] [p0115] step: count(1592), step_time 1406.52, mean_step_time 1435.72, it/s 0.7 | |
| [0130 18:08:34 @multigpu.py:323] [p0574] step: count(1607), step_time 1473.3, mean_step_time 1438.79, it/s 0.7 | |
| [0130 18:08:35 @multigpu.py:323] [p0576] step: count(1617), step_time 1411.52, mean_step_time 1441.49, it/s 0.69 | |
| [0130 18:08:36 @multigpu.py:323] [p0115] step: count(1593), step_time 1405.83, mean_step_time 1432.39, it/s 0.7 | |
| [0130 18:08:36 @multigpu.py:323] [p0574] step: count(1608), step_time 1442.58, mean_step_time 1441.52, it/s 0.69 | |
| [0130 18:08:37 @multigpu.py:323] [p0576] step: count(1618), step_time 1444.46, mean_step_time 1443.6, it/s 0.69 | |
| [0130 18:08:37 @multigpu.py:323] [p0115] step: count(1594), step_time 1419.51, mean_step_time 1430.59, it/s 0.7 | |
| [0130 18:08:37 @multigpu.py:323] [p0574] step: count(1609), step_time 1431.5, mean_step_time 1441.2, it/s 0.69 | |
| [0130 18:08:38 @multigpu.py:323] [p0576] step: count(1619), step_time 1389.02, mean_step_time 1440.55, it/s 0.69 | |
| [0130 18:08:38 @multigpu.py:323] [p0115] step: count(1595), step_time 1395.99, mean_step_time 1430.33, it/s 0.7 | |
| [0130 18:08:38 @multigpu.py:323] [p0574] step: count(1610), step_time 1367.22, mean_step_time 1437.62, it/s 0.7 | |
| [0130 18:08:40 @multigpu.py:323] [p0115] step: count(1596), step_time 1464.15, mean_step_time 1432.48, it/s 0.7 | |
| [0130 18:08:40 @multigpu.py:323] [p0576] step: count(1620), step_time 1519.98, mean_step_time 1446.33, it/s 0.69 | |
| [0130 18:08:40 @multigpu.py:323] [p0574] step: count(1611), step_time 1466.99, mean_step_time 1441.41, it/s 0.69 | |
| [0130 18:08:41 @multigpu.py:323] [p0115] step: count(1597), step_time 1443.96, mean_step_time 1435.08, it/s 0.7 | |
| [0130 18:08:41 @multigpu.py:323] [p0576] step: count(1621), step_time 1432.41, mean_step_time 1447.71, it/s 0.69 | |
| [0130 18:08:41 @multigpu.py:323] [p0574] step: count(1612), step_time 1401.12, mean_step_time 1438.97, it/s 0.69 | |
| [0130 18:08:43 @multigpu.py:323] [p0574] step: count(1613), step_time 1387.32, mean_step_time 1439.1, it/s 0.69 | |
| [0130 18:08:43 @multigpu.py:323] [p0576] step: count(1622), step_time 1411.45, mean_step_time 1446.99, it/s 0.69 | |
| [0130 18:08:43 @multigpu.py:323] [p0115] step: count(1598), step_time 1448.26, mean_step_time 1436.06, it/s 0.7 | |
| [0130 18:08:44 @multigpu.py:323] [p0576] step: count(1623), step_time 1371.52, mean_step_time 1444.83, it/s 0.69 | |
| [0130 18:08:44 @multigpu.py:323] [p0574] step: count(1614), step_time 1392.72, mean_step_time 1436.2, it/s 0.7 | |
| [0130 18:08:44 @multigpu.py:323] [p0115] step: count(1599), step_time 1430.72, mean_step_time 1438.71, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.645635401607 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 18:08:45 @multigpu.py:323] [p0574] step: count(1615), step_time 1387.46, mean_step_time 1436.94, it/s 0.7 | |
| [0130 18:08:46 @multigpu.py:323] [p0576] step: count(1624), step_time 1445.4, mean_step_time 1445.56, it/s 0.69 | |
| [0130 18:08:46 @multigpu.py:323] [p0115] step: count(1600), step_time 1383.25, mean_step_time 1437.31, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.645811170273 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.645811170273 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.645811170273 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.645811683271 , -0.00390666676685 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.645811683271 , 0.224651888013 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.645811683271 , -2.28408169746 ##### | |
| ##### Sending to neptune: value_loss : 0.645811683271 , 1.55937635899 ##### | |
| ##### Sending to neptune: advantage : 0.645811683271 , -0.00111011322588 ##### | |
| ##### Sending to neptune: pred_reward : 0.645811683271 , 0.442633777857 ##### | |
| ##### Sending to neptune: max_logit : 0.645811683271 , 0.208173751831 ##### | |
| [u'loss', -0.00390666676685214, 0.22465188801288605, -2.2840816974639893, 1.5593763589859009, -0.001110113225877285, 0.4426337778568268, 0.2081737518310547] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.645812171102 , 9614416.76 ##### | |
| ##### Sending to neptune: dp_per_s : 0.645812171102 , 89.1838954286 ##### | |
| [u'other', 9614416.76, 89.18389542861547] | |
| receiving | |
| [0130 18:08:47 @multigpu.py:323] [p0574] step: count(1616), step_time 1464.59, mean_step_time 1435.82, it/s 0.7 | |
| [0130 18:08:47 @multigpu.py:323] [p0576] step: count(1625), step_time 1436.61, mean_step_time 1448.45, it/s 0.69 | |
| [0130 18:08:47 @multigpu.py:323] [p0115] step: count(1601), step_time 1436.9, mean_step_time 1436.92, it/s 0.7 | |
| [0130 18:08:48 @multigpu.py:323] [p0574] step: count(1617), step_time 1395.09, mean_step_time 1436.24, it/s 0.7 | |
| [0130 18:08:48 @multigpu.py:323] [p0576] step: count(1626), step_time 1392.04, mean_step_time 1445.58, it/s 0.69 | |
| [0130 18:08:48 @multigpu.py:323] [p0115] step: count(1602), step_time 1412.95, mean_step_time 1433.6, it/s 0.7 | |
| [0130 18:08:50 @multigpu.py:323] [p0574] step: count(1618), step_time 1421.31, mean_step_time 1436.75, it/s 0.7 | |
| [0130 18:08:50 @multigpu.py:323] [p0576] step: count(1627), step_time 1424.94, mean_step_time 1446.88, it/s 0.69 | |
| [0130 18:08:50 @multigpu.py:323] [p0115] step: count(1603), step_time 1481.28, mean_step_time 1435.56, it/s 0.7 | |
| [0130 18:08:51 @multigpu.py:323] [p0574] step: count(1619), step_time 1422.31, mean_step_time 1437.67, it/s 0.7 | |
| [0130 18:08:51 @multigpu.py:323] [p0576] step: count(1628), step_time 1484.54, mean_step_time 1450.62, it/s 0.69 | |
| [0130 18:08:51 @multigpu.py:323] [p0115] step: count(1604), step_time 1440.25, mean_step_time 1436.73, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.647553682195 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:08:53 @multigpu.py:323] [p0574] step: count(1620), step_time 1429.35, mean_step_time 1437.84, it/s 0.7 | |
| [0130 18:08:53 @multigpu.py:323] [p0576] step: count(1629), step_time 1368.05, mean_step_time 1446.19, it/s 0.69 | |
| [0130 18:08:53 @multigpu.py:323] [p0115] step: count(1605), step_time 1469.65, mean_step_time 1437.0, it/s 0.7 | |
| [0130 18:08:54 @multigpu.py:323] [p0574] step: count(1621), step_time 1427.32, mean_step_time 1433.9, it/s 0.7 | |
| [0130 18:08:54 @multigpu.py:323] [p0576] step: count(1630), step_time 1407.52, mean_step_time 1446.95, it/s 0.69 | |
| [0130 18:08:54 @multigpu.py:323] [p0115] step: count(1606), step_time 1435.4, mean_step_time 1436.79, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.648334509134 , 2.7 ##### | |
| [u'online', 2.7] | |
| receiving | |
| [0130 18:08:55 @multigpu.py:323] [p0574] step: count(1622), step_time 1394.61, mean_step_time 1430.28, it/s 0.7 | |
| [0130 18:08:55 @multigpu.py:323] [p0576] step: count(1631), step_time 1451.09, mean_step_time 1447.52, it/s 0.69 | |
| [0130 18:08:56 @multigpu.py:323] [p0115] step: count(1607), step_time 1380.77, mean_step_time 1431.57, it/s 0.7 | |
| [0130 18:08:57 @multigpu.py:323] [p0115] step: count(1608), step_time 1406.7, mean_step_time 1429.63, it/s 0.7 | |
| [0130 18:08:57 @multigpu.py:323] [p0574] step: count(1623), step_time 1581.6, mean_step_time 1429.68, it/s 0.7 | |
| [0130 18:08:57 @multigpu.py:323] [p0576] step: count(1632), step_time 1514.3, mean_step_time 1429.64, it/s 0.7 | |
| [0130 18:08:58 @multigpu.py:323] [p0576] step: count(1633), step_time 1404.11, mean_step_time 1429.56, it/s 0.7 | |
| [0130 18:08:58 @multigpu.py:323] [p0115] step: count(1609), step_time 1422.43, mean_step_time 1427.85, it/s 0.7 | |
| [0130 18:08:58 @multigpu.py:323] [p0574] step: count(1624), step_time 1465.75, mean_step_time 1431.53, it/s 0.7 | |
| [0130 18:09:00 @multigpu.py:323] [p0115] step: count(1610), step_time 1382.41, mean_step_time 1427.22, it/s 0.7 | |
| [0130 18:09:00 @multigpu.py:323] [p0576] step: count(1634), step_time 1471.18, mean_step_time 1430.67, it/s 0.7 | |
| [0130 18:09:00 @multigpu.py:323] [p0574] step: count(1625), step_time 1425.24, mean_step_time 1430.83, it/s 0.7 | |
| [0130 18:09:01 @multigpu.py:323] [p0115] step: count(1611), step_time 1441.52, mean_step_time 1425.42, it/s 0.7 | |
| [0130 18:09:01 @multigpu.py:323] [p0574] step: count(1626), step_time 1396.18, mean_step_time 1428.68, it/s 0.7 | |
| [0130 18:09:01 @multigpu.py:323] [p0576] step: count(1635), step_time 1431.17, mean_step_time 1432.62, it/s 0.7 | |
| [0130 18:09:03 @multigpu.py:323] [p0574] step: count(1627), step_time 1411.9, mean_step_time 1425.61, it/s 0.7 | |
| [0130 18:09:03 @multigpu.py:323] [p0115] step: count(1612), step_time 1476.86, mean_step_time 1428.94, it/s 0.7 | |
| [0130 18:09:03 @multigpu.py:323] [p0576] step: count(1636), step_time 1424.65, mean_step_time 1431.8, it/s 0.7 | |
| [0130 18:09:04 @multigpu.py:323] [p0574] step: count(1628), step_time 1422.29, mean_step_time 1424.59, it/s 0.7 | |
| [0130 18:09:04 @multigpu.py:323] [p0576] step: count(1637), step_time 1418.3, mean_step_time 1432.14, it/s 0.7 | |
| [0130 18:09:04 @multigpu.py:323] [p0115] step: count(1613), step_time 1430.73, mean_step_time 1430.18, it/s 0.7 | |
| [0130 18:09:06 @multigpu.py:323] [p0574] step: count(1629), step_time 1453.18, mean_step_time 1425.68, it/s 0.7 | |
| [0130 18:09:06 @multigpu.py:323] [p0115] step: count(1614), step_time 1443.34, mean_step_time 1431.38, it/s 0.7 | |
| [0130 18:09:06 @multigpu.py:323] [p0576] step: count(1638), step_time 1464.71, mean_step_time 1433.15, it/s 0.7 | |
| [0130 18:09:07 @multigpu.py:323] [p0574] step: count(1630), step_time 1388.58, mean_step_time 1426.75, it/s 0.7 | |
| [0130 18:09:07 @multigpu.py:323] [p0576] step: count(1639), step_time 1360.35, mean_step_time 1431.72, it/s 0.7 | |
| [0130 18:09:07 @multigpu.py:323] [p0115] step: count(1615), step_time 1453.92, mean_step_time 1434.27, it/s 0.7 | |
| [0130 18:09:08 @multigpu.py:323] [p0574] step: count(1631), step_time 1412.25, mean_step_time 1424.01, it/s 0.7 | |
| [0130 18:09:08 @multigpu.py:323] [p0576] step: count(1640), step_time 1414.76, mean_step_time 1426.45, it/s 0.7 | |
| [0130 18:09:08 @multigpu.py:323] [p0115] step: count(1616), step_time 1444.89, mean_step_time 1433.31, it/s 0.7 | |
| [0130 18:09:10 @multigpu.py:323] [p0574] step: count(1632), step_time 1370.91, mean_step_time 1422.5, it/s 0.7 | |
| [0130 18:09:10 @multigpu.py:323] [p0576] step: count(1641), step_time 1455.55, mean_step_time 1427.61, it/s 0.7 | |
| [0130 18:09:10 @multigpu.py:323] [p0115] step: count(1617), step_time 1424.07, mean_step_time 1432.32, it/s 0.7 | |
| [0130 18:09:11 @multigpu.py:323] [p0574] step: count(1633), step_time 1451.77, mean_step_time 1425.72, it/s 0.7 | |
| [0130 18:09:11 @multigpu.py:323] [p0576] step: count(1642), step_time 1383.04, mean_step_time 1426.19, it/s 0.7 | |
| [0130 18:09:11 @multigpu.py:323] [p0115] step: count(1618), step_time 1362.2, mean_step_time 1428.01, it/s 0.7 | |
| [0130 18:09:13 @multigpu.py:323] [p0574] step: count(1634), step_time 1398.64, mean_step_time 1426.02, it/s 0.7 | |
| [0130 18:09:13 @multigpu.py:323] [p0576] step: count(1643), step_time 1435.56, mean_step_time 1429.39, it/s 0.7 | |
| [0130 18:09:13 @multigpu.py:323] [p0115] step: count(1619), step_time 1441.69, mean_step_time 1428.56, it/s 0.7 | |
| [0130 18:09:14 @multigpu.py:323] [p0574] step: count(1635), step_time 1430.21, mean_step_time 1428.15, it/s 0.7 | |
| [0130 18:09:14 @multigpu.py:323] [p0115] step: count(1620), step_time 1395.04, mean_step_time 1429.15, it/s 0.7 | |
| [0130 18:09:14 @multigpu.py:323] [p0576] step: count(1644), step_time 1462.5, mean_step_time 1430.25, it/s 0.7 | |
| [0130 18:09:15 @multigpu.py:323] [p0574] step: count(1636), step_time 1445.55, mean_step_time 1427.2, it/s 0.7 | |
| [0130 18:09:16 @multigpu.py:323] [p0115] step: count(1621), step_time 1430.3, mean_step_time 1428.82, it/s 0.7 | |
| [0130 18:09:16 @multigpu.py:323] [p0576] step: count(1645), step_time 1464.91, mean_step_time 1431.66, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.654332980514 , 1.8 ##### | |
| [u'online', 1.8] | |
| ##### Sending to neptune: active_workers : 0.654333023826 , 3 ##### | |
| receiving | |
| [0130 18:09:17 @multigpu.py:323] [p0574] step: count(1637), step_time 1399.36, mean_step_time 1427.42, it/s 0.7 | |
| [0130 18:09:17 @multigpu.py:323] [p0115] step: count(1622), step_time 1442.2, mean_step_time 1430.28, it/s 0.7 | |
| [0130 18:09:17 @multigpu.py:323] [p0576] step: count(1646), step_time 1404.78, mean_step_time 1432.3, it/s 0.7 | |
| [0130 18:09:18 @multigpu.py:323] [p0574] step: count(1638), step_time 1451.67, mean_step_time 1428.93, it/s 0.7 | |
| [0130 18:09:18 @multigpu.py:323] [p0115] step: count(1623), step_time 1412.34, mean_step_time 1426.84, it/s 0.7 | |
| [0130 18:09:18 @multigpu.py:323] [p0576] step: count(1647), step_time 1444.22, mean_step_time 1433.26, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.655126712455 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:09:20 @multigpu.py:323] [p0574] step: count(1639), step_time 1438.08, mean_step_time 1429.72, it/s 0.7 | |
| [0130 18:09:20 @multigpu.py:323] [p0115] step: count(1624), step_time 1449.13, mean_step_time 1427.28, it/s 0.7 | |
| [0130 18:09:20 @multigpu.py:323] [p0576] step: count(1648), step_time 1473.0, mean_step_time 1432.69, it/s 0.7 | |
| [0130 18:09:21 @multigpu.py:323] [p0574] step: count(1640), step_time 1402.93, mean_step_time 1428.4, it/s 0.7 | |
| [0130 18:09:21 @multigpu.py:323] [p0115] step: count(1625), step_time 1437.78, mean_step_time 1425.69, it/s 0.7 | |
| [0130 18:09:21 @multigpu.py:323] [p0576] step: count(1649), step_time 1437.62, mean_step_time 1436.17, it/s 0.7 | |
| [0130 18:09:23 @multigpu.py:323] [p0574] step: count(1641), step_time 1419.89, mean_step_time 1428.03, it/s 0.7 | |
| [0130 18:09:23 @multigpu.py:323] [p0115] step: count(1626), step_time 1458.74, mean_step_time 1426.85, it/s 0.7 | |
| [0130 18:09:23 @multigpu.py:323] [p0576] step: count(1650), step_time 1415.75, mean_step_time 1436.58, it/s 0.7 | |
| [0130 18:09:24 @multigpu.py:323] [p0574] step: count(1642), step_time 1456.53, mean_step_time 1431.13, it/s 0.7 | |
| [0130 18:09:24 @multigpu.py:323] [p0576] step: count(1651), step_time 1417.01, mean_step_time 1434.87, it/s 0.7 | |
| [0130 18:09:24 @multigpu.py:323] [p0115] step: count(1627), step_time 1436.49, mean_step_time 1429.64, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.656686421898 , 2.6 ##### | |
| [u'online', 2.6] | |
| receiving | |
| [0130 18:09:26 @multigpu.py:323] [p0115] step: count(1628), step_time 1491.6, mean_step_time 1433.88, it/s 0.7 | |
| [0130 18:09:26 @multigpu.py:323] [p0576] step: count(1652), step_time 1495.11, mean_step_time 1433.91, it/s 0.7 | |
| [0130 18:09:26 @multigpu.py:323] [p0574] step: count(1643), step_time 1638.78, mean_step_time 1433.98, it/s 0.7 | |
| [0130 18:09:27 @multigpu.py:323] [p0576] step: count(1653), step_time 1416.91, mean_step_time 1434.55, it/s 0.7 | |
| [0130 18:09:27 @multigpu.py:323] [p0115] step: count(1629), step_time 1430.94, mean_step_time 1434.31, it/s 0.7 | |
| [0130 18:09:27 @multigpu.py:323] [p0574] step: count(1644), step_time 1440.25, mean_step_time 1432.71, it/s 0.7 | |
| [0130 18:09:29 @multigpu.py:323] [p0574] step: count(1645), step_time 1410.67, mean_step_time 1431.98, it/s 0.7 | |
| [0130 18:09:29 @multigpu.py:323] [p0115] step: count(1630), step_time 1455.43, mean_step_time 1437.96, it/s 0.7 | |
| [0130 18:09:29 @multigpu.py:323] [p0576] step: count(1654), step_time 1484.02, mean_step_time 1435.19, it/s 0.7 | |
| [0130 18:09:30 @multigpu.py:323] [p0574] step: count(1646), step_time 1394.32, mean_step_time 1431.89, it/s 0.7 | |
| [0130 18:09:30 @multigpu.py:323] [p0115] step: count(1631), step_time 1378.02, mean_step_time 1434.79, it/s 0.7 | |
| [0130 18:09:30 @multigpu.py:323] [p0576] step: count(1655), step_time 1393.65, mean_step_time 1433.32, it/s 0.7 | |
| [0130 18:09:31 @multigpu.py:323] [p0574] step: count(1647), step_time 1369.83, mean_step_time 1429.78, it/s 0.7 | |
| [0130 18:09:31 @multigpu.py:323] [p0115] step: count(1632), step_time 1396.42, mean_step_time 1430.76, it/s 0.7 | |
| [0130 18:09:31 @multigpu.py:323] [p0576] step: count(1656), step_time 1439.94, mean_step_time 1434.08, it/s 0.7 | |
| [0130 18:09:33 @multigpu.py:323] [p0574] step: count(1648), step_time 1416.27, mean_step_time 1429.48, it/s 0.7 | |
| [0130 18:09:33 @multigpu.py:323] [p0115] step: count(1633), step_time 1430.82, mean_step_time 1430.77, it/s 0.7 | |
| [0130 18:09:33 @multigpu.py:323] [p0576] step: count(1657), step_time 1388.18, mean_step_time 1432.58, it/s 0.7 | |
| [0130 18:09:34 @multigpu.py:323] [p0574] step: count(1649), step_time 1413.59, mean_step_time 1427.5, it/s 0.7 | |
| [0130 18:09:34 @multigpu.py:323] [p0115] step: count(1634), step_time 1411.99, mean_step_time 1429.2, it/s 0.7 | |
| [0130 18:09:34 @multigpu.py:323] [p0576] step: count(1658), step_time 1393.79, mean_step_time 1429.03, it/s 0.7 | |
| [0130 18:09:36 @multigpu.py:323] [p0115] step: count(1635), step_time 1390.83, mean_step_time 1426.05, it/s 0.7 | |
| [0130 18:09:36 @multigpu.py:323] [p0574] step: count(1650), step_time 1473.69, mean_step_time 1431.76, it/s 0.7 | |
| [0130 18:09:36 @multigpu.py:323] [p0576] step: count(1659), step_time 1429.32, mean_step_time 1432.48, it/s 0.7 | |
| [0130 18:09:37 @multigpu.py:323] [p0115] step: count(1636), step_time 1408.14, mean_step_time 1424.21, it/s 0.7 | |
| [0130 18:09:37 @multigpu.py:323] [p0574] step: count(1651), step_time 1388.65, mean_step_time 1430.58, it/s 0.7 | |
| [0130 18:09:37 @multigpu.py:323] [p0576] step: count(1660), step_time 1427.87, mean_step_time 1433.13, it/s 0.7 | |
| [0130 18:09:38 @multigpu.py:323] [p0115] step: count(1637), step_time 1398.41, mean_step_time 1422.93, it/s 0.7 | |
| [0130 18:09:38 @multigpu.py:323] [p0574] step: count(1652), step_time 1414.64, mean_step_time 1432.77, it/s 0.7 | |
| [0130 18:09:39 @multigpu.py:323] [p0576] step: count(1661), step_time 1487.97, mean_step_time 1434.76, it/s 0.7 | |
| [0130 18:09:40 @multigpu.py:323] [p0574] step: count(1653), step_time 1415.11, mean_step_time 1430.93, it/s 0.7 | |
| [0130 18:09:40 @multigpu.py:323] [p0115] step: count(1638), step_time 1471.59, mean_step_time 1428.4, it/s 0.7 | |
| [0130 18:09:40 @multigpu.py:323] [p0576] step: count(1662), step_time 1396.64, mean_step_time 1435.44, it/s 0.7 | |
| [0130 18:09:41 @multigpu.py:323] [p0574] step: count(1654), step_time 1451.77, mean_step_time 1433.59, it/s 0.7 | |
| [0130 18:09:41 @multigpu.py:323] [p0115] step: count(1639), step_time 1495.2, mean_step_time 1431.07, it/s 0.7 | |
| [0130 18:09:41 @multigpu.py:323] [p0576] step: count(1663), step_time 1411.52, mean_step_time 1434.23, it/s 0.7 | |
| [0130 18:09:43 @multigpu.py:323] [p0574] step: count(1655), step_time 1387.5, mean_step_time 1431.45, it/s 0.7 | |
| [0130 18:09:43 @multigpu.py:323] [p0115] step: count(1640), step_time 1416.6, mean_step_time 1432.15, it/s 0.7 | |
| [0130 18:09:43 @multigpu.py:323] [p0576] step: count(1664), step_time 1421.09, mean_step_time 1432.16, it/s 0.7 | |
| [0130 18:09:44 @multigpu.py:323] [p0574] step: count(1656), step_time 1435.78, mean_step_time 1430.97, it/s 0.7 | |
| [0130 18:09:44 @multigpu.py:323] [p0576] step: count(1665), step_time 1384.19, mean_step_time 1428.13, it/s 0.7 | |
| [0130 18:09:44 @multigpu.py:323] [p0115] step: count(1641), step_time 1435.24, mean_step_time 1432.4, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.662334823608 , 1.6 ##### | |
| [u'online', 1.6] | |
| receiving | |
| [0130 18:09:46 @multigpu.py:323] [p0574] step: count(1657), step_time 1428.13, mean_step_time 1432.4, it/s 0.7 | |
| [0130 18:09:46 @multigpu.py:323] [p0576] step: count(1666), step_time 1400.26, mean_step_time 1427.9, it/s 0.7 | |
| [0130 18:09:46 @multigpu.py:323] [p0115] step: count(1642), step_time 1459.13, mean_step_time 1433.24, it/s 0.7 | |
| [0130 18:09:47 @multigpu.py:323] [p0574] step: count(1658), step_time 1380.97, mean_step_time 1428.87, it/s 0.7 | |
| [0130 18:09:47 @multigpu.py:323] [p0576] step: count(1667), step_time 1439.95, mean_step_time 1427.69, it/s 0.7 | |
| [0130 18:09:47 @multigpu.py:323] [p0115] step: count(1643), step_time 1429.71, mean_step_time 1434.11, it/s 0.7 | |
| [0130 18:09:48 @multigpu.py:323] [p0574] step: count(1659), step_time 1468.11, mean_step_time 1430.37, it/s 0.7 | |
| [0130 18:09:48 @multigpu.py:323] [p0576] step: count(1668), step_time 1404.51, mean_step_time 1424.26, it/s 0.7 | |
| [0130 18:09:49 @multigpu.py:323] [p0115] step: count(1644), step_time 1420.53, mean_step_time 1432.68, it/s 0.7 | |
| [0130 18:09:50 @multigpu.py:323] [p0574] step: count(1660), step_time 1433.18, mean_step_time 1431.88, it/s 0.7 | |
| [0130 18:09:50 @multigpu.py:323] [p0576] step: count(1669), step_time 1487.62, mean_step_time 1426.76, it/s 0.7 | |
| [0130 18:09:50 @multigpu.py:323] [p0115] step: count(1645), step_time 1414.96, mean_step_time 1431.54, it/s 0.7 | |
| [0130 18:09:51 @multigpu.py:323] [p0574] step: count(1661), step_time 1415.25, mean_step_time 1431.65, it/s 0.7 | |
| [0130 18:09:51 @multigpu.py:323] [p0576] step: count(1670), step_time 1413.34, mean_step_time 1426.64, it/s 0.7 | |
| [0130 18:09:51 @multigpu.py:323] [p0115] step: count(1646), step_time 1505.24, mean_step_time 1433.86, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.664274540544 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 18:09:53 @multigpu.py:323] [p0574] step: count(1662), step_time 1418.69, mean_step_time 1429.76, it/s 0.7 | |
| [0130 18:09:53 @multigpu.py:323] [p0576] step: count(1671), step_time 1434.48, mean_step_time 1427.52, it/s 0.7 | |
| [0130 18:09:53 @multigpu.py:323] [p0115] step: count(1647), step_time 1434.55, mean_step_time 1433.77, it/s 0.7 | |
| [0130 18:09:54 @multigpu.py:323] [p0115] step: count(1648), step_time 1440.15, mean_step_time 1431.19, it/s 0.7 | |
| [0130 18:09:54 @multigpu.py:323] [p0576] step: count(1672), step_time 1567.38, mean_step_time 1431.13, it/s 0.7 | |
| [0130 18:09:54 @multigpu.py:323] [p0574] step: count(1663), step_time 1667.38, mean_step_time 1431.19, it/s 0.7 | |
| [0130 18:09:56 @multigpu.py:323] [p0574] step: count(1664), step_time 1393.03, mean_step_time 1428.83, it/s 0.7 | |
| [0130 18:09:56 @multigpu.py:323] [p0115] step: count(1649), step_time 1403.24, mean_step_time 1429.81, it/s 0.7 | |
| [0130 18:09:56 @multigpu.py:323] [p0576] step: count(1673), step_time 1419.22, mean_step_time 1431.25, it/s 0.7 | |
| [0130 18:09:57 @multigpu.py:323] [p0576] step: count(1674), step_time 1414.23, mean_step_time 1427.76, it/s 0.7 | |
| [0130 18:09:57 @multigpu.py:323] [p0574] step: count(1665), step_time 1456.1, mean_step_time 1431.1, it/s 0.7 | |
| [0130 18:09:57 @multigpu.py:323] [p0115] step: count(1650), step_time 1469.75, mean_step_time 1430.53, it/s 0.7 | |
| [0130 18:09:59 @multigpu.py:323] [p0576] step: count(1675), step_time 1419.68, mean_step_time 1429.06, it/s 0.7 | |
| [0130 18:09:59 @multigpu.py:323] [p0574] step: count(1666), step_time 1481.67, mean_step_time 1435.47, it/s 0.7 | |
| [0130 18:09:59 @multigpu.py:323] [p0115] step: count(1651), step_time 1466.01, mean_step_time 1434.93, it/s 0.7 | |
| [0130 18:10:00 @multigpu.py:323] [p0576] step: count(1676), step_time 1447.57, mean_step_time 1429.44, it/s 0.7 | |
| [0130 18:10:00 @multigpu.py:323] [p0115] step: count(1652), step_time 1403.98, mean_step_time 1435.3, it/s 0.7 | |
| [0130 18:10:00 @multigpu.py:323] [p0574] step: count(1667), step_time 1482.03, mean_step_time 1441.08, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.666537099679 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 18:10:01 @multigpu.py:323] [p0576] step: count(1677), step_time 1403.16, mean_step_time 1430.19, it/s 0.7 | |
| [0130 18:10:02 @multigpu.py:323] [p0574] step: count(1668), step_time 1409.66, mean_step_time 1440.75, it/s 0.69 | |
| [0130 18:10:02 @multigpu.py:323] [p0115] step: count(1653), step_time 1492.51, mean_step_time 1438.39, it/s 0.7 | |
| [0130 18:10:03 @multigpu.py:323] [p0576] step: count(1678), step_time 1381.72, mean_step_time 1429.59, it/s 0.7 | |
| [0130 18:10:03 @multigpu.py:323] [p0115] step: count(1654), step_time 1420.5, mean_step_time 1438.81, it/s 0.7 | |
| [0130 18:10:03 @multigpu.py:323] [p0574] step: count(1669), step_time 1436.12, mean_step_time 1441.87, it/s 0.69 | |
| [0130 18:10:04 @multigpu.py:323] [p0576] step: count(1679), step_time 1393.89, mean_step_time 1427.82, it/s 0.7 | |
| [0130 18:10:04 @multigpu.py:323] [p0574] step: count(1670), step_time 1387.79, mean_step_time 1437.58, it/s 0.7 | |
| [0130 18:10:04 @multigpu.py:323] [p0115] step: count(1655), step_time 1415.76, mean_step_time 1440.06, it/s 0.69 | |
| [0130 18:10:06 @multigpu.py:323] [p0576] step: count(1680), step_time 1497.64, mean_step_time 1431.3, it/s 0.7 | |
| [0130 18:10:06 @multigpu.py:323] [p0115] step: count(1656), step_time 1409.1, mean_step_time 1440.11, it/s 0.69 | |
| [0130 18:10:06 @multigpu.py:323] [p0574] step: count(1671), step_time 1507.97, mean_step_time 1443.54, it/s 0.69 | |
| [0130 18:10:07 @multigpu.py:323] [p0576] step: count(1681), step_time 1410.04, mean_step_time 1427.41, it/s 0.7 | |
| [0130 18:10:07 @multigpu.py:323] [p0574] step: count(1672), step_time 1387.44, mean_step_time 1442.18, it/s 0.69 | |
| [0130 18:10:07 @multigpu.py:323] [p0115] step: count(1657), step_time 1492.14, mean_step_time 1444.79, it/s 0.69 | |
| [0130 18:10:09 @multigpu.py:323] [p0576] step: count(1682), step_time 1427.56, mean_step_time 1428.95, it/s 0.7 | |
| [0130 18:10:09 @multigpu.py:323] [p0574] step: count(1673), step_time 1457.14, mean_step_time 1444.29, it/s 0.69 | |
| [0130 18:10:09 @multigpu.py:323] [p0115] step: count(1658), step_time 1438.8, mean_step_time 1443.15, it/s 0.69 | |
| [0130 18:10:10 @multigpu.py:323] [p0576] step: count(1683), step_time 1418.8, mean_step_time 1429.32, it/s 0.7 | |
| [0130 18:10:10 @multigpu.py:323] [p0574] step: count(1674), step_time 1430.46, mean_step_time 1443.22, it/s 0.69 | |
| [0130 18:10:10 @multigpu.py:323] [p0115] step: count(1659), step_time 1424.56, mean_step_time 1439.62, it/s 0.69 | |
| [0130 18:10:11 @multigpu.py:323] [p0576] step: count(1684), step_time 1437.82, mean_step_time 1430.15, it/s 0.7 | |
| [0130 18:10:12 @multigpu.py:323] [p0574] step: count(1675), step_time 1401.19, mean_step_time 1443.9, it/s 0.69 | |
| [0130 18:10:12 @multigpu.py:323] [p0115] step: count(1660), step_time 1479.0, mean_step_time 1442.74, it/s 0.69 | |
| [0130 18:10:13 @multigpu.py:323] [p0576] step: count(1685), step_time 1441.35, mean_step_time 1433.01, it/s 0.7 | |
| [0130 18:10:13 @multigpu.py:323] [p0115] step: count(1661), step_time 1384.82, mean_step_time 1440.22, it/s 0.69 | |
| [0130 18:10:13 @multigpu.py:323] [p0574] step: count(1676), step_time 1493.03, mean_step_time 1446.77, it/s 0.69 | |
| [0130 18:10:14 @multigpu.py:323] [p0576] step: count(1686), step_time 1428.42, mean_step_time 1434.42, it/s 0.7 | |
| [0130 18:10:14 @multigpu.py:323] [p0574] step: count(1677), step_time 1398.29, mean_step_time 1445.28, it/s 0.69 | |
| [0130 18:10:14 @multigpu.py:323] [p0115] step: count(1662), step_time 1425.75, mean_step_time 1438.55, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.670702761345 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 18:10:16 @multigpu.py:323] [p0576] step: count(1687), step_time 1457.19, mean_step_time 1435.28, it/s 0.7 | |
| [0130 18:10:16 @multigpu.py:323] [p0574] step: count(1678), step_time 1404.97, mean_step_time 1446.48, it/s 0.69 | |
| [0130 18:10:16 @multigpu.py:323] [p0115] step: count(1663), step_time 1432.49, mean_step_time 1438.69, it/s 0.7 | |
| [0130 18:10:17 @multigpu.py:323] [p0576] step: count(1688), step_time 1400.57, mean_step_time 1435.08, it/s 0.7 | |
| [0130 18:10:17 @multigpu.py:323] [p0574] step: count(1679), step_time 1386.12, mean_step_time 1442.38, it/s 0.69 | |
| [0130 18:10:17 @multigpu.py:323] [p0115] step: count(1664), step_time 1449.27, mean_step_time 1440.13, it/s 0.69 | |
| [0130 18:10:19 @multigpu.py:323] [p0576] step: count(1689), step_time 1439.25, mean_step_time 1432.67, it/s 0.7 | |
| [0130 18:10:19 @multigpu.py:323] [p0574] step: count(1680), step_time 1434.19, mean_step_time 1442.43, it/s 0.69 | |
| [0130 18:10:19 @multigpu.py:323] [p0115] step: count(1665), step_time 1411.0, mean_step_time 1439.93, it/s 0.69 | |
| [0130 18:10:20 @multigpu.py:323] [p0576] step: count(1690), step_time 1420.97, mean_step_time 1433.05, it/s 0.7 | |
| [0130 18:10:20 @multigpu.py:323] [p0574] step: count(1681), step_time 1493.81, mean_step_time 1446.35, it/s 0.69 | |
| [0130 18:10:20 @multigpu.py:323] [p0115] step: count(1666), step_time 1463.45, mean_step_time 1437.84, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.672266149124 , 2.1 ##### | |
| [u'online', 2.1] | |
| ##### Sending to neptune: active_workers : 0.672266205814 , 3 ##### | |
| receiving | |
| [0130 18:10:21 @multigpu.py:323] [p0576] step: count(1691), step_time 1387.7, mean_step_time 1430.71, it/s 0.7 | |
| [0130 18:10:22 @multigpu.py:323] [p0574] step: count(1682), step_time 1396.69, mean_step_time 1445.25, it/s 0.69 | |
| [0130 18:10:22 @multigpu.py:323] [p0115] step: count(1667), step_time 1443.52, mean_step_time 1438.29, it/s 0.7 | |
| [0130 18:10:23 @multigpu.py:323] [p0115] step: count(1668), step_time 1490.48, mean_step_time 1440.81, it/s 0.69 | |
| [0130 18:10:23 @multigpu.py:323] [p0574] step: count(1683), step_time 1577.51, mean_step_time 1440.76, it/s 0.69 | |
| [0130 18:10:23 @multigpu.py:323] [p0576] step: count(1692), step_time 1768.0, mean_step_time 1440.74, it/s 0.69 | |
| [0130 18:10:25 @multigpu.py:323] [p0115] step: count(1669), step_time 1393.01, mean_step_time 1440.29, it/s 0.69 | |
| [0130 18:10:25 @multigpu.py:323] [p0574] step: count(1684), step_time 1442.02, mean_step_time 1443.21, it/s 0.69 | |
| [0130 18:10:25 @multigpu.py:323] [p0576] step: count(1693), step_time 1461.64, mean_step_time 1442.86, it/s 0.69 | |
| [0130 18:10:26 @multigpu.py:323] [p0115] step: count(1670), step_time 1395.54, mean_step_time 1436.58, it/s 0.7 | |
| [0130 18:10:26 @multigpu.py:323] [p0576] step: count(1694), step_time 1374.54, mean_step_time 1440.88, it/s 0.69 | |
| [0130 18:10:26 @multigpu.py:323] [p0574] step: count(1685), step_time 1430.36, mean_step_time 1441.92, it/s 0.69 | |
| [0130 18:10:27 @multigpu.py:323] [p0115] step: count(1671), step_time 1398.47, mean_step_time 1433.21, it/s 0.7 | |
| [0130 18:10:27 @multigpu.py:323] [p0576] step: count(1695), step_time 1413.75, mean_step_time 1440.58, it/s 0.69 | |
| [0130 18:10:27 @multigpu.py:323] [p0574] step: count(1686), step_time 1410.91, mean_step_time 1438.38, it/s 0.7 | |
| [0130 18:10:29 @multigpu.py:323] [p0576] step: count(1696), step_time 1390.11, mean_step_time 1437.71, it/s 0.7 | |
| [0130 18:10:29 @multigpu.py:323] [p0115] step: count(1672), step_time 1461.48, mean_step_time 1436.08, it/s 0.7 | |
| [0130 18:10:29 @multigpu.py:323] [p0574] step: count(1687), step_time 1448.89, mean_step_time 1436.73, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.674562685821 , 1.8 ##### | |
| [u'online', 1.8] | |
| receiving | |
| [0130 18:10:30 @multigpu.py:323] [p0576] step: count(1697), step_time 1453.92, mean_step_time 1440.24, it/s 0.69 | |
| [0130 18:10:30 @multigpu.py:323] [p0115] step: count(1673), step_time 1464.19, mean_step_time 1434.67, it/s 0.7 | |
| [0130 18:10:30 @multigpu.py:323] [p0574] step: count(1688), step_time 1398.54, mean_step_time 1436.17, it/s 0.7 | |
| [0130 18:10:32 @multigpu.py:323] [p0574] step: count(1689), step_time 1437.25, mean_step_time 1436.23, it/s 0.7 | |
| [0130 18:10:32 @multigpu.py:323] [p0115] step: count(1674), step_time 1462.22, mean_step_time 1436.75, it/s 0.7 | |
| [0130 18:10:32 @multigpu.py:323] [p0576] step: count(1698), step_time 1489.01, mean_step_time 1445.61, it/s 0.69 | |
| [0130 18:10:33 @multigpu.py:323] [p0115] step: count(1675), step_time 1410.27, mean_step_time 1436.48, it/s 0.7 | |
| [0130 18:10:33 @multigpu.py:323] [p0576] step: count(1699), step_time 1408.13, mean_step_time 1446.32, it/s 0.69 | |
| [0130 18:10:33 @multigpu.py:323] [p0574] step: count(1690), step_time 1451.62, mean_step_time 1439.42, it/s 0.69 | |
| [0130 18:10:35 @multigpu.py:323] [p0576] step: count(1700), step_time 1407.8, mean_step_time 1441.83, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.676095512443 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.676095512443 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.676095512443 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.676096027493 , -0.0114078214392 ##### | |
| ##### Sending to neptune: policy_loss : 0.676096027493 , -0.820938706398 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.676096027493 , -2.28196120262 ##### | |
| ##### Sending to neptune: value_loss : 0.676096027493 , 1.64269912243 ##### | |
| ##### Sending to neptune: advantage : 0.676096027493 , 0.00343541777693 ##### | |
| ##### Sending to neptune: pred_reward : 0.676096027493 , 0.465741872787 ##### | |
| ##### Sending to neptune: max_logit : 0.676096027493 , 0.206445157528 ##### | |
| [u'loss', -0.0114078214392066, -0.8209387063980103, -2.28196120262146, 1.642699122428894, 0.003435417776927352, 0.4657418727874756, 0.20644515752792358] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.676096511616 , 9639749.34 ##### | |
| ##### Sending to neptune: dp_per_s : 0.676096511616 , 89.2744267437 ##### | |
| [u'other', 9639749.34, 89.27442674373448] | |
| receiving | |
| [0130 18:10:35 @multigpu.py:323] [p0574] step: count(1691), step_time 1400.0, mean_step_time 1434.02, it/s 0.7 | |
| [0130 18:10:35 @multigpu.py:323] [p0115] step: count(1676), step_time 1481.7, mean_step_time 1440.11, it/s 0.69 | |
| [0130 18:10:36 @multigpu.py:323] [p0576] step: count(1701), step_time 1441.7, mean_step_time 1443.41, it/s 0.69 | |
| [0130 18:10:36 @multigpu.py:323] [p0574] step: count(1692), step_time 1439.86, mean_step_time 1436.64, it/s 0.7 | |
| [0130 18:10:36 @multigpu.py:323] [p0115] step: count(1677), step_time 1406.73, mean_step_time 1435.84, it/s 0.7 | |
| [0130 18:10:37 @multigpu.py:323] [p0576] step: count(1702), step_time 1396.92, mean_step_time 1441.88, it/s 0.69 | |
| [0130 18:10:37 @multigpu.py:323] [p0574] step: count(1693), step_time 1393.53, mean_step_time 1433.46, it/s 0.7 | |
| [0130 18:10:37 @multigpu.py:323] [p0115] step: count(1678), step_time 1422.24, mean_step_time 1435.01, it/s 0.7 | |
| [0130 18:10:39 @multigpu.py:323] [p0576] step: count(1703), step_time 1442.18, mean_step_time 1443.05, it/s 0.69 | |
| [0130 18:10:39 @multigpu.py:323] [p0115] step: count(1679), step_time 1388.7, mean_step_time 1433.22, it/s 0.7 | |
| [0130 18:10:39 @multigpu.py:323] [p0574] step: count(1694), step_time 1478.99, mean_step_time 1435.89, it/s 0.7 | |
| [0130 18:10:40 @multigpu.py:323] [p0115] step: count(1680), step_time 1404.89, mean_step_time 1429.51, it/s 0.7 | |
| [0130 18:10:40 @multigpu.py:323] [p0574] step: count(1695), step_time 1387.51, mean_step_time 1435.2, it/s 0.7 | |
| [0130 18:10:40 @multigpu.py:323] [p0576] step: count(1704), step_time 1446.42, mean_step_time 1443.48, it/s 0.69 | |
| [0130 18:10:42 @multigpu.py:323] [p0115] step: count(1681), step_time 1401.27, mean_step_time 1430.33, it/s 0.7 | |
| [0130 18:10:42 @multigpu.py:323] [p0574] step: count(1696), step_time 1426.99, mean_step_time 1431.9, it/s 0.7 | |
| [0130 18:10:42 @multigpu.py:323] [p0576] step: count(1705), step_time 1466.69, mean_step_time 1444.75, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.678271423843 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 18:10:43 @multigpu.py:323] [p0115] step: count(1682), step_time 1426.85, mean_step_time 1430.39, it/s 0.7 | |
| [0130 18:10:43 @multigpu.py:323] [p0574] step: count(1697), step_time 1398.01, mean_step_time 1431.89, it/s 0.7 | |
| [0130 18:10:43 @multigpu.py:323] [p0576] step: count(1706), step_time 1392.96, mean_step_time 1442.97, it/s 0.69 | |
| [0130 18:10:44 @multigpu.py:323] [p0115] step: count(1683), step_time 1428.49, mean_step_time 1430.19, it/s 0.7 | |
| [0130 18:10:45 @multigpu.py:323] [p0576] step: count(1707), step_time 1430.49, mean_step_time 1441.64, it/s 0.69 | |
| [0130 18:10:45 @multigpu.py:323] [p0574] step: count(1698), step_time 1502.63, mean_step_time 1436.77, it/s 0.7 | |
| [0130 18:10:46 @multigpu.py:323] [p0115] step: count(1684), step_time 1463.85, mean_step_time 1430.92, it/s 0.7 | |
| [0130 18:10:46 @multigpu.py:323] [p0574] step: count(1699), step_time 1391.85, mean_step_time 1437.06, it/s 0.7 | |
| [0130 18:10:46 @multigpu.py:323] [p0576] step: count(1708), step_time 1436.71, mean_step_time 1443.44, it/s 0.69 | |
| [0130 18:10:47 @multigpu.py:323] [p0115] step: count(1685), step_time 1387.82, mean_step_time 1429.76, it/s 0.7 | |
| [0130 18:10:47 @multigpu.py:323] [p0574] step: count(1700), step_time 1430.96, mean_step_time 1436.9, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.679671151373 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.679671151373 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.679671151373 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.679671659668 , 0.00126111682039 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.679671659668 , 0.562968075275 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.679671659668 , -2.2817337513 ##### | |
| ##### Sending to neptune: value_loss : 0.679671659668 , 1.88018858433 ##### | |
| ##### Sending to neptune: advantage : 0.679671659668 , -0.0025423753541 ##### | |
| ##### Sending to neptune: pred_reward : 0.679671659668 , 0.480535209179 ##### | |
| ##### Sending to neptune: max_logit : 0.679671659668 , 0.207985624671 ##### | |
| [u'loss', 0.0012611168203875422, 0.5629680752754211, -2.281733751296997, 1.8801885843276978, -0.0025423753540962934, 0.48053520917892456, 0.20798562467098236] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.6796725444 , 9649228.0 ##### | |
| ##### Sending to neptune: dp_per_s : 0.6796725444 , 89.2170627159 ##### | |
| [u'other', 9649228.0, 89.21706271592348] | |
| receiving | |
| [0130 18:10:47 @multigpu.py:323] [p0576] step: count(1709), step_time 1462.95, mean_step_time 1444.63, it/s 0.69 | |
| [0130 18:10:49 @multigpu.py:323] [p0115] step: count(1686), step_time 1456.53, mean_step_time 1429.41, it/s 0.7 | |
| [0130 18:10:49 @multigpu.py:323] [p0574] step: count(1701), step_time 1414.1, mean_step_time 1432.91, it/s 0.7 | |
| [0130 18:10:49 @multigpu.py:323] [p0576] step: count(1710), step_time 1487.65, mean_step_time 1447.96, it/s 0.69 | |
| [0130 18:10:50 @multigpu.py:323] [p0115] step: count(1687), step_time 1401.89, mean_step_time 1427.33, it/s 0.7 | |
| [0130 18:10:50 @multigpu.py:323] [p0574] step: count(1702), step_time 1409.06, mean_step_time 1433.53, it/s 0.7 | |
| [0130 18:10:50 @multigpu.py:323] [p0576] step: count(1711), step_time 1391.33, mean_step_time 1448.14, it/s 0.69 | |
| [0130 18:10:52 @multigpu.py:323] [p0115] step: count(1688), step_time 1571.78, mean_step_time 1431.4, it/s 0.7 | |
| [0130 18:10:52 @multigpu.py:323] [p0574] step: count(1703), step_time 1536.91, mean_step_time 1431.5, it/s 0.7 | |
| [0130 18:10:52 @multigpu.py:323] [p0576] step: count(1712), step_time 1432.06, mean_step_time 1431.35, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.68088431855 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:10:53 @multigpu.py:323] [p0574] step: count(1704), step_time 1428.53, mean_step_time 1430.82, it/s 0.7 | |
| [0130 18:10:53 @multigpu.py:323] [p0576] step: count(1713), step_time 1434.77, mean_step_time 1430.0, it/s 0.7 | |
| [0130 18:10:53 @multigpu.py:323] [p0115] step: count(1689), step_time 1478.01, mean_step_time 1435.65, it/s 0.7 | |
| [0130 18:10:55 @multigpu.py:323] [p0576] step: count(1714), step_time 1380.62, mean_step_time 1430.31, it/s 0.7 | |
| [0130 18:10:55 @multigpu.py:323] [p0574] step: count(1705), step_time 1447.34, mean_step_time 1431.67, it/s 0.7 | |
| [0130 18:10:55 @multigpu.py:323] [p0115] step: count(1690), step_time 1433.78, mean_step_time 1437.56, it/s 0.7 | |
| [0130 18:10:56 @multigpu.py:323] [p0576] step: count(1715), step_time 1425.05, mean_step_time 1430.87, it/s 0.7 | |
| [0130 18:10:56 @multigpu.py:323] [p0574] step: count(1706), step_time 1450.91, mean_step_time 1433.67, it/s 0.7 | |
| [0130 18:10:56 @multigpu.py:323] [p0115] step: count(1691), step_time 1437.53, mean_step_time 1439.51, it/s 0.69 | |
| [0130 18:10:57 @multigpu.py:323] [p0576] step: count(1716), step_time 1372.36, mean_step_time 1429.99, it/s 0.7 | |
| [0130 18:10:58 @multigpu.py:323] [p0115] step: count(1692), step_time 1399.7, mean_step_time 1436.42, it/s 0.7 | |
| [0130 18:10:58 @multigpu.py:323] [p0574] step: count(1707), step_time 1426.04, mean_step_time 1432.53, it/s 0.7 | |
| [0130 18:10:59 @multigpu.py:323] [p0576] step: count(1717), step_time 1474.3, mean_step_time 1431.0, it/s 0.7 | |
| [0130 18:10:59 @multigpu.py:323] [p0115] step: count(1693), step_time 1419.96, mean_step_time 1434.21, it/s 0.7 | |
| [0130 18:10:59 @multigpu.py:323] [p0574] step: count(1708), step_time 1448.96, mean_step_time 1435.05, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.682994077471 , 2.6 ##### | |
| [u'online', 2.6] | |
| receiving | |
| [0130 18:11:00 @multigpu.py:323] [p0576] step: count(1718), step_time 1428.45, mean_step_time 1427.98, it/s 0.7 | |
| [0130 18:11:00 @multigpu.py:323] [p0574] step: count(1709), step_time 1409.53, mean_step_time 1433.67, it/s 0.7 | |
| [0130 18:11:00 @multigpu.py:323] [p0115] step: count(1694), step_time 1501.43, mean_step_time 1436.17, it/s 0.7 | |
| [0130 18:11:02 @multigpu.py:323] [p0576] step: count(1719), step_time 1411.97, mean_step_time 1428.17, it/s 0.7 | |
| [0130 18:11:02 @multigpu.py:323] [p0574] step: count(1710), step_time 1445.18, mean_step_time 1433.34, it/s 0.7 | |
| [0130 18:11:02 @multigpu.py:323] [p0115] step: count(1695), step_time 1403.87, mean_step_time 1435.85, it/s 0.7 | |
| [0130 18:11:03 @multigpu.py:323] [p0576] step: count(1720), step_time 1433.42, mean_step_time 1429.45, it/s 0.7 | |
| [0130 18:11:03 @multigpu.py:323] [p0574] step: count(1711), step_time 1412.82, mean_step_time 1433.99, it/s 0.7 | |
| [0130 18:11:03 @multigpu.py:323] [p0115] step: count(1696), step_time 1435.7, mean_step_time 1433.55, it/s 0.7 | |
| [0130 18:11:05 @multigpu.py:323] [p0576] step: count(1721), step_time 1423.2, mean_step_time 1428.52, it/s 0.7 | |
| [0130 18:11:05 @multigpu.py:323] [p0574] step: count(1712), step_time 1433.86, mean_step_time 1433.69, it/s 0.7 | |
| [0130 18:11:05 @multigpu.py:323] [p0115] step: count(1697), step_time 1434.6, mean_step_time 1434.94, it/s 0.7 | |
| [0130 18:11:06 @multigpu.py:323] [p0576] step: count(1722), step_time 1474.72, mean_step_time 1432.41, it/s 0.7 | |
| [0130 18:11:06 @multigpu.py:323] [p0115] step: count(1698), step_time 1399.99, mean_step_time 1433.83, it/s 0.7 | |
| [0130 18:11:06 @multigpu.py:323] [p0574] step: count(1713), step_time 1499.86, mean_step_time 1439.0, it/s 0.69 | |
| [0130 18:11:07 @multigpu.py:323] [p0576] step: count(1723), step_time 1408.59, mean_step_time 1430.73, it/s 0.7 | |
| [0130 18:11:08 @multigpu.py:323] [p0115] step: count(1699), step_time 1455.65, mean_step_time 1437.18, it/s 0.7 | |
| [0130 18:11:08 @multigpu.py:323] [p0574] step: count(1714), step_time 1492.68, mean_step_time 1439.69, it/s 0.69 | |
| [0130 18:11:09 @multigpu.py:323] [p0576] step: count(1724), step_time 1517.96, mean_step_time 1434.31, it/s 0.7 | |
| [0130 18:11:09 @multigpu.py:323] [p0115] step: count(1700), step_time 1391.7, mean_step_time 1436.52, it/s 0.7 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.685659803549 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.685659803549 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.685659803549 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.685660319659 , -0.00245934864506 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.685660319659 , 0.0409755147994 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.685660319659 , -2.28138971329 ##### | |
| ##### Sending to neptune: value_loss : 0.685660319659 , 1.92561733723 ##### | |
| ##### Sending to neptune: advantage : 0.685660319659 , -0.000262442481471 ##### | |
| ##### Sending to neptune: pred_reward : 0.685660319659 , 0.488827288151 ##### | |
| ##### Sending to neptune: max_logit : 0.685660319659 , 0.209326460958 ##### | |
| [u'loss', -0.0024593486450612545, 0.04097551479935646, -2.2813897132873535, 1.9256173372268677, -0.0002624424814712256, 0.48882728815078735, 0.20932646095752716] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.685660764972 , 9660044.15 ##### | |
| ##### Sending to neptune: dp_per_s : 0.685660764972 , 89.2862143186 ##### | |
| [u'other', 9660044.15, 89.28621431862673] | |
| receiving | |
| [0130 18:11:09 @multigpu.py:323] [p0574] step: count(1715), step_time 1457.41, mean_step_time 1443.18, it/s 0.69 | |
| [0130 18:11:10 @multigpu.py:323] [p0576] step: count(1725), step_time 1414.69, mean_step_time 1431.71, it/s 0.7 | |
| [0130 18:11:10 @multigpu.py:323] [p0115] step: count(1701), step_time 1473.77, mean_step_time 1440.14, it/s 0.69 | |
| [0130 18:11:11 @multigpu.py:323] [p0574] step: count(1716), step_time 1443.67, mean_step_time 1444.02, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.686278829442 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 18:11:12 @multigpu.py:323] [p0576] step: count(1726), step_time 1385.38, mean_step_time 1431.33, it/s 0.7 | |
| [0130 18:11:12 @multigpu.py:323] [p0115] step: count(1702), step_time 1477.13, mean_step_time 1442.66, it/s 0.69 | |
| [0130 18:11:12 @multigpu.py:323] [p0574] step: count(1717), step_time 1432.63, mean_step_time 1445.75, it/s 0.69 | |
| [0130 18:11:13 @multigpu.py:323] [p0576] step: count(1727), step_time 1440.82, mean_step_time 1431.85, it/s 0.7 | |
| [0130 18:11:13 @multigpu.py:323] [p0115] step: count(1703), step_time 1427.43, mean_step_time 1442.6, it/s 0.69 | |
| [0130 18:11:13 @multigpu.py:323] [p0574] step: count(1718), step_time 1400.08, mean_step_time 1440.62, it/s 0.69 | |
| [0130 18:11:15 @multigpu.py:323] [p0576] step: count(1728), step_time 1435.43, mean_step_time 1431.79, it/s 0.7 | |
| [0130 18:11:15 @multigpu.py:323] [p0115] step: count(1704), step_time 1427.15, mean_step_time 1440.77, it/s 0.69 | |
| [0130 18:11:15 @multigpu.py:323] [p0574] step: count(1719), step_time 1447.53, mean_step_time 1443.4, it/s 0.69 | |
| [0130 18:11:16 @multigpu.py:323] [p0576] step: count(1729), step_time 1376.1, mean_step_time 1427.44, it/s 0.7 | |
| [0130 18:11:16 @multigpu.py:323] [p0115] step: count(1705), step_time 1408.15, mean_step_time 1441.79, it/s 0.69 | |
| [0130 18:11:16 @multigpu.py:323] [p0574] step: count(1720), step_time 1473.9, mean_step_time 1445.55, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.68780823138 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 18:11:17 @multigpu.py:323] [p0576] step: count(1730), step_time 1415.12, mean_step_time 1423.82, it/s 0.7 | |
| [0130 18:11:18 @multigpu.py:323] [p0115] step: count(1706), step_time 1437.1, mean_step_time 1440.81, it/s 0.69 | |
| [0130 18:11:18 @multigpu.py:323] [p0574] step: count(1721), step_time 1391.96, mean_step_time 1444.44, it/s 0.69 | |
| [0130 18:11:19 @multigpu.py:323] [p0576] step: count(1731), step_time 1425.59, mean_step_time 1425.53, it/s 0.7 | |
| [0130 18:11:19 @multigpu.py:323] [p0115] step: count(1707), step_time 1398.22, mean_step_time 1440.63, it/s 0.69 | |
| [0130 18:11:19 @multigpu.py:323] [p0574] step: count(1722), step_time 1453.42, mean_step_time 1446.66, it/s 0.69 | |
| [0130 18:11:21 @multigpu.py:323] [p0574] step: count(1723), step_time 1386.74, mean_step_time 1439.15, it/s 0.69 | |
| [0130 18:11:21 @multigpu.py:323] [p0115] step: count(1708), step_time 1543.25, mean_step_time 1439.2, it/s 0.69 | |
| [0130 18:11:21 @multigpu.py:323] [p0576] step: count(1732), step_time 1705.21, mean_step_time 1439.19, it/s 0.69 | |
| [0130 18:11:22 @multigpu.py:323] [p0574] step: count(1724), step_time 1415.99, mean_step_time 1438.53, it/s 0.7 | |
| [0130 18:11:22 @multigpu.py:323] [p0576] step: count(1733), step_time 1415.76, mean_step_time 1438.24, it/s 0.7 | |
| [0130 18:11:22 @multigpu.py:323] [p0115] step: count(1709), step_time 1435.06, mean_step_time 1437.06, it/s 0.7 | |
| [0130 18:11:23 @multigpu.py:323] [p0574] step: count(1725), step_time 1413.88, mean_step_time 1436.85, it/s 0.7 | |
| [0130 18:11:23 @multigpu.py:323] [p0115] step: count(1710), step_time 1420.78, mean_step_time 1436.41, it/s 0.7 | |
| [0130 18:11:23 @multigpu.py:323] [p0576] step: count(1734), step_time 1449.01, mean_step_time 1441.66, it/s 0.69 | |
| [0130 18:11:25 @multigpu.py:323] [p0574] step: count(1726), step_time 1437.26, mean_step_time 1436.17, it/s 0.7 | |
| [0130 18:11:25 @multigpu.py:323] [p0576] step: count(1735), step_time 1410.08, mean_step_time 1440.91, it/s 0.69 | |
| [0130 18:11:25 @multigpu.py:323] [p0115] step: count(1711), step_time 1478.62, mean_step_time 1438.46, it/s 0.7 | |
| [0130 18:11:26 @multigpu.py:323] [p0576] step: count(1736), step_time 1427.03, mean_step_time 1443.64, it/s 0.69 | |
| [0130 18:11:26 @multigpu.py:323] [p0574] step: count(1727), step_time 1438.08, mean_step_time 1436.77, it/s 0.7 | |
| [0130 18:11:26 @multigpu.py:323] [p0115] step: count(1712), step_time 1406.9, mean_step_time 1438.82, it/s 0.7 | |
| [0130 18:11:28 @multigpu.py:323] [p0576] step: count(1737), step_time 1439.18, mean_step_time 1441.88, it/s 0.69 | |
| [0130 18:11:28 @multigpu.py:323] [p0574] step: count(1728), step_time 1446.6, mean_step_time 1436.65, it/s 0.7 | |
| [0130 18:11:28 @multigpu.py:323] [p0115] step: count(1713), step_time 1424.78, mean_step_time 1439.06, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.691006313827 , 1.3 ##### | |
| [u'online', 1.3] | |
| ##### Sending to neptune: active_workers : 0.691006366611 , 3 ##### | |
| receiving | |
| [0130 18:11:29 @multigpu.py:323] [p0574] step: count(1729), step_time 1424.96, mean_step_time 1437.43, it/s 0.7 | |
| [0130 18:11:29 @multigpu.py:323] [p0115] step: count(1714), step_time 1415.67, mean_step_time 1434.77, it/s 0.7 | |
| [0130 18:11:29 @multigpu.py:323] [p0576] step: count(1738), step_time 1476.65, mean_step_time 1444.29, it/s 0.69 | |
| [0130 18:11:31 @multigpu.py:323] [p0115] step: count(1715), step_time 1406.08, mean_step_time 1434.89, it/s 0.7 | |
| [0130 18:11:31 @multigpu.py:323] [p0576] step: count(1739), step_time 1372.57, mean_step_time 1442.33, it/s 0.69 | |
| [0130 18:11:31 @multigpu.py:323] [p0574] step: count(1730), step_time 1460.12, mean_step_time 1438.17, it/s 0.7 | |
| [0130 18:11:32 @multigpu.py:323] [p0115] step: count(1716), step_time 1424.92, mean_step_time 1434.35, it/s 0.7 | |
| [0130 18:11:32 @multigpu.py:323] [p0574] step: count(1731), step_time 1438.66, mean_step_time 1439.46, it/s 0.69 | |
| [0130 18:11:32 @multigpu.py:323] [p0576] step: count(1740), step_time 1510.8, mean_step_time 1446.19, it/s 0.69 | |
| [0130 18:11:33 @multigpu.py:323] [p0574] step: count(1732), step_time 1382.75, mean_step_time 1436.91, it/s 0.7 | |
| [0130 18:11:33 @multigpu.py:323] [p0115] step: count(1717), step_time 1447.9, mean_step_time 1435.01, it/s 0.7 | |
| [0130 18:11:33 @multigpu.py:323] [p0576] step: count(1741), step_time 1415.93, mean_step_time 1445.83, it/s 0.69 | |
| [0130 18:11:35 @multigpu.py:323] [p0574] step: count(1733), step_time 1436.02, mean_step_time 1433.72, it/s 0.7 | |
| [0130 18:11:35 @multigpu.py:323] [p0115] step: count(1718), step_time 1439.82, mean_step_time 1437.0, it/s 0.7 | |
| [0130 18:11:35 @multigpu.py:323] [p0576] step: count(1742), step_time 1452.77, mean_step_time 1444.73, it/s 0.69 | |
| [0130 18:11:36 @multigpu.py:323] [p0115] step: count(1719), step_time 1403.97, mean_step_time 1434.42, it/s 0.7 | |
| [0130 18:11:36 @multigpu.py:323] [p0574] step: count(1734), step_time 1438.25, mean_step_time 1431.0, it/s 0.7 | |
| [0130 18:11:36 @multigpu.py:323] [p0576] step: count(1743), step_time 1377.33, mean_step_time 1443.17, it/s 0.69 | |
| [0130 18:11:38 @multigpu.py:323] [p0115] step: count(1720), step_time 1465.36, mean_step_time 1438.1, it/s 0.7 | |
| [0130 18:11:38 @multigpu.py:323] [p0574] step: count(1735), step_time 1441.22, mean_step_time 1430.19, it/s 0.7 | |
| [0130 18:11:38 @multigpu.py:323] [p0576] step: count(1744), step_time 1451.9, mean_step_time 1439.87, it/s 0.69 | |
| [0130 18:11:39 @multigpu.py:323] [p0574] step: count(1736), step_time 1405.73, mean_step_time 1428.29, it/s 0.7 | |
| [0130 18:11:39 @multigpu.py:323] [p0115] step: count(1721), step_time 1438.44, mean_step_time 1436.34, it/s 0.7 | |
| [0130 18:11:39 @multigpu.py:323] [p0576] step: count(1745), step_time 1456.66, mean_step_time 1441.97, it/s 0.69 | |
| [0130 18:11:41 @multigpu.py:323] [p0115] step: count(1722), step_time 1376.54, mean_step_time 1431.31, it/s 0.7 | |
| [0130 18:11:41 @multigpu.py:323] [p0574] step: count(1737), step_time 1420.84, mean_step_time 1427.7, it/s 0.7 | |
| [0130 18:11:41 @multigpu.py:323] [p0576] step: count(1746), step_time 1445.38, mean_step_time 1444.97, it/s 0.69 | |
| [0130 18:11:42 @multigpu.py:323] [p0115] step: count(1723), step_time 1423.86, mean_step_time 1431.13, it/s 0.7 | |
| [0130 18:11:42 @multigpu.py:323] [p0574] step: count(1738), step_time 1433.22, mean_step_time 1429.36, it/s 0.7 | |
| [0130 18:11:42 @multigpu.py:323] [p0576] step: count(1747), step_time 1447.7, mean_step_time 1445.31, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.694909020795 , 1.3 ##### | |
| [u'online', 1.3] | |
| receiving | |
| [0130 18:11:43 @multigpu.py:323] [p0115] step: count(1724), step_time 1421.89, mean_step_time 1430.86, it/s 0.7 | |
| [0130 18:11:43 @multigpu.py:323] [p0574] step: count(1739), step_time 1427.83, mean_step_time 1428.37, it/s 0.7 | |
| [0130 18:11:44 @multigpu.py:323] [p0576] step: count(1748), step_time 1419.06, mean_step_time 1444.49, it/s 0.69 | |
| [0130 18:11:45 @multigpu.py:323] [p0115] step: count(1725), step_time 1423.5, mean_step_time 1431.63, it/s 0.7 | |
| [0130 18:11:45 @multigpu.py:323] [p0574] step: count(1740), step_time 1404.35, mean_step_time 1424.89, it/s 0.7 | |
| [0130 18:11:45 @multigpu.py:323] [p0576] step: count(1749), step_time 1449.14, mean_step_time 1448.14, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.695835398568 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:11:46 @multigpu.py:323] [p0574] step: count(1741), step_time 1394.8, mean_step_time 1425.04, it/s 0.7 | |
| [0130 18:11:46 @multigpu.py:323] [p0115] step: count(1726), step_time 1435.46, mean_step_time 1431.55, it/s 0.7 | |
| [0130 18:11:46 @multigpu.py:323] [p0576] step: count(1750), step_time 1435.3, mean_step_time 1449.15, it/s 0.69 | |
| [0130 18:11:48 @multigpu.py:323] [p0115] step: count(1727), step_time 1379.68, mean_step_time 1430.62, it/s 0.7 | |
| [0130 18:11:48 @multigpu.py:323] [p0574] step: count(1742), step_time 1454.37, mean_step_time 1425.08, it/s 0.7 | |
| [0130 18:11:48 @multigpu.py:323] [p0576] step: count(1751), step_time 1403.57, mean_step_time 1448.05, it/s 0.69 | |
| [0130 18:11:49 @multigpu.py:323] [p0115] step: count(1728), step_time 1607.47, mean_step_time 1433.83, it/s 0.7 | |
| [0130 18:11:49 @multigpu.py:323] [p0574] step: count(1743), step_time 1562.31, mean_step_time 1433.86, it/s 0.7 | |
| [0130 18:11:49 @multigpu.py:323] [p0576] step: count(1752), step_time 1419.29, mean_step_time 1433.76, it/s 0.7 | |
| [0130 18:11:51 @multigpu.py:323] [p0574] step: count(1744), step_time 1386.85, mean_step_time 1432.4, it/s 0.7 | |
| [0130 18:11:51 @multigpu.py:323] [p0576] step: count(1753), step_time 1400.35, mean_step_time 1432.99, it/s 0.7 | |
| [0130 18:11:51 @multigpu.py:323] [p0115] step: count(1729), step_time 1486.65, mean_step_time 1436.41, it/s 0.7 | |
| [0130 18:11:52 @multigpu.py:323] [p0574] step: count(1745), step_time 1458.63, mean_step_time 1434.64, it/s 0.7 | |
| [0130 18:11:52 @multigpu.py:323] [p0576] step: count(1754), step_time 1454.21, mean_step_time 1433.25, it/s 0.7 | |
| [0130 18:11:52 @multigpu.py:323] [p0115] step: count(1730), step_time 1413.23, mean_step_time 1436.04, it/s 0.7 | |
| [0130 18:11:53 @multigpu.py:323] [p0574] step: count(1746), step_time 1386.86, mean_step_time 1432.12, it/s 0.7 | |
| [0130 18:11:54 @multigpu.py:323] [p0576] step: count(1755), step_time 1419.0, mean_step_time 1433.69, it/s 0.7 | |
| [0130 18:11:54 @multigpu.py:323] [p0115] step: count(1731), step_time 1405.57, mean_step_time 1432.38, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.698198745781 , 1.9 ##### | |
| [u'online', 1.9] | |
| receiving | |
| [0130 18:11:55 @multigpu.py:323] [p0574] step: count(1747), step_time 1453.75, mean_step_time 1432.91, it/s 0.7 | |
| [0130 18:11:55 @multigpu.py:323] [p0115] step: count(1732), step_time 1397.21, mean_step_time 1431.9, it/s 0.7 | |
| [0130 18:11:55 @multigpu.py:323] [p0576] step: count(1756), step_time 1431.6, mean_step_time 1433.92, it/s 0.7 | |
| [0130 18:11:56 @multigpu.py:323] [p0574] step: count(1748), step_time 1367.6, mean_step_time 1428.96, it/s 0.7 | |
| [0130 18:11:56 @multigpu.py:323] [p0576] step: count(1757), step_time 1407.07, mean_step_time 1432.32, it/s 0.7 | |
| [0130 18:11:56 @multigpu.py:323] [p0115] step: count(1733), step_time 1444.28, mean_step_time 1432.87, it/s 0.7 | |
| [0130 18:11:58 @multigpu.py:323] [p0574] step: count(1749), step_time 1424.37, mean_step_time 1428.93, it/s 0.7 | |
| [0130 18:11:58 @multigpu.py:323] [p0576] step: count(1758), step_time 1467.88, mean_step_time 1431.88, it/s 0.7 | |
| [0130 18:11:58 @multigpu.py:323] [p0115] step: count(1734), step_time 1444.77, mean_step_time 1434.33, it/s 0.7 | |
| [0130 18:11:59 @multigpu.py:323] [p0574] step: count(1750), step_time 1439.9, mean_step_time 1427.92, it/s 0.7 | |
| [0130 18:11:59 @multigpu.py:323] [p0576] step: count(1759), step_time 1413.75, mean_step_time 1433.94, it/s 0.7 | |
| [0130 18:11:59 @multigpu.py:323] [p0115] step: count(1735), step_time 1427.89, mean_step_time 1435.42, it/s 0.7 | |
| [0130 18:12:01 @multigpu.py:323] [p0574] step: count(1751), step_time 1468.3, mean_step_time 1429.4, it/s 0.7 | |
| [0130 18:12:01 @multigpu.py:323] [p0576] step: count(1760), step_time 1429.69, mean_step_time 1429.88, it/s 0.7 | |
| [0130 18:12:01 @multigpu.py:323] [p0115] step: count(1736), step_time 1459.78, mean_step_time 1437.16, it/s 0.7 | |
| [0130 18:12:02 @multigpu.py:323] [p0574] step: count(1752), step_time 1401.17, mean_step_time 1430.32, it/s 0.7 | |
| [0130 18:12:02 @multigpu.py:323] [p0576] step: count(1761), step_time 1453.08, mean_step_time 1431.74, it/s 0.7 | |
| [0130 18:12:02 @multigpu.py:323] [p0115] step: count(1737), step_time 1427.12, mean_step_time 1436.12, it/s 0.7 | |
| [0130 18:12:03 @multigpu.py:323] [p0574] step: count(1753), step_time 1443.76, mean_step_time 1430.71, it/s 0.7 | |
| [0130 18:12:04 @multigpu.py:323] [p0576] step: count(1762), step_time 1381.0, mean_step_time 1428.15, it/s 0.7 | |
| [0130 18:12:04 @multigpu.py:323] [p0115] step: count(1738), step_time 1401.29, mean_step_time 1434.2, it/s 0.7 | |
| [0130 18:12:05 @multigpu.py:323] [p0574] step: count(1754), step_time 1450.24, mean_step_time 1431.31, it/s 0.7 | |
| [0130 18:12:05 @multigpu.py:323] [p0576] step: count(1763), step_time 1446.88, mean_step_time 1431.63, it/s 0.7 | |
| [0130 18:12:05 @multigpu.py:323] [p0115] step: count(1739), step_time 1466.91, mean_step_time 1437.34, it/s 0.7 | |
| [0130 18:12:06 @multigpu.py:323] [p0576] step: count(1764), step_time 1357.56, mean_step_time 1426.91, it/s 0.7 | |
| [0130 18:12:06 @multigpu.py:323] [p0574] step: count(1755), step_time 1422.55, mean_step_time 1430.37, it/s 0.7 | |
| [0130 18:12:06 @multigpu.py:323] [p0115] step: count(1740), step_time 1379.79, mean_step_time 1433.07, it/s 0.7 | |
| [0130 18:12:08 @multigpu.py:323] [p0576] step: count(1765), step_time 1472.51, mean_step_time 1427.7, it/s 0.7 | |
| [0130 18:12:08 @multigpu.py:323] [p0574] step: count(1756), step_time 1487.65, mean_step_time 1434.47, it/s 0.7 | |
| [0130 18:12:08 @multigpu.py:323] [p0115] step: count(1741), step_time 1445.5, mean_step_time 1433.42, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.702142093049 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:12:09 @multigpu.py:323] [p0576] step: count(1766), step_time 1390.08, mean_step_time 1424.94, it/s 0.7 | |
| [0130 18:12:09 @multigpu.py:323] [p0115] step: count(1742), step_time 1414.49, mean_step_time 1435.32, it/s 0.7 | |
| [0130 18:12:09 @multigpu.py:323] [p0574] step: count(1757), step_time 1442.0, mean_step_time 1435.53, it/s 0.7 | |
| [0130 18:12:11 @multigpu.py:323] [p0576] step: count(1767), step_time 1466.38, mean_step_time 1425.87, it/s 0.7 | |
| [0130 18:12:11 @multigpu.py:323] [p0574] step: count(1758), step_time 1425.23, mean_step_time 1435.13, it/s 0.7 | |
| [0130 18:12:11 @multigpu.py:323] [p0115] step: count(1743), step_time 1448.84, mean_step_time 1436.57, it/s 0.7 | |
| [0130 18:12:12 @multigpu.py:323] [p0576] step: count(1768), step_time 1411.88, mean_step_time 1425.51, it/s 0.7 | |
| [0130 18:12:12 @multigpu.py:323] [p0115] step: count(1744), step_time 1406.64, mean_step_time 1435.8, it/s 0.7 | |
| [0130 18:12:12 @multigpu.py:323] [p0574] step: count(1759), step_time 1462.57, mean_step_time 1436.86, it/s 0.7 | |
| [0130 18:12:14 @multigpu.py:323] [p0576] step: count(1769), step_time 1486.35, mean_step_time 1427.37, it/s 0.7 | |
| [0130 18:12:14 @multigpu.py:323] [p0574] step: count(1760), step_time 1393.44, mean_step_time 1436.32, it/s 0.7 | |
| [0130 18:12:14 @multigpu.py:323] [p0115] step: count(1745), step_time 1463.1, mean_step_time 1437.78, it/s 0.7 | |
| [0130 18:12:15 @multigpu.py:323] [p0576] step: count(1770), step_time 1415.81, mean_step_time 1426.4, it/s 0.7 | |
| [0130 18:12:15 @multigpu.py:323] [p0574] step: count(1761), step_time 1446.49, mean_step_time 1438.9, it/s 0.69 | |
| [0130 18:12:15 @multigpu.py:323] [p0115] step: count(1746), step_time 1441.33, mean_step_time 1438.08, it/s 0.7 | |
| [0130 18:12:16 @multigpu.py:323] [p0574] step: count(1762), step_time 1413.98, mean_step_time 1436.88, it/s 0.7 | |
| [0130 18:12:16 @multigpu.py:323] [p0115] step: count(1747), step_time 1410.81, mean_step_time 1439.63, it/s 0.69 | |
| [0130 18:12:16 @multigpu.py:323] [p0576] step: count(1771), step_time 1504.78, mean_step_time 1431.46, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.704557807446 , 2.1 ##### | |
| [u'online', 2.1] | |
| receiving | |
| [0130 18:12:18 @multigpu.py:323] [p0115] step: count(1748), step_time 1460.03, mean_step_time 1432.26, it/s 0.7 | |
| [0130 18:12:18 @multigpu.py:323] [p0576] step: count(1772), step_time 1433.75, mean_step_time 1432.18, it/s 0.7 | |
| [0130 18:12:18 @multigpu.py:323] [p0574] step: count(1763), step_time 1467.53, mean_step_time 1432.14, it/s 0.7 | |
| [0130 18:12:19 @multigpu.py:323] [p0574] step: count(1764), step_time 1422.07, mean_step_time 1433.9, it/s 0.7 | |
| [0130 18:12:19 @multigpu.py:323] [p0576] step: count(1773), step_time 1446.41, mean_step_time 1434.48, it/s 0.7 | |
| [0130 18:12:19 @multigpu.py:323] [p0115] step: count(1749), step_time 1464.81, mean_step_time 1431.17, it/s 0.7 | |
| [0130 18:12:21 @multigpu.py:323] [p0574] step: count(1765), step_time 1389.37, mean_step_time 1430.44, it/s 0.7 | |
| [0130 18:12:21 @multigpu.py:323] [p0576] step: count(1774), step_time 1403.14, mean_step_time 1431.93, it/s 0.7 | |
| [0130 18:12:21 @multigpu.py:323] [p0115] step: count(1750), step_time 1480.55, mean_step_time 1434.53, it/s 0.7 | |
| [0130 18:12:22 @multigpu.py:323] [p0576] step: count(1775), step_time 1423.07, mean_step_time 1432.13, it/s 0.7 | |
| [0130 18:12:22 @multigpu.py:323] [p0574] step: count(1766), step_time 1475.22, mean_step_time 1434.86, it/s 0.7 | |
| [0130 18:12:22 @multigpu.py:323] [p0115] step: count(1751), step_time 1412.36, mean_step_time 1434.87, it/s 0.7 | |
| [0130 18:12:24 @multigpu.py:323] [p0576] step: count(1776), step_time 1395.53, mean_step_time 1430.33, it/s 0.7 | |
| [0130 18:12:24 @multigpu.py:323] [p0574] step: count(1767), step_time 1428.71, mean_step_time 1433.61, it/s 0.7 | |
| [0130 18:12:24 @multigpu.py:323] [p0115] step: count(1752), step_time 1423.21, mean_step_time 1436.17, it/s 0.7 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.706616413593 , 1.7 ##### | |
| [u'online', 1.7] | |
| receiving | |
| [0130 18:12:25 @multigpu.py:323] [p0576] step: count(1777), step_time 1471.75, mean_step_time 1433.56, it/s 0.7 | |
| [0130 18:12:25 @multigpu.py:323] [p0574] step: count(1768), step_time 1438.86, mean_step_time 1437.17, it/s 0.7 | |
| [0130 18:12:25 @multigpu.py:323] [p0115] step: count(1753), step_time 1391.3, mean_step_time 1433.53, it/s 0.7 | |
| [0130 18:12:27 @multigpu.py:323] [p0576] step: count(1778), step_time 1478.33, mean_step_time 1434.09, it/s 0.7 | |
| [0130 18:12:27 @multigpu.py:323] [p0574] step: count(1769), step_time 1474.31, mean_step_time 1439.67, it/s 0.69 | |
| [0130 18:12:27 @multigpu.py:323] [p0115] step: count(1754), step_time 1505.91, mean_step_time 1436.58, it/s 0.7 | |
| [0130 18:12:28 @multigpu.py:323] [p0576] step: count(1779), step_time 1450.3, mean_step_time 1435.91, it/s 0.7 | |
| [0130 18:12:28 @multigpu.py:323] [p0574] step: count(1770), step_time 1442.26, mean_step_time 1439.79, it/s 0.69 | |
| [0130 18:12:28 @multigpu.py:323] [p0115] step: count(1755), step_time 1436.9, mean_step_time 1437.03, it/s 0.7 | |
| [0130 18:12:29 @multigpu.py:323] [p0574] step: count(1771), step_time 1409.93, mean_step_time 1436.87, it/s 0.7 | |
| [0130 18:12:29 @multigpu.py:323] [p0576] step: count(1780), step_time 1448.5, mean_step_time 1436.85, it/s 0.7 | |
| [0130 18:12:29 @multigpu.py:323] [p0115] step: count(1756), step_time 1444.44, mean_step_time 1436.27, it/s 0.7 | |
| [0130 18:12:31 @multigpu.py:323] [p0574] step: count(1772), step_time 1430.18, mean_step_time 1438.32, it/s 0.7 | |
| [0130 18:12:31 @multigpu.py:323] [p0576] step: count(1781), step_time 1446.05, mean_step_time 1436.5, it/s 0.7 | |
| [0130 18:12:31 @multigpu.py:323] [p0115] step: count(1757), step_time 1406.18, mean_step_time 1435.22, it/s 0.7 | |
| [0130 18:12:32 @multigpu.py:323] [p0576] step: count(1782), step_time 1418.56, mean_step_time 1438.38, it/s 0.7 | |
| [0130 18:12:32 @multigpu.py:323] [p0574] step: count(1773), step_time 1475.5, mean_step_time 1439.9, it/s 0.69 | |
| [0130 18:12:32 @multigpu.py:323] [p0115] step: count(1758), step_time 1439.71, mean_step_time 1437.14, it/s 0.7 | |
| [0130 18:12:34 @multigpu.py:323] [p0574] step: count(1774), step_time 1418.35, mean_step_time 1438.31, it/s 0.7 | |
| [0130 18:12:34 @multigpu.py:323] [p0576] step: count(1783), step_time 1465.9, mean_step_time 1439.33, it/s 0.69 | |
| [0130 18:12:34 @multigpu.py:323] [p0115] step: count(1759), step_time 1447.71, mean_step_time 1436.18, it/s 0.7 | |
| [0130 18:12:35 @multigpu.py:323] [p0574] step: count(1775), step_time 1432.75, mean_step_time 1438.82, it/s 0.7 | |
| [0130 18:12:35 @multigpu.py:323] [p0576] step: count(1784), step_time 1417.86, mean_step_time 1442.35, it/s 0.69 | |
| [0130 18:12:35 @multigpu.py:323] [p0115] step: count(1760), step_time 1483.61, mean_step_time 1441.37, it/s 0.69 | |
| [0130 18:12:37 @multigpu.py:323] [p0574] step: count(1776), step_time 1427.38, mean_step_time 1435.81, it/s 0.7 | |
| [0130 18:12:37 @multigpu.py:323] [p0576] step: count(1785), step_time 1414.84, mean_step_time 1439.46, it/s 0.69 | |
| [0130 18:12:37 @multigpu.py:323] [p0115] step: count(1761), step_time 1407.42, mean_step_time 1439.47, it/s 0.69 | |
| [0130 18:12:38 @multigpu.py:323] [p0574] step: count(1777), step_time 1370.77, mean_step_time 1432.24, it/s 0.7 | |
| [0130 18:12:38 @multigpu.py:323] [p0576] step: count(1786), step_time 1478.67, mean_step_time 1443.89, it/s 0.69 | |
| [0130 18:12:38 @multigpu.py:323] [p0115] step: count(1762), step_time 1440.08, mean_step_time 1440.75, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.710599316094 , 0.9 ##### | |
| [u'online', 0.9] | |
| ##### Sending to neptune: active_workers : 0.710599354108 , 3 ##### | |
| receiving | |
| [0130 18:12:39 @multigpu.py:323] [p0574] step: count(1778), step_time 1393.35, mean_step_time 1430.65, it/s 0.7 | |
| [0130 18:12:40 @multigpu.py:323] [p0115] step: count(1763), step_time 1404.38, mean_step_time 1438.52, it/s 0.7 | |
| [0130 18:12:40 @multigpu.py:323] [p0576] step: count(1787), step_time 1472.56, mean_step_time 1444.2, it/s 0.69 | |
| [0130 18:12:41 @multigpu.py:323] [p0574] step: count(1779), step_time 1518.58, mean_step_time 1433.45, it/s 0.7 | |
| [0130 18:12:41 @multigpu.py:323] [p0115] step: count(1764), step_time 1478.67, mean_step_time 1442.13, it/s 0.69 | |
| [0130 18:12:41 @multigpu.py:323] [p0576] step: count(1788), step_time 1448.46, mean_step_time 1446.03, it/s 0.69 | |
| [0130 18:12:42 @multigpu.py:323] [p0574] step: count(1780), step_time 1417.89, mean_step_time 1434.67, it/s 0.7 | |
| [0130 18:12:42 @multigpu.py:323] [p0115] step: count(1765), step_time 1425.04, mean_step_time 1440.22, it/s 0.69 | |
| [0130 18:12:42 @multigpu.py:323] [p0576] step: count(1789), step_time 1459.16, mean_step_time 1444.67, it/s 0.69 | |
| [0130 18:12:44 @multigpu.py:323] [p0574] step: count(1781), step_time 1475.84, mean_step_time 1436.14, it/s 0.7 | |
| [0130 18:12:44 @multigpu.py:323] [p0115] step: count(1766), step_time 1432.78, mean_step_time 1439.79, it/s 0.69 | |
| [0130 18:12:44 @multigpu.py:323] [p0576] step: count(1790), step_time 1401.72, mean_step_time 1443.97, it/s 0.69 | |
| [0130 18:12:45 @multigpu.py:323] [p0574] step: count(1782), step_time 1502.9, mean_step_time 1440.59, it/s 0.69 | |
| [0130 18:12:45 @multigpu.py:323] [p0115] step: count(1767), step_time 1451.68, mean_step_time 1441.84, it/s 0.69 | |
| [0130 18:12:45 @multigpu.py:323] [p0576] step: count(1791), step_time 1448.4, mean_step_time 1441.15, it/s 0.69 | |
| [0130 18:12:47 @multigpu.py:323] [p0115] step: count(1768), step_time 1447.9, mean_step_time 1441.23, it/s 0.69 | |
| [0130 18:12:47 @multigpu.py:323] [p0574] step: count(1783), step_time 1478.6, mean_step_time 1441.14, it/s 0.69 | |
| [0130 18:12:47 @multigpu.py:323] [p0576] step: count(1792), step_time 1434.01, mean_step_time 1441.16, it/s 0.69 | |
| [0130 18:12:48 @multigpu.py:323] [p0576] step: count(1793), step_time 1426.57, mean_step_time 1440.17, it/s 0.69 | |
| [0130 18:12:48 @multigpu.py:323] [p0115] step: count(1769), step_time 1436.7, mean_step_time 1439.83, it/s 0.69 | |
| [0130 18:12:48 @multigpu.py:323] [p0574] step: count(1784), step_time 1524.06, mean_step_time 1446.24, it/s 0.69 | |
| [0130 18:12:50 @multigpu.py:323] [p0576] step: count(1794), step_time 1413.62, mean_step_time 1440.69, it/s 0.69 | |
| [0130 18:12:50 @multigpu.py:323] [p0115] step: count(1770), step_time 1430.53, mean_step_time 1437.33, it/s 0.7 | |
| [0130 18:12:50 @multigpu.py:323] [p0574] step: count(1785), step_time 1421.68, mean_step_time 1447.86, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.713830539385 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 18:12:51 @multigpu.py:323] [p0576] step: count(1795), step_time 1453.62, mean_step_time 1442.22, it/s 0.69 | |
| [0130 18:12:51 @multigpu.py:323] [p0115] step: count(1771), step_time 1456.81, mean_step_time 1439.55, it/s 0.69 | |
| [0130 18:12:51 @multigpu.py:323] [p0574] step: count(1786), step_time 1498.47, mean_step_time 1449.02, it/s 0.69 | |
| [0130 18:12:52 @multigpu.py:323] [p0576] step: count(1796), step_time 1380.06, mean_step_time 1441.45, it/s 0.69 | |
| [0130 18:12:53 @multigpu.py:323] [p0115] step: count(1772), step_time 1439.45, mean_step_time 1440.36, it/s 0.69 | |
| [0130 18:12:53 @multigpu.py:323] [p0574] step: count(1787), step_time 1438.11, mean_step_time 1449.49, it/s 0.69 | |
| [0130 18:12:54 @multigpu.py:323] [p0115] step: count(1773), step_time 1373.46, mean_step_time 1439.47, it/s 0.69 | |
| [0130 18:12:54 @multigpu.py:323] [p0576] step: count(1797), step_time 1508.12, mean_step_time 1443.27, it/s 0.69 | |
| [0130 18:12:54 @multigpu.py:323] [p0574] step: count(1788), step_time 1391.79, mean_step_time 1447.13, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.715037188 , 2.0 ##### | |
| [u'online', 2.0] | |
| receiving | |
| [0130 18:12:55 @multigpu.py:323] [p0576] step: count(1798), step_time 1371.25, mean_step_time 1437.91, it/s 0.7 | |
| [0130 18:12:55 @multigpu.py:323] [p0115] step: count(1774), step_time 1477.79, mean_step_time 1438.06, it/s 0.7 | |
| [0130 18:12:55 @multigpu.py:323] [p0574] step: count(1789), step_time 1416.09, mean_step_time 1444.22, it/s 0.69 | |
| [0130 18:12:57 @multigpu.py:323] [p0115] step: count(1775), step_time 1423.69, mean_step_time 1437.4, it/s 0.7 | |
| [0130 18:12:57 @multigpu.py:323] [p0576] step: count(1799), step_time 1497.91, mean_step_time 1440.29, it/s 0.69 | |
| [0130 18:12:57 @multigpu.py:323] [p0574] step: count(1790), step_time 1439.2, mean_step_time 1444.07, it/s 0.69 | |
| [0130 18:12:58 @multigpu.py:323] [p0115] step: count(1776), step_time 1466.31, mean_step_time 1438.5, it/s 0.7 | |
| [0130 18:12:58 @multigpu.py:323] [p0576] step: count(1800), step_time 1453.53, mean_step_time 1440.54, it/s 0.69 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.716016146938 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.716016146938 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.716016146938 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.716016632716 , -0.0104687893763 ##### | |
| ##### Sending to neptune: policy_loss : 0.716016632716 , -0.755453646183 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: xentropy_loss : 0.716016632716 , -2.27917194366 ##### | |
| ##### Sending to neptune: value_loss : 0.716016632716 , 1.69462025166 ##### | |
| ##### Sending to neptune: advantage : 0.716016632716 , 0.00311736739241 ##### | |
| ##### Sending to neptune: pred_reward : 0.716016632716 , 0.461956590414 ##### | |
| ##### Sending to neptune: max_logit : 0.716016632716 , 0.216024249792 ##### | |
| [u'loss', -0.010468789376318455, -0.7554536461830139, -2.279171943664551, 1.6946202516555786, 0.0031173673924058676, 0.46195659041404724, 0.216024249792099] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.71601706273 , 9774144.07 ##### | |
| ##### Sending to neptune: dp_per_s : 0.71601706273 , 89.0824345897 ##### | |
| [u'other', 9774144.07, 89.08243458972638] | |
| receiving | |
| [0130 18:12:58 @multigpu.py:323] [p0574] step: count(1791), step_time 1447.3, mean_step_time 1445.94, it/s 0.69 | |
| [0130 18:13:00 @multigpu.py:323] [p0576] step: count(1801), step_time 1443.6, mean_step_time 1440.42, it/s 0.69 | |
| [0130 18:13:00 @multigpu.py:323] [p0574] step: count(1792), step_time 1389.92, mean_step_time 1443.93, it/s 0.69 | |
| [0130 18:13:00 @multigpu.py:323] [p0115] step: count(1777), step_time 1474.88, mean_step_time 1441.93, it/s 0.69 | |
| [0130 18:13:01 @multigpu.py:323] [p0576] step: count(1802), step_time 1441.11, mean_step_time 1441.55, it/s 0.69 | |
| [0130 18:13:01 @multigpu.py:323] [p0115] step: count(1778), step_time 1417.01, mean_step_time 1440.79, it/s 0.69 | |
| [0130 18:13:01 @multigpu.py:323] [p0574] step: count(1793), step_time 1536.17, mean_step_time 1446.96, it/s 0.69 | |
| [0130 18:13:03 @multigpu.py:323] [p0115] step: count(1779), step_time 1430.11, mean_step_time 1439.91, it/s 0.69 | |
| [0130 18:13:03 @multigpu.py:323] [p0576] step: count(1803), step_time 1462.05, mean_step_time 1441.36, it/s 0.69 | |
| [0130 18:13:03 @multigpu.py:323] [p0574] step: count(1794), step_time 1425.58, mean_step_time 1447.32, it/s 0.69 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.717419414388 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| [0130 18:13:04 @multigpu.py:323] [p0115] step: count(1780), step_time 1381.04, mean_step_time 1434.79, it/s 0.7 | |
| [0130 18:13:04 @multigpu.py:323] [p0576] step: count(1804), step_time 1468.83, mean_step_time 1443.9, it/s 0.69 | |
| [0130 18:13:04 @multigpu.py:323] [p0574] step: count(1795), step_time 1551.57, mean_step_time 1453.26, it/s 0.69 | |
| [0130 18:13:05 @multigpu.py:323] [p0115] step: count(1781), step_time 1463.72, mean_step_time 1437.6, it/s 0.7 | |
| [0130 18:13:06 @multigpu.py:323] [p0576] step: count(1805), step_time 1420.94, mean_step_time 1444.21, it/s 0.69 | |
| [0130 18:13:06 @multigpu.py:323] [p0574] step: count(1796), step_time 1475.27, mean_step_time 1455.66, it/s 0.69 | |
| [0130 18:13:07 @multigpu.py:323] [p0115] step: count(1782), step_time 1433.38, mean_step_time 1437.27, it/s 0.7 | |
| [0130 18:13:07 @multigpu.py:323] [p0576] step: count(1806), step_time 1375.32, mean_step_time 1439.04, it/s 0.69 | |
| [0130 18:13:07 @multigpu.py:323] [p0574] step: count(1797), step_time 1493.17, mean_step_time 1461.78, it/s 0.68 | |
| [0130 18:13:08 @multigpu.py:323] [p0115] step: count(1783), step_time 1401.67, mean_step_time 1437.13, it/s 0.7 | |
| [0130 18:13:08 @multigpu.py:323] [p0576] step: count(1807), step_time 1430.79, mean_step_time 1436.95, it/s 0.7 | |
| [0130 18:13:09 @multigpu.py:323] [p0574] step: count(1798), step_time 1530.55, mean_step_time 1468.64, it/s 0.68 | |
| [0130 18:13:10 @multigpu.py:323] [p0115] step: count(1784), step_time 1447.27, mean_step_time 1435.56, it/s 0.7 | |
| [0130 18:13:10 @multigpu.py:323] [p0576] step: count(1808), step_time 1412.44, mean_step_time 1435.15, it/s 0.7 | |
| 80%|#######9 |799/1000[19:[0130 18:13:10 @multigpu.py:323] [p0574] step: count(1799), step_time 1397.36, mean_step_time 1462.58, it/s 0.68 | |
| [0130 18:13:11 @multigpu.py:323] [p0115] step: count(1785), step_time 1426.48, mean_step_time 1435.63, it/s 0.7 | |
| [0130 18:13:11 @multigpu.py:323] [p0576] step: count(1809), step_time 1425.22, mean_step_time 1433.46, it/s 0.7 | |
| [0130 18:13:12 @multigpu.py:323] [p0574] step: count(1800), step_time 1476.52, mean_step_time 1465.51, it/s 0.68 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.719725394713 , 0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: max_delay : 0.719725394713 , -0.0 ##### | |
| ##### Sending to neptune: min_delay : 0.719725394713 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.719725849695 , -0.00274460855871 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.719725849695 , 0.0771632865071 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.719725849695 , -2.27880167961 ##### | |
| ##### Sending to neptune: value_loss : 0.719725849695 , 1.85032880306 ##### | |
| ##### Sending to neptune: advantage : 0.719725849695 , -0.000355047988705 ##### | |
| ##### Sending to neptune: pred_reward : 0.719725849695 , 0.449805110693 ##### | |
| ##### Sending to neptune: max_logit : 0.719725849695 , 0.217000186443 ##### | |
| [u'loss', -0.00274460855871439, 0.07716328650712967, -2.278801679611206, 1.850328803062439, -0.00035504798870533705, 0.4498051106929779, 0.21700018644332886] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.719726285802 , 9780323.06 ##### | |
| ##### Sending to neptune: dp_per_s : 0.719726285802 , 89.0251181659 ##### | |
| [u'other', 9780323.06, 89.02511816592134] | |
| receiving | |
| [0130 18:13:13 @multigpu.py:323] [p0576] step: count(1810), step_time 1408.54, mean_step_time 1433.8, it/s 0.7 | |
| [0130 18:13:13 @multigpu.py:323] [p0115] step: count(1786), step_time 1458.67, mean_step_time 1436.93, it/s 0.7 | |
| [0130 18:13:13 @multigpu.py:323] [p0574] step: count(1801), step_time 1422.7, mean_step_time 1462.85, it/s 0.68 | |
| [0130 18:13:14 @multigpu.py:323] [p0576] step: count(1811), step_time 1420.61, mean_step_time 1432.41, it/s 0.7 | |
| [0130 18:13:14 @multigpu.py:323] [p0115] step: count(1787), step_time 1511.66, mean_step_time 1439.93, it/s 0.69 | |
| [0130 18:13:14 @multigpu.py:323] [p0574] step: count(1802), step_time 1416.28, mean_step_time 1458.52, it/s 0.69 | |
| [0130 18:13:16 @multigpu.py:323] [p0576] step: count(1812), step_time 1975.63, mean_step_time 1459.49, it/s 0.69 | |
| [0130 18:13:16 @multigpu.py:323] [p0574] step: count(1803), step_time 1497.58, mean_step_time 1459.47, it/s 0.69 | |
| [0130 18:13:16 @multigpu.py:323] [p0115] step: count(1788), step_time 1839.76, mean_step_time 1459.52, it/s 0.69 | |
| [0130 18:13:17 @multigpu.py:323] [p0576] step: count(1813), step_time 1410.21, mean_step_time 1458.67, it/s 0.69 | |
| [0130 18:13:17 @multigpu.py:323] [p0115] step: count(1789), step_time 1418.83, mean_step_time 1458.63, it/s 0.69 | |
| [0130 18:13:17 @multigpu.py:323] [p0574] step: count(1804), step_time 1427.75, mean_step_time 1454.65, it/s 0.69 | |
| [0130 18:13:19 @multigpu.py:323] [p0576] step: count(1814), step_time 1428.95, mean_step_time 1459.44, it/s 0.69 | |
| [0130 18:13:19 @multigpu.py:323] [p0115] step: count(1790), step_time 1435.65, mean_step_time 1458.88, it/s 0.69 | |
| [0130 18:13:19 @multigpu.py:323] [p0574] step: count(1805), step_time 1583.53, mean_step_time 1462.75, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.721848295 , 2.2 ##### | |
| [u'online', 2.2] | |
| receiving | |
| [0130 18:13:20 @multigpu.py:323] [p0576] step: count(1815), step_time 1385.35, mean_step_time 1456.02, it/s 0.69 | |
| [0130 18:13:20 @multigpu.py:323] [p0115] step: count(1791), step_time 1441.33, mean_step_time 1458.11, it/s 0.69 | |
| [0130 18:13:20 @multigpu.py:323] [p0574] step: count(1806), step_time 1451.14, mean_step_time 1460.38, it/s 0.68 | |
| [0130 18:13:22 @multigpu.py:323] [p0576] step: count(1816), step_time 1406.67, mean_step_time 1457.35, it/s 0.69 | |
| [0130 18:13:22 @multigpu.py:323] [p0115] step: count(1792), step_time 1455.17, mean_step_time 1458.89, it/s 0.69 | |
| [0130 18:13:22 @multigpu.py:323] [p0574] step: count(1807), step_time 1446.57, mean_step_time 1460.8, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.722776274681 , 1.5 ##### | |
| [u'online', 1.5] | |
| receiving | |
| 80%|#######9 |799/1000[19:[0130 18:13:23 @multigpu.py:323] [p0576] step: count(1817), step_time 1429.63, mean_step_time 1453.43, it/s 0.69 | |
| [0130 18:13:23 @multigpu.py:323] [p0115] step: count(1793), step_time 1429.46, mean_step_time 1461.69, it/s 0.68 | |
| [0130 18:13:23 @multigpu.py:323] [p0574] step: count(1808), step_time 1495.21, mean_step_time 1465.97, it/s 0.68 | |
| [0130 18:13:24 @multigpu.py:323] [p0576] step: count(1818), step_time 1461.6, mean_step_time 1457.95, it/s 0.69 | |
| [0130 18:13:25 @multigpu.py:323] [p0115] step: count(1794), step_time 1423.89, mean_step_time 1459.0, it/s 0.69 | |
| [0130 18:13:25 @multigpu.py:323] [p0574] step: count(1809), step_time 1456.05, mean_step_time 1467.97, it/s 0.68 | |
| [0130 18:13:26 @multigpu.py:323] [p0576] step: count(1819), step_time 1457.85, mean_step_time 1455.94, it/s 0.69 | |
| [0130 18:13:26 @multigpu.py:323] [p0115] step: count(1795), step_time 1457.69, mean_step_time 1460.7, it/s 0.68 | |
| [0130 18:13:26 @multigpu.py:323] [p0574] step: count(1810), step_time 1477.54, mean_step_time 1469.89, it/s 0.68 | |
| [0130 18:13:27 @multigpu.py:323] [p0576] step: count(1820), step_time 1419.63, mean_step_time 1454.25, it/s 0.69 | |
| [0130 18:13:27 @multigpu.py:323] [p0115] step: count(1796), step_time 1477.3, mean_step_time 1461.25, it/s 0.68 | |
| [0130 18:13:28 @multigpu.py:323] [p0574] step: count(1811), step_time 1431.16, mean_step_time 1469.08, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.724213219682 , 1.2 ##### | |
| [u'online', 1.2] | |
| receiving | |
| [0130 18:13:29 @multigpu.py:323] [p0576] step: count(1821), step_time 1390.79, mean_step_time 1451.61, it/s 0.69 | |
| [0130 18:13:29 @multigpu.py:323] [p0115] step: count(1797), step_time 1504.25, mean_step_time 1462.72, it/s 0.68 | |
| [0130 18:13:29 @multigpu.py:323] [p0574] step: count(1812), step_time 1439.13, mean_step_time 1471.54, it/s 0.68 | |
| [0130 18:13:30 @multigpu.py:323] [p0576] step: count(1822), step_time 1412.46, mean_step_time 1450.17, it/s 0.69 | |
| [0130 18:13:30 @multigpu.py:323] [p0115] step: count(1798), step_time 1406.23, mean_step_time 1462.18, it/s 0.68 | |
| [0130 18:13:31 @multigpu.py:323] [p0574] step: count(1813), step_time 1544.17, mean_step_time 1471.94, it/s 0.68 | |
| [0130 18:13:32 @multigpu.py:323] [p0576] step: count(1823), step_time 1487.29, mean_step_time 1451.44, it/s 0.69 | |
| [0130 18:13:32 @multigpu.py:323] [p0115] step: count(1799), step_time 1443.42, mean_step_time 1462.84, it/s 0.68 | |
| [0130 18:13:32 @multigpu.py:323] [p0574] step: count(1814), step_time 1427.82, mean_step_time 1472.05, it/s 0.68 | |
| [0130 18:13:33 @multigpu.py:323] [p0576] step: count(1824), step_time 1426.19, mean_step_time 1449.31, it/s 0.69 | |
| [0130 18:13:33 @multigpu.py:323] [p0115] step: count(1800), step_time 1415.9, mean_step_time 1464.59, it/s 0.68 | |
| sending debugging info... | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: mean_delay : 0.725740494397 , 0.0 ##### | |
| ##### Sending to neptune: max_delay : 0.725740494397 , -0.0 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: min_delay : 0.725740494397 , -0.0 ##### | |
| [u'delays', [0.0, -0.0, -0.0]] | |
| receiving | |
| ##### Sending to neptune: cost : 0.725740992162 , 0.00403387658298 ##### | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: policy_loss : 0.725740992162 , 0.927618563175 ##### | |
| ##### Sending to neptune: xentropy_loss : 0.725740992162 , -2.27812504768 ##### | |
| ##### Sending to neptune: value_loss : 0.725740992162 , 1.86684274673 ##### | |
| ##### Sending to neptune: advantage : 0.725740992162 , -0.00428262539208 ##### | |
| ##### Sending to neptune: pred_reward : 0.725740992162 , 0.439312964678 ##### | |
| ##### Sending to neptune: max_logit : 0.725740992162 , 0.219251438975 ##### | |
| [u'loss', 0.004033876582980156, 0.9276185631752014, -2.278125047683716, 1.8668427467346191, -0.004282625392079353, 0.43931296467781067, 0.21925143897533417] | |
| receiving | |
| ##### Sending to neptune: active_relus : 0.725741453038 , 9763375.74 ##### | |
| ##### Sending to neptune: dp_per_s : 0.725741453038 , 88.8911625156 ##### | |
| [u'other', 9763375.74, 88.89116251556536] | |
| receiving | |
| [0130 18:13:34 @multigpu.py:323] [p0574] step: count(1815), step_time 1463.02, mean_step_time 1467.63, it/s 0.68 | |
| [0130 18:13:34 @multigpu.py:323] [p0576] step: count(1825), step_time 1403.33, mean_step_time 1448.42, it/s 0.69 | |
| [0130 18:13:35 @multigpu.py:323] [p0115] step: count(1801), step_time 1427.97, mean_step_time 1462.8, it/s 0.68 | |
| [0130 18:13:35 @multigpu.py:323] [p0574] step: count(1816), step_time 1432.76, mean_step_time 1465.5, it/s 0.68 | |
| [0130 18:13:36 @multigpu.py:323] [p0576] step: count(1826), step_time 1400.48, mean_step_time 1449.68, it/s 0.69 | |
| [0130 18:13:36 @multigpu.py:323] [p0115] step: count(1802), step_time 1482.07, mean_step_time 1465.23, it/s 0.68 | |
| [0130 18:13:36 @multigpu.py:323] [p0574] step: count(1817), step_time 1442.84, mean_step_time 1462.98, it/s 0.68 | |
| [0130 18:13:37 @multigpu.py:323] [p0576] step: count(1827), step_time 1420.96, mean_step_time 1449.19, it/s 0.69 | |
| [0130 18:13:38 @multigpu.py:323] [p0115] step: count(1803), step_time 1445.0, mean_step_time 1467.4, it/s 0.68 | |
| [0130 18:13:38 @multigpu.py:323] [p0574] step: count(1818), step_time 1455.15, mean_step_time 1459.21, it/s 0.69 | |
| [0130 18:13:39 @multigpu.py:323] [p0576] step: count(1828), step_time 1433.15, mean_step_time 1450.23, it/s 0.69 | |
| [0130 18:13:39 @multigpu.py:323] [p0115] step: count(1804), step_time 1464.72, mean_step_time 1468.27, it/s 0.68 | |
| [0130 18:13:39 @multigpu.py:323] [p0574] step: count(1819), step_time 1462.78, mean_step_time 1462.48, it/s 0.68 | |
| [0130 18:13:40 @multigpu.py:323] [p0576] step: count(1829), step_time 1441.63, mean_step_time 1451.05, it/s 0.69 | |
| [0130 18:13:40 @multigpu.py:323] [p0115] step: count(1805), step_time 1383.17, mean_step_time 1466.11, it/s 0.68 | |
| [0130 18:13:41 @multigpu.py:323] [p0574] step: count(1820), step_time 1501.29, mean_step_time 1463.72, it/s 0.68 | |
| [0130 18:13:42 @multigpu.py:323] [p0576] step: count(1830), step_time 1455.53, mean_step_time 1453.4, it/s 0.69 | |
| [0130 18:13:42 @multigpu.py:323] [p0115] step: count(1806), step_time 1462.52, mean_step_time 1466.3, it/s 0.68 | |
| [0130 18:13:42 @multigpu.py:323] [p0574] step: count(1821), step_time 1466.15, mean_step_time 1465.9, it/s 0.68 | |
| [0130 18:13:43 @multigpu.py:323] [p0576] step: count(1831), step_time 1391.55, mean_step_time 1451.94, it/s 0.69 | |
| [0130 18:13:43 @multigpu.py:323] [p0115] step: count(1807), step_time 1404.3, mean_step_time 1460.93, it/s 0.68 | |
| [0130 18:13:44 @multigpu.py:323] [p0574] step: count(1822), step_time 1427.68, mean_step_time 1466.47, it/s 0.68 | |
| 80%|#######9 |798/1000[19:[0130 18:13:45 @multigpu.py:323] [p0574] step: count(1823), step_time 1504.06, mean_step_time 1466.79, it/s 0.68 | |
| [0130 18:13:45 @multigpu.py:323] [p0576] step: count(1832), step_time 2273.5, mean_step_time 1466.84, it/s 0.68 | |
| [0130 18:13:45 @multigpu.py:323] [p0115] step: count(1808), step_time 1955.83, mean_step_time 1466.74, it/s 0.68 | |
| [0130 18:13:47 @multigpu.py:323] [p0574] step: count(1824), step_time 1406.97, mean_step_time 1465.75, it/s 0.68 | |
| [0130 18:13:47 @multigpu.py:323] [p0576] step: count(1833), step_time 1409.99, mean_step_time 1466.83, it/s 0.68 | |
| [0130 18:13:47 @multigpu.py:323] [p0115] step: count(1809), step_time 1458.16, mean_step_time 1468.7, it/s 0.68 | |
| [0130 18:13:48 @multigpu.py:323] [p0576] step: count(1834), step_time 1391.96, mean_step_time 1464.98, it/s 0.68 | |
| [0130 18:13:48 @multigpu.py:323] [p0574] step: count(1825), step_time 1469.67, mean_step_time 1460.06, it/s 0.68 | |
| [0130 18:13:48 @multigpu.py:323] [p0115] step: count(1810), step_time 1479.4, mean_step_time 1470.89, it/s 0.68 | |
| [0130 18:13:50 @multigpu.py:323] [p0576] step: count(1835), step_time 1454.65, mean_step_time 1468.44, it/s 0.68 | |
| [0130 18:13:50 @multigpu.py:323] [p0574] step: count(1826), step_time 1450.49, mean_step_time 1460.03, it/s 0.68 | |
| [0130 18:13:50 @multigpu.py:323] [p0115] step: count(1811), step_time 1414.45, mean_step_time 1469.55, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.730388570494 , 2.4 ##### | |
| [u'online', 2.4] | |
| ##### Sending to neptune: active_workers : 0.73038869222 , 3 ##### | |
| receiving | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.730456432435 , 1.1 ##### | |
| [u'online', 1.1] | |
| receiving | |
| [0130 18:13:51 @multigpu.py:323] [p0576] step: count(1836), step_time 1441.33, mean_step_time 1470.18, it/s 0.68 | |
| [0130 18:13:51 @multigpu.py:323] [p0574] step: count(1827), step_time 1471.22, mean_step_time 1461.26, it/s 0.68 | |
| [0130 18:13:51 @multigpu.py:323] [p0115] step: count(1812), step_time 1472.86, mean_step_time 1470.43, it/s 0.68 | |
| [0130 18:13:52 @multigpu.py:323] [p0576] step: count(1837), step_time 1422.47, mean_step_time 1469.82, it/s 0.68 | |
| [0130 18:13:52 @multigpu.py:323] [p0574] step: count(1828), step_time 1382.45, mean_step_time 1455.62, it/s 0.69 | |
| [0130 18:13:53 @multigpu.py:323] [p0115] step: count(1813), step_time 1386.05, mean_step_time 1468.26, it/s 0.68 | |
| [0130 18:13:54 @multigpu.py:323] [p0576] step: count(1838), step_time 1407.76, mean_step_time 1467.13, it/s 0.68 | |
| [0130 18:13:54 @multigpu.py:323] [p0574] step: count(1829), step_time 1432.93, mean_step_time 1454.46, it/s 0.69 | |
| [0130 18:13:54 @multigpu.py:323] [p0115] step: count(1814), step_time 1424.11, mean_step_time 1468.27, it/s 0.68 | |
| [0130 18:13:55 @multigpu.py:323] [p0576] step: count(1839), step_time 1409.52, mean_step_time 1464.71, it/s 0.68 | |
| [0130 18:13:55 @multigpu.py:323] [p0115] step: count(1815), step_time 1428.61, mean_step_time 1466.82, it/s 0.68 | |
| [0130 18:13:55 @multigpu.py:323] [p0574] step: count(1830), step_time 1454.95, mean_step_time 1453.33, it/s 0.69 | |
| [0130 18:13:57 @multigpu.py:323] [p0576] step: count(1840), step_time 1423.07, mean_step_time 1464.88, it/s 0.68 | |
| [0130 18:13:57 @multigpu.py:323] [p0115] step: count(1816), step_time 1405.78, mean_step_time 1463.24, it/s 0.68 | |
| [0130 18:13:57 @multigpu.py:323] [p0574] step: count(1831), step_time 1447.78, mean_step_time 1454.17, it/s 0.69 | |
| [0130 18:13:58 @multigpu.py:323] [p0576] step: count(1841), step_time 1445.77, mean_step_time 1467.63, it/s 0.68 | |
| [0130 18:13:58 @multigpu.py:323] [p0115] step: count(1817), step_time 1432.9, mean_step_time 1459.67, it/s 0.69 | |
| [0130 18:13:58 @multigpu.py:323] [p0574] step: count(1832), step_time 1483.64, mean_step_time 1456.39, it/s 0.69 | |
| [0130 18:14:00 @multigpu.py:323] [p0576] step: count(1842), step_time 1426.93, mean_step_time 1468.35, it/s 0.68 | |
| [0130 18:14:00 @multigpu.py:323] [p0115] step: count(1818), step_time 1473.33, mean_step_time 1463.03, it/s 0.68 | |
| [0130 18:14:00 @multigpu.py:323] [p0574] step: count(1833), step_time 1489.83, mean_step_time 1453.67, it/s 0.69 | |
| [0130 18:14:01 @multigpu.py:323] [p0576] step: count(1843), step_time 1385.74, mean_step_time 1463.28, it/s 0.68 | |
| [0130 18:14:01 @multigpu.py:323] [p0115] step: count(1819), step_time 1469.2, mean_step_time 1464.32, it/s 0.68 | |
| [0130 18:14:01 @multigpu.py:323] [p0574] step: count(1834), step_time 1398.6, mean_step_time 1452.21, it/s 0.69 | |
| [0130 18:14:02 @multigpu.py:323] [p0576] step: count(1844), step_time 1443.21, mean_step_time 1464.13, it/s 0.68 | |
| [0130 18:14:03 @multigpu.py:323] [p0115] step: count(1820), step_time 1427.86, mean_step_time 1464.92, it/s 0.68 | |
| [0130 18:14:03 @multigpu.py:323] [p0574] step: count(1835), step_time 1441.65, mean_step_time 1451.14, it/s 0.69 | |
| [0130 18:14:04 @multigpu.py:323] [p0576] step: count(1845), step_time 1500.38, mean_step_time 1468.98, it/s 0.68 | |
| [0130 18:14:04 @multigpu.py:323] [p0115] step: count(1821), step_time 1393.41, mean_step_time 1463.19, it/s 0.68 | |
| [0130 18:14:04 @multigpu.py:323] [p0574] step: count(1836), step_time 1437.75, mean_step_time 1451.39, it/s 0.69 | |
| [0130 18:14:05 @multigpu.py:323] [p0576] step: count(1846), step_time 1414.1, mean_step_time 1469.66, it/s 0.68 | |
| [0130 18:14:05 @multigpu.py:323] [p0115] step: count(1822), step_time 1463.59, mean_step_time 1462.26, it/s 0.68 | |
| [0130 18:14:06 @multigpu.py:323] [p0574] step: count(1837), step_time 1428.28, mean_step_time 1450.67, it/s 0.69 | |
| [0130 18:14:07 @multigpu.py:323] [p0576] step: count(1847), step_time 1361.17, mean_step_time 1466.67, it/s 0.68 | |
| [0130 18:14:07 @multigpu.py:323] [p0115] step: count(1823), step_time 1388.61, mean_step_time 1459.44, it/s 0.69 | |
| [0130 18:14:07 @multigpu.py:323] [p0574] step: count(1838), step_time 1460.91, mean_step_time 1450.95, it/s 0.69 | |
| [0130 18:14:08 @multigpu.py:323] [p0576] step: count(1848), step_time 1394.95, mean_step_time 1464.76, it/s 0.68 | |
| sending to address tcp://p0112:61216 | |
| ##### Sending to neptune: online_score : 0.735401380526 , 2.3 ##### | |
| [u'online', 2.3] | |
| receiving | |
| [0130 18:14:08 @multigpu.py:323] [p0115] step: count(1824), step_time 1443.5, mean_step_time 1458.38, it/s 0.69 | |
| [0130 18:14:08 @multigpu.py:323] [p0574] step: count(1839), step_time 1438.72, mean_step_time 1449.75, it/s 0.69 | |
| [0130 18:14:10 @multigpu.py:323] [p0576] step: count(1849), step_time 1470.8, mean_step_time 1466.22, it/s 0.68 | |
| [0130 18:14:10 @multigpu.py:323] [p0115] step: count(1825), step_time 1478.5, mean_step_time 1463.15, it/s 0.68 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment