ps_0.log

2017-06-28 03:31:29.606714: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_NO_DEVICE
2017-06-28 03:31:29.606826: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: ip-192-168-2-200
2017-06-28 03:31:29.606839: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: ip-192-168-2-200
2017-06-28 03:31:29.606892: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 375.26.0
2017-06-28 03:31:29.606950: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  375.26  Thu Dec  8 18:36:43 PST 2016
GCC version:  gcc version 6.3.0 20170516 (Debian 6.3.0-18) 
"""
2017-06-28 03:31:29.606973: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 375.26.0
2017-06-28 03:31:29.606981: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 375.26.0
2017-06-28 03:31:29.623957: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:5000, 1 -> 10.40.2.201:5000}
2017-06-28 03:31:29.624019: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 10.40.2.200:5001, 1 -> 10.40.2.201:5001}
2017-06-28 03:31:29.624811: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:5000, 1 -> 10.40.2.201:5000}
2017-06-28 03:31:29.624834: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 10.40.2.200:5001, 1 -> 10.40.2.201:5001}
2017-06-28 03:31:29.626857: I tensorflow/contrib/verbs/rdma.cc:99] Start RdmaAdapter: mlx4_0
2017-06-28 03:31:29.635995: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:5000
2017-06-28 03:31:29.636071: I tensorflow/contrib/verbs/rdma_mgr.cc:56] connecting to remote node /job:worker/replica:0/task:0
2017-06-28 03:31:32.134318: I tensorflow/contrib/verbs/rdma_mgr.cc:56] connecting to remote node /job:worker/replica:0/task:1
2017-06-28 03:31:36.173133: I tensorflow/contrib/verbs/rdma.cc:523] channel already connected
2017-06-28 03:31:37.096942: I tensorflow/contrib/verbs/rdma.cc:523] channel already connected
2017-06-28 03:31:37.097007: I tensorflow/contrib/verbs/rdma_mgr.cc:56] connecting to remote node /job:ps/replica:0/task:1
2017-06-28 03:31:37.097871: I tensorflow/contrib/verbs/rdma.cc:523] channel already connected
2017-06-28 03:31:38.505381: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:38.505519: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_208_init_1/NoOp;0:0;125771371772535595
2017-06-28 03:31:38.505942: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.231184: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.231435: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.231510: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.231539: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.231559: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_208_init_1/NoOp;0:0;125771371772535595
2017-06-28 03:31:39.231614: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.232079: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.232118: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.517811: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.517867: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.517906: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:1/cpu:0;edge_156_report_uninitialized_variables/boolean_mask/Gather;0:0;92681776328865098
2017-06-28 03:31:39.517937: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.517968: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.517999: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.518031: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518046: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_97_report_uninitialized_variables/stack;0:0;92681776328865098
2017-06-28 03:31:39.517967: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_101_report_uninitialized_variables/boolean_mask/Where;0:0;92681776328865098
2017-06-28 03:31:39.518233: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.518267: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518671: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.518758: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.518777: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518784: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.518798: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518808: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518817: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.518859: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518871: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.518886: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.518947: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.518961: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518971: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.518979: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519038: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519054: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519061: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519070: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519118: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519132: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519140: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519149: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519197: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519217: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519224: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519235: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519282: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519299: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519307: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519315: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519361: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519374: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519381: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519390: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519437: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519451: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519458: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519467: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519517: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.519531: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519537: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519545: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519556: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519603: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.519616: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519628: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519638: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519644: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519691: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519703: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.519713: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519722: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519731: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.519741: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519753: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519763: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519774: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.519806: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519835: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.519914: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.519936: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519948: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.519960: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520076: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520096: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.520111: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520124: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520136: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520143: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520152: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520165: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520255: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520274: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520285: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520295: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520302: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520364: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520386: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520398: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520461: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520476: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520483: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520536: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.520551: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520559: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520612: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520625: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520633: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.520660: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520672: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520719: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520774: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.520788: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520795: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.520805: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520814: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520821: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.520832: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520852: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.520866: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.520917: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.520934: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520943: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.520953: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520960: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.520967: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.520975: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.521025: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521039: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521048: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521057: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521064: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521073: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521083: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521092: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:39.521150: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521161: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521170: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521178: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.521224: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521239: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521250: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521257: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521266: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521273: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521280: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521290: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521300: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521308: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521317: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521326: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521335: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521345: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521355: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521365: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521376: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521389: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521398: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521409: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521418: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521425: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521434: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521444: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521454: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521462: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521471: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521481: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521490: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521500: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521510: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521519: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521529: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521539: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521546: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521566: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521576: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521586: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521605: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521616: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521635: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521647: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521656: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521666: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521676: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521701: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521719: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521739: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521757: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521768: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521778: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521796: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.521814: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521849: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.521860: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521869: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521886: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521905: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521927: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521939: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.521962: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.521987: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522008: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522024: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.522034: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522211: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522229: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522241: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522252: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.522267: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522282: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522291: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522300: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522317: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522333: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522355: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.522388: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_97_report_uninitialized_variables/stack;0:0;92681776328865098
2017-06-28 03:31:39.522448: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.522691: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.522708: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783086: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.783213: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783238: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.783253: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783474: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.783591: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.783608: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783802: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.783819: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783886: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.783914: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.783921: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_101_report_uninitialized_variables/boolean_mask/Where;0:0;92681776328865098
2017-06-28 03:31:39.784003: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.784135: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.784156: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.784386: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.784475: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.784488: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.784548: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.784698: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.784788: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.784802: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.784836: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:39.784915: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.785171: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:39.785187: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:39.785201: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:1/cpu:0;edge_156_report_uninitialized_variables/boolean_mask/Gather;0:0;92681776328865098
2017-06-28 03:31:39.785278: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:39.785423: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:39.785439: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.036683: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.036755: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.036872: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_97_report_uninitialized_variables/stack;0:0;88118325191660910
2017-06-28 03:31:45.036873: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037072: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037113: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037126: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037137: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037146: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037161: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037190: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037191: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/cpu:0;edge_156_report_uninitialized_variables/boolean_mask/Gather;0:0;88118325191660910
2017-06-28 03:31:45.037205: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037270: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037282: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037290: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037300: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037312: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037321: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037333: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037345: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037368: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037387: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037402: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037413: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037426: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037437: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037456: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037070: I tensorflow/contrib/verbs/rdma.cc:671] try to send tensor: /job:ps/replica:0/task:0/cpu:0;d89e36729128ca28;/job:worker/replica:0/task:0/gpu:0;edge_101_report_uninitialized_variables/boolean_mask/Where;0:0;88118325191660910
2017-06-28 03:31:45.037480: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037494: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037513: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037524: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037622: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037634: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037643: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037652: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037661: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037672: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037682: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037690: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037699: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037709: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037717: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037729: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037737: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037746: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037756: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037775: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037789: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037817: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037826: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037835: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037846: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037856: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037874: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037898: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037909: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037920: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037940: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.037950: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.037969: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.037981: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.038078: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038087: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.038096: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038106: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.038385: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.038408: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.038427: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038438: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038446: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038457: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.038653: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.038671: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_REQUEST
2017-06-28 03:31:45.038683: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038692: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038699: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038707: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.038871: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.038898: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.038911: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038922: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038929: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.038937: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.039000: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039017: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.039028: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039037: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039044: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039051: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.039138: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039154: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.039164: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039171: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039179: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.039192: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039200: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_REQUEST
2017-06-28 03:31:45.039336: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039355: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.039366: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039374: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039384: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.039393: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039401: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.039411: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039422: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039431: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.039442: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039465: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039484: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039506: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039517: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039526: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039535: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.039545: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039562: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039576: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039595: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039618: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039628: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.039638: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039658: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039680: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039700: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039736: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039926: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.039941: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.039951: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.039961: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_RESPONSE
2017-06-28 03:31:45.040007: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.040177: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.040191: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.040201: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.040212: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.040297: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.040388: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.040920: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.040940: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.040950: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.040963: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_TENSOR_WRITE
2017-06-28 03:31:45.040975: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.040985: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.041083: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK
2017-06-28 03:31:45.041105: I tensorflow/contrib/verbs/rdma.cc:225] sent RDMA message: RDMA_MESSAGE_BUFFER_IDLE
2017-06-28 03:31:45.041083: F tensorflow/contrib/verbs/rdma.cc:786] Check failed: (buffer_size == size_ && rm.data_type_ != DT_STRING) || (buffer_size <= size_ && rm.data_type_ == DT_STRING) tensor and buffer size do not agree! buffer_size = 591 requested tensor size = 583Tensor<type: int64 shape: [0,1] values: >
TensorFlow:  1.2
Model:       vgg16
Mode:        training
Batch size:  64 global
             64 per device
Devices:     ['/job:worker/task:0/gpu:0']
Data format: NCHW
Optimizer:   sgd
Variables:   parameter_server
Sync:        True
==========
Running parameter server 0
2017-06-28 03:31:45.041179: I tensorflow/contrib/verbs/rdma.cc:143] recv RDMA message: RDMA_MESSAGE_ACK