Skip to content

Instantly share code, notes, and snippets.

@aleksas

aleksas/tensor2tensor_imdb_sentiment_analysis.sh

Last active December 3, 2018 07:59
Show Gist options
  • Save aleksas/a8182011193498d70a09172cd5baa8d3 to your computer and use it in GitHub Desktop.
Save aleksas/a8182011193498d70a09172cd5baa8d3 to your computer and use it in GitHub Desktop.
Download ZIP
Run imdb sentiment analysis
Raw
tensor2tensor_imdb_sentiment_analysis.sh
PROBLEM=sentiment_imdb_characters
MODEL=transformer_encoder
HPARAMS=transformer_base_single_gpu
DATA_DIR=$HOME/t2t_data
TMP_DIR=/tmp/t2t_datagen
TRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS
BATCH_SIZE=512
mkdir -p $DATA_DIR $TMP_DIR
t2t-datagen \
--data_dir=$DATA_DIR \
--tmp_dir=$TMP_DIR \
--problem=$PROBLEM
t2t-trainer \
--data_dir=$DATA_DIR \
--problem=$PROBLEM \
--model=$MODEL \
--hparams_set=$HPARAMS \
--output_dir=$TRAIN_DIR \
--tmp_dir=$TMP_DIR \
--batch_size=$BATCH_SIZE \
--worker_gpu=$WORKER_GPU
Raw
tensor2tensor_imdb_sentiment_analysis_output.py
INFO:tensorflow:Found unparsed command-line arguments. Checking if any start with --hp_ and interpreting those as hparams settings.
WARNING:tensorflow:Found unknown flag: --batch_size=512
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py:165: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.
INFO:tensorflow:schedule=continuous_train_and_eval
INFO:tensorflow:worker_gpu=1
INFO:tensorflow:sync=False
WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.
INFO:tensorflow:datashard_devices: ['gpu:0']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:ps_devices: ['gpu:0']
INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_keep_checkpoint_max': 20, '_task_type': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcafc933110>, '_keep_checkpoint_every_n_hours': 10000, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
optimizer_options {
}
}
, 'use_tpu': False, '_tf_random_seed': None, '_num_worker_replicas': 0, '_task_id': 0, 't2t_device_info': {'num_async_replicas': 1}, '_evaluation_master': '', '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_train_distribute': None, '_is_chief': True, '_tf_config': gpu_options {
per_process_gpu_memory_fraction: 1.0
}
, '_save_checkpoints_steps': 1000, '_environment': 'local', '_master': '', '_model_dir': '/root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base', 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7fcafc933150>, '_save_summary_steps': 100}
WARNING:tensorflow:Estimator's model_fn (<function wrapping_model_fn at 0x7fcafc8c7c80>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:ValidationMonitor only works with --schedule=train_and_evaluate
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 600 secs (eval_spec.throttle_secs) or training is finished.
INFO:tensorflow:Reading data files from /root/t2t_data/sentiment_imdb_characters-train*
INFO:tensorflow:partition: 0 num_data_files: 10
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights.
INFO:tensorflow:Setting T2TModel mode to 'train'
INFO:tensorflow:Using variable initializer: uniform_unit_scaling
INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_258_512.bottom
INFO:tensorflow:Transforming 'targets' with class_label_modality_2_512.targets_bottom
INFO:tensorflow:Building model body
INFO:tensorflow:Transforming body output with class_label_modality_2_512.top
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py:703: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
INFO:tensorflow:Base learning rate: 2.000000
INFO:tensorflow:Trainable Variables Total size: 19052546
INFO:tensorflow:Using optimizer Adam
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2018-12-03 00:32:43.738552: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-12-03 00:32:43.800682: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-12-03 00:32:43.801476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1070 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.81GiB
2018-12-03 00:32:43.878655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-12-03 00:32:43.879407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.835
pciBusID: 0000:07:00.0
totalMemory: 7.93GiB freeMemory: 7.84GiB
2018-12-03 00:32:43.880314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1
2018-12-03 00:32:44.242779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-03 00:32:44.242815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1
2018-12-03 00:32:44.242822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y
2018-12-03 00:32:44.242826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N
2018-12-03 00:32:44.243341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-12-03 00:32:44.315487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7713 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1070, pci bus id: 0000:07:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt-15767
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 15768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:loss = 0.07068637, step = 15767
INFO:tensorflow:global_step/sec: 7.91213
INFO:tensorflow:loss = 0.22584555, step = 15867 (12.640 sec)
INFO:tensorflow:global_step/sec: 9.96565
INFO:tensorflow:loss = 0.10638794, step = 15967 (10.034 sec)
INFO:tensorflow:global_step/sec: 9.9829
INFO:tensorflow:loss = 0.22673553, step = 16067 (10.017 sec)
INFO:tensorflow:global_step/sec: 10.0112
INFO:tensorflow:loss = 0.08368318, step = 16167 (9.989 sec)
INFO:tensorflow:global_step/sec: 10.0217
INFO:tensorflow:loss = 0.215715, step = 16267 (9.978 sec)
INFO:tensorflow:global_step/sec: 9.93835
INFO:tensorflow:loss = 0.19894557, step = 16367 (10.062 sec)
INFO:tensorflow:global_step/sec: 9.94434
INFO:tensorflow:loss = 0.16500695, step = 16467 (10.056 sec)
INFO:tensorflow:global_step/sec: 9.97028
INFO:tensorflow:loss = 0.09865117, step = 16567 (10.030 sec)
INFO:tensorflow:global_step/sec: 9.95326
INFO:tensorflow:loss = 0.26081014, step = 16667 (10.047 sec)
INFO:tensorflow:Saving checkpoints for 16768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:global_step/sec: 9.30917
INFO:tensorflow:loss = 0.29037058, step = 16767 (10.742 sec)
INFO:tensorflow:global_step/sec: 9.94707
INFO:tensorflow:loss = 0.12956437, step = 16867 (10.053 sec)
INFO:tensorflow:global_step/sec: 9.94971
INFO:tensorflow:loss = 0.16136491, step = 16967 (10.051 sec)
INFO:tensorflow:global_step/sec: 9.91337
INFO:tensorflow:loss = 0.12763578, step = 17067 (10.088 sec)
INFO:tensorflow:global_step/sec: 9.97755
INFO:tensorflow:loss = 0.07679598, step = 17167 (10.022 sec)
INFO:tensorflow:global_step/sec: 9.96532
INFO:tensorflow:loss = 0.21029404, step = 17267 (10.035 sec)
INFO:tensorflow:global_step/sec: 9.95546
INFO:tensorflow:loss = 0.14709087, step = 17367 (10.045 sec)
INFO:tensorflow:global_step/sec: 9.98741
INFO:tensorflow:loss = 0.19409588, step = 17467 (10.013 sec)
INFO:tensorflow:global_step/sec: 9.92126
INFO:tensorflow:loss = 0.050190523, step = 17567 (10.079 sec)
INFO:tensorflow:global_step/sec: 9.94801
INFO:tensorflow:loss = 0.081334405, step = 17667 (10.052 sec)
INFO:tensorflow:Saving checkpoints for 17768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:global_step/sec: 9.10562
INFO:tensorflow:loss = 0.038878243, step = 17767 (10.983 sec)
INFO:tensorflow:global_step/sec: 9.92217
INFO:tensorflow:loss = 0.038238686, step = 17867 (10.078 sec)
INFO:tensorflow:global_step/sec: 9.90845
INFO:tensorflow:loss = 0.083561644, step = 17967 (10.092 sec)
INFO:tensorflow:global_step/sec: 9.94442
INFO:tensorflow:loss = 0.109358564, step = 18067 (10.056 sec)
INFO:tensorflow:global_step/sec: 9.91609
INFO:tensorflow:loss = 0.09136684, step = 18167 (10.085 sec)
INFO:tensorflow:global_step/sec: 9.90882
INFO:tensorflow:loss = 0.09995653, step = 18267 (10.095 sec)
INFO:tensorflow:global_step/sec: 9.90739
INFO:tensorflow:loss = 0.12213722, step = 18367 (10.090 sec)
INFO:tensorflow:global_step/sec: 9.9119
INFO:tensorflow:loss = 0.09872891, step = 18467 (10.089 sec)
INFO:tensorflow:global_step/sec: 9.9951
INFO:tensorflow:loss = 0.088943064, step = 18567 (10.005 sec)
INFO:tensorflow:global_step/sec: 9.89823
INFO:tensorflow:loss = 0.20490518, step = 18667 (10.103 sec)
INFO:tensorflow:Saving checkpoints for 18768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:global_step/sec: 9.25783
INFO:tensorflow:loss = 0.08762752, step = 18767 (10.802 sec)
INFO:tensorflow:global_step/sec: 9.90454
INFO:tensorflow:loss = 0.023553476, step = 18867 (10.096 sec)
INFO:tensorflow:global_step/sec: 9.90736
INFO:tensorflow:loss = 0.13714536, step = 18967 (10.093 sec)
INFO:tensorflow:global_step/sec: 9.92325
INFO:tensorflow:loss = 0.0923896, step = 19067 (10.077 sec)
INFO:tensorflow:global_step/sec: 9.96013
INFO:tensorflow:loss = 0.19316633, step = 19167 (10.040 sec)
INFO:tensorflow:global_step/sec: 9.92856
INFO:tensorflow:loss = 0.2654686, step = 19267 (10.072 sec)
INFO:tensorflow:global_step/sec: 9.90508
INFO:tensorflow:loss = 0.12172982, step = 19367 (10.096 sec)
INFO:tensorflow:global_step/sec: 9.88051
INFO:tensorflow:loss = 0.081762746, step = 19467 (10.122 sec)
INFO:tensorflow:global_step/sec: 9.94692
INFO:tensorflow:loss = 0.049419664, step = 19567 (10.053 sec)
INFO:tensorflow:global_step/sec: 9.92598
INFO:tensorflow:loss = 0.25150484, step = 19667 (10.075 sec)
INFO:tensorflow:Saving checkpoints for 19768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:global_step/sec: 9.10779
INFO:tensorflow:loss = 0.04782917, step = 19767 (10.980 sec)
INFO:tensorflow:global_step/sec: 9.89414
INFO:tensorflow:loss = 0.09455118, step = 19867 (10.107 sec)
INFO:tensorflow:global_step/sec: 10.0053
INFO:tensorflow:loss = 0.12819602, step = 19967 (9.995 sec)
INFO:tensorflow:global_step/sec: 9.90394
INFO:tensorflow:loss = 0.10249798, step = 20067 (10.097 sec)
INFO:tensorflow:global_step/sec: 9.90123
INFO:tensorflow:loss = 0.08660681, step = 20167 (10.100 sec)
INFO:tensorflow:global_step/sec: 9.9449
INFO:tensorflow:loss = 0.23311824, step = 20267 (10.055 sec)
INFO:tensorflow:global_step/sec: 9.91927
INFO:tensorflow:loss = 0.08787894, step = 20367 (10.082 sec)
INFO:tensorflow:global_step/sec: 9.92413
INFO:tensorflow:loss = 0.09562593, step = 20467 (10.076 sec)
INFO:tensorflow:global_step/sec: 9.89343
INFO:tensorflow:loss = 0.11091391, step = 20567 (10.108 sec)
INFO:tensorflow:global_step/sec: 9.90341
INFO:tensorflow:loss = 0.037016656, step = 20667 (10.098 sec)
INFO:tensorflow:Saving checkpoints for 20768 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:global_step/sec: 9.24064
INFO:tensorflow:loss = 0.049457937, step = 20767 (10.821 sec)
INFO:tensorflow:global_step/sec: 9.9367
INFO:tensorflow:loss = 0.060066514, step = 20867 (10.063 sec)
INFO:tensorflow:global_step/sec: 9.91909
INFO:tensorflow:loss = 0.35424185, step = 20967 (10.082 sec)
INFO:tensorflow:global_step/sec: 9.91975
INFO:tensorflow:loss = 0.20744851, step = 21067 (10.081 sec)
INFO:tensorflow:global_step/sec: 9.90214
INFO:tensorflow:loss = 0.12818135, step = 21167 (10.099 sec)
INFO:tensorflow:global_step/sec: 9.90567
INFO:tensorflow:loss = 0.11052539, step = 21267 (10.095 sec)
INFO:tensorflow:global_step/sec: 9.99491
INFO:tensorflow:loss = 0.045732643, step = 21367 (10.005 sec)
INFO:tensorflow:global_step/sec: 9.88525
INFO:tensorflow:loss = 0.17503393, step = 21467 (10.116 sec)
INFO:tensorflow:Saving checkpoints for 21558 into /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt.
INFO:tensorflow:Loss for final step: 0.15184914.
INFO:tensorflow:Reading data files from /root/t2t_data/sentiment_imdb_characters-dev*
INFO:tensorflow:partition: 0 num_data_files: 1
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Unsetting shared_embedding_and_softmax_weights.
INFO:tensorflow:Setting T2TModel mode to 'eval'
INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0
INFO:tensorflow:Setting hparams.symbol_dropout to 0.0
INFO:tensorflow:Setting hparams.attention_dropout to 0.0
INFO:tensorflow:Setting hparams.dropout to 0.0
INFO:tensorflow:Setting hparams.relu_dropout to 0.0
INFO:tensorflow:Using variable initializer: uniform_unit_scaling
INFO:tensorflow:Transforming feature 'inputs' with symbol_modality_258_512.bottom
INFO:tensorflow:Transforming 'targets' with class_label_modality_2_512.targets_bottom
INFO:tensorflow:Building model body
INFO:tensorflow:Transforming body output with class_label_modality_2_512.top
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2018-12-03-00:42:47
INFO:tensorflow:Graph was finalized.
2018-12-03 00:42:47.628740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0, 1
2018-12-03 00:42:47.628871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-03 00:42:47.628896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1
2018-12-03 00:42:47.628904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N Y
2018-12-03 00:42:47.628910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: Y N
2018-12-03 00:42:47.629281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-12-03 00:42:47.629472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 7713 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1070, pci bus id: 0000:07:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from /root/t2t_train/sentiment_imdb_characters/transformer_encoder-transformer_base/model.ckpt-21558
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2018-12-03 00:43:03.345590: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.87GiB. Current allocation summary follows.
2018-12-03 00:43:03.345700: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 76, Chunks in use: 76. 19.0KiB allocated for chunks. 19.0KiB in use in bin. 458B client-requested in use in bin.
2018-12-03 00:43:03.345738: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.345768: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 3, Chunks in use: 3. 3.5KiB allocated for chunks. 3.5KiB in use in bin. 3.0KiB client-requested in use in bin.
2018-12-03 00:43:03.345796: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 33, Chunks in use: 32. 66.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2018-12-03 00:43:03.345825: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 1, Chunks in use: 1. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 4.0KiB client-requested in use in bin.
2018-12-03 00:43:03.345880: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 7, Chunks in use: 6. 68.0KiB allocated for chunks. 57.8KiB in use in bin. 48.0KiB client-requested in use in bin.
2018-12-03 00:43:03.346005: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346045: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 17, Chunks in use: 16. 548.0KiB allocated for chunks. 516.0KiB in use in bin. 516.0KiB client-requested in use in bin.
2018-12-03 00:43:03.346084: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 4, Chunks in use: 3. 312.5KiB allocated for chunks. 242.5KiB in use in bin. 213.7KiB client-requested in use in bin.
2018-12-03 00:43:03.346110: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346133: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346159: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 1, Chunks in use: 0. 915.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346188: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 24, Chunks in use: 24. 27.66MiB allocated for chunks. 27.66MiB in use in bin. 24.00MiB client-requested in use in bin.
2018-12-03 00:43:03.346213: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346245: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 14, Chunks in use: 12. 57.89MiB allocated for chunks. 48.01MiB in use in bin. 48.00MiB client-requested in use in bin.
2018-12-03 00:43:03.346270: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346299: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346335: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 4, Chunks in use: 2. 216.69MiB allocated for chunks. 108.34MiB in use in bin. 108.34MiB client-requested in use in bin.
2018-12-03 00:43:03.346364: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346392: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-12-03 00:43:03.346420: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 3, Chunks in use: 2. 7.23GiB allocated for chunks. 5.73GiB in use in bin. 5.73GiB client-requested in use in bin.
2018-12-03 00:43:03.346450: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 2.87GiB was 256.00MiB, Chunk State:
2018-12-03 00:43:03.346487: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 1.50GiB | Requested Size: 512B | in_use: 0, prev: Size: 2.87GiB | Requested Size: 2.87GiB | in_use: 1
2018-12-03 00:43:03.346514: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000000 of size 1280
2018-12-03 00:43:03.346537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000500 of size 1280
2018-12-03 00:43:03.346556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000a00 of size 256
2018-12-03 00:43:03.346586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000b00 of size 256
2018-12-03 00:43:03.346605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0000c00 of size 2048
2018-12-03 00:43:03.346624: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0001400 of size 8192
2018-12-03 00:43:03.346641: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0003400 of size 2048
2018-12-03 00:43:03.346660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0003c00 of size 2048
2018-12-03 00:43:03.346679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0004400 of size 2048
2018-12-03 00:43:03.347043: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0004c00 of size 2048
2018-12-03 00:43:03.347094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0005400 of size 2048
2018-12-03 00:43:03.347118: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0005c00 of size 12288
2018-12-03 00:43:03.347140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0008c00 of size 2048
2018-12-03 00:43:03.347163: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0009400 of size 2048
2018-12-03 00:43:03.347186: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0009c00 of size 4096
2018-12-03 00:43:03.347208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a000ac00 of size 32768
2018-12-03 00:43:03.347229: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0012c00 of size 32768
2018-12-03 00:43:03.347252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a001ac00 of size 1050624
2018-12-03 00:43:03.347274: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011b400 of size 2048
2018-12-03 00:43:03.347296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011bc00 of size 2048
2018-12-03 00:43:03.347317: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011c400 of size 2048
2018-12-03 00:43:03.347339: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011cc00 of size 2048
2018-12-03 00:43:03.347362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a011d400 of size 1081344
2018-12-03 00:43:03.347384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0225400 of size 32768
2018-12-03 00:43:03.347403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a022d400 of size 1955840
2018-12-03 00:43:03.347420: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040ac00 of size 2048
2018-12-03 00:43:03.347438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040b400 of size 2048
2018-12-03 00:43:03.347457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a040bc00 of size 71680
2018-12-03 00:43:03.347483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a041d400 of size 1048576
2018-12-03 00:43:03.347502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a051d400 of size 1048576
2018-12-03 00:43:03.347521: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a061d400 of size 2056192
2018-12-03 00:43:03.347539: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0813400 of size 2048
2018-12-03 00:43:03.347556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0813c00 of size 2048
2018-12-03 00:43:03.347574: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0814400 of size 32768
2018-12-03 00:43:03.347595: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081c400 of size 2048
2018-12-03 00:43:03.347613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081cc00 of size 2048
2018-12-03 00:43:03.347631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081d400 of size 2048
2018-12-03 00:43:03.347651: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081dc00 of size 2048
2018-12-03 00:43:03.347672: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081e400 of size 2048
2018-12-03 00:43:03.347699: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081ec00 of size 2048
2018-12-03 00:43:03.347721: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081f400 of size 2048
2018-12-03 00:43:03.347740: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a081fc00 of size 2048
2018-12-03 00:43:03.347757: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0820400 of size 2048
2018-12-03 00:43:03.347776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0820c00 of size 2048
2018-12-03 00:43:03.347798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0821400 of size 10240
2018-12-03 00:43:03.347820: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823c00 of size 256
2018-12-03 00:43:03.347841: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823d00 of size 256
2018-12-03 00:43:03.347863: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823e00 of size 256
2018-12-03 00:43:03.347884: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0823f00 of size 256
2018-12-03 00:43:03.347905: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824000 of size 256
2018-12-03 00:43:03.347926: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824100 of size 256
2018-12-03 00:43:03.347948: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824200 of size 256
2018-12-03 00:43:03.347970: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824300 of size 256
2018-12-03 00:43:03.347990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824400 of size 2048
2018-12-03 00:43:03.348011: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0824c00 of size 2048
2018-12-03 00:43:03.348033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a0825400 of size 71680
2018-12-03 00:43:03.348054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0836c00 of size 32768
2018-12-03 00:43:03.348077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a083ec00 of size 1990656
2018-12-03 00:43:03.348100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0a24c00 of size 1048576
2018-12-03 00:43:03.348122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0b24c00 of size 1048576
2018-12-03 00:43:03.348143: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c24c00 of size 8192
2018-12-03 00:43:03.348169: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c26c00 of size 8192
2018-12-03 00:43:03.348188: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c28c00 of size 256
2018-12-03 00:43:03.348209: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c28d00 of size 2048
2018-12-03 00:43:03.348266: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c29500 of size 2048
2018-12-03 00:43:03.348287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c29d00 of size 12032
2018-12-03 00:43:03.348306: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a0c2cc00 of size 4194304
2018-12-03 00:43:03.348327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a102cc00 of size 32768
2018-12-03 00:43:03.348348: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1034c00 of size 32768
2018-12-03 00:43:03.348367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a103cc00 of size 1048576
2018-12-03 00:43:03.348388: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a113cc00 of size 32768
2018-12-03 00:43:03.348406: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1144c00 of size 1048576
2018-12-03 00:43:03.348427: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1244c00 of size 1048576
2018-12-03 00:43:03.348449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1344c00 of size 32768
2018-12-03 00:43:03.348468: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a134cc00 of size 1966080
2018-12-03 00:43:03.348487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a152cc00 of size 1081344
2018-12-03 00:43:03.348511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1634c00 of size 32768
2018-12-03 00:43:03.348533: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a163cc00 of size 4194304
2018-12-03 00:43:03.348555: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1a3cc00 of size 4194304
2018-12-03 00:43:03.348575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1e3cc00 of size 32768
2018-12-03 00:43:03.348593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a1e44c00 of size 4194304
2018-12-03 00:43:03.348611: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2244c00 of size 34816
2018-12-03 00:43:03.348629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a224d400 of size 32768
2018-12-03 00:43:03.348653: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2255400 of size 34816
2018-12-03 00:43:03.348675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a225dc00 of size 65536
2018-12-03 00:43:03.348698: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a226dc00 of size 1048576
2018-12-03 00:43:03.348716: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a236dc00 of size 4194304
2018-12-03 00:43:03.348735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a276dc00 of size 6166528
2018-12-03 00:43:03.348754: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d4f400 of size 32768
2018-12-03 00:43:03.348772: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d57400 of size 2048
2018-12-03 00:43:03.348793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d57c00 of size 2048
2018-12-03 00:43:03.348812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58400 of size 256
2018-12-03 00:43:03.348829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58500 of size 256
2018-12-03 00:43:03.348847: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58600 of size 256
2018-12-03 00:43:03.348865: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58700 of size 256
2018-12-03 00:43:03.348886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58800 of size 256
2018-12-03 00:43:03.348906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58900 of size 256
2018-12-03 00:43:03.348927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58a00 of size 256
2018-12-03 00:43:03.348946: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58b00 of size 256
2018-12-03 00:43:03.348964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58c00 of size 256
2018-12-03 00:43:03.348982: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58d00 of size 256
2018-12-03 00:43:03.348999: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58e00 of size 256
2018-12-03 00:43:03.349017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d58f00 of size 256
2018-12-03 00:43:03.349034: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59000 of size 256
2018-12-03 00:43:03.349052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59100 of size 256
2018-12-03 00:43:03.349069: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59200 of size 256
2018-12-03 00:43:03.349087: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59300 of size 256
2018-12-03 00:43:03.349112: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59400 of size 256
2018-12-03 00:43:03.349133: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59500 of size 256
2018-12-03 00:43:03.349154: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59600 of size 256
2018-12-03 00:43:03.349179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59700 of size 256
2018-12-03 00:43:03.349200: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59800 of size 256
2018-12-03 00:43:03.349221: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59900 of size 256
2018-12-03 00:43:03.349243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59a00 of size 256
2018-12-03 00:43:03.349263: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59b00 of size 256
2018-12-03 00:43:03.349284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59c00 of size 256
2018-12-03 00:43:03.349308: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59d00 of size 256
2018-12-03 00:43:03.349327: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59e00 of size 256
2018-12-03 00:43:03.349347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d59f00 of size 256
2018-12-03 00:43:03.349369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a000 of size 256
2018-12-03 00:43:03.349389: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a100 of size 256
2018-12-03 00:43:03.349410: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a200 of size 256
2018-12-03 00:43:03.349432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a300 of size 256
2018-12-03 00:43:03.349453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a400 of size 256
2018-12-03 00:43:03.349476: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a500 of size 256
2018-12-03 00:43:03.349494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a600 of size 256
2018-12-03 00:43:03.349511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a700 of size 256
2018-12-03 00:43:03.349532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a800 of size 256
2018-12-03 00:43:03.349552: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5a900 of size 256
2018-12-03 00:43:03.349573: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5aa00 of size 256
2018-12-03 00:43:03.349593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ab00 of size 256
2018-12-03 00:43:03.349613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ac00 of size 256
2018-12-03 00:43:03.349634: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ad00 of size 256
2018-12-03 00:43:03.349654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ae00 of size 256
2018-12-03 00:43:03.349674: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5af00 of size 256
2018-12-03 00:43:03.349695: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b000 of size 256
2018-12-03 00:43:03.349715: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b100 of size 256
2018-12-03 00:43:03.349735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b200 of size 256
2018-12-03 00:43:03.349756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b300 of size 256
2018-12-03 00:43:03.349777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b400 of size 256
2018-12-03 00:43:03.349797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b500 of size 256
2018-12-03 00:43:03.349817: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b600 of size 256
2018-12-03 00:43:03.349838: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b700 of size 256
2018-12-03 00:43:03.349858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b800 of size 256
2018-12-03 00:43:03.349880: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5b900 of size 1024
2018-12-03 00:43:03.349899: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5bd00 of size 256
2018-12-03 00:43:03.349917: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5be00 of size 256
2018-12-03 00:43:03.349938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5bf00 of size 256
2018-12-03 00:43:03.349958: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c000 of size 256
2018-12-03 00:43:03.349978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c100 of size 256
2018-12-03 00:43:03.349998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c200 of size 256
2018-12-03 00:43:03.350018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c300 of size 256
2018-12-03 00:43:03.350039: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5c400 of size 256
2018-12-03 00:43:03.350059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5c500 of size 2048
2018-12-03 00:43:03.350080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5cd00 of size 256
2018-12-03 00:43:03.350101: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5ce00 of size 256
2018-12-03 00:43:03.350121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5cf00 of size 256
2018-12-03 00:43:03.350141: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5d000 of size 512
2018-12-03 00:43:03.350161: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5d200 of size 256
2018-12-03 00:43:03.350185: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d5d300 of size 10496
2018-12-03 00:43:03.350203: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d5fc00 of size 32768
2018-12-03 00:43:03.350222: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a2d67c00 of size 32768
2018-12-03 00:43:03.350239: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a2d6fc00 of size 4194304
2018-12-03 00:43:03.350260: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a316fc00 of size 1048576
2018-12-03 00:43:03.350280: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a326fc00 of size 1048576
2018-12-03 00:43:03.350301: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a336fc00 of size 4194304
2018-12-03 00:43:03.350321: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a376fc00 of size 1048576
2018-12-03 00:43:03.350342: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a386fc00 of size 1048576
2018-12-03 00:43:03.350362: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a396fc00 of size 4194304
2018-12-03 00:43:03.350382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3d6fc00 of size 1048576
2018-12-03 00:43:03.350404: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3e6fc00 of size 1048576
2018-12-03 00:43:03.350426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a3f6fc00 of size 1048576
2018-12-03 00:43:03.350446: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a406fc00 of size 1048576
2018-12-03 00:43:03.350465: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a416fc00 of size 111104
2018-12-03 00:43:03.350482: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a418ae00 of size 937472
2018-12-03 00:43:03.350503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a426fc00 of size 1048576
2018-12-03 00:43:03.350524: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a436fc00 of size 4202496
2018-12-03 00:43:03.350545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a4771c00 of size 4194304
2018-12-03 00:43:03.350565: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a4b71c00 of size 4194304
2018-12-03 00:43:03.350585: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a4f71c00 of size 4194304
2018-12-03 00:43:03.350605: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a5371c00 of size 4194304
2018-12-03 00:43:03.350625: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8a5771c00 of size 56803328
2018-12-03 00:43:03.350646: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8a8d9dc00 of size 56803328
2018-12-03 00:43:03.350666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fc8ac3c9c00 of size 56803328
2018-12-03 00:43:03.350685: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8af9f5c00 of size 56803328
2018-12-03 00:43:03.350704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc8b3021c00 of size 3077142784
2018-12-03 00:43:03.350724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fc96a6b9500 of size 3077142784
2018-12-03 00:43:03.350745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fca21d50e00 of size 1613780992
2018-12-03 00:43:03.350765: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-12-03 00:43:03.350795: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 76 Chunks of size 256 totalling 19.0KiB
2018-12-03 00:43:03.350800: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1024 totalling 1.0KiB
2018-12-03 00:43:03.350805: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB
2018-12-03 00:43:03.350811: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 32 Chunks of size 2048 totalling 64.0KiB
2018-12-03 00:43:03.350815: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4096 totalling 4.0KiB
2018-12-03 00:43:03.350820: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 8192 totalling 24.0KiB
2018-12-03 00:43:03.350826: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 10240 totalling 10.0KiB
2018-12-03 00:43:03.350831: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12032 totalling 11.8KiB
2018-12-03 00:43:03.350836: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12288 totalling 12.0KiB
2018-12-03 00:43:03.350841: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 14 Chunks of size 32768 totalling 448.0KiB
2018-12-03 00:43:03.350847: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 34816 totalling 68.0KiB
2018-12-03 00:43:03.350852: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 65536 totalling 64.0KiB
2018-12-03 00:43:03.350857: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 71680 totalling 70.0KiB
2018-12-03 00:43:03.350862: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 111104 totalling 108.5KiB
2018-12-03 00:43:03.350867: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 17 Chunks of size 1048576 totalling 17.00MiB
2018-12-03 00:43:03.350873: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1050624 totalling 1.00MiB
2018-12-03 00:43:03.350878: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1081344 totalling 2.06MiB
2018-12-03 00:43:03.350883: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1955840 totalling 1.87MiB
2018-12-03 00:43:03.350892: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1966080 totalling 1.88MiB
2018-12-03 00:43:03.350897: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1990656 totalling 1.90MiB
2018-12-03 00:43:03.350903: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2056192 totalling 1.96MiB
2018-12-03 00:43:03.350908: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 4194304 totalling 44.00MiB
2018-12-03 00:43:03.350912: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4202496 totalling 4.01MiB
2018-12-03 00:43:03.350918: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 56803328 totalling 108.34MiB
2018-12-03 00:43:03.350923: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 3077142784 totalling 5.73GiB
2018-12-03 00:43:03.350928: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 5.91GiB
2018-12-03 00:43:03.350935: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 8086972006
InUse: 6348168448
MaxInUse: 7654274048
NumAllocs: 5697987
MaxAllocSize: 3077142784
2018-12-03 00:43:03.350949: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *********************************************************************************___________________
2018-12-03 00:43:03.350977: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at softmax_op_gpu.cu.cc:157 : Resource exhausted: OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/usr/local/bin/t2t-trainer", line 32, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/usr/local/bin/t2t-trainer", line 28, in main
t2t_trainer.main(argv)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 359, in main
execute_schedule(exp)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 306, in execute_schedule
getattr(exp, FLAGS.schedule)()
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py", line 289, in continuous_train_and_eval
self._eval_spec)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
executor.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
self.run_local()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 657, in run_local
eval_result = evaluator.evaluate_and_export()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 847, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 425, in evaluate
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1117, in _evaluate_model
config=self._session_config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/evaluation.py", line 212, in _evaluate_once
session.run(eval_ops, feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 567, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
run_metadata=run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 971, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax = Softmax[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Reshape)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_3/self_attention/multihead_attention/q/Tensordot/Shape/_775 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1161_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op u'transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax', defined at:
File "/usr/local/bin/t2t-trainer", line 32, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/usr/local/bin/t2t-trainer", line 28, in main
t2t_trainer.main(argv)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 359, in main
execute_schedule(exp)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 306, in execute_schedule
getattr(exp, FLAGS.schedule)()
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py", line 289, in continuous_train_and_eval
self._eval_spec)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
executor.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
self.run_local()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 657, in run_local
eval_result = evaluator.evaluate_and_export()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 847, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 425, in evaluate
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1087, in _evaluate_model
features, labels, model_fn_lib.ModeKeys.EVAL, self.config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1155, in wrapping_model_fn
use_tpu=use_tpu)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 1206, in estimator_model_fn
logits, losses_dict = model(features) # pylint: disable=not-callable
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 717, in __call__
outputs = self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 176, in call
sharded_logits, losses = self.model_fn_sharded(sharded_features)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 231, in model_fn_sharded
sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/expert_utils.py", line 231, in __call__
outputs.append(fns[i](*my_args[i], **my_kwargs[i]))
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 265, in model_fn
body_out = self.body(transformed_features)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 1017, in body
nonpadding=features_to_nonpadding(features, "inputs"))
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py", line 1220, in transformer_encoder
vars_3d=hparams.get("attention_variables_3d"))
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/common_attention.py", line 2944, in multihead_attention
dropout_broadcast_dims=dropout_broadcast_dims)
File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/common_attention.py", line 1490, in dot_product_attention
weights = tf.nn.softmax(logits, name="attention_weights")
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1738, in softmax
return _softmax(logits, gen_nn_ops.softmax, axis, name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1680, in _softmax
output = compute_op(logits)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 7097, in softmax
"Softmax", logits=logits, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[221888,3467] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Softmax = Softmax[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_0/self_attention/multihead_attention/dot_product_attention/Reshape)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: transformer_encoder/parallel_0_5/transformer_encoder/transformer_encoder/body/encoder/layer_3/self_attention/multihead_attention/q/Tensordot/Shape/_775 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1161_...rdot/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment