Full magenta polyphony_rnn_train error output and stack trace, including previous steps of dataset generation

terminal_executions.log

# Creating NoteSequences
INPUT_DIRECTORY="/home/lukas/work/aiexperiments/magenta/jazzdata/"
SEQUENCES_TFRECORD=/home/lukas/work/aiexperiments/magenta/jazzdata.tfrecord
convert_dir_to_note_sequences \
> --input_dir=$INPUT_DIRECTORY \
> --output_file=$SEQUENCES_TFRECORD \
> --recursive
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
INFO:tensorflow:Converting files in '/home/lukas/work/aiexperiments/magenta/jazzdata/'.
INFO:tensorflow:Converted 47 files in '/home/lukas/work/aiexperiments/magenta/jazzdata/'.
INFO:tensorflow:Could not parse 0 files.
INFO:tensorflow:Wrote 47 NoteSequence protos to '/home/lukas/work/aiexperiments/magenta/jazzdata.tfrecord'

# Creating SequenceExamples
polyphony_rnn_create_dataset \                                                                                                         [295/1678]
> --input=$SEQUENCES_TFRECORD \
> --output_dir=/home/lukas/work/aiexperiments/magenta/sequence_examples \
> --eval_ratio=0.10
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
INFO:tensorflow:

Completed.

INFO:tensorflow:Processed 47 inputs total. Produced 36 outputs.
INFO:tensorflow:DAGPipeline_PolyExtractor_eval_polyphonic_track_lengths_in_bars:
  [1,10): 9
INFO:tensorflow:DAGPipeline_PolyExtractor_eval_polyphonic_tracks_discarded_more_than_1_program: 324
INFO:tensorflow:DAGPipeline_PolyExtractor_eval_polyphonic_tracks_discarded_too_long: 0
INFO:tensorflow:DAGPipeline_PolyExtractor_eval_polyphonic_tracks_discarded_too_short: 45
INFO:tensorflow:DAGPipeline_PolyExtractor_training_polyphonic_track_lengths_in_bars:
  [1,10): 9
  [10,20): 18
INFO:tensorflow:DAGPipeline_PolyExtractor_training_polyphonic_tracks_discarded_more_than_1_program: 1044
INFO:tensorflow:DAGPipeline_PolyExtractor_training_polyphonic_tracks_discarded_too_long: 45
INFO:tensorflow:DAGPipeline_PolyExtractor_training_polyphonic_tracks_discarded_too_short: 1278
INFO:tensorflow:DAGPipeline_RandomPartition_eval_poly_tracks_count: 7
INFO:tensorflow:DAGPipeline_RandomPartition_training_poly_tracks_count: 40
INFO:tensorflow:DAGPipeline_TranspositionPipeline_eval_skipped_due_to_range_exceeded: 0
INFO:tensorflow:DAGPipeline_TranspositionPipeline_eval_transpositions_generated: 378
INFO:tensorflow:DAGPipeline_TranspositionPipeline_training_skipped_due_to_range_exceeded: 0
INFO:tensorflow:DAGPipeline_TranspositionPipeline_training_transpositions_generated: 2394

# Training
polyphony_rnn_train \
> --run_dir=/home/lukas/work/aiexperiments/magenta/polyphony_rnn/logdir/jazz1 \
> --sequence_example_file=/home/lukas/work/aiexperiments/magenta/sequence_examples/training_poly_tracks.tfrecord \
> --hparams="{'batch_size':64,'rnn_layer_sizes':[64,64]}" \
> --num_training_steps=20000
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
INFO:tensorflow:hparams = {'rnn_layer_sizes': [256, 256, 256], 'decay_rate': 0.95, 'dropout_keep_prob': 0.5, 'batch_size': 64, 'decay_steps': 1000, 'clip_norm': 5, 'initial_learning_rate': 0.001, 'skip_fi
rst_n_losses': 10}
INFO:tensorflow:Train dir: /home/lukas/work/aiexperiments/magenta/polyphony_rnn/logdir/jazz1/train
WARNING:tensorflow:From /home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py:344 in __init__.: __init__ (from tensorflow.python.training.summary_io
) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.FileWriter. The interface and behavior is the same; this is just a rename.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 950M
major: 5 minor: 0 memoryClockRate (GHz) 0.928
pciBusID 0000:01:00.0
Total memory: 3.95GiB
Free memory: 3.26GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0)
INFO:tensorflow:Starting training loop...
INFO:tensorflow:global_step/sec: 0
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
         [[Node: mul_7/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_na$
e="edge_351_mul_7", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'mul_4', defined at:                                                                                                                                                                 [36/1819]
  File "/home/lukas/libs/miniconda2/envs/magenta/bin/polyphony_rnn_train", line 11, in <module>
    sys.exit(console_entry_point())
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 97, in console_entry_point
    tf.app.run(main)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 78, in main
    mode, config, sequence_example_file)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/shared/events_rnn_graph.py", line 143, in build_graph
    tf.to_float(tf.not_equal(labels_flat, no_event_label)) * mask_flat)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 814, in binary_op_wrapper
    return func(x, y, name=name)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 987, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1613, in mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [27776] vs. [28416]
         [[Node: mul_4 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](ToFloat_2, Reshape_4)]]
         [[Node: mul_7/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_nam
e="edge_351_mul_7", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Traceback (most recent call last):
  File "/home/lukas/libs/miniconda2/envs/magenta/bin/polyphony_rnn_train", line 11, in <module>
    sys.exit(console_entry_point())
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 97, in console_entry_point
    tf.app.run(main)
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "/home/lukas/libs/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/polyphony_rnn/polyphony_rnn_train.py", line 93, in main