Created
May 7, 2017 18:01
-
-
Save rizar/4801481258a7f39edf0b4d3e3526f622 to your computer and use it in GitHub Desktop.
Log of my code when run with float16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Using cuDNN version 5105 on context None | |
| Preallocating 14677/16308 Mb (0.900000) on cuda | |
| Mapped name None to device cuda: Tesla P100-SXM2-16GB (0000:0A:00.0) | |
| I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally | |
| I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally | |
| I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally | |
| I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally | |
| I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally | |
| /Tmp/lisa/os_v5/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py:913: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter. | |
| warnings.warn(self.msg_depr % (key, alt_key)) | |
| 2017-05-07 13:57:59,353: dictlearn.main: INFO: Continue an existing job | |
| 2017-05-07 13:58:02,240: /u/bahdanau/Dist/dict_based_learning/dictlearn/lookup.pyc: INFO: Restricting def vocab to 3000 | |
| 2017-05-07 13:58:02,383: root: DEBUG: Model created | |
| 2017-05-07 13:58:02,383: root: DEBUG: Embeddings loaded | |
| 2017-05-07 13:58:02,768: root: INFO: Cost parameters | |
| ['/extractiveqamodel/begin_readout/linear_0.W (400, 1) trained', | |
| '/extractiveqamodel/begin_readout/linear_0.b (1,) trained', | |
| '/extractiveqamodel/bidir/backward.W_cell_to_forget (200,) trained', | |
| '/extractiveqamodel/bidir/backward.W_cell_to_in (200,) trained', | |
| '/extractiveqamodel/bidir/backward.W_cell_to_out (200,) trained', | |
| '/extractiveqamodel/bidir/backward.W_state (200, 800) trained', | |
| '/extractiveqamodel/bidir/backward.initial_cells (200,) trained', | |
| '/extractiveqamodel/bidir/backward.initial_state (200,) trained', | |
| '/extractiveqamodel/bidir/forward.W_cell_to_forget (200,) trained', | |
| '/extractiveqamodel/bidir/forward.W_cell_to_in (200,) trained', | |
| '/extractiveqamodel/bidir/forward.W_cell_to_out (200,) trained', | |
| '/extractiveqamodel/bidir/forward.W_state (200, 800) trained', | |
| '/extractiveqamodel/bidir/forward.initial_cells (200,) trained', | |
| '/extractiveqamodel/bidir/forward.initial_state (200,) trained', | |
| '/extractiveqamodel/bidir_fork.W (600, 800) trained', | |
| '/extractiveqamodel/bidir_fork.b (800,) trained', | |
| '/extractiveqamodel/encoder_fork.W (300, 800) trained', | |
| '/extractiveqamodel/encoder_fork.b (800,) trained', | |
| '/extractiveqamodel/encoder_rnn.W_cell_to_forget (200,) trained', | |
| '/extractiveqamodel/encoder_rnn.W_cell_to_in (200,) trained', | |
| '/extractiveqamodel/encoder_rnn.W_cell_to_out (200,) trained', | |
| '/extractiveqamodel/encoder_rnn.W_state (200, 800) trained', | |
| '/extractiveqamodel/encoder_rnn.initial_cells (200,) trained', | |
| '/extractiveqamodel/encoder_rnn.initial_state (200,) trained', | |
| '/extractiveqamodel/end_readout/linear_0.W (400, 1) trained', | |
| '/extractiveqamodel/end_readout/linear_0.b (1,) trained', | |
| '/extractiveqamodel/lookuptable.W (3000, 300) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_fork.W (300, 800) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_fork.b (800,) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.W_cell_to_forget (200,) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.W_cell_to_in (200,) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.W_cell_to_out (200,) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.W_state (200, 800) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.initial_cells (200,) trained', | |
| '/extractiveqamodel/lstmreaddefinitions/def_rnn.initial_state (200,) trained', | |
| '/extractiveqamodel/meanpoolcombiner/state_transform.W (200, 300) trained', | |
| '/extractiveqamodel/meanpoolcombiner/state_transform.b (300,) trained', | |
| '/extractiveqamodel/question_transform.W (200, 200) trained', | |
| '/extractiveqamodel/question_transform.b (200,) trained'] | |
| 2017-05-07 13:58:02,770: blocks.algorithms: INFO: Taking the cost gradient | |
| 2017-05-07 13:58:03,279: blocks.algorithms: INFO: The cost gradient computation graph is built | |
| 2017-05-07 13:58:03,292: blocks.algorithms: DEBUG: Computing parameter steps... | |
| 2017-05-07 13:58:05,195: blocks.monitoring.evaluators: DEBUG: Compiling initialization and readout functions | |
| 2017-05-07 13:58:05,206: blocks.monitoring.evaluators: DEBUG: Initialization and readout functions compiled | |
| 2017-05-07 13:58:05,228: blocks.monitoring.evaluators: DEBUG: variable to evaluate: length | |
| 2017-05-07 13:58:05,228: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for length | |
| 2017-05-07 13:58:05,233: blocks.monitoring.evaluators: DEBUG: variable to evaluate: batch_size | |
| 2017-05-07 13:58:05,233: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for batch_size | |
| 2017-05-07 13:58:05,235: blocks.monitoring.evaluators: DEBUG: variable to evaluate: mean_cost | |
| 2017-05-07 13:58:05,237: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for mean_cost | |
| 2017-05-07 13:58:05,239: blocks.monitoring.evaluators: DEBUG: variable to evaluate: exact_match_ratio | |
| 2017-05-07 13:58:05,239: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for exact_match_ratio | |
| 2017-05-07 13:58:05,242: blocks.monitoring.evaluators: DEBUG: variable to evaluate: extractiveqamodel__encode_context_unk_ratio | |
| 2017-05-07 13:58:05,242: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for extractiveqamodel__encode_context_unk_ratio | |
| 2017-05-07 13:58:05,244: blocks.monitoring.evaluators: DEBUG: variable to evaluate: lstmreaddefinitions_apply_def_unk_ratio | |
| 2017-05-07 13:58:05,244: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for lstmreaddefinitions_apply_def_unk_ratio | |
| 2017-05-07 13:58:05,247: blocks.monitoring.evaluators: DEBUG: variable to evaluate: num_definitions | |
| 2017-05-07 13:58:05,247: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for num_definitions | |
| 2017-05-07 13:58:05,249: blocks.monitoring.evaluators: DEBUG: variable to evaluate: max_definition_length | |
| 2017-05-07 13:58:05,250: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for max_definition_length | |
| 2017-05-07 13:58:05,253: blocks.monitoring.evaluators: DEBUG: variable to evaluate: total_gradient_norm | |
| 2017-05-07 13:58:05,255: blocks.monitoring.evaluators: DEBUG: Using the default (average over minibatches) aggregation scheme for total_gradient_norm | |
| 2017-05-07 13:58:05,257: blocks.monitoring.evaluators: DEBUG: Compiling initialization and readout functions | |
| 2017-05-07 13:58:05,504: blocks.monitoring.evaluators: DEBUG: Initialization and readout functions compiled | |
| 2017-05-07 13:58:05,657: blocks.monitoring.evaluators: DEBUG: variable to evaluate: length | |
| 2017-05-07 13:58:05,660: blocks.monitoring.evaluators: DEBUG: variable to evaluate: batch_size | |
| 2017-05-07 13:58:05,662: blocks.monitoring.evaluators: DEBUG: variable to evaluate: mean_cost | |
| 2017-05-07 13:58:05,665: blocks.monitoring.evaluators: DEBUG: variable to evaluate: exact_match_ratio | |
| 2017-05-07 13:58:05,667: blocks.monitoring.evaluators: DEBUG: variable to evaluate: extractiveqamodel__encode_context_unk_ratio | |
| 2017-05-07 13:58:05,670: blocks.monitoring.evaluators: DEBUG: variable to evaluate: lstmreaddefinitions_apply_def_unk_ratio | |
| 2017-05-07 13:58:05,672: blocks.monitoring.evaluators: DEBUG: variable to evaluate: num_definitions | |
| 2017-05-07 13:58:05,675: blocks.monitoring.evaluators: DEBUG: variable to evaluate: max_definition_length | |
| 2017-05-07 13:58:05,677: blocks.monitoring.evaluators: DEBUG: Compiling initialization and readout functions | |
| 2017-05-07 13:58:05,744: blocks.monitoring.evaluators: DEBUG: Initialization and readout functions compiled | |
| 2017-05-07 13:58:05,756: blocks.main_loop: INFO: Entered the main loop | |
| 2017-05-07 13:58:05,756: blocks.extensions.saveload: WARNING: No dump found | |
| 2017-05-07 13:58:05,757: blocks.extensions.monitoring: INFO: Monitoring on auxiliary data started | |
| 2017-05-07 13:58:16,027: blocks.main_loop: ERROR: Error occured during training. | |
| Blocks will attempt to run `on_error` extensions, potentially saving data, before exiting and reraising the error. Note that the usual `after_training` extensions will *not* be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately. | |
| {'batch_size': 128, | |
| 'batch_size_valid': 128, | |
| 'coattention': True, | |
| 'compose_type': 'transform_and_sum', | |
| 'data_path': 'squad/squad_from_scratch', | |
| 'def_reader': 'LSTMReadDefinitions', | |
| 'def_word_gating': 'none', | |
| 'dict_path': 'squad/squad_from_scratch/dict4.json', | |
| 'dict_vocab_path': '', | |
| 'dim': 200, | |
| 'emb_dim': 300, | |
| 'embedding_path': '', | |
| 'exclude_top_k': 3000, | |
| 'grad_clip_threshold': 50.0, | |
| 'layout': 'squad', | |
| 'learning_rate': 0.001, | |
| 'max_def_length': 30, | |
| 'max_length': 100, | |
| 'momentum': 0.9, | |
| 'mon_freq_train': 100, | |
| 'mon_freq_valid': 1000, | |
| 'monitor_parameters': False, | |
| 'n_batches': 0, | |
| 'num_input_words': 3000, | |
| 'reuse_word_embeddings': True, | |
| 'save_freq_batches': 1000, | |
| 'save_freq_epochs': 1} | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{true_div,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{Cast{float16}} due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for DeepCopyOp due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for Elemwise{add,no_inplace} due to unsupported float16 | |
| Disabling C code for BatchedDot due to unsupported float16 | |
| Disabling C code for BatchedDot due to unsupported float16 | |
| Disabling C code for BatchedDot due to unsupported float16 | |
| Disabling C code for GpuMaxAndArgmax{axis=(1,)} due to unsupported float16 | |
| Disabling C code for GpuMaxAndArgmax{axis=(1,)} due to unsupported float16 | |
| Traceback (most recent call last): | |
| File "/u/bahdanau/Dist/dict_based_learning/bin/train_extractive_qa.py", line 8, in <module> | |
| main(qa_config_registry, train_extractive_qa) | |
| File "/u/bahdanau/Dist/dict_based_learning/dictlearn/main.py", line 66, in main | |
| call_training_func)() | |
| File "/u/bahdanau/Dist/dict_based_learning/dictlearn/util.py", line 230, in func_wrapper | |
| func(*args, **kwargs) | |
| File "/u/bahdanau/Dist/dict_based_learning/dictlearn/main.py", line 62, in call_training_func | |
| args.params, args.fast_start, args.fuel_server) | |
| File "/u/bahdanau/Dist/dict_based_learning/dictlearn/extractive_qa_training.py", line 234, in train_extractive_qa | |
| main_loop.run() | |
| File "/u/bahdanau/Dist/blocks/blocks/main_loop.py", line 198, in run | |
| reraise_as(e) | |
| File "/u/bahdanau/Dist/blocks/blocks/utils/__init__.py", line 258, in reraise_as | |
| six.reraise(type(new_exc), new_exc, orig_exc_traceback) | |
| File "/u/bahdanau/Dist/blocks/blocks/main_loop.py", line 171, in run | |
| self._run_extensions('before_training') | |
| File "/u/bahdanau/Dist/blocks/blocks/main_loop.py", line 264, in _run_extensions | |
| extension.dispatch(CallbackName(method_name), *args) | |
| File "/u/bahdanau/Dist/blocks/blocks/extensions/__init__.py", line 346, in dispatch | |
| self.do(callback_invoked, *(from_main_loop + tuple(arguments))) | |
| File "/u/bahdanau/Dist/blocks/blocks/extensions/monitoring.py", line 95, in do | |
| value_dict = self._evaluator.evaluate(self.data_stream) | |
| File "/u/bahdanau/Dist/blocks/blocks/monitoring/evaluators.py", line 338, in evaluate | |
| self.process_batch(batch) | |
| File "/u/bahdanau/Dist/blocks/blocks/monitoring/evaluators.py", line 309, in process_batch | |
| numerical_values = self._aggregate_fun(**batch) | |
| File "/u/bahdanau/Dist/theano/theano/compile/function_module.py", line 898, in __call__ | |
| storage_map=getattr(self.fn, 'storage_map', None)) | |
| File "/u/bahdanau/Dist/theano/theano/gof/link.py", line 325, in raise_with_op | |
| reraise(exc_type, exc_value, exc_trace) | |
| File "/u/bahdanau/Dist/theano/theano/compile/function_module.py", line 884, in __call__ | |
| self.fn() if output_subset is None else\ | |
| File "/u/bahdanau/Dist/theano/theano/gof/op.py", line 897, in rval | |
| r = p(n, [x[0] for x in i], o, params) | |
| File "/u/bahdanau/Dist/theano/theano/gof/op.py", line 778, in perform | |
| "Did you used Theano flags mode=FAST_COMPILE?" | |
| theano.gof.utils.MethodNotDefined: ('perform', <class 'theano.gpuarray.reduction.GpuMaxAndArgmax'>, 'GpuMaxAndArgmax', 'Did you used Theano flags mode=FAST_COMPILE? You can use optimizer=fast_compile instead.') | |
| Apply node that caused the error: GpuMaxAndArgmax{axis=(1,)}(GpuElemwise{Composite{((i0 * i1) - i2)}}[(0, 2)]<gpuarray>.0) | |
| Toposort index: 542 | |
| Inputs types: [GpuArrayType<None>(float16, matrix)] | |
| Inputs shapes: [(128, 150)] | |
| Inputs strides: [(300, 2)] | |
| Inputs values: ['not shown'] | |
| Outputs clients: [[InplaceGpuDimShuffle{0,x}(GpuMaxAndArgmax{axis=(1,)}.0)], [GpuElemwise{Composite{(EQ(i0, i1) * EQ(i2, i3))}}[]<gpuarray>(GpuMaxAndArgmax{axis=(1,)}.1, GpuFromHost<None>.0, GpuMaxAndArgmax{axis=(1,)}.1, GpuFromHost<None>.0)]] | |
| HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'. | |
| HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node. | |
| Original exception: | |
| MethodNotDefined: ('perform', <class 'theano.gpuarray.reduction.GpuMaxAndArgmax'>, 'GpuMaxAndArgmax', 'Did you used Theano flags mode=FAST_COMPILE? You can use optimizer=fast_compile instead.') | |
| Apply node that caused the error: GpuMaxAndArgmax{axis=(1,)}(GpuElemwise{Composite{((i0 * i1) - i2)}}[(0, 2)]<gpuarray>.0) | |
| Toposort index: 542 | |
| Inputs types: [GpuArrayType<None>(float16, matrix)] | |
| Inputs shapes: [(128, 150)] | |
| Inputs strides: [(300, 2)] | |
| Inputs values: ['not shown'] | |
| Outputs clients: [[InplaceGpuDimShuffle{0,x}(GpuMaxAndArgmax{axis=(1,)}.0)], [GpuElemwise{Composite{(EQ(i0, i1) * EQ(i2, i3))}}[]<gpuarray>(GpuMaxAndArgmax{axis=(1,)}.1, GpuFromHost<None>.0, GpuMaxAndArgmax{axis=(1,)}.1, GpuFromHost<None>.0)]] | |
| HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'. | |
| HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment