log.txt

11:21:40,746 INFO  ~ Loaded [JCublasBackend] backend
11:21:41,201 INFO  ~ Number of threads used for NativeOps: 32
11:21:41,334 INFO  ~ Backend used: [CUDA]; OS: [Linux]
11:21:41,334 INFO  ~ Cores: [8]; Memory: [23.1GB];
11:21:41,334 INFO  ~ Blas vendor: [CUBLAS]
11:21:41,335 INFO  ~ Device name: [GeForce GTX 1070]; CC: [6.1]; Total/free memory: [8507752448]
Building the dictionary...
Dictionary is ready, size is 40035
Total dictionary size is 40035. Processing the dataset...
Done. Corpus size is 281312
Creating a new network...
Number of parameters: 223265379
Epoch 1
11:21:45,957 INFO  ~ Allocating [LOOP_ITER] workspace of 0 bytes...
11:21:46,117 INFO  ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements
11:21:46,125 INFO  ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements
11:21:46,126 INFO  ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements
11:21:46,138 WARN  ~ Workspace initialization OVER_TIME was selected, but number of cycles isn't positive value!
11:21:46,138 INFO  ~ Allocating [LOOP_FF] workspace of 0 bytes...
11:21:46,138 INFO  ~ Allocating [LOOP_FF] workspace of 0 bytes...
11:21:46,138 INFO  ~ [LOOP_FF] spilled DEVICE array of 1152 bytes, capacity of 288 elements
CUDA error at /home/rkfg/soft/svn-soft/libnd4j/blas/cuda/NativeOps.cu:4801 code=11(cudaErrorInvalidValue) "result" 
Failed on [1110846756352] -> [1110844720640], size: [1152], direction: [2], result: [11]
11:21:46,140 INFO  ~ Allocating [LOOP_ITER] workspace of 3456 bytes...
[WARNING] 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: MemcpyAsync failed: AllocationShape(offset=0, length=288, stride=1, elementSize=4, dataType=FLOAT)
	at org.nd4j.jita.flow.impl.SynchronousFlowController.synchronizeToHost(SynchronousFlowController.java:63)
	at org.nd4j.jita.flow.impl.GridFlowController.synchronizeToHost(GridFlowController.java:36)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.synchronizeThreadDevice(CudaZeroHandler.java:1227)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.synchronizeHostData(AtomicAllocator.java:321)
	at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.getFloat(BaseCudaDataBuffer.java:965)
	at org.nd4j.linalg.jcublas.buffer.CudaFloatDataBuffer.getDouble(CudaFloatDataBuffer.java:237)
	at org.nd4j.linalg.api.shape.Shape.getDouble(Shape.java:163)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.getDouble(BaseNDArray.java:1697)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.getInt(BaseNDArray.java:1666)
	at org.deeplearning4j.nn.layers.feedforward.embedding.EmbeddingLayer.preOutput(EmbeddingLayer.java:93)
	at org.deeplearning4j.nn.layers.feedforward.embedding.EmbeddingLayer.activate(EmbeddingLayer.java:113)
	at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:103)
	at org.deeplearning4j.nn.graph.ComputationGraph.feedForward(ComputationGraph.java:1180)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1053)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:160)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:56)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:59)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:853)
	at dlchat.EncoderDecoderLSTM.train(EncoderDecoderLSTM.java:248)
	at dlchat.EncoderDecoderLSTM.run(EncoderDecoderLSTM.java:201)
	at dlchat.EncoderDecoderLSTM.main(EncoderDecoderLSTM.java:149)
	... 6 more
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[WARNING] thread Thread[ResourceGC thread 0,5,dlchat.EncoderDecoderLSTM] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[ResourceGC thread 0,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[ResourceGC thread 1,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[ResourceGC thread 2,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[ResourceGC thread 3,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[UniGC thread 1,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[NativeRandomDeallocator thread 0,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] thread Thread[Thread-22,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption
[WARNING] NOTE: 7 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=dlchat.EncoderDecoderLSTM,maxpri=10]
java.lang.IllegalThreadStateException
	at java.lang.ThreadGroup.destroy(ThreadGroup.java:778)
	at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.761 s
[INFO] Finished at: 2017-03-30T11:22:01+03:00
[INFO] Final Memory: 77M/2078M
[INFO] ------------------------------------------------------------------------
[WARNING] 
java.lang.RuntimeException: java.lang.InterruptedException
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160)
	at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480)
	at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130)
	at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443)
	at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125)
	... 6 more
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (default-cli) on project dlchat: An exception occured while executing the Java class. null: InvocationTargetException: MemcpyAsync failed: AllocationShape(offset=0, length=288, stride=1, elementSize=4, dataType=FLOAT) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException