Skip to content

Instantly share code, notes, and snippets.

@toyama0919

toyama0919/embulk_error.md

Last active July 29, 2016 00:52
Show Gist options
  • Save toyama0919/53b5b9c20501c4f914450786d16ee803 to your computer and use it in GitHub Desktop.
Save toyama0919/53b5b9c20501c4f914450786d16ee803 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
embulk_error.md

発生するエラーはこれ。

2016-07-29 09:34:28.957 +0900: Embulk v0.8.11
2016-07-29 09:34:30.302 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-s3 (0.2.8)
2016-07-29 09:34:31.190 +0900 [INFO] (0001:transaction): Loaded plugin embulk/output/elasticsearch_ruby from a load path
2016-07-29 09:34:32.575 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 4 = input tasks 1 * 4
2016-07-29 09:34:32.601 +0900 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
org.embulk.exec.PooledBufferAllocator$BufferDoubleReleasedException: Detected double release() call of a buffer
  at org.embulk.exec.PooledBufferAllocator$NettyByteBufBuffer.release(PooledBufferAllocator.java:55)
  at org.embulk.spi.Page.release(Page.java:63)
  at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:404)
  at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:319)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.embulk.exec.PooledBufferAllocator$BufferReleasedBeforeAt
  at org.embulk.exec.PooledBufferAllocator$NettyByteBufBuffer.release(PooledBufferAllocator.java:60)
  at org.embulk.spi.PageReader.close(PageReader.java:166)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
  at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
  at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35)
  at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:293)
  at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
  at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:332)
  at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
  at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)
  at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
  at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:423)
  at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
  at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
  at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
  at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
  at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)
  at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
  at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
  at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
  at Embulk$$OutputPlugin$$JavaAdapter$$OutputAdapter_2045315146.add(Embulk$$OutputPlugin$$JavaAdapter$$OutputAdapter_2045315146.gen:13)
  at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:394)
  ... 5 more
2016-07-29 09:34:34.460 +0900 [INFO] (embulk-output-executor-3): bulk: 1000 success.
2016-07-29 09:34:34.537 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:34:34.646 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:34:35.171 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:34:35.282 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:34:35.489 +0900 [INFO] (embulk-output-executor-3): bulk: 1000 success.
2016-07-29 09:34:35.831 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:34:36.044 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:34:36.273 +0900 [INFO] (embulk-output-executor-3): bulk: 1000 success.
2016-07-29 09:34:36.553 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:34:36.840 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:34:37.100 +0900 [INFO] (embulk-output-executor-3): bulk: 1000 success.
2016-07-29 09:34:37.299 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:34:37.738 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.

.
.
.
(省略)

org.embulk.exec.PartialExecutionException: org.jruby.exceptions.RaiseException: (NameError) uninitialized constant MultiJson::Adapters
  at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)
  at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:572)
  at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
  at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:374)
  at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:370)
  at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
  at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:370)
  at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
  at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
  at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
  at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
  at RUBY.run(uri:classloader:/embulk/runner.rb:84)
  at RUBY.run(uri:classloader:/embulk/command/embulk_run.rb:302)
  at RUBY.<top>(uri:classloader:/embulk/command/embulk_main.rb:2)
  at org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:849)
  at org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2986)
  at org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:951)
  at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:944)
  at Users.toyama_minus_h.bin.embulk.embulk.command.embulk_bundle.<top>(file:/Users/toyama-h/bin/embulk!/embulk/command/embulk_bundle.rb:30)
  at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
  at org.jruby.Ruby.runScript(org/jruby/Ruby.java:833)
  at org.jruby.Ruby.runScript(org/jruby/Ruby.java:825)
  at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:760)
  at org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:579)
  at org.jruby.Main.doRunFromMain(org/jruby/Main.java:425)
  at org.jruby.Main.internalRun(org/jruby/Main.java:313)
  at org.jruby.Main.run(org/jruby/Main.java:242)
  at org.jruby.Main.main(org/jruby/Main.java:204)
  at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
Caused by: org.jruby.exceptions.RaiseException: (NameError) uninitialized constant MultiJson::Adapters
  at org.jruby.RubyModule.const_missing(org/jruby/RubyModule.java:3309)
  at RUBY.load_adapter_from_string_name(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:159)
  at RUBY.load_adapter(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:99)
  at RUBY.load_adapter(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:103)
  at RUBY.use(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:91)
  at RUBY.adapter(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:73)
  at RUBY.current_adapter(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:133)
  at RUBY.dump(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/multi_json-1.12.1/lib/multi_json.rb:139)
  at RUBY.block in __bulkify(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-2.0.0/lib/elasticsearch/api/utils.rb:108)
  at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2355)
  at RUBY.__bulkify(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-2.0.0/lib/elasticsearch/api/utils.rb:108)
  at RUBY.bulk(/project/ipros/ipros-embulk/embulk_bundle/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-2.0.0/lib/elasticsearch/api/actions/bulk.rb:88)
  at RUBY.send(/Users/toyama-h/Dropbox/github/embulk-output-elasticsearch_ruby/lib/embulk/output/elasticsearch_ruby.rb:176)
  at Users.toyama_minus_h.Dropbox.github.embulk_minus_output_minus_elasticsearch_ruby.lib.embulk.output.elasticsearch_ruby.block in add(/Users/toyama-h/Dropbox/github/embulk-output-elasticsearch_ruby/lib/embulk/output/elasticsearch_ruby.rb:131)
  at RUBY.each(uri:classloader:/embulk/page.rb:22)
  at RUBY.add(/Users/toyama-h/Dropbox/github/embulk-output-elasticsearch_ruby/lib/embulk/output/elasticsearch_ruby.rb:121)
  at RUBY.add(uri:classloader:/embulk/output_plugin.rb:114)

4つのoutput threadでthread2がダメなケースです。(0と１と３はOK)

エラーが発生する箇所。

  • rescueするエラーの定義がまずかった(ので全てのエラーを拾ってretryするように直しました)
  • 自社で使ってるやつです

https://github.com/ipros-team/embulk-output-elasticsearch_ruby/blob/ae0071e8199e343fab53d0566b9d7f35cc02c340/lib/embulk/output/elasticsearch_ruby.rb#L185

このraiseしているメッセージがスタックトレースにでてこなくて嵌ってしまいました。

raise "Could not push logs to Elasticsearch after #{retries} retries. #{e.message}"

エラーメッセージは、

uninitialized constant MultiJson::Adapters

どうも、Elasticsearchで使ってるmulti_jsonというgemが遅延load(require)をしているっぽい？

話が少し複雑で、発生する時と発生しない時があります。

  • input-presto => elasticsearch(エラー1回も発生せず)
  • input-s3 => elasticsearch(エラー発生)

修正後も↑のエラー自体は発生します。 (がrescueしてretryするとなぜかうまくいきます。これはElasticsearch gemの問題かも)

2016-07-29 09:44:53.127 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-s3 (0.2.8)
2016-07-29 09:44:54.133 +0900 [INFO] (0001:transaction): Loaded plugin embulk/output/elasticsearch_ruby from a load path
2016-07-29 09:44:55.401 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 4 = input tasks 1 * 4
2016-07-29 09:44:55.428 +0900 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2016-07-29 09:44:56.071 +0900 [WARN] (embulk-output-executor-2): Could not push logs to Elasticsearch, resetting connection and trying again. uninitialized constant MultiJson::Adapters
2016-07-29 09:44:57.294 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:44:57.376 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:44:57.739 +0900 [INFO] (embulk-output-executor-3): bulk: 1000 success.
2016-07-29 09:44:58.035 +0900 [INFO] (embulk-output-executor-0): bulk: 1000 success.
2016-07-29 09:44:58.459 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.
2016-07-29 09:44:59.020 +0900 [INFO] (embulk-output-executor-2): bulk: 1000 success.
2016-07-29 09:45:00.296 +0900 [INFO] (embulk-output-executor-1): bulk: 1000 success.

ここでrescueに入ってます。

2016-07-29 09:44:56.071 +0900 [WARN] (embulk-output-executor-2): Could not push logs to Elasticsearch, resetting connection and trying again. uninitialized constant MultiJson::Adapters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment