Skip to content

Instantly share code, notes, and snippets.

@pcting
Created January 19, 2011 09:38
Show Gist options
  • Save pcting/785914 to your computer and use it in GitHub Desktop.
Save pcting/785914 to your computer and use it in GitHub Desktop.
Flume S3 Sink Failure
2011-01-19 09:34:32,592 [logicalNode nginx_log_collector-20] INFO hdfs.EscapedCustomDfsSink: Opening s3n://XXX/logs/mediasites/date=2011-01-19/hour=09/site=XXX/XXX-log.00000020.20110119-093416586+0000.29141563286990.seq
2011-01-19 09:34:32,592 [logicalNode nginx_log_collector-20] INFO rolling.RollSink: closing RollSink 'collectorSink'
2011-01-19 09:34:32,593 [Roll-TriggerThread-5] WARN rolling.RollSink: TriggerThread interrupted
2011-01-19 09:34:32,593 [logicalNode nginx_log_collector-20] DEBUG collector.CollectorSink: Roll closed, pushing acks for log.00000020.20110119-093416586+0000.29141563286990.seq :: []
2011-01-19 09:34:32,593 [logicalNode nginx_log_collector-20] INFO debug.StubbornAppendSink: Opening Stubborn Append Sink com.cloudera.flume.handlers.debug.StubbornAppendSink@178460d
2011-01-19 09:34:32,593 [logicalNode nginx_log_collector-20] INFO rolling.RollSink: opening RollSink 'collectorSink'
2011-01-19 09:34:32,594 [logicalNode nginx_log_collector-20] INFO debug.InsistentOpenDecorator: Opened MaskDecorator on try 0
2011-01-19 09:34:32,594 [logicalNode nginx_log_collector-20] INFO debug.StubbornAppendSink: Opening Stubborn Append Sink com.cloudera.flume.handlers.debug.StubbornAppendSink@178460d
2011-01-19 09:34:32,594 [logicalNode nginx_log_collector-20] INFO hdfs.EscapedCustomDfsSink: Opening s3n://XXX/logs/mediasites/date=2011-01-19/hour=09/site=XXX/XXX-log.00000020.20110119-093432593+0000.29157570672521.seq
2011-01-19 09:34:32,594 [logicalNode nginx_log_collector-20] INFO debug.InsistentAppendDecorator: append attempt 5 failed, backoff (32000ms): null
2011-01-19 09:34:32,594 [logicalNode nginx_log_collector-20] DEBUG debug.InsistentAppendDecorator:
java.lang.NullPointerException
at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:61)
at org.apache.hadoop.io.compress.GzipCodec.createCompressor(GzipCodec.java:157)
at com.cloudera.flume.handlers.hdfs.CustomDfsSink.open(CustomDfsSink.java:144)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:102)
at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:117)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:167)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:163)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:84)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:106)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:119)
at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:179)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:70)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:70)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:93)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment