gistfile1.txt

2016-04-06T04:30:56.117Z INFO <> [worker-2] com.opentable.logging.KafkaPipeline - STATS -> INPUT:896.78 OUT:24.66 ACK:24.66 DLQ:0.00 DROP:872.12 (messages/sec) processing time 75%:16.10 95%:36.14 99%:66.76 µs/msg)
2016-04-06T04:30:57.588Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:0,host:logging-kafka-qa-sf-1.otenv.com,port:9092 with correlation id 8552 for 1 topic(s) Set(logstash)
2016-04-06T04:30:57.590Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-1.otenv.com:9092 for producing
2016-04-06T04:30:57.592Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-1.otenv.com:9092
2016-04-06T04:30:57.592Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-06T04:31:07.136Z INFO <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Offset commit for group buzzsaw-pp-sf failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
2016-04-06T04:31:07.137Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-06T04:31:07.137Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: The request timed out.
2016-04-06T04:31:08.609Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-10.qasql.opentable.com:9092 for producing
2016-04-06T04:31:12.439Z INFO <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Offset commit for group buzzsaw-pp-sf failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
2016-04-06T04:31:12.439Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-06T04:31:12.439Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: The request timed out.
2016-04-06T04:31:26.117Z INFO <> [worker-2] com.opentable.logging.KafkaPipeline - STATS -> INPUT:812.43 OUT:22.33 ACK:22.32 DLQ:0.00 DROP:790.10 (messages/sec) processing time 75%:16.10 95%:35.41 99%:66.76 µs/msg)
2016-04-06T04:31:42.643Z WARN <> [jvm-pause-alarm] c.o.pausedetector.JvmPauseAlarm - Detected pause of 8.2s!
2016-04-06T04:31:42.645Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167562 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: REQUEST_TIMED_OUT
2016-04-06T04:31:48.679Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167568 on topic-partition loglov3_demo_logstream-3, retrying (9 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:31:48.679Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167566 on topic-partition loglov3_demo_logstream-3, retrying (9 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:31:48.680Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167563 on topic-partition loglov3_demo_logstream-3, retrying (9 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:31:48.729Z WARN <> [jvm-pause-alarm] c.o.pausedetector.JvmPauseAlarm - Detected pause of 6.1s!
2016-04-06T04:31:56.117Z INFO <> [worker-2] com.opentable.logging.KafkaPipeline - STATS -> INPUT:735.68 OUT:20.22 ACK:20.20 DLQ:0.00 DROP:715.45 (messages/sec) processing time 75%:16.02 95%:35.22 99%:66.76 µs/msg)
2016-04-06T04:32:04.242Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-06T04:32:04.242Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: null
2016-04-06T04:32:07.562Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167565 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: REQUEST_TIMED_OUT
2016-04-06T04:32:07.562Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167569 on topic-partition loglov3_demo_logstream-0, retrying (9 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:32:07.562Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167571 on topic-partition loglov3_demo_logstream-1, retrying (8 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:32:07.562Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167567 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: NETWORK_EXCEPTION
2016-04-06T04:32:08.698Z ERROR <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred while committing offsets for group buzzsaw-pp-sf
2016-04-06T04:32:08.699Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: Commit cannot be completed due to group rebalance
2016-04-06T04:32:08.706Z ERROR <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred while committing offsets for group buzzsaw-pp-sf
2016-04-06T04:32:08.706Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed:
2016-04-06T04:32:08.750Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Attempt to join group buzzsaw-pp-sf failed due to unknown member id, resetting and retrying.
2016-04-06T04:32:12.803Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167627 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2016-04-06T04:32:17.802Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 167631 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2016-04-06T04:32:22.648Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-06T04:32:22.649Z WARN <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Failed to send producer request with correlation id 8594 to broker 10 with data for partitions [logstash,0]
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
at kafka.network.BlockingChannel.send(BlockingChannel.scala:92)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
2016-04-06T04:32:22.650Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 3
2016-04-06T04:32:22.750Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:1,host:logging-kafka-qa-sf-2.otenv.com,port:9092 with correlation id 8595 for 1 topic(s) Set(logstash)
2016-04-06T04:32:26.117Z INFO <> [worker-2] com.opentable.logging.KafkaPipeline - STATS -> INPUT:783.10 OUT:20.27 ACK:20.28 DLQ:0.00 DROP:762.83 (messages/sec) processing time 75%:14.50 95%:41.34 99%:62.17 µs/msg)
2016-04-06T04:32:26.758Z INFO <> [ProducerSendThread-pp-buzzsaw-2-mesos-slave3-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-2.otenv.com:9092 for producing