gistfile1.txt

2016-04-05T22:42:00.528Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:2,host:logging-kafka-qa-sf-3.otenv.com,port:9092 with correlation id 6677 for 1 topic(s) Set(logstash)
2016-04-05T22:42:00.532Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-3.otenv.com:9092 for producing
2016-04-05T22:42:00.534Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-3.otenv.com:9092
2016-04-05T22:42:00.534Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-05T22:42:00.538Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-10.qasql.opentable.com:9092 for producing
2016-04-05T22:42:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1140.99 OUT:20.42 ACK:20.42 DLQ:0.00 DROP:1120.55 (messages/sec) processing time 75%:19.98 95%:48.82 99%:841.89 µs/msg)
2016-04-05T22:42:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1144.99 OUT:20.24 ACK:20.25 DLQ:0.00 DROP:1124.73 (messages/sec) processing time 75%:20.26 95%:49.83 99%:807.27 µs/msg)
2016-04-05T22:43:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1138.73 OUT:20.28 ACK:20.28 DLQ:0.00 DROP:1118.44 (messages/sec) processing time 75%:19.35 95%:46.07 99%:618.45 µs/msg)
2016-04-05T22:43:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1146.58 OUT:20.32 ACK:20.32 DLQ:0.00 DROP:1126.25 (messages/sec) processing time 75%:19.26 95%:45.52 99%:402.18 µs/msg)
2016-04-05T22:44:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1129.73 OUT:20.31 ACK:20.31 DLQ:0.00 DROP:1109.42 (messages/sec) processing time 75%:19.45 95%:43.67 99%:455.00 µs/msg)
2016-04-05T22:44:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1134.07 OUT:20.31 ACK:20.31 DLQ:0.00 DROP:1113.75 (messages/sec) processing time 75%:18.84 95%:41.39 99%:397.08 µs/msg)
2016-04-05T22:45:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1120.30 OUT:20.33 ACK:20.33 DLQ:0.00 DROP:1099.96 (messages/sec) processing time 75%:19.48 95%:49.09 99%:580.97 µs/msg)
2016-04-05T22:45:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1118.69 OUT:20.24 ACK:20.24 DLQ:0.00 DROP:1098.43 (messages/sec) processing time 75%:18.33 95%:45.46 99%:235.46 µs/msg)
2016-04-05T22:46:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1110.43 OUT:20.27 ACK:20.27 DLQ:0.00 DROP:1090.14 (messages/sec) processing time 75%:18.81 95%:53.37 99%:555.98 µs/msg)
2016-04-05T22:46:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1115.15 OUT:20.14 ACK:20.15 DLQ:0.00 DROP:1095.00 (messages/sec) processing time 75%:18.44 95%:50.96 99%:600.89 µs/msg)
2016-04-05T22:47:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1102.59 OUT:20.16 ACK:20.17 DLQ:0.00 DROP:1082.43 (messages/sec) processing time 75%:19.28 95%:54.00 99%:600.74 µs/msg)
2016-04-05T22:47:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1110.36 OUT:20.03 ACK:20.04 DLQ:0.00 DROP:1090.34 (messages/sec) processing time 75%:19.30 95%:53.76 99%:611.57 µs/msg)
2016-04-05T22:48:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1105.51 OUT:19.97 ACK:19.97 DLQ:0.00 DROP:1085.53 (messages/sec) processing time 75%:20.03 95%:50.75 99%:621.18 µs/msg)
2016-04-05T22:48:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1105.65 OUT:19.77 ACK:19.77 DLQ:0.00 DROP:1085.88 (messages/sec) processing time 75%:18.17 95%:44.49 99%:521.54 µs/msg)
2016-04-05T22:49:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1090.81 OUT:19.58 ACK:19.59 DLQ:0.00 DROP:1071.26 (messages/sec) processing time 75%:18.43 95%:43.67 99%:556.80 µs/msg)
2016-04-05T22:49:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1094.51 OUT:19.53 ACK:19.53 DLQ:0.00 DROP:1074.98 (messages/sec) processing time 75%:17.85 95%:42.21 99%:449.67 µs/msg)
2016-04-05T22:50:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1088.89 OUT:19.48 ACK:19.48 DLQ:0.00 DROP:1069.41 (messages/sec) processing time 75%:18.68 95%:46.64 99%:556.43 µs/msg)
2016-04-05T22:50:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1105.71 OUT:19.51 ACK:19.51 DLQ:0.00 DROP:1086.19 (messages/sec) processing time 75%:19.33 95%:48.87 99%:547.60 µs/msg)
2016-04-05T22:51:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1097.35 OUT:19.60 ACK:19.60 DLQ:0.00 DROP:1077.73 (messages/sec) processing time 75%:20.17 95%:49.60 99%:565.25 µs/msg)
2016-04-05T22:51:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1107.63 OUT:19.33 ACK:19.33 DLQ:0.00 DROP:1088.29 (messages/sec) processing time 75%:19.86 95%:49.60 99%:689.51 µs/msg)
2016-04-05T22:52:00.564Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:1,host:logging-kafka-qa-sf-2.otenv.com,port:9092 with correlation id 6724 for 1 topic(s) Set(logstash)
2016-04-05T22:52:00.567Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-2.otenv.com:9092 for producing
2016-04-05T22:52:00.569Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-2.otenv.com:9092
2016-04-05T22:52:00.570Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-05T22:52:00.573Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-10.qasql.opentable.com:9092 for producing
2016-04-05T22:52:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1095.62 OUT:19.30 ACK:19.30 DLQ:0.00 DROP:1076.31 (messages/sec) processing time 75%:19.29 95%:45.47 99%:689.36 µs/msg)
2016-04-05T22:52:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1101.64 OUT:19.06 ACK:19.06 DLQ:0.00 DROP:1082.86 (messages/sec) processing time 75%:18.34 95%:44.67 99%:718.43 µs/msg)
2016-04-05T22:53:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1093.99 OUT:19.13 ACK:19.14 DLQ:0.00 DROP:1074.84 (messages/sec) processing time 75%:19.64 95%:53.95 99%:639.56 µs/msg)
2016-04-05T22:53:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1103.96 OUT:19.96 ACK:19.96 DLQ:0.00 DROP:1083.98 (messages/sec) processing time 75%:19.58 95%:54.37 99%:521.78 µs/msg)
2016-04-05T22:54:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1097.34 OUT:20.46 ACK:20.46 DLQ:0.00 DROP:1076.86 (messages/sec) processing time 75%:19.07 95%:44.61 99%:520.50 µs/msg)
2016-04-05T22:54:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1111.44 OUT:20.92 ACK:20.92 DLQ:0.00 DROP:1090.51 (messages/sec) processing time 75%:19.51 95%:52.15 99%:550.32 µs/msg)
2016-04-05T22:55:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1109.96 OUT:21.38 ACK:21.38 DLQ:0.00 DROP:1088.60 (messages/sec) processing time 75%:18.07 95%:50.44 99%:541.62 µs/msg)
2016-04-05T22:55:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1116.10 OUT:21.68 ACK:21.68 DLQ:0.00 DROP:1094.41 (messages/sec) processing time 75%:16.95 95%:47.15 99%:505.45 µs/msg)
2016-04-05T22:56:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1107.37 OUT:22.80 ACK:22.80 DLQ:0.00 DROP:1084.56 (messages/sec) processing time 75%:16.95 95%:47.15 99%:523.83 µs/msg)
2016-04-05T22:56:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1105.80 OUT:23.99 ACK:24.00 DLQ:0.00 DROP:1081.79 (messages/sec) processing time 75%:16.46 95%:43.37 99%:431.48 µs/msg)
2016-04-05T22:57:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1001.60 OUT:21.71 ACK:21.71 DLQ:0.00 DROP:979.87 (messages/sec) processing time 75%:16.49 95%:43.53 99%:431.48 µs/msg)
2016-04-05T22:57:40.470Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-05T22:57:40.470Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: null
2016-04-05T22:57:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:906.28 OUT:19.65 ACK:19.65 DLQ:0.00 DROP:886.62 (messages/sec) processing time 75%:16.49 95%:43.53 99%:431.48 µs/msg)
2016-04-05T22:58:04.779Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 93405 on topic-partition loglov3_demo_logstream-0, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2016-04-05T22:58:04.875Z INFO <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Offset commit for group buzzsaw-pp-sf failed due to NOT_COORDINATOR_FOR_GROUP, will find new coordinator and retry
2016-04-05T22:58:04.877Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-05T22:58:04.877Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: This is not the correct coordinator for this group.
2016-04-05T22:58:08.120Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the group is rebalancing, try to re-join group.
2016-04-05T22:58:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:980.77 OUT:23.67 ACK:23.68 DLQ:0.00 DROP:957.08 (messages/sec) processing time 75%:14.44 95%:42.18 99%:298.30 µs/msg)
2016-04-05T22:58:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1054.47 OUT:27.87 ACK:27.87 DLQ:0.00 DROP:1026.59 (messages/sec) processing time 75%:15.22 95%:48.60 99%:504.55 µs/msg)
2016-04-05T22:59:00.598Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-05T22:59:00.599Z WARN <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Failed to send producer request with correlation id 6770 to broker 10 with data for partitions [logstash,0]
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
at kafka.network.BlockingChannel.send(BlockingChannel.scala:92)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
2016-04-05T22:59:00.602Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 3
2016-04-05T22:59:00.703Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:1,host:logging-kafka-qa-sf-2.otenv.com,port:9092 with correlation id 6771 for 1 topic(s) Set(logstash)
2016-04-05T22:59:00.707Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-2.otenv.com:9092 for producing
2016-04-05T22:59:00.709Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-2.otenv.com:9092
2016-04-05T22:59:00.717Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-13.qasql.opentable.com:9092 for producing
2016-04-05T22:59:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:1027.36 OUT:28.90 ACK:28.90 DLQ:0.00 DROP:998.48 (messages/sec) processing time 75%:17.59 95%:53.58 99%:527.61 µs/msg)
2016-04-05T22:59:55.329Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 93906 on topic-partition loglov3_demo_logstream-0, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2016-04-05T22:59:57.593Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since coordinator is either not started or not valid, marking it as dead.
2016-04-05T22:59:57.594Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483636 dead.
2016-04-05T22:59:58.593Z INFO <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Offset commit for group buzzsaw-pp-sf failed due to NOT_COORDINATOR_FOR_GROUP, will find new coordinator and retry
2016-04-05T22:59:58.593Z INFO <> [worker-0] o.a.k.c.c.i.AbstractCoordinator - Marking the coordinator 2147483634 dead.
2016-04-05T22:59:58.593Z WARN <> [worker-0] o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed: This is not the correct coordinator for this group.
2016-04-05T22:59:59.439Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 93921 on topic-partition loglov3_demo_logstream-3, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2016-04-05T22:59:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:998.72 OUT:29.82 ACK:29.81 DLQ:0.00 DROP:968.89 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:00:28.540Z WARN <> [kafka-producer-network-thread | producer-1] o.a.k.c.producer.internals.Sender - Got error produce response with correlation id 93920 on topic-partition loglov3_demo_logstream-1, retrying (9 attempts left). Error: REQUEST_TIMED_OUT
23:00:28,540 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Elapsed period: Tue Apr 05 22:00:29 UTC 2016
23:00:28,542 |-WARN in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Unexpected periodsElapsed value 0
23:00:28,550 |-INFO in ch.qos.logback.core.rolling.helper.Compressor - GZ compressing [/mnt/mesos/sandbox/logs/application.2016-04-05-22.log] as [/mnt/mesos/sandbox/logs/application.2016-04-05-22.log.gz]
2016-04-05T23:00:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:904.43 OUT:27.00 ACK:27.00 DLQ:0.00 DROP:877.42 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:00:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:818.37 OUT:24.43 ACK:24.44 DLQ:0.00 DROP:793.92 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:01:00.724Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-13.qasql.opentable.com:9092
2016-04-05T23:01:00.725Z WARN <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Failed to send producer request with correlation id 6798 to broker 13 with data for partitions [logstash,0]
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
at kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
at kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
at kafka.network.BlockingChannel.send(BlockingChannel.scala:92)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:526)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
2016-04-05T23:01:00.726Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] k.producer.async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 3
2016-04-05T23:01:00.826Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:1,host:logging-kafka-qa-sf-2.otenv.com,port:9092 with correlation id 6799 for 1 topic(s) Set(logstash)
2016-04-05T23:01:00.833Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-2.otenv.com:9092 for producing
2016-04-05T23:01:00.834Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-2.otenv.com:9092
2016-04-05T23:01:00.838Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-10.qasql.opentable.com:9092 for producing
2016-04-05T23:01:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:740.49 OUT:22.10 ACK:22.11 DLQ:0.00 DROP:718.37 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:01:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:670.02 OUT:20.00 ACK:20.01 DLQ:0.00 DROP:650.01 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:02:00.847Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.client.ClientUtils$ - Fetching metadata from broker id:1,host:logging-kafka-qa-sf-2.otenv.com,port:9092 with correlation id 6812 for 1 topic(s) Set(logstash)
2016-04-05T23:02:00.850Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to logging-kafka-qa-sf-2.otenv.com:9092 for producing
2016-04-05T23:02:00.853Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from logging-kafka-qa-sf-2.otenv.com:9092
2016-04-05T23:02:00.853Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Disconnecting from qa-sf-logkafka-10.qasql.opentable.com:9092
2016-04-05T23:02:00.857Z INFO <> [ProducerSendThread-pp-buzzsaw-1-mesos-slave6-qa-sf.qasql.opentable.com] kafka.producer.SyncProducer - Connected to qa-sf-logkafka-10
2016-04-05T23:02:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:606.26 OUT:18.10 ACK:18.10 DLQ:0.00 DROP:588.15 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:02:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:548.57 OUT:16.38 ACK:16.38 DLQ:0.00 DROP:532.18 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:03:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:496.36 OUT:14.82 ACK:14.82 DLQ:0.00 DROP:481.54 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:03:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:449.13 OUT:13.41 ACK:13.41 DLQ:0.00 DROP:435.71 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:04:29.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:406.39 OUT:12.13 ACK:12.13 DLQ:0.00 DROP:394.25 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)
2016-04-05T23:04:59.800Z INFO <> [worker-1] com.opentable.logging.KafkaPipeline - STATS -> INPUT:367.72 OUT:10.98 ACK:10.98 DLQ:0.00 DROP:356.73 (messages/sec) processing time 75%:19.08 95%:55.21 99%:538.84 µs/msg)