Skip to content

Instantly share code, notes, and snippets.

@ashkrit

ashkrit/SparkMetrics.txt

Created January 27, 2019 05:47
Show Gist options
  • Save ashkrit/fd8bc9dcfc60936ea8b85ff69378e1ea to your computer and use it in GitHub Desktop.
Save ashkrit/fd8bc9dcfc60936ea8b85ff69378e1ea to your computer and use it in GitHub Desktop.
Download ZIP
Raw
SparkMetrics.txt
19/01/27 13:35:57 INFO StageAccumulatorListener: Stage accumulator values:map at SparkMonitoringAdvance.scala:44
19/01/27 13:35:57 INFO StageAccumulatorListener: 8:Some(internal.metrics.memoryBytesSpilled):Some(0)
19/01/27 13:35:57 INFO StageAccumulatorListener: 2:Some(internal.metrics.executorDeserializeCpuTime):Some(468750000)
19/01/27 13:35:57 INFO StageAccumulatorListener: 20:Some(internal.metrics.shuffle.write.writeTime):Some(3082492535)
19/01/27 13:35:57 INFO StageAccumulatorListener: 5:Some(internal.metrics.resultSize):Some(396352)
19/01/27 13:35:57 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[7] at sortBy at SparkMonitoringAdvance.scala:49), which has no missing parents
19/01/27 13:35:57 INFO StageAccumulatorListener: 4:Some(internal.metrics.executorCpuTime):Some(259421875000)
19/01/27 13:35:57 INFO StageAccumulatorListener: 22:Some(internal.metrics.input.recordsRead):Some(42339442)
19/01/27 13:35:57 INFO StageAccumulatorListener: 7:Some(internal.metrics.resultSerializationTime):Some(25)
19/01/27 13:35:57 INFO StageAccumulatorListener: 1:Some(internal.metrics.executorDeserializeTime):Some(592)
19/01/27 13:35:57 INFO StageAccumulatorListener: 10:Some(internal.metrics.peakExecutionMemory):Some(4447578218)
19/01/27 13:35:57 INFO StageAccumulatorListener: 19:Some(internal.metrics.shuffle.write.recordsWritten):Some(41481514)
19/01/27 13:35:57 INFO StageAccumulatorListener: 9:Some(internal.metrics.diskBytesSpilled):Some(0)
19/01/27 13:35:57 INFO StageAccumulatorListener: 18:Some(internal.metrics.shuffle.write.bytesWritten):Some(383161516)
19/01/27 13:35:57 INFO StageAccumulatorListener: 3:Some(internal.metrics.executorRunTime):Some(379425)
19/01/27 13:35:57 INFO StageAccumulatorListener: 21:Some(internal.metrics.input.bytesRead):Some(8989380683)
19/01/27 13:35:57 INFO StageAccumulatorListener: 6:Some(internal.metrics.jvmGCTime):Some(71907)
19/01/27 13:35:57 INFO StageAccumulatorListener: 0:Some(recordCounter):Some(42339442)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment