Skip to content

Instantly share code, notes, and snippets.

View rahulsmahadev's full-sized avatar

Rahul Shivu Mahadev rahulsmahadev

  • Databricks
  • San Francisco
View GitHub Profile
26/04/30 21:35:32 WARN HangingThreadDetector: Potential Hung Thread Detected:
Thread ID: 1910
Thread name: SparkConnectExecuteThread_opId=6e78d900-bffd-4cb8-903b-554eb03722e7
Time Spent Hanging: 29399994 ms
Stuck at Frame: java.base@17.0.16/jdk.internal.misc.Unsafe.park(Native Method)
Latest Stack Trace:
java.base@17.0.16/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@17.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
at java.base@17.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
at java.base@17.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
- test post optimize metadata tagging *** FAILED *** (17 seconds, 315 milliseconds)
java.lang.IllegalStateException: Incrementally Computed State failed checksum check for commit 5 [Map(filePathsFromPreviousVersion -> ["part-00000-9d6358a7-e956-4897-9828-265516870b03-c000.snappy.parquet","part-00000-5bdcf33c-42b0-411b-81e8-29bbdc4fdbbe-c000.snappy.parquet","part-00001-e433660c-436c-4a92-872b-2423055ff03b-c000.snappy.parquet","part-00000-b47e717c-a8a6-471a-9c21-68755ef1bd43-c000.snappy.parquet"], numFilesInPreviousVersion -> 4, removeFilePathsInThisCommit -> [], computedTableSizeBytes -> 5981, computedFiles -> [{"path":"part-00000-9d6358a7-e956-4897-9828-265516870b03-c000.snappy.parquet","partitionValues":{},"size":1000,"modificationTime":1712107119660,"dataChange":false,"stats":"{\"numRecords\":50,\"minValues\":{\"id\":0,\"zCol\":1},\"maxValues\":{\"id\":49,\"zCol\":1},\"nullCount\":{\"id\":0,\"zCol\":0},\"tightBounds\":true}","tags":{"MAX_INSERTION_TIME":"1712107119660000","INSERTION_TIME":"171210711966000
[info] io.delta.standalone.internal.HDFSLogStoreSuite *** ABORTED ***
[info] java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
[info] at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)
[info] at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
[info] at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)
[info] at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167)
[info] at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168)
[info] at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59)
[info] at org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387)
[info] at org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186)