Skip to content

Instantly share code, notes, and snippets.

@nsivabalan
Created July 5, 2021 23:34
Show Gist options
  • Save nsivabalan/fb02509e68797ea222d7109575c70789 to your computer and use it in GitHub Desktop.
Save nsivabalan/fb02509e68797ea222d7109575c70789 to your computer and use it in GitHub Desktop.
21/07/05 20:25:50 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/07/05 20:26:28 WARN HoodieSparkSqlWriter$: hoodie table at s3a://siva-test-bucket-june-16/hudi_testing/hudi_base_2 already exists. Deleting existing data & overwriting with new data.
21/07/05 22:27:52 WARN TaskSetManager: Lost task 438.0 in stage 12.0 (TID 10022, ip-172-31-37-233.us-east-2.compute.internal, executor 2): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:29:11 WARN TaskSetManager: Lost task 378.0 in stage 12.0 (TID 9962, ip-172-31-47-28.us-east-2.compute.internal, executor 12): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:30:00 WARN TaskSetManager: Lost task 417.1 in stage 12.0 (TID 10042, ip-172-31-44-54.us-east-2.compute.internal, executor 10): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:31:20 WARN TaskSetManager: Lost task 391.0 in stage 12.0 (TID 9975, ip-172-31-36-52.us-east-2.compute.internal, executor 3): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:32:19 WARN TaskSetManager: Lost task 432.0 in stage 12.0 (TID 10016, ip-172-31-37-233.us-east-2.compute.internal, executor 1): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:32:33 WARN TaskSetManager: Lost task 447.0 in stage 12.0 (TID 10034, ip-172-31-47-28.us-east-2.compute.internal, executor 11): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:32:44 WARN TaskSetManager: Lost task 407.0 in stage 12.0 (TID 9991, ip-172-31-36-40.us-east-2.compute.internal, executor 14): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:33:10 WARN TaskSetManager: Lost task 448.1 in stage 12.0 (TID 10160, ip-172-31-37-233.us-east-2.compute.internal, executor 2): java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1388)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:81)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:36)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 22 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:143)
at org.apache.hudi.execution.SparkLazyInsertIterable.computeNext(SparkLazyInsertIterable.java:77)
... 24 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Server Error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:141)
... 25 more
Caused by: org.apache.hudi.exception.HoodieRemoteException: Server Error
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:485)
at org.apache.hudi.table.marker.TimelineBasedMarkerFiles.create(TimelineBasedMarkerFiles.java:80)
at org.apache.hudi.table.marker.MarkerFiles.create(MarkerFiles.java:54)
at org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:181)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:83)
at org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:65)
at org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.http.client.HttpResponseException: Server Error
at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69)
at org.apache.http.client.fluent.Response.handleResponse(Response.java:90)
at org.apache.http.client.fluent.Response.returnContent(Response.java:97)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:187)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.createMarker(RemoteHoodieTableFileSystemView.java:483)
... 14 more
21/07/05 22:33:10 ERROR TaskSetManager: Task 423 in stage 12.0 failed 4 times; aborting job
21/07/05 22:33:10 WARN TaskSetManager: Lost task 448.2 in stage 12.0 (TID 10190, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 346.1 in stage 12.0 (TID 10189, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 374.1 in stage 12.0 (TID 10188, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 435.3 in stage 12.0 (TID 10143, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 387.3 in stage 12.0 (TID 10137, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 403.2 in stage 12.0 (TID 10121, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 427.3 in stage 12.0 (TID 10146, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 410.3 in stage 12.0 (TID 10142, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:10 WARN TaskSetManager: Lost task 336.3 in stage 12.0 (TID 10145, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 379.3 in stage 12.0 (TID 10153, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 335.3 in stage 12.0 (TID 10129, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 372.1 in stage 12.0 (TID 10120, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 373.1 in stage 12.0 (TID 10130, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 418.2 in stage 12.0 (TID 10119, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 348.3 in stage 12.0 (TID 10134, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:11 WARN TaskSetManager: Lost task 414.3 in stage 12.0 (TID 10155, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:12 WARN TaskSetManager: Lost task 340.3 in stage 12.0 (TID 10152, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:12 WARN TaskSetManager: Lost task 421.3 in stage 12.0 (TID 10135, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:12 WARN TaskSetManager: Lost task 345.3 in stage 12.0 (TID 10124, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:12 WARN TaskSetManager: Lost task 446.1 in stage 12.0 (TID 10162, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:13 WARN TaskSetManager: Lost task 411.1 in stage 12.0 (TID 10161, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:13 WARN TaskSetManager: Lost task 424.1 in stage 12.0 (TID 10157, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:13 WARN TaskSetManager: Lost task 329.3 in stage 12.0 (TID 10154, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:33:13 WARN TaskSetManager: Lost task 382.1 in stage 12.0 (TID 10166, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:13 WARN TaskSetManager: Lost task 444.1 in stage 12.0 (TID 10164, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:14 WARN TaskSetManager: Lost task 341.0 in stage 12.0 (TID 9925, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:14 WARN TaskSetManager: Lost task 354.3 in stage 12.0 (TID 10148, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:15 WARN TaskSetManager: Lost task 364.3 in stage 12.0 (TID 10144, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:15 WARN TaskSetManager: Lost task 343.3 in stage 12.0 (TID 10147, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:15 WARN TaskSetManager: Lost task 342.3 in stage 12.0 (TID 10150, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:16 WARN TaskSetManager: Lost task 337.3 in stage 12.0 (TID 10149, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 395.1 in stage 12.0 (TID 10169, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 406.1 in stage 12.0 (TID 10159, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 417.3 in stage 12.0 (TID 10125, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 351.3 in stage 12.0 (TID 10132, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 436.2 in stage 12.0 (TID 10127, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 409.3 in stage 12.0 (TID 10131, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 419.1 in stage 12.0 (TID 10141, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 369.3 in stage 12.0 (TID 10139, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 451.1 in stage 12.0 (TID 10168, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 362.1 in stage 12.0 (TID 10167, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:17 WARN TaskSetManager: Lost task 438.3 in stage 12.0 (TID 10177, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 441.3 in stage 12.0 (TID 10151, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 350.3 in stage 12.0 (TID 10171, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 347.1 in stage 12.0 (TID 10173, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 391.2 in stage 12.0 (TID 10172, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 375.1 in stage 12.0 (TID 10179, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 420.3 in stage 12.0 (TID 10138, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 401.1 in stage 12.0 (TID 10158, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 407.1 in stage 12.0 (TID 10180, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 430.1 in stage 12.0 (TID 10184, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:18 WARN TaskSetManager: Lost task 415.1 in stage 12.0 (TID 10165, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:19 WARN TaskSetManager: Lost task 389.1 in stage 12.0 (TID 10182, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:19 WARN TaskSetManager: Lost task 388.1 in stage 12.0 (TID 10187, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:19 WARN TaskSetManager: Lost task 440.1 in stage 12.0 (TID 10170, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:20 WARN TaskSetManager: Lost task 366.0 in stage 12.0 (TID 9950, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:33:20 WARN TaskSetManager: Lost task 378.3 in stage 12.0 (TID 10176, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 433.0 in stage 12.0 (TID 10017, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 399.3 in stage 12.0 (TID 10133, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 405.2 in stage 12.0 (TID 10178, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 339.1 in stage 12.0 (TID 10174, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 383.1 in stage 12.0 (TID 10175, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 320.3 in stage 12.0 (TID 10181, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:21 WARN TaskSetManager: Lost task 365.1 in stage 12.0 (TID 10186, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:22 WARN TaskSetManager: Lost task 356.1 in stage 12.0 (TID 10185, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:22 WARN TaskSetManager: Lost task 404.1 in stage 12.0 (TID 10183, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:23 WARN TaskSetManager: Lost task 416.0 in stage 12.0 (TID 10000, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:24 WARN TaskSetManager: Lost task 413.3 in stage 12.0 (TID 10156, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:24 WARN TaskSetManager: Lost task 376.0 in stage 12.0 (TID 9960, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:24 WARN TaskSetManager: Lost task 384.1 in stage 12.0 (TID 10114, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:24 WARN TaskSetManager: Lost task 408.0 in stage 12.0 (TID 9992, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:33:24 WARN TaskSetManager: Lost task 367.0 in stage 12.0 (TID 9951, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:26 WARN TaskSetManager: Lost task 377.1 in stage 12.0 (TID 10113, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:26 WARN TaskSetManager: Lost task 450.0 in stage 12.0 (TID 10078, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:27 WARN TaskSetManager: Lost task 394.0 in stage 12.0 (TID 9978, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:27 WARN TaskSetManager: Lost task 400.1 in stage 12.0 (TID 10116, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:28 WARN TaskSetManager: Lost task 445.0 in stage 12.0 (TID 10029, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:28 WARN TaskSetManager: Lost task 437.2 in stage 12.0 (TID 10117, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:29 WARN TaskSetManager: Lost task 432.1 in stage 12.0 (TID 10115, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:33:30 WARN TaskSetManager: Lost task 349.0 in stage 12.0 (TID 9933, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:33:30 WARN TaskSetManager: Lost task 368.0 in stage 12.0 (TID 9952, ip-172-31-36-52.us-east-2.compute.internal, executor 3): TaskKilled (Stage cancelled)
21/07/05 22:33:32 WARN TaskSetManager: Lost task 360.0 in stage 12.0 (TID 9944, ip-172-31-36-40.us-east-2.compute.internal, executor 14): TaskKilled (Stage cancelled)
21/07/05 22:33:34 WARN TaskSetManager: Lost task 390.0 in stage 12.0 (TID 9974, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:34 WARN TaskSetManager: Lost task 431.0 in stage 12.0 (TID 10015, ip-172-31-43-189.us-east-2.compute.internal, executor 8): TaskKilled (Stage cancelled)
21/07/05 22:33:35 WARN TaskSetManager: Lost task 447.1 in stage 12.0 (TID 10122, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:36 WARN TaskSetManager: Lost task 434.2 in stage 12.0 (TID 10126, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:36 WARN TaskSetManager: Lost task 353.0 in stage 12.0 (TID 9937, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:37 WARN TaskSetManager: Lost task 386.0 in stage 12.0 (TID 9970, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:38 WARN TaskSetManager: Lost task 392.0 in stage 12.0 (TID 9976, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:33:39 WARN TaskSetManager: Lost task 442.0 in stage 12.0 (TID 10026, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:39 WARN TaskSetManager: Lost task 359.0 in stage 12.0 (TID 9943, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:33:42 WARN TaskSetManager: Lost task 385.0 in stage 12.0 (TID 9969, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:43 WARN TaskSetManager: Lost task 357.0 in stage 12.0 (TID 9941, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:33:44 WARN TaskSetManager: Lost task 371.0 in stage 12.0 (TID 9955, ip-172-31-44-54.us-east-2.compute.internal, executor 9): TaskKilled (Stage cancelled)
21/07/05 22:33:48 WARN TaskSetManager: Lost task 358.0 in stage 12.0 (TID 9942, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:33:49 WARN TaskSetManager: Lost task 449.0 in stage 12.0 (TID 10077, ip-172-31-37-233.us-east-2.compute.internal, executor 1): TaskKilled (Stage cancelled)
21/07/05 22:33:54 WARN TaskSetManager: Lost task 428.0 in stage 12.0 (TID 10012, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:33:55 WARN TaskSetManager: Lost task 344.0 in stage 12.0 (TID 9928, ip-172-31-37-233.us-east-2.compute.internal, executor 2): TaskKilled (Stage cancelled)
21/07/05 22:33:58 WARN TaskSetManager: Lost task 412.0 in stage 12.0 (TID 9996, ip-172-31-43-189.us-east-2.compute.internal, executor 7): TaskKilled (Stage cancelled)
21/07/05 22:34:00 WARN TaskSetManager: Lost task 396.0 in stage 12.0 (TID 9980, ip-172-31-36-52.us-east-2.compute.internal, executor 4): TaskKilled (Stage cancelled)
21/07/05 22:34:01 WARN TaskSetManager: Lost task 370.0 in stage 12.0 (TID 9954, ip-172-31-44-31.us-east-2.compute.internal, executor 5): TaskKilled (Stage cancelled)
21/07/05 22:34:02 WARN TaskSetManager: Lost task 352.0 in stage 12.0 (TID 9936, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:34:03 WARN TaskSetManager: Lost task 425.0 in stage 12.0 (TID 10009, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:34:10 WARN TaskSetManager: Lost task 398.0 in stage 12.0 (TID 9982, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
21/07/05 22:34:14 WARN TaskSetManager: Lost task 443.0 in stage 12.0 (TID 10027, ip-172-31-44-54.us-east-2.compute.internal, executor 10): TaskKilled (Stage cancelled)
21/07/05 22:34:14 WARN TaskSetManager: Lost task 426.0 in stage 12.0 (TID 10010, ip-172-31-36-40.us-east-2.compute.internal, executor 13): TaskKilled (Stage cancelled)
21/07/05 22:34:14 WARN TaskSetManager: Lost task 363.1 in stage 12.0 (TID 10118, ip-172-31-47-28.us-east-2.compute.internal, executor 12): TaskKilled (Stage cancelled)
21/07/05 22:34:15 WARN TaskSetManager: Lost task 393.0 in stage 12.0 (TID 9977, ip-172-31-47-28.us-east-2.compute.internal, executor 11): TaskKilled (Stage cancelled)
21/07/05 22:34:17 WARN TaskSetManager: Lost task 361.0 in stage 12.0 (TID 9945, ip-172-31-44-31.us-east-2.compute.internal, executor 6): TaskKilled (Stage cancelled)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment