Skip to content

Instantly share code, notes, and snippets.

@nsivabalan
Created August 4, 2022 14:47
Show Gist options
  • Select an option

  • Save nsivabalan/a0517ca0881adaf27da32a6b67929ae1 to your computer and use it in GitHub Desktop.

Select an option

Save nsivabalan/a0517ca0881adaf27da32a6b67929ae1 to your computer and use it in GitHub Desktop.
22/08/04 13:52:20 INFO HiveSyncTool: Sync complete for pqm_cow_dhspyam_tacapro_2022_08_04_33_table
22/08/04 13:52:20 INFO metastore: Closed a connection to metastore, current connections: 0
22/08/04 13:52:20 INFO DagScheduler: Finished executing b20fc7ae-e6d9-4b25-90a8-28ba0a54c3f5
22/08/04 13:52:20 WARN DagScheduler: Executing node "first_hive_query" :: {"hive_props":["set hive.input.format = org.apache.hudi.hadoop.HoodieParquetInputFormat"],"name":"25f871d6-4bd3-4c21-85cf-ff5edaf02ca2","hive_queries":[{"select count(*) from testdb.table1":400},{"select count(*) from testdb.table1 group by `_row_key` having count(*) > 1":0}],"config":"first_hive_query"}
22/08/04 13:52:20 INFO DagNode: Executing hive query node 25f871d6-4bd3-4c21-85cf-ff5edaf02ca2
22/08/04 13:52:20 ERROR DagScheduler: Exception executing node
org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:77)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:136)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.sql.SQLException: No suitable driver found for random
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:67)
... 7 more
22/08/04 13:52:20 INFO DagScheduler: Forcing shutdown of executor service, this might kill running tasks
22/08/04 13:52:20 ERROR HoodieTestSuiteJob: Failed to run Test Suite
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.execute(DagScheduler.java:113)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.schedule(DagScheduler.java:68)
at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.runTestSuite(HoodieTestSuiteJob.java:215)
at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:143)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:77)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:136)
... 6 more
Caused by: java.sql.SQLException: No suitable driver found for random
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:67)
... 7 more
22/08/04 13:52:20 INFO SparkUI: Stopped Spark web UI at http://spark-a035ef82691d9e1e-driver-svc.infra.svc:8090
22/08/04 13:52:20 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
22/08/04 13:52:20 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
22/08/04 13:52:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
22/08/04 13:52:20 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/08/04 13:52:20 INFO MemoryStore: MemoryStore cleared
22/08/04 13:52:20 INFO BlockManager: BlockManager stopped
22/08/04 13:52:20 INFO BlockManagerMaster: BlockManagerMaster stopped
22/08/04 13:52:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/08/04 13:52:20 INFO SparkContext: Successfully stopped SparkContext
22/08/04 13:52:20 INFO SparkContext: SparkContext already stopped.
Exception in thread "main" org.apache.hudi.exception.HoodieException: Failed to run Test Suite
at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.runTestSuite(HoodieTestSuiteJob.java:220)
at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.execute(DagScheduler.java:113)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.schedule(DagScheduler.java:68)
at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.runTestSuite(HoodieTestSuiteJob.java:215)
... 13 more
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:143)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieValidationException: Hive query validation failed due to No suitable driver found for random
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:77)
at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:136)
... 6 more
Caused by: java.sql.SQLException: No suitable driver found for random
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode.execute(HiveQueryNode.java:67)
... 7 more
22/08/04 13:52:20 INFO ShutdownHookManager: Shutdown hook called
22/08/04 13:52:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-60b3b994-1b47-4f43-8234-e7a63a50092d
22/08/04 13:52:20 INFO ShutdownHookManager: Deleting directory /var/data/spark-65d47dff-b50a-4111-86ee-6a9103847d88/spark-87d57393-8ecc-49a4-9a5f-b8eadf47c1cc
22/08/04 13:52:20 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
22/08/04 13:52:20 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
22/08/04 13:52:20 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment