Created
June 11, 2020 19:45
-
-
Save lordpretzel/9b5780615338610f58b60aa7d228b43b to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Running sbt | |
| [info] Loading global plugins from /Users/lord_pretzel/.sbt/1.0/plugins | |
| [info] Loading settings for project mimir-caveats-build from metals.sbt ... | |
| [info] Loading project definition from /Users/lord_pretzel/Documents/workspace/mimir-caveats/project | |
| [info] Loading settings for project mimir-caveats from build.sbt ... | |
| [info] Set current project to mimir-caveats (in build file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/) | |
| [info] sbt server started at local:///Users/lord_pretzel/.sbt/1.0/server/3b9af470c5023f52927d/sock | |
| sbt:mimir-caveats> testOnly org.mimirdb.caveats.LogicalPlanRangeSpec -- ex "certain inputs.aggregation - no group-by - aggregtion functions only" | |
| [info] LogicalPlanRangeSpec | |
| [info] DataFrame Range Annotations | |
| [info] Certain inputs | |
| WARNING: An illegal reflective access operation has occurred | |
| WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/lord_pretzel/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.12/3.0.0-preview2/spark-unsafe_2.12-3.0.0-preview2.jar) to constructor java.nio.DirectByteBuffer(long,int) | |
| WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform | |
| WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations | |
| WARNING: All illegal access operations will be denied in a future release | |
| REWRITING PLAN OPERATOR: Aggregate [avg(cast(A#14 as double)) AS X#824] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| REWRITING PLAN OPERATOR: RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| ======================================== | |
| REWRITE OPERATOR TYPE LEAF NODE | |
| ======================================== | |
| -------------------------- | |
| REWRITTEN OPERATOR: | |
| -------------------------- | |
| 'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| ---------------------------------------- | |
| EXPR: avg(cast(A#14 as double)) AS X#824 | |
| GB: None | |
| trace: true | |
| ---------------------------------------- | |
| EXPR: avg(cast(A#14 as double)) | |
| GB: None | |
| trace: true | |
| ---------------------------------------- | |
| EXPR: sum(cast(A#14 as double)) | |
| GB: None | |
| trace: true | |
| ===========> BG EQUALS: true | |
| GROUP BY: None | |
| ---------------------------------------- | |
| EXPR: cast(A#14 as double) | |
| GB: None | |
| trace: true | |
| ---------------------------------------- | |
| EXPR: A#14 | |
| GB: None | |
| trace: true | |
| sum(CASE WHEN (`__CAVEATS_ROW_LB` > 0) THEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) < 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END) ELSE least(0.0D, (CAST(`__CAVEATS_A_LB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_LB` AS DOUBLE) < 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END)) END) | |
| sum(CASE WHEN true THEN (CAST(`A` AS DOUBLE) * `__CAVEATS_ROW_BG`) ELSE 0.0D END) | |
| sum(CASE WHEN (`__CAVEATS_ROW_LB` > 0) THEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) > 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END) ELSE greatest(0.0D, (CAST(`__CAVEATS_A_UB` AS DOUBLE) * CASE WHEN (CAST(`__CAVEATS_A_UB` AS DOUBLE) > 0) THEN `__CAVEATS_ROW_UB` ELSE `__CAVEATS_ROW_LB` END)) END) | |
| ---------------------------------------- | |
| EXPR: count(1) | |
| GB: None | |
| trace: true | |
| ===========> BG EQUALS: true | |
| GROUP BY: None | |
| -------------------------- | |
| REWRITTEN OPERATOR: | |
| -------------------------- | |
| 'Aggregate [CASE WHEN (sum('__CAVEATS_ROW_BG) = 0) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast('A as double) * '__CAVEATS_ROW_BG) ELSE 0.0 END) / cast(sum('__CAVEATS_ROW_BG) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE least(0.0, (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum('__CAVEATS_ROW_UB) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE greatest(0.0, (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum('__CAVEATS_ROW_UB) as double)) END AS __CAVEATS_X_UB#843] | |
| +- 'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| is already annotated? false | |
| base schema: StructType(StructField(X,DoubleType,true)) | |
| row encoder StructType(StructField(X,DoubleType,true), StructField(__CAVEATS_ROW_LB,IntegerType,false), StructField(__CAVEATS_ROW_BG,IntegerType,false), StructField(__CAVEATS_ROW_UB,IntegerType,false), StructField(__CAVEATS_X_LB,DoubleType,true), StructField(__CAVEATS_X_UB,DoubleType,true)) | |
| ================================================================================ | |
| FINAL | |
| ================================================================================ | |
| ============================== QUERY EXECUTION (PLANS) ============================== | |
| == Parsed Logical Plan == | |
| 'Aggregate [CASE WHEN (sum('__CAVEATS_ROW_BG) = 0) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast('A as double) * '__CAVEATS_ROW_BG) ELSE 0.0 END) / cast(sum('__CAVEATS_ROW_BG) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE least(0.0, (cast('__CAVEATS_A_LB as double) * CASE WHEN (cast('__CAVEATS_A_LB as double) < 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN '__CAVEATS_ROW_LB ELSE least(0, '__CAVEATS_ROW_LB) END) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum('__CAVEATS_ROW_UB) = 0) THEN 0.0 ELSE (sum(CASE WHEN ('__CAVEATS_ROW_LB > 0) THEN (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END) ELSE greatest(0.0, (cast('__CAVEATS_A_UB as double) * CASE WHEN (cast('__CAVEATS_A_UB as double) > 0) THEN '__CAVEATS_ROW_UB ELSE '__CAVEATS_ROW_LB END)) END) / cast(sum('__CAVEATS_ROW_UB) as double)) END AS __CAVEATS_X_UB#843] | |
| +- 'Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, 'A AS __CAVEATS_A_LB#832, 'A AS __CAVEATS_A_UB#833, 'B AS __CAVEATS_B_LB#834, 'B AS __CAVEATS_B_UB#835, 'C AS __CAVEATS_C_LB#836, 'C AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| == Analyzed Logical Plan == | |
| X: double, __CAVEATS_ROW_LB: int, __CAVEATS_ROW_BG: int, __CAVEATS_ROW_UB: int, __CAVEATS_X_LB: double, __CAVEATS_X_UB: double | |
| Aggregate [CASE WHEN (sum(cast(__CAVEATS_ROW_BG#830 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast(A#14 as double) * cast(__CAVEATS_ROW_BG#830 as double)) ELSE 0.0 END) / cast(sum(cast(__CAVEATS_ROW_BG#830 as bigint)) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE least(0.0, (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(cast(__CAVEATS_ROW_UB#831 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE greatest(0.0, (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(__CAVEATS_ROW_UB#831 as bigint)) as double)) END AS __CAVEATS_X_UB#843] | |
| +- Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| == Optimized Logical Plan == | |
| Aggregate [CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(A#14 as double) * 1.0)) / cast(sum(1) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)) / cast(sum(1) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(1) = 0) THEN 0.0 ELSE (sum((cast(__CAVEATS_A_UB#833 as double) * 1.0)) / cast(sum(1) as double)) END AS __CAVEATS_X_UB#843] | |
| +- Project [A#14, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833] | |
| +- RelationV2[A#14] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| == Physical Plan == | |
| *(1) HashAggregate(keys=[], functions=[sum(1), sum((cast(A#14 as double) * 1.0)), sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)), sum((cast(__CAVEATS_A_UB#833 as double) * 1.0))], output=[X#842, __CAVEATS_ROW_LB#838, __CAVEATS_ROW_BG#839, __CAVEATS_ROW_UB#840, __CAVEATS_X_LB#841, __CAVEATS_X_UB#843]) | |
| +- *(1) HashAggregate(keys=[], functions=[partial_sum(1), partial_sum((cast(A#14 as double) * 1.0)), partial_sum((cast(__CAVEATS_A_LB#832 as double) * 1.0)), partial_sum((cast(__CAVEATS_A_UB#833 as double) * 1.0))], output=[sum#855L, sum#856, sum#857, sum#858]) | |
| +- *(1) Project [A#14, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833] | |
| +- BatchScan[A#14] CSVScan Location: InMemoryFileIndex[file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv], ReadSchema: struct<A:string> | |
| ============================== SCHEMA ============================== | |
| StructType(StructField(X,DoubleType,true), StructField(__CAVEATS_ROW_LB,IntegerType,false), StructField(__CAVEATS_ROW_BG,IntegerType,false), StructField(__CAVEATS_ROW_UB,IntegerType,false), StructField(__CAVEATS_X_LB,DoubleType,true), StructField(__CAVEATS_X_UB,DoubleType,true)) | |
| ============================== RESULT ============================== | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| | X|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_X_LB|__CAVEATS_X_UB| | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| |1.0| 1| 1| 1| 1.0| 1.0| | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| ================================================================================ | |
| QUERY | |
| ================================================================================ | |
| Aggregate [CASE WHEN (sum(cast(__CAVEATS_ROW_BG#830 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN true THEN (cast(A#14 as double) * cast(__CAVEATS_ROW_BG#830 as double)) ELSE 0.0 END) / cast(sum(cast(__CAVEATS_ROW_BG#830 as bigint)) as double)) END AS X#842, 1 AS __CAVEATS_ROW_LB#838, 1 AS __CAVEATS_ROW_BG#839, 1 AS __CAVEATS_ROW_UB#840, CASE WHEN (sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE least(0.0, (cast(__CAVEATS_A_LB#832 as double) * cast(CASE WHEN (cast(__CAVEATS_A_LB#832 as double) < cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN __CAVEATS_ROW_LB#829 ELSE least(0, __CAVEATS_ROW_LB#829) END as bigint)) as double)) END AS __CAVEATS_X_LB#841, CASE WHEN (sum(cast(__CAVEATS_ROW_UB#831 as bigint)) = cast(0 as bigint)) THEN 0.0 ELSE (sum(CASE WHEN (__CAVEATS_ROW_LB#829 > 0) THEN (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double)) ELSE greatest(0.0, (cast(__CAVEATS_A_UB#833 as double) * cast(CASE WHEN (cast(__CAVEATS_A_UB#833 as double) > cast(0 as double)) THEN __CAVEATS_ROW_UB#831 ELSE __CAVEATS_ROW_LB#829 END as double))) END) / cast(sum(cast(__CAVEATS_ROW_UB#831 as bigint)) as double)) END AS __CAVEATS_X_UB#843] | |
| +- Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| ================================================================================ | |
| RESULT | |
| ================================================================================ | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| | X|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_X_LB|__CAVEATS_X_UB| | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| |1.0| 1| 1| 1| 1.0| 1.0| | |
| +---+----------------+----------------+----------------+--------------+--------------+ | |
| ================================================================================ | |
| QUERY | |
| ================================================================================ | |
| Project [A#14, B#15, C#16, 1 AS __CAVEATS_ROW_LB#829, 1 AS __CAVEATS_ROW_BG#830, 1 AS __CAVEATS_ROW_UB#831, A#14 AS __CAVEATS_A_LB#832, A#14 AS __CAVEATS_A_UB#833, B#15 AS __CAVEATS_B_LB#834, B#15 AS __CAVEATS_B_UB#835, C#16 AS __CAVEATS_C_LB#836, C#16 AS __CAVEATS_C_UB#837] | |
| +- RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| ================================================================================ | |
| RESULT | |
| ================================================================================ | |
| +---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+ | |
| | A| B| C|__CAVEATS_ROW_LB|__CAVEATS_ROW_BG|__CAVEATS_ROW_UB|__CAVEATS_A_LB|__CAVEATS_A_UB|__CAVEATS_B_LB|__CAVEATS_B_UB|__CAVEATS_C_LB|__CAVEATS_C_UB| | |
| +---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+ | |
| | 1| 2| 3| 1| 1| 1| 1| 1| 2| 2| 3| 3| | |
| | 1| 3| 1| 1| 1| 1| 1| 1| 3| 3| 1| 1| | |
| | 2|null| 1| 1| 1| 1| 2| 2| null| null| 1| 1| | |
| | 1| 2|null| 1| 1| 1| 1| 1| 2| 2| null| null| | |
| | 1| 4| 2| 1| 1| 1| 1| 1| 4| 4| 2| 2| | |
| | 2| 2| 1| 1| 1| 1| 2| 2| 2| 2| 1| 1| | |
| | 4| 2| 4| 1| 1| 1| 4| 4| 2| 2| 4| 4| | |
| +---+----+----+----------------+----------------+----------------+--------------+--------------+--------------+--------------+--------------+--------------+ | |
| ================================================================================ | |
| QUERY | |
| ================================================================================ | |
| RelationV2[A#14, B#15, C#16] csv file:/Users/lord_pretzel/Documents/workspace/mimir-caveats/test_data/r.csv | |
| ================================================================================ | |
| RESULT | |
| ================================================================================ | |
| +---+----+----+ | |
| | A| B| C| | |
| +---+----+----+ | |
| | 1| 2| 3| | |
| | 1| 3| 1| | |
| | 2|null| 1| | |
| | 1| 2|null| | |
| | 1| 4| 2| | |
| | 2| 2| 1| | |
| | 4| 2| 4| | |
| +---+----+----+ | |
| 14:44:52.231 [Executor task launch worker for task 167] ERROR org.apache.spark.executor.Executor - Exception in task 5.0 in stage 38.0 (TID 167) | |
| java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080 | |
| at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344) | |
| at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) | |
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) | |
| at java.base/java.lang.Thread.run(Thread.java:830) | |
| Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 | |
| at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| at org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| at org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_1$(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| ... 19 common frames omitted | |
| 14:44:52.231 [Executor task launch worker for task 173] ERROR org.apache.spark.executor.Executor - Exception in task 11.0 in stage 38.0 (TID 173) | |
| java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079 | |
| if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080 | |
| at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344) | |
| at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| at org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) | |
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) | |
| at java.base/java.lang.Thread.run(Thread.java:830) | |
| Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 | |
| at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| at org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| at org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| ... 19 common frames omitted | |
| 14:44:52.253 [task-result-getter-3] ERROR o.a.spark.scheduler.TaskSetManager - Task 11 in stage 38.0 failed 1 times; aborting job | |
| [error] ! certain inputs.aggregation - no group-by - aggregtion functions only | |
| [error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 38.0 failed 1 times, most recent failure: Lost task 11.0 in stage 38.0 (TID 173, 192.168.0.7, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080 | |
| [error] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344) | |
| [error] at org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) | |
| [error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] at org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) | |
| [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) | |
| [error] at java.base/java.lang.Thread.run(Thread.java:830) | |
| [error] Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 | |
| [error] at org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| [error] at org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| [error] at org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| [error] at org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| [error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source) | |
| [error] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| [error] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| [error] ... 19 more | |
| [error] | |
| [error] Driver stacktrace: (DAGScheduler.scala:1989) | |
| [error] org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1989) | |
| [error] org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1977) | |
| [error] org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1976) | |
| [error] org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1976) | |
| [error] org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:956) | |
| [error] org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:956) | |
| [error] org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:956) | |
| [error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2206) | |
| [error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2155) | |
| [error] org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2144) | |
| [error] org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) | |
| [error] org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:758) | |
| [error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2116) | |
| [error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2137) | |
| [error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2156) | |
| [error] org.apache.spark.SparkContext.runJob(SparkContext.scala:2181) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004) | |
| [error] org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| [error] org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| [error] org.apache.spark.rdd.RDD.withScope(RDD.scala:388) | |
| [error] org.apache.spark.rdd.RDD.collect(RDD.scala:1003) | |
| [error] org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:365) | |
| [error] org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3482) | |
| [error] org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2812) | |
| [error] org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3472) | |
| [error] org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100) | |
| [error] org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160) | |
| [error] org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87) | |
| [error] org.apache.spark.sql.Dataset.withAction(Dataset.scala:3468) | |
| [error] org.apache.spark.sql.Dataset.collect(Dataset.scala:2812) | |
| [error] org.mimirdb.utility.Bag$.apply(Bag.scala:41) | |
| [error] org.mimirdb.test.DataFrameMatchers.dfBagEquals(DataFrameMatchers.scala:14) | |
| [error] org.mimirdb.test.DataFrameMatchers.dfBagEquals$(DataFrameMatchers.scala:12) | |
| [error] org.mimirdb.caveats.LogicalPlanRangeSpec.dfBagEquals(LogicalPlanRangeSpec.scala:21) | |
| [error] org.mimirdb.test.DataFrameMatchers.$anonfun$beBagEqualsTo$2(DataFrameMatchers.scala:48) | |
| [error] org.mimirdb.caveats.LogicalPlanRangeSpec.$anonfun$annotBagEqualToDF$1(LogicalPlanRangeSpec.scala:79) | |
| [error] org.mimirdb.caveats.LogicalPlanRangeSpec.annotBagEqualToDF(LogicalPlanRangeSpec.scala:79) | |
| [error] org.mimirdb.caveats.LogicalPlanRangeSpec.$anonfun$new$8(LogicalPlanRangeSpec.scala:366) | |
| [error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344) | |
| [error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| [error] org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| [error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| [error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [error] CAUSED BY | |
| [error] java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, X), StringType), true, false) AS X#1075 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, __CAVEATS_ROW_LB), StringType), true, false) AS __CAVEATS_ROW_LB#1076 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, __CAVEATS_ROW_BG), StringType), true, false) AS __CAVEATS_ROW_BG#1077 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, __CAVEATS_ROW_UB), StringType), true, false) AS __CAVEATS_ROW_UB#1078 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, __CAVEATS_X_LB), StringType), true, false) AS __CAVEATS_X_LB#1079 | |
| [error] if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, __CAVEATS_X_UB), StringType), true, false) AS __CAVEATS_X_UB#1080 (ExpressionEncoder.scala:344) | |
| [error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:344) | |
| [error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| [error] org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| [error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| [error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [error] CAUSED BY | |
| [error] java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5 (rows.scala:174) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174) | |
| [error] org.apache.spark.sql.Row.isNullAt(Row.scala:204) | |
| [error] org.apache.spark.sql.Row.isNullAt$(Row.scala:204) | |
| [error] org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If_5$(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) | |
| [error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:340) | |
| [error] org.apache.spark.sql.SparkSession.$anonfun$createDataFrame$1(SparkSession.scala:350) | |
| [error] org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) | |
| [error] org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) | |
| [error] org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:726) | |
| [error] org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:321) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) | |
| [error] org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) | |
| [error] org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) | |
| [error] org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) | |
| [error] org.apache.spark.rdd.RDD.iterator(RDD.scala:313) | |
| [error] org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) | |
| [error] org.apache.spark.scheduler.Task.run(Task.scala:127) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:441) | |
| [error] org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) | |
| [error] org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:444) | |
| [info] TIP inputs | |
| [info] Caveated inputs | |
| [info] Total for specification LogicalPlanRangeSpec | |
| [info] Finished in 8 seconds, 245 ms | |
| [info] 1 example, 0 failure, 1 error | |
| [error] Error: Total 1, Failed 0, Errors 1, Passed 0 | |
| [error] Error during tests: | |
| [error] org.mimirdb.caveats.LogicalPlanRangeSpec | |
| [error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful | |
| [error] Total time: 16 s, completed Jun 11, 2020, 2:44:57 PM | |
| sbt:mimir-caveats> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment