Skip to content

Instantly share code, notes, and snippets.

@milenkovicm

milenkovicm/data_fusion_plan.txt

Created November 22, 2025 16:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save milenkovicm/8850460d772ce145e53d91a58a71ee25 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save milenkovicm/8850460d772ce145e53d91a58a71ee25 to your computer and use it in GitHub Desktop.
Download ZIP
Q72
Raw
data_fusion_plan.txt
This file has been truncated, but you can view the full file.
DataFrame()
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | SortPreservingMergeExec: [total_cnt@5 DESC, i_item_desc@0 ASC NULLS LAST, w_warehouse_name@1 ASC NULLS LAST, d_week_seq@2 ASC NULLS LAST], fetch=100, metrics=[output_rows=100, elapsed_compute=12µs] |
| | SortExec: TopK(fetch=100), expr=[total_cnt@5 DESC, i_item_desc@0 ASC NULLS LAST, w_warehouse_name@1 ASC NULLS LAST, d_week_seq@2 ASC NULLS LAST], preserve_partitioning=[true], filter=[total_cnt@5 IS NULL OR total_cnt@5 > 1 OR total_cnt@5 = 1 AND i_item_desc@0 < Activities would help. Universally great powers used to throw now problems. In order efficient trees operate quite local, left genes. Hands can testify of OR total_cnt@5 = 1 AND i_item_desc@0 = Activities would help. Universally great powers used to throw now problems. In order efficient trees operate quite local, left genes. Hands can testify of AND w_warehouse_name@1 < Plain, reluctant OR total_cnt@5 = 1 AND i_item_desc@0 = Activities would help. Universally great powers used to throw now problems. In order efficient trees operate quite local, left genes. Hands can testify of AND w_warehouse_name@1 = Plain, reluctant AND d_week_seq@2 < 5182], metrics=[output_rows=1349, elapsed_compute=3.42721ms, row_replacements=4473] |
| | ProjectionExec: expr=[i_item_desc@0 as i_item_desc, w_warehouse_name@1 as w_warehouse_name, d_week_seq@2 as d_week_seq, sum(CASE WHEN promotion.p_promo_sk IS NULL THEN Int64(1) ELSE Int64(0) END)@3 as no_promo, sum(CASE WHEN promotion.p_promo_sk IS NOT NULL THEN Int64(1) ELSE Int64(0) END)@4 as promo, count(Int64(1))@5 as total_cnt], metrics=[output_rows=42545, elapsed_compute=6.251µs] |
| | AggregateExec: mode=FinalPartitioned, gby=[i_item_desc@0 as i_item_desc, w_warehouse_name@1 as w_warehouse_name, d_week_seq@2 as d_week_seq], aggr=[sum(CASE WHEN promotion.p_promo_sk IS NULL THEN Int64(1) ELSE Int64(0) END), sum(CASE WHEN promotion.p_promo_sk IS NOT NULL THEN Int64(1) ELSE Int64(0) END), count(Int64(1))], metrics=[output_rows=42545, elapsed_compute=4.253962ms, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, peak_mem_used=11531816] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=43188, elapsed_compute=3.262586ms] |
| | RepartitionExec: partitioning=Hash([i_item_desc@0, w_warehouse_name@1, d_week_seq@2], 14), input_partitions=14, metrics=[fetch_time=3095.959307168s, repartition_time=2.088789ms, send_time=864.368µs] |
| | AggregateExec: mode=Partial, gby=[i_item_desc@1 as i_item_desc, w_warehouse_name@0 as w_warehouse_name, d_week_seq@2 as d_week_seq], aggr=[sum(CASE WHEN promotion.p_promo_sk IS NULL THEN Int64(1) ELSE Int64(0) END), sum(CASE WHEN promotion.p_promo_sk IS NOT NULL THEN Int64(1) ELSE Int64(0) END), count(Int64(1))], metrics=[output_rows=43188, elapsed_compute=7.286545ms, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, skipped_aggregation_rows=0, peak_mem_used=11620768] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=43245, elapsed_compute=34.678µs] |
| | HashJoinExec: mode=Partitioned, join_type=Right, on=[(cr_item_sk@0, cs_item_sk@0), (cr_order_number@1, cs_order_number@1)], projection=[w_warehouse_name@4, i_item_desc@5, d_week_seq@6, p_promo_sk@7], metrics=[output_rows=43245, elapsed_compute=51.105649ms, build_input_batches=178, build_input_rows=1439749, input_batches=14, input_rows=43245, output_batches=14, build_mem_used=42713912, build_time=43.685873ms, join_time=1.065218ms] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=1439749, elapsed_compute=4.282035ms] |
| | RepartitionExec: partitioning=Hash([cr_item_sk@0, cr_order_number@1], 14), input_partitions=14, metrics=[fetch_time=13.943527ms, repartition_time=18.721383ms, send_time=2.357345ms] |
| | DataSourceExec: file_groups={14 groups: [[Users/marko/TMP/tpcds_data/10g_parquet/catalog_returns.parquet:0..7145905], [Users/marko/TMP/tpcds_data/10g_parquet/catalog_returns.parquet:7145905..14291810], [Users/marko/TMP/tpcds_data/10g_parquet/catalog_returns.parquet:14291810..21437715], [Users/marko/TMP/tpcds_data/10g_parquet/catalog_returns.parquet:21437715..28583620], [Users/marko/TMP/tpcds_data/10g_parquet/catalog_returns.parquet:28583620..35729525], ...]}, projection=[cr_item_sk, cr_order_number], file_type=parquet, metrics=[output_rows=1439749, elapsed_compute=14ns, batches_split=0, bytes_scanned=9439807, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=28ns, metadata_load_time=1.051137ms, page_index_eval_time=28ns, row_pushdown_eval_time=28ns, statistics_eval_time=28ns, time_elapsed_opening=1.506456ms, time_elapsed_processing=11.376863ms, time_elapsed_scanning_total=34.000245ms, time_elapsed_scanning_until_data=7.792623ms] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=43245, elapsed_compute=10.471177ms] |
| | RepartitionExec: partitioning=Hash([cs_item_sk@0, cs_order_number@1], 14), input_partitions=14, metrics=[fetch_time=3095.664858041s, repartition_time=873.29µs, send_time=757.834µs] |
| | ProjectionExec: expr=[cs_item_sk@1 as cs_item_sk, cs_order_number@2 as cs_order_number, w_warehouse_name@3 as w_warehouse_name, i_item_desc@4 as i_item_desc, d_week_seq@5 as d_week_seq, p_promo_sk@0 as p_promo_sk], metrics=[output_rows=43245, elapsed_compute=7.873µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=43245, elapsed_compute=37.138µs] |
| | HashJoinExec: mode=CollectLeft, join_type=Right, on=[(p_promo_sk@0, cs_promo_sk@1)], projection=[p_promo_sk@0, cs_item_sk@1, cs_order_number@3, w_warehouse_name@4, i_item_desc@5, d_week_seq@6], metrics=[output_rows=43245, elapsed_compute=874.42µs, build_input_batches=1, build_input_rows=500, input_batches=14, input_rows=43245, output_batches=14, build_mem_used=19464, build_time=151.832µs, join_time=669.046µs] |
| | DataSourceExec: file_groups={1 group: [[Users/marko/TMP/tpcds_data/10g_parquet/promotion.parquet]]}, projection=[p_promo_sk], file_type=parquet, metrics=[output_rows=500, elapsed_compute=1ns, batches_split=0, bytes_scanned=2641, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=2ns, metadata_load_time=29.376µs, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, time_elapsed_opening=39.416µs, time_elapsed_processing=99.962µs, time_elapsed_scanning_total=535.793µs, time_elapsed_scanning_until_data=505.5µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=43245, elapsed_compute=43.844µs] |
| | HashJoinExec: mode=Partitioned, join_type=Inner, on=[(d_date_sk@0, cs_ship_date_sk@0)], filter=d_date@1 > d_date@0 + IntervalMonthDayNano { months: 0, days: 5, nanoseconds: 0 }, projection=[cs_item_sk@3, cs_promo_sk@4, cs_order_number@5, w_warehouse_name@6, i_item_desc@7, d_week_seq@9], metrics=[output_rows=43245, elapsed_compute=4.558974ms, build_input_batches=14, build_input_rows=73049, input_batches=14, input_rows=45322, output_batches=14, build_mem_used=2534872, build_time=1.633291ms, join_time=2.277627ms] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=73049, elapsed_compute=247.594µs] |
| | RepartitionExec: partitioning=Hash([d_date_sk@0], 14), input_partitions=14, metrics=[fetch_time=6.272958ms, repartition_time=790.301µs, send_time=164.612µs] |
| | RepartitionExec: partitioning=RoundRobinBatch(14), input_partitions=1, metrics=[fetch_time=460.459µs, repartition_time=1ns, send_time=7.92µs] |
| | DataSourceExec: file_groups={1 group: [[Users/marko/TMP/tpcds_data/10g_parquet/date_dim.parquet]]}, projection=[d_date_sk, d_date], file_type=parquet, metrics=[output_rows=73049, elapsed_compute=1ns, batches_split=0, bytes_scanned=875218, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=2ns, metadata_load_time=27.917µs, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, time_elapsed_opening=38.792µs, time_elapsed_processing=322.792µs, time_elapsed_scanning_total=424.292µs, time_elapsed_scanning_until_data=261.459µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=45322, elapsed_compute=6.189345ms] |
| | RepartitionExec: partitioning=Hash([cs_ship_date_sk@0], 14), input_partitions=14, metrics=[fetch_time=3095.594617373s, repartition_time=2.649838ms, send_time=681.655µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=45322, elapsed_compute=9.488566ms] |
| | HashJoinExec: mode=Partitioned, join_type=Inner, on=[(d_date_sk@0, inv_date_sk@4), (d_week_seq@1, d_week_seq@8)], projection=[cs_ship_date_sk@2, cs_item_sk@3, cs_promo_sk@4, cs_order_number@5, w_warehouse_name@7, i_item_desc@8, d_date@9, d_week_seq@10], metrics=[output_rows=45322, elapsed_compute=44.508279ms, build_input_batches=14, build_input_rows=73049, input_batches=933, input_rows=7899706, output_batches=1849, build_mem_used=2534872, build_time=1.150336ms, join_time=43.014718ms] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=73049, elapsed_compute=204µs] |
| | RepartitionExec: partitioning=Hash([d_date_sk@0, d_week_seq@1], 14), input_partitions=14, metrics=[fetch_time=5.72167ms, repartition_time=896.253µs, send_time=108.615µs] |
| | RepartitionExec: partitioning=RoundRobinBatch(14), input_partitions=1, metrics=[fetch_time=416.42µs, repartition_time=1ns, send_time=12.504µs] |
| | DataSourceExec: file_groups={1 group: [[Users/marko/TMP/tpcds_data/10g_parquet/date_dim.parquet]]}, projection=[d_date_sk, d_week_seq], file_type=parquet, metrics=[output_rows=73049, elapsed_compute=1ns, batches_split=0, bytes_scanned=564347, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=2ns, metadata_load_time=26.959µs, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, time_elapsed_opening=35.625µs, time_elapsed_processing=287.666µs, time_elapsed_scanning_total=385.835µs, time_elapsed_scanning_until_data=210.791µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=7899706, elapsed_compute=603.421298ms] |
| | RepartitionExec: partitioning=Hash([inv_date_sk@4, d_week_seq@8], 14), input_partitions=14, metrics=[fetch_time=3091.994912168s, repartition_time=238.415504ms, send_time=68.609008ms] |
| | ProjectionExec: expr=[cs_ship_date_sk@2 as cs_ship_date_sk, cs_item_sk@3 as cs_item_sk, cs_promo_sk@4 as cs_promo_sk, cs_order_number@5 as cs_order_number, inv_date_sk@6 as inv_date_sk, w_warehouse_name@7 as w_warehouse_name, i_item_desc@8 as i_item_desc, d_date@0 as d_date, d_week_seq@1 as d_week_seq], metrics=[output_rows=7899706, elapsed_compute=1.164295ms] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=7899706, elapsed_compute=639.84867ms] |
| | HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(d_date_sk@0, cs_sold_date_sk@0)], projection=[d_date@1, d_week_seq@2, cs_ship_date_sk@4, cs_item_sk@5, cs_promo_sk@6, cs_order_number@7, inv_date_sk@8, w_warehouse_name@9, i_item_desc@10], metrics=[output_rows=7899706, elapsed_compute=202.497007ms, build_input_batches=1, build_input_rows=365, input_batches=3421, input_rows=30402441, output_batches=6825, build_mem_used=13140, build_time=19.005µs, join_time=202.478ms] |
| | CoalescePartitionsExec, metrics=[output_rows=365, elapsed_compute=5.75µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=365, elapsed_compute=1.799µs] |
| | FilterExec: d_year@3 = 1999, projection=[d_date_sk@0, d_date@1, d_week_seq@2], metrics=[output_rows=365, elapsed_compute=22.594µs] |
| | RepartitionExec: partitioning=RoundRobinBatch(14), input_partitions=1, metrics=[fetch_time=3.398833ms, repartition_time=1ns, send_time=6.969µs] |
| | DataSourceExec: file_groups={1 group: [[Users/marko/TMP/tpcds_data/10g_parquet/date_dim.parquet]]}, projection=[d_date_sk, d_date, d_week_seq, d_year], file_type=parquet, predicate=d_year@3 = 1999, pruning_predicate=d_year_null_count@2 != row_count@3 AND d_year_min@0 <= 1999 AND 1999 <= d_year_max@1, required_guarantees=[d_year in (1999)], metrics=[output_rows=20480, elapsed_compute=1ns, batches_split=0, bytes_scanned=742777, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=20480, page_index_rows_pruned=52569, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=10.584µs, metadata_load_time=50.584µs, page_index_eval_time=9.918µs, row_pushdown_eval_time=2ns, statistics_eval_time=9.084µs, time_elapsed_opening=136.584µs, time_elapsed_processing=3.037041ms, time_elapsed_scanning_total=3.171709ms, time_elapsed_scanning_until_data=508.333µs] |
| | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=30402441, elapsed_compute=2.208124186s]
Raw
spark_plan.txt
View raw

(Sorry about that, but we can’t show files that are this big right now.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment