Created
April 2, 2013 14:41
-
-
Save choplin/5292747 to your computer and use it in GitHub Desktop.
a bug of a predicate push down in hive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| create table src_tbl ( | |
| key string, | |
| value string | |
| ); | |
| create table tbl_a ( | |
| key string, | |
| value string | |
| ); | |
| create table tbl_b ( | |
| key string, | |
| value string | |
| ); | |
| EXPLAIN FROM ( | |
| FROM src_tbl | |
| SELECT TRANSFORM(key, value) | |
| USING 'cat' | |
| ) t | |
| INSERT OVERWRITE TABLE tbl_a SELECT key, value WHERE key = 'a' | |
| INSERT OVERWRITE TABLE tbl_b SELECT key, value WHERE key = 'b'; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hive> EXPLAIN FROM ( | |
| > FROM src_tbl | |
| > SELECT TRANSFORM(key, value) | |
| > USING 'cat' | |
| > ) t | |
| > INSERT OVERWRITE TABLE tbl_a SELECT key, value WHERE key = 'a' | |
| > INSERT OVERWRITE TABLE tbl_b SELECT key, value WHERE key = 'b'; | |
| OK | |
| ABSTRACT SYNTAX TREE: | |
| (TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src_tbl))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TRANSFORM (TOK_EXPLIST (TOK | |
| _TABLE_OR_COL key) (TOK_TABLE_OR_COL value)) TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER))))) t)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME tbl_a))) (TOK_SELECT (TOK_SELEXPR (TOK_ | |
| TABLE_OR_COL key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value))) (TOK_WHERE (= (TOK_TABLE_OR_COL key) 'a'))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME tbl_b))) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_CO | |
| L key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value))) (TOK_WHERE (= (TOK_TABLE_OR_COL key) 'b')))) | |
| STAGE DEPENDENCIES: | |
| Stage-2 is a root stage | |
| Stage-8 depends on stages: Stage-2 , consists of Stage-5, Stage-4, Stage-6 | |
| Stage-5 | |
| Stage-0 depends on stages: Stage-5, Stage-4, Stage-7 | |
| Stage-3 depends on stages: Stage-0 | |
| Stage-4 | |
| Stage-6 | |
| Stage-7 depends on stages: Stage-6 | |
| Stage-14 depends on stages: Stage-2 , consists of Stage-11, Stage-10, Stage-12 | |
| Stage-11 | |
| Stage-1 depends on stages: Stage-11, Stage-10, Stage-13 | |
| Stage-9 depends on stages: Stage-1 | |
| Stage-10 | |
| Stage-12 | |
| Stage-13 depends on stages: Stage-12 | |
| STAGE PLANS: | |
| Stage: Stage-2 | |
| Map Reduce | |
| Alias -> Map Operator Tree: | |
| t:src_tbl | |
| TableScan | |
| alias: src_tbl | |
| Select Operator | |
| expressions: | |
| expr: key | |
| type: string | |
| expr: value | |
| type: string | |
| outputColumnNames: _col0, _col1 | |
| Transform Operator | |
| command: cat | |
| output info: | |
| input format: org.apache.hadoop.mapred.TextInputFormat | |
| output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | |
| Filter Operator | |
| predicate: | |
| expr: ((_col0 = 'a') and (_col0 = 'b')) | |
| ... |
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
expr: ((_col0 = 'a') and (_col0 = 'b')) <- all inserts result in 0 row...