Skip to content

Instantly share code, notes, and snippets.

@choplin
Created April 2, 2013 14:41
Show Gist options
  • Select an option

  • Save choplin/5292747 to your computer and use it in GitHub Desktop.

Select an option

Save choplin/5292747 to your computer and use it in GitHub Desktop.
a bug of a predicate push down in hive
create table src_tbl (
key string,
value string
);
create table tbl_a (
key string,
value string
);
create table tbl_b (
key string,
value string
);
EXPLAIN FROM (
FROM src_tbl
SELECT TRANSFORM(key, value)
USING 'cat'
) t
INSERT OVERWRITE TABLE tbl_a SELECT key, value WHERE key = 'a'
INSERT OVERWRITE TABLE tbl_b SELECT key, value WHERE key = 'b';
hive> EXPLAIN FROM (
> FROM src_tbl
> SELECT TRANSFORM(key, value)
> USING 'cat'
> ) t
> INSERT OVERWRITE TABLE tbl_a SELECT key, value WHERE key = 'a'
> INSERT OVERWRITE TABLE tbl_b SELECT key, value WHERE key = 'b';
OK
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src_tbl))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TRANSFORM (TOK_EXPLIST (TOK
_TABLE_OR_COL key) (TOK_TABLE_OR_COL value)) TOK_SERDE TOK_RECORDWRITER 'cat' TOK_SERDE TOK_RECORDREADER))))) t)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME tbl_a))) (TOK_SELECT (TOK_SELEXPR (TOK_
TABLE_OR_COL key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value))) (TOK_WHERE (= (TOK_TABLE_OR_COL key) 'a'))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME tbl_b))) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_CO
L key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value))) (TOK_WHERE (= (TOK_TABLE_OR_COL key) 'b'))))
STAGE DEPENDENCIES:
Stage-2 is a root stage
Stage-8 depends on stages: Stage-2 , consists of Stage-5, Stage-4, Stage-6
Stage-5
Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
Stage-3 depends on stages: Stage-0
Stage-4
Stage-6
Stage-7 depends on stages: Stage-6
Stage-14 depends on stages: Stage-2 , consists of Stage-11, Stage-10, Stage-12
Stage-11
Stage-1 depends on stages: Stage-11, Stage-10, Stage-13
Stage-9 depends on stages: Stage-1
Stage-10
Stage-12
Stage-13 depends on stages: Stage-12
STAGE PLANS:
Stage: Stage-2
Map Reduce
Alias -> Map Operator Tree:
t:src_tbl
TableScan
alias: src_tbl
Select Operator
expressions:
expr: key
type: string
expr: value
type: string
outputColumnNames: _col0, _col1
Transform Operator
command: cat
output info:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Filter Operator
predicate:
expr: ((_col0 = 'a') and (_col0 = 'b'))
...
@choplin
Copy link
Author

choplin commented Apr 2, 2013

expr: ((_col0 = 'a') and (_col0 = 'b')) <- all inserts result in 0 row...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment