Skip to content

Instantly share code, notes, and snippets.

@yashk
Last active January 20, 2017 14:18
Show Gist options
  • Save yashk/df26709f512125deab358d25eb94e6e6 to your computer and use it in GitHub Desktop.
Save yashk/df26709f512125deab358d25eb94e6e6 to your computer and use it in GitHub Desktop.
Which table HiveMetaStoreClient.getPartition queries and does that table have index on partitions

This code is exectuted to get parition list

private MPartition getMPartition(String dbName, String tableName, List<String> part_vals)
     .........
     ........
     query =
          pm.newQuery(MPartition.class,
              "table.tableName == t1 && table.database.name == t2 && partitionName == t3");
      query.declareParameters("java.lang.String t1, java.lang.String t2, java.lang.String t3");
      mparts = (List<MPartition>) query.execute(tableName, dbName, name);
      pm.retrieveAll(mparts);
      commited = commitTransaction();

Here MPartition class in mapped to table and we are doing

select * from where "table.tableName == t1 && table.database.name == t2 && partitionName == t3"

which table maps to MPartition ? Hive uses datanucleus - jdo orm framework to talk to db , so package.jdo maps java class to db table

<class name="MPartition" table="PARTITIONS" identity-type="datastore" detachable="true">
      <index name="UniquePartition" unique="true">
        <column name="PART_NAME"/>
        <column name="TBL_ID"/>
      </index>
      <datastore-identity>
        <column name="PART_ID"/>
      </datastore-identity>
      <field name="partitionName">

This shows PARTITIONS table has index on PART_NAME and TBL_ID

desc PARTITIONS;
+------------------+--------------+--------+-------+-----------+---------+
| Field            | Type         | Null   | Key   |   Default | Extra   |
|------------------+--------------+--------+-------+-----------+---------|
| PART_ID          | bigint(20)   | NO     | PRI   |    <null> |         |
| CREATE_TIME      | int(11)      | NO     |       |    <null> |         |
| LAST_ACCESS_TIME | int(11)      | NO     |       |    <null> |         |
| PART_NAME        | varchar(767) | YES    | MUL   |    <null> |         |
| SD_ID            | bigint(20)   | YES    | MUL   |    <null> |         |
| TBL_ID           | bigint(20)   | YES    | MUL   |    <null> |         |
| LINK_TARGET_ID   | bigint(20)   | YES    | MUL   |    <null> |         |
+------------------+--------------+--------+-------+-----------+---------+
SHOW INDEX FROM PARTITIONS;
+------------+--------------+-----------------+----------------+----------------+-------------+---------------+------------+----------+--------+--------------+-----------+-----------------+
| Table      |   Non_unique | Key_name        |   Seq_in_index | Column_name    | Collation   |   Cardinality |   Sub_part |   Packed | Null   | Index_type   | Comment   | Index_comment   |
|------------+--------------+-----------------+----------------+----------------+-------------+---------------+------------+----------+--------+--------------+-----------+-----------------|
| PARTITIONS |            0 | PRIMARY         |              1 | PART_ID        | A           |        215346 |     <null> |   <null> |        | BTREE        |           |                 |
| PARTITIONS |            0 | UNIQUEPARTITION |              1 | PART_NAME      | A           |        215346 |     <null> |   <null> | YES    | BTREE        |           |                 |
| PARTITIONS |            0 | UNIQUEPARTITION |              2 | TBL_ID         | A           |        215346 |     <null> |   <null> | YES    | BTREE        |           |                 |
| PARTITIONS |            1 | PARTITIONS_N49  |              1 | TBL_ID         | A           |           159 |     <null> |   <null> | YES    | BTREE        |           |                 |
| PARTITIONS |            1 | PARTITIONS_N50  |              1 | SD_ID          | A           |        215346 |     <null> |   <null> | YES    | BTREE        |           |                 |
| PARTITIONS |            1 | PARTITIONS_N51  |              1 | LINK_TARGET_ID | A           |             2 |     <null> |   <null> | YES    | BTREE        |           |                 |
+------------+--------------+-----------------+----------------+----------------+-------------+---------------+------------+----------+--------+--------------+-----------+-----------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment