Created
April 25, 2022 19:35
-
-
Save vvgsrk/f0490965696592a01d397e9b4e0493c1 to your computer and use it in GitHub Desktop.
Generate row rank using event timestamp column with DBT incremental materialization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {# | |
| - this macro creates incremental tables based | |
| ON history DATA - it uses is_incremental() macro TO wrap valid SQL that filters THE rows - Filter will ONLY be applied | |
| ON an incremental run - it uses unique_key PARAMETER which ensures THE existing ROW IS updated IN target TABLE #} | |
| {%- macro generate_row_rank_using_event_ts_incremental(model, keycolumn) -%} | |
| {%- set columns = adapter.get_columns_in_relation(model) -%} | |
| {{ config(materialized = var('incremental'), unique_key = '__uuid') }} | |
| WITH ranked_data AS ( | |
| SELECT | |
| ROW_NUMBER() over ( | |
| PARTITION BY {{ keycolumn }} | |
| ORDER BY | |
| __event_ts DESC | |
| ) AS row_rank, | |
| * | |
| FROM {{ model }} | |
| {% if is_incremental() -%} | |
| WHERE __load_ts > ( SELECT MAX(__load_ts) | |
| FROM {{ this }} ) | |
| {% endif -%} | |
| ) | |
| SELECT | |
| {%- for column in columns %} | |
| "{{ column.name }}" {%- if not loop.last -%}, | |
| {%- endif -%} | |
| {%- endfor -%}, | |
| HASH( | |
| {{ keycolumn }} | |
| ) AS __uuid | |
| FROM | |
| ranked_data | |
| WHERE | |
| row_rank = 1 | |
| ORDER BY __load_ts desc | |
| {% endmacro %} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment