This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Demo script to run a xdb data diff between Databricks and Snowflake with simple API calls | |
""" | |
import os | |
import time | |
from pydantic import BaseModel | |
from typing import Any, List, Optional | |
import requests | |
from tabulate import tabulate |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM quay.io/astronomer/astro-runtime:7.6.0 | |
# Install python packages | |
RUN pip install -r requirements.txt | |
# Switch to root user for installing git | |
USER root | |
# Install git | |
RUN apt-get update && apt-get install -y git |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM quay.io/astronomer/astro-runtime:7.4.2 | |
# Install apache-airflow-providers-google package | |
RUN pip install apache-airflow-providers-google | |
# Switch to root user for installing git | |
USER root | |
# Install git | |
RUN apt-get update && apt-get install -y git |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: 2 | |
models: | |
- name: constraints_example | |
config: | |
constraints_enabled: true | |
columns: | |
- name: id | |
column_type: integer | |
description: I want to describe this one, but I don't want to list all the columns |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from faker import Faker | |
import pandas as pd | |
fake = Faker() | |
def create_rows_faker(num=1): | |
output = [{"name":fake.name(), | |
"address":fake.address(), | |
"name":fake.name(), | |
"email":fake.email(), |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{% macro clone_from_to(from, to) %} | |
{% set sql -%} | |
create schema if not exists {{ target.database }}.{{ to }} clone {{ from }}; | |
{%- endset %} | |
{{ dbt_utils.log_info("Cloning tables/views from schema [" ~ from ~ "] into target schema [" ~ to ~ "]") }} | |
{% do run_query(sql) %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/* | |
Welcome to your first dbt model! | |
Did you know that you can also configure models directly within SQL files? | |
This will override configurations stated in dbt_project.yml | |
Try changing "table" to "view" below | |
*/ | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- force a dependency | |
-- depends_on: analytics.dbt_demo_account_sung.fct_orders | |
select * | |
from -- Need this here, since the actual ref is nested within loops/conditions: | |
-- depends on: analytics.dbt_demo_account_sung.dbt_metrics_default_calendar | |
(with source_query as ( | |
select |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{{ | |
config( | |
materialized = "table_with_constraints" | |
) | |
}} | |
select | |
1 as id, | |
'blue' as color, | |
cast('2019-01-01' as date) as date_day |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{% macro clone_prod_to_target(from) %} | |
{% set sql -%} | |
create schema if not exists {{ target.database }}.{{ target.schema }} clone {{ from }}; | |
{%- endset %} | |
{{ dbt_utils.log_info("Cloning schema " ~ from ~ " into target schema.") }} | |
{% do run_query(sql) %} |
NewerOlder