Skip to content

Instantly share code, notes, and snippets.

View sungchun12's full-sized avatar
🧃
Juicin' up these open source contributor muscles in full force!

Sung Won Chung sungchun12

🧃
Juicin' up these open source contributor muscles in full force!
View GitHub Profile
@sungchun12
sungchun12 / cloudSettings
Last active July 8, 2019 21:42
Visual Studio Code Settings Sync Gist
{"lastUpload":"2019-07-08T21:42:32.794Z","extensionVersion":"v3.3.1"}
@sungchun12
sungchun12 / drop_dbt_cloud_pull_request_schemas.sql
Last active May 17, 2021 19:28
drop dbt Cloud pull request schemas to cleanup your snowflake database organization
--Run this command in your terminal: dbt run-operation drop_dbt_cloud_pull_request_schemas
--Write this as a file in your dbt project directory: macros/drop_dbt_cloud_pull_request_schemas.sql
--Original Source: https://github.com/randypitcherii/hashmap_randy_pitcher_workspace/blob/master/dbt/macros/cleanup/drop_old_relations.sql
{% macro drop_dbt_cloud_pull_request_schemas() %}
{% set cleanup_query %}
-- CTE to query the snowflake information schema to get all the tables and views in scope for cleanup
WITH
{% macro clone_prod_to_target(from) %}
{% set sql -%}
create schema if not exists {{ target.database }}.{{ target.schema }} clone {{ from }};
{%- endset %}
{{ dbt_utils.log_info("Cloning schema " ~ from ~ " into target schema.") }}
{% do run_query(sql) %}
@sungchun12
sungchun12 / table_constraints_demo.sql
Created May 10, 2022 21:03
BigQuery: Table with Constraints Custom Materialization
{{
config(
materialized = "table_with_constraints"
)
}}
select
1 as id,
'blue' as color,
cast('2019-01-01' as date) as date_day
-- force a dependency
-- depends_on: analytics.dbt_demo_account_sung.fct_orders
select *
from -- Need this here, since the actual ref is nested within loops/conditions:
-- depends on: analytics.dbt_demo_account_sung.dbt_metrics_default_calendar
(with source_query as (
select
/*
Welcome to your first dbt model!
Did you know that you can also configure models directly within SQL files?
This will override configurations stated in dbt_project.yml
Try changing "table" to "view" below
*/
@sungchun12
sungchun12 / zero_copy_clone_pre_prod.sql
Last active August 22, 2024 15:50
dbt staging to production gatekeeper macro to prevent problems happening in production after it's too late. Demo Video: https://www.loom.com/share/bcfd2cf3b4b5471683bfc5b24587db3d
{% macro clone_from_to(from, to) %}
{% set sql -%}
create schema if not exists {{ target.database }}.{{ to }} clone {{ from }};
{%- endset %}
{{ dbt_utils.log_info("Cloning tables/views from schema [" ~ from ~ "] into target schema [" ~ to ~ "]") }}
{% do run_query(sql) %}
@sungchun12
sungchun12 / fake_data_example.py
Created September 29, 2022 18:56
Use this to generate fake data in your dbt pipelines as an alternative to dbt seeds with csv files: https://www.loom.com/share/90084f27396746619d4f53f44143faab
from faker import Faker
import pandas as pd
fake = Faker()
def create_rows_faker(num=1):
output = [{"name":fake.name(),
"address":fake.address(),
"name":fake.name(),
"email":fake.email(),
@sungchun12
sungchun12 / constraints_example_snowflake_schema.yml
Last active November 4, 2022 21:22
Add column types, constraints, and default values when creating tables in dbt: https://www.loom.com/share/14020499f5f646b6bc80c909716850fd
version: 2
models:
- name: constraints_example
config:
constraints_enabled: true
columns:
- name: id
column_type: integer
description: I want to describe this one, but I don't want to list all the columns
@sungchun12
sungchun12 / Dockerfile
Created May 10, 2023 08:05
Simple quickstart to dbt Core DAG with bashoperator using BigQuery
FROM quay.io/astronomer/astro-runtime:7.4.2
# Install apache-airflow-providers-google package
RUN pip install apache-airflow-providers-google
# Switch to root user for installing git
USER root
# Install git
RUN apt-get update && apt-get install -y git