Created
March 20, 2021 06:22
-
-
Save tjwaterman99/abf0fda18ef7ea94f796cbf4ed9213f3 to your computer and use it in GitHub Desktop.
Example CI/CD process for dbt on Snowflake
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# For workflow syntax reference, please see the documentation available on Github | |
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions | |
name: CI | |
on: | |
push: | |
schedule: | |
- cron: '50 11 * * *' # Run the `main` branch daily at 11:50am UTC | |
workflow_dispatch: # Allow the workflow to be triggered manually | |
env: | |
SNOWSQL_USER: github | |
SNOWSQL_PWD: ${{ secrets.SNOWSQL_PWD }} | |
SNOWSQL_DATABASE: acme | |
SNOWSQL_SCHEMA: staging | |
SNOWSQL_WAREHOUSE: ci | |
SNOWSQL_ROLE: transformer | |
SNOWSQL_ACCOUNT: ${{ secrets.SNOWSQL_ACCOUNT }} | |
LC_ALL: C.UTF-8 | |
LANG: C.UTF-8 | |
jobs: | |
build: | |
runs-on: ubuntu-latest | |
container: | |
image: docker.pkg.github.com/tjwaterman99/snowflake_dbt/snowflake_dbt:0.1.5 | |
credentials: | |
username: ${{ github.actor }} | |
password: ${{ secrets.GITHUB_TOKEN }} | |
steps: | |
- name: Checkout | |
uses: actions/checkout@v2 | |
- name: pip install | |
run: pip install -r requirements.txt | |
- name: dbt debug | |
run: dbt debug --profiles-dir $PWD | |
- name: dbt deps | |
run: dbt deps --profiles-dir $PWD | |
# Reset the staging database by cloning production | |
- name: Reset staging database | |
run: snowsql -f scripts/reset_staging_db.sql | |
# Load seed files | |
- name: dbt seed | |
run: "dbt seed --full-refresh --profiles-dir $PWD" | |
# Build's the tables and views on the DWH's `staging` schema with dbt | |
- name: dbt run | |
run: dbt run --profiles-dir $PWD | |
# Test the resulting tables to identify any issues before the data lands in production | |
- name: dbt test | |
run: dbt test --profiles-dir $PWD | |
# Clone the staging schema into the production schema once the tests have | |
# validated the data | |
- name: Promote staging | |
run: snowsql -f scripts/promote_staging_db.sql | |
if: github.ref == 'refs/heads/main' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey @francisaddae - yes, this was intended as an illustration for running a CI/CD process for dbt on Snowflake.