NodeJSmith

Table of Contents

• Table of Contents
• So you wanna run dbt on a Databricks job cluster
• Overview
• Existing Literature
• How this differs from the standard process
  • Databricks SQL Warehouse
  • Databricks All Purpose Cluster
  • Databricks Job Cluster
• Steps

NodeJSmith

  - job: precommit
    pool:
      vmImage: ubuntu-latest
    variables:
      PYTHON_VERSION: 3.11
      VENV_ENV_NAME: .venv
      PRE_COMMIT_HOME: $(Pipeline.Workspace)/pre-commit-cache

    steps:
      - task: UsePythonVersion@0

NodeJSmith

Background

Running Python scripts in CI pipelines on Windows agents can sometimes cause UnicodeEncodeErrors that don't seem to make any sense. For example, running a simple command with a tool built in click or running dbx build or prefect build may run fine on your local Windows machine, but fail when running in a CI pipeline.

For example, running the below code:

# example.py
import click