databricks_cli.md

Databricks CLI Setup and Production PySpark Workflow

Prerequisites

• macOS (Apple Silicon or Intel)
• Homebrew installed
• Git installed
• Databricks Workspace
• Databricks account with appropriate permissions

---

Step 1. Check Existing Databricks CLI

Check whether Databricks CLI is already installed.

which databricks

Check the installed version.

databricks --version

Example (old CLI):

Version: 0.17.8

The legacy CLI (0.x) does not support:

databricks auth login

---

Step 2. Remove Legacy CLI (Optional)

If installed using pip:

pip uninstall databricks-cli

or

pip3 uninstall databricks-cli

If installed using Homebrew:

brew uninstall databricks

---

Step 3. Add Databricks Homebrew Repository

brew tap databricks/tap

---

Step 4. Trust the Databricks Homebrew Tap

Newer versions of Homebrew require explicit trust.

Trust the entire tap:

brew trust databricks/tap

or trust only the Databricks formula:

brew trust --formula databricks/tap/databricks

---

Step 5. Install the New Databricks CLI

brew install databricks

---

Step 6. Verify Installation

databricks version

Expected output:

Databricks CLI v1.7.0

---

Step 7. Authenticate to Databricks

Login to your workspace.

databricks auth login --host https://dbc-d40d7b72-eeb2.cloud.databricks.com

When prompted:

Databricks profile name [dbc-d40d7b72-eeb2]:

Press Enter to use the default profile or provide a custom profile name.

The CLI opens a browser for authentication.

After successful login, your credentials are stored in:

~/.databrickscfg

---

Step 8. Validate Authentication

List available authentication profiles.

databricks auth profiles

Display information about the currently authenticated user.

databricks current-user me

Expected output:

{
  "userName": "your-email@company.com",
  ...
}

---

Step 9. Verify Active Profile

cat ~/.databrickscfg

Example:

[dbc-d40d7b72-eeb2]
host = https://dbc-d40d7b72-eeb2.cloud.databricks.com
auth_type = pat

or

auth_type = oauth

---

Step 10. Create a Git Repository

git init
git add .
git commit -m "Initial Commit"
git remote add origin https://github.com/company/customer-etl.git
git push origin main

---

Step 11. Connect Git Repository to Databricks

Inside the Databricks workspace:

1. Workspace
2. Repos
3. Add Repo
4. Connect GitHub
5. Select Repository

Repository structure:

Repos
 └── customer-etl
      ├── src
      ├── configs
      └── requirements.txt

---

Step 12. Create a Compute Cluster

Navigate to:

Compute

Click:

Create Compute

Recommended settings:

• Databricks Runtime LTS
• Photon Enabled
• Autoscaling
• Job Cluster (Production)

---

Step 13. Test the PySpark Script

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read.csv("/mnt/raw/customer.csv", header=True)

df.write.mode("overwrite").format("delta").save("/mnt/gold/customer")

Execute the script interactively from the Databricks workspace.

---

Step 14. Create a Workflow Job

Navigate to:

Workflows

Create:

New Job

Configure:

• Task Type: Python Script
• Path:

/Repos/customer-etl/src/etl.py

Select a cluster or create a Job Cluster.

---

Step 15. Configure Job Parameters

Example:

--env=prod
--date=2026-07-11

Python example:

import argparse

parser = argparse.ArgumentParser()

parser.add_argument("--env")
parser.add_argument("--date")

args = parser.parse_args()

---

Step 16. Install Required Libraries

Example:

pandas
pyarrow
delta-spark
requests

or install from:

requirements.txt

---

Step 17. Schedule the Job

Examples:

• Hourly
• Daily
• Weekly
• Cron

Cron example:

0 2 * * *

Runs daily at 2:00 AM.

---

Step 18. Run the Job

Manual execution:

Run Now

Or using the Databricks CLI:

databricks bundle run customer-etl

---

Step 19. Deploy Using Databricks Asset Bundles

Deploy:

databricks bundle deploy

Validate:

databricks bundle validate

Run:

databricks bundle run customer-etl

---

Step 20. Monitor Jobs

Navigate to:

Workflows

View:

• Job Status
• Run History
• Logs
• Spark UI
• Cluster Details
• Failed Tasks

---

Production Workflow

Developer
    │
    ▼
Write PySpark Code
    │
    ▼
Git Commit
    │
    ▼
Git Push
    │
    ▼
Databricks Repos
    │
    ▼
Workflow Job
    │
    ▼
Cluster Starts
    │
    ▼
PySpark Executes
    │
    ▼
Delta Tables
    │
    ▼
Job Logs
    │
    ▼
Monitoring

---

Useful Databricks CLI Commands

databricks version

databricks auth login --host https://dbc-d40d7b72-eeb2.cloud.databricks.com

databricks auth profiles

databricks current-user me

databricks bundle validate

databricks bundle deploy

databricks bundle run customer-etl

---

Summary

This workflow covers:

• Installing the Databricks CLI
• Trusting the Homebrew tap
• Authenticating to Databricks
• Validating authentication
• Connecting Git repositories
• Creating compute clusters
• Running PySpark jobs
• Creating Databricks Workflow jobs
• Scheduling production jobs
• Deploying with Databricks Asset Bundles
• Monitoring production executions