smurching

~ ❯ npx create-react-app my-app                                                                                                                   Py universe Node 16.13.1 03:18:59 PM

Need to install the following packages:
  create-react-app
Ok to proceed? (y) y
npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.

Creating a new React app in /Users/sid.murching/my-app.

Installing packages. This might take a couple of minutes.

smurching

This gist contains some example bash scripts for triggering and awaiting a one-time job run using existing Databricks CLI APIs.

Rough edges include:

1. Parameter substitution into job JSON (need to implement this ourselves)
2. Writing logic to trigger and await job status
3. Updatability of shell script logic. Any customers that rely on this script would need to update it themselves, whereas updates could easily be pushed to an existing databricks runs submit CLI command with a --wait option (e.g. updating the default job polling interval). However, since we use the Databricks CLI for all API requests, any security/auth patches can be fetched by updating the version of the CLI used in the script.

#2 can be addressed through a --wait option to databricks runs submit. #1 requires implementing parameter substitution and so may be more work, but also isn't as complex - there isn't any branching logic to test, just that parameters are properly passed through.

smurching

# Databricks notebook source
##################################################################################
# Model Training Notebook
##
# This notebook runs the MLflow Regression Pipeline to train and registers an MLflow model in the model registry.
#
# It's run as part of CI (to integration-test model training logic) and by an automated model training job
# defined under ``databricks-config``
#
# NOTE: In general, we recommend that you do not modify this notebook directly, and instead update data-loading