crisptrutski

System design: Python transforms in Metabase

Context

Metabase is an open-source analytics platform that runs on the JVM. It is available as both a self-hosted JAR and a cloud-hosted service. Customers connect Metabase to databases in a variety of environments, including databases outside Metabase's infrastructure and on-premises databases reached through SSH tunnels.

Metabase transforms let users build scheduled data pipelines. A "query transform" runs a saved query and writes its results to a target table in the same database. Dependencies between transforms form a directed acyclic graph, which Metabase uses to run them in the correct order. Today, each transform executes entirely inside the customer's database.

The challenge

crisptrutski

(ns the-grid-search)

;; Solves https://www.hackerrank.com/challenges/the-grid-search/problem

;; Checker has aggressive timeout
(set! *warn-on-reflection* true)
(set! *unchecked-math* true)

(defn first-index-of
  "An alias for the boilerplate to avoid reflection"

crisptrutski

;; Copyright (c) Alan Dipert. All rights reserved.
;; The use and distribution terms for this software are covered by the
;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
;; By using this software in any fashion, you are agreeing to be bound by
;; the terms of this license.
;; You must not remove this notice, or any other, from this software.

(ns alandipert.kahn
  (:require [clojure.set :refer [difference union intersection]]))

crisptrutski

;; Copyright (c) Alan Dipert. All rights reserved.
;; The use and distribution terms for this software are covered by the
;; Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
;; By using this software in any fashion, you are agreeing to be bound by
;; the terms of this license.
;; You must not remove this notice, or any other, from this software.

(ns alandipert.kahn
  (:require
    [clojure.set :refer [union]])

crisptrutski

Keybase proof

I hereby claim:

• I am crisptrutski on github.
• I am crisptrutski (https://keybase.io/crisptrutski) on keybase.
• I have a public key whose fingerprint is F58B C77B 9ECF C9DB 24FE 4832 9CC4 AAF3 BCF9 00F1

To claim this, I am signing this object:

crisptrutski

(def boot-version
  (get (boot.App/config) "BOOT_VERSION" "2.5.5"))

(deftask from-lein
  "Use project.clj as source of truth as far as possible"
  []
  (let [lein-proj (let [l (-> "project.clj" slurp read-string)]
                    (merge (->> l (drop 3) (partition 2) (map vec) (into {}))
                           {:project (second l) :version (nth l 2)}))]
    (merge-env! :repositories (:repositories lein-proj))

crisptrutski

(defn form->operation
  "Given sjacket nodes for a form, return name of the operation.
   Assumes operation is a literal."
  [nodes]
  (->> nodes
       (filter #(= :symbol (:tag %))) first :content
       (filter #(= :name   (:tag %))) first :content
       first))

crisptrutski

#!/usr/bin/ruby
files = Dir["reports*"]
files.each do |f|
  dir = File.join(`cat #{f}|grep DocumentRoot`.lines.first.split(/\s+/).last.split(/\//)[0..-2]) 
  rails_version = `cd #{dir}; rvm exec bundle list|grep ' rack'`.match(/\(([^)]+)\)/)[1]
  deployed_version = `cd #{dir}; cat version`
  deployed_branch = `cd #{dir}; cat config/environments/capistrano/#{f.to_s.gsub(/reports_/, '')}.rb|grep :branch`
  puts "#{f}: #{rails_version}@#{deployed_version} - #{deployed_branch}"
end