MaherSaif

ActiveRecord ETL

I created this to help me run benchmarks/comparisons against Universal ID, but it could serve as the foundation for a robust ETL data pipeline... and it's less than 70 LOC right now! 🤯 🚀

It handles the extract and transform parts of an ETL process and supports the following options:

• only - specify which attributes to include

MaherSaif

stdin → fzf with preview

Setup

1. Add the script below to your PATH
2. Pipe anything from stdin to infzf
3. Enjoy!

Usage

MaherSaif

class BlurhashAnalyzer < ActiveStorage::Analyzer::ImageAnalyzer::Vips
  def metadata
    read_image do |image|
      if rotated_image?(image)
        { width: image.height, height: image.width }
      else
        { width: image.width, height: image.height }
      end.merge blurhash(image)
    end
  end

MaherSaif

from PIL import Image
import os

input_folder = "./public/old"
output_folder = "./public"

if not os.path.exists(output_folder):
    os.makedirs(output_folder)

for filename in os.listdir(input_folder):

MaherSaif

# Builds an SQL insert query for a given record
#
# @param record [ActiveRecord::Base] Record used to build the SQL insert query
# @return [String] SQL insert query
def build_insert_query(record)
  columns = record.class.columns.reject { |col| col.name == record.class.primary_key }
  values = columns.map { |col| record[col.name] }
  insert_manager = Arel::InsertManager.new
  insert_manager.into(record.class.arel_table)
  insert_manager.insert(columns.zip(values)).to_sql

MaherSaif

# Make sure to have prepared statements enabled.
# config/database.yml
# test:
#  database: ...
#  prepared_statements: true

# Add this to the top scope of test/test_helper.rb or spec/rails_helper.rb and
# run tests as usual.

queries = Hash.new(0)

MaherSaif

# syntax = docker/dockerfile:1

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.2.2
FROM registry.docker.com/library/ruby:$RUBY_VERSION-slim as base

# Rails app lives here
WORKDIR /rails

# Set production environment

MaherSaif

# To display flash messages inside XHR responses,
# place this file in app/controllers/concerns/
# then include it in ApplicationController

module RenderFlashNowForXhr
  extend ActiveSupport::Concern

  private

  # Flash messages are not directly available for XHR requests

MaherSaif

  def active_link_to(name = nil, options = nil, **html_options, &block)
    options = block_given? ? name : options

    if current_page?(options)
      html_options[:class] = class_names(html_options[:class], :active)
      html_options[:aria] = html_options.fetch(:aria, {}).merge(current: :page)
    end

    if block_given?
      link_to options, html_options, &block

MaherSaif

# config/environments/test.rb

ActiveSupport.on_load(:active_record_postgresqladapter) do
  # For this optimization to work, you need to recreate your test database
  self.create_unlogged_tables = true
end