davetapley

SQL-Compatible Parquet Export Plan

Goal

Provide GIS and ETL consumers a read-only SQL interface to complete JEFDAQ tabular readings, including the 15-minute observations currently lost by hourly JSON scraping. The service must query materialized cache assets, not JEFDAQ's private ingest database.

Decision

Create a dedicated SqlExportRunner in the product layer. It opens an in-memory DuckDB catalog against the cache directory and exposes only curated, read-only views over tabular Parquet assets.

davetapley

from logging.config import dictConfig
from logging import getLogger
from pathlib import Path
from tempfile import TemporaryDirectory
from threading import Thread


def config(path: Path):
    return {
        'version': 1,

davetapley

# Use of script block for Group-Object -AsHashTable -Property breaks Keys.Clone()
# https://github.com/MicrosoftDocs/PowerShell-Docs/issues/8605

Write-Output "Using -Property Name"
# https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/group-object?view=powershell-7.2#example-1--group-files-by-extension
$files = Get-ChildItem -Path $PSHOME | Group-Object -Property Name -AsHashTable
Write-Output $files.Keys.count

# https://docs.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-hashtable?view=powershell-7.2#iterating-hashtables
$files.Keys.Clone() | ForEach-Object { $files[$_] = 'test' }

davetapley

{"lastUpload":"2020-12-10T20:16:43.484Z","extensionVersion":"v3.4.3"}

davetapley

Keybase proof

I hereby claim:

• I am davetapley on github.
• I am dukedave (https://keybase.io/dukedave) on keybase.
• I have a public key ASDOLHIgLOXrzDLHH3BcAyjkh-kCCVGr4NldjfVZeUHgewo

To claim this, I am signing this object:

davetapley

Keybase proof

I hereby claim:

• I am dukedave on github.
• I am dukedave (https://keybase.io/dukedave) on keybase.
• I have a public key ASDIrQmt-uAMZH32h-lsWcVEvO-Jv8tiDLW_d2KvXMh_FAo

To claim this, I am signing this object:

davetapley

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAu4AAALuCAMAAADGw2P4AAADAFBMVEWhnZbk5OJxbGG0sau7ubOuq6TBvrmkoZrDwbz5+fnc29ivlTHLycV6dmv29fX5/P+cmZHS0c30+f+BfXNuaV2dzv/s9f+RjYSBl0fi4d92al//1dXk8f/x+P/+3l2enHKOsz2ZlY2k0f/UrhuVkonV1NDV6v+JhXyJpkLp9P//mZmh0P/+uLjM5v/c7f/E4v//qqq12Wz/x8e83f97iE202f+rfHXTioas1f/z8vH9/fzh8P/u7uyx2P+p1P/Z7P+o01LS6f/39/d1cGV1fFLG4o7J5P+53P/QzsvB4P/GpiPW6q3a2da6gXyLfEewrqf/zADhuBKUdGz8+/uOioHf3tvf7/99eG7OzMjGhoHo6Obv9//+1zi/3/+bzf/q6ej/8LKKcWfY1tPsk5H/6IuGgnhrZlqopZ73+//8/v+ZzP/////IxsL/8/P/99XGxL/zlZT/6uqEf3Xx8e//9MZsZ1v/4ODs7Oq7nym4trD8/f+Vwjfo89H19PPqvg28zIqMiH6diTx/bWN/enDg8MLH1IWft2iTj4eXp2vKqqb/+eD0xQerqKHlkI6eeHDeu7iliYG2s63/oaH/0h+gz0Hf6cuotoL2+PrpysjwwgndjYrP5/9/dE78ygLRtUXX6//n8//0+en5l5fp7/Tw+OLXm5iav019cmaEhWGKl1+lq4r8mJjSyBd7e2H//PySvTn/++r4zR/b5vG/ko2YyjTswx+bzTdxbFecnYqZyzP6yQPvmpmuuo+lpJH/zxD9/fb/nJysqpzw3t39/vuZzDOXxzWwnJWNjXT/zQWXmYPw8vWzuJ3y9vnq8vqKhnTjoZ7V4+/t8vjm8fuRkXT7zAPwoaD+/v7+//94c2jv7+7n5uT9/v/s6+q+vLen0/+fz/+32//H4/+v1/9waFv7mJf5yAT//v73xgWrrq7e4M75/PLu9dvzzA

davetapley

Before we get started, the violinist subreddit is a fun place to hang out, lots of good questions and other beginners!

I've ended up taking something like a 4 pronged approach, I try to do at least an hour of each twice a week:

Music theory

• I've found this playlist and Michael's videos in general

davetapley

require 'simplecov'
require 'codeclimate-test-reporter'

module Extensions
  module Git
    module ClassMethods
      def info
        {
          head:         head,
          committed_at: committed_at_or_now,

davetapley

>> @store.start_of_day == 0.hours
=> true
>> options_for_select([['foo', 0.hours]], @store.start_of_day)
=> "<option value=\"0\">foo</option>"
>> options_for_select([['foo', 0.hours]], 0.hours)
=> "<option selected=\"selected\" value=\"0\">foo</option>"