NickCrews

I was uploading many csv's in parallel to MyCampaign, for a total of ~660k contacts across 17 uploads, each with 39,998 contacts per csv.

I discovered this by looking at the successful uploads, and all of them have between 50-150 contacts that were "updated". What I expected was that 0 contacts to be updated, and all of them to be created.

In one of the uploads, this is what the csv sheet_01.csv looks like:

person__id,voterfile_vanid,occupation,employer,prefix,first,middle,last,suffix,nickname

NickCrews

[
  {
    "congress_year":2025,
    "vote_date":"July 9, 2025,  06:01 PM",
    "vote_question":"On the Nomination PN12-24",
    "vote_result":"Nomination Confirmed",
    "vote_result_text":"Nomination Confirmed (49-46)",
    "vote_title":"Confirmation: Scott Kupor, of California, to be Director of the Office of Personnel Management",
    "majority_requirement":"1\/2",
    "document_congress":119,

NickCrews

from typing import Iterable, Callable, TypeVar

T = TypeVar("T")
R = TypeVar("R")

def make_none_safe(func: Callable[[Iterable[T]], Iterable[R]], *, batch_size: int | None = None) -> Callable[[Iterable[T | None]], Iterable[R]]:
    """Turn `iterable -> iterable` function into one that is safe for None values.
    
    Consider if you have a function of the form `Iterable[T] -> Iterable[R]`,
    and this function is delicate and will raise an error if it encounters

NickCrews

/*
 * Title: Sync Self-Links
 * License: Public Domain
 * Author: Nick Crews
 * 
 * Description: 
 * You would want to self-link in certain recursive situations.
 * For example, modeling a table of Employees:
 * an employee should have both "manager" and "manages" fields,
 * each of which link back to the same Employees table.

NickCrews

#!/bin/bash

# This script takes the FEC's PostgreSQL dump file and converts it to a directory
# of parquet files.
# See https://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74.s3-us-gov-west-1.amazonaws.com/bulk-downloads/index.html?prefix=bulk-downloads/data-dump/schedules/
# for the PostgreSQL dump file and more info.
# 
# This requires you to
# 1. Have Docker installed and running
# 2. Have the `duckdb` command line tool installed

NickCrews

Adding Strava Global Heatmap to Gaia

This adds the Strava Global Heatmap layer to Gaia GPS, so you can see common tracks on where other people have been outside. Like this:

Steps

1. Log into https://www.gaiagps.com/map
2. On the left sidebar, go to Layers

NickCrews

"""Get the Facebook page IDs for a set of facebook URLs.

Uses Playwright to emulate me going to the page and looking at the page ID in the
actual HTML. I haven't found a more programmatic way to do this without
more complicated developer signup and API keys.

Uses cookies for authentication, as described in
https://github.com/kevinzg/facebook-scraper/blob/392be1eabb43ed301fb7d5c3fd6e10318d26ac27/README.md
"""
from __future__ import annotations

NickCrews

AnyColOrTable = TypeVar("AnyColOrTable", Column, Table, pd.Series, pd.DataFrame)


def convert_to_ibis(
    func: Callable[[ColOrTable], ColOrTable]
) -> Callable[[AnyColOrTable], AnyColOrTable]:
    """Decorator that translates pandas series to Columns and DFs to Tables,
    applies the function, and then converts back to pandas."""

    @functools.wraps(func)

NickCrews

"""coalesce_parquets.py

gist of how to coalesce small row groups into larger row groups.
Solves the problem described in https://issues.apache.org/jira/browse/PARQUET-1115
"""
from __future__ import annotations

from pathlib import Path
from typing import Callable, Iterable, TypeVar

NickCrews