virgulvirgul

Identifying which PDFs are images vs. text

Someone who was in my PDF text extraction session at NICAR 2020 asked how to identify image vs. text PDFs when you have thousands of files and they're a mixture of formats with the end goal of only running OCR software on the image PDFs.

This is how I would approach the problem using command-line tools.

Assumptions

You’re working on a Mac or Linux machine where you have access to some common command-line utilities such as find and sed
This should work under the Windows Subshell for Linux under Windows also

58 bytes of CSS to look great nearly everywhere

When making this website, i wanted a simple, reasonable way to make it look good on most displays. Not counting any minimization techniques, the following 58 bytes worked well for me:

main {
  max-width: 38rem;
  padding: 2rem;
  margin: auto;
}

jq: JSONL ↔︎ JSON conversion

Prerequisites

jq — https://jqlang.github.io/jq/ — "like sed for JSON data"

There are several options available for installing jq. I prefer to use Homebrew: brew install jq

JSONL → JSON

Create a folder to store the databases :

mkdir -p /usr/share/GeoIP

Download Country IP database

wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
gunzip GeoIP.dat.gz

Using Git to Manage a Live Web Site

Overview

As a freelancer, I build a lot of web sites. That's a lot of code changes to track. Thankfully, a Git-enabled workflow with proper branching makes short work of project tracking. I can easily see development features in branches as well as a snapshot of the sites' production code. A nice addition to that workflow is that ability to use Git to push updates to any of the various sites I work on while committing changes.

What / Why

Deploy key is a SSH key set in your repo to grant client read-only (as well as r/w, if you want) access to your repo.

As the name says, its primary function is to be used in the deploy process in replace of username/password, where only read access is needed. Therefore keep the repo safe from the attack, in case the server side is fallen.

How to

Generate a ssh key

	#!/usr/bin/env bash

	# Small shell script to more easily automatically download and transcribe live stream VODs.
	# This uses YT-DLP, ffmpeg and the CPP version of Whisper: https://github.com/ggerganov/whisper.cpp
	# Use `./transcribe-vod help` to print help info.

	# MIT License

	# Copyright (c) 2022 Daniils Petrovs

	# So now you want to finetune that GPT-J-6B on a 3090/TITAN GPU ... okay
	# More exploratory coding. It uses the Huggingface model port, deepspeed and reads all text/md files from a target directory
	# It is a fragment of a larger system with remote editing, but that's another story
	# This is the raw, training tester. Items to look out for:
	# - uses DeepSpeed and has a DS config
	# - to save space uses SGD instead of ADAM
	# - uses gradient checkpointing
	# - freezes 25% of the layers to fit

	# Assumes you can already run https://gist.github.com/kinoc/2d636a68876cd3de7b6e9c9452b61089

	cd() {
	# Check if no arguments to make just typing cd<Enter> work
	# Also check if the first argument starts with a - and let cd handle it
	if [ $# -eq 0 ] \|\| [[ $1 == -* ]]
	then
	builtin cd $@
	return
	fi
	# If path exists, just cd into it
	# (also, using $* and not $@ makes it so you don't have to escape spaces any more)

	from fuzzywuzzy import fuzz
	from fuzzywuzzy import process
	from collections import Counter

	A_title_file = "/tmp/A_titles.txt"
	B_title_file = "/tmp/B_titles.txt"

	# Open the files and get the titles
	A_titles = []
	with open(A_title_file) as f: