kjhealy

Extracting financial disclosure reports and police blotter narratives using OpenAI's Structured Output

tl;dr this demo shows how to call OpenAI's gpt-4o-mini model, provide it with URL of a screenshot of a document, and extract data that follows a schema you define. The results are pretty solid even with little effort in defining the data — and no effort doing data prep. OpenAI's API could be a cost-efficient tool for large scale data gathering projects involving public documents.

OpenAI announced Structured Outputs for its API, a feature that allows users to specify the fields and schema of extracted data, and guarantees that the JSON output will follow that specification.

For example, given a Congressional financial disclosure report, with assets defined in a table like this:

kjhealy

snps <-
    list(r = "~/.config/rstudio/snippets/r.snippets") %>%
    purrr::map(readLines, warn = FALSE) %>%
    purrr::map(paste, collapse = "\n") %>%
    purrr::map(trimws) %>%
    purrr::map(strsplit, split = "(^|\n)snippet ") %>%
    purrr::map_depth(2, ~ .x[.x != ""]) %>%
    purrr::map_depth(2, ~ {
      nm <- gsub("^([^\n\t ]+).*", "\\1", .x)
      names(.x) <- nm

kjhealy

Installing tmux-256color for macOS

• macOS 10.15.5
• tmux 3.1b

macOS has ncurses version 5.7 which does not ship the terminfo description for tmux. There're two ways that can help you to solve this problem.

The Fast Blazing Solution

Instead of tmux-256color, use screen-256color which comes with system. Place this command into ~/.tmux.conf or ~/.config/tmux/tmux.conf(for version 3.1 and later):

kjhealy

## Context: https://twitter.com/grant_mcdermott/status/1493400952878952448

options(collapse_mask = "all") # NB: see `help('collapse-options')`

library(dplyr)
library(data.table)
library(collapse) # Needs to come after library(dplyr) for collapse_mask to work

flights = fread('https://raw.githubusercontent.com/Rdatatable/data.table/master/vignettes/flights14.csv')

kjhealy

This is my personal list of functions that I wrote for converting mov files to mp4!

Command Flags

Flag	Options	Description
-codec:a	libfaac, libfdk_aac, libvorbis	Audio Codec
-quality	best, good, realtime	Video Quality
-b:a	128k, 192k, 256k, 320k	Audio Bitrate
-codec:v	mpeg4, libx264, libvpx-vp9	Video Codec

kjhealy

library(dplyr, warn.conflicts = FALSE)
library(gapminder)
probs <- c(0.1, 0.5, 0.9)

gapminder %>% 
  group_by(continent) %>% 
  summarise(
    probs = probs,
    across(is.numeric & !year, ~ quantile(.x, probs))
  )

kjhealy

# pkgs
pacman::p_load(tidyverse, polite, scales, ggimage, ggforce,
               rvest, glue, extrafont, ggrepel, magick)
loadfonts()

## add_logo function from Thomas Mock
add_logo <- function(plot_path, logo_path, logo_position, logo_scale = 10){

    # Requires magick R Package https://github.com/ropensci/magick

kjhealy

# Animated dplyr joins with gganimate
# * Garrick Aden-Buie
# * garrickadenbuie.com
# * MIT License: https://opensource.org/licenses/MIT

# Note: I used Fira Sans and Fira Mono fonts.
#       Use search and replace to use a different font if Fira is not available.

library(tidyverse)
library(gganimate)

kjhealy

library(XML)
library(ggplot2)

df <- readHTMLTable("http://projects.dailycal.org/paychecker")[[1]]
colnames(df)[4] <- "Salary"
df$Salary <- as.numeric(gsub('[$,]', '', df$Salary))

p <- ggplot(df, aes(x=Department, y=Salary)) + coord_flip()
p + geom_boxplot(aes(color=Rank,
                     x=reorder(Department, Salary, FUN=max))) +

kjhealy

library(XML)
library(ggplot2)

df <- readHTMLTable("http://projects.dailycal.org/paychecker/departments/")[[1]]

DeMoney <- function(x) as.numeric(gsub(",", "", gsub("\\$", "", as.character(x))))

money.columns <- c("All", "Professor", "Associate professor", "Assistant professor", 
                   "Lecturer")