Artem Yegorov

Extracting financial disclosure reports and police blotter narratives using OpenAI's Structured Output

tl;dr this demo shows how to call OpenAI's gpt-4o-mini model, provide it with URL of a screenshot of a document, and extract data that follows a schema you define. The results are pretty solid even with little effort in defining the data — and no effort doing data prep. OpenAI's API could be a cost-efficient tool for large scale data gathering projects involving public documents.

OpenAI announced Structured Outputs for its API, a feature that allows users to specify the fields and schema of extracted data, and guarantees that the JSON output will follow that specification.

For example, given a Congressional financial disclosure report, with assets defined in a table like this:

Permanent Connection Checkout

Rails 7.2 defaults:

Highscore.connection
# => #<ActiveRecord::ConnectionAdapters::SQLite3Adapter:0x000000000080e8 env_name="development" role=:writing>

With config.active_record.permanent_connection_checkout = :disallowed:

Auditing in Database Applications: Ensuring Data Integrity and Compliance

In today's data-driven world, maintaining the integrity and security of information stored in databases is paramount. One crucial aspect of this is auditing - the practice of tracking and logging all changes made to data within a system. This article explores the importance of auditing in database applications and compares several popular auditing solutions.

Why Auditing Matters

Implementing a robust auditing system in your database application offers several key benefits:

Compliance: Many industries are legally required to maintain an audit trail of changes to sensitive data.
Security: Audit logs help detect unauthorized access or potential data breaches.

Simple Template Rendering

I'm experimenting with a simple template rendering solution that leverages Ruby's native String formatting. It feels a little bit like Mustache templates. Note that this demo adds new "formatting specifiers" to support Rainbow color mechanics.

Tip

See the files below for the implementation... and note that this is simply a proof of concept (POC)

Usage

# In your Gemfile
gem "localhost"

# Then, depending on your desired server
gem "falcon"
gem "puma"

	#!/usr/bin/env ruby

	require "bundler/inline"

	gemfile do
	source "https://rubygems.org"
	gem "typhoeus"
	end

	require "uri"

	# frozen_string_literal: true
	# config/initializers/colorized_logger.rb

	# This initializer adds color to the Rails logger output. It's a nice way to
	# visually distinguish log levels.
	module ColorizedLogger
	COLOR_CODES = {
	debug: "\e[36m", # Cyan
	info: "\e[32m", # Green
	warn: "\e[33m", # Yellow

	#!/usr/bin/env ruby

	# make sure you have Ruby installed, should be installed by default on macOS & Linux
	# install Minisky lib with: [sudo] gem install minisky
	# then run with: ruby follow_hashtag.rb somehashtag

	require 'minisky'

	if !File.exist?('config.yml')
	puts "Create a config.yml file with contents like this:"

	require 'prism'
	require 'ripper'
	chars = %w[1 % = a b == * . ( ) , % ? ** & [ ] { } \| ; : => ~ ! ^ \|\| && '' .. ... < > + $a @a] + [' ', "\n", ' if ', ' and ',' rescue ', ' in ']
	p seed: seed = rand(10000)
	srand seed
	$VERBOSE=nil
	(3..10).each do \|n\|
	[chars.size**n, 100000].min.times do \|i\|
	code = n.times.map{chars.sample}.join
	p [n, i] if i % 1000 == 0

	require 'open3'
	require 'reline'

	class Reline::LineEditor
	private def perform_completion(list, _just_show_list)
	_preposing, target, _postposing = retrieve_completion_block
	namespace = IRB.CurrentContext.io.retrieve_doc_namespace(target)
	namespace = namespace.first if namespace.is_a?(Array)
	if namespace.nil? \|\| !('A'..'Z').include?(namespace[0])
	namespace = IRB.conf[:__MAIN__]