dannguyen

Extracting financial disclosure reports and police blotter narratives using OpenAI's Structured Output

tl;dr this demo shows how to call OpenAI's gpt-4o-mini model, provide it with URL of a screenshot of a document, and extract data that follows a schema you define. The results are pretty solid even with little effort in defining the data — and no effort doing data prep. OpenAI's API could be a cost-efficient tool for large scale data gathering projects involving public documents.

OpenAI announced Structured Outputs for its API, a feature that allows users to specify the fields and schema of extracted data, and guarantees that the JSON output will follow that specification.

For example, given a Congressional financial disclosure report, with assets defined in a table like this:

ZaxR

"""
Example:
    # All locations; also locations FROM which we want to find nearest neighbors
    locations = pd.DataFrame({"LOCATION_NAME": ["Chicago, IL", "New York, NY", "San Fransisco, CA"],
                              "LATITUDE": [1, 2, 3],
                              "LONGITUDE": [1, 2, 3],
                              "ID": [1, 2, 3]})
    locations = locations.apply(lambda x: Location(location_name=x['LOCATION_NAME'],
                                                   latitude=x['LATITUDE'],
                                                   longitude=x['LONGITUDE'],

mrmartineau

Stimulus cheatsheet

• Website: https://stimulusjs.org/
• GitHub repo: https://github.com/stimulusjs/stimulus
• Handbook: https://stimulusjs.org/handbook/introduction
• Discourse: https://discourse.stimulusjs.org/

Lifecycle callbacks

• initialize: once, when the controller is first instantiated
• connect: anytime the controller is connected to the DOM

adejones

from openpyxl import load_workbook

def XLSXDictReader(f):
    book = load_workbook(f)
    sheet = book.active
    rows = sheet.max_row
    cols = sheet.max_column
    headers = dict((i, sheet.cell(row=1, column=i).value) for i in range(1, cols))
    def item(i, j):
        return (sheet.cell(row=1, column=j).value, sheet.cell(row=i, column=j).value)

rueycheng

GNU Make Cheatsheet

Based on https://www.gnu.org/software/make/manual/html_node/Quick-Reference.html

Directives

Directive	Description
define variable define variable = define variable := define variable ::= define variable += define variable ?= endef	Define multi-line variables.
undefine variable	Undefining variables.

Kartones

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

• -E: will describe the underlaying queries of the \ commands (cool for learning!)
• -l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

adamwiggins

Make it real

Ideas are cheap. Make a prototype, sketch a CLI session, draw a wireframe. Discuss around concrete examples, not hand-waving abstractions. Don't say you did something, provide a URL that proves it.

Ship it

Nothing is real until it's being used by a real user. This doesn't mean you make a prototype in the morning and blog about it in the evening. It means you find one person you believe your product will help and try to get them to use it.

Do it with style

dminkovsky

from django.http import HttpResponse
from django.utils.functional import wraps
from django.template.context import RequestContext
from django.template.loader import get_template
from django.utils import simplejson as json

def dict_to_template_view(**template_args):
    """
    Takes a function that returns a dict and decorates it into a template-rendering view
    """

jboner

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD