lukauskas

shrayasr

Bulk Inserts via SQLAlchemy and Flask-SQLAlchemy

Problem

I ran into an issue today where I had to perform a bulk insert into a postgres DB. I was already using SQLAlchemy and Flask-SQLAlchemy to manage the connections to the db and I didn't want to have to use things like psycopg2 directly.

Solution

Note: SQLAlchemy provides an ORM. It isn't just an ORM. That is an important thing to be kept in mind. This means that you can bypass choose to not use the ORM layer when you don't want it. The idea with an ORM is to track changes to objects and when you have a case like that is when you'd use the ORM. In a bulk upload scenario, you don't need to track changes to objects. All you care is that everything be pushed into the DB.

bsweger

Useful Pandas Snippets

A personal diary of DataFrame munging over the years.

Data Types and Conversion

Convert Series datatype to numeric (will error if column has non-numeric values)
(h/t @makmanalp)

mattheworiordan

/* Extend the Underscore object with the following methods */

// Rate limit ensures a function is never called more than every [rate]ms
// Unlike underscore's _.throttle function, function calls are queued so that
//   requests are never lost and simply deferred until some other time
//
// Parameters
// * func - function to rate limit
// * rate - minimum time to wait between function calls
// * async - if async is true, we won't wait (rate) for the function to complete before queueing the next request