Go's encoding/csv Reader type takes the novel (to me) approach of deciding that carriage return line feeds (CRLFs) should be replaced with newlines (LFs).
It not only replaces CRLFs that mark then end of one record and the beginning of the next—the encoding of the data—it replaces all CRLFs at the end of any line of text—the data itself.
The CSV:
ID,Data
I mocked up a 60 MB XML by taking all the small samples in your original ZIP archive and just copying them all 200 times, which ended up with over 425k tok elements.
I then profiled your code and found a really bad culprit for chewing up time.
To process that XML took about 35 seconds:
Thu Jun 29 10:50:59 2023 profile.stats
The simplest and (probably) most efficient way to read/write any-sized CSV in Python will always be to use the csv module—bar none.
Its reader provides a very simple interface for iterating the CSV a-row-at-a-time (so never more than one row's worth of memory consumed), and that row can be passed directly to the writer (which will probably be buffered, so minimal sys calls). But, the documentation doesn't show you how to do this, even though it's so simple:
reader → row → process(row) → writerow(row)
The following code has revealed different behavior over time. When I first tested this, I saw behavior like the following where 50 requests were kicked off at once, all requests completed (with a 200), but subsequent requests took longer and longer. The following output also shows a minimum response time of 1.5s, which I think (I remember) shows that Airtable's dynamic back-offs were sticky:
Submitting 50 requests "at once"
All 50 are sent in the span of one second, but each subsequent request takes progressively longer to respond:
started request 42 at Jul 21 11:23:27.565001, ended in 1.50s
For CSV to dataclass...
I orginally had this logic to check if the row of a CSV contained any blank values:
n_cols = len(rows[0])
for row in rows:
if len([x for x in row if x]) != n_cols:
pip install json-streamsh run.shrun.sh calls the run_*.py scripts, which will run gen_json.py to generate three JSON test files of varying size.
The generate JSON looks like:
When I run python3 run_test.py I get:
| Test N | row specs | DT Start | DT End | time (s) |
|---|