Skip to content

Instantly share code, notes, and snippets.

View TariqAHassan's full-sized avatar

Tariq Hassan TariqAHassan

View GitHub Profile
@TariqAHassan
TariqAHassan / PandasConcatWorkaround.adoc
Last active November 6, 2023 22:57
Fast Alternative to pd.concat() for row-wise concatenation

Pandas DataFrames are fantastic. However, concatenating them using standard approaches, such as pandas.concat(), can be very slow with large dataframes. This is a work around for that problem.

Note: this approach assumes that: (a) the goal is a row-wise concatenation (i.e., axis=0) and (b) all dataframes share the same column names.

If these assumptions are not met, this approach could still work…​but it will likely need to be modified.