Incremental CSV saving

incremental_csv_saving.py

import os
import pandas as pd
import time

csv = "file.csv"

for i in range(10):
    row = pd.DataFrame([{"time": pd.Timestamp.now(), "data": i}])
    time.sleep(0.5)
    if i == 9:
        raise Exception("Some error at the end!")

    add_header: bool = not os.path.exists(csv)
    row.to_csv(csv, mode="a", header=add_header, index=False)
    print(f"Saved row {i}")

• This snippet shows how to use the append mode of DataFrame.to_csv() to write chunks of data incrementally to a CSV in a for loop.
• This is useful when the loop that we are running takes a long time to complete and is prone to errors.
• If instead, we dumped the DataFrame after the loop is complete, then any error in the loop would all of our progress, like the error on iteration i=9.