Use Pandas to spread/flatten dictionary's items into individual columns

data.csv

flatten.py

# Thanks:
#    https://stackoverflow.com/a/55279799
#    https://stackoverflow.com/a/25512372
import pandas as pd
import json

def flatten_json(text):
    try:
        if text.startswith('{'):
            return json.loads(text)
    except Exception as e:
        print("Cannot parse:", text)
        print(e)
        return None
    
# Read CSV data using converter function.
df = pd.read_csv('data.csv', escapechar='"', converters={'meta': flatten_json})

# Get all the data as a list of dictionaries
records = json.loads(df.to_json(orient="records"))

# Read all the data back into a Dataframe
flat_df = pd.json_normalize(records)

Notes:

• This strategy works well on smaller data sets. For larger datasets, we're constrained by system memory.
• print statements should use logger instead.
• startswith detection can be made more sophisticated as needed.
• OMG! This actually works.