Simple JSON to CSV Method for Twitter Archive Data

tweet-archive-converter.py

#!/usr/bin/env python3
'''
    IMPORTANT: Please delete the following from the tweet.js file before using this:
     ---> window.YTD.tweet.part0 = <----
    Whilst that remains you cant parse it easily with json.load
'''
import pandas as pd
import json
from pandas.io.json import json_normalize

# process raw json 
data = json.load(open('tweet.js'))
# Create DataFrame
df = pd.DataFrame.from_records(json_normalize(data))
# Write to CSV
df.to_csv("tweets.csv")

Tested with my archive which has roughly 21,000 tweets. The bottle neck is the for loop which took roughly 15 minutes to run. Writing the DataFrame out to CSV was almost instant.

re-wrote as the dataframe append program flow I previously used was inefficient to say the least - eating both memory / CPU and taking roughly 13 minutes to create.

The new method is near instant for just over 20k rows! :)

Worked for me out of the box. TY!

Worked for me out of the box. TY!

Awesome! Ty for the reply it's awesome to see someone else found it useful

Very handy- had to tweak a bit with python 3 on OS X 13.4.1

#!/usr/bin/env python3
'''
    IMPORTANT: Please delete the following from the tweet.js file before using this:
     ---> window.YTD.tweet.part0 = <----
    Whilst that remains you cant parse it easily with json.load
'''
import pandas as pd
import json

# process raw json
data = json.load(open('tweets.js'))
# Create DataFrame
df = pd.DataFrame.from_records(pd.json_normalize(data))
# Write to CSV
df.to_csv("tweets.csv")