Skip to content

Instantly share code, notes, and snippets.

@komly
Created August 28, 2015 12:55
Show Gist options
  • Select an option

  • Save komly/b73d636c57c53c797aa9 to your computer and use it in GitHub Desktop.

Select an option

Save komly/b73d636c57c53c797aa9 to your computer and use it in GitHub Desktop.
#!/usr/bin/env python
import json
import shelve
from urllib2 import urlopen
URL = 'http://54.200.3.128/examples/api.php'
DB_FILENAME = 'data.db'
def get_unique(url, db_filename):
result = []
db = shelve.open(DB_FILENAME)
data = json.load(urlopen(URL))
for entry in data:
if str(entry['link_id']) not in db:
result.append(entry)
db[str(entry['link_id'])] = True
db.close()
return result
print(get_unique(URL, DB_FILENAME))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment