Skip to content

Instantly share code, notes, and snippets.

@maedoc
Last active August 11, 2017 00:54
Show Gist options
  • Save maedoc/a5344865666ace5b6d8ae591cb1815f8 to your computer and use it in GitHub Desktop.
Save maedoc/a5344865666ace5b6d8ae591cb1815f8 to your computer and use it in GitHub Desktop.
faster iteration over batches of rows with uuid ids
from .models import Thing
import time
all_ids = []
size = 10000
num_items = Thing.objects.count()
qs = Thing.objects.order_by('id')
last_id = qs.first().id
count = 0
while True:
tic = time.time()
# this instead of qs[count:count+size]
things = qs.filter(id__gt=last_id)[:size]
if len(things) == 0:
print('done')
break
for thing in things:
all_ids.append(thing.id)
toc = time.time() - tic
last_id = item.id
print(toc, count, last_id)
count += size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment