timed fetching of compressed bitsets from solr

siege_urls.txt

http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=asymptotic+pulse+shapes
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=halo+concentration+gamma+model
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=radial+velocity+M50
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=history+of+cosmic+rays
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=forward+scatter
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=universal+rotation+curve
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=moon+formation
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=fission+moon
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=galactic++winds
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=solar+companion
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=hamilton+spectrograph
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=envelope+accretion+rate+mass
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=refraction+calculation
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=eclipse+1868
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=substellar
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=angular+diameter+redshift
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=H2CO+diffuse+clouds
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=mean+motion+resonance
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=sensor+networks
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=microcalorimeter
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=tidal+comet
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=equipartition+magnetic+field+internal
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=Machine+Learning+Space+Weather
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=Space+Weather+Gaussian+Process
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=dark+energy
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=stardust+comet+wild+dust+model+hazard
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=sunspot+light+bridges
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=catalog+radio
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=protostars
http://adsate:8986/solr/fulltext/select?qt=invenio_query&q=helioseismology+models

siege_urls.txt

Lifting the server siege...      done.                                                                                              Transactions:                    1957 hits
Availability:                 100.00 %
Elapsed time:                  74.76 secs
Data transferred:              27.29 MB
Response time:                  0.08 secs
Transaction rate:              26.18 trans/sec
Throughput:                     0.37 MB/sec
Concurrency:                    2.05
Successful transactions:        1957
Failed transactions:               0
Longest transaction:            0.90
Shortest transaction:           0.04

solr-invenio timings

In [131]: bitset = intbitset.intbitset()

In [132]: url = 'http://adsate:8986/solr/fulltext/select?q=%s&qt=invenio_query'

In [133]: def query(q):
    u = urllib2.urlopen(url % urllib2.quote(q))
    data = u.read()
    bitset.fastload(data)
   .....: 

In [137]: %time query('weak lensing')
CPU times: user 0.00 s, sys: 0.01 s, total: 0.01 s
Wall time: 0.59 s

In [139]: len(bitset)
Out[139]: 14198

In [140]: %time query('galaxy cluster')
CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s
Wall time: 0.16 s

In [142]: len(bitset)
Out[142]: 94849

In [143]: %time l = bitset.tolist()
CPU times: user 0.01 s, sys: 0.00 s, total: 0.02 s
Wall time: 0.02 s

In [145]: l[:100]
Out[145]: 
[11757,   
 11762,   
 24160,   
 24926,   
 25656,   
 40036,   
 41785,   
...

In [146]: %time query('*:*')
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.11 s

In [148]: len(bitset)
Out[148]: 624376

In [150]: %timeit query('*:*')
10 loops, best of 3: 77.8 ms per loop

In [151]: %timeit query('redshift')
10 loops, best of 3: 80.4 ms per loop

In [152]: len(bitset)
Out[152]: 70922