Quick HDF5 benchmark

benchmark.ipynb

I messed up my first comment, sorry for that. I'm pasting it again for reference:
I'm not entirely sure you're aware what happens during each call. I'd suggest trying this:

%%timeit -n1 -r1
f = h5py.File('test.h5', 'r')
print(f['/test'][:2, :3])
f.close()

Benchmark updated following comment by Stuart Berg.

@rossant, a big part of this is that the fancy-indexing code in h5py uses a naive algorithm based on repeated hyperslab selection, which is quadratic in the number of indices. It was designed/tested for small numbers of indices.

The particular example you have here (0 to 10000 in steps of 10) can be mapped to slices (although of course this is not generally true). In this case the results are:

%%timeit f = h5py.File('test.h5','r')
f['/test'][0:10000:10]

100 loops, best of 3: 12 ms per loop

This is a great argument to improve the implementation of fancy indexing in h5py, but I would hesitate to conclude "HDF5 is slow".

@andrewcollette thanks for your comment, I've updated the benchmarks and the post accordingly. And thanks for doing h5py! Despite the problems we've had with HDF5, I actually like the h5py API and how it fits so naturally with NumPy.