nwb_streaming

## set up env via bash:
conda create -n stream_nwb -c conda-forge -y python=3.7 hdf5=1.12.0 jupyter pip
conda activate stream_nwb
pip install nwbwidgets
pip install git+https://github.com/NeurodataWithoutBorders/pynwb
pip install git+https://github.com/hdmf-dev/hdmf
git clone https://github.com/satra/h5py.git
cd h5py
HDF5_DIR=~/Users/bendichter/opt/anaconda3/envs/stream_nwb/ pip install --force-reinstall --no-binary=h5py .


## test out streaming
from pynwb import NWBHDF5IO
from nwbwidgets import nwb2widget
import requests

def get_s3_url(url):
    s3_url = requests.request(url=url, method='head').url
    return s3_url[:s3_url.index('?')]

path = "https://dandiarchive.s3.amazonaws.com/girder-assetstore/58/07/58074d0a5a4a4086afebb4d29912c419"

io = NWBHDF5IO(get_s3_url(url), mode='r', load_namespaces=True, driver='ros3')
nwb = io.read()
nwb2widget(nwb)

@satra thanks, that works indeed!
we are putting it to work on a server and the current solution is still a bit hacky I feel... do you think this h5py build from conda/pip will be available anytime soon?

that's an issue on the conda-forge side that they have not fully figured out: conda-forge/h5py-feedstock#87

i have no idea of a timeline there.

hdf5 is available on conda-forge with ros3 support. the h5py build is currently unavailable due to a failure on arm cross-compiling, but if you have a compiler, you should be able to pip install h5py .

HDF5_DIR=~/Users/bendichter/opt/anaconda3/envs/stream_nwb/ pip install --force-reinstall --no-binary=h5py

you will have to adjust the location of the HDF5 libraries that you install from conda-forge.

@satra It still did not work for Windows even after adjusting for the conda installed hdf5 libraries. Same error

can you check the hdf settings to make sure the driver is in fact enabled? it's a file called libhdf5.settings inside the conda environment where you have installed hdf5 and it should have a line that says (Read-Only) S3 VFD: yes

can you check the hdf settings to make sure the driver is in fact enabled? it's a file called libhdf5.settings inside the conda environment where you have installed hdf5 and it should have a line that says (Read-Only) S3 VFD: yes

It wasn't initially, but even when I changed it to yes, the h5py did not build correctly. I think the folder structure in the conda/envs/stream_nwb/is different in case of linux vs windows. So the build process is not able to look for the correct files when the HDF5_DIR is set to '*/conda/envs/stream_nwb/' in windows. I tried to follow the same steps in linux and it works fine.

you shouldn't change to yes manually. it reflects the hdf5 version that is installed. so if it says no, then your hdf5 library doesn't have the driver enabled. it means probably an incorrect install from conda-forge.

I can reproduce the problem on my windows machine, with both conda-forge installation and manual installation of hdf5.
Seems to be a problem specific to windows. I found this, it might help (search for Windows): https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.0/src/hdf5-1.12.0-RELEASE.txt

we managed to make it work on a Binder deployment, if anyone is interested: https://github.com/catalystneuro/binder-nwb-stream

you may want to file an issue on conda-forge for hdf5-feedstock, if it is not being compiled on windows with ROS3 VFD support.