Skip to content

Instantly share code, notes, and snippets.

@branislav1991

branislav1991/hdf5_dataset_usage.py

Last active July 16, 2020 23:04
Show Gist options
  • Save branislav1991/2c4a225e3066cf1542dacdbc11f51e09 to your computer and use it in GitHub Desktop.
Save branislav1991/2c4a225e3066cf1542dacdbc11f51e09 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hdf5_dataset_usage.py
from torch.utils import data
num_epochs = 50
loader_params = {'batch_size': 100, 'shuffle': True, 'num_workers': 6}
dataset = HDF5Dataset('C:/ml/data', recursive=True, load_data=False,
data_cache_size=4, transform=None)
data_loader = data.DataLoader(dataset, **loader_params)
for i in range(num_epochs):
for x,y in data_loader:
# here comes your training loop
pass
@AJ2019
Copy link

AJ2019 commented Dec 11, 2019

Hi, I tried to create a data_loader using your HDF5Dataset class and got an error, "ValueError: num_samples should be a positive integer value, but got num_samples=0"
Did you face this error before?

Thanks,
Aditya

@prarobinson
Copy link

Same here! Line 9 of hdf5_dataset_usage.py errors out on me. File_path and data all seem to be okay. Any pointers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment