Skip to content

Instantly share code, notes, and snippets.

@remi-or
remi-or / Datasets.py
Last active August 16, 2021 10:54
Snippet for loading datasets
# This snippet requires you to install Hugging Face's datasets module
from datasets import load_dataset
import pandas as pd
Dataframe = pd.DataFrame({})
questions = load_dataset('squad')['train']['question'][:3000]
Dataframe = Dataframe.append(pd.DataFrame({'Text' : questions, 'Source' : 'squad'}))
questions = load_dataset('hotpot_qa', 'distractor')['train']['question'][:3000]