english wikipedia length

en_wiki_length.py

from nlp import load_dataset
from tqdm.auto import tqdm

wiki = load_dataset('wikipedia', '20200501.en', split="train")

batch_size = 1000
total_length = 0
for i in tqdm(range(0, len(wiki), batch_size)):  # loop takes ~1min to run
    batch = wiki[i:i + batch_size]
    total_length += sum(len(sample_text) for sample_text in batch["text"])
    
print(total_length)
# >>> 18067578318

To run this gist, install 🤗nlp with

pip install nlp

And check the details at https://github.com/huggingface/nlp