Skip to content

Instantly share code, notes, and snippets.

@Abhayparashar31

Abhayparashar31/clean_wiki_scrape_data.py

Created October 17, 2022 15:01
Show Gist options
  • Save Abhayparashar31/7a893581b9307e7d0e622acdb41e8278 to your computer and use it in GitHub Desktop.
Save Abhayparashar31/7a893581b9307e7d0e622acdb41e8278 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
clean_wiki_scrape_data.py
import re
def clean_data(data):
text = re.sub(r"\[[0-9]*\]"," ",data)
text = text.lower()
text = re.sub(r'\s+'," ",text)
text = re.sub(r","," ",text)
return text
cleaned_data = clean_data(raw_data)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment