Skip to content

Instantly share code, notes, and snippets.

@avriiil

avriiil/simple-word-tokenizer.py

Created April 5, 2021 19:24
Show Gist options
  • Save avriiil/78e9510de535805f7d1a52e6e66d31c8 to your computer and use it in GitHub Desktop.
Save avriiil/78e9510de535805f7d1a52e6e66d31c8 to your computer and use it in GitHub Desktop.
Download ZIP
Perform a simple word tokenizer on Arabic text
Raw
simple-word-tokenizer.py
from camel_tools.tokenizers.word import simple_word_tokenize
df.tweet_text = df.tweet_text.apply(simple_word_tokenize)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment