Skip to content

Instantly share code, notes, and snippets.

@imarban
Created August 22, 2016 05:55
Show Gist options
  • Save imarban/9bc43f6c2cf33eddd208bdce0ad4a53a to your computer and use it in GitHub Desktop.
Save imarban/9bc43f6c2cf33eddd208bdce0ad4a53a to your computer and use it in GitHub Desktop.
class TextProcessor:
def __init__(self, text, tokenizer_module, sanitizer_module):
self.text = text
self.tokenizer = tokenizer_module
self.sanitizer = sanitizer_module
def get_tokens():
sanitized = self.sanitizer.sanitize(self.text)
tokens = self.tokenizer.tokenize(sanitized)
return tokens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment