Skip to content

Instantly share code, notes, and snippets.

@backupbrain

backupbrain/tokenizer_trim_whitespace.js

Last active October 15, 2018 18:56
Show Gist options
  • Save backupbrain/ddc26a0298d6fc0f6a47be4c32809b5a to your computer and use it in GitHub Desktop.
Save backupbrain/ddc26a0298d6fc0f6a47be4c32809b5a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenizer_trim_whitespace.js
Tokenizer.prototype.trimWhitespace = function(text) {
cleanedText = text.replace(/\n+/g, "\n");
cleanedText = cleanedText.replace(/[ ]+/g, " ");
cleanedText = cleanedText.trim();
return cleanedText;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment