Skip to content

Instantly share code, notes, and snippets.

@backupbrain

backupbrain/tokenizer_get_token_lines.js

Last active October 15, 2018 18:56
Show Gist options
  • Save backupbrain/18c1900a5c8ea8b31df1294e6f910fff to your computer and use it in GitHub Desktop.
Save backupbrain/18c1900a5c8ea8b31df1294e6f910fff to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenizer_get_token_lines.js
Tokenizer.prototype.getTokenLines = function(text) {
lines = text.split("\n");
tokenLines = [];
for (var lineId = 0; lineId < lines.length; lineId++) {
line = lines[lineId];
tokens = line.split(" ");
tokenLines.push(tokens);
}
return tokenLines;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment