Skip to content

Instantly share code, notes, and snippets.

@weaponsforge
Created November 26, 2022 04:20
Show Gist options
  • Save weaponsforge/66f970324ba0f97f1402022d2be55c9e to your computer and use it in GitHub Desktop.
Save weaponsforge/66f970324ba0f97f1402022d2be55c9e to your computer and use it in GitHub Desktop.
js tokenizer
const string = 'Hello, World!'
// Split the a string into words while removing special characters
const words = string.replace(/[!@#$%^&*()_+\-=\[\]{};:"\\|,.<>\/?]/g, '').split(' ').filter(x => x.match(/[a-z]/g))
console.log(words)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment