Skip to content

Instantly share code, notes, and snippets.

@bfarzin

bfarzin/Wiki103_fastai_v1-SP_Tokenizer-Example.ipynb

Last active August 11, 2019 13:39
Show Gist options
  • Save bfarzin/f9407c0b0f2690f36dd6d51c8ef56944 to your computer and use it in GitHub Desktop.
Save bfarzin/f9407c0b0f2690f36dd6d51c8ef56944 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Wiki103_fastai_v1-SP_Tokenizer-Example.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tanny411
Copy link

tanny411 commented Jun 7, 2019

i believe this is how i should use it:
mycust_tok = CustomTokenizer(SPTokenizer,model_prefix)
But i still have the error. Can you help with the full modified code?

@bfarzin
Copy link
Author

bfarzin commented Jun 7, 2019

itos was wrong also. I updated the example above.

@tanny411
Copy link

tanny411 commented Jun 7, 2019

Thank you so much. Sorry I didnt notice the change earlier. It works. Much appreciated.

@bfarzin
Copy link
Author

bfarzin commented Jun 8, 2019

No problem. I am glad I cleaned it up for my own good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment