Skip to content

Instantly share code, notes, and snippets.

@pythonlessons

pythonlessons/transformers_nlp_data_4.py

Created August 24, 2023 10:12
Show Gist options
  • Save pythonlessons/6c30f00a4a7e95998a1e43f2863a3df1 to your computer and use it in GitHub Desktop.
Save pythonlessons/6c30f00a4a7e95998a1e43f2863a3df1 to your computer and use it in GitHub Desktop.
Download ZIP
transformers_nlp_data
Raw
transformers_nlp_data_4.py
tokenized_sentence = detokenizer.texts_to_sequences(["Hello world, how are you?"])[0]
print(tokenized_sentence)
detokenized_sentence = detokenizer.detokenize([tokenized_sentence], remove_start_end=False)
print(detokenized_sentence)
detokenized_sentence = detokenizer.detokenize([tokenized_sentence])
print(detokenized_sentence)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment