Skip to content

Instantly share code, notes, and snippets.

@eustin

eustin/letor-part-1-model-tokeniser.py

Created May 25, 2020 22:35
Show Gist options
  • Save eustin/d3388e80cca81bfe36f7a31076b1bf5a to your computer and use it in GitHub Desktop.
Save eustin/d3388e80cca81bfe36f7a31076b1bf5a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
letor-part-1-model-tokeniser.py
tokeniser = tf.keras.preprocessing.text.Tokenizer()
tokeniser.fit_on_texts(sentences)
print(tokeniser.word_index)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment