JeremiahKamama/num_tokens.py

Created July 9, 2021 06:53

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/JeremiahKamama/de2cd9dde02e9f1e8e579eca78ded8c3.js"></script>
Save JeremiahKamama/de2cd9dde02e9f1e8e579eca78ded8c3 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

num_tokens.py

	num_tokens = [len(tokens) for tokens in train_tokens + test_tokens]
	num_tokens = np.array(num_tokens)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment