This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function inverseDocumentFrequency(documents){ | |
| // calculates the inverse document frequency of every sentence | |
| const words_without_stopwords = prettify(documents); | |
| const sentences = documents.split(".") | |
| sentences[0] = sentences[0].substring(146); | |
| const lengthOfDocuments = sentences.length; | |
| const WordCountDocuments = countWords(words_without_stopwords); | |
| // calculate TF values of all documents |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function inverseDocumentFrequency(document){ | |
| // calculates the inverse document frequency of every sentence | |
| const words_without_stopwords = prettify(document); | |
| const unique_words_set = uniqueWords(words_without_stopwords); | |
| const sentences = document.split(".").map(item => item.trim()); | |
| sentences[0] = sentences[0].substring(146); | |
| const lengthOfDocuments = sentences.length; | |
| // prettifys each sentence so it doesn't have stopwords |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function termFrequency(document){ | |
| // calculates term frequency of each sentence | |
| words_without_stopwords = prettify(document); | |
| // gets rid of trailing spaces | |
| const sentences = document.split(".").map(item => item.trim()); | |
| sentences[0] = sentences[0].substring(146); | |
| const TFVals = countWords(words_without_stopwords) | |
| const unique_words = uniqueWords(words_without_stopwords); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function termFrequency(document){ | |
| // calculates term frequency of each sentence | |
| words_without_stopwords = prettify(document); | |
| // gets rid of trailing spaces | |
| const sentences = document.split(".").map(item => item.trim()); | |
| sentences[0] = sentences[0].substring(146); | |
| const TFVals = countWords(words_without_stopwords) | |
| const unique_words = uniqueWords(words_without_stopwords); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function uniqueWords(words){ | |
| const unique_words_set = new Set(words); | |
| return unique_words = Array.from(unique_words_set); | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function countWords(words){ | |
| // returns a dictionary of {WORD: COUNT} where count is | |
| // how many times that word appears in "words". | |
| const unique_words = uniqueWords(words); | |
| let dict = {}; | |
| // for every single unique word | |
| for (let i = 0; i <= unique_words.length - 1; i++){ | |
| dict[unique_words[i]] = 0 | |
| // see how many times this unique word appears in all words | |
| for (let x = 0; x <= words_without_stopwords.length -1; x++){ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function prettify(document){ | |
| // Turns an array of words into lowercase and removes stopwords | |
| const stopwords = ["a", "share", "linkthese", "about", "above", "after", "again", "against", "all", "am", "an", "and", "any","", "are","aren't","as","at","be","because","been","before","being","below","between","both","but","by","can't","cannot","could","couldn't","did","didn't","do","does","doesn't","doing","don't","down","during","each","few","for","from","further","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","he's","her","here","here's","hers","herself","him","himself","his","how","how's","i","i'd","i'll","i'm","i've","if","in","into","is","isn't","it","it's","its","itself","let's","me","more","most","mustn't","my","myself","no","nor","not","of","off","on","once","only","or","other","ought","our","ours","ourselves","out","over","own","same","shan't","she","she'd","she'll","she's","should","shouldn't","so","some","such","than","that","that's","the","their","theirs","them","them |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| x = range(-5, 5) | |
| all_less_than_zero = (num * num for num in x if num < 0) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # taken from page 87, chapter 3 of Fluent Python by Luciano Ramalho | |
| >>> from unicodedata import name | |
| >>> {chr(i) for i in range(32, 256) if 'SIGN' in name(chr(i), '')} | |
| {'×', '¥', '°', '£', '©', '#', '¬', '%', 'µ', '>', '¤', '±', '¶', '§', '<', '=', '®', '$', '÷', '¢', '+'} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Taken from page 70 chapter 3 of Fluent Python by Luciano Ramalho | |
| DIAL_CODES = [ | |
| (86, 'China'), | |
| (91, 'India'), | |
| (1, 'United States'), | |
| (62, 'Indonesia'), | |
| (55, 'Brazil'), | |
| (92, 'Pakistan'), | |
| (880, 'Bangladesh'), |