Created
July 19, 2012 23:59
-
-
Save arademaker/3147757 to your computer and use it in GitHub Desktop.
count of sentences in a set of markdown files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # A simple script to calculate the number of sentences per markdown | |
| # file in a given directory. | |
| # | |
| # Reference: | |
| # - http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html | |
| # Author: Alexandre Rademaker | |
| import os | |
| import glob | |
| import re | |
| import nltk | |
| sent_tokenizer = nltk.data.load('tokenizers/punkt/portuguese.pickle') | |
| def parseFile(filename): | |
| raw = open(filename).read().decode('utf8') | |
| para = [] | |
| current = [] | |
| for x in raw.split("\n"): | |
| if len(x) > 0: | |
| current.append(x) | |
| else: | |
| para.append( " ".join(current) ) | |
| current = [] | |
| raw_text = "\n".join(para) | |
| sentences = sent_tokenizer.tokenize(raw_text) | |
| return sentences | |
| for myf in glob.glob("*.md"): | |
| print myf, len(parseFile(myf)) |
Author
Thanks Alvaro! Python is no more my most used programming language! Unfortunately, I needed the NLTK module... I have forgotten the glob module!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
You can use
glob('*.md')(after importing withfrom glob import glob) instead ofgetFiles('.').