Skip to content

Instantly share code, notes, and snippets.

@arademaker
Created July 19, 2012 23:59
Show Gist options
  • Select an option

  • Save arademaker/3147757 to your computer and use it in GitHub Desktop.

Select an option

Save arademaker/3147757 to your computer and use it in GitHub Desktop.
count of sentences in a set of markdown files
# A simple script to calculate the number of sentences per markdown
# file in a given directory.
#
# Reference:
# - http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html
# Author: Alexandre Rademaker
import os
import glob
import re
import nltk
sent_tokenizer = nltk.data.load('tokenizers/punkt/portuguese.pickle')
def parseFile(filename):
raw = open(filename).read().decode('utf8')
para = []
current = []
for x in raw.split("\n"):
if len(x) > 0:
current.append(x)
else:
para.append( " ".join(current) )
current = []
raw_text = "\n".join(para)
sentences = sent_tokenizer.tokenize(raw_text)
return sentences
for myf in glob.glob("*.md"):
print myf, len(parseFile(myf))
@turicas

turicas commented Jul 20, 2012

Copy link
Copy Markdown

You can use glob('*.md') (after importing with from glob import glob) instead of getFiles('.').

@arademaker

Copy link
Copy Markdown
Author

Thanks Alvaro! Python is no more my most used programming language! Unfortunately, I needed the NLTK module... I have forgotten the glob module!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment