Skip to content

Instantly share code, notes, and snippets.

@dgadiraju

dgadiraju/python-spark-wordcount.py

Created January 27, 2017 00:45
Show Gist options
  • Save dgadiraju/8d2e0776c25a3f896ccea7f9069fa91e to your computer and use it in GitHub Desktop.
Save dgadiraju/8d2e0776c25a3f896ccea7f9069fa91e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
python-spark-wordcount.py
for i in sc.textFile("/public/randomtextwriter/part-m-00000"). \
flatMap(lambda rec: rec.split(" ")). \
map(lambda rec: (rec, 1)). \
reduceByKey(lambda total, value: total + value). \
take(100):
print(i)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment