SuvroBaner
/ WordCounter.py
Last active
June 21, 2018 08:34
This code will count the number of words in a textfile using Spark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Creating Spark Configuration and Spark Context- | |
| from pyspark import SparkContext, SparkConf | |
| conf = SparkConf().setAppName("Word Counter") | |
| sc = SparkContext(conf = conf) | |
| # Reading the file- | |
| myTextFile = sc.textFile("/Users/bsuvro/spark-2.3.0-bin-hadoop2.7/README.md") | |
| # Removing the empty lines- | |
| non_emptyLines = myTextFile.filter(lambda line: len(line) > 0) |
NewerOlder