Andrea Tommaso Bonanno andreatbonanno

Data Engineer with a strong interest in Big Data, Functional Programming and Machine Learning. 5+ years commercial experience as a Software Engineer.

andreatbonanno / gist:176dec0cf2c52b0941a8ac67597a4758

Created December 5, 2017 18:13

	### Keybase proof

	I hereby claim:

	* I am andreatbonanno on github.
	* I am andreatbonanno (https://keybase.io/andreatbonanno) on keybase.
	* I have a public key ASDRv-rtemwDfZNawiKL284ghOZwv7mFuvEny_kClxtO5go

	To claim this, I am signing this object:

andreatbonanno / CreateHiveTableWithPartitions.scala

Last active November 21, 2017 19:22

Spark: create hive external table with partitions (from partitioned parquet file in hdfs)

	import org.apache.hadoop.fs.{FileSystem, Path}
	import org.apache.spark.sql.SparkSession

	def listHdpFiles(filePath: String, excludeFilesFrom: String = ""): Array[String] = {
	FileSystem
	.get(sc.hadoopConfiguration)
	.listStatus(new Path(filePath))
	.map(fileStatus => fileStatus.getPath.toString)
	.filter(filePath => filePath > excludeFilesFrom)
	}