canimus/pyspark-json.py

Created January 4, 2021 09:41

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/canimus/364eb2fb976fc941fac838541c2337ff.js"></script>
Save canimus/364eb2fb976fc941fac838541c2337ff to your computer and use it in GitHub Desktop.

Download ZIP

PySpark Union JSON

Raw

pyspark-json.py

df = reduce(lambda a,b: a.union(b), [spark.read.json(f'{x}/*result.json') for x in range(195, 202)])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment