Abdul Mannan (Data Engineer|Solution Architect) rao-abdul-mannan
rao-abdul-mannan
/ Pyspark Flatten json
Created
April 20, 2021 08:49
— forked from nmukerje/Pyspark Flatten json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyspark.sql.types import * | |
| from pyspark.sql.functions import * | |
| #Flatten array of structs and structs | |
| def flatten(df): | |
| # compute Complex Fields (Lists and Structs) in Schema | |
| complex_fields = dict([(field.name, field.dataType) | |
| for field in df.schema.fields | |
| if type(field.dataType) == ArrayType or type(field.dataType) == StructType]) | |
| while len(complex_fields)!=0: |
rao-abdul-mannan
/ post-merge
Created
September 25, 2017 11:12
— forked from sindresorhus/post-merge
git hook to run a command after `git pull` if a specified file was changed.In this example it's used to run `npm install` if package.json changed and `bower install` if `bower.json` changed.Run `chmod +x post-merge` to make it executable then put it into `.git/hooks/`.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # MIT © Sindre Sorhus - sindresorhus.com | |
| # git hook to run a command after `git pull` if a specified file was changed | |
| # Run `chmod +x post-merge` to make it executable then put it into `.git/hooks/`. | |
| changed_files="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD)" | |
| check_run() { | |
| echo "$changed_files" | grep --quiet "$1" && eval "$2" |