Created
June 19, 2013 20:50
-
-
Save aishfenton/5817920 to your computer and use it in GitHub Desktop.
Converts file from TSV format with quoted fields into a format compatible with Hadoop Pig.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env awk -f | |
| BEGIN { | |
| SEP="\t" | |
| FS="\"" SEP "\"|^\""; | |
| } | |
| { | |
| out=$2; | |
| for (i=3;i<=NF-1;i++) { | |
| gsub(/\t/, "_", $i); | |
| out=out"" SEP ""$i | |
| }; | |
| # Last column | |
| gsub(/"\r?$/, "", $NF); | |
| gsub(/\t/, "_", $NF); | |
| out=out "" SEP "" $NF | |
| if (NR!=1) {print out} | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Also strips off header row