venkyvb

Is Spark's JSON schema inference too inflexible for your liking?

Common Scenarios:

• Automatic schema inference from Spark is not applying your desired type casting
• You want to completely drop irrelevant fields when parsing
• You want to avoid some highly nested fields simply by casting some outer fields as strings

Step 1: Provide your (ideal) JSON data example

REFERENCE_EXAMPLE = {
  "firstName": "Will",

venkyvb

root
 |-- abatementAge: struct (nullable = true)
 |    |-- code: string (nullable = true)
 |    |-- system: string (nullable = true)
 |    |-- unit: string (nullable = true)
 |    |-- value: long (nullable = true)
 |-- abatementDateTime: string (nullable = true)
 |-- abatementString: string (nullable = true)
 |-- address: array (nullable = true)
 |    |-- element: struct (containsNull = true)