PawaritL

geojson = {
  "type": "Polygon",
  "coordinates": [
    [
      [179.0, 0.0], 
      [-179.0, 0.0], 
      [-179.0, 1.0],
      [179.0, 1.0], 
      [179.0, 0.0]
    ]

PawaritL

Is Spark's JSON schema inference too inflexible for your liking?

Common Scenarios:

• Automatic schema inference from Spark is not applying your desired type casting
• You want to completely drop irrelevant fields when parsing
• You want to avoid some highly nested fields simply by casting some outer fields as strings

Step 1: Provide your (ideal) JSON data example

REFERENCE_EXAMPLE = {
  "firstName": "Will",

PawaritL

def check_crossing(lon1: float, lon2: float, validate: bool = False, dlon_threshold: float = 180.0):
    """
    Assuming a minimum travel distance between two provided longitude coordinates,
    checks if the 180th meridian (antimeridian) is crossed.
    """

    if validate and any([abs(x) > 180.0 for x in [lon1, lon2]]):
        raise ValueError("longitudes must be in degrees [-180.0, 180.0]")   
    return abs(lon2 - lon1) > dlon_threshold

PawaritL

geojson = {
  "type": "Polygon",
  "coordinates": [
    [
      [179.0, 0.0], 
      [-179.0, 0.0], 
      [-179.0, 1.0],
      [179.0, 1.0], 
      [179.0, 0.0]
    ]