Skip to content

Instantly share code, notes, and snippets.

@dlebauer

dlebauer/01-events_spec.md

Last active July 1, 2025 20:32
Show Gist options
  • Save dlebauer/5a2e505bca6afe66ca6aa44d9dd163f7 to your computer and use it in GitHub Desktop.
Save dlebauer/5a2e505bca6afe66ca6aa44d9dd163f7 to your computer and use it in GitHub Desktop.
Download ZIP
Specification for PEcAn events format
Raw
01-events_spec.md

Events format

This specification defines a model-agnostic intermediate format for events, analogous to the PEcAn met standard. It is intended to:

  • Consolidate events from diverse sources (e.g. RS, farmer surveys, literature)
  • Represent ensemble-reesolved realizations of events that can be used in modeling workflows.
  • Be consumed by a model specific event parser like parse.events.<model> that can be called by write.configs.<model>.

This format for events that is based on the format used in the fieldactivity nested within a geoJSON data structure1, but has been modified to be more generalized, and focused on aggregating data from many sources that have varied types of information.

The format is a GeoJSON FeatureCollection. Each Feature in the collection represents a single site. The properties of each feature contain an array of events that occurred at that site.

These properties can be assigned a site_id and handled separately, e.g. as one JSON file for each combination of site / scenario / ensemble member.

They can also be aggregated into GeoJSON and stored for efficient storage and retrieval e.g. in a Parquet file (or other JSON supporting database).

Approach

Minimum requirements depend on use case (model, data source)

MVP spec support generation of model-specific input files (initially, the motivating use-case of SIPNET).

We define a core minimum required parameters - namely event_type, date, location. Beyond that, required fields will vary depending on:

  • event type
  • data source and method used to estimate optional parameters
  • target model or workflow

This will necessarily require flexibility and prevents establishing a broad set of global requirements.

Model-specific translation happens downstream

Events spec should contain model-agnostic information. And the aim is to eliminate or minimize model or method specific information in this format. Model specific information may be stored in additional JSON objects associated with events, but this will not be part of the specification.

Each model will implement its own parse_events.<model> function that is called by write.configs.<model>.

Raw vs processed data products are distinct

  • Upsteram raw/ observed data may include probabilistic or uncertain inputs (e.g. tillage probability, RS index values)
  • Model-ready realizations: deterministic with ensemble based uncertainty

Will evolve iteratively

Rather than defining a universal standard up front, we build toward it by grounding in motivating use-cases and actual implementation.

Structure

Each Feature has a geometry and a properties object. The properties object contains site identifiers and an events array.

Events are represented as a `FeatureCollection` where each `Feature` is a management event.

```json
{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [ ... ]
      },
      "properties": {
        "...": "..."
      }
    }
  ]
}

Feature Properties:

  • site: The site identifier.
  • events: An array of event objects.

Event Object:

  • date: The date of the event in YYYY-MM-DD format.
  • mgmt_operations_event: The type of management event.
  • mgmt_event_notes: General notes for the event.
  • ... and other event-specific properties.

Minimum Working Example

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [ -121.50, 37.20 ]
      },
      "properties": {
        "site": "9900000099",
        "ensemble": "1",
        "events": [
          {
            "mgmt_operations_event": "tillage",
            "date": "2023-09-08",
          },
          {
            "mgmt_operations_event": "planting",
            "date": "2023-09-12",
            "planted_crop": "RYE",
          },
          {
            "mgmt_operations_event": "fertilizer",
            "date": "2019-06-20",
            "fertilizer_type": "fertilizer_type_mineral",
            "N_in_applied_fertilizer": 23,
          },
          {
            "mgmt_operations_event": "harvest",
            "date": "2018-06-12",
            "harvest_crop": "Rye"
          },
          {
            "mgmt_operations_event": "irrigation",
            "date": "2018-06-12",
            "irrig_amount_depth": "5"
          }

        ]
      }
    }
  ]
}

Extended Example

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [ -121.50, 37.20 ]
      },
      "properties": {
        "site": "9900000099",
        "ensemble": "1",
        "events": [
          {
            "mgmt_operations_event": "tillage",
            "date": "2023-09-08",
            "tillage_practice": "tillage_practice_primary",
            "tillage_implement": "TI002"
          },
          {
            "mgmt_operations_event": "planting",
            "date": "2023-09-12",
            "planted_crop": ["RYE", "ZZ3"],
            "planting_material_weight": [158.7, 12.4],
            "planting_notes": "Rye: sangaste"
          },
          {
            "mgmt_operations_event": "fertilizer",
            "date": "2019-06-20",
            "fertilizer_total_amount": 220,
            "N_in_applied_fertilizer": 23,
            "phosphorus_applied_fert": 10,
            "fertilizer_K_applied": 8
          },
          {
            "mgmt_operations_event": "harvest",
            "date": "2018-06-12",
            "harvest_operat_component": "leaf",
            "harvest_area": 4.3,
            "harvest_cut_height": 6,
            "harvest_yield_harvest_dw": 1985,
            "yield_C_at_harvest": 830
          }
        ]
      }
    }
  ]
}

Disambiguating Different Information with Related Meaning

Example: Required fields for tillage intensity

Field Type Required Description
event_type string yes Must be "tillage"
date date yes ISO 8601 date
intensity float optional E.g., 0.3
intensity_method string optional Method used (e.g., "NDVI-diff [link]" or "USDA DB [link]")

If intensity is provided, include intensity_method to describe meaning.

Required and Optional Fields by Event Type

Event Type Required Fields Optional Fields
planting date, crop_type method, seed_density
harvest date, crop_type fraction_removed, method
fertilizer date, amount_N, type amount_C, mineral_vs_organic
tillage date intensity, intensity_method
irrigation date, amount type, method

Note: this - especially optional fields - will necesarrily evolve to handle needs of different data sources, methods, and models.

Event-Specific Properties

For properties with enumerated values, we use a code (e.g., HARM, FECD) from the ICASA Dictionary. There is a lookup table for these in the FieldActivity repository under inst/display_names.csv

planting

  • planted_crop (from CRID code list)
  • planting_material_weight
  • planting_depth
  • planting_material_source
  • planting_notes

harvest

  • harvest_area
  • harvest_crop (from CRID code list)
  • harvest_yield_harvest_dw
  • harv_yield_harv_f_wt
  • yield_C_at_harvest
  • harvest_moisture
  • harvest_method (from HARM code list)
  • harvest_operat_component (from HACOM code list)
  • canopy_height_harvest
  • harvest_cut_height
  • plant_density_harvest
  • harvest_residue_placement (from harvest_residue_placement_choice list)
  • harvest_comments
  • harvest_amount
  • harv_operat_main_product
  • harv_operation_byproduct
  • harv_operat_size_categor
  • harvest_yld_stdev
  • harv_yield_harv_f_wt_sd

tillage

  • tillage_practice (from tillage_practice_choice list)
  • tillage_implement (from TIIMP code list)
  • tillage_operations_depth
  • tillage_treatment_notes

fertilizer

  • fertilizer_type (e.g., fertilizer_type_organic, fertilizer_type_mineral, fertilizer_type_soil_amendment)
  • organic_material (from OMCD code list)
  • animal_fert_usage
  • org_matter_moisture_conc
  • org_matter_carbon_conc
  • fertilizer_product_name
  • fertilizer_material (from FECD code list)
  • fertilizer_material_source
  • fertilizer_applic_method (from FEACD code list)
  • application_depth_fert
  • fertilizer_total_amount
  • N_in_applied_fertilizer
  • N_in_soluble_fertilizer
  • phosphorus_applied_fert
  • fertilizer_K_applied
  • S_in_applied_fertilizer
  • Ca_in_applied_fertilizer
  • Mg_in_applied_fertilizer
  • Na_in_applied_fertilizer
  • Cu_in_applied_fertilizer
  • Zn_in_applied_fertilizer
  • B_in_applied_fertilizer
  • Mn_in_applied_fertilizer
  • Se_in_applied_fertilizer
  • Fe_in_applied_fertilizer
  • other_element_in_applied_fertilizer
  • fertilizer_notes

grazing

  • grazing_species (from grazing_species_choice list)
  • grazing_species_age_group (e.g., 0-1, 1-2, 2-3, 3-5, 5-10, 10+, grazing_species_age_group_mix)
  • livestock_density
  • grazing_intensity
  • grazing_period (date range, may also be represented by start_date and end_date)
  • grazing_type (from grazing_type_choice list)
  • grazing_area
  • grazing_material_removed_prop
  • grazing_starting_height
  • grazing_end_height
  • grazing_notes

mowing

  • mowed_crop (from CRID code list)
  • mowed_area
  • mowing_method (from mowing_method_choices list)
  • mowing_canopy_height
  • mowing_cut_height
  • mowing_notes

chemicals

  • chemical_type (from chemical_type_choice list)
  • chemical_product_name
  • chemical_applic_material (from active_substance code list)
  • chemical_applic_target (from CH_TARGETS code list)
  • chemical_applic_method (from FEACD code list)
  • chemical_applic_amount
  • application_depth_chem
  • application_ph_start
  • application_ph_end
  • chemical_applic_notes

pests

  • plant_pop_reduct_cum

management

  • fuel_amount

irrigation

  • irrigation_operation (from IROP code list, e.g., soil, canopy, drip, furrow, flood)
  • irrig_amount_depth
  • irrigation_applic_depth
  • irrigation_notes

inorg_mulch

  • mulch_type (from MLTP code list)
  • mulch_thickness
  • mulch_cover_fraction
  • mulch_color (from MLCOL code list)
  • mulch_placement_notes

Inorg_mul_rem

  • mulch_type_remove (from MLTP code list)
  • mulch_removal_notes

weeding

  • weeding_notes

bed_prep

  • bed_prep_notes

other

  • other_notes

organic_material (Legacy)

Note: This event type appears in some data sources. It is recommended to record organic material applications as a fertilizer event for consistency.

  • organic_material (from OMCD code list)
  • org_material_applic_meth (alias for fertilizer_applic_method)
  • org_material_appl_depth (alias for application_depth_fert)
  • org_material_applic_amnt (alias for fertilizer_total_amount)
  • org_material_cover
  • org_matter_moisture_conc
  • org_matter_carbon_conc
  • organic_material_N_conc
  • organic_material_P_conc
  • organic_material_K_conc
  • org_material_lignin_conc
  • org_material_c_to_n
  • org_material_notes

measurement (Legacy)

Note: This event type appears in some data sources. It is recommended to record measurements as an observation event for consistency.

  • carbon_soil_tot
  • carbon_soil_tot_sd

Observations from schema / not actually "EVENTS"????

Observations are used in the fieldActivity app so that users can record measurements or observations that are not directly tied to a management event.

Since "observations" are a separate concept from "events", we may want to either a) not use this (as we already have a way of storing this information) or b) optionally create a separate Observations section (outside the agronomic‐events GeoJSON spec) to avoid confusion.

observation

  • observation_type (e.g., soil, vegetation, water, pests, management)
  • observation_notes

observation_type:

soil

  • soil_layer_top_depth
  • soil_layer_base_depth
  • soil_classification_by_layer (from soil_classification_by_layer_choice list)
  • root_depth
  • soil_compactification_depth
  • earthworm_count
  • soil_image (URL)
  • carbon_soil_tot
  • carbon_soil_tot_sd

vegetation

  • growth_stage (from growth_stage_choice list)
  • plant_density
  • specific_leaf_area
  • leaf_area_index
  • canopy_height
  • canopeo_reading
  • canopeo_image (URL)
  • total_biomass_dw
  • tops_C
  • tops_C_std
  • roots_C
  • roots_C_std

water

  • floodwater_depth
  • water_table_depth

Footnotes

  1. The fieldactivity JSON evennt schema is defined in ui_structure.json. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment