Skip to content

Instantly share code, notes, and snippets.

@Kelvinrr

Kelvinrr/RFC13 Draft.MD

Last active December 30, 2024 19:57
Show Gist options
  • Save Kelvinrr/4ed921e89f1692d08c7f3bf571f3f067 to your computer and use it in GitHub Desktop.
Save Kelvinrr/4ed921e89f1692d08c7f3bf571f3f067 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
RFC13 Draft.MD
  • Feature/Process Name: GeoTiff Metadata for ISIS
  • Start Date: 12/19/24
  • RFC PR: DOI-USGS/ISIS3#5618
  • Author: Kelvin Rodriguez, Adam Paquette

Terms

  • PVL - Parameter Value Language, used widely in ISIS Cubes and PDS3 images.
  • ISIS - Integrated Software for Imagers and Spectrometers, well known and widely used processing software for planetary data processing.
  • GTiff - The GeoTiff file format widely used within the planetary science community for data processing and figures in scientific papers.
  • COG - Cloud-optimized Geotiffs, GTiff image with metadata organized to stream from a HTTP server. Here, COG and GTIFF is used interchangeably.
  • Cube - ISIS specific file format. A combination of a PVL header, and structured binary data written as "blobs" to the file
  • Metadata - Data about data. Specifically, any information that a program would need to both encode and decode data written to a file
  • GDAL - Geospatial Data Abstraction Library, well known and widely used software for geospatial image processing.

Summary

ISIS will be changing how libisis I/O handles images and move to leveraging GDAL as its primary engine. This is predominately to enable processing GeoTiffs/COGs which requires ISIS supported metadata to read, write and operate on GeoTiffs from ingestion. We have been successful in reading GeoTiffs using gdal. Here, we propose a structure for ISIS metadata when working with images that are not ISIS cubes.

Motivation

ISIS LTP, NASA SAT findings, and, by extension, the community has identified that a priority for ISIS is to have improved interoperability with other software and enhance our software capability to access data via the cloud, especially in supporting operations in HPC environments. Better interoperability demands we use community-supported standards wherever possible and establish metadata standards for operations novel to our software. We decided to emphasize support for COGs by using GDAL for the underlying I/O operations for DNs and labels. This is expected to allow users to stream images (including ISIS's 2.5D shape models) over the web using GDAL's VSI interface.

ISIS expects formatted label data to support any operation post-ingestion. These data do not match how GDAL works if you use gdal_translate to convert from ISIS Cube to Geotiff or any other format. Beyond some inconsistencies in translation (we have found numbers can completely change on translation), the Cube is exploded into multiple files with a detached label, one for the label, one for each ISIS table, another for DNs. This is because GDAL does not have a metadata format compatible with both ISIS and other GDAL formats without using detached labels. For this reason, we are proposing a single-file metadata format that works with GDAL API. We expect to contribute these changes to GDAL's ISIS3 Cube driver to support it through gdal_translate when converting ISIS3 Cubes to COGs.

We expect users to benefit by:

  1. Allowing scientists to have a better experience working with images on HPC and cloud environments using VSICURL
  2. Enabling compressed images, moderately speeding up I/O times
  3. Enhancing interoperability between ISIS and other non-USGS software

Proposed Solution / Explanation

The proposed solution would allow ISIS to read and write GeoTiffs using GDAL, maintaining Cube reading and writting. To write GTiffs in ISIS, users can set the format of the output file using the same facilities that they would set the format of a Cube. By adding a "+GTiff" to the output image in any ISIS program, the output image produced from the program will be a GeoTiff with all data, and metadata necessary to work with ISIS. Bellow is an example of the format translation:

cubeatt from=initial_cube.cub to=initial_GTiff.tiff+GTiff

The process above, does a one-to-one translation from the ISIS Cube to a GeoTiff. The Cube PVL header is placed onto the GeoTiffs metadata as JSON. The binary blobs that would be placed on the ISIS Cube also have PVL headers that are converted to JSON and placed into the GeoTiff metadata.

Since GeoTiffs cannot store binary data like Cubes, the binary blobs that would be placed into the Cubes are converted into a format that can be written to the GeoTiff. Bellow is what the ISIS PVL blob metadata looks like vs the converted data that will be written to the GeoTiff:

Object = Table
  Name                 = SunPosition
  StartByte            = 493963254
  Bytes                = 112
  Records              = 2
  ByteOrder            = Lsb
  CacheType            = Linear
  SpkTableStartTime    = 297088762.24158
  SpkTableEndTime      = 297088808.37074
  SpkTableOriginalSize = 2.0
  Description          = "Created by spiceinit"
  Kernels              = ($base/kernels/spk/de430.bsp,
                          $base/kernels/spk/mar097.bsp)

  Group = Field
    Name = J2000X
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = J2000Y
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = J2000Z
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = J2000XV
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = J2000YV
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = J2000ZV
    Type = Double
    Size = 1
  End_Group

  Group = Field
    Name = ET
    Type = Double
    Size = 1
  End_Group
End_Object

VS.

{"Table_SunPosition": 
    '{
        "_container_name":"Table",
        "_type":"object",
        "Field_J2000X":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000X",
            "Type":"Double",
            "Size":"1"
        },
        "Field_J2000Y":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000Y",
            "Type":"Double",
            "Size":"1"
        },
        "Field_J2000Z":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000Z",
            "Type":"Double",
            "Size":"1"
        },
        "Field_J2000XV":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000XV",
            "Type":"Double",
            "Size":"1"
        },
        "Field_J2000YV":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000YV",
            "Type":"Double",
            "Size":"1"
        },
        "Field_J2000ZV":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"J2000ZV",
            "Type":"Double",
            "Size":"1"
        },
        "Field_ET":{
            "_container_name":"Field",
            "_type":"group",
            "Name":"ET",
            "Type":"Double",
            "Size":"1"
        },
        "Name":"SunPosition",
        "StartByte":"1",
        "Bytes":"112",
        "Records":"2",
        "ByteOrder":"Lsb",
        "CacheType":"Linear",
        "SpkTableStartTime":"297088762.24158",
        "SpkTableEndTime":"297088808.37074",
        "SpkTableOriginalSize":"2.0",
        "Description":"Created by spiceinit",
        "Kernels":[
            "$base/kernels/spk/de430.bsp",
            "$base/kernels/spk/mar097.bsp"
        ],
        "Data":"4cffffffd401ffffffe62effffffd3ffffffa8ffffffc11d03ffffffffffffff8525495dffffffc157fffffffb03ffffff89ffffff800c404114ffffffd1ffffffa7ffffffb6ffffffecffffffe7ffffffcaffffffbfffffffc6fffffffa48ffffffd7ffffffe1ffffffe637ffffffc0246fffffff93ffffffae39ffffffea25ffffffc074ffffffd83dfffffffa36ffffffb5ffffffb141fffffff6ffffffc264fffffff92effffffd3ffffffa8ffffffc1503cffffffb32a394a5dffffffc1fffffff5fffffff746ffffffceffffff830b4041ffffffc6066d1b4fffffffe3ffffffcaffffffbf6c4f1dffffff80ffffffe1ffffffe637ffffffc03fffffffb246ffffffde39ffffffea25ffffffc0ffffff8fffffffe85e2837ffffffb5ffffffb141"
    }'
}

The PVL metadata for each blob is converted into JSON with the binary data placed at the "Data" key within the JSON. The JSON is written to the GeoTiff under the Domain "USGS", and Keyed based on the name and type of the blob. In the example above the blob is an ISIS "Table". Its type is set to "Table", and name set to "SunPosition" making its metadata key "Table_SunPosition".

Users should not see a change in any workflow that uses Cubes with all data attached to them. Both Cubes and GTiffs should process the same in ISIS and produce the same end results for any given pipeline.

The GTiff format will not support either "Detached" or "External" labels that Cubes support.

Drawbacks

The main drawback for this functionality is adding new bugs into the existing software. If reading and writing GeoTiffs in ISIS or the interoperability between the GeoTiff format and other libraries is not important enough to the project, then these changes should not be considered. It is specifically for the interoperability between ISIS and other software libraries that we believe the risk is worth it.

Once the software is done, we plan to get it in users hands for testing before release.

Alternatives

There are a few alternatives to using GDAL and reading/writing GeoTiffs. There are plenty of other file formats that can manage image data (jpeg, png, gif, etc.), and a metadata sidecar file could we used to manage the data. This approach is bulky and requires users to manage both the image file and the side car file as they process data. Mainly, when moving data around directories, making sure that the image file and the metadata sidecar are collocated. Since ISIS blobs/metadata can be placed directly into the GeoTiff metadata, GeoTiffs do not have the issue that the alternative presents.

There are other Geospatial image formats that we could use as well like Cloud Raster Format, that support writing custom metadata. All of these formats offer the same suite of functionality that GeoTiffs offer.

While alternatives to GeoTiffs exist, we believe that they are the best format to support in ISIS. GeoTiffs are widely used by the planetary science community, are an open source format, and the format is not bogged down by sidecar files. Finally, GeoTiffs support all of the necessary functionality to be compatible with ISIS and support Geospatial image process in a wide varierty of other applications.

Unresolved Questions

Custom metadata will be written to ISIS produced GeoTiffs, so other libraries that want to take advantage of reading/writing that information will need to update. This poses a question to libraries like GDAL for proper metadata translation between file formats. The metadata that ISIS will write also differs from the metadata written by gdal_translate (a gdal application that can translate between file formats). We will need to be in touch with the GDAL team and, ideally, change GDAL to write the same metadata that ISIS will be writing.

Future Possibilities

  • When a format is agreed on and implemented into ISIS, we expect to contribute it to GDAL so users can convert cubes without necessarily using ISIS.
  • COG support + a cloud-optimized vector file format for Control Networks will allow for web-enabled bundle adjustments.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment