[Generate an SOF from directory of FITS files] #fits #sof #esorex

0README.md

Set of Frames (SOF) Files

Here's an example of sof (set of frames) files generated by the XSHooter pipeline when creating a master bias frame.

There are 4 files creates:

1. data.sof
2. input_sof.json
3. output_files.sof
4. products_sof.json

Here is what the ESO esorex docs say about the sof files.

A sof file contains a list of the input data. This data is specified in an sof file (which is just a text file), where each input file is specified with its associated classification and category. The format of each line in the sof file is as follows:

full-path-to-file  classification

The first of the files above data.sof seem to meet these criteria and contains the path to the input bias frames and what appears to be their HIERARCH ESO DPR TYPE keyword together with HIERARCH ESO SEQ ARM (classification tag):

/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:34:46.596.fits BIAS_UVB_1x1
/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:36:19.284.fits BIAS_UVB_1x1
/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:37:52.583.fits BIAS_UVB_1x1
/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:39:24.501.fits BIAS_UVB_1x1
/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:40:58.590.fits BIAS_UVB_1x1

There is also a JSON version of the SOF with much more detailed information. Here is a file called 'input_sof.json' with the same set of files as above:

[
  {
    "purposes": [
      "ACTION_MBIAS_UVB/ACTION_FLUX_SLIT_NOD_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:20:39.700.fits",
    "checksum": "500031168314E955F0A3533660CA7E5A",
    "category": "BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  },
  {
    "purposes": [
      "ACTION_MBIAS_UVB/ACTION_FLUX_SLIT_NOD_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:23:27.505.fits",
    "checksum": "557B300BF8E403413101C543B454C1EC",
    "category": "BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  },
  {
    "purposes": [
      "ACTION_MBIAS_UVB/ACTION_FLUX_SLIT_NOD_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:26:17.621.fits",
    "checksum": "5D5C8D9032EDF7D385F952D92F8CD370",
    "category": "BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  },
  {
    "purposes": [
      "ACTION_MBIAS_UVB/ACTION_FLUX_SLIT_NOD_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:29:05.346.fits",
    "checksum": "4A96894259FD95CAB09416B1711BADAC",
    "category": "BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  },
  {
    "purposes": [
      "ACTION_MBIAS_UVB/ACTION_FLUX_SLIT_NOD_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/raw_data/XSHOO.2017-08-18T21:31:55.631.fits",
    "checksum": "86CF0957ABEAE8AB5074DB6EF4AF863A",
    "category": "BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  }
]

Alongside input_sof.json there are also output_files.sof and products_sof.json

[
  {
    "purposes": [
      "ACTION_MFLAT_SLIT_UVB/ACTION_SCI_SLIT_STARE_UVB"
    ],
    "name": "/Users/Dave/Dropbox/Desktop/xshooter_qub_demo/other_data/reflex_tmp_products/xshooter/xsh_mbias_1/2019-09-30T10:36:13.810/MASTER_BIAS_UVB.fits",
    "checksum": "9149D64DB44F7EBEFF5B22D82E0A8F08",
    "category": "MASTER_BIAS_UVB",
    "class": "org.eso.domain.FitsFile"
  }
]

[
  {
    "name": "\/Users\/Dave\/Dropbox\/Desktop\/xshooter_qub_demo\/other_data\/reflex_tmp_products\/xshooter\/xsh_mbias_1\/2019-09-30T10:37:11.042\/MASTER_BIAS_UVB.fits",
    "category": "MASTER_BIAS_UVB"
  }
]

Generating a SOF File

To generate an sof file for a directory containing FITS files run the command:

python create_sof_from_directory_contents.py /path/to/folder > my_sof_file.sof

You can also filter on arm, binning and frame type:

create_sof_from_directory_contents <directoryPath>
    create_sof_from_directory_contents <directoryPath> <type>
    create_sof_from_directory_contents <directoryPath> <type> <arm>
    create_sof_from_directory_contents <directoryPath> <type> <arm> <xbin> <ybin>
    ```

create_sof_from_directory_contents.py

#!/usr/bin/env python
# encoding: utf-8
"""
*Given a path to a directory, generate an ESORex complient 'set of frames' sof file with the FITS files contained in the directory*

:Author:
    David Young

:Date Created:
    December  9, 2019

Usage:
    create_sof_from_directory_contents <directoryPath>
    create_sof_from_directory_contents <directoryPath> <type>
    create_sof_from_directory_contents <directoryPath> <type> <arm>
    create_sof_from_directory_contents <directoryPath> <type> <arm> <xbin> <ybin>

Options:
    directoryPath         path to the directory containing the FITS frames
    type                  frame type from "HIERARCH ESO DPR TYPE"
    arm                   frame type from "HIERARCH ESO SEQ ARM"
    xbin                  binning in x-axis, CDELT1
    ybin                  binning in y-axis, CDELT2

    -h, --help            show this help message
    -v, --version         show version
    -s, --settings        the settings file
"""
################# GLOBAL IMPORTS ####################
import sys
import os
from fundamentals import tools
from astropy.io import fits


def main(arguments=None):
    """
    *The main function used when ``create_sof_from_directory_contents.py`` is run as a single script from the cl*
    """

    # SETUP THE COMMAND-LINE UTIL SETTINGS
    su = tools(
        arguments=arguments,
        docString=__doc__,
        logLevel="WARNING",
        options_first=False,
        projectName=False
    )
    arguments, settings, log, dbConn = su.setup()

    # UNPACK REMAINING CL ARGUMENTS USING `EXEC` TO SETUP THE VARIABLE NAMES
    # AUTOMATICALLY
    a = {}
    for arg, val in list(arguments.items()):
        if arg[0] == "-":
            varname = arg.replace("-", "") + "Flag"
        else:
            varname = arg.replace("<", "").replace(">", "")
        a[varname] = val
        if arg == "--dbConn":
            dbConn = val
            a["dbConn"] = val
        log.debug('%s = %s' % (varname, val,))

    for d in sorted(os.listdir(a["directoryPath"])):
        if os.path.isfile(os.path.join(a["directoryPath"], d)) and os.path.splitext(d)[-1].lower() == ".fits":
            fitsPath = os.path.abspath(os.path.join(a["directoryPath"], d))
            # OPEN FITS FILE AT HDULIST - HDU (HEADER DATA UNIT) CONTAINS A HEADER AND A DATA ARRAY (IMAGE) OR
            # TABLE.
            with fits.open(fitsPath) as hdul:
                # READ HEADER INTO MEMORY
                hdr = hdul[0].header
                # PRINT FULL FITS HEADER TO STDOUT
                # print(repr(hdr).strip())
                dpr_type = hdr['HIERARCH ESO DPR TYPE'].strip()
                arm = hdr['HIERARCH ESO SEQ ARM']
                xbin = str(int(hdr['CDELT1']))
                ybin = str(int(hdr['CDELT2']))
                catagory = dpr_type + "_" + arm.strip() + "_" + \
                    xbin.strip() + "x" + ybin.strip()
                # CHECK TYPE OF IMAGE
                if a["type"] and dpr_type.lower() != a["type"].lower():
                    continue
                # CHECK ARM
                if a["arm"] and arm.lower() != a["arm"].lower():
                    continue
                # CHECK BINNING
                if a["xbin"] and (xbin.lower() != a["xbin"].lower() or ybin.lower() != a["ybin"].lower()):
                    continue
                print(fitsPath, catagory)

    return

if __name__ == '__main__':
    main()