Trying wrfpy with Cylc

testing-wrfpy-01.md

Note to self: you do not have to copy Cylc directory under wrfpy module installation in Anaconda. That was silly. Actually the package in setup.py had some problems - apparently - but installing from master branch with pip install -e . works fine.

> virtualenv venv
> . venv/bin/active
> pip install -r requirements.txt
...
> pip install -r requirements-docs.txt
...
> pip install -r requirements-test.txt
...
> pip install -r requirements-rtd.txt
...
> pip install -e .
Obtaining file:///home/kinow/Development/python/workspace/wrfpy
Requirement already satisfied: numpy in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (1.16.0)
Requirement already satisfied: Jinja2 in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (2.8)
Requirement already satisfied: MarkupSafe in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (0.23)
Requirement already satisfied: PyYAML in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (3.11)
Requirement already satisfied: f90nml in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (1.0.2)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (2.7.3)
Requirement already satisfied: astropy in ./venv/lib/python3.7/site-packages (from WRFpy==0.2.1) (2.0.9)
Requirement already satisfied: six>=1.5 in ./venv/lib/python3.7/site-packages (from python-dateutil->WRFpy==0.2.1) (1.12.0)
Requirement already satisfied: pytest>=2.8 in ./venv/lib/python3.7/site-packages (from astropy->WRFpy==0.2.1) (4.1.1)
Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages/setuptools-39.1.0-py3.7.egg (from pytest>=2.8->astropy->WRFpy==0.2.1) (39.1.0)
Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.7/site-packages (from pytest>=2.8->astropy->WRFpy==0.2.1) (18.2.0)
Requirement already satisfied: py>=1.5.0 in ./venv/lib/python3.7/site-packages (from pytest>=2.8->astropy->WRFpy==0.2.1) (1.7.0)
Requirement already satisfied: atomicwrites>=1.0 in ./venv/lib/python3.7/site-packages (from pytest>=2.8->astropy->WRFpy==0.2.1) (1.2.1)
Requirement already satisfied: pluggy>=0.7 in ./venv/lib/python3.7/site-packages (from pytest>=2.8->astropy->WRFpy==0.2.1) (0.8.1)
Requirement already satisfied: more-itertools>=4.0.0 in ./venv/lib/python3.7/site-packages (from pytest>=2.8->astropy->WRFpy==0.2.1) (5.0.0)
Installing collected packages: WRFpy
  Found existing installation: WRFpy 0.2.1
    Uninstalling WRFpy-0.2.1:
      Successfully uninstalled WRFpy-0.2.1
  Running setup.py develop for WRFpy
Successfully installed WRFpy

The running the first command in their README file:

> wrfpy --init testsuite
>>> A configuration file has been created at /home/kinow/cylc-suites/testsuite/config.json
>>> Please edit the configuration file before continuing.

This should have created ~/cylc-suites/testsuite/, with some subdirectories, but also the config.json file. Below an example of a working config.json.

{
    "filesystem": {
        "archive_dir": "",
        "boundary_dir": "",
        "obs_dir": "",
        "obs_filename": "",
        "radar_filepath": "",
        "upp_archive_dir": "",
        "upp_dir": "",
        "work_dir": "",
        "wps_dir": "",
        "wrf_dir": "",
        "wrf_run_dir": "",
        "wrfda_dir": ""
    },
    "options_general": {
        "boundary_interval": 1,
        "date_end": "2019-01-01_01",
        "date_start": "2019-01-01_00",
        "fix_urban_temps": "",
        "ref_lat": "",
        "ref_lon": "",
        "run_hours": "1"
    },
    "options_slurm": {
        "slurm_da_wrfvar.exe": "",
        "slurm_geogrid.exe": "",
        "slurm_metgrid.exe": "",
        "slurm_obsproc.exe": "",
        "slurm_real.exe": "",
        "slurm_ungrib.exe": "",
        "slurm_updatebc.exe": "",
        "slurm_wrf.exe": ""
    },
    "options_upp": {
        "upp": "",
        "upp_interval": ""
    },
    "options_urbantemps": {
        "TBL_URB": "",
        "TGL_URB": "",
        "TSLB": "",
        "ah.csv": "",
        "urban_stations": ""
    },
    "options_wps": {
        "geogrid.tbl": "",
        "metgrid.tbl": "",
        "namelist.wps": "/home/kinow/Development/python/workspace/wrfpy/wrfpy/examples/namelist.wps",
        "run_hours": "1",
        "vtable": ""
    },
    "options_wrf": {
        "namelist.input": "/home/kinow/Development/python/workspace/wrfpy/wrfpy/examples/namelist.wps",
        "urbparm.tbl": ""
    },
    "options_wrfda": {
        "be.dat": "",
        "cv_type": "",
        "namelist.wrfda": "",
        "wrfda": "",
        "wrfda_type": ""
    }
}

Note that you must edit the namelist parameters, to point to your local directory.

> wrfpy --create testsuite
> echo $?
0

If you got here with no errors, within the testsuite directory you should now have a suite.rc Cylc suite.

#!Jinja2

{% set START = "20190101T00" %}
{% set STOP  = "20190101T01" %}

[scheduling]
    initial cycle point = {{ START }}
    final cycle point   = {{ STOP }}
    [[dependencies]]
        # Initial cycle point
        [[[R1]]]
            graph = """
                wrf_init => wps => wrf_real => wrfda => wrf_run 
                obsproc_init => obsproc_run => wrfda
            """
        # Repeat every 1 hours, starting 1 hours
        # after initial cylce point
        [[[+PT1H/PT1H]]]
            graph = """
                wrf_run[-PT1H] => wrf_init => wrf_real => wrfda => wrf_run 
                wrfda[-PT1H] => obsproc_init => obsproc_run => wrfda
            """
        # Repeat every 1 hours, starting 1 hours
        # after initial cylce point
        [[[+PT1H/PT1H]]]
            graph = """
                wps[-PT1H] => wps => wrf_init
            """
[runtime]
    [[root]] # suite defaults
        [[[job submission]]]
            method = background

    [[wrf_init]]
        script = """
wrf_init.py $CYLC_TASK_CYCLE_POINT 1
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[obsproc_init]]
        script = """
wrfda_obsproc_init.py $CYLC_TASK_CYCLE_POINT
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[wrf_real]]
        script = """
run_real.py
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[wrf_run]]
        script = """
run_wrf.py
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[obsproc_run]]
        script = """
wrfda_obsproc_run.py $CYLC_TASK_CYCLE_POINT
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[wrfda]]
        script = """
wrfda_run.py $CYLC_TASK_CYCLE_POINT
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            
    [[upp]]
        script = """
upp.py $CYLC_TASK_CYCLE_POINT
"""
        [[[job submission]]]
            method = background
        [[[directives]]]
            

    [[wps]]
        pre-script = """
wps_init.py $CYLC_TASK_CYCLE_POINT 1
"""
        script = """
wps_run.py
"""
        post-script = """
wps_post.py
"""
        [[[environment]]]
            WORKDIR = wps
            CYLC_TASK_WORK_DIR = $WORKDIR
        [[[job submission]]]
            method = background
        [[[directives]]]
            

[visualization]
    initial cycle point = {{ START }}
    final cycle point   = {{ STOP }}
    default node attributes = "style=filled", "fillcolor=grey"

Unfortunately my version of Cylc reports it as invalid.

> cylc validate /home/kinow/cylc-suites/testsuite/
WARNING - deprecated items were automatically upgraded in 'suite definition':
WARNING -  * (6.11.0) [runtime][root][job submission] -> [runtime][root][job] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_init][job submission] -> [runtime][wrf_init][job] - value unchanged
WARNING -  * (6.11.0) [runtime][obsproc_init][job submission] -> [runtime][obsproc_init][job] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_real][job submission] -> [runtime][wrf_real][job] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_run][job submission] -> [runtime][wrf_run][job] - value unchanged
WARNING -  * (6.11.0) [runtime][obsproc_run][job submission] -> [runtime][obsproc_run][job] - value unchanged
WARNING -  * (6.11.0) [runtime][wrfda][job submission] -> [runtime][wrfda][job] - value unchanged
WARNING -  * (6.11.0) [runtime][upp][job submission] -> [runtime][upp][job] - value unchanged
WARNING -  * (6.11.0) [runtime][wps][job submission] -> [runtime][wps][job] - value unchanged
WARNING -  * (6.11.0) [runtime][root][job][method] -> [runtime][root][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_init][job][method] -> [runtime][wrf_init][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][obsproc_init][job][method] -> [runtime][obsproc_init][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_real][job][method] -> [runtime][wrf_real][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][wrf_run][job][method] -> [runtime][wrf_run][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][obsproc_run][job][method] -> [runtime][obsproc_run][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][wrfda][job][method] -> [runtime][wrfda][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][upp][job][method] -> [runtime][upp][job][batch system] - value unchanged
WARNING -  * (6.11.0) [runtime][wps][job][method] -> [runtime][wps][job][batch system] - value unchanged
Valid for cylc-7.8.1

Huh, PTH should presumably be something like PT6H (6 hours).

Oh! You just found a bug in wrfpy @hjoliver! There is some validation for config.json, but not for the run_hours field. Without this value, the period are created as you noted, PTH. Updating the gist with the valid suite.rc now, and reporting the bug to wrfpy devs. Thanks!!