O9-3.tex

\documentclass[11pt,twoside]{article}
\usepackage{asp2014}

\aspSuppressVolSlug
\resetcounters

\bibliographystyle{asp2014}

\markboth{Molenaar}{Easy to deploy and easy to modify data reduction pipelines using KERN and CWL}

\begin{document}

\title{Easy to deploy and easy to modify data reduction pipelines using KERN and CWL}


\author{Gijs~Molenaar$^1$}
\affil{$^1$pythonic.nl; \email{gijs@pythonic.nl}}

\paperauthor{Gijs~Molenaar}{gijs@pythonic.nl}{ORCID}{pythonic.nl}{Author1 Department}{City}{State/Province}{Postal Code}{Country}

% leave these next few aindex lines commented for the editors to enable them. Use Aindex.py to generate them for yourself.
% first presenting author should be the first entry for bold-facing the author index page-reference
%\aindex{Molenaar,~G.}
%\aindex{Author2,~S.}
% remove/add as you need

% leave the ssindex lines commented for the editors to enable them, use Index.py to suggest yours
%\ssindex{FOOBAR!conference!ADASS 2019}
%\ssindex{FOOBAR!organisations!ASP}

% leave the ooindex lines commented for the editors to enable them, use ascl.py to suggest yours
%\ooindex{FOOBAR, ascl:1101.010}
  
\begin{abstract}
In this work we describe a solution to simplify deployment of complex scientific software.
The solution is a combination of organisational and technical practices.
For software version and distribution management we have created KERN\cite{molenaar2018kern},
an superset of Ubuntu/Debian packages.
It should contain most of the standard tools that a radio astronomer needs to work with radio telescope data.
To deploy these software package we are using containerisation technoglogy like Docker and Singularity.
To chain the softwares together in pipelines we use a abstract pipeline modeling framework nammed Common Workflow Language (CWL).
\end{abstract}


\section{KERN, containerisation and Common Workflow Language}

Radio telescopes are producing more data than ever before.
The data rates of future telescopes, like SKA,
will be of such a great scale making it infeasible to keep the end-user astronomer involved in the imaging and calibration step.

However, by making the pipelines open access, modular, easy to deploy and easy to modify,
we believe that involving the end-user astronomer in the calibration decision process might not be that far-fetched.
Creating technical and logistical pipeline infrastructure is pivotal,
and will eventually result in better data products and better science.

Our solution consists of multiple layers.
The base layer is the packaging of all the relevant software,
and regularly releases a bundle of these packages as software distributions.
This created a solid basis for reproducible science.
Secondly, to address the portability and deployment issue,
we adopt container technology with singularity currently as the primary choice.
Lastly, to recombine these packages into data reduction pipelines,
we embrace the Common Workflow Language (CWL).
CWL is an open and free standard software package which is implemented by numerous pipeline running frameworks.

Decomposing the data reduction problem in this matter enables recombining and modifying pipelines in a high-level abstract way,
while also enabling implicit parallelisation due to the functional (programming) nature of the standard.
Furthermore, it represents all available software as with uniform interfaces resulting in simplifying storage,
representation and visualisation of both parameters and (intermediate) results.

At ASTRON and SKA South Africa, successful experiments have been completed based on the described ensemble during recent years.
These are slowly being transferred into production.
Still, there is much more work to be done.
The software stack needs improvements to make it future ready,
and the packaging requires continuous effort in need of financing.

More information can be found on the KERN website: http://kernsuite.info

\end{document}