pietromarchesi

In this example we show how to adapt the example workflow so that we can run it on a cluster using the Slurm Workload Manager (Slurm).

We will write a workflow composed of two tasks, one which creates a file called foo.txt and writes foo in it, and one which reads foo.txt and swaps every occurrence of foo with the name of the cluster node on which the workflow is running.

To do this, we have to slightly change the workflow definition. In particular, we set up a runmode parameter which will allow us to specify from the command line whether we want the workflow to be run locally or on the cluster.

When we define the tasks, we have to pass an additional SlurmInfo object, which contains the specification of the resources we want to allocate to our tasks and other Slurm parameters.

pietromarchesi

Packaging data following the Neo model

General considerations

• A block corresponds to a recording session. Consequently, a full dataset will be composed of several blocks. A block should have annotations regarding the animal and all the dummy codes used in the different fields (e.g. if different trial types have specific numeric labels)
• Segments are used for trials. They should contain annotations regarding trial-specific variables, and event/epoch objects.
• Since this is a multi-area recording, channel indices correspond to a single area/bundle, and tetrode and channel information are ralegated to the annotations.

pietromarchesi

import luigi
import sciluigi
import os

class MyMetaWorkflow(sciluigi.WorkflowTask):

    runmode = luigi.Parameter()
    n_tasks     = luigi.IntParameter()

    def workflow(self):

pietromarchesi

import os

import neo
import numpy as np
from quantities import s
import matplotlib.pyplot as plt

import p_utils

### EXTRACTING AND PREPARING THE DATA ###

pietromarchesi

pietromarchesi