Skip to content

Instantly share code, notes, and snippets.

@akhanf

akhanf/snakemake_iterative_template

Last active October 2, 2020 23:52
Show Gist options
  • Save akhanf/9f1e8f91d2b53be880fa148843bd2ee1 to your computer and use it in GitHub Desktop.
Save akhanf/9f1e8f91d2b53be880fa148843bd2ee1 to your computer and use it in GitHub Desktop.
Download ZIP
Example of recipe for iteratively building a template using recursion (view graph with: snakemake -np --dag | dot | display)
Raw
snakemake_iterative_template
input_images = ['A','B','C']
shell('touch {input_images}')
max_iters = 4
#need this to make sure iterations don't go below 0!
wildcard_constraints:
iteration="[0-9]+"
rule all:
input: 'iter_{iteration}/template.nii.gz'.format(iteration=max_iters)
rule init_template:
input: input_images
output: 'iter_0/template.nii.gz'
shell:
'init_template {input} {output}'
rule reg_to_template:
input:
template = 'iter_{iteration}/template.nii.gz',
image = 'inputs/{image}.nii.gz'
output:
warp = 'iter_{iteration}/{image}_warp_{iteration}.nii.gz'
group: 'reg'
shell:
'reg_to_template {input} {output}'
rule update_template:
input:
images = lambda wildcards: expand('iter_{iteration}/{image}_warp_{iteration}.nii.gz',image=input_images,iteration=int(wildcards.iteration)-1),
template = lambda wildcards: 'iter_{iteration}/template.nii.gz'.format(iteration=int(wildcards.iteration)-1)
output:
template = 'iter_{iteration}/template.nii.gz'
shell:
'update_template {input} {output}'
@akhanf
Copy link
Author

akhanf commented Oct 2, 2020

DAG is visualized here:
dag

@akhanf
Copy link
Author

akhanf commented Oct 2, 2020

Running snakemake -np gives the expected result:

Building DAG of jobs...
Job counts:
    count    jobs
    1    all
    1    init_template
    12    reg_to_template
    4    update_template
    18


[Fri Oct  2 19:43:30 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 17


init_template A B C iter_0/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/B.nii.gz
    output: iter_0/B_warp_0.nii.gz
    jobid: 15
    wildcards: iteration=0, image=B


reg_to_template iter_0/template.nii.gz inputs/B.nii.gz iter_0/B_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/C.nii.gz
    output: iter_0/C_warp_0.nii.gz
    jobid: 16
    wildcards: iteration=0, image=C


reg_to_template iter_0/template.nii.gz inputs/C.nii.gz iter_0/C_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/A.nii.gz
    output: iter_0/A_warp_0.nii.gz
    jobid: 14
    wildcards: iteration=0, image=A


reg_to_template iter_0/template.nii.gz inputs/A.nii.gz iter_0/A_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_0/A_warp_0.nii.gz, iter_0/B_warp_0.nii.gz, iter_0/C_warp_0.nii.gz, iter_0/template.nii.gz
    output: iter_1/template.nii.gz
    jobid: 13
    wildcards: iteration=1


update_template iter_0/A_warp_0.nii.gz iter_0/B_warp_0.nii.gz iter_0/C_warp_0.nii.gz iter_0/template.nii.gz iter_1/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/A.nii.gz
    output: iter_1/A_warp_1.nii.gz
    jobid: 10
    wildcards: iteration=1, image=A


reg_to_template iter_1/template.nii.gz inputs/A.nii.gz iter_1/A_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/B.nii.gz
    output: iter_1/B_warp_1.nii.gz
    jobid: 11
    wildcards: iteration=1, image=B


reg_to_template iter_1/template.nii.gz inputs/B.nii.gz iter_1/B_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/C.nii.gz
    output: iter_1/C_warp_1.nii.gz
    jobid: 12
    wildcards: iteration=1, image=C


reg_to_template iter_1/template.nii.gz inputs/C.nii.gz iter_1/C_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_1/A_warp_1.nii.gz, iter_1/B_warp_1.nii.gz, iter_1/C_warp_1.nii.gz, iter_1/template.nii.gz
    output: iter_2/template.nii.gz
    jobid: 9
    wildcards: iteration=2


update_template iter_1/A_warp_1.nii.gz iter_1/B_warp_1.nii.gz iter_1/C_warp_1.nii.gz iter_1/template.nii.gz iter_2/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/B.nii.gz
    output: iter_2/B_warp_2.nii.gz
    jobid: 7
    wildcards: iteration=2, image=B


reg_to_template iter_2/template.nii.gz inputs/B.nii.gz iter_2/B_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/C.nii.gz
    output: iter_2/C_warp_2.nii.gz
    jobid: 8
    wildcards: iteration=2, image=C


reg_to_template iter_2/template.nii.gz inputs/C.nii.gz iter_2/C_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/A.nii.gz
    output: iter_2/A_warp_2.nii.gz
    jobid: 6
    wildcards: iteration=2, image=A


reg_to_template iter_2/template.nii.gz inputs/A.nii.gz iter_2/A_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_2/A_warp_2.nii.gz, iter_2/B_warp_2.nii.gz, iter_2/C_warp_2.nii.gz, iter_2/template.nii.gz
    output: iter_3/template.nii.gz
    jobid: 5
    wildcards: iteration=3


update_template iter_2/A_warp_2.nii.gz iter_2/B_warp_2.nii.gz iter_2/C_warp_2.nii.gz iter_2/template.nii.gz iter_3/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/A.nii.gz
    output: iter_3/A_warp_3.nii.gz
    jobid: 2
    wildcards: iteration=3, image=A


reg_to_template iter_3/template.nii.gz inputs/A.nii.gz iter_3/A_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/B.nii.gz
    output: iter_3/B_warp_3.nii.gz
    jobid: 3
    wildcards: iteration=3, image=B


reg_to_template iter_3/template.nii.gz inputs/B.nii.gz iter_3/B_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/C.nii.gz
    output: iter_3/C_warp_3.nii.gz
    jobid: 4
    wildcards: iteration=3, image=C


reg_to_template iter_3/template.nii.gz inputs/C.nii.gz iter_3/C_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_3/A_warp_3.nii.gz, iter_3/B_warp_3.nii.gz, iter_3/C_warp_3.nii.gz, iter_3/template.nii.gz
    output: iter_4/template.nii.gz
    jobid: 1
    wildcards: iteration=4


update_template iter_3/A_warp_3.nii.gz iter_3/B_warp_3.nii.gz iter_3/C_warp_3.nii.gz iter_3/template.nii.gz iter_4/template.nii.gz


[Fri Oct  2 19:43:30 2020]
localrule all:
    input: iter_4/template.nii.gz
    jobid: 0


Job counts:
    count    jobs
    1    all
    1    init_template
    12    reg_to_template
    4    update_template
    18
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

@akhanf
Copy link
Author

akhanf commented Oct 2, 2020

But using the --group-components option results in only a single job being submittted:
snakemake -np --group-components reg=3 --cluster "sbatch"

touch A B C
Building DAG of jobs...
Job counts:
	count	jobs
	1	all
	1	init_template
	12	reg_to_template
	4	update_template
	18

[Fri Oct  2 19:49:56 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 17

init_template A B C iter_0/template.nii.gz
Job counts:
	count	jobs
	1	all
	1	init_template
	12	reg_to_template
	4	update_template
	18
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

@akhanf
Copy link
Author

akhanf commented Oct 2, 2020

However, it works as expected if only a subset of the dag is used, when there are no repeated rules, e.g.:
snakemake -np --group-components reg=3 --cluster "sbatch" iter_1/template.nii.gz

touch A B C
Building DAG of jobs...
Job counts:
	count	jobs
	1	init_template
	3	reg_to_template
	1	update_template
	5

[Fri Oct  2 19:51:24 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 4

init_template A B C iter_0/template.nii.gz
[Fri Oct  2 19:51:24 2020]

group job reg (jobs in lexicogr. order):

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/C.nii.gz
        output: iter_0/C_warp_0.nii.gz
        jobid: 3
        wildcards: iteration=0, image=C

    reg_to_template iter_0/template.nii.gz inputs/C.nii.gz iter_0/C_warp_0.nii.gz

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/A.nii.gz
        output: iter_0/A_warp_0.nii.gz
        jobid: 1
        wildcards: iteration=0, image=A

    reg_to_template iter_0/template.nii.gz inputs/A.nii.gz iter_0/A_warp_0.nii.gz

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/B.nii.gz
        output: iter_0/B_warp_0.nii.gz
        jobid: 2
        wildcards: iteration=0, image=B

    reg_to_template iter_0/template.nii.gz inputs/B.nii.gz iter_0/B_warp_0.nii.gz

[Fri Oct  2 19:51:24 2020]
rule update_template:
    input: iter_0/A_warp_0.nii.gz, iter_0/B_warp_0.nii.gz, iter_0/C_warp_0.nii.gz, iter_0/template.nii.gz
    output: iter_1/template.nii.gz
    jobid: 0
    wildcards: iteration=1

update_template iter_0/A_warp_0.nii.gz iter_0/B_warp_0.nii.gz iter_0/C_warp_0.nii.gz iter_0/template.nii.gz iter_1/template.nii.gz
Job counts:
	count	jobs
	1	init_template
	3	reg_to_template
	1	update_template
	5
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment