Skip to content

Instantly share code, notes, and snippets.

@fomightez

fomightez/Data-dependent_conditional_execution_in_Snakemake_from_documentation.ipynb

Last active September 14, 2024 04:53
Show Gist options
  • Save fomightez/b485221c9a9298771433e179b19e52ff to your computer and use it in GitHub Desktop.
Save fomightez/b485221c9a9298771433e179b19e52ff to your computer and use it in GitHub Desktop.
Download ZIP
Data-dependent conditional execution in Snakemake
Display the source blob
Display the rendered blob
Raw
Data-dependent_conditional_execution_in_Snakemake_from_documentation.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Display the source blob
Display the rendered blob
Raw
SO_pt_2_checkpoints_stuff_working_out.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "f7ef44c3-3167-47e4-afc2-d074415310af",
"metadata": {},
"source": [
"# SO https://stackoverflow.com/q/78944571/8508004 pt 2 of checkpoints stuff\n",
"\n",
"My effort to work [MissingWildcard.ipynb](https://github.com/an-way/Sandbox/blob/main/MissingWildcard.ipynb), from https://stackoverflow.com/q/78944571/8508004 , which follows from https://stackoverflow.com/a/78893777/8508004 & https://gist.github.com/fomightez/99f9056649e1ca8128563602b2eb95f9. This effort uses the same environment in MyBinder sessions launched as described in the header of [the notebook there](https://gist.github.com/fomightez/99f9056649e1ca8128563602b2eb95f9).\n",
"\n",
"\n",
"------\n",
"OP's provided Snakefile code at [MissingWildcard.ipynb](https://github.com/an-way/Sandbox/blob/main/MissingWildcard.ipynb) at outset (storing here in case when I ask questions it starts getting edited):\n",
"\n",
"```python\n",
"import os\n",
"\n",
"OUTDIR = \"first_directory\"\n",
"samples = ['sampleA', 'sampleB', 'sampleC']\n",
"SNDDIR = \"final_dir\"\n",
"\n",
"def get_txt_files(wildcards):\n",
" ck_output = checkpoints.make_five_files.get(**wildcards).output[0]\n",
" print([file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])\n",
" return [file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]\n",
"\n",
"def not_empty_files(wildcards):\n",
" ck_output = checkpoints.make_empty_and_notEmpty_files.get(**wildcards).output[0]\n",
" print([file.split(\"_\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])\n",
" return [file.split(\"_\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]\n",
"\n",
"rule all:\n",
" input: \n",
" OUTDIR,\n",
" expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=get_txt_files),\n",
" expand(f\"{SNDDIR}/\"+\"{SMP}_{samples}.txt\", file=not_empty_files,samples=samples)\n",
"\n",
"checkpoint make_five_files:\n",
" output:\n",
" directory(OUTDIR)\n",
" params:\n",
" o = OUTDIR\n",
" shell:\n",
" \"\"\"\n",
" mkdir {output};\n",
" for D in $(seq 1 5); do\n",
" touch {params.o}/$RANDOM.txt\n",
" done\n",
" \"\"\"\n",
"\n",
"rule copy_files:\n",
" input:\n",
" \"{SMP}.txt\"\n",
" output:\n",
" out = \"{SMP}.tsv\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output.out}\n",
" \"\"\"\n",
"\n",
"rule copy2:\n",
" input:\n",
" \"{SMP}.tsv\"\n",
" output:\n",
" \"{SMP}.doc\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output}\n",
" \"\"\"\n",
"\n",
"rule merge:\n",
" input:\n",
" s1 = \"{samples}_1.txt\",\n",
" s2 = \"{samples}_2.txt\"\n",
" output:\n",
" merged = \"{samples}_merged.txt\"\n",
" shell:\n",
" r'''\n",
"\t cat {input.s1} {input.s2} > {output.merge}\n",
"\t '''\n",
"\n",
"checkpoint make_empty_and_notEmpty_files:\n",
" input:\n",
"\t f = \"{SMP}.doc\",\n",
"\t s = \"{samples}_merged.txt\" \n",
" output:\n",
"\t out = \"new/{SMP}_{samples}.txt\"\n",
" shell:\n",
" r'''\n",
" cat {input.f} {input.s} > {output}\n",
" random_num=$(( RANDOM % 2 ))\n",
"\t\tif [ $random_num -eq 1 ]; then \n",
"\t\t\techo \"not_empty\" >> {output} \n",
"\t\tfi\n",
" '''\n",
"\n",
"rule process_not_empty_file:\n",
" input:\n",
" \"new/{SMP}_{samples}.txt\"\n",
" output:\n",
" f\"{SNDDIR}/\"+\"{SMP}_{samples}.txt\"\n",
" shell:\n",
" r'''\n",
" cp {input} {output}\n",
" '''\n",
"```\n",
"\n",
"------"
]
},
{
"cell_type": "markdown",
"id": "79c8097f-7e5f-4819-9efc-053469673747",
"metadata": {},
"source": [
"Use current version of Snakemake matching OP's, which is more recent than when dealing with [previous qestion](https://stackoverflow.com/a/78893777/8508004):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2d28ba94-d826-4a5d-9017-e678c258b52f",
"metadata": {},
"outputs": [],
"source": [
"%pip install snakemake==8.18.2"
]
},
{
"cell_type": "markdown",
"id": "9bc0047e-b2ff-46c9-a28f-53b9c3d6fc07",
"metadata": {},
"source": [
"Next two cells are from OP's code at [MissingWildcard.ipynb](https://github.com/an-way/Sandbox/blob/main/MissingWildcard.ipynb) initially setting things up things by making the following files in the currrent working directory:\n",
"\n",
"```text\n",
"sampleA_1.txt\n",
"sampleB_1.txt\t\n",
"sampleC_1.txt\n",
"sampleA_2.txt\n",
"sampleB_2.txt \n",
"sampleC_2.txt\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e3bb0b03-895e-4769-9609-90fb11965dd0",
"metadata": {},
"outputs": [],
"source": [
"%%writefile create_files.sh\n",
"prefixes=(\"sampleA\" \"sampleB\" \"sampleC\")\n",
"\n",
"for prefix in \"${prefixes[@]}\"; do\n",
" touch \"${prefix}_1.txt\"\n",
" touch \"${prefix}_2.txt\"\n",
"done"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "60a58d44-4f86-4af2-9d52-e3a6d3f41508",
"metadata": {},
"outputs": [],
"source": [
"!bash create_files.sh"
]
},
{
"cell_type": "markdown",
"id": "df1d8dd4-234c-4a33-beec-41d834fcb330",
"metadata": {},
"source": [
"---------"
]
},
{
"cell_type": "markdown",
"id": "f1bd0bd7-c997-4dab-8b3d-67b485729026",
"metadata": {},
"source": [
"**Make Snakefiles** \n",
"Right now, I need two. Makes no sense to me, but cannot make work with one yet."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b1d65db2-0d39-48fe-b909-6027ba316d5d",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile\n",
"import os\n",
"OUTDIR = \"first_directory\"\n",
"samples = ['sampleA', 'sampleB', 'sampleC']\n",
"SNDDIR = \"new\"\n",
"FINALDIR = \"final_dir\"\n",
"\n",
"merged_results = [f\"{name_text}_merged.txt\" for name_text in samples]\n",
"\n",
"def get_txt_files(wildcards):\n",
" #print(f\"'checkpoints.make_five_files.get(**wildcards).output[0]'={checkpoints.make_five_files.get(**wildcards).output[0]}\")\n",
" ck_output = checkpoints.make_five_files.get(**wildcards).output[0]\n",
" #print([file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])\n",
" #print(expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=[file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])) # UNCOMMENT FOR DEBUGGING, to see what `expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=get_txt_files)` would give. I couldn't find how to do it outside since involves checkppoint, but this combination works.\n",
" return [file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]\n",
"\n",
"def not_empty_files(wildcards):\n",
" ck_output = checkpoints.make_empty_and_notEmpty_files.get(**wildcards).output[0]\n",
" print([file.split(\"_\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])\n",
" return [file.split(\"_\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]\n",
"\n",
"def list_of_fake_files(wildcards):\n",
" return [\"FakeA\",\"FakeB\",\"FakeC\"]\n",
"\n",
"rule all:\n",
" input: \n",
" OUTDIR,\n",
" expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=get_txt_files),\n",
" merged_results,\n",
" expand(f\"{SNDDIR}/\"+\"{SMP}_{samples}.txt\", samples=samples, SMP=get_txt_files),\n",
"\n",
"# ---------------Individual Rules---------------------------------------------\n",
"\n",
"# Delete any generated files so can trigger full run easily after cleaning\n",
"'''\n",
"The `touch` commands added make sure files matching each and every pattern of\n",
"output so that the `rm` commands don't throw an error.\n",
"Run with `snakemake -c 8 clean`.\n",
"'''\n",
"rule clean:\n",
" params:\n",
" od = OUTDIR,\n",
" sd = SNDDIR,\n",
" rd = \"record_of_files_sorted\",\n",
" empd = \"empty_files\",\n",
" nempd = \"not_empty_files\",\n",
" shell:\n",
" '''\n",
" touch sampleD_merged.txt\n",
" rm sample*_merged.txt\n",
" rm -rf {params.od}\n",
" rm -rf {params.sd}\n",
" rm -rf {params.rd}\n",
" rm -rf {params.empd}\n",
" rm -rf {params.nempd}\n",
" '''\n",
"\n",
"checkpoint make_five_files:\n",
" output:\n",
" directory(OUTDIR)\n",
" params:\n",
" o = OUTDIR\n",
" shell:\n",
" \"\"\"\n",
" mkdir {output};\n",
" for D in $(seq 1 5); do\n",
" touch {params.o}/$RANDOM.txt\n",
" done\n",
" \"\"\"\n",
"\n",
"rule copy_files:\n",
" input:\n",
" \"{SMP}.txt\"\n",
" output:\n",
" out = \"{SMP}.tsv\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output.out}\n",
" \"\"\"\n",
"\n",
"rule copy2:\n",
" input:\n",
" \"{SMP}.tsv\"\n",
" output:\n",
" \"{SMP}.doc\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output}\n",
" \"\"\"\n",
" \n",
"rule merge:\n",
" input:\n",
" s1 = \"{samples}_1.txt\",\n",
" s2 = \"{samples}_2.txt\"\n",
" output:\n",
" merged = \"{samples}_merged.txt\"\n",
" shell:\n",
" r'''\n",
" cat {input.s1} {input.s2} > {output.merged}\n",
" '''\n",
"\n",
"rule make_empty_and_notEmpty_files:\n",
" input:\n",
" f = os.path.join(OUTDIR, \"{SMP}.doc\"),\n",
" s = \"{samples}_merged.txt\" \n",
" output:\n",
" \"new/{SMP}_{samples}.txt\"\n",
" shell:\n",
" r'''\n",
" mkdir new -p;\n",
" cat {input.f} {input.s} > {output}\n",
" random_num=$(( RANDOM % 2 ))\n",
" if [ $random_num -eq 1 ]; then \n",
" echo \"not_empty\" >> {output} \n",
" fi\n",
" '''\n",
"\n",
"rule process_not_empty_file:\n",
" input:\n",
" \"new/{SMP}_{samples}.txt\"\n",
" params:\n",
" fd = FINALDIR\n",
" output:\n",
" os.path.join(FINALDIR, \"{SMP}_{samples}.txt\")\n",
" shell:\n",
" r'''\n",
" mkdir {params.fd} -p;\n",
" cp {input} {output}\n",
" '''"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "44dd308d-0dbc-47cb-84a1-086b8b92e574",
"metadata": {},
"outputs": [],
"source": [
"%%writefile Snakefile2\n",
"import os\n",
"OUTDIR = \"first_directory\"\n",
"samples = ['sampleA', 'sampleB', 'sampleC']\n",
"SNDDIR = \"new\"\n",
"FINALDIR = \"final_dir\"\n",
"\n",
"merged_results = [f\"{name_text}_merged.txt\" for name_text in samples]\n",
"\n",
"##----------------HELPER FUNCTIONS----------------------------------------------\n",
"def get_txt_file_prefixes(wildcards):\n",
" #print(f\"'checkpoints.make_five_files.get(**wildcards).output[0]'={checkpoints.make_five_files.get(**wildcards).output[0]}\") # UNCOMMENT FOR DEBUGGING\n",
" ck_output = checkpoints.make_five_files.get(**wildcards).output[0]\n",
" #print([file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]) # UNCOMMENT FOR DEBUGGING\n",
" #print(expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=[file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')])) # UNCOMMENT FOR DEBUGGING, to see what `expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=get_txt_file_prefixes)` would give. I couldn't find how to do it outside since involves checkppoint, but this combination works.\n",
" return [file.split(\".\")[0] for file in os.listdir(ck_output) if file.endswith('.txt')]\n",
"\n",
"\n",
"\n",
"def list_of_fake_files(wildcards):\n",
" return [\"FakeA\",\"FakeB\",\"FakeC\"]\n",
"##-----------END OF HELPER FUNCTIONS--------------------------------------------\n",
"\n",
"# SNAKEMAKE RULES---------------------------------------------------------------\n",
"rule all:\n",
" input: \n",
" OUTDIR,\n",
" expand(f\"{OUTDIR}/\"+\"{SMP}.doc\", SMP=get_txt_file_prefixes),\n",
" merged_results,\n",
" expand(f\"{SNDDIR}/\"+\"{SMP}_{samples}.txt\", samples=samples, SMP=get_txt_file_prefixes),\n",
" #expand(f\"{FINALDIR}/\"+\"{SMP}_{samples}.txt\", not_empty_fns=not_empty_file_names),\n",
" expand(f\"record_of_files_sorted/\"+\"{SMP}_{samples}.txt\", samples=samples, SMP=get_txt_file_prefixes),\n",
"\n",
"# ---------------Individual Rules----------------------------------------------\n",
"\n",
"# Delete any generated files so can trigger full run easily after cleaning\n",
"'''\n",
"The `touch` commands added make sure files matching each and every pattern of\n",
"output so that the `rm` commands don't throw an error.\n",
"Run with `snakemake -c 8 clean`.\n",
"'''\n",
"rule clean:\n",
" params:\n",
" od = OUTDIR,\n",
" sd = SNDDIR,\n",
" rd = \"record_of_files_sorted\",\n",
" empd = \"empty_files\",\n",
" nempd = \"not_empty_files\",\n",
" shell:\n",
" '''\n",
" touch sampleD_merged.txt\n",
" rm sample*_merged.txt\n",
" rm -rf {params.od}\n",
" rm -rf {params.sd}\n",
" rm -rf {params.rd}\n",
" rm -rf {params.empd}\n",
" rm -rf {params.nempd}\n",
" '''\n",
"\n",
"checkpoint make_five_files:\n",
" output:\n",
" directory(OUTDIR)\n",
" params:\n",
" o = OUTDIR\n",
" shell:\n",
" \"\"\"\n",
" mkdir {output};\n",
" for D in $(seq 1 5); do\n",
" touch {params.o}/$RANDOM.txt\n",
" done\n",
" \"\"\"\n",
"\n",
"rule copy_files:\n",
" input:\n",
" \"{SMP}.txt\"\n",
" output:\n",
" out = \"{SMP}.tsv\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output.out}\n",
" \"\"\"\n",
"\n",
"rule copy2:\n",
" input:\n",
" \"{SMP}.tsv\"\n",
" output:\n",
" \"{SMP}.doc\"\n",
" shell:\n",
" \"\"\"\n",
" touch {output}\n",
" \"\"\"\n",
"\n",
"rule merge:\n",
" input:\n",
" s1 = \"{samples}_1.txt\",\n",
" s2 = \"{samples}_2.txt\"\n",
" output:\n",
" merged = \"{samples}_merged.txt\"\n",
" shell:\n",
" r'''\n",
" cat {input.s1} {input.s2} > {output.merged}\n",
" '''\n",
"\n",
"rule handle_not_empty:\n",
" input:\n",
" \"new/{SMP}_{samples}.txt\"\n",
" output:\n",
" \"not_empty_files/{SMP}_{samples}.txt\"\n",
" shell:\n",
" \"cp {input} {output}\"\n",
"\n",
"\n",
"rule handle_empty:\n",
" input:\n",
" \"new/{SMP}_{samples}.txt\"\n",
" output:\n",
" \"empty_files/{SMP}_{samples}.txt\"\n",
" shell:\n",
" \"cp {input} {output}\"\n",
"\n",
"\n",
"checkpoint make_empty_and_notEmpty_files:\n",
" input:\n",
" f = os.path.join(OUTDIR, \"{SMP}.doc\"),\n",
" s = \"{samples}_merged.txt\" \n",
" output:\n",
" f\"{SNDDIR}/{{SMP}}_{{samples}}.txt\"\n",
" shell:\n",
" r'''\n",
" mkdir new -p;\n",
" cat {input.f} {input.s} > {output}\n",
" random_num=$(( RANDOM % 2 ))\n",
" if [ $random_num -eq 1 ]; then \n",
" echo \"not_empty\" >> {output} \n",
" fi\n",
" '''\n",
"# input function for the rule sort_and_make_record\n",
"def sort_whether_empty_input(wildcards):\n",
" # decision based on content of output file\n",
" # Important: use the method open() of the returned file!\n",
" # This way, Snakemake is able to automatically download the file if it is generated in\n",
" # a cloud environment without a shared filesystem.\n",
" with checkpoints.make_empty_and_notEmpty_files.get(**wildcards).output[0].open() as f:\n",
" #print(checkpoints.make_empty_and_notEmpty_files.get(**wildcards).output[0])\n",
" #print(type(checkpoints.make_empty_and_notEmpty_files.get(**wildcards).output[0]))\n",
" if f.read().strip() == \"not_empty\":\n",
" return \"not_empty_files/{SMP}_{samples}.txt\"\n",
" else:\n",
" return \"empty_files/{SMP}_{samples}.txt\"\n",
"\n",
"#for record make a file where know for sure what final final is and set that as\n",
"# target in `rule all`. In the documentation example for data-dependent\n",
"# conditional execution, they had files they new from start to finish and so\n",
"# could target those. By setting a file I know as target I can create a flag \n",
"# file and hopefully I'll avoid errors about `{SMP}` & `{samples}`.\n",
"rule sort_and_make_record:\n",
" input:\n",
" sort_whether_empty_input\n",
" output:\n",
" \"record_of_files_sorted/{SMP}_{samples}.txt\"\n",
" shell:\n",
" \"cp {input} {output}\""
]
},
{
"cell_type": "markdown",
"id": "e125f8d9-7fec-419e-b08a-e440fb050e25",
"metadata": {},
"source": [
"-------------\n",
"\n",
"After things set up....\n",
"\n",
"#### Run the file `Snakefile`, which contains my edited version of OP's pipeline:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c7d411bc-0329-4ea8-acc8-cef66d5270a7",
"metadata": {},
"outputs": [],
"source": [
"!snakemake -c 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdd07ab8-7614-4484-adc9-5d8d588d44ff",
"metadata": {},
"outputs": [],
"source": [
"!snakemake -s Snakefile2 -c 1"
]
},
{
"cell_type": "markdown",
"id": "4f447b9a-21db-4aa9-96d6-846f1af70230",
"metadata": {},
"source": [
"Why do the two files work? But I cannot combine into one run? \n",
"Makes no sense to me."
]
},
{
"cell_type": "markdown",
"id": "29a67633-331a-484d-8efd-70619df4fcdc",
"metadata": {},
"source": [
"------------\n",
"\n",
"Uncomment line below and run it to reset things during development."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1a91953d-25a1-43d7-8da3-865729935e1c",
"metadata": {},
"outputs": [],
"source": [
"!snakemake -c 1 clean"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d1f30e81-adb9-4704-8fb0-53ab665fcdf2",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment