John Chilton jmchilton

History Notebooks: Implementation Plan

Overview

This plan implements History Notebooks - markdown documents tied to Galaxy histories that use HID-relative references. The feature enables human-AI collaborative analysis documentation with paths to Pages and Workflow Reports.

Reference Documents:

THE_PROBLEM_AND_GOAL.md - Vision and motivation
RESEARCH_FOR_PLANNING.md - Backend implementation research
RESEARCH_FOR_PLANNING_UX.md - Frontend/UX implementation research

IWC Workflow Integration Plan

Goal

Add link to IWC (Intergalactic Workflow Commission) curated workflows at https://iwc.galaxyproject.org/ in Galaxy's workflow list interface. Implement BOTH a tab approach and a banner approach so you can decide which works better.

Background

IWC has 100+ vetted bioinformatics workflows
Current WorkflowList has 3 tabs: My workflows, Shared with me, Public workflows
No existing IWC integration in frontend
Uses Vue 3 + Bootstrap Vue + TypeScript

OMERO File Source Plugin Review

Question

I'm review a PR. It adds this file lib/galaxy/files/sources/omero.py - which is implementing a Galaxy File Source plugins. I'm worried about the size of the file - it seems large and it interwines plugin implementation logic and API communication a lot - would it make sense to extract the API interaction layer out and reduce the size of the plugin or does it make more sense as is?

Analysis

Based on my analysis, here's my assessment:

WES Task Logs Pagination - Database-Level Implementation Plan

Problem Statement

Current implementation in wes.py:

_build_task_logs() (lines 886-913): Loads ALL WorkflowInvocationSteps into memory, then all related Jobs
Used in 3 places:
1. get_run_tasks() (line 701) - loads all, slices in Python
2. get_run_task() (line 742) - loads all to find one
3. _invocation_to_run_log() (line 868) - populates deprecated field

🎃 Truth or Scare Cocktail Menu

🍸 Squid Ink Basil Gimlet (Gin)

Ingredients: Gin, lime, basil syrup, squid ink
Description: Fresh, herbal.
Garnish: 🌿 Basil leaf

	Found 4 test cases to validate for tool shasta / 0.6.0+galaxy0 (@ /Users/jxc755/workspace/tools-iuc/tools/shasta/shasta.xml)
	Test Case 1: validated
	Test Case 2: validated
	Test Case 3: validated
	Test Case 4: validation failed - Invalid parameter name found kmers\|enrichmentThreshold
	Found 3 test cases to validate for tool bg_diamond_view / 2.0.8 (@ /Users/jxc755/workspace/tools-iuc/tools/diamond/diamond_view.xml)
	Test Case 1: validated
	Test Case 2: validated
	Test Case 3: validation failed - 1 validation error for DynamicModelForTool
	forwardonly

	Server running... press ctrl-c to stop.
	Regenerating: 1 file(s) changed at 2020-07-09 12:50:54
	topics/dev/images/core_plugins_overview.plantuml.txt
	Warning: Empty `slug` generated for ''.
	Warning: Empty `slug` generated for ''.

	[2020-07-09 12:57:05] ERROR Errno::ECONNRESET: Connection reset by peer @ io_fillbuf - fd:17
	/Users/john/miniconda3/envs/galaxy_training_material/lib/ruby/2.4.0/webrick/httpserver.rb:82:in `eof?'
	/Users/john/miniconda3/envs/galaxy_training_material/lib/ruby/2.4.0/webrick/httpserver.rb:82:in `run'
	/Users/john/miniconda3/envs/galaxy_training_material/lib/ruby/2.4.0/webrick/server.rb:308:in `block in start_thread'

	>>> import galaxy.tools.parser
	>>> import galaxy.tools.parser.factory
	>>> tool_source = galaxy.tools.parser.factory.get_tool_source("bwa.xml")
	>>> reqs, _ = tool_source.parse_requirements_and_containers()
	>>> from galaxy.tools.deps.conda_util import CondaTarget
	>>> from galaxy.tools.deps.conda_util import hash_conda_packages
	>>> hash_conda_packages([CondaTarget(req.name, req.version) for req in reqs])
	'28bd91abdde0d474ee5121c1db757e2fe25984a7b393b5288912c68b29b67e02'

	$ mulled-build --recipes-dir ../bioconda-recipes --targets 'samtools=1.3.1,bedtools=2.22' build
	./involucro -v=3 -f /home/john/workspace/galaxy-lib/galaxy/tools/deps/mulled/invfile.lua -set CHANNELS='bioconda,conda-forge,r' -set TEST='true' -set TARGETS='samtools=1.3.1,bedtools=2.22' -set REPO='quay.io/mulled/mulled-v1-a93fa17175d6f093a6423535399b7ad4ac06ceed' build
	[Sep 27 15:58:40] DEBU Run file [/home/john/workspace/galaxy-lib/galaxy/tools/deps/mulled/invfile.lua]
	[Sep 27 15:58:40] STEP Run image [continuumio/miniconda:latest] with command [[rm -rf /data/dist]]
	[Sep 27 15:58:40] DEBU Creating container [step-4ff4f7eeac]
	[Sep 27 15:58:40] DEBU Created container [eb5cdfab2b12 step-4ff4f7eeac], starting it
	[Sep 27 15:58:40] DEBU Container [eb5cdfab2b12 step-4ff4f7eeac] started, waiting for completion
	[Sep 27 15:58:40] DEBU Container [eb5cdfab2b12 step-4ff4f7eeac] completed with exit code [0] as expected
	[Sep 27 15:58:40] DEBU Container [eb5cdfab2b12 step-4ff4f7eeac] removed
	[Sep 27 15:58:40] STEP Run image [

	toolshed.g2.bx.psu.edu/repos/devteam/cummerbund_to_tabular/cummerbund_to_cuffdiff
	toolshed.g2.bx.psu.edu/repos/miller-lab/genome_diversity/gd_coverage_distributions
	toolshed.g2.bx.psu.edu/repos/miller-lab/genome_diversity/gd_population_structure