This script was prepared for development work at the Galaxy Community Conference 2021 CoFest. It has been superseded by a Docker Compose-based project at:
Nate Coraor natefoo
natefoo
/ findbad.py
Created
September 8, 2021 16:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| """ | |
| Finds datasets on disk that shouldn't be there. | |
| This script is not object-store aware. You just point it at directories that are DiskObjectStore backends (e.g. | |
| file_path in the Galaxy config by default). | |
| """ | |
| from __future__ import absolute_import, print_function | |
| import sys |
natefoo
/ 00-run-pulsar-dev.md
Last active
July 21, 2021 02:53
natefoo
/ lamefix.py
Last active
June 22, 2021 16:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python3 | |
| # run with: | |
| # find topics -name \*.md -type f -print0 | xargs -0 python3 lamefix.py | |
| import fileinput | |
| import sys | |
| def fix(mdfileobj): | |
| in_diff = False | |
| min_length = sys.maxsize |
natefoo
/ inputs-to-mem.sql
Created
April 16, 2021 14:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SELECT | |
| subq.*, | |
| (subq.total_size_mb / subq.input_count)::INT8 AS average_size_mb, | |
| (subq.mem_used_mb / subq.total_size_mb)::INT8 AS mem_inputs_ratio, | |
| (subq.mem_used_mb / greatest(subq.total_size_mb / subq.input_count, 1))::INT8 AS mem_avg_input_ratio | |
| FROM | |
| ( | |
| SELECT | |
| subq2.* | |
| FROM |
natefoo
/ windowed_job_readiness_query.sql
Last active
February 12, 2021 00:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SELECT | |
| rank_filter.* | |
| FROM | |
| ( | |
| SELECT | |
| job.id AS job_id, | |
| job.create_time AS job_create_time, | |
| job.update_time AS job_update_time, | |
| job.history_id AS job_history_id, | |
| job.library_folder_id AS job_library_folder_id, |
natefoo
/ jbrowse.pytb
Created
February 11, 2021 15:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| RefactoringTool DEBUG 2021-02-10 22:58:00,404 Refactoring auto_translate_cheetah | |
| galaxy.jobs.runners ERROR 2021-02-10 22:58:01,444 (33368295) Failure preparing job | |
| Traceback (most recent call last): | |
| File "/cvmfs/main.galaxyproject.org/galaxy/lib/galaxy/util/__init__.py", line 1039, in unicodify | |
| value = str(value) | |
| File "/cvmfs/main.galaxyproject.org/venv/lib/python3.6/site-packages/Cheetah/Template.py", line 1053, in __unicode__ | |
| return getattr(self, mainMethName)() | |
| File "cheetah_DynamicallyCompiledCheetahTemplate_1613019481_1763794_97887.py", line 157, in respond | |
| TypeError: 'NoneType' object is not callable |
natefoo
/ run_tag_test_import.sh-session
Last active
January 15, 2021 19:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (ansible)natefoo@fanboy% ansible-playbook -i localhost, tag_test_import.yml | |
| PLAY [localhost] ************************************************************************************************************************************************************************************************************************************ | |
| TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************************************** | |
| [WARNING]: Platform darwin on host localhost is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change the meaning of that path. See | |
| https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information. | |
| ok: [localhost] | |
| TASK [debug] **************************************************************************************************************** |
natefoo
/ twobit.log
Created
December 14, 2020 22:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [g2main@galaxy-web-05 ~]$ grep twobit /srv/galaxy/main/log/zergling0.log | |
| galaxy.tools.data DEBUG 2020-12-09 19:57:30,617 [p:29736,w:0,m:0] [MainThread] Loaded 19 lines from '/cvmfs/data.galaxyproject.org/byhand/location/twobit.loc' for 'twobit' | |
| galaxy.tools.data DEBUG 2020-12-09 19:57:30,617 [p:29736,w:0,m:0] [MainThread] Loaded tool data table 'twobit' from file '/srv/galaxy/main/config/tool_data_table_conf.xml' | |
| galaxy.tools.data DEBUG 2020-12-09 19:57:30,713 [p:29736,w:0,m:0] [MainThread] Loaded 21 lines from '/cvmfs/data.galaxyproject.org/managed/location/twobit.loc' for 'twobit' | |
| galaxy.tools.data DEBUG 2020-12-09 19:57:30,713 [p:29736,w:0,m:0] [MainThread] Loading another instance of data table 'twobit' from file '/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml', attempting to merge content. | |
| galaxy.tools.data DEBUG 2020-12-09 19:57:31,891 [p:29736,w:0,m:0] [MainThread] Loaded 0 lines from '/galaxy-repl/main/tool_data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_twobit_builder/ |
Turns out it's probably Chromium crashing, as determined from running the command line (obtained from ps(1) after clicking to generate an image) and running it by hand:
-bash-4.2$ /var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=TranslateUI,BlinkGenPropertyTrees --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --headless --hide-scrollbars --mute-audio about:blank --no-sandbox --remote-debu
natefoo
/ Multi-host Galaxy Server.md
Last active
November 9, 2020 20:37
usegalaxy.org uses a five host setup:
- 2 x hosts for serving web requests
- 2 x hosts for handling jobs and Pulsar staging
- 1 x host for the database
Web requests are balanced only with a DNS round-robin (i.e. there are two A records for usegalaxy.org), which is not ideal since it does not do any load balancing or health checks. A better setup would include a 6th host with nginx proxying the web hosts.