Skip to content

Instantly share code, notes, and snippets.

View iamh2o's full-sized avatar
💭
autumnal

(major) john (major) iamh2o

💭
autumnal
View GitHub Profile
@sean-smith
sean-smith / OneAPI.md
Created September 3, 2020 21:32
Steps to install the Intel Compiler (ICC) using OneAPI

Intel oneAPI for AWS ParallelCluster

First setup yum to get the right repo:

tee > /tmp/oneAPI.repo << EOF
[oneAPI]
name=Intel(R) oneAPI repository
baseurl=https://yum.repos.intel.com/oneapi
enabled=1
@illusional
illusional / cromwell-slurm-singularity.conf
Last active August 20, 2024 02:28
Cromwell configuration for Slurm + Singularity
# Configure Cromwell to submit jobs to Slurm, including support for Singularity
#
# Author:
# - Michael Franklin <[email protected]>
#
# History:
# - 2020-07-23 - Initial + minor fixes to generalise the script
# - 2024-04-11 - Use bash instead of sh (based on @oneillkza suggestion)
#
# Quickstart:
@GuillaumeDerval
GuillaumeDerval / log_handler_slack.py
Last active June 5, 2021 10:01
A --log-handler-script log script for snakemake that pushes the status to Slack
# License: MIT
import socket
from slacker import Slacker
import datetime
import time
import threading
SLACK_TOKEN = 'xoxb-00000000000-000000000000-000000000000000000000000'
CHANNEL_NAME = "your-channel-on-slack"
@mauler
mauler / http_server_auth.py
Last active May 9, 2025 14:30 — forked from fxsjy/SimpleAuthServer.py
Python3 http.server supporting basic HTTP Auth (username/password)
# Extended python -m http.serve with --username and --password parameters for
# basic auth, based on https://gist.github.com/fxsjy/5465353
from functools import partial
from http.server import SimpleHTTPRequestHandler, test
import base64
import os
class AuthHTTPRequestHandler(SimpleHTTPRequestHandler):
#!/bin/bash
set -ex
gcc_ver=$(gcc --version | head -n 1 | awk '{print $3}')
# Install gcc 4.9 devtoolset
if [[ ! "$gcc_ver" =~ 4\.9\.* ]]; then
sudo yum install epel-release -y
sudo yum install centos-release-scl -y
@adam-garcia
adam-garcia / README.md
Last active April 23, 2024 09:41
Viridis Color Palette Generator

Viridis Color Palette Generator

A simple tool for extracting discrete color palette values from the viridis collection of color palettes.

@indraniel
indraniel / application.conf
Last active August 19, 2024 12:48
Example Cromwell / WDL usage at the MGI
webservice {
port = 8000
interface = 0.0.0.0
instance.name = "reference"
}
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
actor {
default-dispatcher {
@fsteffenhagen
fsteffenhagen / sum_filesize.sh
Last active February 13, 2025 11:01
sum human readable file sizes with numfmt and awk
# Input: list of rows with format: "<filesize> filename", e.g.
# filesizes.txt
#######################
# 1000K file1.txt
# 200M file2.txt
# 2G file3.txt
#
# Output:
cat filesizes.txt | numfmt --from=iec | awk 'BEGIN {sum=0} {sum=sum+$1} END {printf "%.0f\n", sum}'
@Brainiarc7
Brainiarc7 / transient-clustering-gnu-parallel-sshfs.md
Last active September 2, 2024 13:53
How to set up a transient cluster using GNU parallel and SSHFS for distributed jobs (such as FFmpeg media encodes)

Transient compute clustering with GNU Parallel and sshfs:

GNU Parallel is a multipurpose program for running shell commands in parallel, which can often be used to replace shell script loops,find -exec, and find | xargs. It provides the --sshlogin and --sshloginfile options to farm out jobs to multiple hosts, as well as options for sending and retrieving static resources and and per-job input and output files.

For any particular task, however, keeping track of which files need to pushed to and retrieved from the remote hosts is somewhat of a hassle. Furthermore, cancelled or failed runs can leave garbage on the remote hosts, and if input and output files are large, sending them to local disk on the remote hosts is somewhat inefficient.

In a traditional cluster, this problem would be solved by giving all nodes access to a shared filesystem, usually with NFS or something more exotic. However, NFS doesn't wo

@SamStudio8
SamStudio8 / acct_parse.py
Last active November 28, 2023 16:12
SGE QACCT Parsing
"""
Provides functions for the purpose of parsing a Sun Grid Engine (SGE)
accounting file for job metadata.
"""
__author__ = "Sam Nicholls <[email protected]>"
__copyright__ = "Copyright (c) Sam Nicholls"
__version__ = "0.0.32"
__maintainer__ = "Sam Nicholls <[email protected]>"