danking

Shuffler IR Design

UUID is probably a string

ShuffleStart(
  keyFields Array<String>,
  rowType (virtual) Struct,
  TypedCodecSpec codecAndEType
): id UUID

danking

Co-op Resume Filtering

Background

In recent semesters, Hail has received >300 applications for our one open co-op position. We engage in an uncomfortable, severe filtering process to cope with this overwhelming number of applications.

For the Spring 2020 position, Hail received 316 applications. For the upcoming Fall 2020 position, Hail has thus far received 162 applications.

danking

set -ex

cat > foo.sh <<'EOF'
#!/bin/bash
set -ex

cores=$1
script_name=$2

free -ws 5 | grep --line-buffered Mem > free.log &

danking

PRS Scoring of the UK Biobank Using Hail

The run_prs.py script slack_utils.py was created to generate polygenic risk scores (PRS) in the UK Biobank (UKB) using Hail 0.2. Currently, I have hard-coded the file locations in-line with my own file structure and that of the Neale Lab UKB Round 2 GWAS.

NOTE: This is under active development and generally intended for my own use and the use of my collaborators. ADAPT AT YOUR OWN PERIL!!

How to Run

0. This script assumes that you have generated an input file using my process-sumstats and LDPred repos.

danking

{"action": "submitted", "review": {"id": 159193269, "node_id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3MTU5MTkzMjY5", "user": {"login": "hail-ci-test-2", "id": 43152863, "node_id": "MDQ6VXNlcjQzMTUyODYz", "avatar_url": "https://avatars0.githubusercontent.com/u/43152863?v=4", "gravatar_id": "", "url": "https://api.github.com/users/hail-ci-test-2", "html_url": "https://github.com/hail-ci-test-2", "followers_url": "https://api.github.com/users/hail-ci-test-2/followers", "following_url": "https://api.github.com/users/hail-ci-test-2/following{/other_user}", "gists_url": "https://api.github.com/users/hail-ci-test-2/gists{/gist_id}", "starred_url": "https://api.github.com/users/hail-ci-test-2/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/hail-ci-test-2/subscriptions", "organizations_url": "https://api.github.com/users/hail-ci-test-2/orgs", "repos_url": "https://api.github.com/users/hail-ci-test-2/repos", "events_url": "https://api.github.com/users/hail-ci-test-2/events{/privacy}", "received_events

danking

import hail as hl
import pickle
import time
import argparse

def flip_text(base):
    """
    :param StringExpression base: Expression of a single base
    :return: StringExpression of flipped base
    :rtype: StringExpression

danking

2018-08-22 16:55:21 Hail: WARN: This Hail JAR was compiled for Spark 2.2.0, running with Spark 2.2.1.
  Compatibility is not guaranteed.
2018-08-22 16:55:21 SparkContext: INFO: Running Spark version 2.2.1
2018-08-22 16:55:21 SparkContext: INFO: Submitted application: Hail
2018-08-22 16:55:21 SparkContext: INFO: Spark configuration:
spark.app.name=Hail
spark.driver.extraClassPath=./hail-devel-62679fc21687-Spark-2.2.0.jar
spark.driver.extraJavaOptions=-Xss4M -Dlog4j.configuration=file:///tmp/819ef837c3604c95828b61032b37a28e/driver_log4j.properties
spark.driver.maxResultSize=0
spark.driver.memory=41g

danking

-- Logs begin at Wed 2018-08-08 13:03:33 UTC, end at Wed 2018-08-08 13:17:16 UTC. --
Aug 08 13:03:33 localhost kernel: Initializing cgroup subsys cpuset
Aug 08 13:03:33 localhost kernel: Initializing cgroup subsys cpu
Aug 08 13:03:33 localhost kernel: Initializing cgroup subsys cpuacct
Aug 08 13:03:33 localhost kernel: Linux version 4.4.111+ ([email protected]) (gcc version 4.9.x 20150123 (prerelease) (4.9.2_cos_gg_4.9.2-r175-0c5a656a1322e137fa4a251f2ccc6c4022918c0a_4.9.2-r175) ) #1 SMP Thu Feb 1 22:06:37 PST 2018
Aug 08 13:03:33 localhost kernel: Command line: BOOT_IMAGE=/syslinux/vmlinuz.A init=/usr/lib/systemd/systemd boot=local rootwait ro noresume noswap loglevel=7 noinitrd console=ttyS0 vsyscall=emulate security=apparmor systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=true dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 i915.modeset=1 cros_efi root=/dev/dm-0 "dm=1 vroot none ro 1,0 2539520 verity payload=PARTUUID=72C5C8D0-6721-

danking

danking