Skip to content

Instantly share code, notes, and snippets.

View jdanbrown's full-sized avatar

Dan Brown jdanbrown

37 followers · 20 following
View GitHub Profile
@jdanbrown
jdanbrown / gist:9591970
Last active August 29, 2015 13:57

Page collation from:

History of Manu

Peru's Manu is one of the most biologically diverse places on earth. Home to over 1,000 species of birds, 300 species of trees, and countless other life forms, Manu showcases life at its most prolific. But deep within Manu's rain forest also lie stories and histories of Indians and foreign explorers of centuries past. Though their footprints have disappeared over time, these inhabitants and travelers have made deep impressions that have shaped Manu into what it is today.

Home to numerous indigenous Indian tribes, the Peruvian rain forest's most recognized Indian inhabitants were the Incas whose capital was in the Andes but whose empire extended into the cloud forest. With their large empire, the Incas had many contacts with the jungle Indians of Manu. At its height, the Inca empire spanned 3,000 miles (4,800 km) across South America. Inca territory was divided into quarters, with Cuzco, the c

@jdanbrown
jdanbrown / raid_ephemeral.sh
Created April 7, 2014 19:02 — forked from vonion/raid_ephemeral.sh
#!/bin/bash
#
# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
# mounted at /mnt. It should be run early on the first boot of the system.
#
# Beware, This script is NOT fully idempotent.
#
METADATA_URL_BASE="http://169.254.169.254/latest"
@jdanbrown
jdanbrown / raid_ephemeral.sh
Last active August 29, 2015 13:58 — forked from vonion/raid_ephemeral.sh
#!/bin/bash -eu
#
# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
# mounted at /mnt. It should be run early on the first boot of the system.
#
# Beware, This script is NOT fully idempotent.
METADATA_URL_BASE="http://169.254.169.254/latest"
DEBIAN_FRONTEND=noninteractive apt-get -y install mdadm curl
@jdanbrown
jdanbrown / gist:cef65e9082b1f32d085e
Last active August 29, 2015 14:09
Container [pid=25030,containerID=container_1414807043280_0498_01_000002] is running beyond physical memory limits. Current usage: 600.7 MB of 512 MB physical memory used; 1.9 GB of 1.0 GB virtual memory used. Killing container.
Dump of the process-tree for container_1414807043280_0498_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 25173 25103 25030 25030 (python) 8256 847 720211968 109805 python bin/run --email.trailing=PT24h
|- 25030 25028 25030 25030 (bash) 0 3 11354112 361 /bin/bash -c /usr/lib/jvm/java-7-oracle/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx456130560 -Djava.io.tmpdir=/mnt1/yarn/nm/usercache/ops/appcache/application_1414807043280_0498/container_1414807043280_0498_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1414807043280_0498/cont
@jdanbrown
jdanbrown / foo.sh
Created January 31, 2015 02:23
Scrape spark-1.2.0 /environment using pup + jq
$ for header in 'Runtime Information' 'Spark Properties' 'System Properties' 'Classpath Entries' ; do pbpaste | pup 'h4:contains("'"$header"'") > table tr json{}' | jq '{"key":"'"$header"'", "value":[.[].children | {key:.[0].text, value:.[1].text}] | .[1:] | from_entries}' -c ; done | jq --slurp '. | from_entries' --sort-keys
@jdanbrown
jdanbrown / foo.patch
Created January 31, 2015 02:37
$ json-diff <(curl -sfL https://gist.github.com/jdanbrown/027473f2fcec69919f22/raw/3dedaccd3fcf0cc7731a5deebdab4d4b62b6c681/-) <(curl -sfL https://gist.github.com/jdanbrown/7134d82810b4a8240058/raw/0c916ea3f632231b950f9ceb402c0f0339debd96/-)
{
Classpath Entries: {
- /etc/hadoop/conf: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/hadoop/../hadoop-hdfs/./: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/hadoop/../hadoop-hdfs/.//hadoop-hdfs-2.5.0-cdh5.3.0-tests.jar: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/hadoop/../hadoop-hdfs/.//hadoop-hdfs-2.5.0-cdh5.3.0.jar: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/hadoop/../hadoop-hdfs/.//hadoop-hdfs-nfs-2.5.0-cdh5.3.0.jar: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/bin/../lib/hadoop/../hadoop-hdfs/.//hadoop-hdfs-nfs.jar: "System Classpath"
- /opt/cloudera/parcels/CDH-5.3.0-1.c
@jdanbrown
jdanbrown / gist:a00552ef44859b715e7b
Created April 21, 2015 05:17
My traceroute [v0.85]
ip-192-168-0-197.ec2.internal (0.0.0.0) Mon Apr 20 22:16:59 2015
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-tiff-v227.static.monkeybrains.net 0.7% 418 93.0 135.9 34.5 265.2 40.7
2. ip-172-17-18-34.ec2.internal 2.6% 418 154.7 135.1 35.3 232.0 39.7
3. grillo.grillo-mosca-10GB.core.monkeybrains.net 5.0% 418 91.2 135.7 42.1 331.8 43.6
4. lemon.lemon-grillo.core.monkeybrains.net
@jdanbrown
jdanbrown / gist:618cbedc1204fb5e7cb9
Created May 3, 2015 23:55
$ sudo mtr gmail.com
My traceroute [v0.85]
192-195-80-250.public.monkeybrains.net (::) Sun May 3 16:55:14 2015
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-227.IPv6.monkeybrains.net 0.0% 512 35.2 29.1 8.2 86.0 11.3
2. gw-214.IPv6.monkeybrains.net 1.6% 512 29.7 29.0 11.3 92.3 10.1
3. 2607:f598:0:527::1 1.2% 511 40.1 37.0 7.3 143.0 19.7
@jdanbrown
jdanbrown / py
Created March 28, 2017 22:14
Configs for grc/grcat: https://github.com/garabik/grc
# ~/.grc/py
# Skip if not a log line
regexp=^(?!.*\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b)
colours=none
count=stop
======
# Everything before msg: <level> <timestamp> <module> <msg>
regexp=^(\S+\s+){3}
@jdanbrown
jdanbrown / bq.sql
Last active April 4, 2017 01:17
bq: unnest array vs. union select
# TODO(db): Figure out if either of these is better than the other
# - Both give "query resources exceeded" on what_is_junk with all of the homes/addresses fields
def template_unnest_array(table, fields):
df = bq.Query(with_od_homes('''
select * from unnest([
%(array_elems)s
])
order by i
''' % dict(