Moved to https://github.com/kbilsted/Functional-core-imperative-shell/blob/master/README.md
We have a setup that I assume is quite common: A publicly accessible Nimbus running Storm UI. The worker
nodes can only be accessed from the Nimbus (via the LAN). All the nodes have internal DNS names (i.e.
node.lan.example.com
), which is set in the configuration files; they use these DNS names to reach each
other. The Nimbus has an external DNS name (storm.example.com
) for public access. The Nimbus's UI is
behind an Nginx proxy, which provides HTTP Auth and HTTPS.
Because of this setup, the logviewer links in the UI do not work. In order to fix this, we employ an elaborate hack shown in the conf file below. It uses ngx_http_substitutions_filter_module to rewrite content returned by the Storm UI and some complicated URL rewrite tricks to proxy the workers' logviewers through through the Nimbus.
[pytest] | |
python_paths = src/bolts src/spouts | |
testpaths = src/tests |
This document describes how Airflow
jobs (or workflows) get deployed onto production system.
- HOME directory:
/home/airflow
- DAG directory:
$HOME/airflow-git-dir/dags/
- Config directory:
$HOME/airflow-git-dir/configs/
- Unittest directore:
$HOME/airflow-git-dir/tests/
. Preferable, discoverable by bothnose
andpy.test
- Credentials should be accessed by by some library
The Westminster Shorter Catechism | |
================================= | |
The original text of 1647, with the Assembly's proof texts. | |
1. What is the chief end of man? | |
Man's chief end is to glorify God,(1) and to enjoy him for ever.(2) | |
(1) I Cor. 10:31; Rom. 11:36. [ Ps 86; Is 60:21; Rev 4:11 ] |
If you were to give recommendations to your "little brother/sister" on things that they need to do to become a data scientist, what would those things be?
I think the "Data Science Venn Diagram" (http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram) is a great place to start. You need three things to be a good data scientist:
- Statistical knowledge
- Programming/hacking skills
- Domain expertise
hostkey () { | |
for HOST in "$@"; do | |
LIST=$HOST | |
PARTS=(${HOST//./ }) | |
if [[ ${#PARTS[*]} -gt 2 && "${PARTS[0]}" != "$HOST" ]]; then | |
LIST="$LIST ${PARTS[0]}" | |
fi | |
IP=$(dig +short $HOST) | |
if [ -n "$IP" ]; then | |
LIST="$LIST $IP" |
#!/bin/bash | |
virtualenv .env && | |
source .env/bin/activate && | |
brew install libevent && | |
brew install libyaml && | |
pip install cython && | |
pip install numpy && | |
pip install scipy && | |
pip install pandas && | |
pip install matplotlib && |
""" | |
Extract unique Python-Exceptions with their Traceback from a log/text file. | |
Usage:: | |
python extract_exceptions.py -f logfile.txt | |
Furthermore it supports excluding exceptions you don't want to have:: |