cr3a7ure

Submit a Spark job to an existing Amazon EMR cluster

Creates a step in Amazon EMR for a given cluster_id and monitors it's progress using a sensor. A more complex example, that involves cluster creation/termination can be found here.

cr3a7ure

"""

Script to print DAG+Task information in a text format. This can be used to quickly compare against other branches
or versions to confirm the "compiled" version of a DAG is matching expectations.

Usage:
  1. ~/airflow-install (branch) $ ./worker/run --no-name python3 /opt/airflow/utils/dag_dumper.py -l dag1 dag2 -o /opt/airflow/dags/devenv_config/dumps/dag_dump_branch
  2. ~/airflow-install (branch) $ git checkout development
  3. ~/airflow-install (development) $ ./worker/run --no-name python3 /opt/airflow/utils/dag_dumper.py -l dag1 dag2 -o /opt/airflow/dags/devenv_config/dumps/dag_dump_dev
  4. Run comparison against the 2 output files

cr3a7ure

from datetime import timedelta, datetime
import airflow
from airflow import DAG
from airflow.models import Variable
from airflow.contrib.operators.ssh_operator import SSHOperator
from airflow.contrib.hooks.ssh_hook import SSHHook
from airflow.operators.postgres_operator import PostgresOperator

default_args = {
    'owner': 'airflow',

cr3a7ure

# -*- coding: utf-8 -*-
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.postgres_operator import PostgresOperator
from airflow.operators.dummy_operator import DummyOperator

from airflow.hooks.postgres_hook import PostgresHook

from airflow.models import Variable, DAG

from datetime import date, datetime, timedelta

cr3a7ure

#/usr/bin/python3
# -*- coding: utf-8 -*-
 
import logging
import airflow
from airflow import DAG
from datetime import timedelta, datetime
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.operators.http_operator import SimpleHttpOperator

cr3a7ure

# -*- coding: utf-8 -*-
import airflow
from airflow.executors.celery_executor import CeleryExecutor
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.subdag_operator import SubDagOperator

def get_id_list():
    """ idのリストを返す. 例のためとりあえず簡単に0〜99. """

cr3a7ure

## Consumer Throughput: Single consumer thread, no compression
## Consumer Throughput: 3 consumer thread, no compression

bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1

cr3a7ure

Benchmark commands

Producer

Setup

bin/kafka-topics.sh \
  --zookeeper zookeeper.example.com:2181 \
  --create \

cr3a7ure

	  ---			-----------------			----
	  Map			Quick Explanation			Link
	  ---			-----------------			----
<	 <F1>	Causes Netrw to issue help
	 <cr>	Netrw will enter the directory or read the file      |netrw-cr|
	 <del>	Netrw will attempt to remove the file/directory      |netrw-del|
	 <c-h>	Edit file hiding list                                |netrw-ctrl-h|
	 <c-l>	Causes Netrw to refresh the directory listing        |netrw-ctrl-l|
	 <c-r>	Browse using a gvim server                           |netrw-ctrl-r|
	 <c-tab> Shrink/expand a netrw/explore window                |netrw-c-tab|

cr3a7ure

#Unix Toolbox

This document is a collection of Unix/Linux/BSD commands and tasks which are useful for IT work or for advanced users. This is a practical guide with concise explanations, however the reader is supposed to know what s/he is doing.

##Unix Toolbox revision 14.4

The latest version of this document can be found at http://cb.vu/unixtoolbox.xhtml. Replace .xhtml on the link with .pdf for the PDF version and with .book.pdf for the booklet version. On a duplex printer the booklet will create a small book ready to bind. This XHTML page can be converted into a nice PDF document with a CSS3 compliant application (see the script example). See also the about page.
Error reports and comments are m