Bo-chiuan Chen bcchenbc

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
H: preserves hard-links
A: preserves ACLs

Setup

$ pip install click requests

Usage

$ python query_export.py --redash-url "https://app.redash.io/" --api-key ""

This is an unofficial manual for the couchdb Python module I wish I had had.

Installation

pip install couchdb

Importing the module

Docker - How to cleanup (unused) resources

Once in a while, you may need to cleanup resources (containers, volumes, images, networks) ...

delete volumes

// see: https://github.com/chadoe/docker-cleanup-volumes

$ docker volume rm $(docker volume ls -qf dangling=true)

$ docker volume ls -qf dangling=true | xargs -r docker volume rm

Guide on how to create and set up your Django project with webpack, npm and ReactJS :)

Hopefully this will answer "How do I setup or start a Django project?" I was trying to set up my own website, and there was a lot to read, learn and digest! Therefore, I put this together which is a collection of all the guides/blog posts/articles that I found the most useful. At the end of this, you will have your barebones Django app configured and ready to start building :)

NOTE: This guide was built using Django 1.9.5, NodeJS 4+ with NPM 3+

1. Setting up your dev environment

Running Ubuntu 16.04.1 armhf on Qemu

This is a writeup about how to install Ubuntu 16.04.1 Xenial Xerus for the 32-bit hard-float ARMv7 (armhf) architecture on a Qemu VM via Ubuntu netboot.

The setup will create a Ubuntu VM with LPAE extensions (generic-lpae) enabled. However, this writeup should also work for non-LPAE (generic) kernels.

The performance of the resulting VM is quite good, and it allows VMs with >1G ram (compared to 256M on versatilepb and 1G on versatile-a9/versatile-a15). It also supports virtio disks whereas versatile-a9/versatile-a15 only support SD cards via the -sd argument.

	wget http://stedolan.github.io/jq/download/linux64/jq
	aws ec2 describe-instances --filters "Name=tag:Name,Values=$NAME" \
	"Name=instance-state-name,Values=running" \
	\| jq -r \
	".Reservations[] \| .Instances[] \| .InstanceId" \

	aws ec2 describe-volumes --filters \
	"Name=status,Values=available" \
	\| jq -r ".Volumes[] \| .VolumeId" \

	import boto.mturk.connection

	sandbox_host = 'mechanicalturk.sandbox.amazonaws.com'
	real_host = 'mechanicalturk.amazonaws.com'

	mturk = boto.mturk.connection.MTurkConnection(
	aws_access_key_id = 'XXX',
	aws_secret_access_key = 'XXX',
	host = sandbox_host,
	debug = 1 # debug = 2 prints out all requests. but we'll just keep it at 1

	<source>
	type tail
	path /var/log/haproxy.log
	pos_file /var/log/td-agent/haproxy.log.pos
	tag haproxy.http
	format /^(?<syslog_time>.+) (?<source_ip>.+) (?<ps>\w+)\[(?<pid>\d+)\]: (?<c_ip>[\w\.]+):(?<c_port>\d+) \[(?<time>.+)\] (?<f_end>[\w\.-]+) (?<b_end>[\w\.-]+)\/(?<b_server>[^ ]+) (?<tq>[-]?\d+)\/(?<tw>[-]?\d+)\/(?<tc>[-]?\d+)\/(?<tr>[-]?\d+)\/(?<tt>\d+) (?<status_code>\d+) (?<bytes_read>\d+) (?<captured_request_cookie>.+) (?<captured_response_cookie>.+) (?<termination_state>.+) (?<actconn>\d+)\/(?<feconn>\d+)\/(?<beconn>\d+)\/(?<srv_conn>\d+)\/(?<retries>\d+) (?<srv_queue>\d+)\/(?<backend_queue>\d+) \"(?<http_request>.+)\"$/
	time_format %d/%b/%Y:%H:%M:%S.%L
	</source>

	<match .*>

	conf = {
	"sqs-access-key": "",
	"sqs-secret-key": "",
	"sqs-queue-src": "",
	"sqs-queue-dest": "",
	"sqs-region": "ap-southeast-1",
	}

	import boto.sqs
	conn = boto.sqs.connect_to_region(