Duy Mai zduymz

Transforming Code into Beautiful, Idiomatic Python

Notes from Raymond Hettinger's talk at pycon US 2013 video, slides.

The code examples and direct quotes are all from Raymond's talk. I've reproduced them here for my own edification and the hopes that others will find them as handy as I have!

Looping over a range of numbers

for i in [0, 1, 2, 3, 4, 5]:

source: openkb.info

1. Oozie Launcher Job architecture

Oozie Launcher job is a map-only job which will start the jobs which does the real work: eg, Hive, MR, Pig, etc.

(Oozie Launcher MR job [AM/Mapper Container(Hive CLI)])
. -> (MR job-1 spawned by Hive query(stage0) [AM/Mapper/Reducer Containter])
. -> (MR job-2 spawned by Hive query(stage1) [AM/Mapper/Reducer Containter])

2. How to increase the YARN container size for AM or Mapper of Oozie Hive job?

It is controlled by below 4 parameters set in workflow.xml for each Oozie job.

From the Linux kernel documentation:

max_map_count:

This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.
While most applications need less than a thousand maps, certain programs, particularly malloc debuggers, may consume lots of them, e.g., up to one or two maps per allocation.
The default value is 65536.

Bottom line: this setting limits the number of discrete mapped memory areas - on its own it imposes no limit on the size of those areas or on the memory that is usable by a process.

Q: What’s the difference between an accessor method and an instance method?

A: “Accessor method” is just a way of describing one particular kind of instance method, one whose primary purpose is to get or set the value of an instance variable. In all other respects, accessor methods are ordinary instance methods.

Q: I set up an instance variable outside an instance method, but it’s not there when I try to access it. Why? class Widget @size = 'large' def show_size puts "Size: #{@size}" end

$ kitchen init --driver=kitchen-vagrant

driver: This is where we configure the behaviour of the Kitchen Driver - the component that is responsible for creating a machine that we'll use to test our cookbook. Here we set up basics like credentials, ssh usernames, sudo requirements, etc. Each Driver is responsible for requiring and using the configuration here. Under this section we have driver.name: This tells Test Kitchen that we want to use the kitchen-vagrant driver by default unless otherwise specified.
provisioner: This tells Test Kitchen how to run Chef, to apply the code in our cookbook to the machine under test. The default and simplest approach is to use chef-solo, but other options are available, and ultimately Test Kitchen doesn't care how the infrastructure is built - it could theoretically be with Puppet, Ansible, or Perl for all it cares.
platforms: This is a list of operation systems on which we want to run our code. Note that the operating system's version, architecture, cloud

	1. tap repo: brew tap anakros/homebrew-tsocks
	2. install tsocks: brew install --HEAD tsocks
	3. set up socks proxy: ssh -D 5555 server
	4. edit configuration file: vi /usr/local/etc/tsocks.conf
	```
	#local = 192.168.0.0/255.255.255.0
	server = 127.0.0.1
	server_type = 5
	server_port = 8181
	```

	yum install gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison iconv-deve
	curl -s -L get.rvm.io \| bash -s stable
	source /etc/profile.d/rvm.sh
	rvm install 2.3.1
	rvm use 2.3.1 –default

	Chef Solo or Chef Server?

	Chef comes in two flavors: Chef Solo and Chef Server. Chef Solo is a basically a simpler version of Chef Server because it’s designed to be used with a small number of servers. With Chef Solo, you write your recipes on your local computer, upload them to your server(s), and Chef Solo is then called to run them. (A recipe, by the way, is a file containing the commands that will run to provision your server.)

	With Chef Server, you still write your recipes on your local computer, but instead of uploading them to the server you want provisioned, you upload them to a server that’s specifically dedicated to Chef. This server acts as the main repository of all your recipes. The servers you want provisioned will then have a program running on them (referred to as a Chef client) that is in constant communication with your Chef server, and whenever you upload your recipes to Chef server, Chef client will notice this and run them automatically. (Chef Solo is also Chef client; it just doesn’t

	Docker, previously called dotCloud and open-sourced in 2013, is a Linux-only virtual environment (VE) tool, not a VM tool. It builds on LxC (LinuX Containers), which uses the cgroups functionality to enable creation and running of multiple isolated Linux virtual environments (VE) on a single control host. So unlike a VM, a VE like Docker doesn’t create its own virtual computer with a distinct OS and processors and hardware emulation. A VE is VM-lite; it rides on the already existing kernel’s image of the underlying hardware, and only creates a ‘container’ in which to run your apps, and even recreate the OS if you want since the OS is also just another app running on the kernel. It places only a little extra load on the system, so unlike the traditional VM there is very little overhead when using Docker. Because of the shared kernel, Docker’s isolation is not as good as a full VM’s, but it suits many people just fine.

	Vagrant, an open-source product released in 2010, is best described as a VM manager. It allo