Dan Fairs danfairs

Elasticsearch configuration for high sustainable bulk feed

Test on single node, MacBook Pro, 16 GB RAM, 1TB SSD, OS X Maverick

ES 1.1.0 with Java 8, G1 GC, 12 GB heap

/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java -Xms12g -Xmx12g -Djava.awt.headless=true -XX:+UseG1GC -Delasticsearch -Des.foreground=yes -Des.path.home=/Users/es/elasticsearch-1.1.0 -cp :/Users/es/elasticsearch-1.1.0/lib/elasticsearch-1.1.0.jar:/Users/es/elasticsearch-1.1.0/lib/:/Users/es/elasticsearch-1.1.0/lib/sigar/ org.elasticsearch.bootstrap.Elasticsearch

Node

Keybase proof

I hereby claim:

I am danfairs on github.
I am danfairs (https://keybase.io/danfairs) on keybase.
I have a public key ASC0Yu-wF_zELLNx9mkEVloWIzjxa1_MxItY6qRbszjfzQo

To claim this, I am signing this object:

	locust$ vagrant up
	Bringing machine 'default' up with 'virtualbox' provider...
	[default] Box 'precise64' was not found. Fetching box from specified URL for
	the provider 'virtualbox'. Note that if the URL does not have
	a box for this provider, you should interrupt Vagrant now and add
	the box yourself. Otherwise Vagrant will attempt to download the
	full box prior to discovering this error.
	Downloading or copying the box...
	Extracting box...te: 910k/s, Estimated time remaining: 0:00:02))
	Successfully added box 'precise64' with provider 'virtualbox'!

	(fabtools)skater:fabtools danfairs$ FABTOOLS_TEST_BOXES='precise64 opscode-ubuntu-13.04' tox -epy27
	GLOB sdist-make: /Users/danfairs/Envs/fabtools/src/fabtools/setup.py
	py27 inst-nodeps: /Users/danfairs/Envs/fabtools/src/fabtools/.tox/dist/fabtools-0.18.0-dev.zip
	py27 runtests: PYTHONHASHSEED='559934745'
	py27 runtests: commands[0] \| /Users/danfairs/Envs/fabtools/src/fabtools/.tox/py27/bin/python -m unittest discover
	................................[localhost] local: rm -f Vagrantfile
	[localhost] local: vagrant init precise64
	A `Vagrantfile` has been placed in this directory. You are now
	ready to `vagrant up` your first virtual environment! Please read
	the comments in the Vagrantfile as well as documentation on

	$ gcloud compute disks list
	NAME ZONE SIZE_GB TYPE STATUS
	disk100-001 europe-west1-d 100 pd-standard READY
	gke-unrest-micro-pool-199acc6c-3p31 europe-west1-d 100 pd-standard READY
	gke-unrest-micro-pool-199acc6c-4q55 europe-west1-d 100 pd-standard READY

	$ kubectl get pv
	NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
	pv-disk100-001 100Gi RWO Bound default/graphite-statsd-claim 2m

	FROM ubuntu:16.04
	MAINTAINER Dan Fairs <[email protected]>
	RUN apt-get update && apt-get install -y python3-pip
	RUN mkdir /app
	COPY main.py /app
	COPY requirements.txt /app
	RUN pip3 install --upgrade pip
	RUN pip3 install -r /app/requirements.txt
	EXPOSE 8888
	CMD python3 /app/main.py

	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	labels: {name: twist-server, role: app}
	name: twist-server
	spec:
	replicas: 1
	selector:
	matchLabels:
	name: twist-server

	pgbouncer 3585 postgres 27u IPv4 12329827 0t0 TCP 127.0.0.1:43238->127.0.0.1:postgresql (ESTABLISHED)
	postgres 12363 postgres 9u IPv4 12335304 0t0 TCP 127.0.0.1:postgresql->127.0.0.1:43238 (ESTABLISHED)

	// Here's the simple version - first we have a conversion function, that takes a ContentItem and converts to a contentpb.ContentItem.
	func ContentItemToPb(c ContentItem) (contentpb.ContentItem, error) {
	if c == nil {
	return nil, errors.New("ContentItemToPb: ContentItem was nil")
	}
	return &contentpb.ContentItem{
	Id: &contentpb.ContentId{
	SourceId: c.Id.SourceId,
	SourceName: c.Id.SourceName,
	},

	Here's my email address:

	[email protected]