Ankun Yu yakjuly

Optimizing postgresql table for more than 100K inserts per second

Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
Add synchronous_commit = off to postgresql.conf.
Use table inheritance for fast removal of old data:

How to have some fun using the terminal.

How to have some fun using the terminal.

	```
	filter eventName="ConsoleLogin"
	\| stats count(*) as eventCount by userIdentity.userName, sourceIPAddress
	\| sort eventCount desc

	filter not sourceIPAddress =~ /^(?i)123.123.123.123/ and userIdentity.userName =~/^(?i)\w/
	\| stats count(*) as eventCount by eventName, userIdentity.userName, sourceIPAddress
	\| sort eventCount desc

	filter eventName="ConsoleLogin"

	$ docker run --rm -v "$(pwd)/multi-request-json.lua":/multi-request-json.lua -v "$(pwd)/requests.json":/requests.json czerasz/wrk-json wrk -c1 -t1 -d5s -s /multi-request-json.lua https://www.example.com

	multiplerequests: Found 2 requests
	multiplerequests: Found 2 requests

	Running 5s test @ https://www.example.com

	1 threads and 1 connections
	Thread Stats Avg Stdev Max +/- Stdev
	Latency 887.09ms 415.48ms 1.36s 60.00%

	{
	"receipt": {
	"type": "android-playstore",
	"id": "12345678901234567890.1234567890123456",
	"purchaseToken": "purchase token goes here",
	"receipt": "{"orderId":"12345678901234567890.1234567890123456","packageName":"com.example.app","productId":"com.example.app.product","purchaseTime":1417113074914,"purchaseState":0,"purchaseToken":"purchase token goes here"}",
	"signature": "signature data goes here"
	}
	}

	# make directory for new rails app
	mkdir app
	cd app

	# specify ruby version
	echo 2.3.1 > .ruby-version

	# initialize bundler (creates Gemfile)
	bundler init

	Vulnerabilities
	https://www.openssl.org/news/vulnerabilities.html


	https
	openssl s_client -connect www.feistyduck.com:443 -CAfile /etc/ssl/certs

	ftps
	openssl s_client -connect ftp.xxx.com:21 -starttls ftp

	# config/application.rb

	require File.expand_path('../boot', __FILE__)
	require 'rails/all'
	require File.dirname(__FILE__) + '/../lib/silent_logger.rb'

	# Require the gems listed in Gemfile, including any gems
	# you've limited to :test, :development, or :production.
	Bundler.require(:default, Rails.env)

	#use = will_paginate @users, :method => :post, :params => params.slice("q", "keyword"), :form_action => search_service_plans_path


	require 'will_paginate/view_helpers/action_view'

	module WillPaginate
	module ActionView

	def will_paginate(collection = nil, options = {})
	options, collection = collection, nil if collection.is_a? Hash