Patrick Humpal phumpal

Make Ubuntu packages 90% faster by rebuilding them

TL;DR

You can take the same source code package that Ubuntu uses to build jq, compile it again, and realize 90% better performance.

Setting

I use jq for processing GeoJSON files and other open data offered in JSON format. Today I am working with a 500MB GeoJSON file that contains the Alameda County Assessor's parcel map. I want to run a query that prints the city for every parcel worth more than a threshold amount. The program is

I was able to get Ruby 2.5.1 building on M1 by following two of the steps in that GH issue...

rbenv/ruby-build#1691 (comment) rbenv/ruby-build#1691 (comment)

I installed readline and openssl with standard brew install.

brew install readline
brew install openssl

ElasticSearch 5.6 Cheatsheet

Node Decomission

If you need to decomission a node the first thing you should do is transfer all shards from it to other nodes

Start Moving all Shards off a node

This command will tell ElasticSearch to:

Stop sending new shards to node 10.0.0.1
Move all existing shards on node 10.0.0.1 to other nodes in the cluster

High Performance Redis

In response to this brief blog entry, @antirez tweeted for some documentation on high-performance techniques for Redis. What I present here are general high-performance computing (HPC) techniques. The examples are oriented to Redis. but they work well for any program designed to be single- or worker-threaded and asynchronous (e.g. uses epoll).

The motivation for using these techniques is to maximize performance of our system and services. By isolating work, controlling memory, and other tuning, you can achieve significant reduction in latency and increase in throughput.

My perspective comes from the microcosm of my own bare-metal (vs VM), on-premises deployment. It might not be suitable for all scenarios, especially cloud deployments, as I have little experience with HPC there. After some discussion, maybe this can be adapted as [redis.io documentation](https://redis.io/do

	#!/bin/bash

	BUNDLE_PATH=$1

	git_ssh_clone=$(zgrep 'proto=ssh.cmd=git-upload-pack.op done' $BUNDLE_PATH/babeld-logs/babeld.log.1.gz \| grep -o 'ts=[^:]*' \| uniq -c \| sort -nr \| head -n1 \| awk '{printf "%.2f", $1/3600}')
	git_http_clone=$(zgrep 'proto=http.cmd=git-upload-pack.op done' $BUNDLE_PATH/babeld-logs/babeld.log.1.gz \| grep -o 'ts=[^:]*' \| uniq -c \| sort -nr \| head -n1 \| awk '{printf "%.2f", $1/3600}')
	git_ssh_push=$(zgrep 'proto=ssh.cmd=git-receive-pack.op done' $BUNDLE_PATH/babeld-logs/babeld.log.1.gz \| grep -o 'ts=[^:]*' \| uniq -c \| sort -nr \| head -n1 \| awk '{printf "%.2f", $1/60}')
	git_http_push=$(zgrep 'proto=http.cmd=git-receive-pack.op done' $BUNDLE_PATH/babeld-logs/babeld.log.1.gz \| grep -o 'ts=[^:]*' \| uniq -c \| sort -nr \| head -n1 \| awk '{printf "%.2f", $1/60}')

	code_zip=$(grep -E 'GET ./archive/..zip' $BUNDLE_PATH/system-logs/haproxy.log.1 \| cut -c 1-11 \| uniq -c \| sort -rn -k1 \| head -n1 \| awk '{printf "%.2f", $1/10}')

	#!/usr/bin/env python
	from __future__ import print_function

	'''
	Usage:
	./rrd_to_openmetrics.py /path/rrd_directory/ > openmetrics.txt
	'''

	'''
	Directory structure :

	Mute these words in your settings here: https://twitter.com/settings/muted_keywords

	ActivityTweet
	generic_activity_highlights
	generic_activity_momentsbreaking
	RankedOrganicTweet
	suggest_activity
	suggest_activity_feed
	suggest_activity_highlights
	suggest_activity_tweet

	# Let's see the indices already in ES
	GET _cat/indices

	# Setup: delete products index in case it exists
	DELETE products

	# Create product index
	PUT /products
	{
	"settings" : {

	#!/usr/bin/env ansible-playbook -c local
	#
	# AWS STS token update playbook.
	#
	# Updating AWS session tokens with STS can be a pain. But MFA is good. So let's
	# automate the management of the .aws/credentials file to make it not painful!
	#
	# Usage:
	#
	# 1. Save this to a file like /usr/local/bin/aws-sts-token

	source 'https://rubygems.org'

	gem 'hangouts-chat'
	gem 'sinatra'