bbeaudreault’s gists

bbeaudreault / gist:5710269

Last active December 18, 2015 02:19

Alternate implementation of Bookie#getBookieAddress(ServerConfiguration conf) which is configurable to look at a specific interface.

	import java.net.InetAddress;
	import java.net.InetSocketAddress;
	import java.net.InterfaceAddress;
	import java.net.NetworkInterface;
	import java.net.SocketException;
	import java.net.UnknownHostException;

	import org.apache.bookkeeper.conf.ServerConfiguration;

	public class Bookie {

bbeaudreault / gist:5834357

Created June 21, 2013 21:13

Cleanup hadoop jobcache files

	#!/bin/bash

	for DIR in `find /mnt/mapred/local/taskTracker//jobcache/ -maxdepth 0 -type d -mmin +60`; do
	if ! find $DIR \| grep attempt > /dev/null; then
	rm -rf $DIR;
	fi;
	done;

	# There is also another bug that results in jobcache directories being duplicated
	# within the attempt_ directories we filter out above. These directories never go away and so jobs with

bbeaudreault / HMerge.java

Last active December 28, 2015 21:49

Modifications to 0.94.x HMerge tool, to allow running against an online cluster with target table disabled. Important: See README.

	/**
	* Copyright 2009 The Apache Software Foundation
	*
	* Licensed to the Apache Software Foundation (ASF) under one
	* or more contributor license agreements. See the NOTICE file
	* distributed with this work for additional information
	* regarding copyright ownership. The ASF licenses this file
	* to you under the Apache License, Version 2.0 (the
	* "License"); you may not use this file except in compliance
	* with the License. You may obtain a copy of the License at

bbeaudreault / HBaseMapReduceUtils.java

Last active November 8, 2017 06:28

Control # of mappers per RegionServer

	package org.apache.hadoop.hbase.mapreduce;

	import java.io.IOException;
	import java.util.Collections;
	import java.util.List;

	import org.apache.hadoop.hbase.HBaseConfiguration;
	import org.apache.hadoop.hbase.client.Scan;
	import org.apache.hadoop.hbase.util.Bytes;
	import org.apache.hadoop.mapreduce.Job;

bbeaudreault / gist:2994a748da83d9f75085

Created November 30, 2015 23:09

	1 times (0% total cpu)
	"2116134604@qtp-1167186972-0" daemon prio=10 tid=0x7f20df278000 nid=0xb655 in Object.wait() start=2015-11-30 21:17:39
	java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
	- locked <0x00007f19080118c0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

	1 times (2.5% total cpu)
	"RpcServer.responder" daemon prio=10 tid=0x7f20dfb64000 nid=0xb6f9 runnable cpu=2.5% start=2015-11-30 21:17:42
	java.lang.Thread.State: RUNNABLE

bbeaudreault / gist:5214b28319981c18379f

Created March 2, 2016 22:43

	java.lang.AssertionError: Key REDACTED_ROWKEY/2:\x00\x05\x12\xBF8/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0 followed by a smaller key REDACTED_ROWKEY/2:\x00\x03,\x0E0/1456850400248000000/Put/vlen=10/seqid=0 in cf 2
	at org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:787)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:542)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5272)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5423)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5210)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5187)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5174)
	at org.apache.h

bbeaudreault / RpcHandlerUsageSourceImpl.java

Created June 7, 2016 15:05

	package org.apache.hadoop.hbase.regionserver.metrics;

	import java.util.HashMap;
	import java.util.HashSet;
	import java.util.Map;
	import java.util.Set;
	import java.util.Timer;
	import java.util.TimerTask;
	import java.util.concurrent.ConcurrentHashMap;
	import java.util.concurrent.TimeUnit;

bbeaudreault / build_basic_vitess.sh

Created November 21, 2016 19:47

Building basic vitess vtgate and vtctld

	# May need to install the proper version (1.7+) of go before starting
	export GOPATH=~/work
	export PATH=$PATH:$GOPATH/bin
	mkdir -p $GOPATH/src $GOPATH/bin $GOPATH/pkg
	git clone [email protected]:youtube/vitess.git $GOPATH/src/github.com/youtube/vitess
	cd $GOPATH/src/github.com/youtube/vitess
	go get -u github.com/golang/lint/golint github.com/golang/mock/mockgen github.com/kardianos/govendor golang.org/x/tools/cmd/goimports honnef.co/go/unused/cmd/unused code.google.com/p/go.tools/cmd/cover
	govendor sync
	cd go/cmd
	go install ./vtgate ./vtctld

bbeaudreault / build.sh

Last active December 9, 2016 15:58

rpm builder for vitess

	#!/bin/sh -ex

	DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

	vitess_version='2.1.0-alpha'

	source /opt/rh/devtoolset-4/enable
	source /opt/rh/python27/enable

	export GOROOT="/usr/local/go"

bbeaudreault / vtparse.patch

Created March 29, 2017 12:27

Disclaimer: This was the first piece of go I wrote and I've not run this in a few months. It may not work as is, but could be used as a starting point for some hacking. It expects to be run against the general log of mysql, and makes some assumptions about you care about. For instance it ignores a long list of DBA queries

	From a89a8d6144136512574681f36e5e9828c757090d Mon Sep 17 00:00:00 2001
	From: Bryan Beaudreault <[email protected]>
	Date: Wed, 4 Jan 2017 11:23:12 -0500
	Subject: [PATCH] vtparse -- a tool for analyzing query logs, sending them
	through vitess planbuilders, and outputting errors

	---
	go/cmd/vtparse/main.go \| 10 +
	go/cmd/vtparse/plugin_cephbackupstorage.go \| 5 +
	go/cmd/vtparse/plugin_etcd2topo.go \| 22 +

Bryan Beaudreault bbeaudreault