yash yashk

Generating Flame Graphs for Apache Spark

Flame graphs are a nifty debugging tool to determine where CPU time is being spent. Using the Java Flight recorder, you can do this for Java processes without adding significant runtime overhead.

When are flame graphs useful?

Shivaram Venkataraman and I have found these flame recordings to be useful for diagnosing coarse-grained performance problems. We started using them at the suggestion of Josh Rosen, who quickly made one for the Spark scheduler when we were talking to him about why the scheduler caps out at a throughput of a few thousand tasks per second. Josh generated a graph similar to the one below, which illustrates that a significant amount of time is spent in serialization (if you click in the top right hand corner and search for "serialize", you can see that 78.6% of the sampled CPU time was spent in serialization). We used this insight to spee

Shakespeare Sonnet++ Postmortem (incident #465)

Date

2015-10-21

Authors

jennifer
martym

How to add an image to a gist

Create or find a gist that you own.

Clone your gist (replace <hash> with your gist's hash):

# with ssh
git clone [email protected]:<hash>.git mygist

# with https

git clone https://gist.github.com/.git mygist

Currently, there is an explosion of tools that aim to manage secrets for automated, cloud native infrastructure management. Daniel Somerfield did some work classifying the various approaches, but (as far as I know) no one has made a recent effort to summarize the various tools.

This is an attempt to give a quick overview of what can be found out there. The list is alphabetical. There will be tools that are missing, and some of the facts might be wrong--I welcome your corrections. For the purpose, I can be reached via @maxvt on Twitter, or just leave me a comment here.

There is a companion feature matrix of various tools. Comments are welcome in the same manner.

Questions come up some times about why things get bad data. It is hard to pin things down, as maybe you can't modify the code, or even know where to look. Here's an example of how to track down what's making bad trace ids in finagle.

Firstly, download and extract byteman so you can use it.

$ (cd /tmp; wget -q http://downloads.jboss.org/byteman/3.0.5/byteman-download-3.0.5-bin.zip; unzip byteman-download-3.0.5-bin.zip)
$ export BYTEMAN_HOME=/tmp/byteman-download-3.0.5/

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

Clarify and agree on the scope of the system

User cases (description of sequences of events that, taken together, lead to a system doing something useful)
- Who is going to use it?
- How are they going to use it?

For the latest download link check the Apache Maven download page:

https://maven.apache.org/download.cgi

Download Apache Maven

cd~
wget "http://mirror.sdunix.com/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz"

Create directory and install

Apache Hadoop - add native libraries

If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Clone hadoop source code

	#!/bin/bash
	#
	# Restart Bluetooth Module on Mac OS X
	#
	# Requires Blueutil to be installed: http://brewformulas.org/blueutil

	BT="/usr/local/bin/blueutil"

	log() {
	echo "$@"

	@Order(Ordered.LOWEST_PRECEDENCE)
	public class MyFrameworkPropertyDefaultsPostProcessor implements EnvironmentPostProcessor {

	private static final String DEFAULT_MY_FRAMEWORK_PROPERTIES = "defaultMyFrameworkProperties";
	private static final String DEFAULT_MY_FRAMEWORK_PROPERTIES_LOCATION = "/config/myFramework-application.yml";

	private YamlPropertySourceLoader propertySourceLoader = new YamlPropertySourceLoader();

	@Override
	public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {