Noah Zucker noahlz

Troubleshooting Production JVMs with `jcmd`

jcmd is a powerful new tool introduced in Java 7. Along with jstack and jps, it should be in your go-to tool for solving production problems on the JVM. (Come to think of it, with this tool you don't really need jps anymore)

Here's an example session with jcmd:

$ ssh wopr.qa.corp.local
$ jcmd -l
34739 sun.tools.jcmd.JCmd -l

Operations Team should not deploy a new software application to production unless the Development Team complies with the following minimum requirements:

Software Requirements

Packaging: the application must conform to conventions for configuration, directory structure and run scripts:
- Configuration
- Deployment Build
- Run script
Reference Configuration: The reference configuration file included in the package must:

Here's my idea of what goes in a run book for an application:

On which servers it is installed
Databases to which it connects (externally managed, not embedded)
how to start/stop, i.e. actual command line or script to execute
Scheduled batch jobs / EOD processes. Not the actual schedule, but when they generally run (EOD, overnight, etc)
Where to find log files
How to verify that it's actually running
A screen shot of the login page and one or two application pages is helpful.

Scala can be a little obtuse. Yesterday I encountered something like this:

def foo(bar: Bar): Option[Bazz => Buzz => Blah]

It's a function that takes Bar and returns an Option boxing a function that takes a Buzz and returns a Blah. I think some clarifying parenthesis are in order, and probably should be required by the compiler.

When you see something like this:

	trait Color {
	def name: String
	}

	abstract class PrimaryColor(val name: String) extends Color
	case object Red extends PrimaryColor("Red")
	case object Blue extends PrimaryColor("Blue")
	case object Yellow extends PrimaryColor("Yellow")

	trait HasColors {

	(ns hello.core
	(:require [clojure.java.io :as io])
	(:import [java.net ServerSocket]))

	;; Server
	(defn receive-msg [socket]
	(.readLine (io/reader socket)))

	(defn send-msg [socket msg]
	(let [writer (io/writer socket)]

	def divBy3(x: Int) = x % 3 == 0
	// divBy3: (x: Int)Boolean

	def divBy5(x: Int) = x % 5 == 0
	// divBy5: (x: Int)Boolean

	def divByAll(fns: List[(Int => Boolean)])(x: Int) = fns.foldLeft(true)((modZero, fn: (Int => Boolean)) => modZero && fn(x))
	// divByAll: (fns: List[Int => Boolean])(x: Int)Boolean

	val divBy3And5 = divByAll(List(divBy3,divBy5)) _

	// See http://stackoverflow.com/questions/1052476/what-is-scalas-yield/1059501#1059501

	// Compilation error is:
	// [error] found : Seq[String]
	// [error] required: Option[?]
	// [error] child <- foo.children
	// [error] ^
	// [error] one error found

	# http://stackoverflow.com/questions/6012665/read-non-blocking-error-while-using-pxssh-ssh-module-for-python
	# https://github.com/paramiko/paramiko/blob/master/README
	# http://stackoverflow.com/questions/8491194/how-to-handle-socket-errors-as-exceptions-in-python-paramiko

	import paramiko, base64, getpass, socket

	client = paramiko.SSHClient()
	client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

	user = getpass.getuser()


	// Welcome to Scala version 2.9.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_15).
	// Type in expressions to have them evaluated.
	// Type :help for more information.

	def foo(n: Int) = Some(n)
	// foo: (n: Int)Some[Int]

	List(1,2,3,4).map(foo(_))
	// res2: List[Some[Int]] = List(Some(1), Some(2), Some(3), Some(4))

Noah Zucker noahlz

Troubleshooting Production JVMs with jcmd

Software Requirements

Troubleshooting Production JVMs with `jcmd`