kelapure

Resiliency Testing for Cloud Foundry

Application Failure Scenarios

1. [HEAP-OOM] App instance allocates excess Java Heap Memory. App repetitively suffers java heap OOM.
2. [NATIVE-OOM] App instance allocates excess native memory leading to crashes of the warden container.
3. [EXCESS-CPU] App instance is pegged at > 90% CPU for a sustained period of time.
4. [DEADLOCK] App instance is pegged < 50% CPU inspite of sustained increasing request traffic.
5. [GC-PAUSES] App unresponsive for random periods of time.
6. [RANDOM-DEATH] App instance randomly exits.

kelapure

#!/bin/bash

set -e

export PIVNET_TOKEN=GET_FROM_PIVNET_USER_PROFILE_PAGE
export LOCAL_FILE_NAME=pcf-mysql.pivotal
export DOWNLOAD_URL=https://network.pivotal.io/api/v2/products/p-mysql/releases/211/product_files/1755/download
export OPSMGR_HOST=localhost
export OPSMGR_USER=WHATEVER_YOUR_OPSMGR_USER_IS
export OPSMGR_PASSWORD=WHATEVER_YOUR_OPSMGR_PASSWORD_IS

kelapure

package org.cloudfoundry.samples.music.debug;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.Writer;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;

kelapure

while test 1
do
 timeval=`date '+Jun%d.%H.%M'`
./jcmd $PID VM.native_memory summary > NativeMemoryReport.$timeval.Summary.txt
./jcmd $PID VM.native_memory summary.diff > NativeMemoryReport.$timeval.Summary.diff.txt
./jcmd $PID VM.native_memory detail > NativeMemoryReport.$timeval.Detail.txt
./jcmd $PID VM.native_memory detail.diff > NativeMemoryReport.$timeval.DetailDiff.txt
./jstack $PID  > NativeMemoryReport.$timeval.Threads.txt
pmap -x $PID   > NativeMemoryReport.$timeval.pmap.txt
ps aux | grep $PID   > NativeMemoryReport.$timeval.ps.txt

kelapure

For all versions of WebSphere ( that do not override dump agents)

kill -6 <WebSphere Process id>   should produce a core file and leave the server running
kill -11 <WebSphere Process id> should produce a core file and halt the process

PID=` ps -ef | grep java | grep -v "bash\|grep" | awk '{print $2}'`
./jmap -dump:format=b,file=/home/vcap/app/test.hprof $PID

For WebSphere Application Server 7.0 and following, wsadmin scripting may be used to request System Core files by finding the process and issuing the command : See the Knowledge Center for more information: wsadmin script to request core file A jython example for server1 would be:

kelapure

Rohit-Kelapures-MacBook-Pro:spring-music rkelapure$ cf push spring-music -b https://github.com/cloudfoundry-community/tomee-buildpack.git
Using manifest file /Users/rkelapure/git/spring-music/manifest.yml

Creating app spring-music in org EVP / space pse-staging as rkelapure@pivotal.io...
OK

Creating route spring-music-halloween.cfapps.io...
OK

kelapure

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.cloud.bus.BusAutoConfiguration;
import org.springframework.cloud.bus.BusPathMatcher;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;
import org.springframework.util.AntPathMatcher;
import org.springframework.util.PathMatcher;

kelapure

 package com.lmig.pli.rate.auto.autorating.event;

import com.lmig.pli.rate.auto.autorating.config.EventAutoConfig;
import com.lmig.pli.rate.auto.autorating.service.MessageService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.util.Pair;
import org.springframework.integration.annotation.Aggregator;
import org.springframework.integration.annotation.CorrelationStrategy;
import org.springframework.integration.annotation.ReleaseStrategy;

kelapure

FROM websphere-liberty:kernel

ARG REPOSITORIES_PROPERTIES=""

ADD SleepyBatchletSample-1.0.war /config/dropins/

COPY server.xml /config/

RUN if [ ! -z $REPOSITORIES_PROPERTIES ]; then mkdir /opt/ibm/wlp/etc/ \
  && echo $REPOSITORIES_PROPERTIES > /opt/ibm/wlp/etc/repositories.properties; fi \

kelapure

<?xml version="1.0" encoding="UTF-8"?>
<server description="Default server">

    <!-- Enable features -->
    <featureManager>
        <feature>batch-1.0</feature>
        <feature>ssl-1.0</feature>
    </featureManager>

    <!-- To allow access to this server from a remote client host="*" has been added to the following element -->