creating spring boot with prometheus metrics (step by step)

Deployment in openshift

#To be able to deploy the app as service in openshift we can follow various routes.

#I have follwed the binary build and deployment method..
https://access.redhat.com/documentation/en-us/red_hat_jboss_middleware_for_openshift/3/html/red_hat_java_s2i_for_openshift/get_started#source_to_image_s2i_build

#Other methods are also discussed in the recoommened guide above for being able to do deployment. for private source repository 
Image secret is need while doing s2i builds from the repo

#https://developers.redhat.com/blog/2017/02/23/getting-started-with-openshift-java-s2i/ provides other useful ways of deployment

#Deployment using fabric8 maven plugin

 #inject maven dependency 
./mvnw io.fabric8:fabric8-maven-plugin:3.5.30:setup

#deploy the application (make sure you have logged into the cluster and is on the right project)
mvn package fabric8:deploy

#deploy skipping the test
mvn clean fabric8:deploy -DskipTests

For Docker build and docker based deployment
DockerFile

FROM openshift/redhat-openjdk18-openshift

ENV JAVA_APP_JAR demoapp.jar
ENV AB_ENABLED off
ENV AB_JOLOKIA_AUTH_OPENSHIFT true
ENV JAVA_OPTIONS -Xmx256m -Djava.security.egd=file:///dev/./urandom

EXPOSE 8080

ADD target/demoapp.jar /deployments/

#can be build locally and pushed into openshift registry.
oc new-app --from-docker=registry/proj/dockerimage

or be use docker build type in openshift , where build config can pull out Dockerfile from the repository

Deployment of monitoring task

oc process prometheus-template | oc apply -f-

oc process grafana-template -p NAMESPACE=sandbox-promapp | oc apply -f-

#for prometheus to be able to read metrics from pods / services in the same project

oc policy add-role-to-user view -z default

#Add the the following annotation on the desired svc of app to be able to scrape

prometheus.io/path: /actuator/prometheus
prometheus.io/port: "8080"
prometheus.io/scrape: "true"

enable metrics end points

#Firstly this application is created using spring boot 2.0 so if is your application is at 1.5 the one some adjustment might be needed
#helpful article
http://www.ru-rocker.com/2018/03/17/setup-micrometer-prometheus-spring-boot-1-5/


We are using ootb micrometer lib to be able to get metrics from out app to be able to that 
first and fore most add  in the pom.xml

<!-- Micormeter core dependecy  -->
                <dependency>
                        <groupId>io.micrometer</groupId>
                        <artifactId>micrometer-core</artifactId>
                </dependency>
                <!-- Micrometer Prometheus registry  -->
                <dependency>
                        <groupId>io.micrometer</groupId>
                        <artifactId>micrometer-registry-prometheus</artifactId>
                </dependency>
                
 File Once added the prometheus endpoint needs to be enabled addint this under src/main/resources/application.properties
 
 management.endpoint.metrics.enabled=true
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true

run the application now and you should be able to get metrics for your application OOTB: JvmMemoryMetrics , UptimeMetrics and LogbackMetrics about http available
at http://192.168.56.101:8080/actuator/prometheus


Now we can also enable cumulative histograms for SLAs and distribution percentiles by simply providing the below configurations.

management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.sla.http.server.requests=50ms

#All properties available here https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready

example of bucket 
http_server_requests_seconds_bucket{application="demoapplication",exception="None",method="GET",status="200",uri="/actuator/prometheus",le="0.001048576",} 0.92

So We can further customise to expose metrics on our own
add 
import io.micrometer.core.annotation.Timed;
in the controller and 

and @Timed(value = "controller.whereami.requests", histogram = true, percentiles = { 0.95, 0.99 }, extraTags = { "version","v1" })
above controller methods


controller_whereami_requests_seconds{application="demoapplication",exception="None",method="GET",status="200",uri="/whereami",version="v1",quantile="0.95",} 0.071041024
controller_whereami_requests_seconds{application="demoapplication",exception="None",method="GET",status="200",uri="/whereami",version="v1",quantile="0.99",} 0.003866624


Now to be aslo able to provide command tags on your application like application_name="demoapplication"
create a config classs with following things

package com.example.demoboot.config;

import org.springframework.context.annotation.Bean;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.context.annotation.Configuration;
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;

@Configuration
public class MetricConfiguration {

    /**
     * Register common tags application instead of job.
     * This application tag is needed for Grafana dashboard.
     *
     * @return registry with registered tags.
     */
    @Bean
    MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
        return registry -> {
            registry.config().commonTags("application", "demoapplication");
	    //can add more tags with .commonTags("cf.space.id", spaceId)
        };
    }

}

#helpful links
#https://www.baeldung.com/micrometer
#http://www.bytesville.com/springboot-micrometer-prometheus-grafana/
#https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/
#https://www.callicoder.com/spring-boot-actuator/

Springboot App

#install Java
yum install java-1.8.0-openjdk-devel.x86_64

#installing spring boot
wget https://repo.spring.io/release/org/springframework/boot/spring-boot-cli/1.5.9.RELEASE/spring-boot-cli-1.5.9.RELEASE-bin.tar.gz
tar -zxf spring-boot-cli-1.5.9.RELEASE-bin.tar.gz
export SPRING_HOME="${HOME}/spring-1.5.9.RELEASE"
export PATH="${SPRING_HOME}/bin:${PATH}"

#check the spring boot installation
spring  version
`Spring CLI v1.5.9.RELEASE

#Create spring boot project
spring init --artifactId=demoboot --groupId=com.example --dependencies=web,actuator --extract $HOME/demoboot

cd demoboot
mkdir -p  src/main/java/com/example/demoboot
touch src/main/java/com/example/demoboot/DemoBootController.java

#contents of DemoBootController.java

```
package com.example.demoboot;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
/**

Simple REST call that says from where its invoked from
*/
@RestController
public class DemoBootController {
 @GetMapping("/whereami")
 public String whereami() {
    return String.format("Hello from %s", System.getenv().getOrDefault("HOSTNAME", "localhost")); }
}
```

#Installing maven for compilation of the app
tar xzf apache-maven-3.5.4-bin.tar.gz
ln -s apache-maven-3.5.4 maven

vi /etc/profile.d/maven.sh

#contents of maven.sh
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}

source /etc/profile.d/maven.sh

#check the installaion of maven
mvn -version

#Running the application 
mvn spring-boot:run -DskipTests

#Allow access from firewall
firewall-cmd --add-port="8080/tcp" --permanent
firewall-cmd --reload