Small sample of my code... Please read README.txt file first (it is at the bottom of page).

README.txt

This is major part of statistics framework which I have invented and developed. It is designed to be small, fast and easy to use. It is also designed to work in multithreaded environment.

Main design principles in details:
1. Updating statistics should be simple and easily understandable on user side.

User code:
manager.metric( HPA.calls.all ).increment(); // increment number of calls
manager.metric( HPA.calls.failed ).get(); // getting number of failed calls

HPA.call.all and HPA.call.failed is object oriented definition of specific statistic. This definition is created statically at start of server in client code (not part of framework). It serves here as identifier for specific statistic, which should be changed or just read.

2. All aggregation should be done in statistics code, not manually by user. It should be easy to change rules of aggregation without affecting user code.

Aggregation is done by forwarding events. Event is e.g. incrementing or setting specific statistic. Such an event is forwarded to other metrics which can aggregate changes for specific metric. Definition of what should be forwarded is in client code:


public class HpaMain extends HpaMetricGroupDef<HpaMain>
{
  public final Sources sources = new Sources();
  public final Timed timed = new Timed();

  @Override
  public void setupForwarders() {
    sources.forwardGroupTo( timed.min.sources ); // <----- HERE
  }
}


3. Statistics should be used in client code to get information e.g. about global number of events, number of events per request and number of request in specific time (in last minute, last hour, last day etc).

Counting global number of events is easy in above schema. Also timed statistics is quite easy and can be achieved by forwarding statistics events to some group (e.g. Timed as in above example) and then every some period of time these timed statistics can be harvested and sent to receiver. But what about statistics per request, which should also update main counters?

//main processing loop
StatisticManager manager = new DefaultStatisticManager();
...
serveRequest(manager.fork()); //asynchronously starts processing of request; main statistic manager is forked
...

Forked manager is linked with main statistics manager, so changes in forked manager are reflected in main manager.

4. Framework should be fast to be able to work on production machines.

Statistics definitions are forming kind of tree:
HPA.calls.all
HPA.calls.failed
HPA.timed.calls.all
HPA.timed.calls.failed

To make framework fast and still allow it to be easy to use I had to invent some algorithm to make it fast. Resolution was to assign to each statistics group definition (e.g. HPA.calls, HPA.timed, HPA.timed.calls) identifier, which allows to find quickly specific statistic manager responsible for updating specific counter.

Identifier is assigned in such a way that statistic manager, which is handling request, is able to find out very quickly where to look for manager responsible for handling statistics update. You can see it in GroupDef.java -> setupUniqueIds

5. Framework should be small

... and it is indeed small. Whole solution (without tests and client side definitions) is about 850 lines of code. Small size is achieved because of heavy reuse of code through recursive resolution of problem. It was quite hard to write this code, but client side part is very simple to use.

DefaultStatisticManager.java

/*******************************************************************************
 * Copyright (c) 2013 by Marcin Kuszczak. All rights reserved.
 ******************************************************************************/
package net.igsoft.statistic;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;

import net.igsoft.statistic.group.GroupDef;
import net.igsoft.statistic.group.StatisticManagerInternal;
import net.igsoft.statistic.metric.internal.Metric;
import net.igsoft.statistic.metric.internal.MetricDef;
import net.igsoft.statistic.metric.internal.MetricDefBase;

import com.google.common.collect.ImmutableList;

/**
 * @author Marcin Kuszczak
 */
public class DefaultStatisticManager implements StatisticManagerInternal
{
  private final StatisticManagerInternal root;
  private final StatisticManagerInternal parent;
  private final GroupDef groupDef;
  private final ImmutableList<AtomicReference<StatisticManagerInternal>> groups;
  private final ImmutableList<Metric> metrics;

  public DefaultStatisticManager( GroupDef definition ) {
    this( null, null, definition, false );
  }

  protected DefaultStatisticManager( StatisticManagerInternal creator,
                                     StatisticManagerInternal parent,
                                     GroupDef groupDef,
                                     boolean linked )
  {
    this.root = (creator == null) ? this : creator.root();
    this.parent = parent;
    this.groupDef = groupDef;
    this.groups = createGroups( groupDef, linked );
    this.metrics = createMetrics( groupDef, linked ? creator : null );
  }

  @Override
  public StatisticManagerInternal group( GroupDef group ) {
    if ( group.id() == groupDef.id() ) {
      //I am manager of 'group'...
      return this;
    }

    if ( group.id() >= groupDef.leftBound() && group.id() <= groupDef.rightBound() ) {
      //One of my successors is manager of 'group'
      return localGroup( group.path()[groupDef.level()] ).group( group );
    }

    //One of my ancestors is manager of 'group'
    return parent.group( group );
  }

  @Override
  public <T extends Metric> T metric( MetricDefBase<?, T> metric ) {
    return (T) group( metric.group() ).localMetric( metric.index() );
  }

  @Override
  public StatisticManagerInternal fork() {
    return new DefaultStatisticManager( this, parent, groupDef, true );
  }

  @Override
  public StatisticManagerInternal incarnate() {
    return new DefaultStatisticManager( this, parent, groupDef, false );
  }

  @Override
  public StatisticManagerInternal root() {
    return root;
  }

  @Override
  public StatisticManagerInternal localGroup( int index ) {
    return groups.get( index ).get();
  }

  @Override
  public Object localMetric( int index ) {
    return metrics.get( index );
  }

  @Override
  public StatisticManagerInternal parent() {
    return parent;
  }

  @Override
  public GroupDef definition() {
    return groupDef;
  }

  @Override
  public String toString() {
    return "StatisticManager: [" + groupDef.pathName() + "]";
  }

  @Override
  public StatisticManager swap( GroupDef group ) {
    StatisticManagerInternal sm = group( group ).parent();

    if ( sm == null ) {
      throw new IllegalArgumentException( "You can not swap root group." );
    }

    return sm.setLocalGroup( group.index(),
                             new DefaultStatisticManager( sm,
                                                          sm.parent(),
                                                          sm.definition(),
                                                          false ) );
  }

  @Override
  public StatisticManagerInternal setLocalGroup( int index,
                                                 StatisticManagerInternal statisticManager )
  {
    return groups.get( index ).getAndSet( statisticManager );
  }

  private ImmutableList<Metric> createMetrics( GroupDef definition,
                                               StatisticManagerInternal linked )
  {
    ArrayList<Metric> localMetrics = new ArrayList<>( definition.metrics().size() );

    for (MetricDef metric : definition.metrics()) {
      localMetrics.add( metric.index(), metric.incarnate( this, linked ) );
    }

    return ImmutableList.copyOf( localMetrics );
  }

  private ImmutableList<AtomicReference<StatisticManagerInternal>> createGroups( GroupDef definition,
                                                                                 boolean linked )
  {
    List<AtomicReference<StatisticManagerInternal>> localGroups = new ArrayList<>( definition.groupCount() );

    for (GroupDef group : definition.groups()) {
      localGroups.add( group.index(),
                       new AtomicReference<StatisticManagerInternal>( new DefaultStatisticManager( this,
                                                                                                   this,
                                                                                                   group,
                                                                                                   linked ) ) );
    }

    return ImmutableList.copyOf( localGroups );
  }
}

DefaultStatisticManagerTest.java

/*******************************************************************************
 * Copyright (c) 2013 by Marcin Kuszczak. All rights reserved.
 ******************************************************************************/
package net.igsoft.statistic;

import static net.igsoft.statistic.clientdata.application.HpaStatistics.*;
import static org.fest.assertions.api.Assertions.*;
import net.igsoft.statistic.DefaultStatisticManager;
import net.igsoft.statistic.StatisticManager;
import net.igsoft.statistic.metric.LongMetric;

import org.junit.Before;
import org.junit.Test;

/**
 * @author Marcin Kuszczak
 */
public class DefaultStatisticManagerTest
{

  private StatisticManager manager;

  @Before
  public void setUp() {
    manager = new DefaultStatisticManager( HPA );
  }

  @Test
  public void testGetRootGroup() {
    //Given-When-Then
    assertThat( manager.group( HPA ) ).isSameAs( manager );
  }

  @Test
  public void testGetChildGroup() {
    //Given-When-Then
    assertThat( manager.group( HPA.timed.minute ).definition().level() ).isEqualTo( 2 );
    assertThat( manager.group( HPA.timed.minute ).definition().path() ).isEqualTo( new int[] {
        2, 2
    } );
  }

  @Test
  public void testGetParentGroup() {
    //Given-When-Then
    StatisticManager submanager = manager.group( HPA.timed.minute );

    assertThat( submanager.group( HPA.call ).definition().level() ).isEqualTo( 1 );
    assertThat( submanager.group( HPA.call ).definition().path() ).isEqualTo( new int[] {
      1
    } );
  }

  @Test
  public void testGetMetricValue() {
    //Given-When-Then
    assertThat( manager.metric( HPA.call.all ).get() ).isEqualTo( 0 );
  }

  @Test
  public void testIncrementMetric() {
    //Given-When
    LongMetric metric = manager.metric( HPA.call.all );
    metric.increment();

    //Then
    assertThat( manager.metric( HPA.call.all ).get() ).isEqualTo( 1 );
  }

  @Test
  public void testPercentMetric() {
    //Given-When
    LongMetric all = manager.metric( HPA.call.all );
    all.increment();
    all.increment();
    all.increment();
    all.increment();

    LongMetric failed = manager.metric( HPA.call.failed );
    failed.increment();

    //Then
    assertThat( manager.metric( HPA.call.failedPt ).get() ).isEqualTo( 25.0 );
  }

  @Test
  public void testFork() {
    //Given-When
    StatisticManager forked = manager.fork();
    assertThat( forked.parent() ).isSameAs( manager.parent() );
    assertThat( forked.root() ).isSameAs( manager.root() );
  }

  @Test
  public void testForwardingFromForkedGroup() {
    //Given
    StatisticManager original = manager.group( HPA.call );
    StatisticManager forked1 = original.fork();
    StatisticManager forked2 = original.fork();

    //When
    forked1.metric( HPA.call.all ).increment();
    forked2.metric( HPA.call.all ).increment();
    original.metric( HPA.call.all ).increment();

    //Then
    assertThat( forked1.metric( HPA.call.all ).get() ).isEqualTo( 1 );
    assertThat( forked2.metric( HPA.call.all ).get() ).isEqualTo( 1 );
    assertThat( original.metric( HPA.call.all ).get() ).isEqualTo( 3 );

    assertThat( original.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
    assertThat( forked1.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
    assertThat( forked2.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
  }

  @Test
  public void testIncarnatedGroup() {
    //Given
    StatisticManager original = manager.group( HPA.call );
    StatisticManager incarnated1 = original.incarnate();
    StatisticManager incarnated2 = original.incarnate();

    //When
    incarnated1.metric( HPA.call.all ).increment();
    incarnated2.metric( HPA.call.all ).increment();
    original.metric( HPA.call.all ).increment();

    //Then
    assertThat( incarnated1.metric( HPA.call.all ).get() ).isEqualTo( 1 );
    assertThat( incarnated2.metric( HPA.call.all ).get() ).isEqualTo( 1 );
    assertThat( original.metric( HPA.call.all ).get() ).isEqualTo( 1 );

    assertThat( original.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
    assertThat( incarnated1.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
    assertThat( incarnated2.metric( HPA.timed.minute.all ).get() ).isEqualTo( 3 );
  }

  @Test
  public void testSwap() {
    //Given
    StatisticManager original = manager.group( HPA.call );
    StatisticManager old = manager.swap( HPA.call );

    assertThat( original ).isSameAs( old );
    assertThat( original ).isNotSameAs( manager.group( HPA.call ) );
  }
}

GroupDef.java

/*******************************************************************************
 * Copyright (c) 2013 by Marcin Kuszczak. All rights reserved.
 ******************************************************************************/
package net.igsoft.statistic.group;

import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;

import net.igsoft.statistic.CommonEntity;
import net.igsoft.statistic.StatisticException;
import net.igsoft.statistic.metric.internal.MetricDef;

import org.apache.commons.lang3.ArrayUtils;

/**
 * @author Marcin Kuszczak
 */
public abstract class GroupDef implements CommonEntity
{
  protected static final String ROOT_STRING = "root";

  protected GroupDef parent = null;
  protected List<GroupDef> groups;
  protected List<MetricDef> metrics;
  protected int[] path;
  protected String name;
  protected int index;
  protected int level;
  protected int leftBoundAndUniqueId;
  protected int rightBound;

  public abstract GroupDef root();

  public int id() {
    return leftBoundAndUniqueId;
  }

  public int leftBound() {
    return leftBoundAndUniqueId;
  }

  public int rightBound() {
    return rightBound;
  }

  @Override
  public int index() {
    return index;
  }

  public int level() {
    return level;
  }

  public int[] path() {
    return path;
  }

  public List<GroupDef> groups() {
    return groups;
  }

  public List<MetricDef> metrics() {
    return metrics;
  }

  public int groupCount() {
    return groups.size();
  }

  public int metricCount() {
    return metrics.size();
  }

  @Override
  public String name() {
    return name;
  }

  @Override
  public String pathName() {
    if ( parent == null ) {
      return name;
    }

    return parent.pathName() + "." + name;
  }

  @Override
  public String toString() {
    return pathName();
  }

  public void setupForwarders() {
  }

  protected void setup() {
    initialize();
    setupGroupsAndMetrics();
    setupUniqueIds();
    setupSubGroups();
    setupForwarders();
  }

  protected abstract void setRoot( GroupDef root );

  private void setParent( GroupDef parent ) {
    this.parent = parent;
  }

  private void setName( String name ) {
    this.name = name;
  }

  private void setLeftBoundAndUniqueId( int leftBoundAndUniqueId ) {
    this.leftBoundAndUniqueId = leftBoundAndUniqueId;
  }

  private void setRightBound( int rightBound ) {
    this.rightBound = rightBound;
  }

  private void setIndex( int index ) {
    this.index = index;
  }

  private void initialize() {
    this.groups = new ArrayList<>();
    this.metrics = new ArrayList<>();

    if ( parent == null ) {
      this.leftBoundAndUniqueId = 0;
      this.rightBound = Integer.MAX_VALUE;
      this.level = 0;
      this.path = new int[0];
      this.index = 0;
      this.level = 0;
      this.name = ROOT_STRING;
    }
    else {
      this.level = parent.level + 1;
      this.path = ArrayUtils.add( parent.path, this.index );
    }
  }

  private void setupGroupsAndMetrics() {
    if ( parent == null ) {
      setRoot( this ); //ROOT must be set using virtual method to have defined field: 'root' with correct type
    }

    try {
      for (Field field : getClass().getFields()) {

        if ( MetricDef.class.isAssignableFrom( field.getType() ) ) {
          MetricDef metric = (MetricDef) field.get( this );
          metric.setParent( this );
          metric.setName( field.getName() );
          metric.setIndex( metricCount() );
          metrics.add( metric );
        }
        else if ( GroupDef.class.isAssignableFrom( field.getType() ) ) {
          GroupDef group = (GroupDef) field.get( this );
          group.setRoot( root() );
          group.setParent( this );
          group.setName( field.getName() );
          group.setIndex( groupCount() );
          groups.add( group );
        }
      }
    }
    catch (IllegalArgumentException | IllegalAccessException e) {
      throw new StatisticException( "Error while configuring fields.", e );
    }
  }

  private void setupUniqueIds() {
    if ( groups.size() > 0 ) {
      int range = rightBound - leftBoundAndUniqueId - groups.size();
      int subrange = range / groups.size();

      if ( subrange == 0 ) {
        throw new StatisticException( "No available ids for creating statistics definition." );
      }

      for (int i = 0; i < groups.size(); i++) {
        int leftAndId, right;
        leftAndId = leftBoundAndUniqueId + 1 + i + i * subrange;
        right = leftAndId + subrange;

        groups.get( i ).setLeftBoundAndUniqueId( leftAndId );
        groups.get( i ).setRightBound( right );
      }
    }
  }

  private void setupSubGroups() {
    for (GroupDef group : groups) {
      group.setup();
    }
  }
}

HpaMainGroupDef.java

/*******************************************************************************
 * Copyright (c) 2013 by Marcin Kuszczak. All rights reserved.
 ******************************************************************************/
package net.igsoft.statistic.clientdata.application;

import net.igsoft.statistic.clientdata.module.ShardconnectorMainGroupDef;
import net.igsoft.statistic.clientdata.module.ShardconnectorStatistics;

/**
 * @author Marcin Kuszczak
 */
public class HpaMainGroupDef extends HpaMetricGroupDef<HpaMainGroupDef>
{

  public final GenericMetricGroupDef generic = new GenericMetricGroupDef();

  public final CallStatsMetricGroupDef call = new CallStatsMetricGroupDef();
  public final TimedUnitMetricGroupDef timed = new TimedUnitMetricGroupDef();

  public final TestMetricGroupDef metricTest = new TestMetricGroupDef();
  public final TestMetricGroupDef metricTestShadow = new TestMetricGroupDef();

  public final ShardconnectorMainGroupDef shardconnector = ShardconnectorStatistics.SHARDCONNECTOR;

  public HpaMainGroupDef() {
    setup();
  }

  @Override
  public void setupForwarders() {
    metricTest.forwardGroupTo( metricTestShadow );
  }
}