The Kuang program was green, exactly the shade of the T-A ice.
Even as Case watched, it grew gradually more opaque.
-- William Gibson, Neuromancer
The Brooklyn live (and integration) tests are in a pretty poor state at present. Although all the unit tests pass during a build, the 'Brooklyn-Master-Live-*' and 'Brooklyn-Master-Integration' jobs have been failing for months. To see the current status, use your GitHub credentials to log in to Jenkins here:
We do not want to go through another release process like 0.6.0, where we were unable to use the automated tests for QA. Our CI system should normally be "Green" so that any regressions show up as an obvious problem.
I would like to start a project to clean up all of the live and integration tests and get things back to "Green" on our Jenkins box. The goal is to have all the enabled live and integration tests passing, and use this as our acceptance criterion for a 0.7.0 release at the beginning of 2014.
What I would like to propose is that people volunteer to take on some of the 'software' and 'system' projects and to own them, with their live and integration tests, along with the responsibility for fixing any issues in the tests, test setup or entities.
I have started work on the 'software/database' project entities, but at present with my fixes (in a branch on my fork of Brooklyn) only 30 tests pass out of 59, giving 29 failures. These range from issues with cloud image specifications to test configuration and sometimes actual errors in the entity code.
The fixes are part of my 'feature/postgresql-config' branch which is submitted as pull request #1084 and the live test status with this branch can be seen on run #19 of the 'Brooklyn-Andrew-Live-database' job on Jenkins:
- brooklyncentral/brooklyn#1084
- http://brooklyn.builds.cloudsoftcorp.com/job/Brooklyn-Andrew-Live-database/19/
I am happy to take ownership of the database entities and continue working on the tests until they all pass.
The process I would like to propose is as follows:
-
Work on a branch that you regularly rebase from master, and create a series of private Jenkins jobs that run the live and integration tests for your module against that branch. This will allow you to verify your progress. Also, create another job that runs tests using the 'Fixme' profile.
-
Then, move all the currently failing live tests to a group called 'Dead' and merge that change with master. This will give us a green live test run for all the modules.
-
You can then start fixing the broken tests. When you have tests you want to verify, move them out of 'Dead' and into the 'Fixme' group (on your branch) and run the Jenkins job to test. It is important that we test from Jenkins as well as from our own development machines, as the test there run from a freshly provisioned Linux sever and will catch any dependencies on local setup and configuration.
-
Once you have a series of 'Fixme' tests passing, put them back into the 'Live' group and submit a pull request. We can then verify that a 'Live' profile test run has more tests passing using your branch, and merge those back to master.
-
This way we steadily increase the number of passing tests in the main CI jobs, and any failures will indicate a genuine regression.
The next step is to assign each project an owner, so please reply to the list indicating which module you would like to take on. If we start as soon as possible, we can get to the stage of having a minimal set of passing live and integration tests configured, giving us a green build at the beginning of the new year! I will post regular updates to the list on progress to keep everyone informed.
The projects are:
- software/base
- software/database - @grkvlt
- software/messaging
- software/network
- software/nosql
- software/osgi
- software/webapp
- systems/hadoop
- systems/openshift
- systems/whirr
- sandbox/monitoring
Some of these are much smaller than others, so anyone who does not have a lot of spare time could perhaps choose 'software/network' (since it only contains the BindDnsServer entity) or it may make sense for some people to own more than one.
I will set up the new 'Fixme' profile to use for testing, and I will also be available to help out with getting the various Jenkins jobs created and any other issues that arise during testing.
Any suggestions or questions are welcome on-list, and I will also be available on the '#brooklyn' IRC channel on Monday 16 December, at 16h00 GMT for further discussion.
This is a great chance to get involved and improve the quality of the Brooklyn project!