e2e test improvements

e2e_test_improvements.md

First of all, we can extract the current TestCase implementation into an interface:

type Test interface {
	Test(t *testing.T, ctx *TestingContext)
}

then, we can define a few different interfaces that a test can implement to provide additional information that can be used to determine if a test should be executed:

// we can define a set of categories
type Category string
// an interface to provide category info
type CategorisedTest interface {
	Category() Category
}

// we can define a list of environments
type Environment string
// List the environments the test should be running against.
// If the current environment is in the list, it will run the test
type RequireEnvironmentTest interface {
	RequireEnvironments() []Environment
}

// List the environments the test should not be running against
// If the current environment is in the list, it will not run the test
type ExcludeEnvironmentTest interface {
	ExcludeEnvironments() []Environment
}

// We can specify a list of profiles: managed vs. unmanaged
type Profile string

// List the profiles the test can support
// If the current profile is in the list, the test will run
type ForProfileTest interface {
	ForProfile() []Profile
}

Obviously more interfaces can be defined to if additional info is needed.

Each test can implement any one of a combination of the above interfances. Here is an example:

type CRDExistsTest struct {
	Description
}

func (t *CRDExistsTest) Test(t *testing.T, ctx *TestingContext) {
	// run test to check if CRD exists
	TestIntegreatlyCRDExists(t, ctx)
}

func (t *CRDExistsTest) Category() Category {
	return Category.Installation //as an example
}

func (t *CRDExistsTest) ExcludeEnvironments []Environment {
	return []Environment{Environment.OSD} //as an example
}

test := CRDExistsTest{"Verify RHMI CRD Exists"}

Then in the main test, we just need to filter the test cases based on the information. So we can define a few filters:

type TestFilter interface {
	Filter(input []Test) []Test
}

// Filter based on test categories
type CategorisedTestFilter struct {
	Categories []Category
}

func (c *CategorisedTestFilter) Filter(input []Test) []Test {
	out := []Test{}
	for _, t := range input {
		if _, ok := t.(CategorisedTest); ok {
			ct := t.(CategorisedTest)
			// check if c.Categories has ct.Category(), and if that's true
			out = append(out, ct.(Test))
		}
    // if not ok, add it to out as well
	}
	return out
}

// Filter based on current environment
type EnvironmentFilter struct {

}

func (e *EnvironmentFilter) Filter(input []Test) []Test {
	// filter tests based on environments
	return nil
}

and finally we can filter the tests use something like this:

// Filter tests to decide which tests should be running based on command line parameters or environment variables
func filterTests(input []Test) []Test {
	//retrieve categories values from either the command parameters or via environment variables
	f1 := &CategorisedTestFilter{}
	//retrieve environment values from either the command line parameters or via environment variables
	f2 := &EnvironmentFilter{}
	filters := []TestFilter{f1, f2}
	out := input
	for _, f := range filters {
		in := out
		out = f.Filter(in)
	}
	return out
}

There are few benefits with this approach:

1. Each test can define their own conditions to run without changing the main test
2. No need to change any of the existing test if the test doesn't requirement any conditons

yeah, I thinks this makes sense.