e2e test improvements

e2e_test_improvements.md

First of all, we can extract the current TestCase implementation into an interface:

type Test interface {
	Test(t *testing.T, ctx *TestingContext)
}

then, we can define a few different interfaces that a test can implement to provide additional information that can be used to determine if a test should be executed:

// we can define a set of categories
type Category string
// an interface to provide category info
type CategorisedTest interface {
	Category() Category
}

// we can define a list of environments
type Environment string
// List the environments the test should be running against.
// If the current environment is in the list, it will run the test
type RequireEnvironmentTest interface {
	RequireEnvironments() []Environment
}

// List the environments the test should not be running against
// If the current environment is in the list, it will not run the test
type ExcludeEnvironmentTest interface {
	ExcludeEnvironments() []Environment
}

// We can specify a list of profiles: managed vs. unmanaged
type Profile string

// List the profiles the test can support
// If the current profile is in the list, the test will run
type ForProfileTest interface {
	ForProfile() []Profile
}

Obviously more interfaces can be defined to if additional info is needed.

Each test can implement any one of a combination of the above interfances. Here is an example:

type CRDExistsTest struct {
	Description
}

func (t *CRDExistsTest) Test(t *testing.T, ctx *TestingContext) {
	// run test to check if CRD exists
	TestIntegreatlyCRDExists(t, ctx)
}

func (t *CRDExistsTest) Category() Category {
	return Category.Installation //as an example
}

func (t *CRDExistsTest) ExcludeEnvironments []Environment {
	return []Environment{Environment.OSD} //as an example
}

test := CRDExistsTest{"Verify RHMI CRD Exists"}

Then in the main test, we just need to filter the test cases based on the information. So we can define a few filters:

type TestFilter interface {
	Filter(input []Test) []Test
}

// Filter based on test categories
type CategorisedTestFilter struct {
	Categories []Category
}

func (c *CategorisedTestFilter) Filter(input []Test) []Test {
	out := []Test{}
	for _, t := range input {
		if _, ok := t.(CategorisedTest); ok {
			ct := t.(CategorisedTest)
			// check if c.Categories has ct.Category(), and if that's true
			out = append(out, ct.(Test))
		}
    // if not ok, add it to out as well
	}
	return out
}

// Filter based on current environment
type EnvironmentFilter struct {

}

func (e *EnvironmentFilter) Filter(input []Test) []Test {
	// filter tests based on environments
	return nil
}

and finally we can filter the tests use something like this:

// Filter tests to decide which tests should be running based on command line parameters or environment variables
func filterTests(input []Test) []Test {
	//retrieve categories values from either the command parameters or via environment variables
	f1 := &CategorisedTestFilter{}
	//retrieve environment values from either the command line parameters or via environment variables
	f2 := &EnvironmentFilter{}
	filters := []TestFilter{f1, f2}
	out := input
	for _, f := range filters {
		in := out
		out = f.Filter(in)
	}
	return out
}

There are few benefits with this approach:

1. Each test can define their own conditions to run without changing the main test
2. No need to change any of the existing test if the test doesn't requirement any conditons

@wei-lee I like the idea but I would like to propose a couple of changes

I think is better if had just one interface for all the tests, because it would make clear that categories, description, id, ... are important information for each test

For simplicity, I would just go for a list of Environments, and we can always add ExcludedEnvironemnts. If no Environments are passed it means it targets all of them.

We should define exactly what should be a profile and what an environment otherwise we risk that this two-term will be exchanged, but I like the idea of having an additional filter.

Alternative we could also just have two functions in the interface, one for executing the test and the second to retrieve the metadata

type TestMetadata struct {
	id           string
	description  string
	category     Category
	environments []Environment
	profiles     []Profile
}


type Test interface {
	Test(t *testing.T, ctx *TestingContext)
        Metadata() TestMetadata
}

If we keep the same metadata structure in the test cases and in the golang tests suite, we will not have to preserve the automated tests in the test cases directory, because we would be able to extract all the information that we need, for the automated tests, from the golang test suite

Wdyt?

yeah, I thinks this makes sense.