Convert2rhel unit testing discussion 11-10-2021

gistfile1.txt

* Integration tests
  * DevDocTest
  * Hackathon that made the new CI framework work with testing_farm (and
    Artemis) and pakit.  These are used in Fedora and RHEL
    * https://packit.dev/docs/testing-farm/
  * Developers should be able to write these but now that we have a QE team then
    they can write them
  * We might re-evaluate having the developers write these in the future.
  * Runs the whole conversion:
    * Set up the system
    * Run convert2rhel with certain command line params
    * Specify expected results at the end
  * Does not know about the code, just concerned about inputs and end state

* unit testing
  * Generally thought of as the smallest unit possible.  One function (or
    sometimes one class.)
  * Makes heavy use of mocking to isolate the function
  * Pros:
    * Takes less time to run.
    * If a test fails, you know what function an error is occuring in.

* Functional testing
  * A little more flexible definition than unit tests.  Test a "Functional unit"
  * Pro: Tests that the integration between functions works as well (It will
    error if the API of a function changes but the caller is not updated)

* Agreed:
  * We'll continue to use the term unit testing in directories, talking about
    them, etc but we'll lean more towards writing functional tests.  The
    difference is mainly whether we choose to mock something or call
    dependent functions for real.
  * Rules of thumb for when to mock:
    * By default, do not mock.  Use the tmpdir fixture or other means to setup
      the test to run with the values you want.
    * Use mocks if there is a reason to:
      * Test runs too slow when it does actual work (example: actually
        installing packages)
      * Test is flakey if it talks to an actual service (example: talking to
        a web service that rate limits)
      * Test hits an external service (example: talking to a postgres database)
      * The real code requires something that is prohibitive to setup:
        * Example: would have to use auth tokens and you don't want to set that up for CI
  * Almost every PR should come with new tests
    * Not having a unittest is okay with a few things:
      * main(): rely on integration tests instead
        * Our main function is simple (a set of calls to other functions) so
          there's not much to test.
      * Some functions are so simple (example: one conditional and then call
        a python stdlib function) that we don't need to test them.

* Briefly discussed test organization:
  * When to use parameterization and when to make a separate test function?
    * Discussion: https://github.com/oamg/convert2rhel/pull/357#discussion_r745893426
    * Some sample criteria for choosing to make a new test function instead:
      * Use a new test if it would require adding a conditional to the current
        test's code.
      * Use a new test if it you would have to create different mock objects
        for it in the current test.
      * Use a new test if the output that you check for has to follow
        a different pattern (type, format string, regex, exception, etc) for
        the new parameter.