Testing Notes

testing_notes.txt

Testing Notes
  Tradeoffs (high-level):
    ref? https://kentcdodds.com/blog/unit-vs-integration-vs-e2e-tests#cost-%EF%BF%A0-heap--
    . Cost: Time to write/maintain/fix/run tests in Dev/Dev
    . Feedback Speed: Longer time to get feedback.
    . Confidence: How confident do we feel that passing tests mean users will actually experience the same behavior in production.
    . Confidence vs Cost/Feedback Speed
  Accessibility concerns to test?
    . Flows, like multiple keyboard navigation steps
    . Screen reader testing
    . color contrast
  Multi-device/browser testing
    . We don't currently do this
    . We might be able to with Selenium Grid
    . Likely works FE unit/component integration tests too
  Test Types?
    . Manual E2E exploratory testing
      + finds gaps in business requirements and acceptance criteria (even at the continuous release cases)
      +/- can test multi-element color contrast, assuming humans can actually tell the difference and know the requirements
    . Manual E2E scenario testing
      + good for one-offs (no regression value)
      + visual styling
      + 95% confidence
      - humans error: we are bad at making 0 mistakes on manual 1000 actions.
      - to slow to ever achieve continuous releases
      + provides a second chance per sprint to think of other scenarios we didn't consider
      + allows testing accessibility interaction (e.g., slider navigation)
      +/- can test multi-element color contrast, assuming humans can actually tell the difference and know the requirements
    . Automated E2E
      + Provides more time for exploratory testing
      +/- Provides plenty time to practice meditation to promote mental stability and decrease anxiety
      + Things that cannot be test by other types of tests
      + 99% confidence in theory representative of actual user experience (given stable env, correct test code, and correct config setup)
      - 80% confident for us now in non-flaky groups
      - 0% confident for us now in flaky gorups
      + Integration tests across front and backend
      - Expensive to run
      - Slow/Expensive to get feedback on
      - Slow/Expensive to write (in Java, with selenium specifically)
      + FE Integration can potentially be toggled to use real data for E2E tests, just by disabling mock service worker.
    . FE Manual Audio tests (same scenarios for E2E)
      - requires OS
      - browsers' behavior differs
      - screen readers' behavior differs
      Are there good automated tools?
    . FE Component Integration tests
      + they let you calculate all the possible combinations of component inputs
      + they let you run many more combinations of component inputs than E2E can
      + with msw, they can allow us to do multi-page flows with 0 load time.
      + allow us to test checkout pages without previous pages
      - requires way to prevent people from testing subcomponents in parent component
      + easy to mock/override by merging objects
      + json objects serializable by the front and backend
      + allow testing multi-component accessibility requirements like H2/H3 nesting for screen readers
      + can cover accessibility flows, like multiple keyboard navigation steps
      - color contrast: axe produces false errors sicne it can't "see" element overlays
    . FE unit tests
      + test specific method in isolation, without rendering
      + fastest of all FE options
      - requires a whole lot of tests for 90+% coverage
      + eases granular debugging on data framworks that lack mature dev tools
      - does not tell you if the function being tested is called in the right location
      - does not tell you if the function being tested is called period
      - value per time spent writing test is low
    . FE Visual snapshot tests
      summary? compare the screen before/after changes
      + test themes
      + test visual regressions
      + color contrast
      + test image compression
      + can catch base styles interaction with cobrand style bugs (when using stylesheets)
      - image storage on dev and CI
      + opportunity to test screenshots of Billy's resting zen face (when not running automated test)
      - browser variance can change rendering
      - browser version variance can change rendering
      - os variance can change rendering
      - you have to lock in os, browser, and version, to do these, so they get out of date quick (or set your pixel variance threshold so loose that the tests yield false passes)
    . FE Code snapshot tests
      summary? render your HTML.  Compare character-by-character against previous render.
      - if non-functional things change, like renaming a class, it will break tests when the visuals don't change.
      - break on almost every kind of refactor
      - lose value because they break so often updating them becomes a habit
    . FE static tests (linters)
      + fast accessibility checking of component attribute accessibility while writing
      - don't test multiple components together, like H2/H3 nesting for screen readers
    . BE Integration tests
    . BE Unit tests
    . BE Static tests (linters)
  Tools?
    E2E
      Test Runner?
        . Selenium (current)
        . Cypress
    FE Integration
      Test runner:
        @web/test-runner
      Test runner:
        @web/test-runner
      Test runner:
        chai
      Stubbing/Mocking funcs:
        Sinon
      Stubbing/Mocking network requests:
        Mock service worker
      Stubbing/Mocking browser location, localstorage, sessionstorage:
        Not necessary in web test runner, since it actually uses a browser.
      Stubbing/Mocking (timers for animations, scrolling, etc):
        Allow instantly finishing timers for fast tests
        Unsure if web test runner supports this, or... what.
        We can monkey patch setTimeout
    FE Unit
      Test runner:
        Karma (runs them in browser)
      Test Framework:
        Mocha (describe(), test(), it())
      Assertion/Expecation Library:
        Chai (expect())
      Stubbing/Mock:
        Sinon
    FE Static
    BE Integration?
      Switchfly Working, but we don't know much.
      Test Runner
        Junit
      Framework
        Junit
      Assertions
        Junit
      Stubs/mocks
        springframework.test.web.servlet.MockMvc (for database transactions)
        springframework.test.web.servlet.ResultActions;
        easymock (internal services)
    BE Unit
      Test runner:
        Switchly Junit
      Test Framework:
        Junit
      Assertion/Expecation Library:
        Hamcrest
        Junit built in
        Testng
      Stubbing/Mock:
        EZMock
    BE Static