Fixing tests: Quick notes #codeblog

codeblog.md

Had to spend some hours fixing various tests. What could go wrong? Quite a few things I'll capture below:

• CircleCI login problem. To debug these, it would be useful to ssh to the build box, but it hasn't been possible for some weeks. Today I realised it's because the Github project owner is a different account, and since Github doesn't support the same ssh ID for different accounts, I made a custom ssh key, added it to Github, and could then ssh into the Circle box.
• There was a call using Mocha's .any_instance.stubs which was never restored. Some refactoring brought this bug to the open as tests maybe ran in a different order. There doesn't appear to be any way to globally reset all mocks, so I just added .any_instance.unstub to undo this in the same test.
• Another refactor inadvertently removed Sidekiq's "fake" framework, which is now in the superclass setup, so it will always be called. Sidekiq::Worker.clear_all and Sidekiq::Testing.fake! called every time.
• Added error logging to a "Headless controller" class which is used for batch REST calls. Absence of error logging was slowing down diagnosis of Circle bugs.

Along the way, I refactored the main test superclass to isolate all "reset" commands into a single TestResetter module, which is used by regular tests, controller tests, and integration tests. It does a bunch of things, like clearing Redis and Memcached, resetting Timecop, and setting some "initial count" values (so tests can check what was created/deleted)

I should note some of these were time-consuming because the tests worked fine in isolation, a kind of Heisenbug caused by side effects of earlier tests. e.g. with the mock any_instance call, it was breaking completely different tests which worked fine when run on their own. That's why it was useful to add extra logging.