Rails 8.1 breaks tests if you have MT_NO_PLUGINS set. Which a certain @tenderlove told me to always set!!

SUMMARY.md

Rails 8.1, Minitest plugins, and "0 runs"

Problem statement

After upgrading Rails from 8.0.x to 8.1.x, running system tests like

CI=true bin/rails test:system
CI=true bin/rails test test/system/login_test.rb -v

started reporting

0 runs, 0 assertions, 0 failures, 0 errors, 0 skips

even though the test/system/*_test.rb files existed and previously ran fine on Rails 8.0.3/8.0.4.

Root cause

Between Rails 8.0.4 and 8.1.0, commit b1589fa7d9 ("Let minitest parse ARGV") changed how Rails loads tests:

• In Rails 8.0.x, Rails::TestUnit::Runner.run(argv) directly called load_tests(argv) and then require "active_support/testing/autorun". Rails itself always required the test files, independent of any Minitest plugins.
• In Rails 8.1.x, Runner.run(args) now only:
  • requires active_support/testing/autorun (which wires up Minitest.autorun),
  • sets @@load_test_files = true, and
  • registers an at_exit hook to restore ARGV.

The actual loading of test files was moved into the Rails Minitest plugin (minitest/rails_plugin.rb). That plugin:

• Parses CLI arguments into options[:test_files] in plugin_rails_options, then
• In a final opts.on block, if Rails::TestUnit::Runner.load_test_files is true, calls Rails::TestUnit::Runner.load_tests(options.fetch(:test_files, [])) to require the tests.

At the same time, the environment included:

MT_NO_PLUGINS=1

Minitest's run implementation short‑circuits plugin loading when this env var is set:

def self.run(args = [])
  self.load_plugins unless args.delete("--no-plugins") || ENV["MT_NO_PLUGINS"]
  ...
end

With MT_NO_PLUGINS=1, Minitest.load_plugins never runs, so minitest/rails_plugin.rb is never required, and plugin_rails_options never executes. In Rails 8.1, that means:

• Rails::TestUnit::Runner.load_test_files is true, but
• Runner.load_tests is never called, so no _test.rb files are required, and
• Minitest runs with zero runnables, reporting 0 runs.

This explains why the same environment worked on Rails 8.0.3/8.0.4 (where Runner.run eagerly called load_tests(argv) itself) but fails starting in 8.1.0 (which now depends on the plugin).

Summary / fix

• The regression appears at Rails 8.1.0 and persists in 8.1.1.
• The behavior is not caused by Capybara, Playwright, or other app gems; it comes from the interaction between:
  • Rails 8.1's new test‑loading behavior (Rails::TestUnit::Runner.run + minitest/rails_plugin.rb), and
  • the global environment variable MT_NO_PLUGINS=1.
• When MT_NO_PLUGINS=1 is set, the Rails plugin that loads tests is disabled, so no tests are required and the suite reports 0 runs.

Practical fix for this app: ensure MT_NO_PLUGINS is unset (or empty) when running Rails tests, e.g.

unset MT_NO_PLUGINS
CI=true bin/rails test:system

# or for one-off runs
env -u MT_NO_PLUGINS CI=true bin/rails test test/system/login_test.rb -v

If we ever truly need to run without Minitest plugins, a separate override of Rails::TestUnit::Runner.run (restoring pre‑8.1 behavior by calling load_tests(args) directly) could be added, but for now the simplest and most consistent solution is to avoid exporting MT_NO_PLUGINS into Rails test runs.