For us this turned out to be that we had cached html generated by the rails 3 app that had the ASCII-8BIT encoding, clearing all html caches fixed this.
To arrive at this, we first added this debug code:
# if you reuse this: ensure it matches the method in your version of rails
# as the method is overwritten by this monkeypatch
# config/initializers/debug_encoding_errors.rb
ActiveSupport::SafeBuffer