Feedback loop speed in one of the biggest contributing factors to overall development time. The faster you get results, the faster you can move on to other things. A fast enough test suite is therefore critical to teams' success, and is worth investing some time at the beginning to save in the long run.
Below is a list of techniques for speeding up a Rails test suite. It is not comprehensive, but should definitely provide some quick wins. This list of techniques assumes you're using minitest, but most everything should translate over to rspec by simply replacing test/test_helper.rb with spec/spec_helper.rb.