Helpful Settings When Running RSpec with parallel_tests

This article describes a few settings that are useful when running tests with parallel_tests, especially for reproducing and investigating failures that occur during CI runs.

`parallel_tests`

In CI environments, we often run RSpec in parallel to speed things up.

For that, there’s a gem called parallel_tests.

It’s very handy for utilizing multi-core CPUs efficiently.

While the README covers setup well enough, debugging becomes trickier when parallel execution introduces flaky tests.

In this post, I’ll share a few TIPS for handling flaky tests when running RSpec with parallel_tests in CI.

TIPS

Fix the seed across all processes

Using RSpec’s seed is helpful for reproducing failures with the same test order.

When running tests in parallel, each process gets its own seed value.

For debugging, though, it’s more efficient if all processes share the same seed.

This doesn’t mean always running with a fixed seed — rather, you generate a random number once and pass it explicitly to all processes.

Without --seed, RSpec decides the seed internally here:

https://github.com/rspec/rspec-core/blob/v3.13.2/lib/rspec/core/ordering.rb#L147-L152

It uses rand(0xFFFF). So, by passing the same value via --seed, you can align all processes.

Example:

$ ruby -e "puts rand(0xffff)"
22357
$ ruby -e "puts rand(0xffff)"
7574
$ ruby -e "puts rand(0xffff)"
11717

Then run your tests like this:

$ bundle exec parallel_rspec -- --seed $(ruby -e "puts rand(0xFFFF)") -- spec/

For more about using --seed to deal with flaky tests in RSpec, I wrote another article:
https://dev.to/hamajyotan/tame-your-flaky-rspec-tests-by-fixing-the-seed-ffl

See which files each process is responsible for

When you run parallel_rspec, test files are distributed across processes.
However, by default, you can’t tell which process is handling which files from the output.

To make this visible, you can use RSpec’s before(:suite) hook:

# spec/rails_helper.rb
RSpec.configure do |config|
  config.before(:suite) do
    files = config.files_to_run
    normalized = files.map { Pathname.new(File.absolute_path(it)).relative_path_from(Rails.root) }
    banner = "PID (#{Process.pid}) #{normalized.count} files to run:"
    puts [banner, *normalized].join("\n\t")

    # ...
  end

  # ...
end

Docs:
https://github.com/rspec/rspec-core/blob/v3.13.2/lib/rspec/core/configuration.rb#L1094-L1098

Now each process will output something like:

PID (1075695) 5 files to run:
    spec/controllers/bars_controller_spec.rb
    spec/controllers/foos_controller_spec.rb
    spec/models/bar_spec.rb
    spec/models/baz_spec.rb
    spec/models/foo_spec.rb

PID (1075696) 4 files to run:
    spec/controllers/bazs_controller_spec.rb
    spec/controllers/hoges_controller_spec.rb
    spec/controllers/root_controller_spec.rb
    spec/models/hoge_spec.rb

See the execution order of files in each process

Sometimes failures depend on the execution order of test files.
With the previous TIPS, you know which seed and which file set was assigned to a process, but not the actual execution order.

Suppose CI shows this failure:

PID (1075695) 5 files to run:
    spec/controllers/bars_controller_spec.rb
    spec/controllers/foos_controller_spec.rb
    spec/models/bar_spec.rb
    spec/models/baz_spec.rb
    spec/models/foo_spec.rb

Randomized with seed 54242

You could reproduce it locally like this:

bundle exec rspec --seed 54242 \
    spec/controllers/bars_controller_spec.rb \
    spec/controllers/foos_controller_spec.rb \
    spec/models/bar_spec.rb \
    spec/models/baz_spec.rb \
    spec/models/foo_spec.rb

Of course, this works fine, but what if the CI execution log looked like the following?
(Here I’m using the progress format)

...F...........

Somehow, you can guess that spec/models/foo_spec.rb was executed relatively early.
Even if your future self knows that this file was actually the second one to run, at this point you’d still need to include all five files in order to reproduce the failure.
(Of course, there’s always the “cheat” of relying on an experienced developer’s intuition…)

That’s why, if you can see the actual execution order of the test files, you can streamline the reproduction steps.
Let’s update the earlier code as follows:

 RSpec.configure do |config|
   config.before(:suite) do
-    files = config.files_to_run
+    files = config.world.ordered_example_groups.map { it.file_path }
     normalized = files.map { Pathname.new(File.absolute_path(it)).relative_path_from(Rails.root) }
     banner = "PID (#{Process.pid}) #{normalized.count} files to run:"
     puts [banner, *normalized].join("\n\t")

     # ...
   end

   # ...
 end

After making the above adjustments, the output will appear as follows.
The order of the files has changed, hasn’t it?

PID (1075695) 5 files to run:
    spec/controllers/foos_controller_spec.rb
    spec/models/foo_spec.rb
    spec/models/bar_spec.rb
    spec/models/baz_spec.rb
    spec/controllers/foos_controller_spec.rb

Then, RSpec proceeds with the tests in the order they are output here.
Also, since the failing test was in spec/models/foo_spec.rb, you don’t need to run anything after that to reproduce the issue.
This shows that the following is sufficient for local reproduction. The reproduction steps are significantly streamlined!
This saves a lot of time.

bundle exec rspec --seed 54242 \
    spec/controllers/foos_controller_spec.rb \
    spec/models/foo_spec.rb

⚠️ Note: Both world and ordered_example_groups are private APIs:

So be aware they may change without notice.

Conclusion

parallel_tests is excellent for speeding up CI, but its parallelism often makes debugging flaky tests more challenging.
With the TIPS outlined above, reproducing and investigating such failures becomes much more manageable.

Even if your test results look messy at first, with the right setup you can methodically trace the root cause — and keep your CI pipeline stable.

Source link