Can test-prof run in parallel?

[WaKeMaTTa]

Can test-prof run in parallel using gem flatware or gem parallel_tests ?

[palkan]

Never tried myself, but as I see these gems run subprocesses/workers and distribute tests among them.

Thus test-prof should work but only in the context of the current worker (i.e. profiling results won't be aggregated–each worker will generate its report).

If you want to generate file-based reports (factory flamegraphs, tag prof charts or stackprof/rubyprof reports) you should use different output_dir-s for each worker:

TestProf.configure do |config|
  config.output_dir = "tmp/test_prof_#{ENV['TEST_ENV_NUMBER']}"
end

[WaKeMaTTa]

Interesting 🤔

I will experiment and give feedback here.

[WaKeMaTTa]

I tried to use FPROF=1 with bundle exec flatware rspec and i get this error. I think is because @stack is shared between the parallel tests.

An error occurred in an `after(:context)` hook.
Failure/Error: @stack.pop.rollback

NoMethodError:
  undefined method `rollback' for nil:NilClass
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/transaction.rb:179:in `rollback_transaction'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract/database_statements.rb:221:in `rollback_transaction'
# /home/runner/project/vendor/bundle/ruby/2.5.0/bundler/gems/test-prof-846bada07a92/lib/test_prof/before_all/adapters/active_record.rb:19:in `rollback_transaction'
# /home/runner/project/vendor/bundle/ruby/2.5.0/bundler/gems/test-prof-846bada07a92/lib/test_prof/before_all.rb:25:in `rollback_transaction'
# /home/runner/project/vendor/bundle/ruby/2.5.0/bundler/gems/test-prof-846bada07a92/lib/test_prof/recipes/rspec/before_all.rb:22:in `block in before_all'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:366:in `instance_exec'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:366:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:507:in `block in run_owned_hooks_for'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:506:in `each'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:506:in `run_owned_hooks_for'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:459:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:574:in `block in run_after_context_hooks'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/memoized_helpers.rb:186:in `block in isolate_for_context_hook'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/memoized_helpers.rb:182:in `instance_exec'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/memoized_helpers.rb:182:in `isolate_for_context_hook'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:573:in `run_after_context_hooks'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:600:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `block (3 levels) in run_specs'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `map'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `block (2 levels) in run_specs'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/configuration.rb:1894:in `with_suite_hooks'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:113:in `block in run_specs'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/reporter.rb:79:in `report'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:112:in `run_specs'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:87:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:71:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/rspec.rb:25:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:38:in `block in listen'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:32:in `loop'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:32:in `listen'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:18:in `block (2 levels) in spawn'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:15:in `fork'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:15:in `block in spawn'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:14:in `times'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/lib/flatware/worker.rb:14:in `spawn'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-rspec-1.0.0/lib/flatware/rspec/cli.rb:12:in `rspec'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
# /home/runner/project/vendor/bundle/ruby/2.5.0/gems/flatware-1.0.0/bin/flatware:6:in `<top (required)>'
# /home/runner/project/vendor/bundle/ruby/2.5.0/bin/flatware:23:in `load'
# /home/runner/project/vendor/bundle/ruby/2.5.0/bin/flatware:23:in `<top (required)>'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli/exec.rb:74:in `load'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli/exec.rb:74:in `kernel_load'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli/exec.rb:28:in `run'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli.rb:463:in `exec'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli.rb:27:in `dispatch'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/cli.rb:18:in `start'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/exe/bundle:30:in `block in <top (required)>'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
# /home/runner/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/exe/bundle:22:in `<top (required)>'
# /home/runner/.rbenv/versions/2.5.1/bin/bundle:23:in `load'
# /home/runner/.rbenv/versions/2.5.1/bin/bundle:23:in `<main>'
# 
#   Showing full backtrace because every line was filtered out.
#   See docs for RSpec::Configuration#backtrace_exclusion_patterns and
#   RSpec::Configuration#backtrace_inclusion_patterns for more information.

[palkan]

think is because @stack is shared between the parallel tests

@stack is from ActiveRecord internals; it's thread-safe (and actually, we using processes here, so it shouldn't be a problem).

Looks like transaction is rollbacked somewhere else (in the test itself or in the application code?).

I've pushed a small fix to master to check for open transactions. Could you try it?

Also, it could be helpful to turn verbose ActiveRecord logging (https://test-prof.evilmartians.io/#/logging) to see what's going on with DB.

[WaKeMaTTa]

It works. Thank you.

---

I'm using a machine with 4 CPUs. The build was 20 min.

Now is 7 min using flatware. 🆒

[palkan]

It works. Thank you.

Cool! Do you see any warning during the test run?

[WaKeMaTTa]

No, I don't see warnings

[zauzaj]

We're using this for the last couple of months.
test-prof + parallel_tests = winnig combination

[palkan]

@zauzaj Could you tell a bit more about this integration? That would be great if we add an example to the documentation.

[zauzaj]

@palkan Yes sure I will take some time to make PR with some details if you agree with that?

[palkan]

Sounds good. Thanks!

[XenosRbel]

Hi, how to you run and configure test-prof with parallel-test ?

[palkan]

I believe, most functionality should work out-of-the-box; the only caveat is that reports would be generated per-process.

So, profiling with multiple processes would be tricky; l'd recommend to use profilers without parallelization.