Success stories

[palkan]

This thread (not a an issue at all) aims to collect the stories of detecting and fixing test suites bottlenecks. Not necessary related to TestProf, btw.

That would help others to improve their tests and would help me to make the TestProf gem better.

Please, comment on this issue to share your story!

[palkan]

And excerpt from Appaloosa Store blog:

We noticed that we were creating way too much records with FactoryGirl ...

So we started removing these associations, set by hands inside test or turn them into traits:

factory :authorization do |auth|
  auth.trait :with_user do |auth|
    auth.user { create :user }
  end
end

We did that for most of our factories. This saved around 4–6min on the all test suite. Some factories remain unchanged but it can be tricky when they are used in tens of tests.

/cc @benoittgt

[benoittgt]

Yep! I confirm. And we still have some "leaky" factories. By default, factories should not have (in database) association created.

[lcguida]

By detecting some factories' records creation with testprof and FactoryBot I could determine that we were creating a lot of records unecesary. To make it worse, those records (which are central records in our app) were creating a lot of dependencies trough callbacks.

I found a single test that was really using those dependecies.

So I overwrote the factory instance callback method to do nothing like this:

FactoryGirl.define do

  factory :my_model do
    # factory fields

   after(:build) do |obj|
     def obj.callback_method; end
   end

  end
end

Since we create this my_model a lot, this reduced my tests time in 30%. That's not bad for 3 lines of code.

[dgilperez]

After checking some specs with Factory Profiler and StackProf, and then using a combination of FactoryDefault, AnyFixture and let_it_be, apart from other strategies, we just reduced times in one folder of particularly slow specs, from 25 minutes to 16 seconds. Yes, you read well.

And we are just starting with the refactor.

I. Literally. Love. You.

Pattern we're following:

require 'test_prof/any_fixture/dsl'

# Activate AnyFixture DSL (fixture) through refinements
using TestProf::AnyFixture::DSL

RSpec.shared_context 'Shared context for Bla', fixtures: :bla do
  before(:all) do
    fixture(:foo_f) { create :foo }
    fixture(:bar_f) { create :bar }
  end
  
  before do 
    set_factory_default :foo, foo # just this one, the other is not needed as a default
  end

  let_it_be(:foo) { fixture(:foo_f) }
  let_it_be(:bar) { fixture(:bar_f) }
end

Currently thinking about just dropping the let_it_be in favour of a let. Makes sense?

[palkan]

@dgilperez

from 25 minutes to 16 seconds

Wow! 🙀

let_it_be(:foo) { fixture(:foo_f) }
let_it_be(:bar) { fixture(:bar_f) }

Using let_it_be with fixture doesn't make a lot of sense, since the value is already cached globally.

Using let in this case would be even faster (a little bit).

[palkan]

Discourse + let_it_be (https://twitter.com/samsaffron/status/1125602558024699904)

our spec suite at @discourse and our 11146 spec suite now takes 6:06s down for 7:10s

See also discourse/discourse#7414

[palkan]

Cut Your RSpec/Minitest Runtime With TestProf
https://medium.com/better-programming/cut-your-rspec-minitest-runtime-with-testprof-d19e55783050

[pirj]

We've cut our spec run time on my previous project four-fold.

Part of the analysis was made with the help of the awesome factory profiler. This research gave birth to a monkey patch @DNNX authored that helped to reuse models created by factories with complex interdependencies. @DNNX would you be up to open source that brilliant and flawless piece of code, and maybe contribute it to test-prof?

At some point, under the pressure of getting specs under a given execution time, let_it_be became a widely used tool, which helped to reduce the run time of many many specs, with minimal effort and with immediate results. We counted thousands of let_it_bes in our specs.

[DNNX]

This is the patch (two files):

module Monkey
  # This module is an extension to `FactoryBot::FactoryRunner` which provides
  # a cached version of the `#run` method.
  module FactoryRunnerWithCaching
    # `RSpecExtension` module is designed to be prepended to RSpec's config,
    # so that if one calls `create` from a spec directly, the `create` call
    # would reuse any associated records.
    module RSpecExtension
      def create(*)
        ::FactoryBotAssociationCache.run_within { super }
      end

      def build(*)
        ::FactoryBotAssociationCache.run_within { super }
      end

      def build_stubbed(*)
        ::FactoryBotAssociationCache.run_within { super }
      end
    end

    # Runs a factory - either "creating", "building" or "build_stubbing" a new object.
    # Most of the implementation is copied from `FactoryBot::Runner#run`, but
    # caching support is added at the end of the method, so that instead of
    # going through the regular way of constructing the whole object and its
    # associations.
    def run(runner_strategy = @strategy, &block)
      FactoryBotAssociationCache.instance.fetch(
        name: @name,
        strategy: runner_strategy,
        traits: @traits,
        overrides: @overrides
      ) { super }
    end
  end
end

::FactoryBot::FactoryRunner.prepend(::Monkey::FactoryRunnerWithCaching)

# Caches factory bot objects
class FactoryBotAssociationCache
  def self.run_within
    instance.enable!
    yield
  ensure
    instance.disable!
  end

  def self.instance
    Thread.current[:factory_bot_association_cache] ||= new
  end

  def initialize
    @cache = {}
    @enabled = false
  end

  def enable!
    @enabled = true
  end

  def disable!
    @enabled = false
    @cache.clear
  end

  def enabled?
    @enabled
  end

  # Fetches the factory from the cache by its name, strategy, traits and overrides.
  # Only `create` strategy is cached now.
  # If a set of `overrides` is a _subset_ of overrides of an object in the cache,
  # then this object is considered a match and returned. Example.
  #
  # ```
  # FactoryBotAssociationCache.run_within do
  #   a = create(:user, full_name: 'A', status: :active) # adds talent to the cache, as well as its associations
  #   b = create(:user, full_name: 'A')                  # returns previously cached `a`
  #   c = create(:user)                                  # returns previously cached `a`
  #   d = create(:user, :active)                         # creates a new object in the cache because traits are different
  #   e = create(:user, full_name: 'E')                  # creates a new object in the cache
  # @param name [Symbol] factory name, e.g. `:user` or `:post`
  # @param strategy [Symbol] strategy name, e.g. `:create` or `:build_stubbed`
  # @param traits [Array<Symbol>|nil] trait names, e.g. `[:active, :extended]`
  # @param overrides [Hash] attribute overrides, e.g. `{full_name: 'John Doe', status: :active}`
  # @yieldreturn ActiveRecord object which is cached if there was no match in the cache
  # @return ActiveRecord object either from the cache or created by the given block
  def fetch(name:, strategy:, traits:, overrides:)
    return yield unless enabled?
    return yield if traits.include?(:skip_factory_bot_cache)
    return yield unless strategy == :create

    key = [name, strategy, traits.to_set]
    subcache = (@cache[key] ||= {})
    _, candidate = subcache.find { |sub_overrides, _| overrides <= sub_overrides }
    candidate || (subcache[overrides] = yield)
  end
end

Not sure if it's suitable for inclusion to test-prof as it may potentially break tests in subtle ways. It wasn't the case in our situation, but every codebase is different, so I'd use this with caution. Anyway, like @pirj said, it gave tremendous boost to our spec speed without even touching existing tests (with a very few easily fixable exceptions).

[lawrencegripper]

Coming here to say a massive thankyou for the library and it's awesome documentation 🙇

Just used test-prof at work, on our deployment tooling which is a rails project. Specifically RubyProf and RSpecDissect where super useful. With around 3 hours of playing I got the following results:

Before: 287 seconds (4:47)
After: 185 seconds (3:05)

One of the issues highlighted by RubyProf digging was a very inefficient method on the hot path in lots of tests.

Outputting the flat wall clock rubyprof output showed this lovely method being called how many times! (this is a single rspec file)

29.330     0.112     0.000    29.218   145768  *Deployment::Application#initialize

Highly recommend folks take a look at the RubyProf output once they've got wins out of the Let/Before bits.

Cool thing, as well as making tests quicker, this improved our response time in production by a huge chunk too at p95.

[palkan]

Thanks for sharing! A great example of how profiling tests may help in improving production performance.

[palkan]

And thanks for your support!