This project is discontinued because of technical difficulty, and left here only for reference.

If I were to start this again from scratch, I'd store test-to-code mapping not in git repo in each project, but in some external storage in order not to clutter project git history.

TTNT: Test This, Not That!

Stop running tests which are clearly not affected by the change you introduced in your commit!

Started as a Google Summer of Code 2015 project with mentoring organization Ruby on Rails, with idea based on Aaron Patterson's article "Predicting Test Failures".

Goal of this project

rails/rails has a problem that CI builds take hours to finish. This project aims to solve that problem by making it possible to run only tests related to changes introduced in target commits/branches/PRs.

Terminology

• test-to-code mapping
  • mapping which maps test (file name) to code (file name and line number) executed on that test run for a given commit
  • this will be used to determine tests that are affected by changes in code
• base commit
  • the commit to which the tests you should run will be calculated (e.g. the latest commit of master branch)
  • this commit should have test-to-code mapping
• target commit
  • the commit on which you want to select tests you should run (e.g. HEAD of your feature branch)
  • this commit does not have to have test-to-code mapping

Current Status

This project is still in an early stage and we are experimenting the best approach to solve the problem.

Currently, this program does:

• Generate test-to-code mapping with $ rake ttnt:test:anchor
• Select tests related to the change between base commit and current HEAD, and run the selected tests

Installation

Add this line to your application's Gemfile:

gem 'ttnt'

And then execute:

$ bundle

Or install it yourself as:

$ gem install ttnt

Define Rake tasks

TTNT allows you to define its tasks according to an existing Rake::TestTask object like:

require 'rake/testtask'
require 'ttnt/testtask'

t = Rake::TestTask.new do |t|
  t.libs << 'test'
  t.name = 'task_name'
end

TTNT::TestTask.new(t)

This will define 2 tasks: ttnt:task_name:anchor and ttnt:task_name:run. Usage for those tasks are described later in this document.

You can also instantiate a new TTNT::TestTask object and specify certain options like:

require 'ttnt/testtask'

TTNT::TestTask.new do |t|
  t.code_files = FileList['lib/**/*.rb'] - FileList['lib/vendor/**/*.rb']
  t.test_files = 'test/**/*_test.rb'
end

You can specify the same options as Rake::TestTask. Additionally, there is an option which is specific to TTNT:

• code_files
  • Specifies code files TTNT uses to select tests. Changes in files not listed here do not affect the test selection. Defaults to all files under the directory Rakefile resides.

Requirements

Developed and only tested under ruby version 2.2.2.

Usage

Produce test-to-code mapping for a given commit

If you defined TTNT rake task as described above, you can run following command to produce test-to-code mapping:

$ rake ttnt:my_test_name:anchor

Select tests

If you defined TTNT rake task as described above, you can run following command to run selected tests.

$ rake ttnt:my_test_name:run

Options

You can run test files one by one by setting ISOLATED environment variable:

$ rake ttnt:my_test_name:run ISOLATED=1

With isolated option, you can set FAIL_FAST environment variable to stop running successive tests after a test has failed:

$ rake ttnt:my_test_name:run ISOLATED=1 FAIL_FAST=1

Current Limitations

• Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives)
• Only supports git
• Only supports MiniTest
• Only select test files, not fine-grained test cases
• And a lot more!

This gem can only produce test-to-code mapping "from a single test file to code lines executed" (not fine-grained mapping "from a single test case to code lines executed"). This is due to the way Ruby's coverage library works. Details are covered in my proposal.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/Genki-S/ttnt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

I really ❤️ getting interesting ideas, so please don't hesitate to open a issue to share your ideas for me! Any comment will be valuable especially in this early development stage. I am collecting interesting ideas which I cannot start working on soon here at Trello.

License

The gem is available as open source under the terms of the MIT License.