TestDetta

Validate on your repository

Our evaluation runs on open-source libraries. Your code has its own patterns, so check TestDetta on it before you let it decide what runs. This takes a couple of weeks of normal pull requests and changes nothing for your team.

The idea

Keep running the full suite as the gate. Next to it, ask TestDetta what it would have selected. Every test class that fails in the full run must be in that selection. If one is not, that is a miss, and we want to hear about it.

Steps

  1. Record maps on the default branch. With GitHub Actions, add the record step from the workflow; elsewhere, run td record on every commit to the default branch and keep the maps (other CI systems).
  2. On every pull request, write down the selection. Without changing your test step: 
    td affected --format json > selection.json
  3. Run the full suite as you do today, with a TRX report so failures can be read: 
    dotnet test --logger trx --results-directory TestResults
  4. Compare. For every failed test, its class must be selected. A class is selected when its project's mode is All, or when its project is Filtered and the class is in classes: 
    jq -r '.tests[] | select(.mode == "All") | .project' selection.json      # projects that run completely
jq -r '.tests[] | select(.mode == "Filtered") | .classes[]' selection.json  # classes that run
  5. Keep both files of any run where a failed class was not selected, and send them with the -v log of td affected to support@testdetta.com.

Only failing tests can show a miss, and a couple of weeks may bring few failures. You can add some: on a scratch branch, break a method on purpose (return a wrong value) and check that the test that catches it is selected. Do not merge that branch.

When you are confident

Replace the full test step with td test, or the action's test mode. Keep recording on the default branch: it runs every test there, so anything a selection could ever hide is still caught before release.

How we test it

The same check runs at scale on 41 open-source repositories: members are changed one at a time and every test class that fails must have been selected. Out of 2,941 changes that broke a test, 3 were missed, and each was fixed with a test that fails without the fix. The corpus in numbers.