Validate on your repository

Our evaluation runs on open-source libraries. Your code has its own patterns, so check TestDetta on it before you let it decide what runs. This takes a couple of weeks of normal pull requests and changes nothing for your team. The GitHub Action's shadow mode and audit do these checks for you; the steps below do them by hand with td affected.

The idea

Keep running the full suite as the gate. Next to it, ask TestDetta what it would have selected. Every test class that fails in the full run must be in that selection. If one is not, that is a miss, and we want to hear about it.

Steps

  1. Record maps on the default branch. With GitHub Actions, add the record step from the workflow; elsewhere, run td record on every commit to the default branch and keep the maps (other CI systems).
  2. On every pull request, write down the selection. Without changing your test step:
    td affected --format json > selection.json
  3. Run the full suite as you do today, with a TRX report so failures can be read:
    dotnet test --logger trx --results-directory TestResults
  4. Compare. For every failed test, its class must be selected. A class is selected when its project's mode is All, or when its project is Filtered and the class is in classes:
    jq -r '.tests[] | select(.mode == "All") | .project' selection.json      # projects that run completely
    jq -r '.tests[] | select(.mode == "Filtered") | .classes[]' selection.json  # classes that run
  5. Keep both files of any run where a failed class was not selected, and send them with the -v log of td affected to support@testdetta.com.

Only failing tests can show a miss, and a couple of weeks may bring few failures. You can add some: on a scratch branch, break a method on purpose (return a wrong value) and check that the test that catches it is selected. Do not merge that branch.

When you are confident

Replace the full test step with td test, or the action's test mode. Keep recording on the default branch: it runs every test there, so anything a selection could ever hide is still caught before release.

How we test it

The same check runs at scale on 53 open-source repositories: members are changed one at a time and every test class that fails must have been selected. Out of 3,071 changes that broke a test in the latest round, judged strictly (a crash or hang outside the selection counts as a miss), 5 were missed, each a known limitation: code that runs once or on another thread credited to another class (three), and a source generator whose output was not read (two). Real misses in earlier rounds were fixed with a test that fails without the fix. The corpus in numbers.