Time saved

Every run says how much test time it saved. The numbers stay in your CI, so adding them up over a month is a few lines of script, not a dashboard you have to trust.

On every run

A coverage map records how long each test class took. Before running anything, TestDetta adds up the recorded time of the whole suite and of the classes it selected. The GitHub job summary, and the Markdown that --summary-file writes for other CI systems, say:

Recorded test time: 22.4 s of 5 m 59 s (93% saved); 7 test project(s) without timings are not counted.

The console prints the same numbers with a bar: Recorded test time 22.4 s instead of 5 m 59 s, then 93% saved. They count test time only. Building and restoring happen either way, except for test projects that are skipped entirely: those are not built at all, which saves more than the line shows.

Projects without timings, for example because no map was recorded for them yet, are left out of both numbers, and the line says how many; the JSON counts them in projectsWithoutTimings.

In JSON

td affected --format json carries the same numbers for scripts, for the whole run and for each test project:

{
  "estimate": {
    "fullSeconds": 358.7,
    "selectedSeconds": 22.4,
    "savedSeconds": 336.3,
    "projectsWithoutTimings": 7
  }
}

Adding it up

Keep the JSON of every pull request run as a build artifact. Summing savedSeconds over any period gives the test time saved, with nothing leaving your CI:

td affected --format json > testdetta.json      # in each pull request run

# later, over the files of a month:
jq -s 'map(.estimate.savedSeconds // 0) | add / 3600' runs/*.json   # hours saved

Multiply by the number of parallel jobs that run the tests (an OS or framework matrix) for runner minutes, and by your runner's price per minute for cost. The calculator below does the same with estimates.

What the numbers mean

Estimating it first

Enter your own numbers. The estimate assumes your test time falls the way it did for the median repository in our evaluation, where a change ran 35% of the test classes. Your repository will differ; measure it on your own code.

82 hof waiting for test results saved per month
Test minutes saved per run
7.8 min
Runner minutes saved per month
4,914
Runner cost saved per month
$29
Developer time saved per month
$4,914

21 working days a month. Only test time is counted: builds run either way.