Time saved
Every run says how much test time it saved. The numbers stay in your CI, so adding them up over a month is a few lines of script, not a dashboard you have to trust.
On every run
A coverage map records how long each test class took. Before running anything, TestDetta adds up the recorded time of the whole suite and of the classes it selected. The GitHub job summary, and the Markdown that --summary-file writes for other CI systems, say:
Recorded test time: 22.4 s of 5 m 59 s (93% saved); 7 test project(s) without timings are not counted.
The console prints the same numbers with a bar: Recorded test time 22.4 s instead of 5 m 59 s, then 93% saved. They count test time only. Building and restoring happen either way, except for test projects that are skipped entirely: those are not built at all, which saves more than the line shows.
Projects without timings, for example because no map was recorded for them yet, are left out of both numbers, and the line says how many; the JSON counts them in projectsWithoutTimings.
In JSON
td affected --format json carries the same numbers for scripts, for the whole run and for each test project:
{
"estimate": {
"fullSeconds": 358.7,
"selectedSeconds": 22.4,
"savedSeconds": 336.3,
"projectsWithoutTimings": 7
}
}
Adding it up
Keep the JSON of every pull request run as a build artifact. Summing savedSeconds over any period gives the test time saved, with nothing leaving your CI:
td affected --format json > testdetta.json # in each pull request run
# later, over the files of a month:
jq -s 'map(.estimate.savedSeconds // 0) | add / 3600' runs/*.json # hours saved
Multiply by the number of parallel jobs that run the tests (an OS or framework matrix) for runner minutes, and by your runner's price per minute for cost. The calculator below does the same with estimates.
What the numbers mean
- Times come from the recording run, where tests run one at a time with coverage. Absolute times in a normal run are usually shorter; the share saved is what carries over.
- A test project that runs completely, for example after a build file changed, saves nothing, and the line says so.
Estimating it first
Enter your own numbers. The estimate assumes your test time falls the way it did for the median repository in our evaluation, where a change ran 35% of the test classes. Your repository will differ; measure it on your own code.