GitHub Actions
Record a map for every commit on the default branch, and let pull requests run only the tests their changes affect. The action fetches the TestDetta package it pins in tool.lock from nuget.org, checks its SHA-512 before running it, and keeps the maps in the Actions cache.
Workflow
on:
push:
branches: [main]
pull_request:
permissions:
contents: read
actions: read # lets pull requests find the newest coverage map in the Actions cache
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7 # pin to a commit SHA
with:
fetch-depth: 0 # TestDetta compares with the merge base
persist-credentials: false
- uses: actions/setup-dotnet@v6 # pin to a commit SHA
with:
global-json-file: global.json # without a global.json: dotnet-version: 10.0.x
# Also the full test run of main; to leave out projects that cannot run here,
# see "Recording only part of the suite" below.
- if: github.event_name == 'push'
uses: TestDetta/action@8750db18099a2af0436788d20dc591e94b550c07 # v0.5.2
with:
mode: record
- if: github.event_name == 'pull_request'
uses: TestDetta/action@8750db18099a2af0436788d20dc591e94b550c07 # v0.5.2
with:
mode: test
dotnet-test-args: --logger trx
Full history is required. In test, record and audit modes the action stops with an error on a shallow clone; shadow mode warns and runs your plain dotnet test instead. Check out with fetch-depth: 0.
How the maps get to pull requests
- Record mode saves the maps in the Actions cache under
testdetta-coverage-<commit>. Pull requests can read caches of their base branch. - A pull request uses the map of its merge base, or the newest one recorded up to
max-map-agefirst-parent commits before it. - Finding that newer-than-exact map lists the repository's caches, which needs
actions: read. Without it, only a map at the merge base is found, and the step says so in a notice. - Without any map,
mode: teststill works, deciding by name only.
When tests fail while recording, the map is still saved before the job fails, so pull requests keep a map to work with.
Merge the workflow before relying on it: the first push to the default branch records the first map. Pull requests whose merge base is older than the first map find none and select by name until they merge or rebase onto the default branch. After upgrading from TestDetta 0.1.x, its maps are of an older format and are ignored with the warning "record the maps again" until the default branch records again; pull requests that run before that select more (see Limitations).
Permissions
| Permission | Why |
|---|---|
contents: read | Check out the repository. |
actions: read | List the Actions caches to find the newest map within max-map-age. |
With persist-credentials: false a private repository cannot be fetched again later in the job; fetch-depth: 0 already brings every branch, so the base branch is there.
Inputs
| Input | Default | Description |
|---|---|---|
mode | test on pull requests (run affected tests), shadow on pull requests while you evaluate TestDetta (run every test and report what the selection would have left out), record on the default branch (record coverage maps), audit on a schedule (break your code one change at a time in a copy and check the selection ran every test that failed). Required. | |
base | the PR's base branch | Branch to compare with in test and shadow modes. Outside pull requests, set it. |
configuration | Debug | Build configuration for building and running the tests. |
projects | all test projects | Record mode: .csproj paths, one per line. |
parallel | 1 | Record mode: test host processes per test project; each still runs its tests one at a time. The suite must tolerate several test processes at once. |
no-build | false | Record mode: true to record the build an earlier step made with the same configuration. |
dotnet-test-args | Test, shadow, audit and record modes: extra dotnet test arguments, separated by spaces. Give the filter and settings your own test job uses: TestDetta keeps them in every run and while recording (see Your test configuration). In shadow mode, name your solution here to run it as one command; with testing-platform projects, build in a step before and add --no-build here (see Shadow mode). | |
max-map-age | 50 | Test, shadow and audit modes: commits a map may be behind the merge base; the changes in between are bridged. 0 uses only an exact map. |
drift-fallback | all | Test, shadow and audit modes: when the changes since an older map include a build file, configuration or a global using, all runs the affected test projects completely; name decides them by name (faster, blind to calls made through configuration or reflection). |
non-code-files | readers | Test, shadow and audit modes: what a changed file in a project folder that is not code (README.md, a JSON file) runs. readers runs only the test classes the coverage map saw read it, when the build does not use the file; project runs its project and every project depending on it completely; see --non-code-files. |
not-build-inputs | Test, shadow and audit modes: non-code files your build does not use, as repository-relative patterns separated by ; (docs/**;*.yaml). Like documentation files, they count as build inputs only when MSBuild builds, embeds or copies them or a target names them; see --not-build-inputs. | |
shadow-analysis-timeout, shadow-rerun-timeout, shadow-rerun-idle-timeout, shadow-base-timeout, shadow-base-idle-timeout, shadow-classify-budget | 10m, 15m, 5m, 30m, 10m, 45m | Shadow mode: the time limits of the analysis and of the reruns that classify failures; see the options of the same names. |
shadow-artifact-suffix | Shadow mode: added to the report artifact's name, for example your matrix values, to tell the legs of a matrix apart at a glance. The name is unique without it. | |
audit-target-broken | 100 | Audit mode: stop once this many changes broke tests. A lower number does not lower what graduation asks for (below). |
audit-max-mutants | no limit | Audit mode: stop after this many changes ran tests. |
audit-budget | 4h | Audit mode: start no change after this long (90s, 15m, 1h30m; none for no limit); a change still running then finishes, so the audit can end after the budget, and the report says by how much. Leave that headroom below the job's timeout-minutes. |
audit-parallel | 1 | Audit mode: changes tested at once, each in its own copy; memory use grows with it. |
github-token | github.token | Test, shadow and audit modes: lists the Actions caches to find the newest map (needs actions: read). |
license | Deprecated and ignored: TestDetta needs no license. Still accepted so existing workflows keep working; the step writes a notice to the log when it is set. Remove it from your workflow. | |
working-directory | . | Any directory inside the repository to analyze. |
setup-dotnet | true | Install the .NET 10 SDK TestDetta runs on. Set to false if the job already has it. |
Shadow mode
To try TestDetta without skipping anything, use mode: shadow on pull requests. Every test your command runs, runs, and the job's result is that run's; the job summary and the report artifact show what the selection would have run, the measured minutes, and any failing class it would have left out. The artifact is named testdetta-shadow-<run>-<attempt>-<job>-…, one per job and matrix leg (shadow-artifact-suffix adds your own part), and is kept for up to 90 days (your repository's artifact retention setting can shorten it). Name your solution in dotnet-test-args to run it as one command, as your own job does; see td test --shadow.
TestDetta itself makes no network call in shadow mode. The Action downloads the TestDetta package from nuget.org (checked against its SHA-512) and lists and restores the maps in your Actions cache, as in test mode. No step before the tests fails the job: if one cannot do its part (a shallow checkout, the download, a missing base branch, an invalid input), or TestDetta ends without a result of the tests, the step runs your plain dotnet test with the same arguments instead, and the job summary says why. A package that does not match its checksum is never run. A failed upload of the report never fails the job either.
- if: github.event_name == 'pull_request'
uses: TestDetta/action@8750db18099a2af0436788d20dc591e94b550c07 # v0.5.2
with:
mode: shadow
dotnet-test-args: Shop.slnx
Test projects on the testing platform (Microsoft.Testing.Platform) need one more step. TestDetta reads the report option such a project takes from its build output, because an option the project does not know would fail its run, and it does not guess it from package references. On a fresh checkout there is no build output yet: the projects then run one by one without a report, and the report names each one with "no test report: the project was not built before the run", so their failures cannot be compared with the selection. Build first, with the same configuration, and pass --no-build:
- if: github.event_name == 'pull_request'
run: dotnet build Shop.slnx -c Debug
- if: github.event_name == 'pull_request'
uses: TestDetta/action@8750db18099a2af0436788d20dc591e94b550c07 # v0.5.2
with:
mode: shadow
configuration: Debug
dotnet-test-args: Shop.slnx --no-build
Audit on a schedule
To check the selection on your own code, use mode: audit on a schedule, for example weekly on the branch record mode records. It breaks members of your code one change at a time in a copy of the working tree, runs every test that can see each change, and checks that the selection ran every test class that failed; see td audit. Your checkout and its build output are only read. The job summary shows the verdict and every miss with its decision chain; the artifact testdetta-audit-<run>-<attempt> keeps the JSON report for up to 90 days. It uses the newest map within max-map-age of the commit it runs on. TestDetta itself makes no network call: the Action downloads the package and finds the maps as in test mode, and the builds in the copy restore packages as yours do. It runs for hours on your runners: give the job a timeout-minutes well above audit-budget, since a change still running when the budget ends finishes first (all its runs, which can take several times the full run of the test projects that load it); the summary then says how long the audit ran past the budget and which changes ran on. A job killed at its timeout reports the change it cut short as stopped.
on:
schedule:
- cron: '0 3 * * 1' # Mondays, 03:00 UTC
workflow_dispatch:
permissions:
contents: read
jobs:
audit:
runs-on: ubuntu-latest
timeout-minutes: 360 # the most a GitHub-hosted runner allows: 2 h of headroom after audit-budget
permissions:
contents: read
actions: read # finds the newest coverage map in the Actions cache
steps:
- uses: actions/checkout@v7 # pin to a commit SHA
with:
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-dotnet@v6 # pin to a commit SHA
with:
global-json-file: global.json # without a global.json: dotnet-version: 10.0.x
- uses: TestDetta/action@8750db18099a2af0436788d20dc591e94b550c07 # v0.5.2
with:
mode: audit
audit-budget: 4h
Shadow mode's graduation asks for one audit run with at least 100 changes that broke tests, none missed, and a worst-case 95% upper bound of the miss rate of at most 3.0%: the worst case counts every change whose runs could not tell (inconclusive) or that was cut short (stopped) as a miss, and the bound is shown rounded up. The summary and the report say in one line whether the run meets this and, if not, why, for example "not yet: 87 changes broke tests, 100 needed" or "no: worst case 4.1% > 3.0% because 5 inconclusive and 0 stopped are counted as misses". A run limited to some members with --member never counts.
Job summary
In test mode, TestDetta writes its selection to the job summary: the changed files and the base it compared with, each test project with whether it ran filtered, completely or not at all and why, and the test results. Paths, class names and reasons from the repository are written as code, so file names in a pull request cannot inject links or mentions into the page.
Recording only part of the suite
Recording runs the whole suite once with coverage, which on the default branch is also your full test run. Leave out projects that cannot run on the runner, such as integration tests that need a database, with projects. Pull requests then decide those projects by name.
If an earlier step already built the solution with the same configuration, set no-build: true. For a suite that tolerates several test processes at once, parallel spreads each project's test classes over that many processes.