Limitations

Where TestDetta cannot see something, it selects more. This page lists those places, so you know when a selection is wider than it needs to be, and the known cases where a test can be missed.

Test frameworks and runners

Both dotnet test runners are supported. The runner is read from global.json ("test": { "runner": "Microsoft.Testing.Platform" }, required for xUnit v3 on the .NET 10 SDK).

Each test project is built and run in the directory of the nearest global.json at or above it, so a file kept next to the solution (src/global.json) counts as it does for a developer there. --dry-run prints such commands as (cd src && dotnet test ...).

Recording and selection are tested end to end with xUnit v2 (VSTest), xUnit v3 and TUnit (Microsoft.Testing.Platform), and MSTest and NUnit under both runners, including projects that turn on their framework's own parallelization.

Tests that fail while recording

Test classes that fail while recording are always selected when their project is affected: a failing test stops early, so its coverage is incomplete.

When tests fail and no TRX report says which (NUnit under Microsoft.Testing.Platform without Microsoft.Testing.Extensions.TrxReport), the map is not used and selection falls back to names.

Maps recorded by an earlier version

A map holds what the recorder of its version could see. When a version of TestDetta records something differently in a way the map cannot show, it raises the map format, and maps of an earlier format are ignored with a warning to "record the maps again": their test projects are selected by names, as without a map, which selects more, until the maps are recorded again. The first recording on the default branch after upgrading writes new maps; pull requests that run before it select more than they need to.

Maps recorded by TestDetta 0.1.x are of format 2 and are ignored by later versions. A test that builds a fixture type itself (new StoreFixture()) now keeps the rest of its code in its own class; a format 2 map could credit that code to the fixture's users and miss the test. The recorder now also notes which classes read a type's shape through reflection or a JSON schema (its attributes, properties and base types), so a change to that metadata selects them; a format 2 map has no such record.

Maps recorded by TestDetta 0.2.0 to 0.2.2 are of format 3 and are ignored by later versions. xUnit sets a collection fixture (ICollectionFixture<T>) up for every class of its collection, and a class may depend on what the setup did (a container it started, a connection string it set) without asking for the fixture; the recorder now notes the classes of each collection, so a change to the setup selects them. A format 3 map has no such record and could miss them.

Maps recorded by TestDetta 0.3.0 are of format 4 and are ignored by later versions. xUnit v3 sets an assembly fixture ([assembly: AssemblyFixture(typeof(X))]) up for every class of the test assembly; when X lives in a shared test library rather than in the test assembly, the recorder now credits its setup to every class, as it already did for a fixture declared in the test assembly. A format 4 map counted that setup as code outside every test and could miss the classes that name nothing of it.

Code that runs once per process

Caches, lazy singletons and EF model building run once per test process, and are credited to the first test class that triggers them. Changes to code that ran inside a static constructor or a cache-filling call (ConcurrentDictionary.GetOrAdd, Lazy<T>.Value, LazyInitializer, ConditionalWeakTable, IMemoryCache.GetOrCreate), or that also ran outside every test, are traced by name, and the classes the map saw run that code are selected as well. A value kept in a field, such as a lazy singleton (_x ??= new X()), is credited to every test class that reads the field.

A plugin loaded into the test process stays loaded for the rest of it. A class that scans its types (Assembly.GetTypes and the like) is selected when the plugin changes, and one that reads a type's attributes or properties when that metadata changes, even when an earlier class loaded the plugin. A class that only looks at Type objects an earlier class kept in a static field, such as counting the plugin types it found or checking their names, without loading, scanning or reading their attributes itself, is not: a type added to or removed from the plugin may not select it. The same holds for a class that lists a plugin's types through Module.GetTypes, or looks one up by name with Assembly.GetType(name), and does not read their attributes: neither call is recorded as a scan.

A cache written by hand, such as a Dictionary checked with TryGetValue, or a static flag that makes a check run once (if (s_done == 0) { SelfTest(); s_done = 1; }), can still hide its users: the code counts for the class that happened to run first. In the corpus this hid two changes (NSec's algorithm self-tests). Keep running the full suite on the default branch; recording does that.

State one test class leaves for the next

A test class can behave differently depending on what an earlier class in the same process left behind, such as a global configuration it set and never reset. The map records the path the class took in the recorded order, so a change to the code it would run alone or in another order may not select it. The full run in the same order does not fail it either. In an earlier corpus round this hid one change (NLog's LogManagerTests). Tests that reset shared state in their setup avoid it.

Code a test runs on other threads

Code that runs for a test on a thread its async flow does not reach, such as an in-process test server handling the test's HTTP request, is credited to the test class that started last. When test classes overlap, or the framework prepares the next class before the current one ends, that code can count for another class, and a change to it may not select the class whose request ran it. In the corpus this hid one change (an ASP.NET validation filter behind a TUnit test's request in ManagedCode.Communication).

Project files

Project files are evaluated by MSBuild (one dotnet msbuild call, cached in .testdetta/cache) for references, compiled and embedded items, assembly names and target frameworks, in the default configuration and without a restore.

What the code cannot show

TestDetta sees your source code, your project files and what your tests ran while recording. It cannot see:

Package version changes are seen: a PackageVersion change in Directory.Packages.props runs the projects that reference the package, and a PackageReference change is a change to the project file.

Preprocessor symbols

C# files are parsed with each target framework's real preprocessor symbols, and also with no symbols and with every symbol their #ifs name, so code under any active condition is seen.

Target frameworks

For a test project that targets several frameworks, what is selected depends on the frameworks the tests run for. The same change can select more tests in one run and fewer in another:

So a CI job that runs one framework can skip tests that only another framework's build could fail. If your test projects target several frameworks, run every framework somewhere, for example in a nightly full run.

Skipped tests

Tests skipped while a map was recorded have no coverage. A class whose tests were all skipped is selected on another platform, where it may run, and skipped on the recording platform. Tests skipped for other reasons, such as a missing Docker daemon, are covered only by recording where the tests run.

Source generators

Code written by source generators is not in the source tree. td record reads it from the PDBs of the recorded assemblies (portable or embedded) and keeps the names it mentions in the map, so name tracing sees generated callers. Assemblies built without a portable PDB give nothing.

When the generated code of a test project cannot be read, neither when the map was recorded nor now, a change to the generator is taken to change nothing there, and the project's tests are not selected for it. In the corpus this hid two changes (Mediator's source generator, read as empty for two of its three test projects). A later version is planned to fix this.

F#

F# code is analysed like C#, and so is Visual Basic. F# signature files (.fsi) are not: a change to one runs everything that depends on its project. A change to an inline function is traced by name to its callers, because the compiler copies its body into them and a coverage map credits them instead.