Understanding How Catch2 Discovers Tests

Header-Only Implications

Catch2 uses static initialization via macros such as TEST_CASE to register test functions into a global registry. This mechanism depends heavily on linker behavior and compilation unit structure. In complex systems, tests may not be linked into the final binary unless explicitly referenced, especially when using shared libraries or LTO (Link Time Optimization).

// Simple Catch2 test case
TEST_CASE("Basic math", "[math]") {
    REQUIRE(1 + 1 == 2);
}

Root Causes of Test Case Discovery Failures

1. Dead-Stripped Object Files

When test definitions reside in object files that are never referenced by main(), some linkers (like ld or lld) may strip these files during optimization, even if they contain Catch2 test cases. As a result, the static initializers never run and tests don't appear in the final binary.

2. Shared Library Loading Issues

If tests are compiled into shared libraries but not explicitly loaded at runtime, their static initializers do not execute. This results in missing test registrations during Catch2's discovery phase. Preloading such shared objects or forcing dynamic loading is necessary.

// Force load with dlopen (POSIX systems)
void* handle = dlopen("./libmytests.so", RTLD_NOW);
if (!handle) {
    std::cerr << "Failed to load test lib: " << dlerror();
}

3. Custom Main() Functions

Catch2 allows users to define custom main() functions using CATCH_CONFIG_RUNNER. However, misconfigurations or incomplete initialization in such custom runners may skip the test registry setup entirely.

#define CATCH_CONFIG_RUNNER
#include "catch2/catch.hpp"

int main(int argc, char* argv[]) {
    return Catch::Session().run(argc, argv);
}

Diagnosing the Problem

Checklist for Diagnosing Missing Tests

• Run with --list-tests and verify expected test cases are registered.
• Check linker map to confirm test object files are included.
• Validate that no linker flags like --gc-sections or /OPT:REF are stripping test symbols.
• Confirm that all shared libraries containing test cases are loaded at runtime.
• Ensure CATCH_CONFIG_MAIN or CATCH_CONFIG_RUNNER is defined in only one translation unit.

Using Verbose Mode

Catch2 supports a verbose mode that prints out test discovery and filtering. Running with --verbosity high can expose skipped tests or malformed patterns.

./tests --list-tests --verbosity high

Architectural and CI Implications

Impact on Test Coverage and Quality Gates

In CI environments, false positives due to skipped tests can lead to undetected regressions. Coverage tools may report inflated metrics because test cases appear to exist but were never executed.

Monolithic vs. Modular Test Binaries

Combining all test cases into a monolithic binary can mitigate static initialization issues, at the cost of slower test runs. Alternatively, modular test suites require runtime checks to verify loading and execution of all modules.

Fixing the Issue

Reliable Linking of Test Code

Ensure that test object files are always linked into the final binary. A common pattern is to define a dummy function in each test object and call it explicitly from a central test registry file to force inclusion.

// In each test file
void force_link_test_math() {}

// In main file
extern void force_link_test_math();
int main(...) {
    force_link_test_math();
    return Catch::Session().run(...);
}

CI Pipeline Safeguards

• Enforce test counts per binary in CI checks.
• Log test discovery explicitly before running tests.
• Fail builds if known test modules are absent or zero tests are detected.

Best Practices

• Never assume test cases are included—validate with --list-tests.
• Avoid overusing LTO or aggressive linker optimizations in test builds.
• Use explicit test registries in large test suites.
• Force-load all shared libraries containing tests in runtime-linked setups.
• Use CATCH_CONFIG_MAIN or CATCH_CONFIG_RUNNER consistently and correctly.

Conclusion

Catch2 is a powerful framework, but its reliance on static initialization and header-only compilation can be a double-edged sword in enterprise-grade systems. Understanding the subtleties of test registration, linking, and runtime behavior is essential to avoid silent failures. By adopting explicit loading patterns, rigorous CI checks, and better visibility into test discovery, development teams can ensure test reliability and avoid hidden regressions in production systems.

FAQs

1. Why do some Catch2 test cases not appear in the binary?

They may be optimized out by the linker if not explicitly referenced or included, especially when aggressive flags like LTO or dead-code elimination are enabled.

2. Can Catch2 tests be compiled into shared libraries?

Yes, but you must ensure those libraries are dynamically loaded at runtime; otherwise, the static initializers won't execute and tests won't register.

3. How do I verify all test modules were loaded?

Use the --list-tests flag and compare the output against expected test suites. Also, log shared object loading explicitly when using dlopen or similar mechanisms.

4. Does Catch2 support parallel test execution?

Not natively. However, tests can be split manually or executed in parallel using custom runners or CI orchestration tools like CTest, Ninja, or custom scripts.

5. What's the best way to catch missing test registration in CI?

Run a sanity check using --list-tests and fail the build if fewer than expected tests are found. Maintain a test manifest to track counts and module presence.