Troubleshooting Concordion Test Failures in Enterprise Java Projects

Details: Category: Testing Frameworks; By Mindful Chase; 08.Aug; Hits: 21

Concordion is a specification-by-example testing framework designed to keep automated acceptance tests readable and synchronized with business requirements. In enterprise projects, Concordion is often integrated with large-scale Java applications, CI/CD pipelines, and domain-driven design initiatives. However, one of the more complex and rarely discussed issues is the silent failure of tests or incomplete execution in large Concordion suites. These problems often emerge only in CI environments or when test suites scale across multiple modules. This article addresses these subtle issues by analyzing the root causes, architectural risks, diagnostic strategies, and best practices for stable and predictable Concordion test execution at scale.

Mindful Chase

Writing Code, Writing Stories

tbd

Experience

tbd

More to Explore

Understanding the Problem

Silent Failures and Partial Test Execution

In complex projects using Concordion, test failures may go unnoticed if:

Concordion test fixtures are misconfigured
JUnit runners are not correctly discovering specifications
Tests execute in a non-deterministic order across CI environments
Output files are silently overwritten or skipped

This results in teams mistakenly believing all tests passed when, in fact, some specs never executed or their results were never captured.

Architectural Implications

Concordion's Convention-over-Configuration Model

Concordion relies on strict naming conventions and directory structures. For example, the fixture class name must match the .html spec file. In microservice or multi-module builds, this structure often breaks due to refactoring, test relocation, or dynamic class loading.

CI/CD Pipeline Interaction

CI pipelines (e.g., Jenkins, GitLab CI) may invoke test runners without full classpath awareness. Combined with conditional test inclusion/exclusion (e.g., JUnit filters), this leads to tests being skipped silently if Concordion specs aren't loaded properly.

Diagnosing the Root Cause

1. Enable Verbose Output in CI Logs

mvn -Dconcordion.output.verbose=true test

This ensures you see all spec executions, including skipped or failed mappings between fixtures and HTML files.

2. Validate Fixture-to-Spec Mapping

@RunWith(ConcordionRunner.class)
public class OrderSummaryTest {}
# Must match /OrderSummary.html in test resources

Mismatches between class and file names prevent the test from executing without throwing an error.

3. Use Concordion Logging Extension

Use the Concordion Logging Extension to log lifecycle events and determine which specs were loaded or ignored.

4. Check for Parallel Execution Artifacts

If tests are executed in parallel without isolating output directories, files may overwrite each other or get skipped:

concordion.output.dir=target/concordion/<thread-name>

Common Pitfalls

1. Incorrect Maven/Gradle Plugin Configuration

Failing to bind the Concordion plugin correctly results in specs not being generated or tests being excluded from the build lifecycle.

2. Static vs Dynamic Classpath Issues

When running from IDE vs CI/CD, classpath differences can cause Concordion to behave differently. Specs may load in IntelliJ but fail on Jenkins.

3. Test Output Overwrites

Concordion writes output per spec. Without proper naming or configuration, parallel builds may overwrite reports or skip file generation entirely.

Step-by-Step Fixes

1. Standardize Fixture Naming and Locations

Ensure the test class and HTML spec match in both name and package path. Example:

src/test/java/specs/customer/OrderTest.java
src/test/resources/specs/customer/OrderTest.html

2. Enable Test Discovery Validation

Implement a test that loads all classes annotated with @RunWith(ConcordionRunner.class) to ensure no fixtures are missing:

Reflections reflections = new Reflections("specs");
Set<Class<?>> testClasses = reflections.getTypesAnnotatedWith(RunWith.class);

3. Isolate Output in CI Environments

Use build-specific directories to avoid overwrites:

-Dconcordion.output.dir=target/concordion/${BUILD_ID}

4. Integrate Concordion Result Validation

After the test phase, parse the output folder to ensure each expected HTML report exists and is non-empty. Fail the build if not.

5. Use a Custom Concordion Command for Test Status Reporting

Define a command extension that logs results centrally or into the CI dashboard in real time.

Best Practices

Document naming and directory conventions clearly across teams
Run Concordion in a controlled test harness with predictable output
Validate fixture-spec mappings during code review and CI pre-flight checks
Include a sanity test that ensures minimum test execution thresholds
Never assume IDE success equals CI success—test both environments

Conclusion

Silent failures in Concordion testing suites stem from its convention-driven model and limited feedback mechanisms. By enforcing strict naming, directory, and CI output isolation standards, teams can eliminate phantom successes and ensure reliable specification testing. Treat Concordion not just as a testing tool, but as a specification contract that must be audited and validated continuously.

FAQs

1. Why do some Concordion specs not execute?

This usually occurs when the Java fixture class name doesn't match the HTML spec file, or if the spec isn't on the classpath.

2. How do I ensure Concordion outputs are generated during CI builds?

Use explicit output directory settings and verify files exist post-build using a CI script or plugin.

3. Can Concordion run in parallel safely?

Yes, but only with isolated output directories. Parallel execution must avoid shared write targets.

4. Is it possible to fail builds when a Concordion test is skipped?

Yes, implement post-test validations that check for missing reports and fail the build if any expected output is absent.

5. How do I debug Concordion tests that work locally but not in CI?

Compare the effective classpath, logging output, and plugin configurations. Differences in environments often lead to inconsistent behavior.

Contact Us