Understanding Bazel's Build Model

Hermetic Builds and Sandboxing

Bazel ensures that build actions are hermetic—meaning they depend only on declared inputs. It runs actions in sandboxed environments to guarantee consistency. While this increases reliability, any undeclared dependency leads to confusing runtime errors or missing outputs.

Dependency Graph (Action Graph)

Bazel constructs a directed acyclic graph (DAG) of actions and targets. Any cyclic dependency or incorrect labeling can cause invalidations, excessive rebuilds, or build failures without obvious root causes.

Common Troubleshooting Scenarios

1. Persistent Rebuilds Despite No Code Changes

This often results from non-hermetic tools (e.g., embedded timestamps) or incorrectly configured outputs that Bazel treats as changed on every build.

2. Misleading Cache Behavior

Bazel's remote or local cache can serve outdated artifacts if file digests or metadata are incorrectly computed due to misdeclared inputs.

3. External Repository Fetch Failures

When using http_archive or git_repository, network issues or checksum mismatches can break reproducibility across environments or CI agents.

4. Language Toolchain Mismatch

In polyglot projects (Java, Python, Go), Bazel requires explicit toolchain configuration. Improper toolchain resolution causes non-obvious errors like ABI mismatches or broken macros.

Diagnostic Techniques

Inspecting Action Graphs

bazel aquery 'mnemonic("CppCompile", //path/to:target)'

This reveals low-level action metadata, helping diagnose input/output mismatches and over-triggered rules.

Tracking File Hash Changes

bazel build --experimental_generate_json_trace_profile=trace.json //my/target

Use this to visualize changes across builds and identify why Bazel reruns actions even when source files remain unchanged.

Debugging Stuck or Hanging Builds

Enable verbose logging and sandbox debug:

bazel build --sandbox_debug --verbose_failures --explain=explain.log //target

The explain.log file contains why actions were invalidated.

Architectural Pitfalls

Non-Deterministic Build Rules

Custom rules that generate outputs with timestamps or dynamic content cause cache misses and rebuilds. All rule outputs must be deterministic and reproducible.

Misconfigured Visibility and Labels

Incorrect visibility declarations or inconsistent use of alias() targets can break transitive dependencies and build graph resolution.

Toolchain Ambiguity

Without explicit toolchain_type configuration, Bazel may resolve incompatible compilers, causing subtle runtime issues in cross-platform builds.

Step-by-Step Fix Guide

1. Enforce Hermetic Outputs

Audit all custom genrules and scripts for non-hermetic operations. Replace dynamic inputs (like timestamps or uname) with fixed values.

2. Declare All Inputs Explicitly

genrule(
    name = "generate_foo",
    srcs = ["input.txt"],
    outs = ["output.txt"],
    cmd = "cat $(SRCS) > $(OUTS)",
)

Never rely on implicit file access or environment variables without declaring them via srcs or tools.

3. Use bazel clean --expunge Selectively

This clears all caches but should be used only after significant build rule changes. Overuse leads to long rebuild times and undermines cache efficiency.

4. Stabilize External Dependencies

http_archive(
    name = "zlib",
    urls = ["https://..."],
    strip_prefix = "zlib-1.2.11",
    sha256 = "abc123...",
)

Always pin versions and verify checksums for external repositories to ensure reproducibility.

5. Validate Toolchains and Targets

bazel query --output=build //my/target | grep toolchain

Ensure correct toolchains are resolved based on platform, CPU, and Bazel configuration flags.

Best Practices

  • Use .bazelrc to enforce consistent build flags across environments.
  • Write deterministic genrules and avoid global temp files.
  • Enable remote cache for distributed CI systems with --remote_cache.
  • Use bazel query and aquery to audit dependencies regularly.
  • Prefer rules_xxx (community rules) over custom logic when available.

Conclusion

Bazel excels at reproducible, scalable builds—but only when build rules, inputs, and toolchains are tightly controlled. Common missteps around non-hermetic rules, cache misuse, or incorrect dependency declarations can sabotage CI pipelines. With disciplined diagnostics, explicit declarations, and platform-aware configuration, Bazel becomes a highly reliable tool for managing enterprise-grade build systems.

FAQs

1. Why does Bazel rebuild targets even when nothing changed?

Likely due to non-hermetic outputs (e.g., generated files with timestamps) or undeclared dependencies triggering invalidation.

2. How do I inspect why a target was rebuilt?

Use --explain with a log file to capture and analyze build action invalidation reasons.

3. What is the best way to manage external dependencies in Bazel?

Use http_archive or git_repository with version pinning and checksums to ensure reproducibility across machines and CI.

4. Can I mix Bazel with traditional build tools?

Yes, but integration is complex. Use rules_foreign_cc or rules_jvm_external to wrap native and Maven artifacts cleanly.

5. How can I debug flaky builds in Bazel?

Use bazel aquery and sandbox logs to trace non-deterministic behavior or dependency errors. Rebuild with verbose logs enabled for full traceability.