Understanding Bazel's Build Model

Determinism and Hermeticity

Bazel's core philosophy is deterministic builds: the same input always yields the same output. It enforces hermeticity by sandboxing build actions and tracking all dependencies explicitly.

Incrementality and Caching

Bazel reuses outputs from previous builds using content-based hashing. Problems arise when rules are misconfigured or tools produce inconsistent outputs, leading to invalid cache hits or misses.

Common Enterprise-Level Issues

1. Non-Deterministic Outputs

Some build tools write timestamps or absolute paths into outputs, violating hermeticity and causing cache invalidation.

jar cf build/libs/myapp.jar *.class

Fix: Strip timestamps or use --normalize options if supported.

2. Action Conflicts

Two targets trying to write to the same output file cause action conflicts.

Error: file already being written by //lib:foo and //lib:bar

Solution: Refactor targets to write to unique paths or consolidate outputs.

3. Remote Cache Inconsistencies

When remote cache is shared across platforms or branches without isolation, corrupted builds may occur.

Use cache keys with platform suffixes or separate instances per major branch.

4. Genrule Side Effects

Genrules can produce side effects outside the declared outputs, leading to unreliable builds.

genrule(
  name = "generate",
  outs = ["out.txt"],
  cmd = "./generate.sh > out.txt; cp out.txt ../somewhere"
)

Fix: Avoid accessing directories outside declared outputs.

Diagnostics and Logging

1. Enable Verbose Logs

Use flags for detailed analysis of build actions and cache keys.

bazel build --experimental_execution_log_file=log.json --sandbox_debug //target

2. Use Bazel Query and cquery

Inspect the dependency graph or rule attributes:

bazel query "deps(//myapp:target)" --output=graph
bazel cquery "//myapp:target" --output=starlark

3. Analyze Output Base

Explore Bazel's output base to diagnose lingering or conflicting build artifacts:

bazel info output_base
ls -l $OUTPUT_BASE/execroot/

Advanced Fixes and Techniques

1. Enforce Output Consistency

Use bazel-diff or third-party reproducibility tools to detect output drift.

2. Declare All Inputs and Tools

Ensure all tools used in genrules are declared in tools or inputs:

genrule(
  name = "compile",
  srcs = ["schema.json"],
  tools = ["//tools:jsonc"],
  outs = ["output.h"],
  cmd = "$(location //tools:jsonc) $(SRCS) > $@"
)

3. Fix External Dependency Flakiness

Lock external repositories with commit hashes and mirror URLs to avoid resolution failures.

http_archive(
  name = "zlib",
  urls = ["https://mirror.example.com/zlib.tar.gz"],
  sha256 = "..."
)

Remote Execution Considerations

1. Platform Mismatch

Set execution platform flags to avoid cache poisoning across incompatible OS or architectures.

--platforms=@local_config_platform//:host

2. Missing Toolchain Declarations

Remote builds fail when toolchains are implicitly assumed. Use toolchains attribute explicitly in rules.

3. Monitor Execution Logs

Use remote build event protocol (BEP) logs to visualize action execution times and remote cache hit/miss ratios.

Best Practices

• Write hermetic rules with declared inputs, outputs, and tools.
• Isolate remote cache per branch or platform.
• Use bazel analyze-profile to tune CI builds.
• Enforce strict visibility to prevent dependency sprawl.
• Regularly audit genrules and legacy macro behaviors.

Conclusion

Bazel's high-performance and deterministic builds are invaluable for scaling large codebases. However, its power comes with complexity. Subtle misconfigurations can cascade into opaque failures or performance degradation. By adopting hermetic design, strict dependency declaration, and vigilant remote execution hygiene, teams can harness Bazel effectively in enterprise CI/CD workflows and maintain long-term build reliability.

FAQs

1. Why are my builds flaky even with remote caching enabled?

Likely due to non-hermetic rules or mismatched platforms. Ensure output consistency and platform tagging.

2. How do I debug cache misses in Bazel?

Use --experimental_execution_log_file and bazel aquery to trace input digests and cache keys.

3. Can I use Bazel with Docker for full isolation?

Yes. Use rules_docker or run actions in containerized toolchains for reproducibility and OS-level isolation.

4. How do I handle external dependencies safely?

Pin versions with commit hashes, provide mirror URLs, and verify checksums to avoid repo flakiness.

5. What is the best way to track build performance regressions?

Use bazel analyze-profile and remote build event protocol logs to compare build time deltas over time.