Root Cause: Shippable's Container Reuse Model

Understanding Shippable's Job Architecture

Shippable jobs are container-based, and reuse base images for speed. However, if these base images are stale or mutable (i.e., lack content hashes), the pipeline's behavior can vary across runs—even without source code changes.

Common Symptoms

• Pipeline passes locally but fails on Shippable's environment
• Flaky tests when build caches are re-used
• Stuck builds due to stale or invalid Docker layers
• Security scanning fails due to outdated dependencies

Architectural Analysis

Ephemeral vs Persistent Runners

Shippable supports both persistent workers (hosted internally) and ephemeral cloud-based runners. When persistent nodes aren't cleaned, dangling containers or volumes can leak into future builds, breaking idempotency.

Shared Pipeline Resources

Teams often share pipeline YAML templates and container registries across services. Without version-locking or image pinning, this sharing introduces downstream failures triggered by subtle changes to unrelated services.

Diagnostic Approaches

1. Enabling Build Debug Mode

Append shippable_post scripts with verbose logs to capture the build's runtime context. This includes inspecting the filesystem, image layers, and network availability:

jobs:
  - name: ci_build_debug
    type: runSh
    steps:
      - IN: repo_name
      - TASK:
          script:
            - df -h
            - docker images
            - env

2. Audit Cached Layers

Use the Shippable cache viewer or inspect /tmp/cache contents if using on-prem workers. Compare cache timestamps with known commits to identify staleness.

3. Use Immutable Image Digests

Instead of referencing tags like python:3.9, use SHA digests to pin dependencies:

image: python@sha256:ab234f...890

Step-by-Step Fixes

1. Force Clean Builds

Periodically invalidate caches and rebuild from scratch to ensure consistency. Use the UI or Shippable CLI:

shippable reset --build-cache --all-pipelines

2. Pin Docker Layers Explicitly

Update your pipeline YAML to use specific digests instead of floating tags:

jobs:
  - name: myjob
    type: runSh
    image: my-registry/app@sha256:123abc...

3. Prune Worker Nodes

Automate the pruning of Docker containers, volumes, and images on persistent nodes every 24 hours:

docker system prune -af --volumes

4. Validate Environment Checksums

Create checksum baselines for your build environments. Store these in CI logs and compare them during post-merge builds to catch image drift:

md5sum /etc/os-release

Best Practices for Long-Term Stability

• Use ephemeral workers when possible to ensure clean state
• Pin all container images by digest, not tag
• Perform weekly validation of pipeline config vs registry versions
• Centralize caching policies to avoid drift between services
• Log container diffs and prune stale resources regularly

Conclusion

Shippable's CI/CD model provides flexibility, but it requires discipline in image management, cache control, and build environment hygiene. Inconsistent builds often stem from seemingly unrelated architectural decisions—like unpinned containers or shared caches. By incorporating deterministic pipeline practices, stronger version pinning, and periodic clean-ups, teams can restore confidence in Shippable-driven delivery pipelines and reduce pipeline entropy over time.

FAQs

1. Why do Shippable builds pass locally but fail remotely?

Usually due to differences in base images, cache contents, or environment variables that drift between local and cloud environments.

2. Is it safe to run Docker prune in CI environments?

Yes, if properly scoped. Use it in ephemeral or non-production runners and automate it with scheduled tasks to keep environments clean.

3. What's the best way to verify that a Docker image hasn't changed?

Use SHA256 digests and image manifest validation. Avoid using tags alone, especially 'latest', which can point to mutable content.

4. Can Shippable support immutable infrastructure patterns?

Partially. While it lacks native immutability enforcement, you can simulate it with digest-pinned images and strict pipeline gating logic.

5. How can I detect when caches are causing flakiness?

Introduce a parallel cache-free build job and compare outputs. If failures vanish in the clean build, stale caches are likely the issue.