Skip to main content
The DevInfra team must ensure that CI remains fast by avoiding any unneeded work. Monitor your cache hit rate and be vigilant about repairing regressions. For example, someone introduces a protoc plugin that stamps a date into the output. This is non-deterministic and means that the files produced will be different on each build, so anything (transitively) depending on them will be a cache miss. Non-determinism higher in the build graph causes more cascading cache misses and should be addressed first.
Remote Build Execution is sometimes sold as a solution, because it spreads the work over many machines, but if the work is unneeded this is the wrong solution because costs are excessive. The best answer is simply “less execution”.
To locate non-determinisms, there are a few approaches:
  1. Run two builds, collecting the --execution_log_binary_file. Ideally these builds are on different machines but at the same VCS snapshot.
  2. Run bazel dump --action_cache and see if the digestKey matches expectations.
  3. Diff the two execution logs from step 1 yourself. The Aspect CLI can read a compact execution log from a task in AXL, so you can compare the two runs on the dimensions that matter:
    • Actions: Changed outputs, timing, or ordering
    • Files: Unexpected file modifications
    • Environment: Non-hermetic environment variables
    • Arguments: Inconsistent build flags
Read more: