Your remote cache already knows which tests should run. You're just not asking it yet.
[`bazel-diff`](https://github.com/Tinder/bazel-diff) is the tool most teams reach for.
It hashes every target from its rule, its attributes, and the hashes of its inputs
(a Merkle tree), then diffs two revisions. It works, and we've used it. But it's not without its limitations:
it sits one level above the action cache key, which is what actually decides whether an action re-runs. `bazel-diff`
only sees target definitions and source file contents. It never notices that two different inputs can still produce a byte-identical output that's already in the cache.
It overreports.
Lockfiles make this especially obvious, and it isn't just a Python problem. By default, `bazel-diff` treats
each external repo as an opaque blob. Anything that re-runs a repository rule re-hashes
the whole repo, flagging every target under it. Bump one package in `uv.lock` and all of
`@pypi` is dirty. Touch `pnpm-lock.yaml` and `@npm` goes with it. Edit `go.mod` and the
generated Go module repos follow.
The [`--fineGrainedHashExternalRepos`](https://github.com/Tinder/bazel-diff) flag narrows that to individual changed
targets, which helps a lot. Even so, `bazel-diff` is still hashing targets. But the deeper problem remains: you can change a
target's inputs, Bazel rebuilds the action, and the output comes out byte-identical. The action cache would have served it.
`bazel-diff` still flags it.
We wanted to catch that case (target changed, output didn't) by reading the cache directly. That's what pushed us
to build `aspect cache diff`.
Like any analysis tool, `aspect cache diff` overreports by default — there is no cheap way to rule out every false positive without actually building something. It ships two modes: **overreport** (the default) runs nothing; **precise** runs `bazel build` on the changed frontier first and produces an exact answer. Both are described [below](#inverting-the-hits).
Two design constraints: use only a remote cache with no
remote-execution cluster, and read the verdict straight out of Bazel's
[`--remote_grpc_log`](https://bazel.build/reference/command-line-reference#flag--remote_grpc_log),
no `cquery` and no `aquery`.
## Why not just run `bazel test //...`?
Bazel already serves cached test results. Why not just run everything and let it skip what hasn't changed?
A single Bazel server can only parallelize within one machine. [`--jobs`](https://bazel.build/reference/command-line-reference#flag--jobs) parallelizes across local cores,
or across an RBE (Remote Build Execution) cluster if you have one. Without RBE, every affected test runs on one box,
bounded by its core count. You could scale your machine, but you'd get sub-linear returns, have an oversized machine sitting idle for 99% of small-scale builds, and
pay for cores whether you use them or not. You can't run more tests concurrently than one Bazel server can schedule.
The way you scale without RBE is to shard across many CI runners, each running a slice.
To shard intelligently, you need the affected set as data first — a list of labels
you can partition before anything runs. That's what `aspect cache diff` produces, cheaply, without executing anything.
The probe runs on a small coordinator, prints the affected labels, and a scheduler fans
them out. Running `bazel test //...` to completion on one machine is the serialization bottleneck you're trying to break.
## Probe the cache, run nothing
Bazel has a little-known flag, [`--experimental_remote_require_cached`](https://bazel.build/reference/command-line-reference#flag--experimental_remote_require_cached),
that intercepts the remote-execution decision (the moment Bazel is about to dispatch an action to a remote executor).
With it, when an action isn't already in the remote cache, Bazel refuses to run it. It returns `EXECUTION_DENIED`
before scheduling any work, local or remote. Pair it with `--remote_grpc_log` and Bazel writes one `LogEntry` per RPC,
including a `GetActionResult` for every action it checked, tagged with the target label and action mnemonic.
So the probe is a single invocation that executes nothing:
```sh theme={null}
bazel test //... \
--experimental_remote_require_cached \
--remote_grpc_log=probe.binpb \
--keep_going
```
Every action is either a cache hit (served, nothing run) or a miss (denied, nothing run).
Then we check the log.
One problem: with a cache but no executor, the remote-execution decision never fires.
A cache miss falls back to local execution, ungated, and the flag does nothing. So we give
Bazel an executor to gate against. It's an in-process dummy that implements just enough of the
[Remote Execution API](https://github.com/bazelbuild/remote-apis) for Bazel to route
actions through the remote-execution path. `--experimental_remote_require_cached`
intercepts each miss and returns `EXECUTION_DENIED` before anything reaches the dummy.
## Inverting the hits
Under `--keep_going`, a cache miss deep in a test's dependency tree trips require-cached at
that action. The test's own `TestRunner` lookup never gets reached. The flat log
shows the miss frontier, the deepest missed action on each branch, not a clean per-test
verdict.
The trick is inverting the hits instead of chasing the misses:
> A test's `TestRunner` action is a cache hit if and only if its entire transitive closure
> was cached, which means the test is unaffected.
So `affected = tests(//...) − {tests whose TestRunner lookup hit}`. And since every log
entry carries the target label, the tests are self-labeling. No sentinel actions to wire up.
Two modes come out of this:
* **`overreport`** (the default): runs nothing and reverse-deps from each missed target to
its dependent tests. Cheap, but it flags tests that may actually be cached if
their direct inputs haven't changed.
* **`precise`**: first runs `bazel build` on the missing non-test actions to resolve deps that are
merely uncached, then counts a test as affected only if its `TestRunner` action misses, a genuine input change.
There's no blind spot: a `TestRunner` action key already folds in the content
digests of its runfiles, so a data dep that rebuilt to a different output moves the test's key (correctly affected),
and one that rebuilt to a byte-identical output leaves it alone (correctly skipped). Same
transitive-closure guarantee, read off a single action. It builds the changed frontier
but still never runs a test, since `bazel build` can't execute `TestRunner`.
`overreport` is the default because it runs nothing. `precise` has to `bazel build` the
changed frontier first. The coordinator running the probe doesn't have to be small — you can size it up
or point it at RBE to keep build times fast. Tests are often the long tail of CI: wide end-to-end suites
that need to be sharded across many runners to finish in time. The build frontier in `precise` mode
is bounded to a single pass; the false positives `overreport` emits fan out across all your shards.
Reach for `precise` only when a falsely-flagged test costs more to run than the frontier costs to build.
Even in `overreport` mode, `aspect cache diff` handles repository rule changes more precisely than
`bazel-diff`. Where `bazel-diff` marks an entire external repo as dirty when its lockfile changes,
`aspect cache diff` lets Bazel re-evaluate the repository rules and propagates only the actual
downstream effects — touch one package in `uv.lock` and only what was genuinely affected gets flagged,
not all of `@pypi`. In a large monorepo with a single directory change, `overreport` resolves most of
the graph exactly; only the targets blocked by an uncached upstream action are unknowable without
building first.
## What the probe has to get right
Two things have to be true for the probe to produce a correct affected set, and both took us longer to pin
down than we'd like to admit.
**Flag parity between the probe and your baseline.** Local and remote execution compute the same action digest
given the same flags. But any flag that touches an action's environment, [`--action_env`](https://bazel.build/reference/command-line-reference#flag--action_env)
for instance, changes that digest. If the CI run that populates the cache uses a flag the probe doesn't
(or vice versa), every action looks like a miss and the probe reports everything as affected.
The fix is flag hygiene: the run that fills the cache and the run that reads it must see identical flags.
**Defeating two layers of local short-circuiting.** Run the probe twice in the same output base
and the second run reports nothing, because Bazel never queries the remote cache at all.
The Bazel server is long-lived and keeps its [Skyframe](https://bazel.build/reference/skyframe) graph alive
between invocations. On a re-run, Skyframe sees an action's node is already evaluated with unchanged inputs
and returns the memoized result without re-entering the execution path. The remote cache lookup lives inside
that path, so it never fires.
Underneath Skyframe is the on-disk action cache: a persistent record in the output base of
which actions Bazel has run. It survives server restarts. Even on a cold server, if the on-disk
cache says an action is up-to-date, Bazel marks it as done and skips it. Again, no remote lookup.
So we open both gates:
```sh theme={null}
--nokeep_state_after_build --nouse_action_cache
```
[`--nokeep_state_after_build`](https://bazel.build/reference/command-line-reference#flag--keep_state_after_build)
discards the in-memory Skyframe state when the build finishes, so the next probe
re-evaluates from scratch. [`--nouse_action_cache`](https://bazel.build/reference/command-line-reference#flag--use_action_cache)
ignores the on-disk action cache, so no action is presumed up-to-date. Every action
falls through to its normal execution path, where the first step (with a remote cache
configured) is a `GetActionResult` keyed by the action digest.
That lookup is the whole point, and it's cheap: a single small gRPC call that returns
hit-or-miss from metadata alone. No outputs downloaded, nothing executed. Neither flag is part
of the action key, so the digest we look up is byte-identical to the one your baseline
cached.
Clearing these two caches instead of running `bazel clean` is also what keeps the probe fast on a warm runner:
the output base survives, so external repos stay extracted across runs.
Opening those gates isn't free. Throwing away Skyframe and bypassing the action cache
forces Bazel to redo analysis and re-check every action from scratch. The probe's
analysis phase runs slower than a warm `bazel test //...` that reuses all of that incremental state. We're
deliberately discarding the incrementality Bazel works hard to preserve. For a normal
build, that's a regression; for the probe it's the entire point, because that same
short-circuiting is what hides the cache lookups we need to read.
It still beats the multiple Bazel invocations `bazel-diff` requires. To diff two revisions,
`bazel-diff` checks out the base, runs `generate-hashes`, checks out your change, runs
`generate-hashes` again, then diffs the two JSON files. Two full Bazel invocations across two checkouts, and
the checkout in the middle invalidates Skyframe, so there's nothing incremental to reuse
between them. The probe runs once, against your working tree, with the cache your mainline
CI already filled as the baseline. No second checkout, no second invocation,
and the result is keyed on action digests instead of target hashes.
## What you get
Nothing to install separately. `aspect cache diff` ships in the
[Aspect CLI](/docs/cli/install), the same `aspect` binary you already run Bazel through.
```
$ aspect cache diff
//services/api:integration_test ← cache miss in //services/api:lib (CppCompile)
//services/api:smoke_test ← cache miss in //services/api:lib (CppCompile)
Affected 2 of 318 test target(s).
```
Affected labels stream to stdout, so you can pipe them straight into
`xargs aspect test`. The reasons, the "caused by" target, and the summary go to stderr, so
the two streams stay separable. `--format=json` emits a structured output for a
scheduler to consume. `--exec` runs the affected set in one shot:
```sh theme={null}
aspect cache diff --exec="aspect test"
```
On a very large graph, passing labels through `xargs` or `--exec` can hit the shell's
argument-length ceiling. Redirect stdout to a file and hand it to `aspect test` via
`--target-pattern-file` instead — no argument-length limit:
```sh theme={null}
aspect cache diff > affected.txt
aspect test --target-pattern-file=affected.txt
```
## Grouping the affected set into CI jobs
Because the affected labels come out as plain data, you aren't stuck running them as one
undifferentiated batch on one runner. You can partition the set along boundaries your team already
reasons about and schedule each group as its own CI job, a Buildkite step or a GitHub
Actions matrix leg:
* **By language:** Go tests on a runner sized for them, the TypeScript suite on another,
the slow integration tests on a third.
* **By label hierarchy:** `//services/...` on one job, `//libs/...` on another, so a change
confined to one area only spins up the runners it actually needs.
This is the sharding from earlier, made deliberate. Instead of slicing the affected set
into arbitrary chunks, you slice along ownership and test behavior boundaries. Each group gets
its own resource profile, and its own pass/fail signal, all running in parallel. The probe is cheap enough to run first
on a small coordinator, so you only pay for the runners a given change requires.
## The honest caveats
This is a cache-shaped tool, so it inherits the cache's limits.
* **Baseline coverage bounds accuracy.** A cache hit definitively means unaffected. A miss means
changed, never-uploaded, or evicted. Against a fully-populated baseline it's exact;
against a sparse cache it overreports. Your mainline CI's normal cache uploads are the
baseline, no extra job to run.
* **Always-local actions are always affected.** Anything tagged `no-remote`, `local`, or
`no-cache` bypasses the remote path; require-cached can't gate it, so it runs locally
during the probe and reads as affected.
The payoff is a different failure mode from `bazel-diff`. Instead of a Merkle tree amplifying
every leaf change up the graph, you get exactly what the cache knows changed, computed from
the same action keys your build already uses, with nothing executed to find out.