Measuring results - Aspect Build

A trial typically ends in a comparison with your current CI. Measure the same things on your current CI and on the dark-launched job, over the same commits, and agree with your Aspect contact at the start what result counts as a success.

What to measure

Metric	Where it comes from	Why it matters
Pull request wall time, p50 and p90	Your CI provider’s job durations	What developers wait for. p90 is the one they complain about
Queue time	Your CI provider: time from trigger to a job starting	A fast build behind a slow queue is still slow
Cache hit rate	The Build Results UI, per invocation	How much work the build skipped
CI compute cost	Your CI provider’s bill today. Self-hosted: the deployment’s or your cloud bill	Makes the ROI case: what the faster builds cost or save

Take the current-CI numbers over a period of ordinary traffic before you start, as the trial’s first step says.

Comparing fairly

Same commits. The dark-launched job runs on the commits your current CI already builds, so both sides see the same changes.
Warm against warm. The first builds on a new deployment are cold by definition. Compare after the cache and the runners have seen a few days of traffic.
Tune before you judge. Runner sizes, group splits and scaling limits change the numbers. Work through Optimizing CI runners and Optimizing your Bazel build before you take the final measurement.
Count what you excluded. Tests you tagged out to get green aren’t in the comparison. Say how many there are, and what it would take to bring them back.

​What to measure

​Comparing fairly

What to measure

Comparing fairly