android-14.0.0_r22, target aosp_arm64-userdebug, 176,702
ninja actions. Deployed on Aspect Workflows self-hosted in AWS (us-east-2).
Executive summary
Four configurations, spanning the full spectrum a team actually experiences, from the worst case (a fresh machine with nothing warm) to the steady state (no changes):- Full, fully-uncached build: 24m 38s with remote execution, versus 34m 07s on a 96-vCPU machine alone: 81,463 of 81,481 actions executed on the remote fleet, zero failures. On the part of the build that can be parallelized, remote execution is 3.9x faster (14m 12s down to 3m 36s); the end-to-end ratio is diluted by AOSP’s serial packaging tail, which no platform can parallelize.
- Incremental build with no changes: 2m 38s, most of which is
repo sync. Warm, persistent runners keep the source tree between builds: the first-ever run on a fresh machine paid 39m 21s ofrepo sync; every run since has paid about 90 seconds. - The open-source AOSP build is the worst case for remote execution, not the best. It is small enough that fixed costs and the serial tail make up ~58% of wall-clock. Production AOSP-derived builds are typically several times larger, and nearly all of the added volume is parallelizable compile and link work, which is exactly what the fleet accelerates. Modeled on the measured phase profile, a build that takes ~4 hours locally lands around ~1.5 hours with remote execution (~2.5x) at the measured fleet size, and toward ~1 hour (~3.3–3.9x) with the fleet scaled 2–4x to match the larger build. See Extrapolating to larger AOSP builds below.
Results
The four benchmarks below use the same build on the same infrastructure. What varies is what was already warm (the source tree, the output directory, the caches) and whether remote execution was enabled.
The remote-execution row is the honest full-build number: cache reuse was explicitly
disabled, so every action was genuinely executed. 81,463 actions ran on the remote
fleet, 18 fell back to the local machine, and none failed.
The two warm full builds differ in exactly one thing: whether remote execution is on. It
takes the build step from 34m 07s to 24m 38s, a 1.38x end-to-end improvement. Isolating ninja, it goes from 29m 10s to
19m 24s, a 1.50x improvement.
Those figures understate what remote execution does to the work it can actually affect.
Roughly 5 minutes of every run is source sync and soong analysis, and 15 minutes is a serial
tail that runs on the CI machine in both configurations. On the parallelizable part of the
build, remote execution is 3.9x faster: 14m 12s down to 3m 36s.
Phase-by-phase comparison
The two warm full builds execute an identical action graph, and the last 10% is the same 17,669 actions in each, so the phases compare directly.
Remote execution is roughly 4x on the parallelizable part of the build, the portion that
scales with fleet size. The serial tail is marginally slower under remote execution, because
those actions form a dependency chain in which each one pays a network round trip that no
amount of parallelism recovers. The tail accounts for about 44% of the remote-execution run’s
wall-clock and is unaffected by fleet size, cache state, or worker count.
Execution profile
Within the remote-execution run, the fleet absorbs the parallel bulk of the build almost immediately:
90% of all actions complete in the first 3.6 minutes. The remaining time is AOSP’s
serial tail (R8/dex, APEX hiddenapi encoding, image assembly, and signing), a dependency
chain of large, mostly single-threaded steps that runs on the CI machine.
The serial tail is a property of the AOSP build graph, not of the execution platform. It is
unaffected by fleet size or cache state, and is present in every configuration, including a
purely local build. It is the reason a larger fleet doesn’t reduce total build time beyond
a point; see Sizing guidance below.
First build on a fresh runner (worst case)
The cold-start benchmark: a newly-launched runner with no source tree, no output directory, and nothing warm anywhere. This is what a team’s first build looks like, and what every build would look like without persistent runners.
More than half the wall-clock is the cold
repo sync: the cost runner persistence
removes. Every subsequent run on this page synced the same tree in under two minutes.
Full build with remote execution
Every action executed remotely, with no action-cache reuse of any kind.Remote execution outcome
Measured fromrbe_metrics.txt, uploaded as a workflow artifact on the run:
Action mix by ninja edge description: approximately 47,600
clang++, 18,300 clang, 1,100
javac, 1,000 turbine, plus jar, d8, signapk and zip steps.
The 18 local fallbacks are actions that exceeded the configured per-action memory ceiling
(large C++ links) and were retried on the CI machine, a deliberate trade-off described
under Sizing guidance.
Full build on a single machine
The control: an identical source tree, an identical emptied output directory and the same 96-vCPU CI machine, with remote execution disabled. All 176,702 actions ran locally.
Sync and analysis cost the same as the remote-execution run, to within a few seconds: they’re
unaffected by where actions execute. The difference is entirely in ninja: 29m 10s
locally against 19m 24s with the fleet.
A 96-vCPU machine is already a strong local baseline. The comparison understates
what remote execution offers a team whose CI machines are smaller, since the remote figure
barely depends on the size of the machine driving it; see Sizing guidance.
Incremental build
The tree is fully built and nothing has changed. soong detects no work and only the packaging ninja re-runs.
No actions reached the remote fleet; there was nothing to execute. That leaves
repo sync
as 60% of a build that did nothing: in the incremental loop the cost is syncing source, not
compiling.
Configuration
CI runner
The machine that drives the build, runs soong, and executes the serial tail.
The AOSP workspace lives on the instance-store NVMe rather than EBS, so source sync, the
output tree and all local I/O are on directly-attached storage.
Remote execution fleet
Workers run the container image AOSP’s build system requests, so remote actions execute in
the toolchain environment they expect.
Remote cache and storage tier
A full cold AOSP build writes roughly 250–400 GB into the content-addressable store.
Repeated builds deduplicate to nearly nothing.
AOSP build configuration
Tools routed to remote execution through reclient: C++ compile, C++ link, javac, turbine,
R8, D8, jar, zip, signapk, ABI dumper, ABI linker and clang-tidy. Metalava runs locally.
Every tool is configured for local fallback, so any action the fleet cannot serve is retried
on the CI machine rather than failing the build.
Sizing guidance
Fleet size stops mattering after the parallel phase, for a build this size. With 90% of actions complete in the first 3.6 minutes, adding workers shortens that window, not the serial tail that follows. For the OSS build, 512 slots is already past the point of diminishing returns and a smaller fleet would produce a similar total time. The economics invert for larger builds: a production build’s parallel phase runs for hours rather than minutes, keeping a bigger fleet busy long enough to pay for itself; see Scaling the fleet beyond 512 slots under the extrapolation section. The CI machine can be much smaller. The measurements above use a 96-vCPU runner for both configurations, which flatters the local baseline: it is close to the largest single machine worth pointing at an AOSP build. With remote execution, that machine’s job is to run soong, drive the graph and execute the serial tail, none of which needs 96 cores. A substantially smaller runner should land near the same total time. Memory ceiling versus slot count. Each worker provides 64 GiB across 16 concurrent actions. A 3.5 GB per-action ceiling keeps all 16 slots usable, at the cost of a small number of very large C++ links falling back to the CI machine (18 of 81,481 actions here). Workloads with many large links can trade slots for headroom: for example, 8 slots at 7 GB. Cold-start behaviour. Scaling a fleet from zero costs several minutes before full capacity is available, because cloud capacity provisioning ramps progressively. A small minimum fleet size or a warm pool removes this from the critical path. The runs above were measured with the fleet already warm. Warm, persistent runners eliminate the largest fixed cost. The very first build on this deployment ran on a fresh machine and paid 39m 21s ofrepo sync
(run 31758099068);
cold syncs against AOSP’s servers range 35–50 minutes with rate limiting. Aspect Workflows
runners persist between builds (configurable idle timeout; 60 minutes here), so every
benchmark run on this page paid 1m 32s to 1m 49s instead: the source tree survives on the
runner’s local NVMe and syncs incrementally. For the incremental loop, where repo sync is
60% of a 2m 38s build, runner persistence is the optimization that pays off most.
Extrapolating to larger AOSP builds
The open-source AOSP build measured above is, counterintuitively, close to the worst case for remote execution. It is small enough that the two components a fleet can’t accelerate, source sync plus soong analysis (~5 min) and the serial packaging tail (~15 min), account for 58% of the remote run’s wall-clock. Production AOSP-derived builds (vendor HALs, platform apps, product-specific stacks) are commonly three to five times the compile volume, and nearly all of that added volume is parallelizable compile and link work. The serial tail does not grow with it: a product build still assembles one set of images, and its packaging-phase action count is similar to the OSS build’s, growing only modestly with the number of APKs. That inverts the proportions. Modelling a production build that takes ~4 hours on the same class of CI hardware where the OSS build takes ~1 hour, using the phase behaviour measured above:
Model assumptions, stated so they can be challenged:
- Added volume is parallel-phase work. What distinguishes a product build from OSS is compile and link volume: the exact work measured at 3.9x above.
- The serial tail is roughly constant in absolute terms, give or take 15–20% growth for additional APK dexing and signing, because packaging structure does not scale with source volume.
- The parallel-phase speedup holds at 3.9x. This is conservative: in the OSS run the 512-slot fleet was idle after the first 3.6 minutes, so a longer parallel phase amortizes the same fleet better, and the fleet itself can be sized to the workload. With a fleet matched to a 4-hour build and output-download pruning enabled, the parallel-phase ratio has room above 3.9x.
Scaling the fleet beyond 512 slots
For the small OSS build, a larger fleet buys nothing: the 512 slots were already idle after 3.6 minutes. A 4-hour-class build is different: its ~195-minute parallel phase keeps a fleet busy long enough that fleet size becomes a real lever. Scaling the measured configuration by 2–4x, with a stated scheduling-efficiency discount per step (90% / 85% / 80% of linear, to account for narrowing graph width, cache-tier load, and client pipeline depth):
Two readings of that table matter equally. First, a fleet sized to the build pushes a
4-hour build toward one hour: the 2x step alone recovers another 22 minutes. Second, the
returns diminish exactly as Amdahl predicts: the fixed ~46 minutes of sync, analysis and
serial tail is the floor no fleet reaches past, so each doubling buys roughly half what the
previous one did. The economically sensible operating point for a 4-hour-class build is
likely 2–3x the measured fleet; beyond that, effort is better spent on the serial tail
itself.
Scaling prerequisites, all configuration rather than architecture: the client’s remote
pipeline depth (
NINJA_REMOTE_NUM_JOBS) scales with slot count; the cache/storage tier is
sized at deployment time to match (its shard topology is fixed once data is resident); and
output-download pruning keeps the CI machine’s network from becoming the funnel at higher
action rates.
Two further effects compound this in production use, both measured above rather than
modeled: real CI runs are not fully uncached, so unchanged subtrees hit the shared remote
cache and land well under the uncached figure; and no-change runs complete in minutes,
independent of build size.
Cost considerations
This page’s focus is build time; cost depends on fleet utilization and cache hit rate, so what follows is a planning model with its assumptions stated. Two facts frame it: a remote fleet adds compute that a local build does not consume, and a large dedicated runner spends most of a long build under-utilized: dozens of vCPUs busy for the few minutes of wide parallelism, then near-idle through the serial tail. Which effect wins is a question of utilization, and utilization is set by build concurrency and caching.A modeled scenario: one production Android repo
Assume caching is already in place: warm persistent trees and a shared build cache. That’s the right baseline, because caching is table stakes; what this models is what adding remote execution is worth on top of it. The load is a typical day for a large organization shipping a production Android platform: ~300 presubmit delta builds, ~40 postsubmit integration builds, and ~15 full builds (the nightly target matrix plus release candidates), roughly 355 builds a day with dozens in flight at peak hours. Today’s architecture is dedicated 64-vCPU runners. The comparison moves the same load to 16-vCPU runners plus one shared, demand-scaled fleet (up to 2,048 slots, the 4x row of the fleet-scaling table above). Percentages are computed from on-demand list prices for the instance types in this report; only the relative results are shown:
The gains concentrate where work actually executes. A cached presubmit only compiles its
delta, so remote execution trims it rather than transforms it; a full build executes
everything, and that’s where 4 hours becomes ~70 minutes.
The outcome, on this load:
- Compute cost: ~50% lower than the same load on fully autoscaled large runners, and ~90% lower than a static pool sized for peak. Real AOSP runner pools sit between those bounds: a cold runner pays a ~39-minute source sync before its first build (measured above), which pushes teams toward keeping runners warm rather than scaling to zero.
- One demand-scaled fleet serves everything. The whole day is ~2,500–3,000 slot-hours of remote execution, absorbed at 60–70% fleet utilization; peak presubmit bursts queue for minutes, not hours.
- ~100 hours of cumulative build waiting removed per day across the team.
- The fleet only pays for executed actions. Caching already covers repeated work in both columns; remote execution buys speed on the work that remains, and its cost scales with that work rather than with the fleet’s size on paper.
- Load keeps the fleet utilized. 355 builds a day is comfortably enough; the same fleet serving a handful of builds a week would be idle-heavy, and the comparison inverts.
- Runner downsizing compounds across the pool: every concurrent build needs a runner, and each one is a quarter the size.
Verifying these results
Every run is public and reproducible:- Build logs: the “Build AOSP” step of each linked run contains full ninja output with per-action timestamps.
- Artifacts: every run of the
AOSP Build workflow
uploads its full build logs as downloadable artifacts: the build log and the complete
reclient logs, including
rbe_metrics.txtwith per-action completion statuses and a per-action record log. Download them from any run’s page for full build details beyond what this page summarizes. - Build definition: aspect-build/basic-aosp-build contains the build script, the reclient configuration and the CI workflow.