> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# AOSP build benchmark on Aspect Workflows

> Measured results for a full AOSP android-14.0.0_r22 build on self-hosted Aspect Workflows, with complete infrastructure configuration and links to the public CI runs.

Aspect Workflows runs Android Open Source Project builds on remote execution. This page
reports measured results for a full AOSP build, the infrastructure it ran on, and links to
the public CI runs so every number can be checked against its source.

**Build under test:** AOSP `android-14.0.0_r22`, target `aosp_arm64-userdebug`, 176,702
ninja actions. Deployed on Aspect Workflows self-hosted in AWS (`us-east-2`).

## Executive summary

Four configurations, spanning the full spectrum a team actually experiences, from the worst
case (a fresh machine with nothing warm) to the steady state (no changes):

|                  | Configuration                                            | Build time  |
| ---------------- | -------------------------------------------------------- | ----------- |
| Worst case       | Fresh runner: cold source sync + full local build        | **1h 16m**  |
| Baseline         | Warm runner, full uncached build, single 96-vCPU machine | **34m 07s** |
| Remote execution | Warm runner, full uncached build, 512-slot fleet         | **24m 38s** |
| Steady state     | Warm runner, no changes                                  | **2m 38s**  |

* **Full, fully-uncached build: 24m 38s** with remote execution, versus **34m 07s** on a
  96-vCPU machine alone: **81,463 of 81,481 actions executed on the remote fleet, zero
  failures**. On the part of the build that can be parallelized, remote execution is
  **3.9x faster** (14m 12s down to 3m 36s); the end-to-end ratio is diluted by AOSP's
  serial packaging tail, which no platform can parallelize.
* **Incremental build with no changes: 2m 38s**, most of which is `repo sync`. Warm,
  persistent runners keep the source tree between builds: the first-ever run on a fresh
  machine paid **39m 21s** of `repo sync`; every run since has paid about 90 seconds.
* **The open-source AOSP build is the worst case for remote execution, not the best.** It
  is small enough that fixed costs and the serial tail make up \~58% of wall-clock.
  Production AOSP-derived builds are typically several times larger, and nearly all of the
  added volume is parallelizable compile and link work, which is exactly what the fleet
  accelerates. Modeled on the measured phase profile, a build that takes \~4 hours locally
  lands around **\~1.5 hours with remote execution (\~2.5x)** at the measured fleet size, and
  **toward \~1 hour (\~3.3–3.9x)** with the fleet scaled 2–4x to match the larger build. See
  *Extrapolating to larger AOSP builds* below.

**Why self-hosted.** The deployment measured here runs entirely inside one AWS account:
source, artifacts, and caches never leave the VPC, which matters when the tree contains
vendor IP. Because the platform is yours, every knob behind these numbers is tunable per
workload: fleet size, per-action memory ceilings, worker pools, and the exact container
image AOSP's build system expects. The focus of this page is **build time**; cost economics
depend on usage pattern and are treated separately in *Cost considerations* below. Every
number on this page links to a public CI run and its uploaded logs.

## Results

The four benchmarks below use the same build on the same infrastructure. What varies is
what was already warm (the source tree, the output directory, the caches) and whether
remote execution was enabled.

| Benchmark                                         | Build step     | soong timer | Actions executed |
| ------------------------------------------------- | -------------- | ----------- | ---------------- |
| First build: fresh runner, cold sync (worst case) | **1h 16m 06s** | 35:58       | 176,702          |
| Full build, single 96-vCPU machine                | **34m 07s**    | 32:25       | 176,702          |
| Full build, remote execution                      | **24m 38s**    | 22:38       | 176,702          |
| Incremental build, nothing changed                | **2m 38s**     | 01:01       | 576              |

The remote-execution row is the honest full-build number: cache reuse was explicitly
disabled, so every action was genuinely executed. **81,463 actions ran on the remote
fleet**, 18 fell back to the local machine, and none failed.

The two warm full builds differ in exactly one thing: whether remote execution is on. It
takes the build step from 34m 07s to 24m 38s, a **1.38x** end-to-end improvement. Isolating ninja, it goes from 29m 10s to
19m 24s, a **1.50x** improvement.

Those figures understate what remote execution does to the work it can actually affect.
Roughly 5 minutes of every run is source sync and soong analysis, and 15 minutes is a serial
tail that runs on the CI machine in both configurations. **On the parallelizable part of the
build, remote execution is 3.9x faster**: 14m 12s down to 3m 36s.

### Phase-by-phase comparison

The two warm full builds execute an identical action graph, and the last 10% is the same
17,669 actions in each, so the phases compare directly.

| Component                      | Remote execution | Single machine | Ratio             |
| ------------------------------ | ---------------- | -------------- | ----------------- |
| `repo sync` and soong analysis | 5m 02s           | 4m 47s         | 1.0x (fixed cost) |
| **First 90% of actions**       | **3m 36s**       | **14m 12s**    | **3.9x faster**   |
| Last 10%: the serial tail      | 15m 36s          | 14m 54s        | 0.95x             |
| **Build step total**           | **24m 38s**      | **34m 07s**    | **1.38x**         |

Remote execution is roughly **4x on the parallelizable part of the build**, the portion that
scales with fleet size. The serial tail is marginally slower under remote execution, because
those actions form a dependency chain in which each one pays a network round trip that no
amount of parallelism recovers. The tail accounts for about 44% of the remote-execution run's
wall-clock and is unaffected by fleet size, cache state, or worker count.

### Execution profile

Within the remote-execution run, the fleet absorbs the parallel bulk of the build almost
immediately:

| Progress | Elapsed from ninja start | Sustained rate    |
| -------- | ------------------------ | ----------------- |
| 10%      | 0.3 min                  | \~1,100 actions/s |
| 30%      | 0.8 min                  | \~1,600 actions/s |
| 50%      | 1.7 min                  | \~490 actions/s   |
| 70%      | 2.5 min                  | \~770 actions/s   |
| 90%      | 3.6 min                  | \~430 actions/s   |
| 100%     | 19.4 min                 | —                 |

**90% of all actions complete in the first 3.6 minutes.** The remaining time is AOSP's
serial tail (R8/dex, APEX hiddenapi encoding, image assembly, and signing), a dependency
chain of large, mostly single-threaded steps that runs on the CI machine.

<Note>
  The serial tail is a property of the AOSP build graph, not of the execution platform. It is
  unaffected by fleet size or cache state, and is present in every configuration, including a
  purely local build. It is the reason a larger fleet doesn't reduce total build time beyond
  a point; see <b>Sizing guidance</b> below.
</Note>

All four benchmarks are public:

| Scenario                               | Public CI run                                                                                |
| -------------------------------------- | -------------------------------------------------------------------------------------------- |
| First build, fresh runner (worst case) | [run 31758099068](https://github.com/aspect-build/basic-aosp-build/actions/runs/31758099068) |
| Full build, remote execution           | [run 31847444276](https://github.com/aspect-build/basic-aosp-build/actions/runs/31847444276) |
| Incremental build                      | [run 31850749685](https://github.com/aspect-build/basic-aosp-build/actions/runs/31850749685) |
| Full build, single machine             | [run 31851206170](https://github.com/aspect-build/basic-aosp-build/actions/runs/31851206170) |

## First build on a fresh runner (worst case)

The cold-start benchmark: a newly-launched runner with no source tree, no output directory,
and nothing warm anywhere. This is what a team's first build looks like, and what every
build would look like without persistent runners.

| Phase                               | Duration       |
| ----------------------------------- | -------------- |
| `repo sync` (from nothing)          | **39m 21s**    |
| soong analysis and ninja generation | 3m 27s         |
| ninja execution                     | 32m 40s        |
| **Build step total**                | **1h 16m 06s** |

More than half the wall-clock is the cold `repo sync`: the cost runner persistence
removes. Every subsequent run on this page synced the same tree in under two minutes.

## Full build with remote execution

Every action executed remotely, with no action-cache reuse of any kind.

| Phase                                     | Duration    |
| ----------------------------------------- | ----------- |
| Prerequisites                             | 3s          |
| `repo sync` (source tree already present) | 1m 49s      |
| lunch / build configuration               | 8s          |
| soong analysis and ninja generation       | 3m 13s      |
| ninja execution                           | 19m 24s     |
| **Build step total**                      | **24m 38s** |

### Remote execution outcome

Measured from `rbe_metrics.txt`, uploaded as a workflow artifact on the run:

| Metric                         | Value                     |
| ------------------------------ | ------------------------- |
| Actions dispatched to reclient | 81,481                    |
| Executed remotely              | **81,463** (99.98%)       |
| Local fallback                 | 18 (0.02%)                |
| Failures                       | 0                         |
| Cache hits                     | 0 (disabled for this run) |

Action mix by ninja edge description: approximately 47,600 `clang++`, 18,300 `clang`, 1,100
`javac`, 1,000 `turbine`, plus `jar`, `d8`, `signapk` and `zip` steps.

The 18 local fallbacks are actions that exceeded the configured per-action memory ceiling
(large C++ links) and were retried on the CI machine, a deliberate trade-off described
under *Sizing guidance*.

## Full build on a single machine

The control: an identical source tree, an identical emptied output directory and the same
96-vCPU CI machine, with remote execution disabled. All 176,702 actions ran locally.

| Phase                               | Duration    |
| ----------------------------------- | ----------- |
| `repo sync`                         | 1m 32s      |
| soong analysis and ninja generation | 3m 15s      |
| ninja execution                     | 29m 10s     |
| **Build step total**                | **34m 07s** |

Sync and analysis cost the same as the remote-execution run, to within a few seconds: they're
unaffected by where actions execute. The difference is entirely in ninja: 29m 10s
locally against 19m 24s with the fleet.

A 96-vCPU machine is already a strong local baseline. The comparison understates
what remote execution offers a team whose CI machines are smaller, since the remote figure
barely depends on the size of the machine driving it; see *Sizing guidance*.

## Incremental build

The tree is fully built and nothing has changed. soong detects no work and only the
packaging ninja re-runs.

| Phase                                        | Duration   |
| -------------------------------------------- | ---------- |
| Prepare build directory                      | 8s         |
| `repo sync`                                  | 1m 34s     |
| soong analysis                               | 2s         |
| ninja (`no work to do`, 576 packaging edges) | 57s        |
| **Build step total**                         | **2m 38s** |

No actions reached the remote fleet; there was nothing to execute. That leaves `repo sync`
as 60% of a build that did nothing: in the incremental loop the cost is syncing source, not
compiling.

## Configuration

### CI runner

The machine that drives the build, runs soong, and executes the serial tail.

| Property      | Value                                    |
| ------------- | ---------------------------------------- |
| Instance type | `c6id.24xlarge`                          |
| vCPU / memory | 96 vCPU / 192 GiB                        |
| Local storage | 2 x 1,425 GB NVMe SSD (instance store)   |
| Root volume   | 2,048 GB EBS                             |
| Network       | 37.5 Gbps                                |
| OS            | Ubuntu 24.04.4 LTS                       |
| CI system     | GitHub Actions, self-hosted runner group |

The AOSP workspace lives on the instance-store NVMe rather than EBS, so source sync, the
output tree and all local I/O are on directly-attached storage.

### Remote execution fleet

| Property                      | Value                          |
| ----------------------------- | ------------------------------ |
| Worker instance type          | `m6id.4xlarge`                 |
| Per worker                    | 16 vCPU / 64 GiB / 950 GB NVMe |
| Workers (maximum)             | 32                             |
| Concurrent actions per worker | 16                             |
| **Total execution slots**     | **512**                        |
| Memory ceiling per action     | 3,584 MiB                      |
| Warm pool                     | 2 pre-initialized workers      |
| Scaling                       | 0 to 32, demand-driven         |

Workers run the container image AOSP's build system requests, so remote actions execute in
the toolchain environment they expect.

### Remote cache and storage tier

| Property      | Value                                        |
| ------------- | -------------------------------------------- |
| Engine        | BuildBarn storage nodes                      |
| Shards        | 4, mirrored                                  |
| Instance type | `im4gn.large` (2 vCPU / 8 GiB / 937 GB NVMe) |
| Usable cache  | \~3.7 TiB                                    |
| Frontend      | 3–18 tasks, auto-scaled                      |

A full cold AOSP build writes roughly 250–400 GB into the content-addressable store.
Repeated builds deduplicate to nearly nothing.

### AOSP build configuration

| Setting                  | Value                                                                                    |
| ------------------------ | ---------------------------------------------------------------------------------------- |
| Branch                   | `android-14.0.0_r22`                                                                     |
| Target                   | `aosp_arm64-userdebug`                                                                   |
| Platform version         | Android 14 (`UQ1A.240205.002`)                                                           |
| `repo sync` parallelism  | 4 jobs                                                                                   |
| Local ninja parallelism  | 96                                                                                       |
| Remote ninja parallelism | 800                                                                                      |
| ccache                   | Disabled (`USE_CCACHE=0`) on the full-build benchmarks; enabled on the incremental build |

Tools routed to remote execution through reclient: C++ compile, C++ link, javac, turbine,
R8, D8, jar, zip, signapk, ABI dumper, ABI linker and clang-tidy. Metalava runs locally.
Every tool is configured for local fallback, so any action the fleet cannot serve is retried
on the CI machine rather than failing the build.

## Sizing guidance

**Fleet size stops mattering after the parallel phase, for a build this size.** With 90%
of actions complete in the first 3.6 minutes, adding workers shortens that window, not the
serial tail that follows. For the OSS build, 512 slots is already past the point of
diminishing returns and a smaller fleet would produce a similar total time. The economics
invert for larger builds: a production build's parallel phase runs for hours rather than
minutes, keeping a bigger fleet busy long enough to pay for itself; see *Scaling the fleet
beyond 512 slots* under the extrapolation section.

**The CI machine can be much smaller.** The measurements above use a 96-vCPU runner for both
configurations, which flatters the local baseline: it is close to the largest single machine
worth pointing at an AOSP build. With remote execution, that machine's job is to run soong,
drive the graph and execute the serial tail, none of which needs 96 cores. A substantially
smaller runner should land near the same total time.

**Memory ceiling versus slot count.** Each worker provides 64 GiB across 16 concurrent
actions. A 3.5 GB per-action ceiling keeps all 16 slots usable, at the cost of a small
number of very large C++ links falling back to the CI machine (18 of 81,481 actions here).
Workloads with many large links can trade slots for headroom: for example, 8 slots at 7 GB.

**Cold-start behaviour.** Scaling a fleet from zero costs several minutes before full
capacity is available, because cloud capacity provisioning ramps progressively. A small
minimum fleet size or a warm pool removes this from the critical path. The runs above were
measured with the fleet already warm.

**Warm, persistent runners eliminate the largest fixed cost.** The very first build on this
deployment ran on a fresh machine and paid **39m 21s** of `repo sync`
([run 31758099068](https://github.com/aspect-build/basic-aosp-build/actions/runs/31758099068));
cold syncs against AOSP's servers range 35–50 minutes with rate limiting. Aspect Workflows
runners persist between builds (configurable idle timeout; 60 minutes here), so every
benchmark run on this page paid 1m 32s to 1m 49s instead: the source tree survives on the
runner's local NVMe and syncs incrementally. For the incremental loop, where `repo sync` is
60% of a 2m 38s build, runner persistence is the optimization that pays off most.

## Extrapolating to larger AOSP builds

The open-source AOSP build measured above is, counterintuitively, close to the worst case
for remote execution. It is small enough that the two components a fleet can't accelerate,
source sync plus soong analysis (\~5 min) and the serial packaging tail (\~15 min), account
for **58% of the remote run's wall-clock**. Production AOSP-derived builds (vendor HALs,
platform apps, product-specific stacks) are commonly three to five times the compile volume,
and nearly all of that added volume is parallelizable compile and link work. The serial tail
does not grow with it: a product build still assembles one set of images, and its
packaging-phase action count is similar to the OSS build's, growing only modestly with the
number of APKs.

That inverts the proportions. Modelling a production build that takes **\~4 hours** on the
same class of CI hardware where the OSS build takes \~1 hour, using the phase behaviour
measured above:

| Phase                        | OSS build (\~1 h local) | Production build (\~4 h local) | Production with RBE                 |
| ---------------------------- | ----------------------- | ------------------------------ | ----------------------------------- |
| Source sync + soong analysis | \~8 min                 | \~15 min                       | \~15 min                            |
| Parallelisable actions       | \~25 min (42%)          | \~195 min (**81%**)            | **\~50 min** (at the measured 3.9x) |
| Serial packaging tail        | \~26 min                | \~30 min                       | \~31 min                            |
| **Total**                    | \~60 min                | **\~4 h**                      | **\~1.5 h (\~2.5x)**                |

Model assumptions, stated so they can be challenged:

1. **Added volume is parallel-phase work.** What distinguishes a product build from OSS is
   compile and link volume: the exact work measured at 3.9x above.
2. **The serial tail is roughly constant in absolute terms**, give or take 15–20% growth for
   additional APK dexing and signing, because packaging structure does not scale with source
   volume.
3. **The parallel-phase speedup holds at 3.9x.** This is conservative: in the OSS run the
   512-slot fleet was idle after the first 3.6 minutes, so a longer parallel phase amortizes
   the same fleet better, and the fleet itself can be sized to the workload. With a fleet
   matched to a 4-hour build and output-download pruning enabled, the parallel-phase ratio
   has room above 3.9x.

Sensitivity: doubling the assumed tail still yields \~2.0x; holding the tail constant and
achieving 5x on the parallel phase yields \~2.8x. The justified range is **2.2–2.8x
end-to-end for a 4-hour-class build, \~2.5x central**, against the 1.38x measured on the
small OSS build. The direction of the relationship is the important part: **the larger the
AOSP build, the larger the share of it that remote execution accelerates.**

### Scaling the fleet beyond 512 slots

For the small OSS build, a larger fleet buys nothing: the 512 slots were already idle after
3.6 minutes. A 4-hour-class build is different: its \~195-minute parallel phase keeps a fleet
busy long enough that fleet size becomes a real lever. Scaling the measured configuration by
2–4x, with a stated scheduling-efficiency discount per step (90% / 85% / 80% of linear, to
account for narrowing graph width, cache-tier load, and client pipeline depth):

| Fleet                       | Slots | Parallel phase | Build total  | vs \~4 h local |
| --------------------------- | ----- | -------------- | ------------ | -------------- |
| 1x (measured configuration) | 512   | \~50 min       | **\~1h 36m** | **2.5x**       |
| 2x                          | 1,024 | \~28 min       | **\~1h 14m** | **3.3x**       |
| 3x                          | 1,536 | \~20 min       | **\~1h 06m** | **3.6x**       |
| 4x                          | 2,048 | \~16 min       | **\~1h 02m** | **3.9x**       |
| Limit (unbounded fleet)     | —     | 0              | \~46 min     | \~5.2x         |

Two readings of that table matter equally. First, **a fleet sized to the build pushes a
4-hour build toward one hour**: the 2x step alone recovers another 22 minutes. Second, the
returns diminish exactly as Amdahl predicts: the fixed \~46 minutes of sync, analysis and
serial tail is the floor no fleet reaches past, so each doubling buys roughly half what the
previous one did. The economically sensible operating point for a 4-hour-class build is
likely 2–3x the measured fleet; beyond that, effort is better spent on the serial tail
itself.

Scaling prerequisites, all configuration rather than architecture: the client's remote
pipeline depth (`NINJA_REMOTE_NUM_JOBS`) scales with slot count; the cache/storage tier is
sized at deployment time to match (its shard topology is fixed once data is resident); and
output-download pruning keeps the CI machine's network from becoming the funnel at higher
action rates.

Two further effects compound this in production use, both measured above rather than
modeled: real CI runs are not fully uncached, so unchanged subtrees hit the shared remote
cache and land well under the uncached figure; and no-change runs complete in minutes,
independent of build size.

## Cost considerations

This page's focus is build time; cost depends on fleet utilization and cache hit
rate, so what follows is a planning model with its assumptions stated. Two facts frame it: a
remote fleet adds compute that a local build does not consume, and a large dedicated runner
spends most of a long build under-utilized: dozens of vCPUs busy for the few minutes of
wide parallelism, then near-idle through the serial tail. Which effect wins is a question
of utilization, and utilization is set by build concurrency and caching.

### A modeled scenario: one production Android repo

Assume caching is already in place: warm persistent trees and a shared build cache. That's
the right baseline, because caching is table stakes; what this models is what *adding
remote execution* is worth on top of it.

The load is a typical day for a large organization shipping a production Android platform:
**\~300 presubmit delta builds, \~40 postsubmit integration builds, and \~15 full builds**
(the nightly target matrix plus release candidates), roughly 355 builds a day with dozens
in flight at peak hours.

Today's architecture is dedicated 64-vCPU runners. The comparison moves the same load to
16-vCPU runners plus one shared, demand-scaled fleet (up to 2,048 slots, the 4x row of the
fleet-scaling table above). Percentages are computed from on-demand list prices for the
instance types in this report; only the relative results are shown:

| Build type                       | Per day | Today (64-vCPU runners, cached) | Adding RBE (16-vCPU runners + shared fleet) |
| -------------------------------- | ------- | ------------------------------- | ------------------------------------------- |
| Presubmit (delta, warm tree)     | \~300   | \~25 min median                 | **\~15 min median**                         |
| Postsubmit integration           | \~40    | \~40 min                        | **\~22 min**                                |
| Full build (nightly matrix, RCs) | \~15    | \~4 h                           | **\~1 h 10 m**                              |

The gains concentrate where work actually executes. A cached presubmit only compiles its
delta, so remote execution trims it rather than transforms it; a full build executes
everything, and that's where 4 hours becomes \~70 minutes.

The outcome, on this load:

* **Compute cost: \~50% lower** than the same load on fully autoscaled large runners, and
  **\~90% lower** than a static pool sized for peak. Real AOSP runner pools sit between
  those bounds: a cold runner pays a \~39-minute source sync before its first build
  (measured above), which pushes teams toward keeping runners warm rather than scaling
  to zero.
* **One demand-scaled fleet serves everything.** The whole day is \~2,500–3,000 slot-hours
  of remote execution, absorbed at 60–70% fleet utilization; peak presubmit bursts queue
  for minutes, not hours.
* **\~100 hours of cumulative build waiting removed per day** across the team.

The basis: the OSS benchmark measured \~31 slot-hours of remote execution for a full build
(512 slots busy for 3.6 minutes), so a 4x-volume full build is \~125 slot-hours cold, and
\~60% of that on typical nightlies since the existing cache covers unchanged subtrees.
Cached presubmit deltas average \~4 slot-hours. The runner shrinks from 64 to 16 vCPU
because under remote execution it only runs soong, drives the graph, and executes the
serial tail.

What the model depends on:

* **The fleet only pays for executed actions.** Caching already covers repeated work in
  both columns; remote execution buys speed on the work that remains, and its cost scales
  with that work rather than with the fleet's size on paper.
* **Load keeps the fleet utilized.** 355 builds a day is comfortably enough; the same fleet
  serving a handful of builds a week would be idle-heavy, and the comparison inverts.
* **Runner downsizing compounds across the pool**: every concurrent build needs a runner,
  and each one is a quarter the size.

At this load, engineer time waiting on builds dwarfs the compute in either column, and
it's exactly the number this page measures.

## Verifying these results

Every run is public and reproducible:

* **Build logs**: the "Build AOSP" step of each linked run contains full ninja output with
  per-action timestamps.
* **Artifacts**: every run of the
  [AOSP Build workflow](https://github.com/aspect-build/basic-aosp-build/actions/workflows/aosp-build.yaml)
  uploads its full build logs as downloadable artifacts: the build log and the complete
  reclient logs, including `rbe_metrics.txt` with per-action completion statuses and a
  per-action record log. Download them from any run's page for full build details beyond
  what this page summarizes.
* **Build definition**: [aspect-build/basic-aosp-build](https://github.com/aspect-build/basic-aosp-build)
  contains the build script, the reclient configuration and the CI workflow.