Targeting remote execution worker pools

Remote execution is available on Aspect Enterprise today and coming soon to Aspect Cloud. A deployment’s remote execution fleet is divided into worker pools. Each pool runs a particular container image on a particular instance type, and advertises a set of platform properties describing itself. A Bazel action reaches a pool when the exec platform it was configured with requests exactly that set of properties.

The rule that catches everyone

REv2 platform matching is whole-set equality, not a subset match. A worker advertising three properties is reachable only by a client requesting all three, with the same values. Request two of them and nothing matches; request a fourth and nothing matches. An action whose platform no pool advertises has nowhere to run: depending on the scheduler it waits or is rejected, and Bazel runs it locally instead only if --remote_local_fallback is set.

A newly added pool that sits idle while actions run locally is almost always a platform mismatch. Compare the properties the pool advertises against the exec_properties on your exec platform, character for character, before looking anywhere else.

What a pool advertises

Two sources, merged:

Derived properties, from the pool’s configured operating system and container image, typically OSFamily and container-image.
Explicit properties, set on the pool, which merge over the derived ones.

So a pool configured with a Linux image and an explicit Pool = "large" advertises three properties:

OSFamily       = Linux
container-image = docker://<the pool's image>
Pool           = large

A pool can also drop a derived property when it’s configured. It then advertises only what remains, and your exec platform has to mirror that.

The Bazel side

An exec platform requests properties with exec_properties:

platforms/BUILD.bazel

platform(
    name = "linux_x86_64_remote",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    exec_properties = {
        "OSFamily": "Linux",
        "container-image": "docker://your-registry/rbe-base@sha256:...",
    },
)

Register it only for builds that execute remotely: registered for every build, it would configure local actions for the remote platform too. Keep it in a --config group of your own:

.bazelrc

build:rbe --extra_execution_platforms=//platforms:linux_x86_64_remote

Then turn that group on wherever the build reaches remote execution. Bazel merges a group’s lines across every rc, so a line in your .bazelrc extends a group aspect setup bazelrc writes:

The build reaches remote execution through	Add
`bazel --config=aspect-<name>-exec`, from the repository rc	`build:aspect-<name>-exec --config=rbe`
`bazel --config=aspect-exec` on a Workflows runner, or a machine rc written with `--remote=exec`	`build:aspect-exec --config=rbe`
`aspect <task> --remote=exec`, or `--workflows:remote-exec` on a Workflows runner	`--config=rbe` on the same call. These wire the executor directly, with no `--config` group

<name> is the deployment name aspect auth status shows. Aspect support gives you the exact property set each pool advertises; on a self-hosted deployment, it’s in the pool’s configuration. Copy it verbatim.

Routing specific actions to specific pools

The usual reason for a second pool is that some actions need something the default workers don’t have, such as more memory. When the second pool runs the same image as the default one, a Pool property is all that separates them, and a target can ask for it directly:

server/BUILD.bazel

cc_test(
    name = "load_test",
    srcs = ["load_test.cc"],
    exec_properties = {"Pool": "large"},
)

The target’s exec_properties merge into its exec platform’s, so the action requests OSFamily, container-image and Pool together, exactly what the large pool advertises.

If the second pool runs a different image, Pool alone won’t match. The merged request still carries the default platform’s container-image. Give that pool its own platform with its own image and a constraint of your own, register it alongside the default one, and select it with exec_compatible_with on the target.

Operating systems

Pool OS	Notes
Linux	The default. Docker-based tests are supported on workers.
macOS	Aspect Enterprise, Workflows 6.0 and later: EC2 Mac on AWS, or Macs you provide. Lets iOS and macOS targets build and test remotely. See macOS remote execution.
FreeBSD	AWS deployments, on a pinned module based on Workflows 5.18; not in the 6.x releases. See .

Checking what actually happened

The Build Results UI shows, per invocation, how many actions executed remotely versus locally. If a build you expected to fan out runs actions locally under --remote_local_fallback, or has actions that wait or are rejected, the platform isn’t matching. --toolchain_resolution_debug='.*' and Bazel’s execution log show which platform each action resolved to, which is the fastest way to see what your build is actually requesting.

Remote execution: what the fleet is and how it scales.
Parallelize remote execution: getting more actions in flight once pools match.
Configuration options: requesting a new pool on an Aspect Enterprise deployment hosted by Aspect.

​The rule that catches everyone

​What a pool advertises

​The Bazel side

​Routing specific actions to specific pools

​Operating systems

​Checking what actually happened

​Related

The rule that catches everyone

What a pool advertises

The Bazel side

Routing specific actions to specific pools

Operating systems

Checking what actually happened

Related