Skip to main content
This guide provides step-by-step instructions for setting up Mac runners for Remote Build Execution (RBE) in Aspect Workflows. Mac RBE runners are optimized for building and testing macOS and iOS applications using Bazel.

Overview

Mac RBE workers use a specialized storage architecture designed for macOS constraints:
  • RAID0 APFS volume at /private/var/bb-storage for maximum throughput
  • File-backed Content Addressable Storage (CAS) on RAID (100 GB block file, 400 MB key-location map by default), with an in-memory fallback for hosts where FileVault can’t be disabled
  • File pool for build execution scratch space, sized from TOTAL_STORAGE_GB (50 GB by default with one runner)
  • bb-worker runs as root (required for the NFSv4 mount); bb-runner process(es) run as a dedicated _bb service account created automatically—no GUI session or auto-login required
  • EC2-native cluster membership: Mac workers reach the storage shards and scheduler the same way Linux workers do, over the cluster’s service-discovery namespace

Hardware Requirements

Supported Mac Hardware

  • AWS EC2 Mac instances (production): mac1.metal (Intel), mac2.metal (M1), mac2-m2.metal, mac2-m2pro.metal—Mac RBE workers deploy as EC2-native cluster members on Dedicated Hosts
  • Architecture: Both Intel (x86_64/amd64) and Apple Silicon (arm64/aarch64) are supported
This guide covers the AWS EC2 deployment path, which mac_init.sh assumes. Local Mac hardware (Mac mini, Mac Studio, Mac Pro) can be used for isolated development or testing against a standalone BuildBarn stack, but that’s a separate, unsupported workflow outside the scope of this guide.

Storage Requirements

  • Minimum: 700 GB available space
  • Recommended: External EBS volumes (AWS) or Thunderbolt/USB drives with 1 TB+ capacity
  • Multiple drives: Automatically configured as RAID0 for maximum performance
  • Single drive: Works but provides lower throughput than RAID configuration
On AWS EC2 Mac instances, if you attach an EBS volume while the instance is running, you MUST REBOOT before the volume becomes visible. EBS hotplug on Mac instances requires a reboot.

AWS EC2 Deployment Prerequisites

When deploying Mac RBE workers on AWS EC2 via the Aspect Workflows Terraform module, several AWS account-level resources must exist before the first terraform apply.

1. AWS License Manager Service-Linked Role

AWS License Manager requires a service-linked role to manage Dedicated Hosts on your behalf. This is a one-time setup per AWS account:
If the role already exists, this command returns an error that can be safely ignored.

2. License Manager License Configuration

AWS requires an explicit License Manager license configuration to be associated with both the Host Resource Group and the launch template. Create a minimal one with no enforcement limits:
Note the LicenseConfigurationArn from the output—you’ll need it in the next two steps.

3. EC2 Dedicated Host Resource Group

Mac instances on AWS require Dedicated Host tenancy. For Auto Scaling Groups, a Host Resource Group is required so that AWS can automatically allocate and release Dedicated Hosts as the ASG scales. Create a Host Resource Group referencing the license configuration from the previous step (adjust allowed-host-families for mac1, mac2, mac2-m2, etc.):
auto-allocate-host and auto-release-host allow AWS to provision and release Dedicated Hosts automatically as the ASG scales. This is required because mac instances have a 24-hour minimum allocation period—hosts cannot be released sooner regardless of whether an instance is running on them.
Note the GroupArn returned from this command—you’ll need it in the next step.

4. Terraform Configuration

Pass both ARNs in the executor’s ec2 block. The license_configuration_arn must match what is associated with the Host Resource Group, or AWS will reject the launch template.
To look up the latest AWS macOS AMI for your region:
macOS AMIs must be explicitly provided—there is no SSM-managed default AMI path for Mac instances, unlike Linux and Windows. The instance_image field is required when platform = "Darwin".
mac2.metal (Apple Silicon) Dedicated Hosts are scarce in most AWS regions. Set minimum = 0 initially—the ASG infrastructure will deploy but no host will be allocated. Request capacity via an AWS Support ticket before scaling up. mac1.metal (Intel) is generally more available.

Step 1: Storage Setup

The storage setup script prepares external drives for RBE workloads using RAID0 APFS volumes.

Running the Partition Script

The mac_partition.sh script automatically detects and configures external drives:

What the Script Does

  1. Detects external drives using diskutil list physical external (AWS-recommended method)
  2. Safety checks: Excludes disk0 (boot disk) and internal Apple SSDs
  3. Creates RAID0 if multiple drives are detected, or formats a single drive as APFS
  4. Mounts with nobrowse flag at /private/var/bb-storage for LaunchDaemon access
  5. Creates build directories: worker/build and worker/cache
  6. Disables Spotlight indexing on the RAID volume
  7. Adds /etc/fstab entry for automatic mounting on boot

Storage Architecture

mac_partition.sh creates the following layout on the RAID volume:
mac_init.sh populates it with the CAS storage and filepool worker.jsonnet actually reads from:
The active build directory itself is a separate NFSv4 virtual mount at /var/build/root, not a path under /private/var/bb-storage.
Why /private/var/ instead of /Volumes/?macOS DiskArbitration mounts volumes in /Volumes/ within user sessions, making them inaccessible to LaunchDaemons (system-level services). The nobrowse mount flag combined with /private/var/ ensures the storage is accessible to root system processes at boot time.

Environment Variables

Step 2: Worker Initialization

After storage is configured, initialize the RBE worker components using the mac_init.sh script.

Running the Init Script

mac_init.sh fetches its configuration verbatim from shared SSM parameters—the same common.libsonnet/worker.jsonnet/runner.jsonnet artifacts Linux workers use—rather than rendering Mac-specific templates. A handful of environment variables must be set in addition to its positional arguments; your deployment’s Terraform configuration provisions the actual SSM parameter paths.
Required Arguments:
  • bb_scheduler_address: Scheduler endpoint (for example, scheduler.workflows.internal)—used to gate worker startup until the scheduler is reachable
  • namespace: The cluster’s service-discovery namespace (for example, workflows.internal)—Mac workers are EC2-native cluster members and reach the scheduler and storage shards via Cloud Map service-discovery names built from this namespace, the same as Linux workers
Optional Arguments:
  • otel_collector_endpoint: OpenTelemetry collector endpoint for metrics
  • use_tls: Set to 1 for TLS, 0 for plaintext (auto-detected from bb_scheduler_address if not specified)
Required Environment Variables:
  • COMMON_CONFIG_SSM_PATH, WORKER_CONFIG_SSM_PATH, RUNNER_CONFIG_SSM_PATH: SSM parameter paths for the shared config files
  • WORKER_COMMON_ENV: space-separated KEY=VALUE pairs carrying cluster-topology settings (NUM_SHARDS, INFIX, COMMON_NAMESPACE, MIRRORED, ISCC_ENABLED) the worker needs to address storage shards directly

Example

What the Init Script Does

  1. Creates the _bb service account using dscl—a hidden system service user (UID 305+) with no shell and no home directory, following the macOS convention for service accounts like _www and _mysql. bb-worker itself runs as root (required for the NFSv4 mount); bb-runner process(es) run as _bb
  2. Verifies storage is mounted at /private/var/bb-storage
  3. Pre-allocates CAS block files on RAID (skipped entirely when CAS_MODE=memory)
  4. Downloads BuildBarn binaries: bb-worker and bb-runner for the detected architecture
  5. Code signs binaries with ad-hoc signatures (required for macOS security)
  6. Fetches common.libsonnet/worker.jsonnet/runner.jsonnet verbatim from SSM—no host-side templating
  7. Installs OpenTelemetry Collector (if endpoint provided) for metrics export
  8. Creates LaunchDaemons, injecting every setting worker.jsonnet/runner.jsonnet need as environment variables—bb-worker/bb-runner resolve them from their own process environment at startup
  9. Starts every runner service first, then waits for the scheduler to become DNS-resolvable and TCP-listening before starting bb-worker (bounded retry, ~5 minutes by default—covers Cloud Map registration lag on a fresh deployment)
  10. Performs health checks against bb-worker and each bb-runner

Installed Components

After successful initialization:
  • Binaries: /usr/local/aspectbuild/bin/ (bb-worker, bb-runner, otelcol)
  • Configs: /usr/local/aspectbuild/config/ (worker.jsonnet, runner.jsonnet)
  • Logs: /usr/local/var/log/aspectbuild/ (bb-worker.log, bb-runner.log)
  • LaunchDaemons: /Library/LaunchDaemons/build.aspect.*.plist

Step 3: macOS Security Configuration

Service Management

Services always run via LaunchDaemon in the system launchctl domain (/Library/LaunchDaemons/), starting at boot with no interactive user session or auto-login required. bb-worker runs as root (required for the NFSv4 mount(2) call, mirroring the SYS_ADMIN capability Linux workers need for FUSE); each bb-runner runs as the _bb service account, created automatically by the init script.

Full Disk Access

Both bb-worker and bb-runner require Full Disk Access (FDA). The init script prints the exact steps after completing setup.
Grant FDA to the binary files (not the user account):
  1. Open System Settings → Privacy & Security → Full Disk Access
  2. Click + and add /usr/local/aspectbuild/bin/bb-worker
  3. Click + and add /usr/local/aspectbuild/bin/bb-runner
  4. Ensure both have checkmarks
After granting FDA, restart the services (repeat the bb-runner-{index} lines for each configured runner):

Additional Security Considerations

  • Code signing: Binaries are automatically signed with ad-hoc signatures (codesign -s -)
  • Quarantine removal: xattr -cr removes quarantine flags from downloaded binaries
  • nobrowse mount: Required for LaunchDaemon access to storage volumes

iOS simulator prerequisites

Mac RBE workers that execute iOS simulator tests require additional one-time setup after worker initialization. Skipping these steps causes test actions to fail or time out on first execution.

Install iOS simulator runtime

The iOS simulator runtime isn’t bundled with Xcode and must be downloaded separately. If it isn’t pre-installed, the first test run triggers an automatic download mid-build, which causes the action to time out.
Run this on the worker Mac before the worker starts serving test actions. The download is several gigabytes and can take 10–30 minutes depending on network speed.
Verify the runtime installed successfully:
Expected output:

Full Disk Access for iOS test execution

bb-runner requires Full Disk Access to write action output files within the virtual filesystem during iOS test execution. Without it, test actions fail immediately with operation not permitted when the runner attempts to open stdout/stderr capture paths. Ensure bb-runner is granted Full Disk Access as described in the Full Disk Access section preceding this one. This is required for all iOS simulator test actions, not just compilation.

Step 4: Configuration Details

Worker Configuration

The worker is configured with the following key settings:

Runner Configuration

cleanTemporaryDirectories and cleanProcessTable are currently disabled on Darwin: enabling them caused worker readiness-check failures in prior hardware testing. bb-remote-execution has real Darwin implementations for both (native process-table scanning and directory cleanup), so this isn’t a known platform limitation—the root cause hasn’t been isolated yet.

Communication Ports

Step 5: Verification and Monitoring

Health Checks

The init script automatically performs health checks. You can manually verify:

Service Management

mac_init.sh always names runner services with an index suffix, even with NUM_RUNNERS=1—there is no unindexed build.aspect.bb-runner.plist.

Storage Verification

Troubleshooting

Worker Fails to Start

Symptom: Worker process crashes or doesn’t appear in pgrep output. Solutions:
  1. Check Full Disk Access permissions (most common issue)
  2. Verify storage is mounted at /private/var/bb-storage with nobrowse flag
  3. Review error logs: tail -20 /usr/local/var/log/aspectbuild/bb-worker.err
  4. Ensure scheduler address is reachable: nc -zv <scheduler_address> 8983—mac_init.sh already polls this automatically for up to SCHEDULER_WAIT_ATTEMPTS × 5 seconds (default ~5 minutes) before starting bb-worker; check its console output for polling attempts if setup seems to hang here

Permission Denied Errors

Symptom: Logs show “Permission denied” or “Operation not permitted” errors. Solutions:
  1. Grant Full Disk Access to both bb-worker and bb-runner
  2. Verify storage mount has correct permissions: ls -la /private/var/bb-storage
  3. Ensure binaries are code-signed: codesign -dv /usr/local/aspectbuild/bin/bb-worker

Storage Not Accessible to LaunchDaemon

Symptom: Worker starts but can’t access /private/var/bb-storage. Solutions:
  1. Verify mount has nobrowse flag: mount | grep nobrowse
  2. Check /etc/fstab entry exists for the volume UUID
  3. Remount with nobrowse:

RAID Creation Fails

Symptom: mac_partition.sh fails when creating RAID. Solutions:
  1. Ensure disks are completely unmounted: sudo diskutil unmountDisk force /dev/diskN
  2. Check for existing RAID: diskutil appleRAID list
  3. Use FORCE_RECREATE=1 to tear down existing RAID
  4. Verify disks are not the boot disk or internal SSD

Worker Fails with mmap or “Operation not permitted” on CAS Files

Symptom: bb-worker.err shows an mmap or “operation not permitted” error referencing the CAS block files. Cause: File-backed CAS (CAS_MODE=file) uses mmap on APFS files, which requires FileVault to be disabled. Solutions:
  1. Ensure FileVault is disabled: fdesetup status
  2. Verify the RAID volume is mounted with nobrowse: mount | grep bb-storage
  3. Fall back to in-memory CAS: re-run with CAS_MODE=memory sudo -E ./mac_init.sh ...

Health Check Timeout

Symptom: Init script times out waiting for services to respond. Solutions:
  1. Check if runner started first: pgrep -f bb-runner
  2. Verify ports aren’t in use: lsof -i :9980 -i :9981
  3. Review logs for startup errors
  4. Increase timeout by editing the health_check() function

Metrics Not Appearing

Symptom: OTEL collector configured but metrics not reaching endpoint. Solutions:
  1. Verify OTEL collector is running: pgrep -f otelcol
  2. Check OTEL logs: tail -f /usr/local/var/log/aspectbuild/otel-collector.log
  3. Verify endpoint is reachable: nc -zv <otel_endpoint> 4317
  4. Check TLS settings in /usr/local/aspectbuild/config/otel/config.yaml

EBS Volume Not Detected (AWS EC2)

Symptom: Script reports “No suitable drives found” after attaching EBS. Solutions:
  1. REBOOT the instance - EBS hotplug requires a reboot on Mac instances
  2. After reboot, verify disk appears: diskutil list physical external
  3. If still not visible, check AWS console that volume is attached
  4. Ensure volume is attached to the correct instance

Platform Configuration for Mac Builds

After setting up Mac RBE workers, configure Bazel to target them.

Create Mac Platform Definition

In your repository’s platforms/BUILD.bazel:

Update .bazelrc

Using Mac RBE

Advanced Configuration

Environment Variables

The following environment variables customize mac_init.sh behavior:
TOTAL_STORAGE_GB/MAX_BLOB_SIZE_GB (used by worker.jsonnet’s sizing math) and CAS_FILE_SIZE_BYTES/FILEPOOL_SIZE_BYTES (what’s actually pre-allocated on disk) are configured independently. Mismatched values won’t error, but the worker’s expected sizes can silently diverge from what’s on disk—keep them consistent when changing NUM_RUNNERS or storage sizing.

Custom Storage Sizing

Multiple Worker Instances

For testing different configurations on the same Mac:
  1. Use different install directories: INSTALL_DIR=/usr/local/aspectbuild-test
  2. Use different storage paths: STORAGE_BASE_DIR=/private/var/bb-storage-test
  3. Change service names in LaunchDaemon plist files
Diagnostics ports are fixed (bb-worker: 9980, each bb-runner: 9981 + index)—two independent worker instances on the same host will conflict on these ports.

Scaling Parallel Build Actions

Each bb-runner process runs at concurrency 1 by design. To run more build actions in parallel, increase NUM_RUNNERS and re-run mac_init.sh—it creates one LaunchDaemon per runner, each with its own Unix socket and diagnostics port:
Hand-editing /usr/local/aspectbuild/config/worker.jsonnet isn’t persistent—it’s fetched fresh from SSM every time mac_init.sh runs.

References

  • Script locations: infrastructure/modules/workflows/remote/capacity/
    • mac_partition.sh - Storage setup (infrastructure/modules/workflows/remote/capacity/mac_partition.sh:1)
    • mac_init.sh - Worker initialization (infrastructure/modules/workflows/remote/capacity/mac_init.sh:1)
  • BuildBarn documentation: bb-remote-execution
  • macOS Launch Daemons: Creating Launch Daemons
  • AWS EC2 Mac Instances: EC2 Mac Instances User Guide