Overview
Mac RBE workers use a specialized storage architecture designed for macOS constraints:- RAID0 APFS volume at
/private/var/bb-storagefor maximum throughput - File-backed Content Addressable Storage (CAS) on RAID (100 GB block file, 400 MB key-location map by default), with an in-memory fallback for hosts where FileVault can’t be disabled
- File pool for build execution scratch space, sized from
TOTAL_STORAGE_GB(50 GB by default with one runner) bb-workerruns as root (required for the NFSv4 mount);bb-runnerprocess(es) run as a dedicated_bbservice account created automatically—no GUI session or auto-login required- EC2-native cluster membership: Mac workers reach the storage shards and scheduler the same way Linux workers do, over the cluster’s service-discovery namespace
Hardware Requirements
Supported Mac Hardware
- AWS EC2 Mac instances (production): mac1.metal (Intel), mac2.metal (M1), mac2-m2.metal, mac2-m2pro.metal—Mac RBE workers deploy as EC2-native cluster members on Dedicated Hosts
- Architecture: Both Intel (x86_64/amd64) and Apple Silicon (arm64/aarch64) are supported
This guide covers the AWS EC2 deployment path, which
mac_init.sh assumes. Local Mac hardware (Mac mini, Mac Studio, Mac Pro) can be used for isolated development or testing against a standalone BuildBarn stack, but that’s a separate, unsupported workflow outside the scope of this guide.Storage Requirements
- Minimum: 700 GB available space
- Recommended: External EBS volumes (AWS) or Thunderbolt/USB drives with 1 TB+ capacity
- Multiple drives: Automatically configured as RAID0 for maximum performance
- Single drive: Works but provides lower throughput than RAID configuration
AWS EC2 Deployment Prerequisites
When deploying Mac RBE workers on AWS EC2 via the Aspect Workflows Terraform module, several AWS account-level resources must exist before the firstterraform apply.
1. AWS License Manager Service-Linked Role
AWS License Manager requires a service-linked role to manage Dedicated Hosts on your behalf. This is a one-time setup per AWS account:2. License Manager License Configuration
AWS requires an explicit License Manager license configuration to be associated with both the Host Resource Group and the launch template. Create a minimal one with no enforcement limits:LicenseConfigurationArn from the output—you’ll need it in the next two steps.
3. EC2 Dedicated Host Resource Group
Mac instances on AWS require Dedicated Host tenancy. For Auto Scaling Groups, a Host Resource Group is required so that AWS can automatically allocate and release Dedicated Hosts as the ASG scales. Create a Host Resource Group referencing the license configuration from the previous step (adjustallowed-host-families for mac1, mac2, mac2-m2, etc.):
auto-allocate-host and auto-release-host allow AWS to provision and release Dedicated Hosts automatically as the ASG scales. This is required because mac instances have a 24-hour minimum allocation period—hosts cannot be released sooner regardless of whether an instance is running on them.GroupArn returned from this command—you’ll need it in the next step.
4. Terraform Configuration
Pass both ARNs in the executor’sec2 block. The license_configuration_arn must match what is associated with the Host Resource Group, or AWS will reject the launch template.
Step 1: Storage Setup
The storage setup script prepares external drives for RBE workloads using RAID0 APFS volumes.Running the Partition Script
Themac_partition.sh script automatically detects and configures external drives:
What the Script Does
- Detects external drives using
diskutil list physical external(AWS-recommended method) - Safety checks: Excludes disk0 (boot disk) and internal Apple SSDs
- Creates RAID0 if multiple drives are detected, or formats a single drive as APFS
- Mounts with
nobrowseflag at/private/var/bb-storagefor LaunchDaemon access - Creates build directories:
worker/buildandworker/cache - Disables Spotlight indexing on the RAID volume
- Adds
/etc/fstabentry for automatic mounting on boot
Storage Architecture
mac_partition.sh creates the following layout on the RAID volume:
mac_init.sh populates it with the CAS storage and filepool worker.jsonnet actually reads from:
The active build directory itself is a separate NFSv4 virtual mount at
/var/build/root, not a path under /private/var/bb-storage.Why
/private/var/ instead of /Volumes/?macOS DiskArbitration mounts volumes in /Volumes/ within user sessions, making them inaccessible to LaunchDaemons (system-level services). The nobrowse mount flag combined with /private/var/ ensures the storage is accessible to root system processes at boot time.Environment Variables
Step 2: Worker Initialization
After storage is configured, initialize the RBE worker components using themac_init.sh script.
Running the Init Script
mac_init.sh fetches its configuration verbatim from shared SSM parameters—the same common.libsonnet/worker.jsonnet/runner.jsonnet artifacts Linux workers use—rather than rendering Mac-specific templates. A handful of environment variables must be set in addition to its positional arguments; your deployment’s Terraform configuration provisions the actual SSM parameter paths.
bb_scheduler_address: Scheduler endpoint (for example,scheduler.workflows.internal)—used to gate worker startup until the scheduler is reachablenamespace: The cluster’s service-discovery namespace (for example,workflows.internal)—Mac workers are EC2-native cluster members and reach the scheduler and storage shards via Cloud Map service-discovery names built from this namespace, the same as Linux workers
otel_collector_endpoint: OpenTelemetry collector endpoint for metricsuse_tls: Set to1for TLS,0for plaintext (auto-detected frombb_scheduler_addressif not specified)
COMMON_CONFIG_SSM_PATH,WORKER_CONFIG_SSM_PATH,RUNNER_CONFIG_SSM_PATH: SSM parameter paths for the shared config filesWORKER_COMMON_ENV: space-separatedKEY=VALUEpairs carrying cluster-topology settings (NUM_SHARDS,INFIX,COMMON_NAMESPACE,MIRRORED,ISCC_ENABLED) the worker needs to address storage shards directly
Example
What the Init Script Does
- Creates the
_bbservice account usingdscl—a hidden system service user (UID 305+) with no shell and no home directory, following the macOS convention for service accounts like_wwwand_mysql.bb-workeritself runs as root (required for the NFSv4 mount);bb-runnerprocess(es) run as_bb - Verifies storage is mounted at
/private/var/bb-storage - Pre-allocates CAS block files on RAID (skipped entirely when
CAS_MODE=memory) - Downloads BuildBarn binaries:
bb-workerandbb-runnerfor the detected architecture - Code signs binaries with ad-hoc signatures (required for macOS security)
- Fetches
common.libsonnet/worker.jsonnet/runner.jsonnetverbatim from SSM—no host-side templating - Installs OpenTelemetry Collector (if endpoint provided) for metrics export
- Creates LaunchDaemons, injecting every setting
worker.jsonnet/runner.jsonnetneed as environment variables—bb-worker/bb-runnerresolve them from their own process environment at startup - Starts every runner service first, then waits for the scheduler to become DNS-resolvable and TCP-listening before starting
bb-worker(bounded retry, ~5 minutes by default—covers Cloud Map registration lag on a fresh deployment) - Performs health checks against
bb-workerand eachbb-runner
Installed Components
After successful initialization:- Binaries:
/usr/local/aspectbuild/bin/(bb-worker, bb-runner, otelcol) - Configs:
/usr/local/aspectbuild/config/(worker.jsonnet, runner.jsonnet) - Logs:
/usr/local/var/log/aspectbuild/(bb-worker.log, bb-runner.log) - LaunchDaemons:
/Library/LaunchDaemons/build.aspect.*.plist
Step 3: macOS Security Configuration
Service Management
Services always run via LaunchDaemon in thesystem launchctl domain (/Library/LaunchDaemons/), starting at boot with no interactive user session or auto-login required. bb-worker runs as root (required for the NFSv4 mount(2) call, mirroring the SYS_ADMIN capability Linux workers need for FUSE); each bb-runner runs as the _bb service account, created automatically by the init script.
Full Disk Access
Grant FDA to the binary files (not the user account):- Open System Settings → Privacy & Security → Full Disk Access
- Click + and add
/usr/local/aspectbuild/bin/bb-worker - Click + and add
/usr/local/aspectbuild/bin/bb-runner - Ensure both have checkmarks
bb-runner-{index} lines for each configured runner):
Additional Security Considerations
- Code signing: Binaries are automatically signed with ad-hoc signatures (
codesign -s -) - Quarantine removal:
xattr -crremoves quarantine flags from downloaded binaries - nobrowse mount: Required for LaunchDaemon access to storage volumes
iOS simulator prerequisites
Mac RBE workers that execute iOS simulator tests require additional one-time setup after worker initialization. Skipping these steps causes test actions to fail or time out on first execution.Install iOS simulator runtime
The iOS simulator runtime isn’t bundled with Xcode and must be downloaded separately. If it isn’t pre-installed, the first test run triggers an automatic download mid-build, which causes the action to time out.Full Disk Access for iOS test execution
bb-runner requires Full Disk Access to write action output files within the virtual filesystem during iOS test execution. Without it, test actions fail immediately with operation not permitted when the runner attempts to open stdout/stderr capture paths.
Ensure bb-runner is granted Full Disk Access as described in the Full Disk Access section preceding this one. This is required for all iOS simulator test actions, not just compilation.
Step 4: Configuration Details
Worker Configuration
The worker is configured with the following key settings:Runner Configuration
cleanTemporaryDirectories and cleanProcessTable are currently disabled on Darwin: enabling them caused worker readiness-check failures in prior hardware testing. bb-remote-execution has real Darwin implementations for both (native process-table scanning and directory cleanup), so this isn’t a known platform limitation—the root cause hasn’t been isolated yet.Communication Ports
Step 5: Verification and Monitoring
Health Checks
The init script automatically performs health checks. You can manually verify:Service Management
mac_init.sh always names runner services with an index suffix, even with NUM_RUNNERS=1—there is no unindexed build.aspect.bb-runner.plist.Storage Verification
Troubleshooting
Worker Fails to Start
Symptom: Worker process crashes or doesn’t appear inpgrep output.
Solutions:
- Check Full Disk Access permissions (most common issue)
- Verify storage is mounted at
/private/var/bb-storagewithnobrowseflag - Review error logs:
tail -20 /usr/local/var/log/aspectbuild/bb-worker.err - Ensure scheduler address is reachable:
nc -zv <scheduler_address> 8983—mac_init.shalready polls this automatically for up toSCHEDULER_WAIT_ATTEMPTS× 5 seconds (default ~5 minutes) before startingbb-worker; check its console output for polling attempts if setup seems to hang here
Permission Denied Errors
Symptom: Logs show “Permission denied” or “Operation not permitted” errors. Solutions:- Grant Full Disk Access to both
bb-workerandbb-runner - Verify storage mount has correct permissions:
ls -la /private/var/bb-storage - Ensure binaries are code-signed:
codesign -dv /usr/local/aspectbuild/bin/bb-worker
Storage Not Accessible to LaunchDaemon
Symptom: Worker starts but can’t access/private/var/bb-storage.
Solutions:
- Verify mount has
nobrowseflag:mount | grep nobrowse - Check
/etc/fstabentry exists for the volume UUID - Remount with nobrowse:
RAID Creation Fails
Symptom:mac_partition.sh fails when creating RAID.
Solutions:
- Ensure disks are completely unmounted:
sudo diskutil unmountDisk force /dev/diskN - Check for existing RAID:
diskutil appleRAID list - Use
FORCE_RECREATE=1to tear down existing RAID - Verify disks are not the boot disk or internal SSD
Worker Fails with mmap or “Operation not permitted” on CAS Files
Symptom: bb-worker.err shows an mmap or “operation not permitted” error referencing the CAS block files.
Cause: File-backed CAS (CAS_MODE=file) uses mmap on APFS files, which requires FileVault to be disabled.
Solutions:
- Ensure FileVault is disabled:
fdesetup status - Verify the RAID volume is mounted with
nobrowse:mount | grep bb-storage - Fall back to in-memory CAS: re-run with
CAS_MODE=memory sudo -E ./mac_init.sh ...
Health Check Timeout
Symptom: Init script times out waiting for services to respond. Solutions:- Check if runner started first:
pgrep -f bb-runner - Verify ports aren’t in use:
lsof -i :9980 -i :9981 - Review logs for startup errors
- Increase timeout by editing the
health_check()function
Metrics Not Appearing
Symptom: OTEL collector configured but metrics not reaching endpoint. Solutions:- Verify OTEL collector is running:
pgrep -f otelcol - Check OTEL logs:
tail -f /usr/local/var/log/aspectbuild/otel-collector.log - Verify endpoint is reachable:
nc -zv <otel_endpoint> 4317 - Check TLS settings in
/usr/local/aspectbuild/config/otel/config.yaml
EBS Volume Not Detected (AWS EC2)
Symptom: Script reports “No suitable drives found” after attaching EBS. Solutions:- REBOOT the instance - EBS hotplug requires a reboot on Mac instances
- After reboot, verify disk appears:
diskutil list physical external - If still not visible, check AWS console that volume is attached
- Ensure volume is attached to the correct instance
Platform Configuration for Mac Builds
After setting up Mac RBE workers, configure Bazel to target them.Create Mac Platform Definition
In your repository’splatforms/BUILD.bazel:
Update .bazelrc
Using Mac RBE
Advanced Configuration
Environment Variables
The following environment variables customizemac_init.sh behavior:
Custom Storage Sizing
Multiple Worker Instances
For testing different configurations on the same Mac:- Use different install directories:
INSTALL_DIR=/usr/local/aspectbuild-test - Use different storage paths:
STORAGE_BASE_DIR=/private/var/bb-storage-test - Change service names in LaunchDaemon plist files
Scaling Parallel Build Actions
Eachbb-runner process runs at concurrency 1 by design. To run more build actions in parallel, increase NUM_RUNNERS and re-run mac_init.sh—it creates one LaunchDaemon per runner, each with its own Unix socket and diagnostics port:
Hand-editing
/usr/local/aspectbuild/config/worker.jsonnet isn’t persistent—it’s fetched fresh from SSM every time mac_init.sh runs.References
- Script locations:
infrastructure/modules/workflows/remote/capacity/mac_partition.sh- Storage setup (infrastructure/modules/workflows/remote/capacity/mac_partition.sh:1)mac_init.sh- Worker initialization (infrastructure/modules/workflows/remote/capacity/mac_init.sh:1)
- BuildBarn documentation: bb-remote-execution
- macOS Launch Daemons: Creating Launch Daemons
- AWS EC2 Mac Instances: EC2 Mac Instances User Guide

