> ## Documentation Index
> Fetch the complete documentation index at: https://aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# Mac Remote Build Execution Setup

This guide provides step-by-step instructions for setting up Mac runners for Remote Build Execution (RBE) in Aspect Workflows. Mac RBE runners are optimized for building and testing macOS and iOS applications using Bazel.

## Overview

Mac RBE workers use a specialized storage architecture designed for macOS constraints:

* **RAID0 APFS volume** at `/private/var/bb-storage` for maximum throughput
* **File-backed Content Addressable Storage (CAS)** on RAID (100 GB block file, 400 MB key-location map by default), with an in-memory fallback for hosts where FileVault can't be disabled
* **File pool** for build execution scratch space, sized from `TOTAL_STORAGE_GB` (50 GB by default with one runner)
* **`bb-worker` runs as root** (required for the NFSv4 mount); **`bb-runner` process(es) run as a dedicated `_bb` service account** created automatically—no GUI session or auto-login required
* **EC2-native cluster membership**: Mac workers reach the storage shards and scheduler the same way Linux workers do, over the cluster's service-discovery namespace

## Hardware Requirements

### Supported Mac Hardware

* **AWS EC2 Mac instances** (production): mac1.metal (Intel), mac2.metal (M1), mac2-m2.metal, mac2-m2pro.metal—Mac RBE workers deploy as EC2-native cluster members on Dedicated Hosts
* **Architecture**: Both Intel (x86\_64/amd64) and Apple Silicon (arm64/aarch64) are supported

> [!NOTE]
>
> This guide covers the AWS EC2 deployment path, which `mac_init.sh` assumes. Local Mac hardware (Mac mini, Mac Studio, Mac Pro) can be used for isolated development or testing against a standalone BuildBarn stack, but that's a separate, unsupported workflow outside the scope of this guide.

### Storage Requirements

* **Minimum**: 700 GB available space
* **Recommended**: External EBS volumes (AWS) or Thunderbolt/USB drives with 1 TB+ capacity
* **Multiple drives**: Automatically configured as RAID0 for maximum performance
* **Single drive**: Works but provides lower throughput than RAID configuration

> [!WARNING]
>
> On AWS EC2 Mac instances, if you attach an EBS volume while the instance is running, you **MUST REBOOT** before the volume becomes visible. EBS hotplug on Mac instances requires a reboot.

## AWS EC2 Deployment Prerequisites

When deploying Mac RBE workers on AWS EC2 via the Aspect Workflows Terraform module, several AWS account-level resources must exist before the first `terraform apply`.

### 1. AWS License Manager Service-Linked Role

AWS License Manager requires a service-linked role to manage Dedicated Hosts on your behalf. This is a one-time setup per AWS account:

```bash theme={null}
aws iam create-service-linked-role --aws-service-name license-manager.amazonaws.com
```

If the role already exists, this command returns an error that can be safely ignored.

### 2. License Manager License Configuration

AWS requires an explicit License Manager license configuration to be associated with both the Host Resource Group and the launch template. Create a minimal one with no enforcement limits:

```bash theme={null}
aws license-manager create-license-configuration \
  --name "mac2-metal" \
  --license-counting-type "Core" \
  --license-count 0 \
  --no-license-count-hard-limit
```

Note the `LicenseConfigurationArn` from the output—you'll need it in the next two steps.

### 3. EC2 Dedicated Host Resource Group

Mac instances on AWS require [Dedicated Host tenancy](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-hosts-overview.html). For Auto Scaling Groups, a **Host Resource Group** is required so that AWS can automatically allocate and release Dedicated Hosts as the ASG scales.

Create a Host Resource Group referencing the license configuration from the previous step (adjust `allowed-host-families` for `mac1`, `mac2`, `mac2-m2`, etc.):

```bash theme={null}
aws resource-groups create-group \
  --name mac2-metal-hosts \
  --configuration '[
    {
      "Type": "AWS::EC2::HostManagement",
      "Parameters": [
        {"Name": "allowed-host-families",                    "Values": ["mac2"]},
        {"Name": "auto-allocate-host",                       "Values": ["true"]},
        {"Name": "auto-release-host",                        "Values": ["true"]},
        {"Name": "allowed-host-based-license-configurations","Values": ["<license-configuration-arn>"]}
      ]
    },
    {
      "Type": "AWS::ResourceGroups::Generic",
      "Parameters": [
        {"Name": "allowed-resource-types", "Values": ["AWS::EC2::Host"]},
        {"Name": "deletion-protection",    "Values": ["UNLESS_EMPTY"]}
      ]
    }
  ]'
```

> [!NOTE]
>
> `auto-allocate-host` and `auto-release-host` allow AWS to provision and release Dedicated Hosts automatically as the ASG scales. This is required because mac instances have a **24-hour minimum allocation period**—hosts cannot be released sooner regardless of whether an instance is running on them.

Note the `GroupArn` returned from this command—you'll need it in the next step.

### 4. Terraform Configuration

Pass both ARNs in the executor's `ec2` block. The `license_configuration_arn` must match what is associated with the Host Resource Group, or AWS will reject the launch template.

```hcl theme={null}
execution = {
  executors = {
    mac = {
      platform = "Darwin"
      workers = [{
        scaling = { minimum = 0, maximum = 2 }
        max_concurrency = 4
        ec2 = {
          instance_type             = "mac2.metal"
          instance_image            = "<your-macos-ami-id>"
          host_resource_group_arn   = "<group-arn-from-step-3>"
          license_configuration_arn = "<license-arn-from-step-2>"
        }
      }]
    }
  }
}
```

To look up the latest AWS macOS AMI for your region:

```bash theme={null}
aws ec2 describe-images --owners amazon \
  --filters "Name=name,Values=amzn-ec2-macos-*" \
            "Name=architecture,Values=arm64_mac" \
  --query 'sort_by(Images,&CreationDate)[-1].ImageId'
```

> [!WARNING]
>
> macOS AMIs must be explicitly provided—there is no SSM-managed default AMI path for Mac instances, unlike Linux and Windows. The `instance_image` field is required when `platform = "Darwin"`.

> [!WARNING]
>
> `mac2.metal` (Apple Silicon) Dedicated Hosts are scarce in most AWS regions. Set `minimum = 0` initially—the ASG infrastructure will deploy but no host will be allocated. Request capacity via an AWS Support ticket before scaling up. `mac1.metal` (Intel) is generally more available.

## Step 1: Storage Setup

The storage setup script prepares external drives for RBE workloads using RAID0 APFS volumes.

### Running the Partition Script

The `mac_partition.sh` script automatically detects and configures external drives:

```bash theme={null}
# Auto-detect external/EBS drives and configure storage
sudo ./mac_partition.sh

# Specify specific disks (replace diskN with actual disk identifiers)
TARGET_DISKS="disk2 disk3" sudo ./mac_partition.sh

# Dry run to preview changes without modifying disks
DRY_RUN=1 ./mac_partition.sh

# Force recreation of existing RAID (WARNING: destroys data)
FORCE_RECREATE=1 sudo ./mac_partition.sh
```

### What the Script Does

1. **Detects external drives** using `diskutil list physical external` (AWS-recommended method)
2. **Safety checks**: Excludes disk0 (boot disk) and internal Apple SSDs
3. **Creates RAID0** if multiple drives are detected, or formats a single drive as APFS
4. **Mounts with `nobrowse` flag** at `/private/var/bb-storage` for LaunchDaemon access
5. **Creates build directories**: `worker/build` and `worker/cache`
6. **Disables Spotlight indexing** on the RAID volume
7. **Adds `/etc/fstab` entry** for automatic mounting on boot

### Storage Architecture

`mac_partition.sh` creates the following layout on the RAID volume:

```
/private/var/bb-storage/          # RAID0 APFS volume (nobrowse mount)
└── worker/
    ├── build/
    └── cache/
```

`mac_init.sh` populates it with the CAS storage and filepool `worker.jsonnet` actually reads from:

```
/private/var/bb-storage/
├── cas/
│   ├── blocks                    # CAS block file (CAS_MODE=file only)
│   └── key-location-map
└── worker/
    └── filepool.block            # Build execution scratch space
```

> [!NOTE]
>
> The active build directory itself is a separate NFSv4 virtual mount at `/var/build/root`, not a path under `/private/var/bb-storage`.

> [!NOTE]
>
> **Why `/private/var/` instead of `/Volumes/`?**
>
> macOS DiskArbitration mounts volumes in `/Volumes/` within user sessions, making them inaccessible to LaunchDaemons (system-level services). The `nobrowse` mount flag combined with `/private/var/` ensures the storage is accessible to root system processes at boot time.

### Environment Variables

| Variable | Description | Default |
| :- | :- | :- |
| `TARGET_DISKS` | Space-separated disk identifiers (for example, "disk2 disk3") | Auto-detect |
| `DRY_RUN` | Set to 1 to preview changes without applying | 0 |
| `SKIP_CONFIRM` | Set to 1 to skip confirmation prompts (dangerous!) | 0 |
| `FORCE_RECREATE` | Set to 1 to tear down existing RAID and recreate | 0 |
| `STORAGE_MOUNT_PATH` | Target mount path for APFS volume | `/private/var/bb-storage` |
| `BUILD_CACHE_PCT` | Percentage of RAID for build cache | 74 (yields \~700 GB on 953 GB RAID) |

## Step 2: Worker Initialization

After storage is configured, initialize the RBE worker components using the `mac_init.sh` script.

### Running the Init Script

`mac_init.sh` fetches its configuration verbatim from shared SSM parameters—the same `common.libsonnet`/`worker.jsonnet`/`runner.jsonnet` artifacts Linux workers use—rather than rendering Mac-specific templates. A handful of environment variables must be set in addition to its positional arguments; your deployment's Terraform configuration provisions the actual SSM parameter paths.

```bash theme={null}
sudo COMMON_CONFIG_SSM_PATH=<path> WORKER_CONFIG_SSM_PATH=<path> RUNNER_CONFIG_SSM_PATH=<path> \
     WORKER_COMMON_ENV="NUM_SHARDS=<n> INFIX=<infix> COMMON_NAMESPACE=<namespace> MIRRORED=<0|1> ISCC_ENABLED=<0|1>" \
     ./mac_init.sh <bb_scheduler_address> <namespace> [otel_collector_endpoint] [use_tls]
```

**Required Arguments:**

* `bb_scheduler_address`: Scheduler endpoint (for example, `scheduler.workflows.internal`)—used to gate worker startup until the scheduler is reachable
* `namespace`: The cluster's service-discovery namespace (for example, `workflows.internal`)—Mac workers are EC2-native cluster members and reach the scheduler and storage shards via Cloud Map service-discovery names built from this namespace, the same as Linux workers

**Optional Arguments:**

* `otel_collector_endpoint`: OpenTelemetry collector endpoint for metrics
* `use_tls`: Set to `1` for TLS, `0` for plaintext (auto-detected from `bb_scheduler_address` if not specified)

**Required Environment Variables:**

* `COMMON_CONFIG_SSM_PATH`, `WORKER_CONFIG_SSM_PATH`, `RUNNER_CONFIG_SSM_PATH`: SSM parameter paths for the shared config files
* `WORKER_COMMON_ENV`: space-separated `KEY=VALUE` pairs carrying cluster-topology settings (`NUM_SHARDS`, `INFIX`, `COMMON_NAMESPACE`, `MIRRORED`, `ISCC_ENABLED`) the worker needs to address storage shards directly

### Example

```bash theme={null}
# With auto-detected TLS (uses TLS for non-internal addresses)
sudo COMMON_CONFIG_SSM_PATH=/aw/mycluster/common.libsonnet \
     WORKER_CONFIG_SSM_PATH=/aw/mycluster/worker/worker.jsonnet \
     RUNNER_CONFIG_SSM_PATH=/aw/mycluster/worker/runner.jsonnet \
     WORKER_COMMON_ENV="NUM_SHARDS=3 INFIX=remote COMMON_NAMESPACE=workflows.internal MIRRORED=0 ISCC_ENABLED=0" \
     ./mac_init.sh scheduler.workflows.aspect.build workflows.internal

# With explicit TLS disabled (internal network)—the same env vars are required but omitted below for brevity
sudo COMMON_CONFIG_SSM_PATH=... WORKER_CONFIG_SSM_PATH=... RUNNER_CONFIG_SSM_PATH=... WORKER_COMMON_ENV="..." \
     ./mac_init.sh scheduler.internal workflows.internal "" 0

# With OpenTelemetry metrics collection
sudo COMMON_CONFIG_SSM_PATH=... WORKER_CONFIG_SSM_PATH=... RUNNER_CONFIG_SSM_PATH=... WORKER_COMMON_ENV="..." \
     ./mac_init.sh scheduler.workflows.aspect.build workflows.internal otel-collector.internal
```

### What the Init Script Does

1. **Creates the `_bb` service account** using `dscl`—a hidden system service user (UID 305+) with no shell and no home directory, following the macOS convention for service accounts like `_www` and `_mysql`. `bb-worker` itself runs as **root** (required for the NFSv4 mount); `bb-runner` process(es) run as `_bb`
2. **Verifies storage** is mounted at `/private/var/bb-storage`
3. **Pre-allocates CAS block files** on RAID (skipped entirely when `CAS_MODE=memory`)
4. **Downloads BuildBarn binaries**: `bb-worker` and `bb-runner` for the detected architecture
5. **Code signs binaries** with ad-hoc signatures (required for macOS security)
6. **Fetches `common.libsonnet`/`worker.jsonnet`/`runner.jsonnet` verbatim** from SSM—no host-side templating
7. **Installs OpenTelemetry Collector** (if endpoint provided) for metrics export
8. **Creates LaunchDaemons**, injecting every setting `worker.jsonnet`/`runner.jsonnet` need as environment variables—`bb-worker`/`bb-runner` resolve them from their own process environment at startup
9. **Starts every runner service first**, then **waits for the scheduler** to become DNS-resolvable and TCP-listening before starting `bb-worker` (bounded retry, \~5 minutes by default—covers Cloud Map registration lag on a fresh deployment)
10. **Performs health checks** against `bb-worker` and each `bb-runner`

### Installed Components

After successful initialization:

* **Binaries**: `/usr/local/aspectbuild/bin/` (bb-worker, bb-runner, otelcol)
* **Configs**: `/usr/local/aspectbuild/config/` (worker.jsonnet, runner.jsonnet)
* **Logs**: `/usr/local/var/log/aspectbuild/` (bb-worker.log, bb-runner.log)
* **LaunchDaemons**: `/Library/LaunchDaemons/build.aspect.*.plist`

## Step 3: macOS Security Configuration

### Service Management

Services always run via **LaunchDaemon** in the `system` launchctl domain (`/Library/LaunchDaemons/`), starting at boot with no interactive user session or auto-login required. `bb-worker` runs as **root** (required for the NFSv4 `mount(2)` call, mirroring the `SYS_ADMIN` capability Linux workers need for FUSE); each `bb-runner` runs as the `_bb` service account, created automatically by the init script.

### Full Disk Access

> [!WARNING]
>
> Both `bb-worker` and `bb-runner` require Full Disk Access (FDA). The init script prints the exact steps after completing setup.

Grant FDA to the **binary files** (not the user account):

1. Open **System Settings** → **Privacy & Security** → **Full Disk Access**
2. Click **+** and add `/usr/local/aspectbuild/bin/bb-worker`
3. Click **+** and add `/usr/local/aspectbuild/bin/bb-runner`
4. Ensure both have checkmarks

After granting FDA, restart the services (repeat the `bb-runner-{index}` lines for each configured runner):

```bash theme={null}
sudo launchctl bootout system/build.aspect.bb-worker
sudo launchctl bootout system/build.aspect.bb-runner-0
sudo launchctl bootstrap system /Library/LaunchDaemons/build.aspect.bb-runner-0.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/build.aspect.bb-worker.plist
```

### Additional Security Considerations

* **Code signing**: Binaries are automatically signed with ad-hoc signatures (`codesign -s -`)
* **Quarantine removal**: `xattr -cr` removes quarantine flags from downloaded binaries
* **nobrowse mount**: Required for LaunchDaemon access to storage volumes

## iOS simulator prerequisites

Mac RBE workers that execute iOS simulator tests require additional one-time setup after worker initialization. Skipping these steps causes test actions to fail or time out on first execution.

### Install iOS simulator runtime

The iOS simulator runtime isn't bundled with Xcode and must be downloaded separately. If it isn't pre-installed, the first test run triggers an automatic download mid-build, which causes the action to time out.

> [!WARNING]
>
> Run this on the worker Mac **before** the worker starts serving test actions. The download is several gigabytes and can take 10–30 minutes depending on network speed.

```bash theme={null}
xcodebuild -downloadPlatform iOS
```

Verify the runtime installed successfully:

```bash theme={null}
xcrun simctl list runtimes | grep iOS
```

Expected output:

```
iOS 18.x (18.x - 22Fxxx) - com.apple.CoreSimulator.SimRuntime.iOS-18-x
```

### Full Disk Access for iOS test execution

`bb-runner` requires Full Disk Access to write action output files within the virtual filesystem during iOS test execution. Without it, test actions fail immediately with `operation not permitted` when the runner attempts to open stdout/stderr capture paths.

Ensure `bb-runner` is granted Full Disk Access as described in the [Full Disk Access](#full-disk-access) section preceding this one. This is required for all iOS simulator test actions, not just compilation.

## Step 4: Configuration Details

### Worker Configuration

The worker is configured with the following key settings:

| Setting | Value | Description |
| :- | :- | :- |
| **CAS Backend** | File-backed (default) or in-memory | `CAS_MODE=file` (default: 100 GB block file + 400 MB key-location map) or `CAS_MODE=memory` (FileVault fallback) |
| **File Pool** | RAID filesystem | Scratch space for build execution, sized from `TOTAL_STORAGE_GB` / `NUM_RUNNERS` |
| **Download Concurrency** | 16 | Parallel input downloads (`MAX_DOWNLOAD_CONCURRENCY`) |
| **Upload Concurrency** | 16 | Parallel output uploads (`MAX_UPLOAD_CONCURRENCY`) |
| **Runner Count** | 1 (configurable) | Number of parallel `bb-runner` processes (`NUM_RUNNERS`); each runs at concurrency 1 |
| **Cache Replacement** | Least Recently Used | Eviction policy for the directory cache and file-backed CAS integrity cache |
| **Diagnostics Port** | 9980 | HTTP endpoint for `bb-worker` metrics and health |

### Runner Configuration

| Setting | Value | Description |
| :- | :- | :- |
| **Worker ↔ Runner IPC** | Unix socket | `/usr/local/aspectbuild/runner-{index}.sock`, one per runner |
| **Diagnostics Port** | 9981 + index | Each runner gets its own port—`bb-worker` and one or more `bb-runner`s share a single host and can't all bind the same port |
| **Temp Directory Handling** | `setTmpdirEnvironmentVariable` | Redirects `TMPDIR` into the per-action build directory, auto-cleaned by `bb-worker` |
| **Xcode Support** | Enabled | `appleXcodeDeveloperDirectories`, discovered automatically from installed Xcode versions |

> [!NOTE]
>
> `cleanTemporaryDirectories` and `cleanProcessTable` are currently disabled on Darwin: enabling them caused worker readiness-check failures in prior hardware testing. `bb-remote-execution` has real Darwin implementations for both (native process-table scanning and directory cleanup), so this isn't a known platform limitation—the root cause hasn't been isolated yet.

### Communication Ports

| Component | Port | Protocol | Description |
| :- | :- | :- | :- |
| Worker → Scheduler | 8983 | gRPC | Work assignment and coordination; the worker also polls this port at boot to gate startup on scheduler readiness |
| Worker → Storage Shards | 8981 | gRPC | Direct, sharded CAS and action cache access—same as Linux workers |
| Storage Shard Replication | 8982 | gRPC | Inter-shard replication |
| Worker ↔ Runners | — | Unix socket | `/usr/local/aspectbuild/runner-{index}.sock` |
| Worker Diagnostics | 9980 | HTTP | Prometheus metrics, health checks |
| Runner Diagnostics | 9981 + index | HTTP | Prometheus metrics, health checks (one port per runner) |
| OTEL Collector → Endpoint | 4317 | gRPC/OTLP | Telemetry export |

## Step 5: Verification and Monitoring

### Health Checks

The init script automatically performs health checks. You can manually verify:

```bash theme={null}
# Check if services are running
pgrep -f bb-worker
pgrep -f bb-runner   # matches all runner processes regardless of index

# Check worker metrics endpoint
curl http://localhost:9980/metrics

# Check runner metrics endpoint(s)—one per runner index (9981, 9982, ... for NUM_RUNNERS > 1)
curl http://localhost:9981/metrics

# View worker logs
tail -f /usr/local/var/log/aspectbuild/bb-worker.log

# View runner error logs (one file per runner index, e.g. bb-runner-0.err)
tail -f /usr/local/var/log/aspectbuild/bb-runner-0.err
```

### Service Management

> [!NOTE]
>
> `mac_init.sh` always names runner services with an index suffix, even with `NUM_RUNNERS=1`—there is no unindexed `build.aspect.bb-runner.plist`.

```bash theme={null}
# Check service status
sudo launchctl print system/build.aspect.bb-worker
sudo launchctl print system/build.aspect.bb-runner-0

# Stop services
sudo launchctl bootout system/build.aspect.bb-worker
sudo launchctl bootout system/build.aspect.bb-runner-0

# Start services
sudo launchctl bootstrap system /Library/LaunchDaemons/build.aspect.bb-worker.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/build.aspect.bb-runner-0.plist

# Restart services (after config changes)
sudo launchctl bootout system/build.aspect.bb-worker
sudo launchctl bootstrap system /Library/LaunchDaemons/build.aspect.bb-worker.plist
```

### Storage Verification

```bash theme={null}
# Verify RAID mount
mount | grep bb-storage

# Should show: /dev/diskN on /private/var/bb-storage (apfs, local, nobrowse, ...)

# Check available space
df -h /private/var/bb-storage

# Verify build directories
ls -la /private/var/bb-storage/worker/
```

## Troubleshooting

### Worker Fails to Start

**Symptom**: Worker process crashes or doesn't appear in `pgrep` output.

**Solutions**:

1. Check Full Disk Access permissions (most common issue)
2. Verify storage is mounted at `/private/var/bb-storage` with `nobrowse` flag
3. Review error logs: `tail -20 /usr/local/var/log/aspectbuild/bb-worker.err`
4. Ensure scheduler address is reachable: `nc -zv <scheduler_address> 8983`—`mac_init.sh` already polls this automatically for up to `SCHEDULER_WAIT_ATTEMPTS` × 5 seconds (default \~5 minutes) before starting `bb-worker`; check its console output for polling attempts if setup seems to hang here

### Permission Denied Errors

**Symptom**: Logs show "Permission denied" or "Operation not permitted" errors.

**Solutions**:

1. Grant Full Disk Access to both `bb-worker` and `bb-runner`
2. Verify storage mount has correct permissions: `ls -la /private/var/bb-storage`
3. Ensure binaries are code-signed: `codesign -dv /usr/local/aspectbuild/bin/bb-worker`

### Storage Not Accessible to LaunchDaemon

**Symptom**: Worker starts but can't access `/private/var/bb-storage`.

**Solutions**:

1. Verify mount has `nobrowse` flag: `mount | grep nobrowse`
2. Check `/etc/fstab` entry exists for the volume UUID
3. Remount with nobrowse:
   ```bash theme={null}
   sudo umount /private/var/bb-storage
   sudo mount -t apfs -o nobrowse <device> /private/var/bb-storage
   ```

### RAID Creation Fails

**Symptom**: `mac_partition.sh` fails when creating RAID.

**Solutions**:

1. Ensure disks are completely unmounted: `sudo diskutil unmountDisk force /dev/diskN`
2. Check for existing RAID: `diskutil appleRAID list`
3. Use `FORCE_RECREATE=1` to tear down existing RAID
4. Verify disks are not the boot disk or internal SSD

### Worker Fails with `mmap` or "Operation not permitted" on CAS Files

**Symptom**: `bb-worker.err` shows an `mmap` or "operation not permitted" error referencing the CAS block files.

**Cause**: File-backed CAS (`CAS_MODE=file`) uses `mmap` on APFS files, which requires FileVault to be disabled.

**Solutions**:

1. Ensure FileVault is disabled: `fdesetup status`
2. Verify the RAID volume is mounted with `nobrowse`: `mount | grep bb-storage`
3. Fall back to in-memory CAS: re-run with `CAS_MODE=memory sudo -E ./mac_init.sh ...`

### Health Check Timeout

**Symptom**: Init script times out waiting for services to respond.

**Solutions**:

1. Check if runner started first: `pgrep -f bb-runner`
2. Verify ports aren't in use: `lsof -i :9980 -i :9981`
3. Review logs for startup errors
4. Increase timeout by editing the `health_check()` function

### Metrics Not Appearing

**Symptom**: OTEL collector configured but metrics not reaching endpoint.

**Solutions**:

1. Verify OTEL collector is running: `pgrep -f otelcol`
2. Check OTEL logs: `tail -f /usr/local/var/log/aspectbuild/otel-collector.log`
3. Verify endpoint is reachable: `nc -zv <otel_endpoint> 4317`
4. Check TLS settings in `/usr/local/aspectbuild/config/otel/config.yaml`

### EBS Volume Not Detected (AWS EC2)

**Symptom**: Script reports "No suitable drives found" after attaching EBS.

**Solutions**:

1. **REBOOT the instance** - EBS hotplug requires a reboot on Mac instances
2. After reboot, verify disk appears: `diskutil list physical external`
3. If still not visible, check AWS console that volume is attached
4. Ensure volume is attached to the correct instance

## Platform Configuration for Mac Builds

After setting up Mac RBE workers, configure Bazel to target them.

### Create Mac Platform Definition

In your repository's `platforms/BUILD.bazel`:

```bazel theme={null}
platform(
    name = "mac_rbe",
    exec_properties = {
        "OSFamily": "Darwin",
        "cpu": "arm64",  # or "amd64" for Intel Macs
    },
)
```

### Update `.bazelrc`

```bazel theme={null}
# Mac Remote Execution configuration
build:mac-rbe --host_platform=//platforms:mac_rbe
build:mac-rbe --extra_execution_platforms=//platforms:mac_rbe
build:mac-rbe --jobs=50
build:mac-rbe --remote_executor=grpcs://<your-scheduler-endpoint>:8983
build:mac-rbe --remote_timeout=3600
```

### Using Mac RBE

```bash theme={null}
# Build with Mac RBE
bazel build --config=mac-rbe //your/ios:app

# Test with Mac RBE
bazel test --config=mac-rbe //your/macos:tests
```

## Advanced Configuration

### Environment Variables

The following environment variables customize `mac_init.sh` behavior:

| Variable | Description | Default |
| :- | :- | :- |
| `BB_SERVICE_USER` | Service account name for `bb-runner` process(es) | `_bb` |
| `INSTALL_DIR` | Installation directory for binaries and configs | `/usr/local/aspectbuild` |
| `STORAGE_BASE_DIR` | Base path for RAID storage | `/private/var/bb-storage` |
| `CAS_MODE` | CAS backend: `file` (default) or `memory` (FileVault fallback) | `file` |
| `CAS_FILE_SIZE_BYTES` | CAS block file size in bytes (file mode only) | `107374182400` (100 GiB) |
| `CAS_KEYMAP_SIZE_BYTES` | CAS key-location map size in bytes (file mode only) | `419430400` (400 MiB) |
| `FILEPOOL_SIZE_BYTES` | Filepool block file size in bytes, pre-allocated on disk | `53687091200` (50 GiB) |
| `NUM_RUNNERS` | Number of parallel `bb-runner` processes | `1` |
| `TOTAL_STORAGE_GB` | Drives `worker.jsonnet`'s CAS block-count math and filePool sizing—keep consistent with `FILEPOOL_SIZE_BYTES`/`CAS_FILE_SIZE_BYTES` above | `100` |
| `MAX_BLOB_SIZE_GB` | Drives CAS block-count math | `10` |
| `MAX_DOWNLOAD_CONCURRENCY` | Parallel input downloads | `16` |
| `MAX_UPLOAD_CONCURRENCY` | Parallel output uploads | `16` |
| `RUNNER_ENVIRONMENT` | JSON object of extra environment variables passed to build actions | `{}` |
| `WORKER_COMMON_ENV` | Space-separated `KEY=VALUE` cluster-topology extVars (`NUM_SHARDS`, `INFIX`, `COMMON_NAMESPACE`, `MIRRORED`, `ISCC_ENABLED`) | *(required, no default)* |
| `SCHEDULER_WAIT_ATTEMPTS` | Number of 5-second polls waiting for the scheduler before giving up | `60` (\~5 minutes) |
| `BROWSER_URL` | Build results browser (`bb-browser`) URL | *(empty)* |
| `COMMON_CONFIG_SSM_PATH` | SSM parameter path for `common.libsonnet` | *(required, no default)* |
| `WORKER_CONFIG_SSM_PATH` | SSM parameter path for `worker.jsonnet` | *(required, no default)* |
| `RUNNER_CONFIG_SSM_PATH` | SSM parameter path for `runner.jsonnet` | *(required, no default)* |
| `OTEL_CONFIG_SSM_PATH` | SSM parameter path for the OTEL collector config template | *(used only if `otel_collector_endpoint` is provided)* |

> [!WARNING]
>
> `TOTAL_STORAGE_GB`/`MAX_BLOB_SIZE_GB` (used by `worker.jsonnet`'s sizing math) and `CAS_FILE_SIZE_BYTES`/`FILEPOOL_SIZE_BYTES` (what's actually pre-allocated on disk) are configured independently. Mismatched values won't error, but the worker's expected sizes can silently diverge from what's on disk—keep them consistent when changing `NUM_RUNNERS` or storage sizing.

### Custom Storage Sizing

```bash theme={null}
# Increase the filepool to 500GB and keep TOTAL_STORAGE_GB consistent
export FILEPOOL_SIZE_BYTES=536870912000
export TOTAL_STORAGE_GB=1000
sudo -E ./mac_init.sh <scheduler> <namespace>
```

### Multiple Worker Instances

For testing different configurations on the same Mac:

1. Use different install directories: `INSTALL_DIR=/usr/local/aspectbuild-test`
2. Use different storage paths: `STORAGE_BASE_DIR=/private/var/bb-storage-test`
3. Change service names in LaunchDaemon plist files

> [!WARNING]
>
> Diagnostics ports are fixed (`bb-worker`: `9980`, each `bb-runner`: `9981 + index`)—two independent worker instances on the same host will conflict on these ports.

### Scaling Parallel Build Actions

Each `bb-runner` process runs at concurrency 1 by design. To run more build actions in parallel, increase `NUM_RUNNERS` and re-run `mac_init.sh`—it creates one LaunchDaemon per runner, each with its own Unix socket and diagnostics port:

```bash theme={null}
sudo NUM_RUNNERS=4 ./mac_init.sh <scheduler> <namespace>
```

> [!NOTE]
>
> Hand-editing `/usr/local/aspectbuild/config/worker.jsonnet` isn't persistent—it's fetched fresh from SSM every time `mac_init.sh` runs.

## References

* **Script locations**: `infrastructure/modules/workflows/remote/capacity/`
  * `mac_partition.sh` - Storage setup (infrastructure/modules/workflows/remote/capacity/mac\_partition.sh:1)
  * `mac_init.sh` - Worker initialization (infrastructure/modules/workflows/remote/capacity/mac\_init.sh:1)
* **BuildBarn documentation**: [bb-remote-execution](https://github.com/buildbarn/bb-remote-execution)
* **macOS Launch Daemons**: [Creating Launch Daemons](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html)
* **AWS EC2 Mac Instances**: [EC2 Mac Instances User Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.