> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# Parallelize remote execution

> Bazel remote execution tuning: avoid client-side throttling so remote execution workers run more actions in parallel.

With high parallelism on remote execution, Bazel can throttle execution on the client side while remote capacity sits idle: hundreds of actions ready to run, and a small fraction executing.

## Symptoms

You're likely hitting this if you see:

* Many actions ready/queued, for example, 768 actions ready.
* Only a small number actually running, for example, 72 actions running.
* Abundant remote executor capacity, for example, 500+ container instances available.
* Build profiles showing `acquiringSemaphore` traces.
* Slower builds than expected despite high `--jobs` values.

## Prerequisites

* A working [remote execution](/docs/aspect-workflows/platform/features/remote-execution) setup.
* Bazel 7.0 or later.
* High parallelism (for example, `--jobs` set to a multiple of host CPUs).

## Diagnosing parallelization issues

To find out whether client-side throttling is limiting your build:

### Step 1: Generate a build profile

```bash theme={null}
bazel build //... --profile=profile.json
```

### Step 2: Force a rebuild

To avoid cache hits when profiling:

```bash theme={null}
bazel build //... \
  --profile=profile.json \
  --action_env=CACHE_BUST=$(date +%s)
```

### Step 3: Inspect the profile

Look for `acquiringSemaphore` traces. They mean Bazel is waiting on internal semaphores rather than using the remote executors.

## Configuration

These Bazel flags reduce client-side throttling and raise concurrency. Put them in your workspace `.bazelrc`, in the `rbe` group that registers your exec platforms, so they apply only when a build executes remotely. [RBE platforms](/docs/aspect-workflows/platform/guides/remote-execution-worker-pools#the-bazel-side) shows how to turn that group on from the `--config` group [`aspect setup bazelrc`](/docs/cli/tasks/setup_bazelrc) writes, or from an Aspect CLI task.

### Bazel flags

```bazel theme={null}
# Disable client-side throttling for remote builds
build:rbe --noexperimental_throttle_remote_action_building
build:rbe --noexperimental_throttle_action_cache_check

# Reduce client memory for Merkle trees
build:rbe --experimental_remote_discard_merkle_trees

# Remove additional semaphore bottlenecks
build:rbe --noexperimental_use_semaphore_for_jobs

# Address Bazel 7 Skymeld regression
build:rbe --noexperimental_merged_skyframe_analysis_execution
```

What each flag does:

| Flag                                                  | Description                                                                                                                                                                                                                                                                                                                              |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--noexperimental_throttle_remote_action_building`    | Disables the `remoteActionBuildingSemaphore`, which can significantly slow down highly parallel remote builds. It keeps a client from overwhelming the remote executors, and becomes a bottleneck when you have substantial remote capacity. <br /> <br />See [bazelbuild/bazel#20478](https://github.com/bazelbuild/bazel/issues/20478) |
| `--noexperimental_throttle_action_cache_check`        | Removes throttling when checking the action cache, so more cache lookups run in parallel.                                                                                                                                                                                                                                                |
| `--experimental_remote_discard_merkle_trees`          | Discards Merkle tree data structures once they're no longer needed, reducing client-side memory.                                                                                                                                                                                                                                         |
| `--noexperimental_use_semaphore_for_jobs`             | Disables an internal semaphore that can limit job concurrency.                                                                                                                                                                                                                                                                           |
| `--noexperimental_merged_skyframe_analysis_execution` | Disables Skymeld, which merges the analysis and execution phases. Skymeld has a known regression in Bazel 7 that slows remote execution down to actions running one at a time. <br /><br /> See [bazelbuild/bazel#22233](https://github.com/bazelbuild/bazel/issues/22233)                                                               |

### Memory flags that cost incrementality

If the Bazel client runs out of memory at high parallelism, these flags reduce what the Bazel server holds:

| Flag                          | What it discards                                             |
| ----------------------------- | ------------------------------------------------------------ |
| `--heuristically_drop_nodes`  | Skyframe nodes Bazel judges it won't need again              |
| `--notrack_incremental_state` | The dependency graph Bazel uses to rebuild incrementally     |
| `--nokeep_state_after_build`  | The analysis cache and build state, once the build completes |

They cost warm incrementality: the next build on the same Bazel server re-analyzes from scratch. On [Workflows runners](/docs/aspect-workflows/platform/features/ci-runners), which keep the Bazel server and its analysis cache warm between jobs, that gives up most of what a warm runner saves. Use them only where the server isn't reused, and fix memory with a larger client or `--host_jvm_args` first.

### Job parallelism

A common pattern is a multiple of the host CPU count:

```bazel theme={null}
# Example: for a 24-core machine, 32 * 24 host CPUs
build:rbe --jobs=768
```

The exact multiplier depends on your workload characteristics and remote executor capacity.

## Test and validation

To check the flags help:

1. Run a clean build with profiling enabled:
   ```bash theme={null}
   bazel clean
   bazel build //... \
     --config=aspect-exec \
     --profile=profile-optimized.json \
     --action_env=CACHE_BUST=$(date +%s)
   ```
2. Check the build output to confirm more actions are running concurrently.
3. Analyze the profile to verify that `acquiringSemaphore` traces are fewer or gone.
4. Compare build times before and after applying these flags.

## Before you keep a flag

* **Add flags one at a time.** Profile after each and drop any that doesn't help your workload, or that breaks the build.
* **Check your Bazel version.** Flag behavior and support vary between Bazel releases; the release notes list known issues with experimental flags.
* **Check remote capacity.** These flags remove client-side bottlenecks; the worker pool still needs the capacity for the extra actions in flight.