> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# Fixing Bazel out-of-memory problems

> Troubleshoot and fix Bazel's out-of-memory issues in both JVM and system contexts with practical solutions and tips.

export const BlogPost = ({title, date, authors, tags, image, children}) => {
  const tagList = tags ? tags.split(", ").filter(Boolean) : [];
  const tagSlug = t => t.toLowerCase().replace(/&/g, "").replace(/\+/g, "").replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
  const formattedDate = date ? new Date(date + "T00:00:00").toLocaleDateString("en-US", {
    year: "numeric",
    month: "long",
    day: "numeric"
  }) : "";
  return <section className="w-full flex justify-center px-4 py-12 md:py-16">
      <div style={{
    maxWidth: "800px",
    width: "100%"
  }}>
        {image && (typeof image === "string" ? <img noZoom src={image} alt={title} className="w-full rounded-xl mb-8" style={{
    maxHeight: "400px",
    objectFit: "cover"
  }} /> : <div className="blog-post-hero-image">{image}</div>)}
        <h1 className="text-3xl md:text-4xl font-bold text-zinc-900 dark:text-white">
          {title}
        </h1>
        <div className="flex flex-wrap items-center gap-3 mt-4 text-sm text-zinc-500 dark:text-zinc-400">
          {authors && <span>{authors}</span>}
          {authors && formattedDate && <span>·</span>}
          {formattedDate && <span>{formattedDate}</span>}
        </div>
        {tagList.length > 0 && <div className="flex flex-wrap gap-2 mt-3">
            {tagList.map(tag => <a key={tag} href={"/blog/tags/" + tagSlug(tag)} className="px-2 py-0.5 rounded-full text-xs bg-zinc-100 dark:bg-zinc-800 text-zinc-600 dark:text-zinc-400 hover:bg-blue-100 dark:hover:bg-blue-900/40 hover:text-blue-700 dark:hover:text-blue-300 transition">
                {tag}
              </a>)}
          </div>}
        <hr className="my-8 border-zinc-200 dark:border-zinc-700" />
        <div className="prose dark:prose-invert max-w-none">{children}</div>
      </div>
    </section>;
};

export const MarketingPage = () => <div className="marketing-page-marker" style={{
  display: "none"
}} />;

export const Section = ({children, className = "", gray = false, dark = false, id}) => <section id={id} className={`w-full flex justify-center px-4 py-16 md:py-24 ${gray ? "bg-gray-50 dark:bg-zinc-900" : dark ? "bg-zinc-900 dark:bg-zinc-950" : ""} ${className}`}>
    <div className="w-full" style={{
  maxWidth: "1140px"
}}>
      {children}
    </div>
  </section>;

<MarketingPage />

<BlogPost title="Fixing Bazel out-of-memory problems" date="2022-08-20" authors="Alex Eagle" tags="Bazel, Performance">
  <img noZoom src="https://mintcdn.com/aspectbuild/x1L7Iep716jCyJVo/images/blog/hashnode/hn-9dab9f15b9.jpeg?fit=max&auto=format&n=x1L7Iep716jCyJVo&q=85&s=7c58a1e73d917e5d2c6397d181218beb" alt="" className="blog-post-cover" width="1600" height="1068" data-path="images/blog/hashnode/hn-9dab9f15b9.jpeg" />

  Memory management is a generally hard topic in computer systems operations. Debugging it inside a cloud-hosted build system is even worse!

  There are two potential problems:

  * The Bazel server runs in a JVM, and it internally tries to allocate more objects than the max heap size its allowed.
  * Bazel spawns subprocesses (called "actions", including test actions) and they collectively exhaust the memory in the machine or VM that Bazel runs in.

  I'll cover these scenarios separately since they're mostly unrelated.

  > Of course, the Bazel JVM heap does occupy system memory, so they're related in the sense that a smaller Bazel server footprint would allow for more actions to run, but I've never considered that to be a potential remediation.

  ## Bazel server out-of-memory

  How to tell this is happening:

  * Bazel exits with code 33 (see [`ExitCodes.java`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/util/ExitCode.java#L58))
  * Check the output of `bazel info | grep heap` if the Bazel server is still running, see if it is near the max.

  Some things you can do about it:

  * Give it more RAM! Assuming the system has some available, you can use the [`host_jvm_args` startup flag](https://docs.bazel.build/versions/main/command-line-reference.html#flag--host_jvm_args) to adjust the usual JVM parameters like `-Xmx2g`.
  * Always turn on the [`--heap_dump_on_oom` flag](https://bazel.build/reference/command-line-reference#flag--heap_dump_on_oom) so that you get extra information in this case.
  * "Memory saving mode" can be useful if you're hosting Bazel in ephemeral CI workers where you expect every build to be cold, however that's slow and not recommended. If you do that, you can avoid Bazel tracking incremental state which saves some memory. [https://bazel.build/configure/memory](https://bazel.build/configure/memory)
  * Roll up your sleeves and figure out what's consuming so much memory in Bazel's JVM. Start from [Bazel's documentation on memory profiling](https://bazel.build/rules/performance#memory-profiling). An example can be rulesets where data is repeated rather than using depsets, an example analysis: [https://github.com/aspect-build/rules\_js/pull/391](https://github.com/aspect-build/rules_js/pull/391)

  ## System out-of-memory

  Bazel schedules actions (build steps and test runners) based on the amount of system resources it thinks are available, and using some heuristic about how much RAM a typical action requires. Two kinds of things can go wrong, either Bazel thinks more RAM is available than the system actually has free, or Bazel underestimates the resources to be reserved for a given action.

  By default, Bazel's max concurrency is based on the heuristic that each action needs one CPU core, so the `--jobs` flag default is the number of (maybe virtual) CPUs on the machine. Note that Bazel reports progress with a "X running" indicator which might lead you to believe that the concurrency is actually higher, but that's a misleading message because it can include actions that are queued waiting for resources

  <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Did you know when <a href="https://twitter.com/bazelbuild?ref_src=twsrc%5Etfw">@bazelbuild</a> prints progress like <br />\[12 / 100] 32 actions, 30 running<br />That "running" count includes "remote-cache" spawns! If you have --jobs=16 (the default on 16 core) the other 14 of them aren't actually running, they're queued for "local" spawn. <a href="https://t.co/a3w7TWZVTM">[https://t.co/a3w7TWZVTM](https://t.co/a3w7TWZVTM)</a></p>— Alex 🦅 Eagle (@Jakeherringbone) <a href="https://twitter.com/Jakeherringbone/status/1559942657756975104?ref_src=twsrc%5Etfw">August 17, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8" />

  How to know this is happening:

  * The Bazel server gets killed by the operating system like `Bazel server terminated abruptly (error code: 14, error message: 'Socket Closed', log file: ...)`

  Some things you can do about it:

  * If Bazel is running inside a container, it may calculate available RAM based on the host system rather than what is allocated for the container, due to [Default local resources should respect cgroup limits on Linux](https://github.com/bazelbuild/bazel/issues/3886). The remediation is to explicitly tell Bazel's scheduler what the container limits are by setting the [`--local_ram_resources` flag](https://bazel.build/docs/user-manual#local-resources) to match the container runtime.
  * Reduce `--jobs` so that fewer things run concurrently. This is a blunt approach and makes all builds take longer, but saves you the effort of figuring out which actions didn't get enough resource reservation.
  * Figure out which actions consume a lot of RAM, and tell Bazel's scheduler to reserve more resources for them. For tests, use the [`test_size` attribute](https://bazel.build/reference/be/common-definitions#common-attributes-tests) - a larger size gets more reserved memory per the table in that documentation. For build actions, the Bazel team [recommends](https://github.com/bazelbuild/bazel/issues/14601) using [execution properties](https://docs.bazel.build/versions/main/exec-groups.html#using-execution-groups-to-set-execution-properties) though to be honest, that looks really complex and I haven't used it myself.
  * Try out the [`--experimental_local_memory_estimate` flag](https://docs.bazel.build/versions/main/command-line-reference.html#flag--experimental_local_memory_estimate) to make Bazel smarter about knowing the available system resources at the time it's scheduling the subprocess to spawn.
  * Investigate using [Remote Build Execution](https://bazel.build/remote/rbe) so that heavy workloads move off the machine and run on a cloud of executors.
</BlogPost>