# About Aspect Build Source: https://site.aspect.build/about Aspect Build commercializes Bazel, Google's open source build system, through expert support and the Aspect Workflows developer productivity platform.

Aspect Build was founded in 2021 to commercialize Bazel, Google's open sourced build system, now stewarded by the Linux Foundation. Our team includes early Bazel contributors and maintainers who built and operated Bazel at Google scale.

Our insight: Bazel provides a highly efficient and scalable core build engine, but organizations outside Google need a richer ecosystem of tools, integrations, and services to adopt it successfully. We began with Bazel services and then released the Aspect Workflows developer productivity platform.

Our company name is a nod to Bazel's core concepts. In Bazel, an "aspect" is a powerful extension that enriches the build graph with additional insights, actions, and dependencies, unlocking deeper analysis and automation.

Meanwhile, code and builds are structured in "BUILD" files. Just like Bazel aspects enhance the build process, Aspect Build enhances your developer experience with expert services and our developer productivity platform.

Aspect Bazel rules repositories on GitHub

# RBE is fast enough for AI agents. But, is it safe enough? Source: https://site.aspect.build/blog/ai-agents-rbe Everyone's solving 'make RBE fast enough for agents.' Nobody's solving 'make RBE safe enough to let agents near it.' Self-hosted Workflows gives your AI its own execution plane — in your own VPC, without extending the trust boundary of your human CI. AI coding agents and remote build execution (RBE) are a natural fit. Agents iterate fast: run a build, watch it fail, fix it, then rebuild. Everyone always asks if their executor fleet can keep up. Almost always, they find that it can. After all, RBE was built exactly for these kinds of parallel, high-frequency builds. The real question is should agents have access to that infrastructure at all? When an AI coding agent runs on your RBE cluster, it’s working in the same environment as your engineers, with access to the same executor pool, the same secrets, and the same artifact cache your release builds depend on. Your RBE cluster was designed with human engineers in mind. Someone authenticated, accountable, with a job title and an oncall rotation is lurking on the other end. An agent doesn’t have any of that. Yet most teams have quietly handed it full CI-level trust anyway. We ran into this with a healthcare technology company. Their fix: a second Aspect Workflows deployment that runs nothing but AI coding agents, isolated from their human CI behind its own network boundary, with its own access audit trail. ## When agents cross the trust threshold When you’re a small team, running a handful of agents occasionally, sharing a cluster is no big deal. The risk is low because the scope is narrow. It’s not fine when you realize your agent has unfettered access to infrastructure that’s not set up to be audited that way. There are a few concrete signs it’s time to isolate your agents: * **Agents are running non-stop:** Your nightly batch job touches the cluster for an hour then clocks out. An agent loop, on the other hand, is active during business hours. It spawns builds on every commit and hangs out on your executor fleet all day. * **Build load is blending:** Your human CI builds and agent retry loops compete for the same executors. When agent-driven builds gang up on your human builds and push them back in the queue, it’s not an executor problem. It’s actually two workloads that were never meant to share a queue — you just found out the hard way when your PR is stuck behind an agent’s umpteenth retry. * **Your audit trail can’t tell the difference:** Your logs show that a build ran on executor X. They don’t tell you whether X was a person or an agent (at least not without additional tools). Do you know what an agent touched last week? Or even two days ago? That gap in logging is a compliance nightmare in regulated environments and a forensic dead-end anywhere else. * **Compliance isn’t reactive:** Some teams have security conversations after something breaks or an audit asks a question you can’t answer. For teams in certain industries (healthcare, finance, defense, cybersecurity) compliance comes before an incident. Either way, the need will arrive. And the infrastructure you’ve been running on was never built to answer those questions. ## Blast radius: what an agent can actually get its hands on The trust argument is abstract. The blast radius is concrete. When your agent is unruly, gets compromised, or just operates with the speed and energy of an overcaffeinated toddler, here’s what it can reach on a shared cluster. ### The artifact cache Bazel’s remote cache is keyed by action hash. An agent looking around a large change set generates action hashes across a wide slice of the build graph. Every query fills the cache’s hot tier with entries your engineers will probably never need and pushes out the ones they probably do. A build that should’ve been fast turns into a partial cold rebuild, not because of any changes in the code, but because an agent went on a querying quest an hour earlier. That’s a correctness-adjacent problem: your cache is now shaped by how agents wander, not by how your team actually builds. Without any changes to infrastructure you can create separate cache namespaces, set with `--remote_instance_name` in your Bazel config. Your human CI reads and writes to one namespace and agents get their own. Take it even further and tier your storage: fast, low-latency storage for humans and a cheaper tier for agents, where a cache miss costs less because the agent rebuilds anyway. ### The executor pool The agent retry loop is a tight one: build, fail, fix, rebuild. Enough of those running at once and your executor pool is at 100%, leaving your human CI waiting in line. The fix is queue prioritization: human builds get a dedicated capacity reservation, agent builds get burst capacity. Most RBE platforms support priority classes, but somebody has to configure it. It doesn’t happen by default. On a dedicated cluster, this doesn’t come up, because agents physically can’t reach the pool human builds depend on. ### The secrets those executors can reach RBE executors have access to things like API tokens, signing keys, registry credentials, and other secrets your build actions ask for. An agent running wild on the same fleet has the same reach as your engineers. If you haven’t explicitly scoped what an agent can touch, you’ve implicitly given it access to all the things. You ask it to "fix the failing test" and it queries the whole graph, hitting targets that touch credentials, and burning signing resources, all without leaving a trace that tells you it wasn’t a person. A dedicated cluster means two IAM configurations instead of one, and the agent one is short. Access to the cache. That’s it — no signing keys, no registry writes, nothing that touches a deploy. The scope is explicit because you made it that way, not because you’re hoping no one asks. ## Compliance: the audit trail problem For a healthcare technology company, the whole thing didn’t start as a performance conversation. It started with an audit. When it comes to regulated industries the concerns are specific: who had access to the executor that ran this build, and what could they reach? On a shared cluster, the honest answer is "whoever was in the CI IAM policy at the time, including any agents that happened to be running." That just doesn’t satisfy an auditor and it doesn’t tend to hold up in a security incident review either. A dedicated agent cluster makes the audit trail clean. Every action lands in your cloud’s native audit log, tagged to that cluster automatically. No one has to build a special agent-tracking dashboard. Agent activity is visibly separate from human activity because there are two clusters, not with a tool that’s cobbled together. When an auditor asks what the agent accessed, the answer is bound by what that cluster was allowed to reach and that’s a short list. This carries over to data residency. If you have requirements for where code artifacts live or get processed, an air-gapped or single-region Workflows deployment keeps agent build data inside the same jurisdictional lines as everything else, regardless of what the agent is doing. As a practical side benefit, it’s simpler to reason about. CI executors need write access to your artifact registry, signing keys for release builds, and deployment credentials. Agents don’t typically need any of that. Scoping those permissions differently on one cluster is awkward. With two clusters, there are two policies, and the agent policy is short. ## Cost attribution: governance follows isolation Once you’ve split execution planes, you can actually see what agents cost. On a shared cluster, agent and human CI builds are a blended bill. You can watch the spend as agents become more active, but you can’t easily separate what drove it. It’s easily fixed by tagging compute resources by workload. Compute resources in the agent cluster get a tag like `workload: agent`; human CI gets `workload: ci`. Most cloud billing dashboards already group by tag, so per-workload cost shows up without additional tooling. The same pattern works if multiple teams are running their own agent clusters — each team’s infrastructure cost is a separate line. The more useful number is cost per unit of output rather than raw compute cost. For human CI that’s cost per merged PR or per successful build. For agents, it’s cost per accepted commit or cost per resolved issue. Those numbers take more manual work to pull together, but worth computing once to establish a baseline. Dedicated infrastructure adds a bit of fixed overhead, but it also makes the comparison honest. You can’t do that with a blended bill. Chargeback models differ from team to team. Some teams bill agent infrastructure back to the team running the agents; others treat it as a shared platform cost. Either way, attribution only works if the workloads were separated at the infrastructure level. You can’t make a governance decision about agent adoption off a number that mixes two different things together. ## What this looks like in practice The healthcare technology company runs two [Aspect Workflows](https://aspect.build/platform) deployments in their AWS account. One handles all their human CI. The other handles nothing but agent-driven builds, behind its own network boundary, with its own IAM policies and its own audit trail. The agent cluster reads from the same cache the human CI fills, so agents aren’t starting from nothing every time. Writes go to a separate namespace, so they can’t step on your CI cache. Executor pools are separate. IAM policies are separate. Billing tags are separate. Running two clusters costs the same operational overhead as one. Aspect engineers deploy and operate both either way. What the company walks away with is a compliance posture they can describe to an auditor and a trust boundary they can point to. ## Get your agents off shared CI Everyone’s busy solving fast enough. We’ve got that part down. The trust boundary problem has been largely ignored. It’s the one that shows up in audits, in incident postmortems, and on the day an agent-driven build quietly touches a signing key it shouldn’t have. Give agents their own Workflows deployment and their own execution plane: separate VPC, separate IAM, separate audit trail. It still reads from the same artifact cache your human CI fills. It just doesn’t get to write to it, or touch anything else. If you want to see what this looks like for your infrastructure, [reach out and we’ll walk through it](https://aspect.build/request-demo). # Angular with Bazel Source: https://site.aspect.build/blog/angular-with-bazel Learn how to integrate Angular and Bazel using rules_js for better performance and compatibility. We recently released [rules\_js](https://github.com/aspect-build/rules_js/) 1.0.0, a faster and more compatible approach to integrating JavaScript tooling under Bazel. Now that ng-conf 2022 is kicking off and all our friends are back in Salt Lake City, it's a perfect time to update how Angular and Bazel work well together using rules\_js. ## First a bit of background… the Angular CLI The [Angular CLI](https://angular.io/cli) is the standard developer tool for Angular applications. From dev server to production bundling, the CLI takes an "all-in-one" approach for a good, lowest-effort developer experience. Under the hood the CLI is primarily a thin wrapper over the [Angular Architect library](https://github.com/angular/angular-cli/tree/14.2.x/packages/angular_devkit/architect). Angular Architect allows various tools to be integrated into the CLI via plugins such as a devserver, webpack bundling, sass integration, and test frameworks, along with the core Angular Compiler (`ngc`). Architect combines those tools into one for the full Angular CLI experience. ## Angular Architect in Bazel The simplest method of building an Angular application under Bazel is the same as under the Angular CLI: use Angular Architect. Simply invoke the Angular Architect tool from Bazel for the same all-in-one experience as the Angular CLI. For example, a single Bazel target to compile an Angular application (named `my-app`, created by the Angular CLI): ```python theme={null} load("@npm//:@angular-devkit/architect-cli/package_json.bzl", architect_cli = "bin") architect_cli.architect( name = "my-app", args = ["my-app:build"], srcs = glob(["**/*.ts", "**/*.sass", "**/*.html"]), ) ``` See the [Angular Architect example](https://github.com/aspect-build/bazel-examples/tree/main/angular) for a full example. But all-in-one is not the idiomatic Bazel style: if anything in the application changes the entire application must recompile. While easier to implement, Angular Architect is not the ideal Bazel experience. ## `ngc` in Bazel To truly benefit from Bazel the compilation must be split into independent actions that can be individually executed and cached. Each action coordinated by Angular Architect can instead be coordinated by Bazel. When Bazel is coordinating the actions the real benefits of Bazel will be seen such as parallelization, caching, test caching, remote execution/caching and all the other benefits that come with Bazel. The primary action is compiling Angular TypeScript including component templates, css and the various annotations such as `@Injectable`. The Angular Compiler (`ngc`), is a drop-in replacement for the TypeScript compiler (`tsc`). The `ts_project` rule can be customized to use `ngc` as the compiler binary. Bazel's macros provide a simple way to define your own "syntax sugar". We'll start by declaring the `ngc` compiler target, and an `ng_project` macro that makes it easy to declare these. **tools/BUILD.bazel** ```python theme={null} load("@npm//:@angular/compiler-cli/package_json.bzl", compiler_cli = "bin") compiler_cli.ngc_binary(name = "ngc") ``` **tools/ng.bzl** ```python theme={null} load("@aspect_rules_ts//ts:defs.bzl", "ts_project") # Macro to wrap Angular's ngc compiler def ng_project(name, **kwargs): ts_project( name = name, # NGC compiler, do not use the standard tsc worker tsc = "//tools:ngc", supports_workers = False, # Any other ts_project() or generic args **kwargs ) ``` Now Angular code can be fully compiled with the `ng_project` macro including TypeScript, component HTML and CSS, directives, `@Injectable`s etc. An example of a `ng_project` target: **my-app/BUILD.bazel** ```python theme={null} load("//tools:ng.bzl", "ng_project") ng_project( name = "my-app", srcs = glob(["**/*.ts", "**/*.css", "**/*.html"]), deps = [ "//:node_modules/@angular/core", ... ], ) ``` This is a single `ng_project` rule, but an application will most likely be divided into many BUILD files, creating many independently compiled and cached Bazel targets. Other features of the Angular CLI such as Sass preprocessing, webpack bundling, a devserver, testing etc. will be configured as independent Bazel targets. Under the hood the tools will be the same as Angular Architect but now coordinated by Bazel. Or, you can swap out some of the tools, essentially making your own custom, incremental build system for Angular just with a few lines of Bazel's macros. For example, bundling the application with [rules\_webpack](https://github.com/aspect-build/rules_webpack) ```python theme={null} load("@aspect_rules_webpack//webpack:defs.bzl", "webpack_bundle") webpack_bundle( name = "bundle", entry_point = "main.js", srcs = [":my-app"], ) ``` For a complete example see the angular-ngc Bazel example: [https://github.com/jbedard/bazel-examples/tree/angular-ngc/angular-ngc](https://github.com/jbedard/bazel-examples/tree/angular-ngc/angular-ngc) # Announcing Remote Build Execution Source: https://site.aspect.build/blog/announcing-remote-build-execution Aspect Workflows now includes Remote Build Execution, speeding development by offloading computation to worker machines Remote Build Execution (RBE) is a technique for off-loading computation of a wide build and test graph to a farm of worker machines. It can vastly speed up development when changes affect a large subgraph of a monorepo. Aspect is pleased to announce that this is now a supported feature of our Workflows platform on both GCP and AWS, in use on our OSS rulesets and rolling out to users today! ## What is Remote Build Execution (RBE)? Bazel has a built-in scheduler. It tries to parallelize build steps as much as possible given the estimated available resources. For example, on an eight-core machine, it might run eight different test actions concurrently, if it determines other resources can allow it (for example, available system memory). Bazel will queue actions which otherwise might have been able to run immediately when resources on the “local” machine are exhausted. Remote Build Execution allows additional resources on other computers to be added to the build. Instead of queuing, Bazel then uses a remote API to send RPC calls to that “farm” of computers. The inputs are identified (and uploaded, if needed), then an RPC call schedules the action to run remotely. In some cases this makes the overall build faster. In this post we’ll discuss which cases those are, so you can decide if RBE is right for your organization. ## Remote Build Execution is misunderstood Aspect’s competitors have offered Remote Build Execution from the beginning. Their pricing is based on usage (either on the upper bound on the number of executors or on Cloud Compute resource consumption.) Perhaps due to this perverse incentive, they have positioned RBE as the way to accelerate build and test and encourage every user to adopt it. From reading their website, an engineer can reasonably come away with the false impression that RBE is the first step to speeding up a slow build. However this naive view usually results in much higher costs. You may have heard me in a conference talk describe RBE as “the performance optimization of last resort”. Our first goal is “Minimal Execution”, which is where a build is incremental thanks to a very high cache hit rate. Of course “Minimal Execution” is faster and cheaper than either Local or Remote execution. That’s why Aspect doesn’t have a usage-based pricing model! Later in this article, I’ll dig more into the reasons that a small-to-medium sized codebase and team might not get enough benefits from RBE to make it worthwhile. For now, just be aware that the highest-order factor for ANY Bazel project is the remote cache, which is how highly incremental builds can skip work, and is the bottleneck for Bazel to look up cache hits to avoid re-work. Aspect’s largest users like Airtable do benefit from Remote Build Execution. These companies have hundreds of engineers working on a codebase with millions of lines of code. Even after minimizing the amount of execution, their build graph shape still lends itself to wide parallelism on a typical product engineer’s change. ## About Aspect’s RBE Aspect has open-source in our DNA. So it was obvious from the beginning that we’d build on excellent, well-maintained and battle-tested Remote Cache and Execution software. We chose Buildbarn! It powers hundreds of developers at Apple and has a strong Slack community. (Apple’s open-source office doesn’t like to publicize their projects, so it’s not obvious that Buildbarn is built there!) Our Buildbarn remote cache deployments have been live at every Workflows customer for the last 18 months. The cache is highly available and scalable, and has been rock-solid. So adding RBE was “just” a matter of enabling another Buildbarn component. In practice, it was not that easy. Although Buildbarn has an “[example deployments](https://github.com/buildbarn/bb-deployments)” repository, there’s a lot of missing documentation. Users are forced to read comments on the protocol buffer definitions to understand a lot of the fields. If you decide to deploy Buildbarn yourself, and run into problems, we recommend our partner [Meroton](https://meroton.com/) for professional services. Moving execution off of Bazel’s host machine has the architectural benefit of separation of concerns. You can choose compute instance types to match the profile of your Bazel actions, rather than needing to bend the actions to fit the available resources on the machine where Bazel runs. Separating the environment where the CI system executes steps from the environment where actions execute also adds security and maintainability benefits. This is especially true when the remote cache and execution is accessed from developer machines. As a special case of this separation, some builds need to execute actions on various hardware platforms. Aspect has several autonomous driving customers with custom circuit boards. Remote execution allows Bazel to run on a standard and inexpensive instance type like AWS Graviton, while some test logic can execute on a runner process on the board. Bazel itself understands execution platforms, and Buildbarn allows workers on multiple platforms within a single cluster. Aspect Workflows includes the configuration needed to connect Bazel and Buildbarn ensuring that each action runs on the right hardware and operating system. Customers may wish to distribute a "platform-compatible" workload over a variety of compute options, with various scaling parameters and compute specificity. Buildbarn accounts for this with a concept called "size classes". With no effort required by the user, Buildbarn may schedule a single action across the gamut of size class options available, and uses this data to schedule that action in the future on the least expensive size class where it is likely to succeed. This has the advantage of still delivering fast results, but optimizing the workload over time to reduce costs. RBE also serves as a stop-gap when a build is non-incremental, typically when the Remote Cache is unavailable or has been intentionally cleared as part of an infrastructure rollout. Watch for a formal Case Study on Aspect Workflows RBE at some of our customers, coming soon! ## When to consider Local rather than RBE As I've said at conference talks, we consider RBE to be the "performance optimization of last resort". We believe strongly that reducing execution is the first step, to minimize cloud compute costs. Here are other factors that make Bazel’s Local execution strategy a better choice than RBE for small-to-medium sized repositories: 1. When typical developer activity results in invalidating many expensive test actions, compute costs will be high (regardless of whether execution is local or remote). If your organization’s budget doesn’t allow for re-computing all these tests, you may want to run them less frequently (i.e. only after merge, or nightly). We often see this in robotics or autonomous driving, where physics simulations run as part of a test. This “test selection” strategy is in contrast to the typical Bazel idiom which is to “test everything affected”. 2. RBE requires strictness in configuring Bazel’s toolchains and hermeticity to account for the “host” and “execution” platforms differing. This can be a big task, and gets harder as you work through the “long tail” of atypical build actions. As our competitor writes on one of their case studies: > Migrating a Bazel project to remote execution can be a daunting task 3. A wide cache miss is often caused by an infra engineer who changed an SDK version, or some other configuration change that invalidates a lot of the graph. In that use case, the engineer probably doesn't expect the same performance as a product engineer making routine changes. 4. It’s easy to add more Local resources. By provisioning a larger machine for Bazel, the built-in scheduling will parallelize build actions over the available resources such as multiple CPU cores. By occupying such a machine for short time periods, we can avoid the high cost implications of these expensive instance types. 5. Your build graph may not be amenable to parallelization. An action cannot run until its inputs are available, so when a build is slow due to the “critical path” of such serialized actions, adding more compute resources doesn’t make the build faster. ## Try Aspect Workflows Try Aspect's RBE solution by requesting a free trial of Aspect Workflows. Our engineering team will deploy an instance to try out with your real workload, and gather the data to help decide whether RBE is a net benefit for your team! # Aspect Build, built on Bazel Source: https://site.aspect.build/blog/aspect-build Aspect Build offers Bazel solutions, including a rules docsite, BUILD file automation, and a more user-friendly CLI. Explore more at aspect.build We've been hard at work building a great consulting business at [Aspect](https://aspect.build). We are helping some large companies like Robinhood and Boston Dynamics migrate their build, test, and CI systems to Bazel. However, it's clear from all our engagements that Bazel itself is incomplete. We are excited to announce some products we've built to help fill in this gap. ## Rules Docsite Bazel doesn't come with "batteries included". It only understands a few languages in the built-in distribution like Java, Python, and C++. For everything else, it relies on plugins called "rulesets". The first problem our customers found with rulesets is that the documentation is lacking. There is usually a very simple example of usage, and then as soon as you need something more real, you are stuck reading sources. So we've launched the first Bazel rules docsite, at [https://aspect.build/docs](https://aspect.build/docs) Our docsite scrapes the generated API output of canonical rulesets, using the resulting protocol buffers to drive a custom site. It's blazing fast because the content is all build-time pre-rendered. Other features of our docsite: * **Unified**: All the Bazel rules documented in a single place. * **Search**: You can search across the documentation for all rulesets. * **Versioned**: Every documentation page is permalinked to an exact version and won't change. You can select the version of a ruleset that you use. * **Deep-links**: You can link directly to an attribute of a rule, so it's quicker to help out your fellow humans. Ultimately our goal is to improve the upstream documentation across the rulesets by making the documentation easy to edit in-place, with the convenience of a wiki, through a standard pull request process. ## `BUILD` file generation Bazel's `BUILD` files are wonderfully explicit and self-contained, but largely mirror dependency information that was already evident in the source files. At most clients we've observed that Product Engineers don't want to learn about build systems, and we think they shouldn't need to. Aspect is making a big investment in [Gazelle](https://github.com/bazelbuild/bazel-gazelle), which automates maintenance of `BUILD` files. We upstreamed the [Python Gazelle plugin](https://github.com/bazelbuild/rules_python/tree/main/gazelle/README.md) to rules\_python and have started work on plugins for TypeScript and other languages. ## Rules Authors SIG We've long been the core maintainers of [rules\_nodejs](https://github.com/bazelbuild/rules_nodejs) and also help to maintain [rules\_python](https://github.com/bazelbuild/rules_python) and [rules\_docker](https://github.com/bazelbuild/rules_docker). After working with our clients, we have a much better appreciation for how much companies depend on these rules, yet how little maintenance effort goes into them. Sadly many corporate contributors haven't been able to get their changes merged, and maintainers haven't benefitted from the resources these large companies can provide. For this reason, we led the creation of the first Special Interest Group (SIG) under Bazel's [new SIG program](https://bazel.build/sig.html). Find it at [SIG Rules Authors](https://github.com/bazelbuild/community/blob/main/sigs/rules-authors/CHARTER.md). ## More usable CLI We see that most engineers interact with Bazel on the command-line, and many are baffled by its complex error messages and unfamiliar terminology. It's an expert tool, being used by engineers who would rather not become build system experts. `aspect` is a new command-line interface wrapping Bazel. The `aspect` CLI is interactive, and customizable for your organization's developer workflows. It's compatible with `bazel` so you can switch back and forth easily. While it's currently a pre-release, we are excited for the future of this tool. Read more at [https://aspect.build](https://aspect.build). ## We can't wait to build a better Bazel! If you're as excited about build tooling as we are, then let's work together! Visit aspect.build or participate in the SIG as a funder or a coder. # Start a new Bazel project with aspect init Source: https://site.aspect.build/blog/aspect-init-and-starters aspect init is now a native Aspect CLI command, and the language starter templates live at github.com/aspect-starters with a Use this template button. Standing up a Bazel workspace takes more boilerplate than it should. `.bazelversion`, `.bazelrc` with sane flags, toolchains, formatting and linting wiring, package-manager wiring, and CI that *maybe works*. It's a rite of passage no one asked for. So we packed the whole setup into a single command. ## `aspect init` is now built into the CLI `aspect init` is a native [Aspect CLI](/docs/cli) command (requires [v2026.25.11](https://github.com/aspect-build/aspect-cli/releases/tag/v2026.25.11) or newer). No separate tool, nothing to download by hand. Just install the CLI and run: ```bash theme={null} aspect init my-project --preset go cd my-project aspect build //... ``` That's it. You have a fresh bazel setup. Omit `--preset` and you get an interactive picker. Available presets: `minimal`, `shell`, `go`, `js`, `py`, `java`, `kotlin`, `cpp`, `rust`, `ruby`, `scala`, and `kitchen-sink` (every language plus OCI containers, protobuf, and release stamping). What generated project comes wired up: * 🧱 The latest Bazel (bzlmod) with curated flags via [`bazelrc-preset.bzl`](https://github.com/bazel-contrib/bazelrc-preset.bzl) * 🧰 A hermetic dev environment via [`bazel_env.bzl`](https://github.com/buildbuddy-io/bazel_env.bzl) and [`rules_multitool`](https://github.com/theoremlp/rules_multitool) * 🎨 Formatting and linting with [`rules_lint`](https://github.com/aspect-build/rules_lint) * 📦 Native package-manager integration for the chosen languages * ⚙️ Working GitHub Actions CI that runs `aspect build`/`test`/`lint`/`format` on ephemeral runners * 📌 A pinned Aspect CLI version so your whole team and CI use identical tooling. ## The template repos, if you prefer Each preset is also published as a GitHub *template repository* under [github.com/aspect-starters](https://github.com/aspect-starters). Go to the language you want — say [aspect-starters/go](https://github.com/aspect-starters/go) — and hit **Use this template** button (or fork it, or just `git clone`). Super handy if you want a playground, a training repo, or a link to send to the next person who asks "how do I set up Bazel for X?" Building a polyglot monorepo? `aspect init --preset kitchen-sink` or start from [aspect-starters/kitchen-sink](https://github.com/aspect-starters/kitchen-sink). ## One source of truth The CLI command and the template repos render from the same templates, so they never drift. The source of truth is [aspect-build/aspect-workflows-template](https://github.com/aspect-build/aspect-workflows-template): CI there renders and builds every preset before anything lands, then republishes the `aspect-starters` repos on each release. Found a rough edge or want to improve a template? File issues and PRs there. This replaces the older wizard-based `init` (which relied on an external scaffolding tool) and Aspect's previous starter repos. Same idea, much better execution, and now it's part of our CLI. [Install the Aspect CLI](/docs/cli/install) and run `aspect init`. # Automated testing of each commit != CI Source: https://site.aspect.build/blog/automated-testing-of-each-commit-ci Explore how microservice architectures hinder true Continuous Integration and why API contracts alone can't prevent integration issues. Learn how to fix it. > Cross-posted from my [Medium blog from 2019](https://medium.com/@Jakeherringbone/automated-testing-of-each-commit-ci-6f718d93d0da) I’ve had some chances on a recent trip to get more first-hand engagement with some enterprise-scale Angular customers. What I learned reinforces the impression I’ve gotten about our industry as micro-service architecture rolls out: we no longer do Continuous Integration (CI). These companies seem self-assured that they have good testing practices: they have an automated test suite, and they run it continuously. That’s really great, and catches some bugs earlier and makes happier developers who don’t have to go back as frequently to debug through the sludge of years-old code. So you have the C in CI, no problem. We must remember what the I in CI stands for. What are we integrating? Any large organization requires breaking down work into departments. Ever wondered why a space agency has a control room with such a huge number of desks with controllers? Rocketry and space flight is such a massive technical undertaking, with different scientific and engineering disciplines working together, that they’ve built an operations process which gives them immediate access to information from each. Imagine if instead of the big control room, the astronauts were just talking to a “frontend” team who then had to consult what the “backend” team thought, sometimes with a several day round-trip, and that in turn was using outdated manuals for the spacecraft. Each department might be doing the right thing on their own, but they are not “integrated”. They don’t act as a single unit. The problem I see in companies adopting a microservice architecture is similar. Each team has tested their own components and are ready for deployment. Then when it’s time to get some new software up there to our astronauts (or other users), we discover that it doesn’t work in the QA environment — taking a day to trace the change in some other department. This is really bad for the business. If it takes weeks to integrate the software each time we want to deploy, then critical business initiatives have to build in extra time budget for software changes. This delay in shipping causes a terrible feedback loop in the organization. As it takes weeks to release a change through “the process”, teams want to have more autonomy to have their own release schedule. It seems like it should accelerate the process, but of course this exacerbates the problem. These new autonomous units only run their own tests, and so the interactions are untested when used with the other systems it will have to integrate with in production. The usual software architect or consultant replies, “This is not a problem because rigorous API contracts are drawn at the boundaries of each service. Each part is tested against that API”. It sounds like a great answer. Does it work? There’s a guy who worked on the C++ team at Google named Hyrum Wright. He had to make global changes across Google’s monorepo to migrate everyone to newer versions of some base library API. That API had been specified, and was tested against, and the API was not being changed. Yet library changes would inevitably cause a bunch of test failures across Google. What were we doing wrong? There’s now an observation called [“Hyrum’s Law”](https://www.hyrumslaw.com/) based on this experience: > With a sufficient number of users of an API,\ > it does not matter what you promise in the contract:\ > all observable behaviors of your system\ > will be depended on by somebody. Another way to say this is, while your API surface is constrained in spirit, it is unconstrained in practice. Change how a result is sorted? Someone relied on the prior ordering. Now comes the incorrect conclusion from some architects: “So then that was a bug in the client of the API. The contract never guaranteed that the data would be sorted. Shame on them”. What is the business to make of this? The client software team made an avoidable error and is therefore negligent? Passing the blame this way doesn’t actually solve the business problem: these integration errors happen and are preventable. API contracts are not a sufficient way to prevent them. As software engineers, we know that even if we are quite diligent, we’ll make accidental assumptions that happen to work today. The solution to developers making human mistakes is to add QA which catches it. Therefore, we should be writing (and continuously running) tests that exercise the *entire stack* we’ll deploy on: integrating it continuously. Only this can assure reliable delivery of our new code. And remember, the astronauts are depending on us. ## Let's fix it! In my article [https://blog.aspect.dev/cboi-continuous-build-occasional-integration](https://blog.aspect.dev/cboi-continuous-build-occasional-integration) I talk about the technical details of how to get Continuous Integration back. # Bazel: Avoiding eager fetches Source: https://site.aspect.build/blog/avoid-eager-fetches Learn how to identify and prevent eager fetches in Bazel builds, optimizing dependency management for efficient development workflows Bazel manages your dependencies, and fetches them to a users machine when they are needed for a build. That's great, as it ensures all developers on the project have the same dependencies installed without having to think about it. When working well, these fetches are lazy and fine-grained: users only download what's needed for the specific targets they requested to build or test. However it's easy to de-optimize by introducing an "eager fetch". This is when Bazel downloads some dependencies which aren't actually needed for the current build, just because they are referenced during the analysis phase when the BUILD files are read. These fetches are only a problem on the first build, since the resulting "external repositories" are reused by Bazel for subsequent re-builds. However they are still annoying that first time, and if the repository gets invalidated (maybe because the user switches branches to one that's rebased before some change to the dependency listing) then they have to wait again. They are extra-annoying when the "fetch" includes some subsequent slow install steps, like compiling a program that was just downloaded. > Bazel has a ["repository cache"](https://bazel.build/docs/build#repository-cache) but this is often misunderstood. It does *not* cache the external repository that Bazel installed on the disk - rather it only caches certain network fetches which had a sha256 sum or integrity hash and were fetched by the Bazel built-in downloader. Tools like `npm` and `pip` do their own fetches, so those aren't cached by Bazel (though they might be cached somewhere else on disk by those tools). Even if you avoid a network fetch, any computation performed to "install" those dependencies is never cached by Bazel and has to be re-done if the external repository is invalidated. If you change branches back to the original one, the invalidation is just as expensive; there's no re-use of the prior state in a X -> Y -> X sequence. This article explains how these eager fetches get triggered, how to remediate them and how to prevent regressions. ## WORKSPACE eager fetches These are the worst kind, because they happen for every single build regardless of the dependency graph or which targets the user requests. Bazel must evaluate the complete `WORKSPACE` file to understand what third-party dependencies exist for the build. Let's say the WORKSPACE file contains this content: ```python theme={null} load("@rules_python//python:pip.bzl", "pip_parse") pip_parse( name = "my_deps", requirements_lock = "//path/to:requirements_lock.txt", ) load("@my_deps//:requirements.bzl", "install_deps") install_deps() ``` The penultimate line loads from the `@my_deps` repository, which means that repository must be eagerly fetched. Whatever work happens in `pip_parse` will happen for every single build, even for developers who aren't doing anything Python-related. In this case, `pip_parse` does need to fetch metadata about Python dependencies, so this isn't free. Use the Bazel profile to help you determine whether an eager fetch is a problem in your builds. ## Mitigating In some cases, you can refactor the WORKSPACE to remove the fetch. One approach is to "vendor" - check in the result of the expensive computation rather than perform it on-the-fly, and add a test to the repo ensuring it stays up-to-date. That test will still need to fetch the external repository, but other builds won't. Continuing the example above, I added documentation for `pip_parse` showing how you could `load` the `requirements.bzl` file from within your repo: [https://github.com/bazelbuild/rules\\\_python/blob/main/docs/pip.md#vendoring-the-requirementsbzl-file](https://github.com/bazelbuild/rules\\_python/blob/main/docs/pip.md#vendoring-the-requirementsbzl-file) If you do this, there will no longer be a `load` statement from `@my_deps` in the WORKSPACE, which should fix the eager fetch. Another approach is to defer the work from a repository rule to an action that runs later in the BUILD graph. This generally requires changes to the rules you're using, so I won't try to give an example for end-users to follow. Again, you should first profile your build to understand which eager fetches are really a problem in practice. External repositories are locally cached by Bazel and shouldn't be invalidated often. ## BUILD eager fetches BUILD files also contain `load` statements, causing the external repository being loaded to be fetched. Unlike the WORKSPACE case above, the behavior depends on whether Bazel needs to analyze the BUILD file, which is the case if it is transitively referenced from targets the user requests to build or test. For example, in this BUILD file, we load from under the `@npm` repository: ```python theme={null} # Content of //pkg1:BUILD load("@npm//@bazel/typescript:index.bzl", "ts_project") package(default_visibility = ["//visibility:public"]) ts_project( name = "a", srcs = glob(["*.ts"]), declaration = True, tsconfig = "//:tsconfig.json", deps = [ "@npm//@types/node", "@npm//tslib", ], ) filegroup(name = "b") ``` As a result, if the user asks to build `a`, or *any target* in the `pkg1` package such as `b`, then the full fetch of `@npm` will be eager. This also happens if the user asks for a target which directly or indirectly load's *any target* in this package, such as `//pkg2:c` shown here: ```python theme={null} # Content of //pkg2:BUILD filegroup(name = "c", srcs = ["//pkg1:b"]) ``` Another example comes from using the `requirements` helper provided by rules\_python. If you use the suggested pattern ```python theme={null} load("@pip//:requirements.bzl", "requirement") py_library( name = "foo", ... deps = [ requirement("requests"), ], ) ``` this also causes an eager-fetch of whatever is in `@pip` which might cause all Python dependencies to be downloaded! I added this warning to the [documentation for pip\_install](https://github.com/bazelbuild/rules_python/blob/main/docs/pip.md#pip_install): > Note that this convenience comes with a cost. Analysis of any BUILD file which loads the requirements helper in this way will cause an eager-fetch of all the pip dependencies, even if no python targets are requested to be built. In a multi-language repo, this may cause developers to fetch dependencies they don't need, so consider using the long form for dependencies if this happens. ## Mitigating For BUILD fetches, the shape of the BUILD file graph matters. As with any programming language, it's a design smell when your imports come from many different unrelated places. Try to avoid BUILD files that load from external repositories and *also* contain other targets which don't use those repositories. Another mitigation is to reduce the size of the external repository being fetched. In the npm example example, we fetched `@npm` which might have a large number of packages, and if it uses `npm_install` or `yarn_install` from `build_bazel_rules_nodejs`, then *all* of those packages had to be installed just to get the `@bazel/typescript` one actually needed by this build. You could have a separate `package.json` listing file with a small number of dependencies loaded by BUILD files, with another `npm_install` repository rule fetching a repo like `@npm_bazel_deps`. This would still be eager-fetched, but it would be much faster. In some cases the eager fetch is just for syntax sugar. In the rules\_python `requirement` example, we could have just used `@pypi__requests//:pkg` in the `deps`, with no `load` statement at all. You can file issues as well. This takes longer to resolve, but it's healthier for the ecosystem to report these problems. Often the maintainers of the ruleset you use don't know about the problem, because they only build targets that require fetching the repo and have a smaller project, so they just don't observe the negative effects. [https://github.com/bazelbuild/rules\\\_nodejs/issues/3193](https://github.com/bazelbuild/rules\\_nodejs/issues/3193) is the issue for the example above. ## Preventing regression Eager fetches are subtle, and easy to introduce into your Bazel build. You only notice them when the external repository is invalidated, and are only outraged enough to file issues when you're building something you feel is unrelated. "Why am I re-compiling a python interpreter from source just to run my Go test??" You can't write a test within Bazel to catch this condition (as far as I know). But you can formulate something outside of Bazel, and then run this as a separate step/pipeline on CI. There are a couple methods. The first is clumsy but reproduces exactly what you observe: write a test to the effect of "If I do a clean Bazel build of target `//:foo`, then look in the external directory, I should not observe the presence of the unrelated repo `bar`." See [https://github.com/aspect-build/bazel-examples/tree/main/eager-fetch](https://github.com/aspect-build/bazel-examples/tree/main/eager-fetch) for both the working example, and a sample `test_no_eager_fetch.sh` script which makes the assertion. The second is to use a `bazel query` to detect a path through the dependency graph from a target to an undesired external repo. We'll just query for all the packages that some targets depend on, then grep for those in a repo, and if the result is non-empty then we found an eager fetch: ```plaintext theme={null} bazel query --output=package 'let targets = set(//some:target //some/other:target) in buildfiles(deps($targets))' | uniq | sort | grep @slow_repo ``` # Bazel 9 Upstream Prebuilt Protobuf Source: https://site.aspect.build/blog/bazel-9-protobuf Bazel 9 includes a prebuilt upstream protobuf compiler. Bazel 9.0 is a major long-term support (LTS) release. It contains new features and backwards incompatible changes. * Bzlmod is now always enabled, and all WORKSPACE logic has been removed from Bazel ([#26131](https://github.com/bazelbuild/bazel/issues/26131)). The [Bzlmod migration tool is available](https://bazel.build/external/migration_tool). * All C++-related [rules](https://github.com/bazelbuild/bazel/issues/26131) are [removed](https://bazel.build/external/migration_tool) from Bazel and must be loaded from @rules\_cc, as part of the [Starlark effort](https://github.com/bazelbuild/bazel/issues/23043). * [Bazel 9](https://github.com/bazelbuild/bazel/issues/23043) ships with the latest protobuf module (version 33.4), which includes support for a prebuilt protobuf compiler. This prebuilt protobuf compiler follows up on our blog article [never compile protoc again](/blog/never-compile-protoc-again) which ended with the promise “We’re hoping to upstream our `toolchains_protoc` to the protobuf repository, so that the default Bazel experience will be fast.” This is now (nearly) true as of the [v33.4 release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4). We thank the [Google Protobuf](https://github.com/protocolbuffers/protobuf) team for sponsoring this work and reviewing the implementation, including changes to the release process. ## Enabling the Protobufs toolchains feature Bazel 9 flips the [`--incompatible_enable_proto_toolchain_resolution`](https://registry.build/flag/bazel/?flag=incompatible_enable_proto_toolchain_resolution) flag to true. This means Bazel is responsible for resolving the symbol `@protobuf//bazel/private:proto_toolchain_type` to a “concrete” toolchain that provides the right `protoc` binary for your execution platform. This is also the case for a toolchain\_type for each language stub generator. If you’re on Bazel 9, there’s nothing to do, but earlier Bazels require you set the flag. ## Opting-in Until [https://github.com/protocolbuffers/protobuf/pull/25313](https://github.com/protocolbuffers/protobuf/pull/25313) lands and is released, you need to opt-in to using the `protoc` binaries from [https://github.com/protocolbuffers/protobuf/releases](https://github.com/protocolbuffers/protobuf/releases). Add to your `.bazelrc`: ```bash theme={null} common --@protobuf//bazel/toolchains:prefer_prebuilt_protoc ``` > note, if you have a `repo_name=com_google_protobuf` you’ll have to adapt the `@protobuf` name ## Enforcing You can enforce that no rules do the “Wrong Thing” of directly referencing the `cc_binary` target `@protobuf//:protoc` by following our snippets: [https://github.com/aspect-build/toolchains\_protoc#ensure-protobuf-and-grpc-never-built](https://github.com/aspect-build/toolchains_protoc#ensure-protobuf-and-grpc-never-built) If this fails to build, it means that there’s a bug you should find or report. The [https://github.com/aspect-build/toolchains\_protoc](https://github.com/aspect-build/toolchains_protoc) and [https://github.com/bazelbuild/rules\_proto](https://github.com/bazelbuild/rules_proto) repositories are now archived, completing the deprecation period. ## Next Need help migrating from WORKSPACE to Bzlmod or other steps to move to Bazel 9? View [Aspect Build services](http://www.aspect.build/services) and email us at [hello@aspect.build](mailto:hello@aspect.build) or ping us on Bazel Slack. We’re happy to support you. # Bazel can write to the source folder! Source: https://site.aspect.build/blog/bazel-can-write-to-the-source-folder Bazel can write to the source folder for specific needs, using `bazel run` and `bazel test` to maintain consistency Bazel is Google's open-sourced build tool. When used internally at Google, it comes along with a bunch of idioms which Googlers naturally take for granted, and associate with Bazel. These can accidentally become part of the accepted dogma around Bazel migration. Most frequently, the accident I see is a false perception "Bazel cannot write to the source folder, so you can no longer check in generated files, nor have them in the sources but ignored from VCS". ## Typically you shouldn't do it Intermediate outputs in Bazel are meant to be used directly as inputs to another target in the build. For example, if you generate language-specific client stubs from a `.proto` file, those stay in the `bazel-out` folder and a later compiler step should be configured to read them from there. However there are plenty of cases where outputs do need to go in the source folder: * workaround for an editor plugin that only knows to read in the source folder and can't be configured to look in bazel-out * "golden" or "snapshot" files used for tests * generated documentation that's checked in next to sources * files that you need to be able to search or browse from your version control GUI ## Yes you can do it If you restrict yourself to only `bazel build` and `bazel test`, then it's true that neither of these commands can mutate the source tree. Bazel is strictly a transform tool from the sources to its own bazel-out folder. However, `bazel run` has no such limitation, and in fact always sets an environment variable `BUILD_WORKSPACE_DIRECTORY` which makes it easy to find your sources and modify them. This leads us to the "Write to Sources" pattern for Bazel. We'll use `bazel run` to make the updates, and `bazel test` to make sure developers don't allow the file in the source folder to drift from what Bazel generates. Note that this pattern does have one downside, compared with build tools that allow a build to directly output into the source tree. Until you run the tests, it's possible that you're working against an out-of-date file in the source folder. This could mean you spend some time developing, only to find on CI that the generated file needs to be updated, and then after updating it, you have to make some fixes to the code you wrote. The easiest way to use this pattern is with rules that already exist for this purpose. Aspect has a [`write_source_files`](https://registry.bazel.build/modules/bazel_lib#lib-write_source_files-bzl) rule, and another option is [`updatesrc`](https://github.com/cgrindel/bazel-starlib/tree/main/updatesrc) from Chuck Grindel. You can also assemble the parts yourself, directly in a `BUILD.bazel` file. Here's the basic recipe, which I've adapted to many scenarios. For example, many of the core Bazel rulesets now use this pattern to keep their generated API markdown files in sync with the sources. ```python theme={null} load("@bazel_skylib//rules:diff_test.bzl", "diff_test") load("@bazel_skylib//rules:write_file.bzl", "write_file") # Config: # Map from some source file to a target that produces it. # This recipe assumes you already have some such targets. _GENERATED = { "some-source": "//:generated.txt", # ... } # Create a test target for each file that Bazel should # write to the source tree. [ diff_test( name = "check_" + k, # Make it trivial for devs to understand that if # this test fails, they just need to run the updater # Note, you need bazel-skylib version 1.1.1 or greater # to get the failure_message attribute failure_message = "Please run: bazel run //:update", file1 = k, file2 = v, ) for [k, v] in _GENERATED.items() ] # Generate the updater script so there's only one target for devs to run, # even if many generated files are in the source folder. write_file( name = "gen_update", out = "update.sh", content = [ # This depends on bash, would need tweaks for Windows "#!/usr/bin/env bash", # Bazel gives us a way to access the source folder! "cd $BUILD_WORKSPACE_DIRECTORY", ] + [ # Paths are now relative to the workspace. # We can copy files from bazel-bin to the sources "cp -fv bazel-bin/{1} {0}".format( k, # Convert label to path v.replace(":", "/"), ) for [k, v] in _GENERATED.items() ], ) # This is what you can `bazel run` and it can write to the source folder sh_binary( name = "update", srcs = ["update.sh"], data = _GENERATED.values(), ) ``` You may want to tweak the recipe, for example if the output files are markdown I'll append ".md" to the keys. If your files follow a convention you might be able to configure it with just a list rather than a dictionary. # Device management: tools on your developers PATH Source: https://site.aspect.build/blog/bazel-devenv Explore easier developer tool distribution with Bazel and direnv for seamless environment setup and management in your development workflow “Device Management”, or MDM, is that thing which forces your work computer to have security software installed. It has the ability to push tools to developer machines too - however it’s owned by the security team at your company. While it would be convenient, I’ve found it’s difficult for the Developer Platform team to use that to distribute the "canonical developer environment”. Some folks use a devcontainer or VDI (Virtual Desktop Infrastructure) to describe the tooling you need installed, but that’s heavy and slow. If we’re using Bazel, it already has features to give a hermetic environment, right? A year ago, I wrote an article describing how to get tools on your developers local machine using Bazel, and today I have an update. In the technique from my original post ([run tools installed by bazel](/blog/run-tools-installed-by-bazel)), users have to change their behavior, typing `./tools/my-tool` rather than just `my-tool`. Retraining developers to have a different command under their fingers is hard, especially for things they run all the time. Shortly after I wrote that post, Fabian Meumertzheim created [https://github.com/buildbuddy-io/bazel\_env.bzl](https://github.com/buildbuddy-io/bazel_env.bzl). This is an alternative technique I’ll write about today. It puts the tools on the `$PATH` instead, fixing the ergonomic issue from the first technique — however there are always trade-offs! This one requires engineers manually install the `direnv` tool before they get setup. ## How direnv works **direnv** is a command-line tool that automatically sets and unsets environment variables when you `cd` into or out of a directory. It’s especially useful for managing project-specific environment variables, such as secrets, configuration settings, or language versions (like Python or Node.js versions). * You place a `.envrc` file in your project directory. * This `.envrc` file contains shell commands to export environment variables or run setup scripts. * When you enter the directory, `direnv` loads the `.envrc` file and applies the environment changes. * When you leave the directory, it automatically reverts those changes. Here’s how it looks when I enter Aspect’s monorepo (named “silo”): ```plaintext theme={null} alexeagle@aspect-build ~ % cd Projects/silo direnv: loading ~/Projects/silo/.envrc direnv: export ~PATH ``` Installing `direnv` isn’t just a matter of getting the program on your machine. It also needs to hook into your shell (it supports `bash`, `zsh` and many others). And finally, you must explicitly allow each `.envrc` to be trusted. To follow this pattern, you’ll need to instruct your developers to install this first tool manually. Fortunately they’ll get a reminder of the instructions in the next step. ## bazel\_env creates a .envrc After installing `bazel_env.bzl` you’ll have a runnable Bazel target, typically `bazel run //:_bazel_env` or `bazel run //tools:bazel_env`. The `bazel_env` target is defined with a dictionary that maps a tool name to put on the PATH, to some other target that provides it. What kinds of targets can those be? Take a look at the example: [https://github.com/buildbuddy-io/bazel\_env.bzl/blob/main/examples/BUILD.bazel](https://github.com/buildbuddy-io/bazel_env.bzl/blob/main/examples/BUILD.bazel) 1. Binary targets for programs you author yourself in the monorepo 2. Tools provided by a toolchain, like `go`, `node`, `pnpm`, `cargo`, etc 3. With [https://github.com/theoremlp/rules\_multitool](https://github.com/theoremlp/rules_multitool) you can run `multitool` to update a `tools.lock.json` file, and all of these tools are installed. 4. CLI utilities distributed by a package manager, like console scripts from PyPI, `bin` entries from the `package.json` of NPM packages, Go utilities (see `scaffold` example here), and so on. ```plaintext theme={null} % bazel run //tools:bazel_env INFO: Analyzed target //tools:bazel_env (580 packages loaded, 72963 targets configured). INFO: Found 1 target... Target //tools:bazel_env up-to-date: bazel-bin/tools/bazel_env_all_tools INFO: Elapsed time: 9.399s, Critical Path: 0.45s INFO: Running command line: bazel-bin/tools/bazel_env.sh ====== bazel_env ====== ✅ direnv is installed ✅ direnv added bazel-out/bazel_env-opt/bin/tools/bazel_env/bin to PATH Tools available in PATH: * aws: @aws * pnpm: @pnpm * gofumpt: @@rules_multitool~~multitool~multitool//tools/gofumpt:gofumpt * jsonnetfmt: @@rules_multitool~~multitool~multitool//tools/jsonnetfmt:jsonnetfmt * shfmt: @@rules_multitool~~multitool~multitool//tools/shfmt:shfmt * terraform: //tools:terraform * yamlfmt: @@rules_multitool~~multitool~multitool//tools/yamlfmt:yamlfmt * ruff: @@rules_multitool~~multitool~multitool//tools/ruff:ruff * shellcheck: @@rules_multitool~~multitool~multitool//tools/shellcheck:shellcheck * buf: @@rules_multitool~~multitool~multitool//tools/buf:buf * buildozer: @@rules_multitool~~multitool~multitool//tools/buildozer:buildozer * docker-compose: @@rules_multitool~~multitool~multitool//tools/docker-compose:docker-compose * diesel-cli: @@rules_multitool~~multitool~multitool//tools/diesel-cli:diesel-cli * etcdctl: @@rules_multitool~~multitool~multitool//tools/etcdctl:etcdctl * grpcurl: @@rules_multitool~~multitool~multitool//tools/grpcurl:grpcurl * ibazel: @@rules_multitool~~multitool~multitool//tools/ibazel:ibazel * multitool: @@rules_multitool~~multitool~multitool//tools/multitool:multitool * otel-cli: @@rules_multitool~~multitool~multitool//tools/otel-cli:otel-cli * otelcol-contrib: @@rules_multitool~~multitool~multitool//tools/otelcol-contrib:otelcol-contrib * promtool: @@rules_multitool~~multitool~multitool//tools/promtool:promtool * pyrra: @@rules_multitool~~multitool~multitool//tools/pyrra:pyrra * tflint: @@rules_multitool~~multitool~multitool//tools/tflint:tflint * tfsec: @@rules_multitool~~multitool~multitool//tools/tfsec:tfsec * buildifier: @buildifier_prebuilt//:buildifier * scaffold: @com_github_hay_kot_scaffold//:scaffold * node: $(NODE_PATH) * cargo: $(CARGO) * rustfmt: $(RUSTFMT) Toolchains available at stable relative paths: * nodejs: bazel-out/bazel_env-opt/bin/tools/bazel_env/toolchains/nodejs * rust: bazel-out/bazel_env-opt/bin/tools/bazel_env/toolchains/rust direnv: loading ~/Projects/silo/.envrc direnv: export ~PATH ``` ## Guardrails There are a few things that can go wrong, which are worth pointing out. You’ll note that the tools are actually installed under a named output folder, `bazel-out/bazel_env-opt` - what if the user runs a `bazel clean`? This case is well handled, since `direnv` is able to report errors in the `.envrc` file with a custom message: ```plaintext theme={null} alexeagle@aspect-build silo % bazel clean INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes. direnv: loading ~/Projects/silo/.envrc direnv: ERROR[bazel_env.bzl]: Run 'bazel run //tools:bazel_env' to regenerate bazel-out/bazel_env-opt/bin/tools/bazel_env/bin direnv: export ~PATH ``` Any time the tools change, everyone has to run it again. It’s also not lazy (at least not yet). When you run `bazel_env` it needs to fetch all the tools, even those you never plan to run. See [https://github.com/buildbuddy-io/bazel\_env.bzl/issues/14](https://github.com/buildbuddy-io/bazel_env.bzl/issues/14) As a workaround, you can have multiple `bazel_env` targets, but users have to choose the right one for the work they intend to do. # Bazel for SONiC: What We've Learned and Contributed Source: https://site.aspect.build/blog/bazel-for-sonic We explore the case for adopting Bazel across SONiC Foundation projects and update the SONiC and Bazel community on Aspect Build contributions. As the [Linux Foundation](https://www.linuxfoundation.org/)’s [**SONiC Foundation**](https://sonicfoundation.dev/) continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: **how to build, test, and release SONiC components with speed, consistency, and confidence**. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software. This is exactly where **Bazel**, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers **SONiC** (**S**oftware for **O**pen **N**etworking **i**n the **C**loud) contributors, adopters and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale. Below, we explore the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering. *** ## 1. Reproducible Builds Are No Longer Optional SONiC today supports an expanding matrix of devices, ASICs, and software packages. That's its strength—but also its build challenge. Makefiles, ad-hoc scripts, and hand-rolled toolchains multiply variability and increase onboarding time, especially for new vendors. Bazel provides hermetic, reproducible builds, ensuring that: * The same inputs always produce the same outputs * Dependencies are fetched deterministically and cached * Toolchains and container images are versioned and immutable * Builds can run anywhere—from a developer's laptop to CI runners to cloud build farms This consistency dramatically reduces "it works on my machine" issues. For SONiC's multi-vendor ecosystem, it means any contributor can build and test with confidence against a shared standard. *** ## 2. A Universal Build System for a Polyglot Codebase SONiC involves C++, Python, Go, Rust, Docker, kernel modules, switch-vendor SDKs, and more. Bazel excels here: * First-class multi-language support * Extensibility through Starlark rules * Deterministic container and image builds * Cross-compilation support for ARM, x86, PowerPC, and ASIC-specific toolchains Instead of maintaining fragmented build logic across repositories, SONiC developers can standardize on a single system that handles everything from low-level DPDK components to high-level services. *** ## 3. Cloud-Native CI/CD That Scales With the SONiC Community Bazel was designed for massive scale and parallelism. For SONiC maintainers, this directly translates into: * Faster builds through fine-grained caching * Incremental rebuilds that recompile only what changed * Remote execution to distribute builds across clusters * Remote caching to avoid duplicated work between developers and CI jobs * Unified pipelines across all repos and languages As the community continues to expand, this infrastructure lets new contributors ramp up quickly and ensures that CI is fast, stable, and cost-efficient. *** ## 4. Stronger Security and Compliance Networking software is increasingly subject to regulatory scrutiny. SONiC vendors must prove the provenance, integrity, and patch level of every component. Bazel strengthens security by: * Locking down external dependencies * Ensuring reproducibility (critical for supply-chain audits) * Providing deterministic SBOM generation * Enforcing hermetic builds that eliminate environment drift * Integrating easily with tools for SLSA, sigstore, and in-toto For organizations integrating SONiC into large-scale infrastructure, this reduces risk and simplifies compliance workflows. *** ## 5. Better Collaboration Between Vendors and the Community One of SONiC's greatest strengths is its vendor-neutral ecosystem. Bazel reinforces this mission: * All vendors compile with the same rules and toolchains * Contributors can share Starlark extensions for SDKs or hardware targets * Build logic becomes collaborative, reviewable, and testable—just like code * Onboarding new vendors becomes faster and less error-prone Instead of each party maintaining private build scripts, SONiC can establish a shared, open-standard build vocabulary. *** ## 6. Future-Proofing SONiC for the Next Decade SONiC is evolving quickly—into new form factors, new silicon, new service models, and new deployment patterns. With Bazel, SONiC gains a foundation that: * Scales horizontally as code volume increases * Easily supports new languages or frameworks * Provides deterministic releases that downstream integrators can trust * Enables advanced workflows like distributed testing or reproducible builds in the cloud Bazel is not a short-term patch; it is an investment in SONiC's long-term velocity and stability. *** ## The Legacy Build System: Complexity at Scale To understand why Bazel is necessary, it's worth examining what SONiC's build process looked like before: a Makefile-based system built around a "slave" container image that encapsulated system dependencies. ### Arbitrary Commands, Arbitrary Problems The legacy system consists of **317 Makefile rules** (`.mk` files) and dependency files (`.dep` files), one for nearly every package, container, and component in SONiC. Each rule is allowed to invoke arbitrary commands: `apt-get install`, `pip install`, `dget`, shell scripts, custom build logic. On the surface, this sounds reasonable—the "slave" container provides isolation. But at SONiC's scale, this approach becomes a liability: **Brittleness**: Each recipe fetches dependencies from the internet on-demand. A network hiccup, a removed package from Debian archives, a deprecated Python module on PyPI, and the entire build fails. There's no guarantee that tomorrow's build will work the same way as today's. No pinning, no snapshot repositories, no explicit version control of dependencies. **Non-Hermeticity**: Because recipes are allowed to fetch and execute arbitrary commands, the build's output depends on what's available on the internet *right now*. Build two identical SONiC commits a week apart, and they may produce different binaries. The build machine's OS and installed packages are implicit dependencies—if you upgrade Ubuntu on your build server, you risk silently changing SONiC's outputs. **Unmaintainability at Scale**: With 317 separate rule files, coordinating changes is nightmarish. A Debian package update might require changes to multiple recipes. Dependency injection during build (where rules can declare new dependencies at build time) means the full dependency graph is opaque until runtime. You don't know what's actually going to be built until you run `make`. **Hidden Complexity**: The "slave" container approach obscures the real problem. Developers think they're building in a consistent environment because it's Docker-based, but the container itself is built by the same brittle, non-hermetic process. You're wrapping chaos in a box and calling it reproducibility. ### Recipes Inject Dependencies During Build Making matters worse, the legacy system allows rules to **inject dependencies dynamically during the build process**. A recipe isn't just a fixed input-output mapping; it's a script that can decide at runtime what to depend on. This makes the dependency graph fundamentally unknowable until build execution. You can't reason about what the artifact will contain—you have to run it and see. ### The Scale Problem SONiC spans dozens of C/C++ components, multiple Python packages, Go binaries, container images, kernel modules, and ASIC-specific toolchains. The legacy system required maintainers to write and maintain Makefiles for all of this, with no unifying framework. Each component has its own recipe, its own build logic, its own fragile internet-based dependency fetching. When SONiC grew to support ARM architectures, multiple Debian versions (Stretch, Buster, Bullseye, Bookworm), and dozens of ASIC vendors, the number of recipes multiplied. The build system became a maintenance burden that grew faster than the codebase itself. *** ## Bazel is decades ahead as a build system—but it’s not autopilot. You still need a good engineer to make it fit the problem. Bazel is uniquely suited to address the intricate build and test challenges faced by the SONiC project. As SONiC evolves to support a growing range of devices, languages, and architectures, the need for a robust build system becomes paramount. Bazel’s hermetic, reproducible builds help ensure that the same inputs consistently yield the same outputs—an essential property in a diverse ecosystem. With deterministic dependency management and strong cross-compilation support, Bazel aligns well with SONiC’s goals of consistency across platforms and contributors. But Bazel isn’t a silver bullet. It’s decades ahead as a build system, yet its architecture doesn’t solve the problem automagically—real success still depends on good engineering: clear rule boundaries, disciplined dependencies, and a toolchain/packaging strategy that makes Bazel’s guarantees real in practice. ## The Deep Technical Challenge: Hermetic Package Dependency Resolution Building a polyglot, multi-architecture system like SONiC using Bazel exposed a fundamental tension between how traditional Linux package managers work and how Bazel is designed to operate. The solution has required solving multiple interconnected problems that most build systems never encounter. The sections below summarize some of the Bazel for SONiC issues that [Aspect Build](http://www.aspect.build) and has helped to identify and address. ### The .deb Package Overlay Problem Debian packages are designed to be installed sequentially into a shared filesystem. When you run `apt-get install`, each package unpacks its files into the same root directory—creating a virtual "overlay" of files from potentially hundreds of packages. Circular dependencies are resolved through this overlay: library A might contain a symlink to library B, which is provided by a completely different package. The order of installation handles conflicts, and the running system sees a unified merged filesystem. Bazel, by contrast, abhors large, opaque target outputs. Each target should be hermetic and reproducible, with explicit dependencies and minimal side effects. Representing "unpack all these packages and overlay them" as a single Bazel target creates an enormous, unwieldy build artifact that defeats caching, incremental builds, and reproducibility. Yet SONiC needs precisely this—a complete, consistent sysroot with hundreds of interdependent Debian packages. The solution: *Bazel needed to compute the transitive closure of package dependencies, resolve symlinks ahead of time, and produce a metadata representation of the merged filesystem without materializing the entire overlay*\*\*.\*\* Rather than creating a giant unpacked sysroot target, `rules_distroless` generates a "Contents" file—a snapshot mapping every filename to the package that provides it. This allows Bazel to: * Resolve symlink chains before build time (avoiding "dangling symlink" errors that plague container builds) * Determine the final location of each library or header file despite circular package dependencies * Share the metadata across builds without replicating hundreds of MB of unpacked packages * Enable fine-grained caching at the package level, not the sysroot level ### The RPATH Nightmare The ELF dynamic linker, [`ld.so`](http://ld.so), searches for shared libraries in a precise order: 1. Directories in the binary's `DT_RPATH` attribute (if `DT_RUNPATH` is absent) 2. `LD_LIBRARY_PATH` environment variable 3. Directories in `DT_RUNPATH` (the modern preferred approach) 4. System cache (`/etc/`[`ld.so`](http://ld.so)`.cache`) 5. Default paths (`/lib`, `/usr/lib`, and architecture-specific variants) By default, binaries built in SONiC expect their dependencies in `/usr/lib`, `/lib`, and architecture-specific variants like `/usr/lib/aarch64-linux-gnu`. But Bazel builds place outputs in `bazel-out/`, a completely different path structure. The linker has no idea where to find [`libyang.so`](http://libyang.so)`.2.0` when it's buried in `bazel-out/aarch64-linux-gnu/bin/external/com_github_sonic_net_sonic_mgmt_common/lib/`[`libyang.so`](http://libyang.so)`.2.0`. The fix requires embedding `RPATH` entries directly into the binary at link time—telling the linker "look in `$ORIGIN/../lib` for my dependencies." But this introduces new challenges: * **Relocation**: A binary linked with `RPATH=$ORIGIN/../lib` expects a specific directory structure. If that binary is later used as a tool in another Bazel action (where it's relocated to a different path), the `RPATH` becomes invalid. * **Transitive Dependencies**: When a binary depends on library A, which depends on library B, the linker must find all three—but each may have different `RPATH` settings. Bazel must ensure the transitive closure is visible to the linker without creating massive, monolithic targets. SONiC solved this through **Bazel Configurations and Transitions** —a mechanism that applies different compiler flags to different parts of the dependency graph. Some targets (like Python's C extensions) are compiled with `-fPIC` and packaged into a sysroot. Everything else uses the standard compilation model. Bazel's transitions ensure these different compilation modes never mix, preventing linker errors where position-dependent code tries to reference position-independent code. ### The RPATH Nightmare Continues: Finding Dependencies in the Sandbox When Bazel runs a test target, it constructs a temporary sandbox directory containing only the files that test explicitly depends on. A test might link against a dynamic C++ library, which depends on [`libc.so.6`](http://libc.so), which transitively depends on [`ld-linux-x86-64.so.2`](http://ld-linux-x86-64.so). At test runtime, none of these libraries are in the system `/usr/lib`—they're scattered across `bazel-out/` in the test's sandbox. **The breakthrough**: Rather than relying on `LD_LIBRARY_PATH` (which is fragile and defeats reproducibility), `rules_distroless` ensures that when a test binary is linked, its `RPATH` contains the exact paths to all transitive dependencies *within that test's sandbox*. The binary becomes self-contained—it knows exactly where to find every library it needs, relative to its own location. A test can now be run anywhere, on any machine, and it will find its dependencies without environment variable manipulation. This is genuinely hermetic testing: the test's success or failure depends only on the code and its declared dependencies, not on what happens to be installed on the developer's machine. ### Container Runtime: Binaries Finding Dependencies Across Layers When a SONiC container starts, thousands of binaries are present—p4rt, swss, gnmi, monitoring agents, and hundreds of utilities. None of them are statically linked. Each binary must find its shared libraries at runtime. In a traditional Docker/container image built with a Dockerfile, everything is installed into standard locations (`/lib`, `/usr/lib`). The dynamic linker searches these paths by default. Simple and naive. But SONiC containers built with Bazel take a different approach. Multiple packages provide the same functionality (e.g., multiple versions of [`libprotoc.so`](http://libprotoc.so)), but only one should be "selected" for the final image. Furthermore, if SONiC switches from Debian bookworm to Debian trixie, or moves between Ubuntu LTS versions, the exact libraries available change. Hardcoding `/usr/lib/x86_64-linux-gnu/`[`libfoo.so.1`](http://libfoo.so) in a binary's `RPATH` would break when the library moves or changes versions. **The solution**: Bazel generates a container image where *the RPATH of every binary is already computed to find the exact libraries that will be present in that specific image*. When p4rt starts in the container, its `RPATH` contains the paths where `rules_distroless` placed [`libprotoc.so`](http://libprotoc.so), [`libyang.so`](http://libyang.so), and everything else. The binary finds its dependencies through the RPATH, not through the system's default search paths. This means: * The container is self-describing: a binary's dependencies are "baked in" rather than discovered at runtime * Moving from Debian bookworm to trixie doesn't break existing binaries—Bazel recomputes the RPATH for the new distro * Libraries can be placed anywhere in the image without breaking binaries * Container images are reproducible: given the same Bazel build configuration, the same binaries will find the same libraries every time ### Liberation from Host Distro Dependencies Perhaps the most profound impact: `rules_distroless` decouples the build machine's operating system from the built artifact. Traditionally, if you build SONiC on Ubuntu 24.04 and then try to run the binaries on Ubuntu 22.04, you hit subtle glibc incompatibilities. Some binaries depend on functions only available in Ubuntu 24's glibc version. Others link against the "wrong" version of OpenSSL. Developers spend weeks debugging "works on my machine" failures. With `rules_distroless`, the build process explicitly pins every single Debian package using Debian snapshot repositories. When you build SONiC on an Ubuntu 22.04 machine, the build ignores the system's installed packages and fetches exact, pinned versions from the snapshots. The build output contains binaries linked against those exact packages, regardless of what's on the build machine. This means: * A developer on Ubuntu 20.04 can build SONiC targeting Debian bookworm without installing bookworm libraries locally * CI/CD runners can be any modern Linux distro—the build is isolated from the host * Build results are reproducible across machines because dependency versions are explicit, not implicit * Upgrading the build machine's OS doesn't accidentally change SONiC's binaries The build machine becomes nearly irrelevant. You're no longer asking "how do I build this on my machine?" You're asking "given a Bazel configuration and a snapshot of the Debian archive, what do I build?" The answer is the same everywhere, because Bazel controls everything. ### Radical Hermeticity: Explicit Dependencies, Zero Implicit Assumptions The commitment to hermeticity goes far beyond RPATH tuning and package pinning. SONiC's C/C++ compilation is built with compiler flags that reject the very concept of "system libraries": ```plaintext theme={null} -nostdinc -nostdinc++ -nostdlib ``` These flags tell the compiler: "There are no standard libraries. There are no default include paths. Everything must be explicitly declared." Without these flags, a compiler would silently fall back to the build machine's `/usr/include` and `/usr/lib`. A developer on Ubuntu 24.04 might accidentally use headers from Ubuntu 24's glibc, even if the target is Debian bookworm. With `-nostdinc -nostdinc++ -nostdlib`, that accident becomes impossible—the build fails loudly if a dependency is missing. Every header file, every standard library function, every bit of C runtime must be explicitly provided as a Bazel dependency. This guarantees that: * A build on any machine produces identical binaries * Adding a dependency automatically updates the build configuration (nothing hidden) * Removing unused dependencies is impossible—they're explicitly declared * Switching between libc implementations or versions is straightforward—just change the declared dependency ### Auto-Generated cc\_library Targets: Turning .deb Packages into Bazel Dependencies But declaring every libc header and every system library explicitly would require thousands of manual `cc_library` rules. That's where `rules_distroless` shines. When `rules_distroless` processes a Debian package, it doesn't just extract files. It **automatically generates Bazel** `cc_library` targets that encapsulate: * All header files from the package (C standard library headers, C++ standard library headers, architecture-specific headers) * All shared object libraries * The correct include paths and linker settings * All transitive dependencies, automatically resolved From a SONiC developer's perspective, instead of hoping that libc headers and libraries exist somewhere on the system, they explicitly declare: ```python theme={null} cc_binary( name = "my_tool", srcs = ["main.cc"], deps = [ "@debian//libc6", "@debian//libstdc++", ], ) ``` The Bazel build system now understands exactly which Debian packages this binary depends on. If you want to upgrade libc, you change one line in MODULE.bazel. If you want to use a different libc or switch distributions entirely, the build system tracks it explicitly. **This auto-generation is the key to polyrepo hermetic builds.** In a monorepo, one team can write all the libc rules. But SONiC spans multiple GitHub repositories. Each repo's BUILD files can independently declare which Debian packages it needs. The `rules_distroless` machinery automatically generates compatible `cc_library` targets, and Bazel's module system ensures they all use the same versions across all repositories. No more silent fallback to system libraries. No more "it works on my machine but not in CI." Every dependency is explicit, versionable, and traceable. ### The Performance Paradox Computing transitive closure of dependencies across a polyrepo, resolving symlink chains, and managing RPATH dynamically sounds expensive — and it is. But doing it at build time, once, and caching the result is far cheaper than doing it at container runtime or — worse — debugging "symbol not found" errors weeks later in production. The trick is recognizing what can and cannot be cached: * **Package metadata** (which files each package provides, symlink targets) is stable and highly cacheable * **Sysroot layout** (which package provides which file when all dependencies are considered) is computed once and reused * **Binary relocations** (embedding RPATH in binaries) happens at link time and is reproducible * **Container assembly** (final layer selection) happens downstream and doesn't affect upstream caching SONiC's Bazel infrastructure treats these as separate concerns, allowing massive parallelism and cache reuse even though the final build is complex. *** ## Solving the Python Dependency Maze: Explicit, Versioned, Cross-Compilable Python presents a unique challenge in reproducible builds. Unlike C/C++ where dependencies are system packages installed via a package manager, Python's ecosystem fetches from PyPI—a centralized repository where package availability and versioning can change. The standard approach to managing Python dependencies in Bazel has been `rules_python`, but it has limitations that made it unsuitable for SONiC's polyglot, multi-architecture environment. ### The PyPI Problem at Scale SONiC uses Python extensively: configuration generation tools, management daemons, testing frameworks, and utilities. Each Python package has transitive dependencies on other packages, many of which are compiled extensions (`.so` files). The challenge is that `rules_python`'s traditional `pip_parse` implementation: * Assumes a single target architecture (no cross-compilation support) * Doesn't pin package versions deterministically * Struggles with compiled Python extensions that depend on system libraries * Doesn't integrate well with polyrepo ecosystems where multiple repositories have different Python dependency requirements In SONiC's case, you might need to build the same Python package for x86-64 and ARM64 simultaneously. The standard tooling wasn't built for that. ### Enter Aspect Rules Python: Modern Dependency Management To solve these problems, [**Aspect Rules Python**](https://github.com/aspect-build/rules_py) (`aspect_rules_py`) provides a modern reimplementation of Python dependency management in Bazel. It replaces the traditional `pip_parse` with a new implementation based on **uv**, a fast, production-grade Python package resolver. Key improvements: **Explicit Versioning**: Dependencies are pinned in a lock file (similar to `requirements.lock`), ensuring that every build uses the exact same package versions. No surprises from PyPI changes. **Cross-Compilation Support**: Unlike the original `rules_python`, `aspect_rules_py` can build Python packages for different architectures simultaneously. This is critical for SONiC: the same Python source code can be compiled as a wheel for x86-64, then recompiled for ARM64, with all dependencies resolved correctly for each target. **Integration with Hermetic Sysroots**: When a Python package has compiled extensions that depend on system libraries (e.g., a C extension that links against `libyang`), the sysroot provided by `rules_distroless` makes the correct headers and libraries available. `aspect_rules_py` integrates seamlessly with this sysroot, so the package's build automatically uses the right C compiler flags, header locations, and link paths for the target architecture. **Polyrepo-Friendly**: Each SONiC repository declares and resolves its own Python dependency closure independently. `aspect_rules_py` uses uv to compute the complete transitive dependency graph for each repository, generating a lock file that pins all transitive dependencies. Because each repo has its own lock file, different repositories can use different versions of shared dependencies without conflict—Bazel's module system keeps them isolated. ### Enabling Container Cross-Compilation Combined with the Debian package management improvements (`rules_distroless`), `aspect_rules_py` enables a powerful capability: **building complete container images for different architectures within a single Bazel build**. Before, cross-compiling a SONiC container for ARM64 required: 1. Running the build on an ARM64 machine (or emulating it, which was slow) 2. Managing different Python dependency versions for different architectures 3. Handling the mismatch between the build machine's Python environment and the target architecture With `aspect_rules_py` and `rules_distroless`, Bazel can: 1. Resolve Python dependencies for the target architecture (e.g., ARM64) 2. Build wheels for that architecture using the sysroot's C compiler and libraries 3. Layer those wheels into the container image alongside the pinned Debian packages 4. All within a single Bazel invocation on any machine This means a developer on an x86-64 laptop can type `bazel build //path/to:docker-swss-arm64` and get a fully cross-compiled container image without any special setup, emulation, or native ARM64 hardware. *** ## A Call to Action for the SONiC Foundation The SONiC community is at an inflection point. As deployments reach hyperscale and the contributor base grows more diverse, the project needs a build and test system that matches its ambitions. Bazel offers exactly that: a modern, reproducible, cloud-scale platform that empowers contributors, accelerates releases, and strengthens the entire ecosystem. Adopting Bazel across SONiC Foundation projects will: * Reduce fragmentation * Increase developer productivity * Improve security and auditing * Enable faster and more reliable releases * Provide a consistent experience for every vendor and contributor The benefits compound over time—and they're aligned with SONiC's vision of an open, interoperable, high-performance network OS. **Let's give SONiC the build system it deserves**. ### Accelerating Innovation Through Reproducible, Scalable, Cloud-Native Build Systems In this article, we have explored the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering. We’ve also updated the Bazel and SONiC communities on some of our recent contributions to empower Bazel for SONiC. As the SONiC Foundation continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: **how to build, test, and release SONiC components with speed, consistency, and confidence**. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software. This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC contributors and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale. ## Next Steps Interested in learning more about how to succeed with Bazel for SONiC? [Schedule time to talk with us](https://calendly.com/aspect-build/intro?back=1\&month=2026-01) or email us at [hello@aspect.build](mailto:hello@aspect.build). # Bazel market growth, year over year Source: https://site.aspect.build/blog/bazel-market-2023 Bazel adoption grew by 56% in 2023, with around 950 companies now using it. Learn more about the growth and its implications for businesses and developers. A year ago, I wrote about how Google hadn't produced any adoption numbers for Bazel. However, companies like ours must make informed investment decisions, and so must a lot of individuals deciding whether the Bazel ecosystem is "big enough". This could be as simple as "should I convince my husband that it's worth me spending my nights and weekends to write a book about Bazel", or "is it economical for my consulting company to advertise our services by hosting a Bazel podcast". On the other hand, it could involve institutional investors who are putting millions of dollars into a business plan that is counting on projected revenue from users to make Bazel more usable by enterprises. Here's that post from last year: [https://blog.aspect.dev/estimating-bazel-adoption](https://blog.aspect.dev/estimating-bazel-adoption). The summary: in the absence of data from Google, we used estimated **600 companies were using Bazel**. ## This year Fast-forward to BazelCon this year. In the keynote, there was a wide open opportunity for the Bazel team to provide some adoption numbers. Even though Bazel still doesn't have telemetry built-in, there's a simple proxy - the same one we always used for the Angular JavaScript framework: distinct active users of the documentation site. Sadly, while the Bazel team is interested in providing this data, there are bureaucratic obstacles in privacy policy and legal standing in the way, so they still haven't published anything. I remain optimistic that we'll get it eventually! Fortunately, my co-founder Greg is willing to do all the work to reproduce our analysis from last year, and our result is that **there are now about 950 companies using Bazel, a 56% increase from last year.** ## Details I won't repeat the methodology we used, as you can read about it in the post from last year. Essentially, we take two different approaches to list which companies use Bazel, then see how many companies appear in both lists. Assuming a uniform "density", this gives us an approximation of how many Bazel users appear on *neither* list, and so we can extrapolate a total. Last year we published this data: ```plaintext theme={null} TBA = [# public list] * [#private list] / [# overlap] = 218 * 127 / 46 = 602 ``` Updating for November 2023, we now arrive at: ```plaintext theme={null} TBA = [# public list] * [#private list] / [# overlap] = 311 * 228 / 75 = 945 ``` # Fixing Bazel out-of-memory problems Source: https://site.aspect.build/blog/bazel-oom Troubleshoot and fix Bazel's out-of-memory issues in both JVM and system contexts with practical solutions and tips. Memory management is a generally hard topic in computer systems operations. Debugging it inside a cloud-hosted build system is even worse! There are two potential problems: * The Bazel server runs in a JVM, and it internally tries to allocate more objects than the max heap size its allowed. * Bazel spawns subprocesses (called "actions", including test actions) and they collectively exhaust the memory in the machine or VM that Bazel runs in. I'll cover these scenarios separately since they're mostly unrelated. > Of course, the Bazel JVM heap does occupy system memory, so they're related in the sense that a smaller Bazel server footprint would allow for more actions to run, but I've never considered that to be a potential remediation. ## Bazel server out-of-memory How to tell this is happening: * Bazel exits with code 33 (see [`ExitCodes.java`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/util/ExitCode.java#L58)) * Check the output of `bazel info | grep heap` if the Bazel server is still running, see if it is near the max. Some things you can do about it: * Give it more RAM! Assuming the system has some available, you can use the [`host_jvm_args` startup flag](https://docs.bazel.build/versions/main/command-line-reference.html#flag--host_jvm_args) to adjust the usual JVM parameters like `-Xmx2g`. * Always turn on the [`--heap_dump_on_oom` flag](https://bazel.build/reference/command-line-reference#flag--heap_dump_on_oom) so that you get extra information in this case. * "Memory saving mode" can be useful if you're hosting Bazel in ephemeral CI workers where you expect every build to be cold, however that's slow and not recommended. If you do that, you can avoid Bazel tracking incremental state which saves some memory. [https://bazel.build/configure/memory](https://bazel.build/configure/memory) * Roll up your sleeves and figure out what's consuming so much memory in Bazel's JVM. Start from [Bazel's documentation on memory profiling](https://bazel.build/rules/performance#memory-profiling). An example can be rulesets where data is repeated rather than using depsets, an example analysis: [https://github.com/aspect-build/rules\_js/pull/391](https://github.com/aspect-build/rules_js/pull/391) ## System out-of-memory Bazel schedules actions (build steps and test runners) based on the amount of system resources it thinks are available, and using some heuristic about how much RAM a typical action requires. Two kinds of things can go wrong, either Bazel thinks more RAM is available than the system actually has free, or Bazel underestimates the resources to be reserved for a given action. By default, Bazel's max concurrency is based on the heuristic that each action needs one CPU core, so the `--jobs` flag default is the number of (maybe virtual) CPUs on the machine. Note that Bazel reports progress with a "X running" indicator which might lead you to believe that the concurrency is actually higher, but that's a misleading message because it can include actions that are queued waiting for resources

Did you know when @bazelbuild prints progress like
\[12 / 100] 32 actions, 30 running
That "running" count includes "remote-cache" spawns! If you have --jobs=16 (the default on 16 core) the other 14 of them aren't actually running, they're queued for "local" spawn. [https://t.co/a3w7TWZVTM](https://t.co/a3w7TWZVTM)
— Alex 🦅 Eagle (@Jakeherringbone) August 17, 2022