As the [Linux Foundation](https://www.linuxfoundation.org/)’s [**SONiC Foundation**](https://sonicfoundation.dev/) continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: **how to build, test, and release SONiC components with speed, consistency, and confidence**. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software.
This is exactly where **Bazel**, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers **SONiC** (**S**oftware for **O**pen **N**etworking **i**n the **C**loud) contributors, adopters and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.
Below, we explore the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering.
***
## 1. Reproducible Builds Are No Longer Optional
SONiC today supports an expanding matrix of devices, ASICs, and software packages. That's its strength—but also its build challenge. Makefiles, ad-hoc scripts, and hand-rolled toolchains multiply variability and increase onboarding time, especially for new vendors.
Bazel provides hermetic, reproducible builds, ensuring that:
* The same inputs always produce the same outputs
* Dependencies are fetched deterministically and cached
* Toolchains and container images are versioned and immutable
* Builds can run anywhere—from a developer's laptop to CI runners to cloud build farms
This consistency dramatically reduces "it works on my machine" issues. For SONiC's multi-vendor ecosystem, it means any contributor can build and test with confidence against a shared standard.
***
## 2. A Universal Build System for a Polyglot Codebase
SONiC involves C++, Python, Go, Rust, Docker, kernel modules, switch-vendor SDKs, and more. Bazel excels here:
* First-class multi-language support
* Extensibility through Starlark rules
* Deterministic container and image builds
* Cross-compilation support for ARM, x86, PowerPC, and ASIC-specific toolchains
Instead of maintaining fragmented build logic across repositories, SONiC developers can standardize on a single system that handles everything from low-level DPDK components to high-level services.
***
## 3. Cloud-Native CI/CD That Scales With the SONiC Community
Bazel was designed for massive scale and parallelism. For SONiC maintainers, this directly translates into:
* Faster builds through fine-grained caching
* Incremental rebuilds that recompile only what changed
* Remote execution to distribute builds across clusters
* Remote caching to avoid duplicated work between developers and CI jobs
* Unified pipelines across all repos and languages
As the community continues to expand, this infrastructure lets new contributors ramp up quickly and ensures that CI is fast, stable, and cost-efficient.
***
## 4. Stronger Security and Compliance
Networking software is increasingly subject to regulatory scrutiny. SONiC vendors must prove the provenance, integrity, and patch level of every component.
Bazel strengthens security by:
* Locking down external dependencies
* Ensuring reproducibility (critical for supply-chain audits)
* Providing deterministic SBOM generation
* Enforcing hermetic builds that eliminate environment drift
* Integrating easily with tools for SLSA, sigstore, and in-toto
For organizations integrating SONiC into large-scale infrastructure, this reduces risk and simplifies compliance workflows.
***
## 5. Better Collaboration Between Vendors and the Community
One of SONiC's greatest strengths is its vendor-neutral ecosystem. Bazel reinforces this mission:
* All vendors compile with the same rules and toolchains
* Contributors can share Starlark extensions for SDKs or hardware targets
* Build logic becomes collaborative, reviewable, and testable—just like code
* Onboarding new vendors becomes faster and less error-prone
Instead of each party maintaining private build scripts, SONiC can establish a shared, open-standard build vocabulary.
***
## 6. Future-Proofing SONiC for the Next Decade
SONiC is evolving quickly—into new form factors, new silicon, new service models, and new deployment patterns. With Bazel, SONiC gains a foundation that:
* Scales horizontally as code volume increases
* Easily supports new languages or frameworks
* Provides deterministic releases that downstream integrators can trust
* Enables advanced workflows like distributed testing or reproducible builds in the cloud
Bazel is not a short-term patch; it is an investment in SONiC's long-term velocity and stability.
***
## The Legacy Build System: Complexity at Scale
To understand why Bazel is necessary, it's worth examining what SONiC's build process looked like before: a Makefile-based system built around a "slave" container image that encapsulated system dependencies.
### Arbitrary Commands, Arbitrary Problems
The legacy system consists of **317 Makefile rules** (`.mk` files) and dependency files (`.dep` files), one for nearly every package, container, and component in SONiC. Each rule is allowed to invoke arbitrary commands: `apt-get install`, `pip install`, `dget`, shell scripts, custom build logic. On the surface, this sounds reasonable—the "slave" container provides isolation. But at SONiC's scale, this approach becomes a liability:
**Brittleness**: Each recipe fetches dependencies from the internet on-demand. A network hiccup, a removed package from Debian archives, a deprecated Python module on PyPI, and the entire build fails. There's no guarantee that tomorrow's build will work the same way as today's. No pinning, no snapshot repositories, no explicit version control of dependencies.
**Non-Hermeticity**: Because recipes are allowed to fetch and execute arbitrary commands, the build's output depends on what's available on the internet *right now*. Build two identical SONiC commits a week apart, and they may produce different binaries. The build machine's OS and installed packages are implicit dependencies—if you upgrade Ubuntu on your build server, you risk silently changing SONiC's outputs.
**Unmaintainability at Scale**: With 317 separate rule files, coordinating changes is nightmarish. A Debian package update might require changes to multiple recipes. Dependency injection during build (where rules can declare new dependencies at build time) means the full dependency graph is opaque until runtime. You don't know what's actually going to be built until you run `make`.
**Hidden Complexity**: The "slave" container approach obscures the real problem. Developers think they're building in a consistent environment because it's Docker-based, but the container itself is built by the same brittle, non-hermetic process. You're wrapping chaos in a box and calling it reproducibility.
### Recipes Inject Dependencies During Build
Making matters worse, the legacy system allows rules to **inject dependencies dynamically during the build process**. A recipe isn't just a fixed input-output mapping; it's a script that can decide at runtime what to depend on. This makes the dependency graph fundamentally unknowable until build execution. You can't reason about what the artifact will contain—you have to run it and see.
### The Scale Problem
SONiC spans dozens of C/C++ components, multiple Python packages, Go binaries, container images, kernel modules, and ASIC-specific toolchains. The legacy system required maintainers to write and maintain Makefiles for all of this, with no unifying framework. Each component has its own recipe, its own build logic, its own fragile internet-based dependency fetching. When SONiC grew to support ARM architectures, multiple Debian versions (Stretch, Buster, Bullseye, Bookworm), and dozens of ASIC vendors, the number of recipes multiplied.
The build system became a maintenance burden that grew faster than the codebase itself.
***
## Bazel is decades ahead as a build system—but it’s not autopilot. You still need a good engineer to make it fit the problem.
Bazel is uniquely suited to address the intricate build and test challenges faced by the SONiC project. As SONiC evolves to support a growing range of devices, languages, and architectures, the need for a robust build system becomes paramount. Bazel’s hermetic, reproducible builds help ensure that the same inputs consistently yield the same outputs—an essential property in a diverse ecosystem. With deterministic dependency management and strong cross-compilation support, Bazel aligns well with SONiC’s goals of consistency across platforms and contributors.
But Bazel isn’t a silver bullet. It’s decades ahead as a build system, yet its architecture doesn’t solve the problem automagically—real success still depends on good engineering: clear rule boundaries, disciplined dependencies, and a toolchain/packaging strategy that makes Bazel’s guarantees real in practice.
## The Deep Technical Challenge: Hermetic Package Dependency Resolution
Building a polyglot, multi-architecture system like SONiC using Bazel exposed a fundamental tension between how traditional Linux package managers work and how Bazel is designed to operate. The solution has required solving multiple interconnected problems that most build systems never encounter.
The sections below summarize some of the Bazel for SONiC issues that [Aspect Build](http://www.aspect.build) and has helped to identify and address.
### The .deb Package Overlay Problem
Debian packages are designed to be installed sequentially into a shared filesystem. When you run `apt-get install`, each package unpacks its files into the same root directory—creating a virtual "overlay" of files from potentially hundreds of packages. Circular dependencies are resolved through this overlay: library A might contain a symlink to library B, which is provided by a completely different package. The order of installation handles conflicts, and the running system sees a unified merged filesystem.
Bazel, by contrast, abhors large, opaque target outputs. Each target should be hermetic and reproducible, with explicit dependencies and minimal side effects. Representing "unpack all these packages and overlay them" as a single Bazel target creates an enormous, unwieldy build artifact that defeats caching, incremental builds, and reproducibility. Yet SONiC needs precisely this—a complete, consistent sysroot with hundreds of interdependent Debian packages.
The solution: *Bazel needed to compute the transitive closure of package dependencies, resolve symlinks ahead of time, and produce a metadata representation of the merged filesystem without materializing the entire overlay*\*\*.\*\* Rather than creating a giant unpacked sysroot target, `rules_distroless` generates a "Contents" file—a snapshot mapping every filename to the package that provides it. This allows Bazel to:
* Resolve symlink chains before build time (avoiding "dangling symlink" errors that plague container builds)
* Determine the final location of each library or header file despite circular package dependencies
* Share the metadata across builds without replicating hundreds of MB of unpacked packages
* Enable fine-grained caching at the package level, not the sysroot level
### The RPATH Nightmare
The ELF dynamic linker, [`ld.so`](http://ld.so), searches for shared libraries in a precise order:
1. Directories in the binary's `DT_RPATH` attribute (if `DT_RUNPATH` is absent)
2. `LD_LIBRARY_PATH` environment variable
3. Directories in `DT_RUNPATH` (the modern preferred approach)
4. System cache (`/etc/`[`ld.so`](http://ld.so)`.cache`)
5. Default paths (`/lib`, `/usr/lib`, and architecture-specific variants)
By default, binaries built in SONiC expect their dependencies in `/usr/lib`, `/lib`, and architecture-specific variants like `/usr/lib/aarch64-linux-gnu`. But Bazel builds place outputs in `bazel-out/`, a completely different path structure. The linker has no idea where to find [`libyang.so`](http://libyang.so)`.2.0` when it's buried in `bazel-out/aarch64-linux-gnu/bin/external/com_github_sonic_net_sonic_mgmt_common/lib/`[`libyang.so`](http://libyang.so)`.2.0`.
The fix requires embedding `RPATH` entries directly into the binary at link time—telling the linker "look in `$ORIGIN/../lib` for my dependencies." But this introduces new challenges:
* **Relocation**: A binary linked with `RPATH=$ORIGIN/../lib` expects a specific directory structure. If that binary is later used as a tool in another Bazel action (where it's relocated to a different path), the `RPATH` becomes invalid.
* **Transitive Dependencies**: When a binary depends on library A, which depends on library B, the linker must find all three—but each may have different `RPATH` settings. Bazel must ensure the transitive closure is visible to the linker without creating massive, monolithic targets.
SONiC solved this through **Bazel Configurations and Transitions** —a mechanism that applies different compiler flags to different parts of the dependency graph. Some targets (like Python's C extensions) are compiled with `-fPIC` and packaged into a sysroot. Everything else uses the standard compilation model. Bazel's transitions ensure these different compilation modes never mix, preventing linker errors where position-dependent code tries to reference position-independent code.
### The RPATH Nightmare Continues: Finding Dependencies in the Sandbox
When Bazel runs a test target, it constructs a temporary sandbox directory containing only the files that test explicitly depends on. A test might link against a dynamic C++ library, which depends on [`libc.so.6`](http://libc.so), which transitively depends on [`ld-linux-x86-64.so.2`](http://ld-linux-x86-64.so). At test runtime, none of these libraries are in the system `/usr/lib`—they're scattered across `bazel-out/` in the test's sandbox.
**The breakthrough**: Rather than relying on `LD_LIBRARY_PATH` (which is fragile and defeats reproducibility), `rules_distroless` ensures that when a test binary is linked, its `RPATH` contains the exact paths to all transitive dependencies *within that test's sandbox*. The binary becomes self-contained—it knows exactly where to find every library it needs, relative to its own location. A test can now be run anywhere, on any machine, and it will find its dependencies without environment variable manipulation. This is genuinely hermetic testing: the test's success or failure depends only on the code and its declared dependencies, not on what happens to be installed on the developer's machine.
### Container Runtime: Binaries Finding Dependencies Across Layers
When a SONiC container starts, thousands of binaries are present—p4rt, swss, gnmi, monitoring agents, and hundreds of utilities. None of them are statically linked. Each binary must find its shared libraries at runtime.
In a traditional Docker/container image built with a Dockerfile, everything is installed into standard locations (`/lib`, `/usr/lib`). The dynamic linker searches these paths by default. Simple and naive.
But SONiC containers built with Bazel take a different approach. Multiple packages provide the same functionality (e.g., multiple versions of [`libprotoc.so`](http://libprotoc.so)), but only one should be "selected" for the final image. Furthermore, if SONiC switches from Debian bookworm to Debian trixie, or moves between Ubuntu LTS versions, the exact libraries available change. Hardcoding `/usr/lib/x86_64-linux-gnu/`[`libfoo.so.1`](http://libfoo.so) in a binary's `RPATH` would break when the library moves or changes versions.
**The solution**: Bazel generates a container image where *the RPATH of every binary is already computed to find the exact libraries that will be present in that specific image*. When p4rt starts in the container, its `RPATH` contains the paths where `rules_distroless` placed [`libprotoc.so`](http://libprotoc.so), [`libyang.so`](http://libyang.so), and everything else. The binary finds its dependencies through the RPATH, not through the system's default search paths. This means:
* The container is self-describing: a binary's dependencies are "baked in" rather than discovered at runtime
* Moving from Debian bookworm to trixie doesn't break existing binaries—Bazel recomputes the RPATH for the new distro
* Libraries can be placed anywhere in the image without breaking binaries
* Container images are reproducible: given the same Bazel build configuration, the same binaries will find the same libraries every time
### Liberation from Host Distro Dependencies
Perhaps the most profound impact: `rules_distroless` decouples the build machine's operating system from the built artifact.
Traditionally, if you build SONiC on Ubuntu 24.04 and then try to run the binaries on Ubuntu 22.04, you hit subtle glibc incompatibilities. Some binaries depend on functions only available in Ubuntu 24's glibc version. Others link against the "wrong" version of OpenSSL. Developers spend weeks debugging "works on my machine" failures.
With `rules_distroless`, the build process explicitly pins every single Debian package using Debian snapshot repositories. When you build SONiC on an Ubuntu 22.04 machine, the build ignores the system's installed packages and fetches exact, pinned versions from the snapshots. The build output contains binaries linked against those exact packages, regardless of what's on the build machine.
This means:
* A developer on Ubuntu 20.04 can build SONiC targeting Debian bookworm without installing bookworm libraries locally
* CI/CD runners can be any modern Linux distro—the build is isolated from the host
* Build results are reproducible across machines because dependency versions are explicit, not implicit
* Upgrading the build machine's OS doesn't accidentally change SONiC's binaries
The build machine becomes nearly irrelevant. You're no longer asking "how do I build this on my machine?" You're asking "given a Bazel configuration and a snapshot of the Debian archive, what do I build?" The answer is the same everywhere, because Bazel controls everything.
### Radical Hermeticity: Explicit Dependencies, Zero Implicit Assumptions
The commitment to hermeticity goes far beyond RPATH tuning and package pinning. SONiC's C/C++ compilation is built with compiler flags that reject the very concept of "system libraries":
```plaintext theme={null}
-nostdinc -nostdinc++ -nostdlib
```
These flags tell the compiler: "There are no standard libraries. There are no default include paths. Everything must be explicitly declared." Without these flags, a compiler would silently fall back to the build machine's `/usr/include` and `/usr/lib`. A developer on Ubuntu 24.04 might accidentally use headers from Ubuntu 24's glibc, even if the target is Debian bookworm. With `-nostdinc -nostdinc++ -nostdlib`, that accident becomes impossible—the build fails loudly if a dependency is missing.
Every header file, every standard library function, every bit of C runtime must be explicitly provided as a Bazel dependency. This guarantees that:
* A build on any machine produces identical binaries
* Adding a dependency automatically updates the build configuration (nothing hidden)
* Removing unused dependencies is impossible—they're explicitly declared
* Switching between libc implementations or versions is straightforward—just change the declared dependency
### Auto-Generated cc\_library Targets: Turning .deb Packages into Bazel Dependencies
But declaring every libc header and every system library explicitly would require thousands of manual `cc_library` rules. That's where `rules_distroless` shines.
When `rules_distroless` processes a Debian package, it doesn't just extract files. It **automatically generates Bazel** `cc_library` targets that encapsulate:
* All header files from the package (C standard library headers, C++ standard library headers, architecture-specific headers)
* All shared object libraries
* The correct include paths and linker settings
* All transitive dependencies, automatically resolved
From a SONiC developer's perspective, instead of hoping that libc headers and libraries exist somewhere on the system, they explicitly declare:
```python theme={null}
cc_binary(
name = "my_tool",
srcs = ["main.cc"],
deps = [
"@debian//libc6",
"@debian//libstdc++",
],
)
```
The Bazel build system now understands exactly which Debian packages this binary depends on. If you want to upgrade libc, you change one line in MODULE.bazel. If you want to use a different libc or switch distributions entirely, the build system tracks it explicitly.
**This auto-generation is the key to polyrepo hermetic builds.** In a monorepo, one team can write all the libc rules. But SONiC spans multiple GitHub repositories. Each repo's BUILD files can independently declare which Debian packages it needs. The `rules_distroless` machinery automatically generates compatible `cc_library` targets, and Bazel's module system ensures they all use the same versions across all repositories.
No more silent fallback to system libraries. No more "it works on my machine but not in CI." Every dependency is explicit, versionable, and traceable.
### The Performance Paradox
Computing transitive closure of dependencies across a polyrepo, resolving symlink chains, and managing RPATH dynamically sounds expensive — and it is. But doing it at build time, once, and caching the result is far cheaper than doing it at container runtime or — worse — debugging "symbol not found" errors weeks later in production.
The trick is recognizing what can and cannot be cached:
* **Package metadata** (which files each package provides, symlink targets) is stable and highly cacheable
* **Sysroot layout** (which package provides which file when all dependencies are considered) is computed once and reused
* **Binary relocations** (embedding RPATH in binaries) happens at link time and is reproducible
* **Container assembly** (final layer selection) happens downstream and doesn't affect upstream caching
SONiC's Bazel infrastructure treats these as separate concerns, allowing massive parallelism and cache reuse even though the final build is complex.
***
## Solving the Python Dependency Maze: Explicit, Versioned, Cross-Compilable
Python presents a unique challenge in reproducible builds. Unlike C/C++ where dependencies are system packages installed via a package manager, Python's ecosystem fetches from PyPI—a centralized repository where package availability and versioning can change. The standard approach to managing Python dependencies in Bazel has been `rules_python`, but it has limitations that made it unsuitable for SONiC's polyglot, multi-architecture environment.
### The PyPI Problem at Scale
SONiC uses Python extensively: configuration generation tools, management daemons, testing frameworks, and utilities. Each Python package has transitive dependencies on other packages, many of which are compiled extensions (`.so` files). The challenge is that `rules_python`'s traditional `pip_parse` implementation:
* Assumes a single target architecture (no cross-compilation support)
* Doesn't pin package versions deterministically
* Struggles with compiled Python extensions that depend on system libraries
* Doesn't integrate well with polyrepo ecosystems where multiple repositories have different Python dependency requirements
In SONiC's case, you might need to build the same Python package for x86-64 and ARM64 simultaneously. The standard tooling wasn't built for that.
### Enter Aspect Rules Python: Modern Dependency Management
To solve these problems, [**Aspect Rules Python**](https://github.com/aspect-build/rules_py) (`aspect_rules_py`) provides a modern reimplementation of Python dependency management in Bazel. It replaces the traditional `pip_parse` with a new implementation based on **uv**, a fast, production-grade Python package resolver.
Key improvements:
**Explicit Versioning**: Dependencies are pinned in a lock file (similar to `requirements.lock`), ensuring that every build uses the exact same package versions. No surprises from PyPI changes.
**Cross-Compilation Support**: Unlike the original `rules_python`, `aspect_rules_py` can build Python packages for different architectures simultaneously. This is critical for SONiC: the same Python source code can be compiled as a wheel for x86-64, then recompiled for ARM64, with all dependencies resolved correctly for each target.
**Integration with Hermetic Sysroots**: When a Python package has compiled extensions that depend on system libraries (e.g., a C extension that links against `libyang`), the sysroot provided by `rules_distroless` makes the correct headers and libraries available. `aspect_rules_py` integrates seamlessly with this sysroot, so the package's build automatically uses the right C compiler flags, header locations, and link paths for the target architecture.
**Polyrepo-Friendly**: Each SONiC repository declares and resolves its own Python dependency closure independently. `aspect_rules_py` uses uv to compute the complete transitive dependency graph for each repository, generating a lock file that pins all transitive dependencies. Because each repo has its own lock file, different repositories can use different versions of shared dependencies without conflict—Bazel's module system keeps them isolated.
### Enabling Container Cross-Compilation
Combined with the Debian package management improvements (`rules_distroless`), `aspect_rules_py` enables a powerful capability: **building complete container images for different architectures within a single Bazel build**.
Before, cross-compiling a SONiC container for ARM64 required:
1. Running the build on an ARM64 machine (or emulating it, which was slow)
2. Managing different Python dependency versions for different architectures
3. Handling the mismatch between the build machine's Python environment and the target architecture
With `aspect_rules_py` and `rules_distroless`, Bazel can:
1. Resolve Python dependencies for the target architecture (e.g., ARM64)
2. Build wheels for that architecture using the sysroot's C compiler and libraries
3. Layer those wheels into the container image alongside the pinned Debian packages
4. All within a single Bazel invocation on any machine
This means a developer on an x86-64 laptop can type `bazel build //path/to:docker-swss-arm64` and get a fully cross-compiled container image without any special setup, emulation, or native ARM64 hardware.
***
## A Call to Action for the SONiC Foundation
The SONiC community is at an inflection point. As deployments reach hyperscale and the contributor base grows more diverse, the project needs a build and test system that matches its ambitions.
Bazel offers exactly that: a modern, reproducible, cloud-scale platform that empowers contributors, accelerates releases, and strengthens the entire ecosystem.
Adopting Bazel across SONiC Foundation projects will:
* Reduce fragmentation
* Increase developer productivity
* Improve security and auditing
* Enable faster and more reliable releases
* Provide a consistent experience for every vendor and contributor
The benefits compound over time—and they're aligned with SONiC's vision of an open, interoperable, high-performance network OS. **Let's give SONiC the build system it deserves**.
### Accelerating Innovation Through Reproducible, Scalable, Cloud-Native Build Systems
In this article, we have explored the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering. We’ve also updated the Bazel and SONiC communities on some of our recent contributions to empower Bazel for SONiC.
As the SONiC Foundation continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: **how to build, test, and release SONiC components with speed, consistency, and confidence**. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software.
This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC contributors and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.
## Next Steps
Interested in learning more about how to succeed with Bazel for SONiC? [Schedule time to talk with us](https://calendly.com/aspect-build/intro?back=1\&month=2026-01) or email us at [hello@aspect.build](mailto:hello@aspect.build).