> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# rctx.download custom headers coming to Bazel 7.1

> Learn how to use HTTP headers in Bazel's downloader for fetching Docker layers, Alpine packages, and more. Improve caching and avoid common HTTP issues.

export const BlogPost = ({title, date, authors, tags, image, children}) => {
  const tagList = tags ? tags.split(", ").filter(Boolean) : [];
  const tagSlug = t => t.toLowerCase().replace(/&/g, "").replace(/\+/g, "").replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
  const formattedDate = date ? new Date(date + "T00:00:00").toLocaleDateString("en-US", {
    year: "numeric",
    month: "long",
    day: "numeric"
  }) : "";
  return <section className="w-full flex justify-center px-4 py-12 md:py-16">
      <div style={{
    maxWidth: "800px",
    width: "100%"
  }}>
        {image && (typeof image === "string" ? <img noZoom src={image} alt={title} className="w-full rounded-xl mb-8" style={{
    maxHeight: "400px",
    objectFit: "cover"
  }} /> : <div className="blog-post-hero-image">{image}</div>)}
        <h1 className="text-3xl md:text-4xl font-bold text-zinc-900 dark:text-white">
          {title}
        </h1>
        <div className="flex flex-wrap items-center gap-3 mt-4 text-sm text-zinc-500 dark:text-zinc-400">
          {authors && <span>{authors}</span>}
          {authors && formattedDate && <span>·</span>}
          {formattedDate && <span>{formattedDate}</span>}
        </div>
        {tagList.length > 0 && <div className="flex flex-wrap gap-2 mt-3">
            {tagList.map(tag => <a key={tag} href={"/blog/tags/" + tagSlug(tag)} className="px-2 py-0.5 rounded-full text-xs bg-zinc-100 dark:bg-zinc-800 text-zinc-600 dark:text-zinc-400 hover:bg-blue-100 dark:hover:bg-blue-900/40 hover:text-blue-700 dark:hover:text-blue-300 transition">
                {tag}
              </a>)}
          </div>}
        <hr className="my-8 border-zinc-200 dark:border-zinc-700" />
        <div className="prose dark:prose-invert max-w-none">{children}</div>
      </div>
    </section>;
};

export const MarketingPage = () => <div className="marketing-page-marker" style={{
  display: "none"
}} />;

export const Section = ({children, className = "", gray = false, dark = false, id}) => <section id={id} className={`w-full flex justify-center px-4 py-16 md:py-24 ${gray ? "bg-gray-50 dark:bg-zinc-900" : dark ? "bg-zinc-900 dark:bg-zinc-950" : ""} ${className}`}>
    <div className="w-full" style={{
  maxWidth: "1140px"
}}>
      {children}
    </div>
  </section>;

<MarketingPage />

<BlogPost title="rctx.download custom headers coming to Bazel 7.1" date="2024-01-03" authors="Şahin Yort" tags="Bazel">
  <img noZoom src="https://mintcdn.com/aspectbuild/x1L7Iep716jCyJVo/images/blog/hashnode/hn-386af5b2db.jpeg?fit=max&auto=format&n=x1L7Iep716jCyJVo&q=85&s=246d29047fa4f5bd66661ca04c19cfa2" alt="" className="blog-post-cover" width="1600" height="1067" data-path="images/blog/hashnode/hn-386af5b2db.jpeg" />

  Bazel is responsible for fetching files from the internet in most cases. From a `WORKSPACE` file you may have seen this as `http_archive` or `http_file`. Behind the scenes, when writing a repository rule, this uses the `repository_ctx#download` API, [https://bazel.build/rules/lib/builtins/repository\_ctx#download](https://bazel.build/rules/lib/builtins/repository_ctx#download) (or the `download_and_extract` variant).

  > Note that some rulesets decided not to use the Bazel downloader. For example, rules\_go uses `go mod download` [https://pkg.go.dev/cmd/go#hdr-Download\_modules\_to\_local\_cache](https://pkg.go.dev/cmd/go#hdr-Download_modules_to_local_cache) and rules\_python uses `pip download`.

  The Bazel Downloader API is great because the files are always cached in the Bazel repository\_cache. If you enable `--experimental_remote_downloader` you can even use your Remote Cache to serve as a read-through proxy of the files. The Downloader also has a rich configuration ability.

  As ruleset authors, we've relied heavily on this Bazel API. But, we ran into some cases that didn't work. For example, the Docker registry is an HTTP server with lots of surprises. At least that has been our experience implementing a [repository rule](https://bazel.build/extending/repo) for rules\_oci for fetching container images from registries like DockerHub.

  ## Docker pull and HTTP Headers

  In rules\_oci, we don't want to run `docker pull`. We want to use the Bazel downloader to get the benefits listed above.

  Internally, container images are just a few tar archives called "layers" which are put together via a "Manifest" file, which is a straightforward `json` file. Like everything else in the software industry, this json file also has its [variations](https://github.com/moby/moby/tree/master/image/spec). We are mostly interested in the [v2](https://distribution.github.io/distribution/spec/manifest-v2-2/) version as it is widely used, while v1 is deprecated and not in use.

  However, some people don't like to move with everybody else. So this encouraged Docker to implement a fallback behavior where if you are trying to request a manifest from a registry, and you don't have `Accept` HTTP header set with specific mime types, then the registry will happily assume that you are an old person trying to fetch an image that has the v2 manifest and gladly downgrade the manifest to v1.

  Unfortunately Bazel's downloader had no way to set headers, so it was not suitable for fetching Docker layers. We had to workaround this by reverting to `curl` in our repository rule. But then the downloaded files are no longer cached, and so this is not a principled solution.

  Thanks to Bazel being open-source, this was correctable! I filed the issue for this: [https://github.com/bazelbuild/bazel/issues/17829](https://github.com/bazelbuild/bazel/issues/17829). Then I worked with the Bazel team to land a fix for it. Thanks to Fabian and Tiago for reviewing the PR and getting it landed!

  The feature that allows setting arbitrary HTTP headers is already on HEAD and available on the [documentation](https://bazel.build/rules/lib/builtins/repository_ctx#download) page but has not been cherry-picked into Bazel's 7.1-release branch yet. However, it can be used today by putting `2b8885ed954e58d09d47290d47882a7298c88334` into `.bazelversion` file.

  ## Alpine

  This problem of setting headers is not unique to Docker registries. Alpine packages also have a property that also requires us to send some headers. However, this case is a little different than Docker one. Alpine uses an [archive format](https://wiki.alpinelinux.org/wiki/Apk_spec) which is simply multiple gzip streams of tars combined. It is a lot like \`.deb\` [archives](https://en.wikipedia.org/wiki/Deb_\(file_format\)).

  These three segments are;

  * `signature`: contains signature

  * `control`: contains metadata information about the package.

  * `data`: contains files such as libs and executables.

  The problem with `.apk` packages is that unlike the `control` , `data` segments, the `signature` segment is not guaranteed to be stable, meaning its contents might change without notice. This is a problem in the Bazel world because in order to be reproducible everything we fetch from the internet has to be identical even after five years from now.

  Luckily the signature segment is optional, some packages don't even have it, and can be omitted if needed. However, since an `.apk` is a single archive consisting of three `.tar` archives we can't just get the last two. This is where we need to use [HTTP range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests) to only fetch the last two segments of an archive so that it's always stable. In order to do that, we need to send a `Range` header to tell the HTTP server to stream only the range of bytes we specified.

  ## Trying it out

  Here's an example of an oci\_pull rule that demonstrates how setting the `Accept` header changes the response of the registry.

  ```python theme={null}
  def _simple_pull_impl(rctx):
      url = "https://index.docker.io/v2/fluxcd/flux/manifests/1.25.4"
      # Get the token: curl -fsSL "https://auth.docker.io/token?service=registry.docker.io&scope=repository:fluxcd/flux:pull"
      auth_pattern = {
          url: { "type": "pattern", "pattern": "Bearer <password>", "password": "<token>" }
      }
      headers = {
          "Accept": "application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v2+json",
      }
      rctx.download(
          url = url,
          auth = auth_pattern,
          # If you comment out the headers argument, the schema version will be 1.
          headers = headers,
          output = "manifest.json"
      )
      manifest = json.decode(rctx.read("manifest.json"))
      print("schemaVersion: %s" % manifest["schemaVersion"])

      rctx.file("BUILD.bazel", """filegroup(name = "manifest", srcs = ["manifest.json"])""")

  simple_pull = repository_rule(
      implementation = _simple_pull_impl,
  )
  ```

  With HTTP headers the sky is the limit. We have only talked about a few specific problems, but this feature allows Bazel users to do much more than that.

  Imagine if you want to download a big archive from the Artifactory but it takes too much time. Well, you can accelerate the fetching by downloading different parts of the archive in parallel by making multiple HTTP requests simultaneously. I'll post more on that later.
</BlogPost>