The term "CD" is ambiguous. Some engineers use it to mean "Continuous Deployment", in which changes are automatically released, e.g. into a "dev" environment.
Aspect recommends that Continuous Delivery is modeled as the step of the pipeline where built artifacts are uploaded from the build machine to a well-known repository location. This could be a container image registry like Docker Hub, a blob store like AWS S3, or even a database.
This makes a clear separation of responsibilities between CI, CD and Deployment:
* The CI pipeline runs all the tests to confirm the repo is in a shippable state
* The CD pipeline should then upload only the artifacts that are:
* configured with `BUILD.bazel` files. Product engineers don't need to worry about setting up CD
* green: it can prove that all relevant tests are passing
* changed from a previous build
* The deployment system
* locates and "promotes" artifacts to the next environment, such as "dev", "staging", or "prod".
## Build vs. Buy
The recommendations in this guide can be applied in two ways:
* DevInfra teams may wish to implement and operate a custom system for their organization, or
* Use [Aspect Workflows](https://aspect.build/workflows), which provides this feature out-of-the-box.
## What is "deliverable"
A deliverable artifact is one that contains both the binary or files to push, as well as the "pushing" logic that knows how to perform the upload. It might also send a message to the deployment system to trigger an auto-deployment of the new artifact.
In Bazel terms, this means a deliverable should be an executable program that can be `bazel run`.
### Container Images
The rules\_oci [`oci_push`](https://github.com/bazel-contrib/rules_oci/blob/main/docs/push.md#oci_push) or rules\_docker [`container_push`](https://github.com/bazelbuild/rules_docker/blob/master/docs/container.md#container_push) rules can both be executed with `bazel run` to push a Docker image to a registry like Docker Hub.
Therefore these rules are considered "deliverable".
### Git Push
Sometimes artifacts belong in a separate code repository. For example, an SDK built from the API definitions in a monorepo needs to be published.
See an example `git_push` executable [in this repository](https://github.com/aspect-build/bazel-examples/tree/main/git_push).
### S3 upload
See [s3\_sync](https://github.com/aspect-build/rules_aws/tree/main/examples/release_to_s3).
## Which targets to deliver
A `bazel query` expression is the most convenient way to locate deliverable targets. Users may choose a tagging scheme for their workspace (i.e. "all targets with `tags = ['artifact']`"), or deliver well-known rule kinds (i.e. `oci_push`), or both.
## Which changes to deliver
To optimize time and money, it's best to deliver only "changed" targets. This avoids wasted time and resources uploading the same artifact repeatedly. It also means that release engineers won't have to sort through a massive list of duplicates when choosing a release.
There are two approaches for choosing "changed" targets:
1. Predict the changes based on a version control delta. For example you could `git diff` between the hash being delivered and the "prior successful" delivery hash, then use a tool like [bazel-diff](https://github.com/Tinder/bazel-diff) or [target-determinator](https://github.com/bazel-contrib/target-determinator) to produce a list of targets that might be affected by those changes.
2. Determine empirically based on what is actually different. This requires determinism, so it must only use [unstamped](https://github.com/bazel-contrib/bazel-lib/blob/main/docs/stamping.md) build results (`--nostamp`). In most cases a green CI run just completed, so these unstamped outputs are easily available.
Aspect recommends following the second approach because the first has some downsides:
* [bazel-diff](https://github.com/Tinder/bazel-diff) is incorrect and will sometimes miss affected targets, so they aren't delivered.
* [target-determinator](https://github.com/bazel-contrib/target-determinator) is slow and may hurt the "service level indicator" of time between pushing a hotfix and being able to release that fix.
* It will over-deliver, because sometimes a source change doesn't actually factor into whether the release binary changes, such as for a comment-only change.
The rest of this section provides more details about the second approach (determine empirically).
To determine whether Workflows should deliver that executable target on a particular commit, it is first hashed using the [`aspect outputs` command](https://docs.aspect.build/cli/commands/aspect_outputs/) with a special pseudo-mnemonic "ExecutableHash", for example:
```sh theme={null}
$ aspect outputs 'attr("tags", "\bdeliverable\b", //...)' ExecutableHash
//cli:release h1:cj8OUC3l3fIr3Zxnffk6y7gukLOJmiWRCAQoqadg66Y=
//workflows/rosetta:release h1:kjHVajw+Nta2kh3Epcd32DkZxTE1NHA8b5N7hCNFNSM=
```
You need a lookup database to store previously delivered hashes. If the hash value matches one previously seen, then skip delivery of that target.
### Debugging changes to deliver
You can run the `aspect outputs` command locally to understand whether a given change to a source file results in a new executable. Sometimes the result may be surprising. For example, if a comment in a `.go` source file is changed, the compiler produces the same `.a` file as a result, so the hash seen on the uploader executable is unchanged.
Another scenario that won't change the executable is when some production configuration is changed. For example, you may use Helm charts to deploy to Kubernetes. If these aren't included inside the image, then changes to these files won't cause a new delivery.
## Perform the delivery
Run each deliverable target with [stamping](https://registry.bazel.build/modules/bazel_lib#lib-stamping-bzl) enabled. You can do this in a script which reads the targets from a manifest file, essentially `cat $delivery_manifest | xargs -N1 bazel run --stamp`.