Custom machine images - Aspect Build

Aspect Workflows runs Bazel directly on the host machine, not in a container, to avoid Docker-in-Docker complications. Runners therefore use a cloud machine image, not a container image. Aspect publishes open-source starter machine images, but you may need to build a custom image with Packer when:

You require tight control over security patches and versions of the OS and packages.
The starter images don’t include fixes for every vulnerability you track.
Your non-hermetic build or tests require specific system-level packages.

Step 1: Prepare the build machine

Download Packer for your operating system. To have Bazel manage the Packer binary instead, see the starter images repository linked in step 5.
Authenticate with your cloud provider (Amazon Web Services or Google Cloud Platform). Packer needs permission to create EC2 or Compute Engine instances.

Step 2: Choose a base image

AWS

Access the AWS console.
Access the EC2 Dashboard and select AMIs from the left sidebar.

Start with an Amazon-supplied image, for example Amazon Linux 2023 (al2023-ami). See Packer’s Getting Started with AWS for building AMIs.

GCP

Access the GCP console.
Navigate to Compute Engine > Images.
Search for the image that suits your needs, for example debian-13 or ubuntu-2404-lts.

See Packer’s Google Compute Builder for building GCP images.

Step 3: Create a Packer script

Packer scripts use the HashiCorp Configuration Language (HCL), like Terraform, with the .pkr.hcl file extension.

Add the Packer plugin for your cloud provider to your Packer script:
- For AWS use Amazon EBS.
- For GCP use Google Cloud Platform.
Create a locals block for the values you reuse:

locals {
  # This is a public Amazon Linux 2 image in us-east-2
  # We use this AMI because it already contains everything needed to interact with AWS
  # Name: amzn2-ami-kernel-5.10-hvm-2.0.20220719.0-x86_64-gp2
  source_ami = "ami-051dfed8f67f095f5"
  region = "us-east-2"
  platform = "linux/amd64"
}

Create a source block and a build block, following the Packer documentation.

Step 4: Dependencies

Minimal dependencies

Every custom machine image needs:

Dependency	Purpose
`fuse`	Required for the high-performance remote cache configuration.
`git`	Used for fetching source code for testing.
`mdadm`	Required when mounting NVMe drives with raid 0.
`rsync`	Used during bootstrap.
`rsyslog`	Used for system logging.

AWS AMI requirements

AWS AMIs also need:

Dependency	Purpose
`amazon-cloudwatch-agent`	Used for gathering logs.
`amazon-ssm-agent`	Required for Amazon Web Services SSM support.

amazon-ssm-agent comes pre-installed on Amazon Linux 2 and Amazon Linux 2023 base AMIs.

GCP image requirements

GCP images also need:

Dependency	Purpose
`google-osconfig-agent`	Google operational monitoring tools used to collect and alarm on critical telemetry.

GitHub Actions requirements

Runners for GitHub Actions also need:

Dependency	Purpose
`libicu`	Needed by the GitHub Actions agent. See Runner v2.303 not working on AmazonLinux2023 due to missing ICU package.

Recommended dependencies

Dependency	Purpose
`zip`	Required by Bazel if any tests create archives of undeclared test outputs.
`patch`	May be used by certain rule sets and package managers during dependency fetching.

Step 5: Add dependencies for your build

Install the packages your build needs, such as Docker, with apt-get or yum depending on your Linux distribution:

build {

  ...

  provisioner "shell" {
    inline = [
      # Install additional dependencies
      "sudo apt-get update",
      "sudo apt-get --assume-yes install clang-13 libgdal-dev openjdk-17-jdk-headless libtinfo5"
    ]
  }
}

The Aspect Workflows starter images repository has complete Packer files and a Bazel setup that runs a hermetic Packer binary.

Step 6: Test a new image

Test image changes on a separate canary runner group before they reach the runners your developers depend on. In your Workflows Terraform configuration:

Add a data "aws_ami" or data "google_compute_image" block that selects the canary image.
Add an entry to resource_types with image_id set to the canary image ID.
Add a runner group under runners (for example in runners.bk.groups) that selects the new resource type.
Run terraform apply.
Point one or more jobs in your CI configuration at the canary runner group’s queue.

To test the new image fully, force a complete, non-incremental build by adding cache-busting environment variables for actions and repository rules to .bazelrc:

# cache-bust for testing canary, values are arbitrary
build --repo_env=CACHE_BUST=001 --action_env=CACHE_BUST=001

​Step 1: Prepare the build machine

​Step 2: Choose a base image

​AWS

​GCP

​Step 3: Create a Packer script

​Step 4: Dependencies

​Minimal dependencies

​AWS AMI requirements

​GCP image requirements

​GitHub Actions requirements

​Recommended dependencies

​Step 5: Add dependencies for your build

​Step 6: Test a new image

Step 1: Prepare the build machine

Step 2: Choose a base image

AWS

GCP

Step 3: Create a Packer script

Step 4: Dependencies

Minimal dependencies

AWS AMI requirements

GCP image requirements

GitHub Actions requirements

Recommended dependencies

Step 5: Add dependencies for your build

Step 6: Test a new image