CAPI Images

The Image Builder can be used to build images intended for use with Kubernetes CAPI providers. Each provider has its own format of images that it can work with. For example, AWS instances use AMIs, and vSphere uses OVAs.

Prerequisites

Packer and Ansible are used for building these images. This tooling has been forked and extended from the Wardroom project.

• Packer version >= 1.6.0
• Goss plugin for Packer version >= 1.2.0
• Ansible version >= 2.10.0

If any needed binaries are not present, they can be installed to images/capi/.bin with the make deps command. This directory will need to be added to your $PATH.

Providers

• AWS
• Azure
• CloudStack
• DigitalOcean
• GCP
• IBM Cloud
• Nutanix
• OCI
• 3DSOutscale
• OpenStack
• OpenStack remote image building
• Raw
• VirtualBox
• vSphere
• Proxmox

Make targets

Within this repo, there is a Makefile located at images/capi/Makefile that can be used to create the default images.

Run make or make help to see the current list of targets. The targets are categorized into Dependencies, Builds, and Cleaning. The Dependency targets will check that your system has the proper tools installed to run the build for your specific provider. If the dependencies are not present, they will be installed.

Configuration

The images/capi/packer/config directory includes several JSON files that define the default configuration for the images:

Due to OS differences, Windows images has additional configuration in the packer/config/windows folder. See Windows documentation for more details.

Customization

Several variables can be used to customize the image build.

The variables found in packer/config/*.json or packer/<provider>/*.json should not need to be modified directly. For customization it is better to create a JSON file with your changes and provide it via the PACKER_VAR_FILES environment variable. Variables set in this file will override any previous values. Multiple files can be passed via PACKER_VAR_FILES, with the last file taking precedence over any others.

Examples

Passing custom Kubernetes version

PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.3' --var 'kubernetes_semver=v1.28.3' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.3-1.1'" make ...

Passing a single extra var file

PACKER_VAR_FILES=var_file_1.json make ...

Passing multiple extra var files

PACKER_VAR_FILES="var_file_1.json var_file_2.json" make ...

Passing in extra packages to the image

If you wanted to install the RPMs nfs-utils and net-tools, create a file called extra_vars.json and populate with the following:

{
  "extra_rpms": "\"nfs-utils net-tools\""
}

Note that since the extra_rpms variable is a string, and we need the string to be quoted to preserve the space when placed on the command line, having the escaped double-quotes is required.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=extra_vars.json make build-node-ova-local-photon-3

Configuring Containerd at runtime

Containerd default configuration has the following imports value:

imports = ["/etc/containerd/conf.d/*.toml"]

This allows you to place files at runtime in /etc/containerd/conf.d/ that will then be merged with the rest of containerd configuration.

For example to enable containerd metrics, create a file /etc/containerd/conf.d/metrics.toml with the following:

[metrics]
  address = "0.0.0.0:1338"
  grpc_histogram = false

Disabling default repos and using an internal package mirror

A common use-case within enterprise environments is to have a package repository available on an internal network to install from rather than reaching out to the internet. To support this, you can inject custom repository definitions into the image, and optionally disable the use of the default ones.

For example, to build an image using only an internal mirror, create a file called internal_repos.json and populate it with the following:

{
  "disable_public_repos": "true",
  "extra_repos": "/home/<user>/mirror.repo",
  "remove_extra_repos": "true"
}

This example assumes that you have an RPM repository definition available at /home/<user>/mirror.repo, and it is correctly configured to point to your internal mirror. It will be added to the image within /etc/yum.repos.d, with all existing repositories found with /etc/yum.repos.d disabled by setting disable_public_repos to "true". Furthermore, the (optional) use of "remove_extra_repos" means that at the end of the build, the repository definition that was added will be removed. This is useful if the image you are building will be shared externally and you do not wish to include a file with internal network services and addresses.

For Ubuntu images, the process works the same but you would need to add a .list file pointing to your DEB package mirror.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=internal_repos.json make build-node-ova-local-photon-3

Setting up an HTTP Proxy

The Packer tool itself honors the standard env vars of HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. If these variables are set and exported, they will be honored if Packer needs to download an ISO during a build. However, in order to use these proxies with Ansible (for use during package installation or binary download), we need to pass them via JSON file.

For example, to set the HTTP_PROXY env var for the Ansible stage of the build, create a proxy.json and populate it with the following:

{
  "http_proxy": "http://proxy.corp.com"
}

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=proxy.json make build-node-ova-local-photon-3

Loading additional components using additional_components.json

{
  "additional_executables": "true",
  "additional_executables_destination_path": "/path/to/dest",
  "additional_executables_list": "http://path/to/exec1,http://path/to/exec2",
  "additional_s3": "true",
  "additional_s3_endpoint": "https://path-to-s3-endpoint",
  "additional_s3_access": "S3_ACCESS_KEY",
  "additional_s3_secret": "S3_SECRET_KEY",
  "additional_s3_bucket": "some-bucket",
  "additional_s3_object": "path/to/object",
  "additional_s3_destination_path": "/path/to/dest",
  "additional_s3_ceph": "true",
  "additional_registry_images": "true",
  "additional_registry_images_list": "plndr/kube-vip:0.3.4,plndr/kube-vip:0.3.3",
  "additional_url_images": "true",
  "additional_url_images_list": "http://path/to/image1.tar,http://path/to/image2.tar",
  "load_additional_components": "true"
}

Using ansible_user_vars to pass custom variables

{
  "ansible_user_vars": "var1=value1 var2={{ user `myvar2`}}",
  "myvar2": "value2"
}

Enabling Ansible custom roles

Put the Ansible role files in the ansible/roles directory.

{
  "firstboot_custom_roles_pre": "setupRole",
  "node_custom_roles_post": "role1 role2"
}

Note, for backwards compatibility reasons, the variable custom_role_names is still accepted as an alternative to node_custom_roles_post, and they are functionally equivalent.