CAPI Images

The Image Builder can be used to build images intended for use with Kubernetes CAPI providers. Each provider has its own format of images that it can work with. For example, AWS instances use AMIs, and vSphere uses OVAs.

Prerequisites

Packer and Ansible are used for building these images. This tooling has been forked and extended from the Wardroom project.

If any needed binaries are not present, they can be installed to images/capi/.bin with the make deps command. This directory will need to be added to your $PATH.

Providers

Make targets

Within this repo, there is a Makefile located at images/capi/Makefile that can be used to create the default images.

Run make or make help to see the current list of targets. The targets are categorized into Dependencies, Builds, and Cleaning. The Dependency targets will check that your system has the proper tools installed to run the build for your specific provider. If the dependencies are not present, they will be installed.

Configuration

The images/capi/packer/config directory includes several JSON files that define the default configuration for the images:

FileDescription
packer/config/ansible-args.jsonA common set of variables that are sent to the Ansible playbook
packer/config/cni.jsonThe version of Kubernetes CNI to install
packer/config/containerd.jsonThe version of containerd to install and customizations specific to the containerd runtime
packer/config/kubernetes.jsonThe version of Kubernetes to install. The default version is kept at n-2. See Customization section below for overriding this value

Due to OS differences, Windows images has additional configuration in the packer/config/windows folder. See Windows documentation for more details.

Customization

Several variables can be used to customize the image build.

VariableDescriptionDefault
firstboot_custom_roles_pre
firstboot_custom_roles_post
node_custom_roles_pre
node_custom_roles_post
Each of these four variables allows for giving a space delimited string of custom Ansible roles to run at different times. The “pre” roles run as the very first thing in the playbook (useful for setting up environment specifics like networking changes), and the “post” roles as the very last (useful for undoing those changes, custom additions, etc). Note that the “post” role does run before the “sysprep” role in the “node” playbook, as the “sysprep” role seals the image. If the role is placed in the ansible/roles directory, it can be referenced by name. Otherwise, it must be a fully qualified path to the role.""
disable_public_reposIf set to "true", this will disable all existing package repositories defined in the OS before doing any package installs. The extra_repos variable must be set for package installs to succeed."false"
extra_debsThis can be set to a space delimited string containing the names of additional deb packages to install""
extra_reposA space delimited string containing the names of files to add to the image containing repository definitions. The files should be given as absolute paths.""
extra_rpmsThis can be set to a space delimited string containing the names of additional RPM packages to install""
http_proxyThis can be set to URL to use as an HTTP proxy during the Ansible stage of building""
https_proxyThis can be set to URL to use as an HTTPS proxy during the Ansible stage of building""
kubernetes_deb_versionThis can be set to the version of kubernetes which will be installed in debian based image"1.23.10-00"
kubernetes_rpm_versionThis can be set to the version of kubernetes which will be installed in rpm based image"1.23.10-0"
kubernetes_semverThis can be set to semantic verion of kubernetes which will be installed in the image"v1.23.10"
kubernetes_seriesThis can be set to series version kubernetes which will be installed in the image"v1.23"
no_proxyThis can be set to a comma-delimited list of domains that should be exluded from proxying during the Ansible stage of building""
reenable_public_reposIf set to "false", the package repositories disabled by setting disable_public_repos will remain disabled at the end of the build."true"
remove_extra_reposIf set to "true", the package repositories added to the OS through the use of extra_repos will be removed at the end of the build."false"
pause_imageThis can be used to override the default pause image used to hold the network namespace and IP for the pod."k8s.gcr.io/pause:3.6"
pip_conf_fileThe path to a file to be copied into the image at /etc/pip.conf for use as a global config file. This file will be removed at the end of the build if remove_extra_repos is true.""
containerd_additional_settingsThis is a string, base64 encoded, that contains additional configuration for containerd. It must be version 2 and not contain the pause image configuration block. See image-builder/images/capi/ansible/roles/containerd/templates/etc/containerd/config.toml for the template.null
load_additional_componentsIf set to "true", the load_additional_components role will be executed. This needs to be set to "true" if any of additional_url_images, additional_registry_images or additional_executables are set to "true""false"
additional_url_imagesSet this to "true" to load addtional container images using a tar url. additional_url_images_list var should be set to a comma seperated string of tar urls of the container images."false"
additional_registry_imagesSet this to "true" to load addtional container images using their registry url. additional_registry_images_list var should be set to a comma seperated string of registry urls of the container images."false"
additional_executablesSet this to "true" to load addtional executables from a url. additional_executables_list var should be set to a comma seperated string of urls. additional_executables_destination_path should be set to the destination path of the executables."false"
ansible_user_varsA space delimited string that the user can pass to use in the ansible roles""
containerd_config_fileCustom containerd config file a user can pass to override the default. Use ansible_user_vars to pass this var""

The variables found in packer/config/*.json or packer/<provider>/*.json should not need to be modified directly. For customization it is better to create a JSON file with your changes and provide it via the PACKER_VAR_FILES environment variable. Variables set in this file will override any previous values. Multiple files can be passed via PACKER_VAR_FILES, with the last file taking precedence over any others.

Examples

Passing a single extra var file
PACKER_VAR_FILES=var_file_1.json make ...
Passing multiple extra var files
PACKER_VAR_FILES="var_file_1.json var_file_2.json" make ...
Passing in extra packages to the image

If you wanted to install the RPMs nfs-utils and net-tools, create a file called extra_vars.json and populate with the following:

{
  "extra_rpms": "\"nfs-utils net-tools\""
}

Note that since the extra_rpms variable is a string, and we need the string to be quoted to preserve the space when placed on the command line, having the escaped double-quotes is required.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=extra_vars.json make build-node-ova-local-photon-3
Configuring Containerd at runtime

Containerd default configuration has the following imports value:

imports = ["/etc/containerd/conf.d/*.toml"]

This allows you to place files at runtime in /etc/containerd/conf.d/ that will then be merged with the rest of containerd configuration.

For example to enable containerd metrics, create a file /etc/containerd/conf.d/metrics.toml with the following:

[metrics]
  address = "0.0.0.0:1338"
  grpc_histogram = false
Disabling default repos and using an internal package mirror

A common use-case within enterprise environments is to have a package repository available on an internal network to install from rather than reaching out to the internet. To support this, you can inject custom repository definitions into the image, and optionally disable the use of the default ones.

For example, to build an image using only an internal mirror, create a file called internal_repos.json and populate it with the following:

{
  "disable_public_repos": "true",
  "extra_repos": "/home/<user>/mirror.repo",
  "remove_extra_repos": "true"
}

This example assumes that you have an RPM repository definition available at /home/<user>/mirror.repo, and it is correctly configured to point to your internal mirror. It will be added to the image within /etc/yum.repos.d, with all existing repositories found with /etc/yum.repos.d disabled by setting disable_public_repos to "true". Furthermore, the (optional) use of "remove_extra_repos" means that at the end of the build, the repository definition that was added will be removed. This is useful if the image you are building will be shared externally and you do not wish to include a file with internal network services and addresses.

For Ubuntu images, the process works the same but you would need to add a .list file pointing to your DEB package mirror.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=internal_repos.json make build-node-ova-local-photon-3
Setting up an HTTP Proxy

The Packer tool itself honors the standard env vars of HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. If these variables are set and exported, they will be honored if Packer needs to download an ISO during a build. However, in order to use these proxies with Ansible (for use during package installation or binary download), we need to pass them via JSON file.

For example, to set the HTTP_PROXY env var for the Ansible stage of the build, create a proxy.json and populate it with the following:

{
  "http_proxy": "http://proxy.corp.com"
}

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=proxy.json make build-node-ova-local-photon-3
Loading addtional components using additional_components.json
{
  "additional_executables": "true",
  "additional_executables_destination_path": "/path/to/dest",
  "additional_executables_list": "http://path/to/exec1,http://path/to/exec2",
  "additional_registry_images": "true",
  "additional_registry_images_list": "plndr/kube-vip:0.3.4,plndr/kube-vip:0.3.3",
  "additional_url_images": "true",
  "additional_url_images_list": "http://path/to/image1.tar,http://path/to/image2.tar",
  "load_additional_components": "true"
}
Using ansible_user_vars to pass custom variables
{
  "ansible_user_vars": "var1=value1 var2={{ user `myvar2`}}",
  "myvar2": "value2"
}
Enabling Ansible custom roles

Put the Ansible role files in the ansible/roles directory.

{
  "firstboot_custom_roles_pre": "setupRole",
  "node_custom_roles_post": "role1 role2"
}

Note, for backwards compatibility reasons, the variable custom_role_names is still accepted as an alternative to node_custom_roles_post, and they are functionally equivalent.