Kubernetes Image Builder

This is the official documentation for the Kubernetes Image Builder.

Introduction

The Kubernetes Image Builder is a SIG Cluster Lifecycle sponsored project with the purpose to consolidate several existing projects for building node images to run conformant Kubernetes clusters. Over time, the Image Builder will present a consistent, unified tool to create images across cloud and infrastructure providers, along with all the needed hooks for customization to meet business needs.

Summary

Tutorials

CAPI Images

The Image Builder can be used to build images intended for use with Kubernetes CAPI providers. Each provider has its own format of images that it can work with. For example, AWS instances use AMIs, and vSphere uses OVAs.

Prerequisites

Packer and Ansible are used for building these images. This tooling has been forked and extended from the Wardroom project.

• Packer version >= 1.6.0
• Goss plugin for Packer version >= 1.2.0
• Ansible version >= 2.10.0

If any needed binaries are not present, they can be installed to images/capi/.bin with the make deps command. This directory will need to be added to your $PATH.

Providers

• AWS
• Azure
• CloudStack
• DigitalOcean
• GCP
• OCI
• OpenStack
• Raw
• VirtualBox
• vSphere

Make targets

Within this repo, there is a Makefile located at images/capi/Makefile that can be used to create the default images.

Run make or make help to see the current list of targets. The targets are categorized into Dependencies, Builds, and Cleaning. The Dependency targets will check that your system has the proper tools installed to run the build for your specific provider. If the dependencies are not present, they will be installed.

Configuration

The images/capi/packer/config directory includes several JSON files that define the default configuration for the images:

Due to OS differences, Windows images has additional configuration in the packer/config/windows folder. See Windows documentation for more details.

Customization

Several variables can be used to customize the image build.

The variables found in packer/config/*.json or packer/<provider>/*.json should not need to be modified directly. For customization it is better to create a JSON file with your changes and provide it via the PACKER_VAR_FILES environment variable. Variables set in this file will override any previous values. Multiple files can be passed via PACKER_VAR_FILES, with the last file taking precedence over any others.

Examples

Passing a single extra var file

PACKER_VAR_FILES=var_file_1.json make ...

Passing multiple extra var files

PACKER_VAR_FILES="var_file_1.json var_file_2.json" make ...

Passing in extra packages to the image

If you wanted to install the RPMs nfs-utils and net-tools, create a file called extra_vars.json and populate with the following:

{
  "extra_rpms": "\"nfs-utils net-tools\""
}

Note that since the extra_rpms variable is a string, and we need the string to be quoted to preserve the space when placed on the command line, having the escaped double-quotes is required.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=extra_vars.json make build-node-ova-local-photon-3

Configuring Containerd at runtime

Containerd default configuration has the following imports value:

imports = ["/etc/containerd/conf.d/*.toml"]

This allows you to place files at runtime in /etc/containerd/conf.d/ that will then be merged with the rest of containerd configuration.

For example to enable containerd metrics, create a file /etc/containerd/conf.d/metrics.toml with the following:

[metrics]
  address = "0.0.0.0:1338"
  grpc_histogram = false

Disabling default repos and using an internal package mirror

A common use-case within enterprise environments is to have a package repository available on an internal network to install from rather than reaching out to the internet. To support this, you can inject custom repository definitions into the image, and optionally disable the use of the default ones.

For example, to build an image using only an internal mirror, create a file called internal_repos.json and populate it with the following:

{
  "disable_public_repos": "true",
  "extra_repos": "/home/<user>/mirror.repo",
  "remove_extra_repos": "true"
}

This example assumes that you have an RPM repository definition available at /home/<user>/mirror.repo, and it is correctly configured to point to your internal mirror. It will be added to the image within /etc/yum.repos.d, with all existing repositories found with /etc/yum.repos.d disabled by setting disable_public_repos to "true". Furthermore, the (optional) use of "remove_extra_repos" means that at the end of the build, the repository definition that was added will be removed. This is useful if the image you are building will be shared externally and you do not wish to include a file with internal network services and addresses.

For Ubuntu images, the process works the same but you would need to add a .list file pointing to your DEB package mirror.

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=internal_repos.json make build-node-ova-local-photon-3

Setting up an HTTP Proxy

The Packer tool itself honors the standard env vars of HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. If these variables are set and exported, they will be honored if Packer needs to download an ISO during a build. However, in order to use these proxies with Ansible (for use during package installation or binary download), we need to pass them via JSON file.

For example, to set the HTTP_PROXY env var for the Ansible stage of the build, create a proxy.json and populate it with the following:

{
  "http_proxy": "http://proxy.corp.com"
}

Then, execute the build (using a Photon OVA as an example) with the following:

PACKER_VAR_FILES=proxy.json make build-node-ova-local-photon-3

Loading addtional components using additional_components.json

{
  "additional_executables": "true",
  "additional_executables_destination_path": "/path/to/dest",
  "additional_executables_list": "http://path/to/exec1,http://path/to/exec2",
  "additional_registry_images": "true",
  "additional_registry_images_list": "plndr/kube-vip:0.3.4,plndr/kube-vip:0.3.3",
  "additional_url_images": "true",
  "additional_url_images_list": "http://path/to/image1.tar,http://path/to/image2.tar",
  "load_additional_components": "true"
}

Using ansible_user_vars to pass custom variables

{
  "ansible_user_vars": "var1=value1 var2={{ user `myvar2`}}",
  "myvar2": "value2"
}

Enabling custom_role on Ansible

Put the Ansible role files in the ansible/roles directory.

{
  "custom_role": "true",
  "custom_role_names": "role1,role2,role3"
}

Quick Start

In this tutorial we will cover the basics of how to download and execute the Image Builder.

Installation

As a set of scripts and Makefiles that rely on Packer and Ansible, there is image builder binary/application to install. Rather we need to download the tooling from the GitHub repo and make sure that the Packer and Ansible are installed.

To get the latest image-builder source on your machine, choose one of the following methods:

Tarball download:

curl -L https://github.com/kubernetes-sigs/image-builder/tarball/master -o image-builder.tgz
mkdir image-builder
tar xzf image-builder.tgz --strip-components 1 -C image-builder
rm image-builder.tgz
cd image-builder/images/capi

Or, if you’d like to keep tracking the repo updates (requires git):

git clone git@github.com:kubernetes-sigs/image-builder.git
cd image-builder/images/capi

Dependencies

Once you are within the capi directory, you can execute make or make help to see all the possible make targets. Before we can build an image, we need to make sure that Packer and Ansible are installed on your system. You may already have them, Mac users may have them installed via brew, or you may have downloaded them directly.

If you want the image-builder to install these tools for you, they can be installed by executing make deps. This will install dependencies into image-builder/images/capi/.bin if they are not already on your system. make deps will first check if Ansible and Packer are available and if they are, will use the existing installations.

Looking at the output from make deps, if Ansible or Packer were installed into the .bin directory, you’ll need to add that to your PATH environment variable before they can be used. Assuming you are still in images/capi, you can do that with the following:

export PATH=$PWD/.bin:$PATH

Builds

With the CAPI image builder installed and dependencies satisfied, you are now ready to build an image. In general, this is done via make targets, and each provider (e.g. AWS, GCE, etc.) will have different requirements for what information needs to be provided (such as cloud provider authentication credentials). Certain providers may have dependencies that are not satisfied by make deps, for example the vSphere provider needs access to a hypervisor (VMware Fusion on macOS, VMware Workstation on Linux). See the specific documentation for your desired provider for more details.

Building Images for AWS

Prerequisites for Amazon Web Services

• An AWS account
• The AWS CLI installed and configured

Building Images

The build prerequisites for using image-builder for building AMIs are managed by running:

make deps-ami

From the images/capi directory, run make build-ami-<OS>, where <OS> is the desired operating system. The available choices are listed via make help.

To build all available OS’s, uses the -all target. If you want to build them in parallel, use make -j. For example, make -j build-ami-all.

In the output of a successful make command is a list of created AMIs. To format them you can copy the output and pipe it through this to get a desired table:

echo 'us-fake-1: ami-123
us-fake-2: ami-234' | column -t | sed 's/^/| /g' | sed 's/: //g' | sed 's/ami/| ami/g' | sed 's/$/ |/g'
| us-fake-1 | ami-123 |
| us-fake-2 | ami-234 |

Note: If making the images public (the default), you must use one of the Public CentOS images as a base rather than a Marketplace image.

Configuration

In addition to the configuration found in images/capi/packer/config, the ami directory includes several JSON files that define the default configuration for the different operating systems.

Common AWS options

This table lists several common options that a user may want to set via PACKER_VAR_FILES to customize their build behavior. This is not an exhaustive list, and greater explanation can be found in the Packer documentation for the Amazon AMI builder.

In the below examples, the parameters can be set via variable file and the use of PACKER_VAR_FILES. See Customization for examples.

Examples

Building private AMIs

Set ami_groups="" and snapshot_groups="" parameters to ensure you end up with a private AMI. Both parameters default to "all".

Encrypted AMIs

Set encrypted=true for encrypted AMIs to allow for use with EC2 instances backed by encrypted root volumes. You must also set ami_groups="" and snapshot_groups="" for this to work.

Sharing private AMIs with other AWS accounts

Set ami_users="012345789012,0123456789013" to make your AMI visible to a select number of other AWS accounts, and snapshot_users="012345789012,0123456789013" to allow the EBS snapshot backing the AMI to be copied.

If you are using encrypted root volumes in multiple accounts, you will want to build one unencrypted AMI in a root account, setting snapshot_users, and then use your own methods to copy the snapshot with encryption turned on into other accounts.

Limiting AMI Regions

By default images are copied to many of the available AWS regions. See ami_regions in AWS options for the default list. The list of all available regions can be obtained running:

aws ec2 describe-regions --query "Regions[].{Name:RegionName}" --output text | paste -sd "," -

To limit the regions, provide the ami_regions variable as a comma-delimited list of AWS regions.

For example, to build all images in us-east-1 and copy only to us-west-2 set ami_regions="us-west-2".

Required Permissions to Build the AWS AMIs

The Packer documentation for the Amazon AMI builder supplies a suggested set of minimum permissions.

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action" : [
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CopyImage",
        "ec2:CreateImage",
        "ec2:CreateKeypair",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteKeyPair",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSnapshot",
        "ec2:DeleteVolume",
        "ec2:DeregisterImage",
        "ec2:DescribeImageAttribute",
        "ec2:DescribeImages",
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume",
        "ec2:GetPasswordData",
        "ec2:ModifyImageAttribute",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifySnapshotAttribute",
        "ec2:RegisterImage",
        "ec2:RunInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource" : "*"
  }]
}

Testing Images

Connect remotely to an instance created from the image and run the Node Conformance tests using the following commands:

Initialize a CNI

As root:

(copied from containernetworking/cni)

mkdir -p /etc/cni/net.d
wget -q https://github.com/containernetworking/cni/releases/download/v0.7.0/cni-amd64-v0.7.0.tgz
tar -xzf cni-amd64-v0.7.0.tgz --directory /etc/cni/net.d
cat >/etc/cni/net.d/10-mynet.conf <<EOF
{
    "cniVersion": "0.2.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.22.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
    "cniVersion": "0.2.0",
    "name": "lo",
    "type": "loopback"
}
EOF

Run the e2e node conformance tests

As a non-root user:

wget https://dl.k8s.io/$(< /etc/kubernetes_community_ami_version)/kubernetes-test.tar.gz
tar -zxvf kubernetes-test.tar.gz kubernetes/platforms/linux/amd64
cd kubernetes/platforms/linux/amd64
sudo ./ginkgo --nodes=8 --flakeAttempts=2 --focus="\[Conformance\]" --skip="\[Flaky\]|\[Serial\]|\[sig-network\]|Container Lifecycle Hook" ./e2e_node.test -- --k8s-bin-dir=/usr/bin --container-runtime=remote --container-runtime-endpoint unix:///var/run/containerd/containerd.sock --container-runtime-process-name /usr/local/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd\"]}"

Building Images for Azure

These images are designed for use with Cluster API Provider Azure (CAPZ). Learn more on using custom images with CAPZ.

Prerequisites for Azure

• An Azure account
• The Azure CLI installed and configured
• Set environment variables for AZURE_SUBSCRIPTION_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET
• Set optional environment variables RESOURCE_GROUP_NAME, STORAGE_ACCOUNT_NAME & AZURE_LOCATION to override the default values

Building Images

The build prerequisites for using image-builder for building Azure images are managed by running:

make deps-azure

Building Managed Images in Shared Image Galleries

From the images/capi directory, run make build-azure-sig-ubuntu-1804

Building VHDs

From the images/capi directory, run make build-azure-vhd-ubuntu-1804

If building the windows images from a Mac there is a known issue with connectivity. Please see details on running MacOS with ansible.

Hyper-V Generation 2 VHDs

Most of the images built from the images/capi directory for Azure will be Hyper-V Generation 1 images. There are also a few available configurations to build Generation 2 VMs. The naming pattern is identical to Generation 1 images, with -gen2 appended to the end of the image name. For example:

# Generation 1 image
make build-azure-sig-ubuntu-1804

# Generation 2 image
make build-azure-sig-ubuntu-1804-gen2

Generation 2 images may only be used with Shared Image Gallery, not VHD.

Configuration

Common Azure options

This table lists several common options that a user may want to set via PACKER_VAR_FILES to customize their build behavior. This is not an exhaustive list, and greater explanation can be found in the Packer documentation for the Azure ARM builder.

Developer

If you are adding features to image builder than it is sometimes useful to work with the images directly. This section gives some tips.

Provision a VM directly from a VHD

After creating a VHD, create a managed image using the url output from make build-azure-vhd-<image> and use it to create the VM:

az image create -n testvmimage -g cluster-api-images --os-type <Windows/Linux> --source <storage url for vhd file>
az vm create -n testvm --image testvmimage -g cluster-api-images

Debugging packer scripts

There are several ways to debug packer scripts: https://www.packer.io/docs/other/debugging.html

Building Images for CloudStack

Hypervisor

The image is built using KVM hypervisor as a qcow2 image. Following which, it can be converted into ova for VMware and vhd for XenServer.

Prerequisites for building images

Images can only be built on Linux Machines, and has been tested on Ubuntu 18.04 LTS. Execute the following command to install qemu-kvm and other packages if you are running Ubuntu 18.04 LTS.

Installing packages to use qemu-img

$ sudo -i
# apt install qemu-kvm libvirt-bin qemu-utils

If you’re on Ubuntu 20.04 LTS, then execute the following command to install qemu-kvm packages.

$ sudo -i
# apt install qemu-kvm libvirt-daemon-system libvirt-clients virtinst cpu-checker libguestfs-tools libosinfo-bin

Adding your user to the kvm group

$ sudo usermod -a -G kvm <yourusername>
$ sudo chown root:kvm /dev/kvm

Then exit and log back in to make the change take place.

Building Images

The build prerequisites for using image-builder for building cloudstack images are managed by running:

$ cd image-builder/images/capi
$ make deps-qemu

KVM Hypervisor

From the images/capi directory, run make build-qemu-xxxx-yyyy. The image is built and located in images/capi/output/BUILD_NAME+kube-KUBERNETES_VERSION. Please replace xxxx with the OS distribution and yyyy with the OS version depending on WHAT you want to build the image for.

For building a ubuntu-2004 based capi image, run the following commands -

$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi/
$ cat > extra_vars.json <<EOF
{
  "ansible_user_vars": "provider=cloudstack"
}
EOF
$ PACKER_VAR_FILES=extra_vars.json make clean build-qemu-ubuntu-2004

XenServer Hypervisor

Run the following script to ensure the required dependencies are met :

$ ./hack/ensure-vhdutil.sh

Follow the preceding steps to build the qcow2 capi template for KVM. It will display the location of the template to the terminal as shown :

$ make build-qemu-ubuntu-2004
.............................
Builds finished. The artifacts of successful builds are:
qemu: VM files in directory: ./output/ubuntu-2004-kube-v1.21.10

Here the build-name is ubuntu-2004-kube-v1.21.10

One completed, run the following commands to convert the template to a XenServer compatible template

$ ./hack/convert-cloudstack-image.sh ./output/<build-name>/<build-name> x

Creating XenServer Export for ubuntu-2004-kube-v1.21.10
NOTE: For better performance, we will do the overwritten convert!
Done! Convert to ubuntu-2004-kube-v1.21.10.vhd.
Back up source to ubuntu-2004-kube-v1.21.10.vhd.bak.
Converting to ubuntu-2004-kube-v1.21.10-xen.vhd.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>Done!
Created .vhd file, now zipping
ubuntu-2004-kube-v1.21.10 exported for XenServer: ubuntu-2004-kube-v1.21.10-xen.vhd.bz2

VMware Hypervisor

Run the following script to ensure the required dependencies are met :

$ ./hack/ensure-ovftool.sh

Follow the preceding steps to build the qcow2 capi template for KVM. It will display the location of the template to the terminal as shown :

$ make build-qemu-ubuntu-2004
.............................
Builds finished. The artifacts of successful builds are:
qemu: VM files in directory: ./output/ubuntu-2004-kube-v1.21.10

Here the build-name is ubuntu-2004-kube-v1.21.10

One completed, run the following commands to convert the template to a VMware compatible template

$ ./hack/convert-cloudstack-image.sh ./output/<build-name>/<build-name> v

Creating VMware Export for ubuntu-2004-kube-v1.21.10
/usr/bin/ovftool: line 10: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
Opening VMX source: ubuntu-2004-kube-v1.21.10-vmware.vmx
Opening OVA target: ubuntu-2004-kube-v1.21.10-vmware.ova
Writing OVA package: ubuntu-2004-kube-v1.21.10-vmware.ova
Transfer Completed
Completed successfully

Prebuilt Images

For convenience, prebuilt images can be found here

Building Images for DigitalOcean

Prerequisites for DigitalOcean

• A DigitalOcean account
• The DigitalOcean CLI (doctl) installed and configured
• Set environment variables for DIGITALOCEAN_ACCESS_TOKEN,

Building Images

The build prerequisites for using image-builder for building Digital Ocean images are managed by running:

make deps-do

From the images/capi directory, run make build-do-<OS> where <OS> is the desired operating system. The available choices are listed via make help.

Configuration

In addition to the configuration found in images/capi/packer/config, the digitalocean directory includes several JSON files that define the default configuration for the different operating systems.

Building CAPI Images for Google Cloud Platform (GCP)

Prerequisites

Create Service Account

From your google cloud console, follow these instructions to create a new service account with Editor permissions. Thereafter, generate a JSON Key and store it somewhere safe.

Use cloud shell to install ansible, packer and proceed with building the CAPI compliant vm image.

Install Ansible and Packer

Start by launching the google cloud shell.

# Export the GCP project id you want to build images in
$ export GCP_PROJECT_ID=<project-id>

# Export the path to the service account credentials created in the step above
$ export GOOGLE_APPLICATION_CREDENTIALS=</path/to/serviceaccount-key.json>

# If you dont have the image-builder repository
$ git clone https://github.com/kubernetes-sigs/image-builder.git

$ cd image-builder/images/capi/
# Run the target make deps-gce to install ansible and packer
$ make deps-gce

Run the Make target to generate GCE images.

From images/capi directory, run make build-gce-ubuntu-<version> command depending on which ubuntu version you want to build the image for.

For instance, to build an image for ubuntu 18-04, run

$ make build-gce-ubuntu-1804

To build all gce ubuntu images, run

make build-gce-all

Configuration

The gce sub-directory inside images/capi/packer stores JSON configuration files for Ubuntu OS.

List Images

List all images by running the following command in the console

$ gcloud compute images list --project ${GCP_PROJECT_ID} --no-standard-images

NAME                                         PROJECT            FAMILY                      DEPRECATED  STATUS
cluster-api-ubuntu-1804-v1-17-11-1603233313  myregistry-292303  capi-ubuntu-1804-k8s-v1-17              READY
cluster-api-ubuntu-2004-v1-17-11-1603233874  myregistry-292303  capi-ubuntu-2004-k8s-v1-17              READY

Delete Images

To delete images from gcloud shell, run following

$ gcloud compute images delete [image 1] [image2]

where [image 1] and [image 2] refer to the names of the images to be deleted.

Building Images for OpenStack

Hypervisor

The image is built using KVM hypervisor.

Prerequisites for QCOW2

Execute the following command to install qemu-kvm and other packages if you are running Ubuntu 18.04 LTS.

Installing packages to use qemu-img

$ sudo -i
# apt install qemu-kvm libvirt-bin qemu-utils

If you’re on Ubuntu 20.04 LTS, then execute the following command to install qemu-kvm packages.

$ sudo -i
# apt install qemu-kvm libvirt-daemon-system libvirt-clients virtinst cpu-checker libguestfs-tools libosinfo-bin

Adding your user to the kvm group

$ sudo usermod -a -G kvm <yourusername>
$ sudo chown root:kvm /dev/kvm

Then exit and log back in to make the change take place.

Building Images

The build prerequisites for using image-builder for building qemu images are managed by running:

cd image-builder/images/capi
make deps-qemu

Building QCOW2 Image

From the images/capi directory, run make build-qemu-ubuntu-xxxx. The image is built and located in images/capi/output/BUILD_NAME+kube-KUBERNETES_VERSION. Please replace xxxx with 1804 or 2004 depending on the version you want to build the image for.

For building a ubuntu-2004 based capi image, run the following commands -

$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi/
$ make build-qemu-ubuntu-2004

Building Images for Oracle Cloud Infrastructure (OCI)

Prerequisites

• An OCI account
• The OCI plugin for Packer supports three options for authentication . You may use any of these options when building the Cluster API images.

Building Images

The build prerequisites for using image-builder for building OCI images are managed by running the following command from images/capi directory.

make deps-oci

From the images/capi directory, run make build-do-<OS> where <OS> is the desired operating system. The available choices are listed via make help.

Configuration

In addition to the configuration found in images/capi/packer/config, the oci directory includes several JSON files that define the default configuration for the different operating systems.

Common options

This table lists several common options that a user must set via PACKER_VAR_FILES to customize their build behavior. This is not an exhaustive list, and greater explanation can be found in the Packer documentation for the OCI builder.

Steps to create Packer VAR file

Create a file with the following contents and name it as oci.json

{
  "compartment_ocid": "Fill compartment OCID here",
  "subnet_ocid": "Fill Subnet OCID here",
  "availability_domain": "Fill Availability Domain here"
}

Example make command with Packer VAR file

PACKER_VAR_FILES=oci.json make build-oci-oracle-linux-8

Build an Arm based image

Building an Arm based image requires some overrides to use the correct installation files . An example for an oci.json file for Arm is shown below. The parameters for containerd, crictl and kubernetes has to point to the corresponding URL for Arm. The containerd SHA can be changed appropriately, the containerd version is defined in images/capi/packer/config/containerd.json.

{
  "compartment_ocid": "Fill compartment OCID here",
  "subnet_ocid": "Fill Subnet OCID here",
  "availability_domain": "Fill Availability Domain here",
  "shape": "VM.Standard.A1.Flex",
  "containerd_url": "https://github.com/containerd/containerd/releases/download/v{{user `containerd_version`}}/cri-containerd-cni-{{user `containerd_version`}}-linux-arm64.tar.gz",
  "crictl_url": "https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{user `crictl_version`}}/crictl-v{{user `crictl_version`}}-linux-arm64.tar.gz",
  "kubernetes_rpm_repo": "https://packages.cloud.google.com/yum/repos/kubernetes-el7-aarch64",
  "containerd_sha256": "9ac616b5f23c1d10353bd45b26cb736efa75dfef31a2113baff2435dbc7becb8"
}

Building Raw Images (Baremetal)

Hypervisor

The image is built using KVM hypervisor.

Prerequisites for QCOW2

Execute the following command to install qemu-kvm and other packages if you are running Ubuntu 18.04 LTS.

Installing packages to use qemu-img

$ sudo -i
# apt install qemu-kvm libvirt-bin qemu-utils

If you’re on Ubuntu 20.04 LTS, then execute the following command to install qemu-kvm packages.

$ sudo -i
# apt install qemu-kvm libvirt-daemon-system libvirt-clients virtinst cpu-checker libguestfs-tools libosinfo-bin

Adding your user to the kvm group

$ sudo usermod -a -G kvm <yourusername>
$ sudo chown root:kvm /dev/kvm

Then exit and log back in to make the change take place.

Building Images

The build prerequisites for using image-builder for building raw images are managed by running:

cd image-builder/images/capi
make deps-raw

Building QCOW2 Image

From the images/capi directory, run make build-raw-ubuntu-xxxx. The image is built and located in images/capi/output/BUILD_NAME+kube-KUBERNETES_VERSION. Please replace xxxx with 1804 or 2004 depending on the version you want to build the image for.

For building a ubuntu-2004 based capi image, run the following commands -

$ git clone https://github.com/kubernetes-sigs/image-builder.git
$ cd image-builder/images/capi/
$ make build-qemu-ubuntu-2004

Building Images for vSphere

Hypervisor

The images may be built using one of the following hypervisors:

NOTE If you want to build all available OS’s, uses the -all target. If you want to build them in parallel, use make -j. For example, make -j build-node-ova-local-all.

The vsphere builder supports building against a remote VMware vSphere using standard API.

vmware-vmx builder

During the dev process it’s uncommon for the base OS image to change, but the image building process builds the base image from the ISO every time and thus adding a significant amount of time to the build process.

To reduce the image building times during development, one can use the build-node-ova-local-base-<OS> target to build the base image from the ISO. By setting source_path variable in vmx.json to the *.vmx file from the output, it can then be re-used with the build-node-ova-local-vmx-<OS> build target to speed up the process.

vsphere-clone builder

vsphere-base builder allows you to build one time base OVAs from iso images using the kickstart process. It leaves the user builder intact in base OVA to be used by clone builder later. vSphere-clone builder builds on top of base OVA by cloning it and ansiblizing it. This saves time by allowing repeated iteration on base OVA without installing OS from scratch again and again. Also, it uses link cloning and create_snapshot feature to clone faster.

Prerequisites for vSphere builder

Complete the vsphere.json configuration file with credentials and informations specific to the remote vSphere hypervisor used to build the ova file. This file must have the following format (cluster can be replace by host):

{
    "vcenter_server":"FQDN of vcenter",
    "username":"vcenter_username",
    "password":"vcenter_password",
    "datastore":"template_datastore",
    "folder": "template_folder_on_vcenter",
    "cluster": "esxi_cluster_used_for_template_creation",
    "network": "network_attached_to_template",
    "insecure_connection": "false",
    "template": "base_template_used_by_clone_builder",
    "create_snbapshot": "creates a snaphot on base OVA after building",
    "linked_clone": "Uses link cloning in vsphere-clone builder: true, by default"
}

If you prefer to use a different configuration file, you can create it with the same format and export PACKER_VAR_FILES environment variable containing the full path to it.

Building Images

The build prerequisites for using image-builder for building OVAs are managed by running:

make deps-ova

From the images/capi directory, run make build-node-ova-<hypervisor>-<OS>, where <hypervisor> is your target hypervisor (local or vsphere) and <OS> is the desired operating system. The available choices are listed via make help.

OVA Creation

When the final OVA is created, there are two methods that can be used for creation. By default, an OVF file is created, the manifest is created using SHA256 sums of the OVF and VMDK, and then tar is used to create an OVA containing the OVF, VMDK, and the manifest.

Optionally, ovftool can be used to create the OVA. This has the advantage of validating the created OVF, and has greater chances of producing OVAs that are compliant with more versions of VMware targets of Fusion, Workstation, and vSphere. To use ovftool for OVA creation, set the env variable IB_OVFTOOL to any non-empty value. Optionally, args to ovftool can be passed by setting the env var IB_OVFTOOL_ARGS like the following:

IB_OVFTOOL=1 IB_OVFTOOL_ARGS="--allowExtraConfig" make build-node-ova-<hypervisor>-<OS>

Configuration

In addition to the configuration found in images/capi/packer/config, the ova directory includes several JSON files that define the configuration for the images:

Photon specific options

Enabling .local lookups via DNS

Photon uses systemd-resolved defaults, which means that .local will be resolved using Multicast DNS. If you are deploying to an environment where you require DNS resolution .local, then add leak_local_mdns_to_dns=yes in ansible_user_vars.

RHEL

When building the RHEL image, the OS must register itself with the Red Hat Subscription Manager (RHSM). To do this, the current supported method is to supply a username and password via environment variables. The two environment variables are RHSM_USER and RHSM_PASS. Although building RHEL images has been tested via this method, if an error is encountered during the build, the VM is deleted without the machine being unregistered with RHSM. To prevent this, it is recommended to build with the following command:

PACKER_FLAGS=-on-error=ask RHSM_USER=user RHSM_PASS=pass make build-node-ova-<hypervisor>-rhel-7

The addition of PACKER_FLAGS=-on-error=ask means that if an error is encountered, the build will pause, allowing you to SSH into the machine and unregister manually.

Output

The images are built and located in images/capi/output/BUILD_NAME+kube-KUBERNETES_VERSION

Uploading Images

The images are uploaded to the GCS bucket capv-images. The path to the image depends on the version of Kubernetes:

Uploading the images requires the gcloud and gsutil programs, an active Google Cloud account, or a service account with an associated key file. The latter may be specified via the environment variable KEY_FILE.

hack/image-upload.py --key-file KEY_FILE BUILD_DIR

First the images are checksummed (SHA256). If a matching checksum already exists remotely then the image is not re-uploaded. Otherwise the images are uploaded to the GCS bucket.

Listing Available Images

Once uploaded the available images may be listed using the gsutil program, for example:

gsutil ls gs://capv-images/release

Downloading Images

Images may be downloaded via HTTP:

Testing Images

Accessing the Images

Accessing Local VMs

After the images are built, the VMs from they are built are prepped for local testing. Simply boot the VM locally with Fusion or Workstation and the machine will be initialized with cloud-init data from the cloudinit directory. The VMs may be accessed via SSH by using the command hack/image-ssh.sh BUILD_DIR capv.

Accessing Remote VMs

After deploying an image to vSphere, use hack/image-govc-cloudinit.sh VM to snapshot the image and update it with cloud-init data from the cloudinit directory. Start the VM and now it may be accessed with ssh -i cloudinit/id_rsa.capi capv@VM_IP. This hack necessitate the govc utility from VMmare

Initialize a CNI

As root:

(copied from containernetworking/cni)

mkdir -p /etc/cni/net.d
curl -LO https://github.com/containernetworking/plugins/releases/download/v0.7.0/cni-plugins-amd64-v0.7.0.tgz
tar -xzf cni-plugins-amd64-v0.7.0.tgz --directory /etc/cni/net.d
cat >/etc/cni/net.d/10-mynet.conf <<EOF
{
    "cniVersion": "0.2.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.22.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
    "cniVersion": "0.2.0",
    "name": "lo",
    "type": "loopback"
}
EOF

Run the e2e node conformance tests

As a non-root user:

curl -LO https://dl.k8s.io/$(</etc/kubernetes-version)/kubernetes-test-linux-amd64.tar.gz
tar -zxvf kubernetes-test-linux-amd64.tar.gz
cd kubernetes/test/bin
sudo ./ginkgo --nodes=8 --flakeAttempts=2 --focus="\[Conformance\]" --skip="\[Flaky\]|\[Serial\]|\[sig-network\]|Container Lifecycle Hook" ./e2e_node.test -- --k8s-bin-dir=/usr/bin --container-runtime=remote --container-runtime-endpoint unix:///var/run/containerd/containerd.sock --container-runtime-process-name /usr/local/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd\"]}"

The cloudinit Directory

The cloudinit contains files that:

• Are example data used for testing
• Are not included in any of the images
• Should not be used in production systems

For more information about how the files in the cloudinit directory are used, please refer to the section on accessing the images.

Windows

Configuration

The images/capi/packer/config/windows directory includes several JSON files that define the default configuration for the Windows images:

Service Manager

Image-builder provides you two ways to configure windows services. The default is setup using nssm which configures a windows service for kubelet by running {{ kubernetes_install_path }}\StartKubelet.ps1 allowing easy editing of command arguments in the startup file. The alternate is to use the windows native sc.exe which uses the kubelet argument --windows-service to install kubelet as a native windows service with the command line arguments configured on the service. Nssm handles service restarts, if you are using sc.exe you may wish to configure the service restart options on kubelet. To avoid starting kubelet to early, image-builder sets the kubelet service to manual which you should consider changing once the node has joined a cluster.

Important: sc.exe does not support kubeadm KUBELET_KUBEADM_ARGS which is used by Cluster API to pass extra user args

Wins (by rancher)

As a workaround for the lack of privileged containers on windows nodes, SIG Windows utilise Wins to allow pods to run processes on hosts, an example is when you are using kube-proxy or a cni provider in a pod. This means that by default we install wins using image-builder. This may be undesirable, if you do not need pod to have the ability to run processes on the host. Additionally, Alpha support for Windows Host Processes, which is the windows equivalent for privileged containers, will be introduced in 1.22 which negates the need for wins entirely. To skip installing wins on the host you can add "wins_url": "" to your variables.

When using containerd, due to lack of host networking, there is currently no way to run cni inside a pod. To work around this, containerd needs to be installed with an additional custom Ansible module or after the image has been created. With CAPI, the addtional configuration can be done with pre/postKubeadm scripts when the node is created.

Windows Updates

When building Windows images it is necessary to install OS and Security updates. Image Builder provides two variables to allow choosing which updates get installed which can be used together or seperately (with individual KBs installed first).

To specify the update categories to check, provide a value for windows_updates_categories in packer/config/windows/common.json.

Example: Install all available updates from all categories.
"windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups"

Published Cloud Provider images such as Azure or AWS are regularly updated so it may be preferable to specify individual patches to install. This can be achieved by specifying the KB numbers of required updates.

To choose individual updates, provide a value for windows_updates_kbs in packer/config/windows/common.json.

Example: "windows_updates_kbs": "KB4580390 KB4471332".

OpenSSH Server

If a connection to the Microsoft Updates server is not possible, you may use the Win32 port of OpenSSH located on GitHub. To do this, you can set the ssh_source_url to the location of the desired OpenSSH Version from https://github.com/PowerShell/Win32-OpenSSH/releases/

Example: "ssh_source_url": "https://github.com/PowerShell/Win32-OpenSSH/releases/download/V8.6.0.0p1-Beta/OpenSSH-Win64.zip"

Using the Ansible Scripts directly

Ansible doesn’t run on directly on Windows (wsl works) but can used to configure a remote Windows host. For faster development you can create a VM and run Ansible against the Windows VM directly with out using packer. This document gives the high level steps to use Ansible from Linux machine.

Set up Windows machine

Follow the documentation for configuring WinRm on the Windows machine: https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html#winrm-setup. Note the ConfigureRemotingForAnsible.ps1 is for development only. Refer to Ansible WinRM documentation for details for advance configuration.

After WinRM is installed you can edit or /etc/ansible/hosts file with the following:

[winhost]    
<windows ip>

[winhost:vars]
ansible_user=username
ansible_password=<your password>
ansible_connection=winrm
ansible_winrm_server_cert_validation=ignore

Then run: ansible-playbook -vvv node_windows.yml --extra-vars "@example.vars.yml

MacOS with ansible

The Winrm connection plugin for Ansible on MacOS causes connection issues which can result in ERROR! A worker was found in a dead state. See https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html#what-is-winrm for more details.

To fix the issue on MacOS is to set the no_proxy environment variable. Example:

'no_proxy=* make build-azure-vhd-windows-2019'

Building Images for VirtualBox

Hypervisor

The image is built using Oracle VM VirtualBox hypervisor.

Installing VirtualBox package

Oracle VirtualBox install instructions and packages are available at the official page.

Building Images

Validating

The build prerequisites for using image-builder for building vbox images are managed by running:

cd image-builder/images/capi
make deps-vbox

Generating a VirtualBox image

Only Windows 2019 images are available for VirtualBox hypervisor for now, to build local images for development are made by running:

cd image-builder/images/capi
make build-node-vbox-local-windows-2019

Windows ISO download

This field should point to an Windows evaluation ISO, it can be found at Microsoft Evaluation Center page. Make sure you have the correct Windows Server version.

{
  "os_iso_url": "file:/path/en_windows_server_2019_x64_dvd_4cb967d8.iso"
}

Testing CAPI Images

GOSS

Goss is a YAML based serverspec alternative tool for validating a server’s configuration. It is used in conjunction with packer-provisioner-goss to test if the images have all requisite components to work with cluster API.

Support Matrix

*For stock server-specs shipped with repo

Prerequisites for Running GOSS

GOSS runs as a part of image building through a packer provisioner. Supported arguments are passed through file: packer/config/goss-args.json

{
  "goss_arch": "amd64",
  "goss_download_path": "",
  "goss_entry_file": "goss/goss.yaml",
  "goss_format": "json",
  "goss_inspect_mode": "true",
  "goss_remote_folder": "",
  "goss_remote_path": "",
  "goss_skip_install": "",
  "goss_tests_dir": "packer/goss",
  "goss_url": "",
  "goss_format_options": "pretty",
  "goss_vars_file": "packer/goss/goss-vars.yaml",
  "goss_version": "0.3.16"
}

Supported values for some of the arguments can be found here.

Enabling the goss_inspect_mode lets you build image even if goss tests fail.

Manually setup GOSS

• Start a VM from capi image
• Copy complete goss dir packer/goss to remote machine
• Download and setup GOSS (use version from goss-args) on the remote machine. Instructions
• Custom goss version can be installed if testing custom server-specs supported by higher version of GOSS.
• All the variables used in GOSS are declared in packer/goss/goss-vars.yaml
• Add more custom serverspec to corresponding GOSS files. Like, goss-command.yaml or goss-kernel-params.yaml

    some_cli --version:
      exit-status: 0
      stdout: [{{ .Vars.some_cli_version }}]
      stderr: []
      timeout: 0
  

• Add more custom variables to corresponding GOSS file goss-vars.yaml.

  some_cli_version: "1.4.5+k8s-1"
  

• Fill the variable values in goss-vars.yaml or specify in --vars-inline while executing GOSS in below steps
• Render the goss template to fix any problems with parsing variable and serverspec yamls

    sudo goss -g goss/goss.yaml --vars /tmp/goss/goss-vars.yaml --vars-inline '{"ARCH":"amd64","OS":"Ubuntu","PROVIDER":"aws", some_cli_version":"1.3.4"}' render
  

• Run the GOSS tests

    sudo goss -g goss/goss.yaml --vars /tmp/goss/goss-vars.yaml --vars-inline '{"ARCH":"amd64","OS":"Ubuntu","PROVIDER":"aws", some_cli_version":"1.3.4"}' validate --retry-timeout 0s --sleep 1s -f json -o pretty
  

Using a container image to build a custom image

This image building approach eliminates the need to manually install and maintain pre-requisite packages like Ansible, Packer, libraries etc. It requires only Docker installed on your machine. All dependencies are handled in Docker while building the container image. This stable container image can be used and reused as a basis for building your own custom images.

Image builder uses GCR to store promoted images in a central registry. Latest container images can be found here - Staging and GA

Building a Container Image

Run the docker build target of Makefile

make docker-build

Using a Container Image

The latest container image can be pulled from GCR as below -

docker pull gcr.io/k8s-staging-scl-image-builder/cluster-node-image-builder-amd64:v0.1.11

Examples

• AMI

  • If the AWS-CLI is already installed on your machine, you can simply mount the ~/.aws folder that stores all the required credentials.

  docker run -it --rm -v /Users/<user>/.aws:/home/imagebuilder/.aws k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11 build-ami-ubuntu-2004
  

  • Another alternative is to use an aws-creds.env file to load the credentials and pass it during docker run.

    AWS_ACCESS_KEY_ID=xxxxxxx
    AWS_SECRET_ACCESS_KEY=xxxxxxxx
    AWS_DEFAULT_REGION=xxxxxx
    

      docker run -it --rm --env-file aws-creds.env k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11 build-ami-ubuntu-2004
  

• AZURE

  • You’ll need an az-creds.env file to load environment variables AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET

    AZURE_SUBSCRIPTION_ID=xxxxxxx
    AZURE_TENANT_ID=xxxxxxx
    AZURE_CLIENT_ID=xxxxxxxx
    AZURE_CLIENT_SECRET=xxxxxx
    

  docker run -it --rm --env-file az-creds.env k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11 build-azure-sig-ubuntu-2004
  

• vSphere OVA

  • vsphere.json configuration file with user and hypervisor credentials. A template of this file can be found here

  • Docker’s --net=host option to ensure http server starts with the host IP and not the docker container IP. This option is Linux specific and thus implies that it can be run only from a Linux machine.

  docker run -it --rm --net=host --env PACKER_VAR_FILES=/home/imagebuilder/vsphere.json -v <complete path of vsphere.json>:/home/imagebuilder/vsphere.json k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.11 build-node-ova-vsphere-ubuntu-2004
  

In addition to this, further customizations can be done as discussed here.

Table of Contents

A | C | E | G | K | O | V

A

AWS

Amazon Web Services

AMI

Amazon Machine Image

C

CAPA

Cluster API Provider AWS

CAPG

Cluster API Provider GCP

CAPI

The Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management. It provides optional, additive functionality on top of core Kubernetes.

source

CAPV

Cluster API Provider vSphere

CAPZ

Cluster API Prover Azure

E

ESXi

ESXi (formerly ESX) is an enterprise-class, type-1 hypervisor developed by VMware. ESXi provides strong separation between VMs and itself, providing strong security boundaries between the guest and host operating systems. ESXi can be used as a standalone entity, without vCenter but this is extremely uncommon and feature limited as without a higher level manager (vCenter). ESXi cannot provide its most valuable features, like High Availability, vMotion, workload balancing and vSAN (a software defined storage stack).

G

GOSS

Goss is a YAML based serverspec alternative tool for validating a server’s configuration. It is used in conjunction with packer-provisioner-goss to test if the images have all requisite components to work with cluster API.

K

K8s

Kubernetes

Kubernetes

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

source

O

OVA

Open Virtual Appliance

A single package containing a pre-configured virtual machine, usually based on OVF.

OVF

Open Virtualization Format

An open standard for packaging and distributing virtual appliances or, more generally, software to be run in virtual machines.

source

V

vCenter

vCenter can be thought of as the management layer for ESXi hosts. Hosts can be arranged into Datacenters, Clusters or resources pools, vCenter is the centralized monitoring and management control plane for ESXi hosts allow centralized management, integration points for other products in the VMware SDDC stack and third party solutions, like backup, DR or networking overlay applications, such as NSX. vCenter also provides all of the higher level features of vSphere such as vMotion, vSAN, HA, DRS, Distributed Switches and more.

VirtualBox

VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use.

VM

A VM is an abstraction of an operating system from the physical machine by creating a “virtual” representation of the physical hardware the OS expects to interact with, this includes but is not limited to CPU instruction sets, memory, BIOS, PCI buses, etc. A VM is an entirely self-contained entity and shares no components with the host OS. In the case of vSphere the host OS is ESXi (see below).

vSphere

vSphere is the product name of the two core components of the VMware Software Defined Datacenter (SDDC) stack, they are vCenter and ESXi.