Skip to main content
Version: 2.5.x ๐Ÿ’€

Kubernetes backend

The Kubernetes backend executes steps inside standalone Pods. A temporary PVC is created for the lifetime of the pipeline to transfer files between steps.

Images from private registriesโ€‹

In order to pull private container images defined in your pipeline YAML you must provide registry credentials in Kubernetes Secret. As the Secret is Agent-wide, it has to be placed in namespace defined by WOODPECKER_BACKEND_K8S_NAMESPACE. Besides, you need to provide the Secret name to Agent via WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES.

Job specific configurationโ€‹

Resourcesโ€‹

The Kubernetes backend also allows for specifying requests and limits on a per-step basic, most commonly for CPU and memory. We recommend to add a resources definition to all steps to ensure efficient scheduling.

Here is an example definition with an arbitrary resources definition below the backend_options section:

steps:
- name: 'My kubernetes step'
image: alpine
commands:
- echo "Hello world"
backend_options:
kubernetes:
resources:
requests:
memory: 200Mi
cpu: 100m
limits:
memory: 400Mi
cpu: 1000m

You can use Limit Ranges if you want to set the limits by per-namespace basis.

Runtime classโ€‹

runtimeClassName specifies the name of the RuntimeClass which will be used to run this Pod. If no runtimeClassName is specified, the default RuntimeHandler will be used. See the Kubernetes documentation for more information on specifying runtime classes.

Service accountโ€‹

serviceAccountName specifies the name of the ServiceAccount which the Pod will mount. This service account must be created externally. See the Kubernetes documentation for more information on using service accounts.

Node selectorโ€‹

nodeSelector specifies the labels which are used to select the node on which the job will be executed.

Labels defined here will be appended to a list which already contains "kubernetes.io/arch". By default "kubernetes.io/arch" is inferred from the agents' platform. One can override it by setting that label in the nodeSelector section of the backend_options. Without a manual overwrite, builds will be randomly assigned to the runners and inherit their respective architectures.

To overwrite this, one needs to set the label in the nodeSelector section of the backend_options. A practical example for this is when running a matrix-build and delegating specific elements of the matrix to run on a specific architecture. In this case, one must define an arbitrary key in the matrix section of the respective matrix element:

matrix:
include:
- NAME: runner1
ARCH: arm64

And then overwrite the nodeSelector in the backend_options section of the step(s) using the name of the respective env var:

[...]
backend_options:
kubernetes:
nodeSelector:
kubernetes.io/arch: "${ARCH}"

You can use PodNodeSelector admission controller if you want to set the node selector by per-namespace basis.

Tolerationsโ€‹

When you use nodeSelector and the node pool is configured with Taints, you need to specify the Tolerations. Tolerations allow the scheduler to schedule Pods with matching taints. See the Kubernetes documentation for more information on using tolerations.

Example pipeline configuration:

steps:
- name: build
image: golang
commands:
- go get
- go build
- go test
backend_options:
kubernetes:
serviceAccountName: 'my-service-account'
resources:
requests:
memory: 128Mi
cpu: 1000m
limits:
memory: 256Mi
nodeSelector:
beta.kubernetes.io/instance-type: p3.8xlarge
tolerations:
- key: 'key1'
operator: 'Equal'
value: 'value1'
effect: 'NoSchedule'
tolerationSeconds: 3600

Volumesโ€‹

To mount volumes a PersistentVolume (PV) and PersistentVolumeClaim (PVC) are needed on the cluster which can be referenced in steps via the volumes option. Assuming a PVC named woodpecker-cache exists, it can be referenced as follows in a step:

steps:
- name: "Restore Cache"
image: meltwater/drone-cache
volumes:
- woodpecker-cache:/woodpecker/src/cache
settings:
mount:
- "woodpecker-cache"
[...]

Security contextโ€‹

Use the following configuration to set the Security Context for the Pod/container running a given pipeline step:

steps:
- name: test
image: alpine
commands:
- echo Hello world
backend_options:
kubernetes:
securityContext:
runAsUser: 999
runAsGroup: 999
privileged: true
[...]

Note that the backend_options.kubernetes.securityContext object allows you to set both Pod and container level security context options in one object. By default, the properties will be set at the Pod level. Properties that are only supported on the container level will be set there instead. So, the configuration shown above will result in something like the following Pod spec:

kind: Pod
spec:
securityContext:
runAsUser: 999
runAsGroup: 999
containers:
- name: wp-01hcd83q7be5ymh89k5accn3k6-0-step-0
image: alpine
securityContext:
privileged: true
[...]

You can also restrict a container's syscalls with seccomp profile

backend_options:
kubernetes:
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/audit.json

or restrict a container's access to resources by specifying AppArmor profile

backend_options:
kubernetes:
securityContext:
apparmorProfile:
type: Localhost
localhostProfile: k8s-apparmor-example-deny-write
note

AppArmor syntax follows KEP-24.

Annotations and labelsโ€‹

You can specify arbitrary annotations and labels to be set on the Pod definition for a given workflow step using the following configuration:

backend_options:
kubernetes:
annotations:
workflow-group: alpha
io.kubernetes.cri-o.Devices: /dev/fuse
labels:
environment: ci
app.kubernetes.io/name: builder

In order to enable this configuration you need to set the appropriate environment variables to true on the woodpecker agent: WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEP and/or WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEP.

Tips and tricksโ€‹

CRI-Oโ€‹

CRI-O users currently need to configure the workspace for all workflows in order for them to run correctly. Add the following at the beginning of your configuration:

workspace:
base: '/woodpecker'
path: '/'

See this issue for more details.

Configurationโ€‹

These env vars can be set in the env: sections of the agent.

WOODPECKER_BACKEND_K8S_NAMESPACEโ€‹

Default: woodpecker

The namespace to create worker Pods in.

WOODPECKER_BACKEND_K8S_VOLUME_SIZEโ€‹

Default: 10G

The volume size of the pipeline volume.

WOODPECKER_BACKEND_K8S_STORAGE_CLASSโ€‹

Default: empty

The storage class to use for the pipeline volume.

WOODPECKER_BACKEND_K8S_STORAGE_RWXโ€‹

Default: true

Determines if RWX should be used for the pipeline volume's access mode. If false, RWO is used instead.

WOODPECKER_BACKEND_K8S_POD_LABELSโ€‹

Default: empty

Additional labels to apply to worker Pods. Must be a YAML object, e.g. {"example.com/test-label":"test-value"}.

WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEPโ€‹

Default: false

Determines if additional Pod labels can be defined from a step's backend options.

WOODPECKER_BACKEND_K8S_POD_ANNOTATIONSโ€‹

Default: empty

Additional annotations to apply to worker Pods. Must be a YAML object, e.g. {"example.com/test-annotation":"test-value"}.

WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEPโ€‹

Default: false

Determines if Pod annotations can be defined from a step's backend options.

WOODPECKER_BACKEND_K8S_SECCTX_NONROOTโ€‹

Default: false

Determines if containers must be required to run as non-root users.

WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMESโ€‹

Default: empty

Secret names to pull images from private repositories. See, how to Pull an Image from a Private Registry.