Advanced usage

Advanced YAML syntax

YAML has some advanced syntax features that can be used like variables to reduce duplication in your pipeline config:

Anchors & aliases

You can use YAML anchors & aliases as variables in your pipeline config.

To convert this:

steps:
  - name: test
    image: golang:1.18
    commands: go test ./...
  - name: build
    image: golang:1.18
    commands: build

Just add a new section called variables like this:

+variables:
+  - &golang_image 'golang:1.18'

 steps:
   - name: test
-    image: golang:1.18
+    image: *golang_image
     commands: go test ./...
   - name: build
-    image: golang:1.18
+    image: *golang_image
     commands: build

Map merges and overwrites

variables:
  - &base-plugin-settings
    target: dist
    recursive: false
    try: true
  - &special-setting
    special: true
  - &some-plugin codeberg.org/6543/docker-images/print_env

steps:
  - name: develop
    image: *some-plugin
    settings:
      <<: [*base-plugin-settings, *special-setting] # merge two maps into an empty map
    when:
      branch: develop

  - name: main
    image: *some-plugin
    settings:
      <<: *base-plugin-settings # merge one map and ...
      try: false # ... overwrite original value
      ongoing: false # ... adding a new value
    when:
      branch: main

Sequence merges

variables:
  pre_cmds: &pre_cmds
    - echo start
    - whoami
  post_cmds: &post_cmds
    - echo stop
  hello_cmd: &hello_cmd
    - echo hello

steps:
  - name: step1
    image: debian
    commands:
      - <<: *pre_cmds # prepend a sequence
      - echo exec step now do dedicated things
      - <<: *post_cmds # append a sequence
  - name: step2
    image: debian
    commands:
      - <<: [*pre_cmds, *hello_cmd] # prepend two sequences
      - echo echo from second step
      - <<: *post_cmds

References

• Official YAML specification
• YAML cheat sheet

Persisting environment data between steps

One can create a file containing environment variables, and then source it in each step that needs them.

steps:
  - name: init
    image: bash
    commands:
      - echo "FOO=hello" >> envvars
      - echo "BAR=world" >> envvars

  - name: debug
    image: bash
    commands:
      - source envvars
      - echo $FOO

Declaring global variables

As described in Global environment variables, you can define global variables:

WOODPECKER_ENVIRONMENT=first_var:value1,second_var:value2

Note that this tightly couples the server and app configurations (where the app is a completely separate application). But this is a good option for truly global variables which should apply to all steps in all pipelines for all apps.

Docker in docker (dind) setup

warning

This set up will only work on trusted repositories and for security reasons should only be used in private environments. See project settings to enable "trusted" mode.

The snippet below shows how a step can communicate with the docker daemon running in a docker:dind service.

note

If your goal is to build/publish OCI images, consider using the Docker Buildx Plugin instead.

First we need to define a service running a docker with the dind tag. This service must run in privileged mode:

services:
  - name: docker
    image: docker:dind # use 'docker:<major-version>-dind' or similar in production
    privileged: true
    ports:
      - 2376

Next, we need to set up TLS communication between the dind service and the step that wants to communicate with the docker daemon (unauthenticated TCP connections have been deprecated as of docker v27 and will result in an error in v28).

This can be achieved by letting the daemon generate TLS certificates and share them with the client through an agent volume mount (/opt/woodpeckerci/dind-certs in the example below).

services:
  - name: docker
    image: docker:dind # use 'docker:<major-version>-dind' or similar in production
    privileged: true
+    environment:
+      DOCKER_TLS_CERTDIR: /dind-certs
+    volumes:
+      - /opt/woodpeckerci/dind-certs:/dind-certs
     ports:
       - 2376

In the docker client step:

1. Set the DOCKER_* environment variables shown below to configure the connection with the daemon. These generic docker environment variables that are framework-agnostic (e.g. frameworks like TestContainers, Spring Boot Docker Compose do all respect them).
2. Mount the volume to the location where the daemon has created the certificates (/opt/woodpeckerci/dind-certs)

Test the connection with the docker client:

steps:
  - name: test
    image: docker:cli # in production use something like 'docker:<major version>-cli'
+    environment:
+      DOCKER_HOST: "tcp://docker:2376"
+      DOCKER_CERT_PATH: "/dind-certs/client"
+      DOCKER_TLS_VERIFY: "1"
+    volumes:
+      - /opt/woodpeckerci/dind-certs:/dind-certs
    commands:
      - docker version

This step should output the server and client version information if everything has been set up correctly.

Full example:

steps:
  - name: test
    image: docker:cli # use 'docker:<major-version>-cli' or similar in production
    environment:
      DOCKER_HOST: 'tcp://docker:2376'
      DOCKER_CERT_PATH: '/dind-certs/client'
      DOCKER_TLS_VERIFY: '1'
    volumes:
      - /opt/woodpeckerci/dind-certs:/dind-certs
    commands:
      - docker version

services:
  - name: docker
    image: docker:dind # use 'docker:<major-version>-dind' or similar in production
    privileged: true
    environment:
      DOCKER_TLS_CERTDIR: /dind-certs
    volumes:
      - /opt/woodpeckerci/dind-certs:/dind-certs
    ports:
      - 2376