Testing

Backend

Unit Tests

We use default golang unit tests with "github.com/stretchr/testify/assert" to simplify testing.

Tests that need test-only code, like the dummy backend, start with //go:build test. They only run if the test build tag is set, see testing & linting code.

Mocks

Mocks of interfaces are generated by mockery as configured in .mockery.yaml.

After you change an interface, regenerate them with:

make generate

If an interface has no mock yet, just add it to .mockery.yaml in a dedicated pull request.

Integration Tests

The tests in e2e/ start a server and an agent inside the test process and let them run whole pipelines. The server uses an in-memory SQLite database and a mocked forge, the agent uses the dummy backend. So no forge and no container runtime are needed. Run them with:

make test-e2e

Most test cases are scenarios in e2e/scenarios/fixtures/. A scenario is a workflow file and a JSON file of the same name that describes the expected result:

02_step_failure.yaml

skip_clone: true

steps:
  - name: build
    image: dummy
    commands:
      - echo building
    environment:
      STEP_EXIT_CODE: '1'

  - name: deploy
    image: dummy
    commands:
      - echo deploying

02_step_failure.json

{
  "name": "step failure stops pipeline",
  "event": "push",
  "expected_status": "failure",
  "expected_workflows": [{ "name": "woodpecker", "status": "failure", "error": "" }],
  "expected_steps": [
    { "name": "build", "status": "failure", "exit_code": 1 },
    { "name": "deploy", "status": "skipped", "exit_code": 0 }
  ]
}

A scenario with multiple workflows is a folder that contains one YAML file per workflow and a scenario.json. New scenarios are picked up automatically, you do not have to write Go code for them.

Cases that need more than a comparison of the final result, like canceling or restarting a pipeline, are written as Go tests in e2e/scenarios/ and use the helpers of e2e/setup/.

Dummy backend

There is a special backend called dummy which does not execute any commands, but emulates how a typical backend should behave. To enable it you need to build the agent or cli with the test build tag.

An example pipeline config would be:

when:
  event: manual

steps:
  - name: echo
    image: dummy
    commands: echo "hello woodpecker"
    environment:
      SLEEP: '1s'

services:
  echo:
    image: dummy
    commands: echo "i am a service"

This could be executed via woodpecker-cli --log-level trace exec --backend-engine dummy example.yaml:

9:18PM DBG pipeline/pipeline.go:94 > executing 2 stages, in order of: CLI=exec
9:18PM DBG pipeline/pipeline.go:104 > stage CLI=exec StagePos=0 Steps=echo
9:18PM DBG pipeline/pipeline.go:104 > stage CLI=exec StagePos=1 Steps=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:75 > create workflow environment taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:176 > prepare CLI=exec step=echo
9:18PM DBG pipeline/pipeline.go:203 > executing CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:81 > start step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:167 > tail logs of step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:209 > complete CLI=exec step=echo
[echo:L0:0s] StepName: echo
[echo:L1:0s] StepType: service
[echo:L2:0s] StepUUID: 01J10P578JQE6E25VV1A2DNQN9
[echo:L3:0s] StepCommands:
[echo:L4:0s] ------------------
[echo:L5:0s] echo ja
[echo:L6:0s] ------------------
[echo:L7:0s] 9:18PM DBG pipeline/pipeline.go:176 > prepare CLI=exec step=echo
9:18PM DBG pipeline/pipeline.go:203 > executing CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:81 > start step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:167 > tail logs of step echo taskUUID=01J10P578JQE6E25VV1EQF0745
[echo:L0:0s] StepName: echo
[echo:L1:0s] StepType: commands
[echo:L2:0s] StepUUID: 01J10P578JQE6E25VV1DFSXX1Y
[echo:L3:0s] StepCommands:
[echo:L4:0s] ------------------
[echo:L5:0s] echo ja
[echo:L6:0s] ------------------
[echo:L7:0s] 9:18PM TRC pipeline/backend/dummy/dummy.go:108 > wait for step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:187 > stop step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:209 > complete CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:208 > delete workflow environment taskUUID=01J10P578JQE6E25VV1EQF0745

There are also environment variables to alter step behavior:

• SLEEP: 10 will let the step wait 10 seconds
• EXPECT_TYPE allows to check if a step is a clone, service, plugin or commands
• STEP_START_FAIL: true if set will simulate a step to fail before actually being started (e.g. happens when the container image cannot be pulled)
• STEP_TAIL_FAIL: true if set will error when we simulate to read from stdout for logs
• STEP_EXIT_CODE: 2 if set will be used as exit code, default is 0
• STEP_OOM_KILLED: true simulates a step being killed by memory constrains
• STEP_WAIT_FAIL: true if set will error when we wait for the step to finish (e.g. happens when the kubernetes backend cannot pull the image)
• STEP_STATE_ERROR: true if set will report an error in the step state next to its exit code
• STEP_DESTROY_FAIL: true if set will error when we remove the finished step

You can let the setup of a whole workflow fail by setting it's UUID to WorkflowSetupShouldFail.