Buildkite Test Agent Setup¶

We are using Buildkite for Windows and macOS testing. The build machines and buildkite-agent must be set up before use.

Windows Test Agent Setup¶

Create the user “testbot” on the machine. Use the password for ddevtestbot@gmail.com, available in 1Password.
In admin PowerShell, wsl --install.
(WSL2/Docker Desktop and traditional Windows only): Install either Docker Desktop for Windows (from the release notes page, which is best maintained) or Rancher Desktop. With Rancher Desktop, turn off Kubernetes. (Rancher Desktop on Windows appears to be too unreliable to use at this point in its history.)
In admin PowerShell, Set-ExecutionPolicy -Scope "CurrentUser" -ExecutionPolicy "Unrestricted".
In admin PowerShell, download and run windows_buildkite_start.ps1 with curl <url> -O windows_buildkite_start.ps1.
Install items as needed; git, jq, mysql-cli, golang, make are only required for a traditional Windows test machine. choco install -y git jq mysql-cli golang make mkcert netcat zip.
After restart, in administrative Git Bash window, Rename-Computer <testbot-win10(home|pro)-<description>-1 and then export BUILDKITE_AGENT_TOKEN=<token>.
(Traditional Windows test runner only): Download and run windows_buildkite_setup.sh.
If using Rancher Desktop, adjust the /c/buildkite-agent/buildkite-agent.cfg file to set rancher-desktop=true in the tags instead of docker-desktop. If using Docker Desktop, set docker-desktop=true.
(Traditional Windows test runner only): Download and run windows_postinstall.sh.
(Traditional Windows or Docker Desktop WSL2 Only) Launch Docker. It may require you to take further actions.
- Check “Start Docker Desktop when you sign in” or the equivalent with Rancher Desktop.
- Check “Add the *.docker.internal names to the host’s /etc/hosts file”
- Uncheck “SBOM Indexing”
- Under “Resources” uncheck “Resource Saver”
After starting Docker Desktop or Rancher Desktop, set the correct Docker context in the Git Bash window with docker context use desktop-linux (Docker Desktop) or docker context use default (Rancher Desktop).
Log into Chrome with the user ddevtestbot@gmail.com and enable Chrome Remote Desktop.
(Traditional Windows test runner only): Enable gd, fileinfo, and curl extensions in /c/tools/php*/php.ini.
Set the “Sleep after time” setting in settings to never.
Install winaero tweaker and “Enable user autologin checkbox”. Set up the machine to automatically log in on boot. Then run netplwiz, provide the password for the main user, uncheck “require a password to log in”.
(Traditional Windows test runner only): Set the buildkite-agent service to run as the testbot user and use delayed start: Choose “Automatic, delayed start” and on the “Log On” tab in the services widget it must be set up to log in as the testbot user, so it inherits environment variables and home directory (and has testbot Git config, etc).
(Traditional Windows test runner only): git config --global --add safe.directory '*'.
(Traditional Windows test runner only): Manually run testbot_maintenance.sh, curl -sL -O https://raw.githubusercontent.com/ddev/ddev/main/.buildkite/testbot_maintenance.sh && bash testbot_maintenance.sh.
(Traditional Windows test runner only): Run .buildkite/sanetestbot.sh to check your work.
(Traditional Windows test runner only): Reboot the machine and do a test run. (On Windows, the machine name only takes effect on reboot.)
(Traditional Windows test runner only): Verify that go, ddev, git-bash are in the path.
In “Advanced Windows Update Settings” enable “Receive updates for other Microsoft products” to make sure you get WSL2 kernel upgrades. Make sure to run Windows Update to get the latest kernel.
Turn off the settings that cause the “windows experience” prompts after new upgrades:
In PowerShell: wsl.exe --update. Watch for the elevation to complete, it may require elevation.

Both Docker Desktop/WSL2 and Docker-ce/WSL2¶

The Ubuntu distro should be set up with the user buildkite-agent and the password used for ddevtestbot@gmail.com.
Stop Docker Desktop if it is running.
Log into Chrome with the user ddevtestbot@gmail.com and enable Chrome Remote Desktop.
Windows Terminal should be installed. Set “Ubuntu” (or this distro) as the default and have it start on Windows startup. Enable “copy on select” in behaviors.
nc.exe -L -p 9003 on Windows to trigger and allow Windows Defender.
For Mirrored Mode (normal for these) edit the ~/.wslconfig on Windows to add appropriate WSL2 settings:
```
[wsl2]
networkingMode=Mirrored
[experimental]
hostAddressLoopback=true
```
Before running the setup script, configure mkcert CAROOT on Windows (PowerShell as the testbot user). This must happen first so that mkcert -install in WSL2 installs the Windows CA (not a separate Linux-local one) into the Linux trust store:
```
choco install mkcert -y   # if not already installed
mkcert -install
setx CAROOT (mkcert -CAROOT)
setx WSLENV "CAROOT/up"
wsl --terminate Ubuntu    # restart distro so WSLENV takes effect
```
In the Ubuntu distro:
1. export BUILDKITE_AGENT_TOKEN=<token> with the token from 1Password BUILDKITE_AGENT_TOKEN.
2. export BUILDKITE_DOCKER_TYPE=dockerforwindows or export BUILDKITE_DOCKER_TYPE=wsl2
3. Optionally export NGROK_TOKEN=<token> with the NGROK_TOKEN from 1Password ngrok.com nopaid account.
4. Run the script wsl2-test-runner-setup.sh in the Ubuntu distro. This script reads CAROOT from the Windows registry via powershell.exe, exports it before calling mkcert -install, and then creates /etc/buildkite-agent/hooks/environment to repeat this for every buildkite-agent job (since systemd does not propagate WSLENV).
Restart the distro with wsl.exe -t Ubuntu and then restart it by opening the Ubuntu window.
If using Docker Desktop, start Docker Desktop.
In ~/workspace/ddev/.buildkite, run ./testbot_maintenance.sh.
In ~/workspace/ddev/.buildkite, run ./sanetestbot.sh to check your work.

Icinga2 monitoring setup for WSL2 instances¶

Icinga Director web UI, configure the host on monitor.ddev.com, normally making a copy of an existing identical item.
Deploy the new host using Icinga Director.

sudo icinga2 node wizard to configure the agent, see docs.

buildkite-agent@tb-wsldd-16:~$ sudo icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]:

Starting the Agent/Satellite setup routine...

Please specify the common name (CN) [tb-wsldd-16.localdomain]: tb-wsldd-16

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): monitor.ddev.com

Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): monitor.ddev.com
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

Version:             3
Subject:             CN = monitor.ddev.com
Issuer:              CN = Icinga CA
Valid From:          Feb 22 22:34:09 2026 GMT
Valid Until:         Mar 26 22:34:09 2027 GMT
Serial:              02:0e:20:7e:a1:a9:f0:b4:8d:07:63:49:46:fb:d7:90:29:fd:c4:bc

Signature Algorithm: sha256WithRSAEncryption
Subject Alt Names:   monitor.ddev.com
Fingerprint:         4A 99 90 F6 F4 F7 F9 20 1B DD 9D 51 EE 86 50 C7 BB 38 09 9B B0 40 4A E0 52 C9 52 2D A1 B8 72 D5

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
(Hint: # icinga2 pki ticket --cn 'tb-wsldd-16'): c5b9fa649792abcde3941862a664be0fe8d126
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Local zone name [tb-wsldd-16]:
Parent zone name [master]:

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]:

Do you want to disable the inclusion of the conf.d directory [Y/n]:
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!
buildkite-agent@tb-wsldd-16:~$ sudo systemctl restart icinga2

Restart sudo systemctl restart icinga2
On monitor.ddev.com edit /usr/local/bin/check_buildkite_agents.sh to include the hostname of the new instance.

macOS Docker Desktop Test Agent Setup (Intel and Apple Silicon)¶

Create the user “testbot” on the machine. Use the password for ddevtestbot@gmail.com, available in 1Password.
Change the name of the machine to something in keeping with current style, perhaps testbot-macos-arm64-8. This is done in Settings → General → About → Name and in Sharing → Computer Name and in Sharing → Local Hostname.
Download and install Chrome and log the browser into the account used for test runners. It will pick up the Chrome Remote Desktop setup as a result. Configure Chrome Remote Desktop to serve. When this is done, the machine will be available for remote access and most other tasks can be done using Chrome Remote Desktop.
The machine should be on the correct network and have a static IP handed out by DHCP. IP addresses are listed in /etc/hosts on pi.ddev.site, so this one should be added.
Power should be set up as in .
Auto login should be set up as in , see automatically log in on boot.
Remote login should be enabled as in .
Automatic updates should be set to mostly security only as in .
Set the time zone to US MT (nearest city: Denver, Colorado).
sudo mkdir -p /usr/local/bin && sudo chown -R testbot /usr/local/bin
Install Homebrew /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
After installing Homebrew follow the instructions it gives you at the end to add brew to your PATH.
Install everything you’ll need with brew install buildkite/buildkite/buildkite-agent bats-core composer ddev/ddev/ddev git golang jq mysql-client@8.0 mkcert netcat p7zip && brew install --cask docker iterm2 ngrok && brew link --force mysql-client.
Run ngrok authtoken <token> with token for free account from 1Password.
Run mkcert -install.
If Docker Desktop will be deployed, run Docker manually and go through its configuration routine.
If OrbStack will be deployed, install it from orbstack.dev.
- Install with Docker only.
- Click “Sign in” in the lower left to sign in with OrbStack credentials (normal test runner gmail address; it will receive an email with a login code).
- Configure it to automatically start and download updates, see .
If Rancher Desktop will be deployed, install it.
- Turn off kubernetes.
- Set virtualization to VZ
- Set mount type to VirtioFS
Run iTerm. You may need to allow full disk access permissions.
Run mkdir ~/workspace && cd ~/workspace && git clone https://github.com/ddev/ddev.
git config --global --add safe.directory '*'.
Edit /usr/local/etc/buildkite-agent/buildkite-agent.cfg or /opt/homebrew/etc/buildkite-agent/buildkite-agent.cfg to add
- the agent token (from agents tab, “Reveal Agent Token”).
- the agent name (the name of the machine).
- tags, like "os=macos,architecture=arm64,osvariant=sonoma,dockertype=dockerformac,rancher-desktop=true,orbstack=true,docker-desktop=true"
- build-path="~/tmp/buildkite-agent/builds"
Run brew services start buildkite-agent.
Run bash ~/workspace/ddev/.buildkite/testbot_maintenance.sh.
Run bash ~/workspace/ddev/.buildkite/sanetestbot.sh to check your work.
The testbot user’s SSH account is used for monitoring, so ssh-keygen and then add the public key id_testbot from 1Password to ~/.ssh/authorized_keys and chmod 600 ~/.ssh/authorized_keys.
Add the new machine to Icinga by copying an existing Icinga service to the new one. This is done in Icinga Director → Services → Single Services → Select a Service → Clone → Deploy. The new service has to have by-ssh-address set to the name of the test runner, and that address needs to be added to pi.ddev.site’s /etc/hosts file.
If zsh is the shell configured, add /etc/zshenv so that /usr/local/bin/docker will be picked up:
```
PATH=$PATH:/usr/local/bin:/opt/homebrew/bin
```
In macOS Settings visit “full disk access” and grant access to buildkite-agent, docker, iterm, orbstack. This may prevent startup modal dialogs that prevent buildkite-agent or docker from continuing properly.

Additional Colima macOS setup¶

brew install colima
colima start vz --cpu 4 --memory 6 --disk 60 --vm-type=vz --mount-type=virtiofs --dns=1.1.1.1
colima stop vz

Then the Buildkite agent must be configured with tags colima_vz=true.

Additional Lima macOS setup¶

limactl create --name=lima-vz --vm-type=vz --mount-type=virtiofs --mount-writable --mount="~/:w" --memory=6 --cpus=4 --disk=60 template:docker
limactl start lima-vz
docker context use lima-lima-vz

Then the Buildkite agent must be configured with tags lima=true.

Running Targeted Builds on Specific Pipelines¶

To test a branch against only selected pipelines (e.g. WSL2 only) or to run a subset of tests without waiting for the full matrix:

Push your branch to upstream.
In the Buildkite dashboard, open the pipeline you want (e.g. “wsl2-docker-inside”).
Click New Build and set the branch to your branch name.
Expand Environment Variables and add:
```
TESTARGS=-run TestCheck -failfast
```
Adjust the -run pattern to match the tests you want. Use a common prefix where possible. If you need | for alternation, wrap the regular expression in single quotes so the shell does not interpret the pipe as a pipeline operator:
```
TESTARGS=-run 'TestCheckLiveConnectivity|TestCheckMkcertInstallation' -failfast
```
TESTARGS defaults to -failfast when not set, so include -failfast explicitly if you want it.
Click Create Build.

Use [skip ci] in your commit message to prevent all pipelines from triggering automatically when you push, then manually trigger only the pipelines you need.