Merge branch 'weekly-cleanup/document-adding-test-runners-des-513'

6 files changed, 250 insertions, 88 deletions

diff --git a/.github/workflows/desktop-e2e.yml b/.github/workflows/desktop-e2e.yml
index 96000a9c64..8594ae32bf 100644
--- a/.github/workflows/desktop-e2e.yml
+++ b/.github/workflows/desktop-e2e.yml
@@ -1,3 +1,11 @@
+# Workflow for triggering `test-manager` on select platforms.
+#
+# This is a rather complex workflow. The complexity mainly stems from these sources:
+# * figuring out which platforms to test on which runners (prepare-matrices)
+# * figuring out if the app and e2e-tests should be built on the runner (build-{linux,windows,macos})
+# or if we should download the artifacts from https://releases.mullvad.net/desktop/
+# * compiling the output from the different runners and executed platforms.
+---
 name: Desktop - End-to-end tests
 on:
   schedule:
@@ -16,6 +24,8 @@ jobs:
       - name: Generate matrix for Linux builds
         shell: bash
         run: |
+          # A list of VMs to run the tests on. These refer to the names defined
+          # in $XDG_CONFIG_DIR/mullvad-test/config.json on the runner.
           all='["debian11","debian12","ubuntu2004","ubuntu2204","ubuntu2304","ubuntu2310","fedora39","fedora38","fedora37","fedora36"]'
           oses="${{ github.event.inputs.oses }}"
           echo "OSES: $oses"
diff --git a/test/README.md b/test/README.md
index c39785e9a9..d9381e2e62 100644
--- a/test/README.md
+++ b/test/README.md
@@ -95,7 +95,7 @@ TARGET=x86_64-pc-windows-gnu ./build.sh
 
 # Building base images
 
-See [`BUILD_OS_IMAGE.md`](./BUILD_OS_IMAGE.md) for how to build images for running tests on.
+See [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md) for how to build images for running tests on.
 
 # Running tests
 
@@ -107,7 +107,7 @@ Here is an example of how to create a new OS configuration and then run all test
 
 ```bash
 # Create or edit configuration
-# The image is assumed to contain a test runner service set up as described in ./BUILD_OS_IMAGE.md
+# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
 cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
     --package-type deb --architecture x64 \
     --artifacts-dir /opt/testing \
diff --git a/test/BUILD_OS_IMAGE.md b/test/docs/BUILD_OS_IMAGE.md
index fbe249e656..f837047c41 100644
--- a/test/BUILD_OS_IMAGE.md
+++ b/test/docs/BUILD_OS_IMAGE.md
@@ -1,46 +1,77 @@
-This document explains how to create base OS images and run test runners on them.
+# Creating a test image
 
-For macOS, the host machine must be macOS. All other platforms assume that the host is Linux.
+This document explains how to create base OS images and run test runners on them.
+This guide is written from the perspective of a Linux user, but it should work on Windows as well.
 
-# Configuring a user in the image
+## Prerequisites
 
-`test-manager` assumes that a dedicated user named `test` (with password `test`) is configured in any guest system which it should control.
-Also, it is strongly recommended that a new image should have passwordless `sudo` set up and `sshd`  running on boot,
-since this will greatly simplify the bootstrapping of `test-runner` and all needed binary artifacts (MullvadVPN App, GUI tests ..).
-The legacy method of pre-building a test-runner image is detailed [further down in this document](#).
+You need to have [QEMU](https://www.qemu.org/) installed.
 
-# Creating a base Linux image
+# Linux
 
 These instructions use Debian, but the process is pretty much the same for any other distribution.
 
+## Creating a virtual machine
+
 On the host, start by creating a disk image and installing Debian on it:
 
 ```
-wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.5.0-amd64-netinst.iso
-mkdir -p os-images
-qemu-img create -f qcow2 ./os-images/debian.qcow2 5G
-qemu-system-x86_64 -cpu host -accel kvm -m 4096 -smp 2 -cdrom debian-11.5.0-amd64-netinst.iso -drive file=./os-images/debian.qcow2
+wget -O debian.iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.5.0-amd64-netinst.iso
+qemu-img create -f qcow2 ./debian.qcow2 5G
+qemu-system-x86_64 -cpu host -accel kvm -m 4096 -smp 2 -cdrom debian.iso -drive file=./debian.qcow2
 ```
 
-## Dependencies to install in the image
+## Installing Linux
 
-`xvfb` must be installed on the guest system. You will also need
-`wireguard-tools` and some additional libraries. They are likely already
-installed if gnome is installed.
+Follow the distribution's installation process. The only important detail is to set up the [user](#User) correctly.
 
-### Debian/Ubuntu
+### User
+
+The testframework expects there to be a regular user in the guest which it can login as.
+The `test-manager` makes no assumptions about the login credentials of this user, but they have to be provided as outlined in the [test-manager configuration specification](../test-manager/docs/config.md).
 
+To better illustrate the remaining steps of the installation process, we assume that a regular system user called `test` has been created.
+
+#### sudo
+
+The user should be able to execute `sudo` without a password.
+
+One way of accomplishing this is to add the `test` user to the `wheel` group
 ```bash
-apt install libnss3 libgbm1 libasound2 libatk1.0-0 libatk-bridge2.0-0 libcups2 libgtk-3-0 wireguard-tools xvfb
+gpasswd -a test wheel
 ```
 
-### Fedora
+and edit `/etc/sudoers` to allow members of `wheel` to execute commands without a password
+```bash
+sudo visudo
+```
+
+Then comment out
+```bash
+## Allows people in group wheel to run all commands
+# %wheel        ALL=(ALL)       ALL
+```
+
+and add
+```bash
+## Same thing without a password
+%wheel  ALL=(ALL)       NOPASSWD: ALL
+```
+
+#### ssh
 
+Make sure that `sshd.service` is enabled on boot.
 ```bash
-dnf install libnss3 libgbm1 libasound2 libatk1.0-0 libatk-bridge2.0-0 libcups2 libgtk-3-0 wireguard-tools xorg-x11-server-Xvfb
+systemctl enable sshd.service
 ```
 
-# Creating a base Windows image
+## Finishing setup
+
+Now you are done! If the VM was configured correctly, `test-manager` will be able to install the required dependencies and run the test suite using the new OS image.
+Now you should [add your new VM to the test-manager config](./test-manager/README.md#configuring-test-manager)
+
+
+# Windows
 
 ## Windows 10
 
@@ -54,8 +85,6 @@ dnf install libnss3 libgbm1 libasound2 libatk1.0-0 libatk-bridge2.0-0 libcups2 l
     qemu-system-x86_64 -cpu host -accel kvm -m 4096 -smp 2 -cdrom <YOUR ISO HERE> -drive file=./os-images/windows10.qcow2
     ```
 
-    (For Windows 11, see the notes below.)
-
 ## Windows 11
 
 * Download an ISO: https://www.microsoft.com/software-download/windows11
@@ -110,7 +139,99 @@ do the following:
 1. Press shift-F10 to open a command prompt.
 1. Type `oobe\BypassNRO` and press enter.
 
-# Creating a testrunner image (Legacy method)
+
+### Bootstrapping the test runner
+
+The test runner needs to be started on boot, with the test runner image mounted at `E:`.
+This can be achieved as follows:
+
+* Restart the VM:
+
+    ```
+    qemu-system-x86_64 -cpu host -accel kvm -m 4096 -smp 2 -drive file="./os-images/windows10.qcow2"
+    ```
+
+* In the guest admin `cmd`, add the test runner as a scheduled task:
+
+    ```
+    schtasks /create /tn "Mullvad Test Runner" /sc onlogon /tr "\"E:\test-runner.exe\" \\.\COM1 serve" /rl highest
+    ```
+
+    Further changes might be required to prevent the task from stopping unexpectedly. In the
+    Task Scheduler (`taskschd.msc`), change the following settings for the runner task:
+
+    * Disable "Start the task only if the computer is on AC power".
+    * Disable "Stop task if it runs longer than ...".
+    * Enable "Run task as soon as possible after a scheduled start is missed".
+    * Enable "If the task fails, restart every: 1 minute".
+
+* In the guest, disable Windows Update.
+
+    * Open `services.msc`.
+
+    * Open the properties for `Windows Update`.
+
+    * Set "Startup type" to "Disabled". Also, click "stop".
+
+* In the guest, disable SmartScreen.
+
+    * Go to "Reputation-based protection settings" under
+      Start > Settings > Update & Security > Windows Security > App & browser control.
+
+    * Set "Check apps and files" to off.
+
+* (Windows 11) In the guest, disable Smart App Control
+
+    * Go to "Smart App Control" under
+      Start > Settings > Privacy & security > Windows Security > App & browser control.
+
+    * Set it to off.
+
+* Enable autologon by creating or editing the following registry values (all of type REG_SZ):
+
+    * Set the current user in
+      `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultUserName`.
+
+    * Set the password in
+      `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultPassword`.
+
+    * Set `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\AutoAdminLogon` to 1.
+
+* Shut down.
+
+## Finishing setup
+
+Now you are done! If the VM was configured correctly, `test-manager` will be able to run the test suite using the new OS image.
+Now you should [add your new VM to the test-manager config](./test-manager/README.md#configuring-test-manager)
+
+# macOS 🚧
+
+_Instructions for building an OS image based on macOS are still under construction._
+
+# Legacy methods
+
+The following instructions are either completely deprecated or needed very seldom. We keep these for future reference.
+
+# Linux
+
+## Test runner dependencies
+
+`xvfb` and `wireguard-tools` must be installed on the guest system.
+You will also need some additional libraries, but these are most likely already installed if `gnome` is installed.
+
+### Debian/Ubuntu
+
+```bash
+apt install wireguard-tools xvfb libnss3 libgbm1 libasound2 libatk1.0-0 libatk-bridge2.0-0 libcups2 libgtk-3-0
+```
+
+### Fedora
+
+```bash
+dnf install wireguard-tools xorg-x11-server-Xvfb nss mesa-libgbm atk alsa-lib-devel at-spi2-atk gtk3
+```
+
+## Creating a test runner image (Legacy method)
 
 The [build-runner-image.sh](./scripts/build-runner-image.sh) script produces a
 virtual disk containing the test runner binaries, which must be mounted when
@@ -121,8 +242,6 @@ you want to build a testrunner-image for.
 
 ## Bootstrapping test runner (Legacy method)
 
-### Linux
-
 The testing image needs to be mounted to `/opt/testing`, and the test runner needs to be started on
 boot.
 
@@ -153,7 +272,7 @@ boot.
 ### Note about SELinux (Fedora)
 
 SELinux prevents services from executing files that do not have the `bin_t` attribute set. Building
-the test runner image stripts extended file attributes, and `e2tools` does not yet support setting
+the test runner image strips extended file attributes, and `e2tools` does not yet support setting
 these. As a workaround, we currently need to reapply these on each boot.
 
 First, set `bin_t` for all files in `/opt/testing`:
@@ -176,62 +295,3 @@ ExecStart=/opt/testing/test-runner /dev/ttyS0 serve
 [Install]
 WantedBy=multi-user.target
 ```
-
-### Windows
-
-The test runner needs to be started on boot, with the test runner image mounted at `E:`.
-This can be achieved as follows:
-
-* Restart the VM:
-
-    ```
-    qemu-system-x86_64 -cpu host -accel kvm -m 4096 -smp 2 -drive file="./os-images/windows10.qcow2"
-    ```
-
-* In the guest admin `cmd`, add the test runner as a scheduled task:
-
-    ```
-    schtasks /create /tn "Mullvad Test Runner" /sc onlogon /tr "\"E:\test-runner.exe\" \\.\COM1 serve" /rl highest
-    ```
-
-    Further changes might be required to prevent the task from stopping unexpectedly. In the
-    Task Scheduler (`taskschd.msc`), change the following settings for the runner task:
-
-    * Disable "Start the task only if the computer is on AC power".
-    * Disable "Stop task if it runs longer than ...".
-    * Enable "Run task as soon as possible after a scheduled start is missed".
-    * Enable "If the task fails, restart every: 1 minute".
-
-* In the guest, disable Windows Update.
-
-    * Open `services.msc`.
-
-    * Open the properties for `Windows Update`.
-
-    * Set "Startup type" to "Disabled". Also, click "stop".
-
-* In the guest, disable SmartScreen.
-
-    * Go to "Reputation-based protection settings" under
-      Start > Settings > Update & Security > Windows Security > App & browser control.
-
-    * Set "Check apps and files" to off.
-
-* (Windows 11) In the guest, disable Smart App Control
-
-    * Go to "Smart App Control" under
-      Start > Settings > Privacy & security > Windows Security > App & browser control.
-
-    * Set it to off.
-
-* Enable autologon by creating or editing the following registry values (all of type REG_SZ):
-
-    * Set the current user in
-      `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultUserName`.
-
-    * Set the password in
-      `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultPassword`.
-
-    * Set `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\AutoAdminLogon` to 1.
-
-* Shut down.
diff --git a/test/docs/GITHUB_ACTIONS.md b/test/docs/GITHUB_ACTIONS.md
new file mode 100644
index 0000000000..86c052a208
--- /dev/null
+++ b/test/docs/GITHUB_ACTIONS.md
@@ -0,0 +1,14 @@
+# `test-manager` in Github Actions
+
+There exists a Workflow for running `test-manager` on a selection of different
+platforms. The Workflow file can be found
+[here](https://github.com/mullvad/mullvadvpn-app/blob/main/.github/workflows/desktop-e2e.yml).
+
+## Adding a new platform
+
+Basically everything related to adding a new virtual machine (VM) to the Workflow is a manual process, but it is fairly straightforward. Most steps (1-3) are performed outside of git, on both your local machine (1) as well as the GitHub runner (2-3). The last step (4) warrants a pull request, as it will make change(s) to the Workflow file.
+
+1. Create a new VM based on [these instructions](./BUILD_OS_IMAGE.md)
+2. Upload the newly-assembled VM to the GitHub Actions runner
+3. Add an entry for the VM in the [test-manager config file](../test-manager/docs/config.md) on the GitHub runner
+4. Update [this Workflow](https://github.com/mullvad/mullvadvpn-app/blob/main/.github/workflows/desktop-e2e.yml) in the [Mullvad App repository](https://github.com/mullvad/mullvadvpn-app/). This will enable GitHub Actions to trigger the `test-manager` with the new VM!
diff --git a/test/test-manager/README.me b/test/test-manager/README.md
index 8e0da31375..16a1b8ac6c 100644
--- a/test/test-manager/README.me
+++ b/test/test-manager/README.md
@@ -45,3 +45,9 @@ Error>`
 // Run a UI test. Panic if any assertion in it fails!
 test_manager::tests::ui::run_test(&rpc, &["gui-test.spec"]).await.unwrap()
 ```
+
+# Configuring `test-manager`
+
+`test-manager` uses a configuration file to keep track of available virtual machines it can use for testing purposes.
+
+More details can be found in [this configuration format document](./docs/config.md).
diff --git a/test/test-manager/docs/config.md b/test/test-manager/docs/config.md
new file mode 100644
index 0000000000..f86315241d
--- /dev/null
+++ b/test/test-manager/docs/config.md
@@ -0,0 +1,72 @@
+# Introduction
+
+This document outlines the format of the configuration used by `test-manager` to perform end-to-end tests in virtualized environments.
+
+# Format
+
+The configuration is a JSON document with two values:
+```json
+{
+    "mullvad_host": <optional string>,
+    "vms": <document>
+}
+```
+
+The configurable values are prone to change, and for the time being it is probably a good idea to get acquainted with the [Rust struct called "Config"](../src/config.rs) from which the configuration is serialized.
+To get started, `test-manager` provides the `test-manager set` command to add and edit VM configurations.
+It is also recommended to view the [example section](#Examples) further down this document.
+
+# Location
+
+The configuration is assumed to exist in `$XDG_CONFIG_HOME/mullvad-test/config.json` (most likely `$HOME/.config/mullvad-test/config.json`) on Linux and `$HOME/Library/Application Support/mullvad-test/config.json` on macOS.
+
+# Examples
+
+## Minimal
+
+The minimal valid configuration does not contain any virtual machines
+```json
+{
+    "mullvad_host": "stagemole.eu",
+    "vms": { }
+}
+```
+
+## Complete
+
+A configuration containing one Debian 12 VM and one Windows 11 VM
+```json
+{
+    "mullvad_host": "stagemole.eu",
+    "vms": {
+        "debian12": {
+          "vm_type": "qemu",
+          "image_path": "$VM_IMAGES/debian12.qcow2",
+          "os_type": "linux",
+          "package_type": "deb",
+          "architecture": "x64",
+          "provisioner": "ssh",
+          "ssh_user": "test",
+          "ssh_password": "test",
+          "disks": [],
+          "artifacts_dir": "/opt/testing",
+          "tpm": false
+        },
+        "windows11": {
+            "vm_type": "qemu",
+            "image_path": "$VM_IMAGES/windows11.qcow2",
+            "os_type": "windows",
+            "package_type": null,
+            "architecture": null,
+            "provisioner": "noop",
+            "ssh_user": null,
+            "ssh_password": null,
+            "disks": [
+              "$TESTRUNNER_IMAGES/windows-test-runner.img"
+            ],
+            "artifacts_dir": "E:\\",
+            "tpm": false
+        }
+    }
+}
+```