Writing Tests

WARNING

Contributors willing to work on tests should be familiar with NixOS test framework.

Overview

All tests live inside the tests directory and are automatically imported into the test suite using a custom mkTest function which allows various ways to define the tests:

1. As a simple attribute set, which will be used as the test machine NixOS module

   {
     home-manager.users.root.programs.nixkraken.enable = true;
   }

2. As a list of attribute sets, which will be used as the test machines NixOS modules

   [
     {
       home-manager.users.root.programs.nixkraken.enable = true;
     }
     # ...other machines
   ]

3. As an attribute set with machine and extraOptions attributes, respectively used as the test machine(s) NixOS module(s) and additional test options

   {
     machine = {
       home-manager.users.root.programs.nixkraken.enable = true;
     };
   
     # Or multiple machines
     # machine = [
     #   {
     #     home-manager.users.root.programs.nixkraken.enable = true;
     #   }
     # ]
   
     extraOptions = {
       skipTypeCheck = true;
       extraPythonPackages = p: with p; [ numpy ]; };
   }

4. As a function called with the pkgs attribute set containing nixpkgs, which can return either of the previous attribute sets

   pkgs:
   
   {
     environment.systemPackages = with pkgs; [
       jq
     ];
   }

   # Variant
   { jq, ... }:
   
   {
     environment.systemPackages = [
       jq
     ];
   }

INFO

• all tests use one or more machines
• all tests have OCR capabilities enabled (which cannot be disabled)
• all test machines share default configuration (which is not overwritable)
• using the test option is disallowed in favor of testScript (see test rules about files below)

Flake Exposure

Tests are exposed as legacyPackages Flake outputs rather than packages for the following reasons:

• they are still runnable/buildable with nix run and nix build
• they are not validated by nix flake check (.#tests is a namespace, not a derivation)
• they are not built by Garnix (avoids CI overhead and inevitable build failures due to previous point)

Rules

Each test must follow the rules described in this section.

Naming Convention

Use a clear, concise and descriptive name in kebab case

• This is correct: accept-eula
• This is incorrect: acceptEula, accept_eula, …

Directory Structure

Each test has its own subdirectory, matching its name.

Example:

tests
└── accept-eula
    ├── default.nix
    └── test.py

Required Files

At a minimum, each test should define two files:

• default.nix: defines the machine module and, optionally, extra test options beyond default ones
• test.py: contains the Python test logic (automatically loaded in testScript test option - read the dedicated section for further details)

Additional files relevant to the test can be added in the test directory. Look at the datetime test for a real-world example.

Taking Screenshots

When graphical output is being validated, screenshots must be produced using the expression below:

# Take a screenshot of the machine
machine1.screenshot('snapshot')

The test framework will generate screenshots in PNG format in the derivation output.

Use Subtests

Even when a test is testing a single thing, use subtest as shown below:

with subtest("Test name"):
    # Actual test code

See minimal example for details.

About test.py

As previously noted, all tests must define a test.py file containing the Python test logic. Find below some useful details about it:

• test machines are named machine{id}, where {id} is the machine index starting at 1
• each test machine is exposed as a Machine object identified by its name (i.e., machine1, machine2, …)
• the Machine object provides methods to interact with the matching test machine:
  • execute shell commands
  • get a textual representation of the machine screen
  • take screenshots of the machine display
  • send arbitrary typing sequences
  • simulate pressing keys
  • wait for various operations like X server start, window to appear, text to be displayed, …
  • …and much more
• the t object exposes all assertions from Python's unittest.TestCase

GitKraken Quirks

• Because GitKraken is a graphical application, most tests require starting an X server
• GitKraken will fail to run under root user unless --no-sandbox flag is used
• waiting for GitKraken window succeeds before the window is actually drawn on screen, requiring a sleep workaround

Minimal Example

# pyright: reportUndefinedVariable=false

# Wait for graphical server
machine1.wait_for_x()

with subtest("Test name"):
    # GitKraken won't launch unless '--no-sandbox' is set when running as root
    # Disable splashscreen with '--show-splashscreen' ('-s') set to false
    machine1.succeed("gitkraken --no-sandbox -s false >&2 &")

    # Wait for window to show up
    # WARN: for some reason, this succeeds a few seconds before the window actually
    #       shows up on screen, hence the 15 seconds sleep workaround (which is required)
    machine1.wait_for_window("GitKraken Desktop")
    machine1.sleep(15)

    # Dummy test example, actual tests should go here
    machine1.succeed("true")

    # Take a screenshot of GitKraken
    machine1.screenshot("snapshot")

# Exit GitKraken
machine1.succeed("pkill -f gitkraken")

Shared Configuration

The _common directory holds NixOS modules shared across all tests to avoid repetition and ensure a consistent environment.

tests/_common
├── base-config.nix
├── default.nix
├── display.nix
└── nixkraken.nix

Additionally, all tests have NixKraken force-enabled (with enabled option) and are using the latest GitKraken version available, unless stated otherwise.

default.nix

Imports the minimum required configuration to be able to use NixKraken in tests.

It automatically imports display.nix and nixkraken.nix, so that tests only need to import _common for being able to test NixKraken.

INFO

The _common directory is imported by default in all tests.

display.nix

Defines NixOS configuration options to enable display in tests.

It configures X11 and a display/window manager, required by most tests since GitKraken is a graphical application.

It should remain mostly stable, updated only for compatibility with future NixOS versions.

nixkraken.nix

Enables both Home Manager and NixKraken in tests.

This is (obviously?) required to be able to test NixKraken.

WARNING

Whenever nixpkgs gets updated in flake.nix, the Home Manager version defined in this file must be updated accordingly.

This is due to the way Home Manager works: both Home Manager and nixpkgs versions must be in sync.

base-config.nix

Defines a basic working GitKraken configuration which will give a usable UI on app launch.

This configuration is optional. It is useful for tests that needs a working app to perform tests.

A good example of this is the datetime test, which needs to open the repository view to perform its tests.