Skip to content

UI Tests

Core Development
Debugging GIMP
UI Tests

GIMP has UI tests. A UI test starts a real GIMP, clicks menus and buttons like a person, and then checks the result. This page tells you how to run the tests, how to write a new test, and what to do when a test fails.

What is a UI test⁠

GIMP has two types of tests:

  • Unit tests in app/tests/ and libgimp/tests/. They check one part of the code and do not open a window.
  • UI tests in app/tests/uitests/. They open GIMP and use its interface.

The UI tests do not need a screen. Each test starts a small, private desktop: its own D-Bus, its own accessibility bus and a headless mutter. GIMP runs in this desktop. At the end, the test removes all of it. Nothing on your own desktop changes.

The tests use dogtail to talk to GIMP. Dogtail does not look at pixels. It reads the accessibility tree: the list of windows, menus and buttons that GIMP gives to screen readers. A test finds a button in this tree and tells it to “click”.

Requirements⁠

The UI tests run on Linux only. On Windows and macOS, meson turns them off.

You need these on your system:

  • mutter
  • dbus-run-session (part of D-Bus)
  • at-spi2-core
  • dogtail, for the same Python that meson uses:
pip install dogtail

When you configure the build, meson checks for these. Look for this line in the summary at the end of meson setup:

UI tests             yes

If you see no, the text in brackets tells you what is missing.

Run the tests⁠

All UI tests are in the meson suite ui. From your source directory:

meson test -C _build --suite ui

To run one test, give its name:

meson test -C _build layer-add

To see all the output of a test, add --verbose:

meson test -C _build --verbose layer-add

The UI tests run one at a time. Each test takes some seconds, because GIMP must start each time.

Read a test: layer-add⁠

Before you write a test, read one. The file app/tests/uitests/layer-add.py checks that Layer > New Layer adds a layer. Here are its parts.

1. Start GIMP⁠

Python
from uicommon import close_welcome_dialog, dump_tree, fail, find_first, \
    gimp_session, wait_for

GIMP = sys.argv[1]
MUTTER = sys.argv[2]

with gimp_session(GIMP, MUTTER) as (session, app, proc):
    close_welcome_dialog(app)

Meson gives the test two arguments: the GIMP program and the mutter program. gimp_session() starts the private desktop and GIMP in it. GIMP uses a new, empty configuration folder, so each test starts clean.

app is GIMP in the accessibility tree. All searches start from it.

A new configuration always shows the Welcome dialog. So, close it first.

2. Click a menu item⁠

Python
def click_menu_item(menu_name, item_name):
    menu = find_first(app, lambda n: n.roleName == "menu" and n.name == menu_name)
    if menu is None:
        fail("menu %r not found via AT-SPI:\n%s" % (menu_name, dump_tree(app)))
    menu.do_action_named("click")

    item = wait_for(
        lambda: find_first(app, lambda n: n.roleName == "menu item"
                           and n.name == item_name),
        10)
    if item is None:
        fail("menu item %r not found under %r" % (item_name, menu_name))
    item.do_action_named("click")

Each item in the tree has a role (menu, menu item, push button, dialog…) and a name (the text you see, for example New...). find_first() looks for the first item with the role and the name that you ask for.

do_action_named("click") clicks the item. Dogtail does not move a mouse pointer. It tells the widget directly that it got a click.

The menu item does not show at once. wait_for() tries again and again, for up to 10 seconds, until it finds the item.

3. Confirm a dialog⁠

Python
def confirm_dialog(name_substr, button_name="OK"):
    dlg = wait_for(
        lambda: find_first(app, lambda n: n.roleName == "dialog"
                           and name_substr in (n.name or "").lower()),
        30)
    ...
    btn = find_first(dlg, lambda n: "button" in n.roleName and n.name == button_name)
    ...
    btn.do_action_named("click")

This waits for the dialog, then finds the OK button inside the dialog and clicks it. The search starts from dlg, not from app. Thus, it cannot find an OK button in a different window.

4. Do the task and check the result⁠

Python
click_menu_item("File", "New...")
confirm_dialog("new image")

win = wait_for(image_window, 15)
if "1 layer" not in win.name:
    fail("expected a fresh image to report '1 layer', got: %r" % win.name)

click_menu_item("Layer", "New Layer...")
confirm_dialog("new layer")

win = wait_for(two_layers, 15)
if win is None:
    fail("expected '2 layers' in image window title after adding a layer:\n%s"
         % dump_tree(app))

How does the test know that a layer was added? It reads the title of the image window. By default, the title includes the number of layers, for example [Untitled]-1.0 (RGB color 8-bit gamma integer, GIMP built-in sRGB, 1 layer) 1920x1080. After New Layer, the title must say 2 layers.

If the check is not correct, fail() stops the test and writes the reason. The message includes dump_tree(app), so you can see what was in the tree at that time.

5. End⁠

Python
sys.exit(0)

Exit code 0 tells meson that the test passed. When the with block ends, gimp_session() stops GIMP and removes the private desktop.

Write a new test⁠

1. Make the file⁠

Copy the test that is nearest to your task. For a menu and a dialog, copy layer-add.py. Give the new file a short name, for example app/tests/uitests/my-test.py. Change the comment at the top: tell what the test does and how it checks the result.

This is the minimum test:

Python
#!/usr/bin/env python3
#
# <what this test does, and how it checks the result>
#
# args: <gimp-exe> <mutter-exe>  (gimpmain_exe, mutter_prog, see meson)
# exit: 0 pass, 1 fail, 99 hard error (infra bug: mutter/dogtail/session boot)

import sys

from uicommon import close_welcome_dialog, dump_tree, fail, find_first, \
    gimp_session, wait_for

GIMP = sys.argv[1]
MUTTER = sys.argv[2]

with gimp_session(GIMP, MUTTER) as (session, app, proc):
    close_welcome_dialog(app)

    # do the task here, then check the result.

sys.exit(0)

2. Add it to meson⁠

Open app/tests/uitests/meson.build and add the name (without .py) to the ui_tests list:

meson.build
ui_tests = [
  ...
  { 'name': 'menu-action' },
  { 'name': 'my-test' },
]

The default time limit is 240 seconds. If your test needs more time, set it:

meson.build
  { 'name': 'my-test', 'timeout': 300 },

3. Find the widgets⁠

To find a widget, you need its role and its name. The easiest way is to print the tree. Put this line in your test, at the point where you want to look:

Python
sys.stderr.write(dump_tree(app))

Then run the test with --verbose. You get one line for each widget:

application: 'gimp'
  frame: 'GNU Image Manipulation Program'
    ...
      menu: 'File'
        menu item: 'New...'

Use these roles and names in find_first(). Remove the line when you are done.

4. Check the result⁠

A test that only clicks is not a test. It must check that GIMP did the correct thing. Look for a place where GIMP shows the result:

  • The title of the image window shows the size and the number of layers. See new-image.py and layer-add.py.
  • A text field in a dialog shows a value. color-swap.py reads the color from the hex field in the Colors dialog.
  • A menu name can show a value. zoom.py reads the zoom level from the View > Zoom menu name.
  • A file on disk. export-image.py checks the PNG file, and preferences.py reads the saved gimprc file.

Use an unusual value when you can. For example, new-image.py uses the size 321x123. If the test passes, you know that GIMP used your value and not a default value.

5. Run it⁠

Run your test many times. A test that passes only some of the time is worse than no test.

meson test -C _build --repeat 5 my-test

Then run it as CI does, with no screen (see below).

Helpers in uicommon.py⁠

All tests share app/tests/uitests/uicommon.py:

Helper What it does
gimp_session(gimp, mutter) Starts the private desktop and GIMP. Stops both at the end.
close_welcome_dialog(app) Closes the Welcome dialog, if it shows.
find_first(node, test) Gives the first item below node for which test is true, or None.
find_all(node, test) Gives a list of all the items for which test is true.
wait_for(fn, timeout) Calls fn again and again until it gives something that is not None, or the time ends.
fail(msg) Writes FAIL: msg and stops the test with exit code 1.
hard_error(msg) Writes ERROR: msg and stops with exit code 99. Use it only when the test system is broken, not GIMP.
dump_tree(node) Gives the tree below node as text.

Rules for good tests⁠

  • Wait, do not sleep. GIMP can be slower on CI than on your computer. Always use wait_for() when you look for a widget that must show.
  • Search in the correct place. Two widgets can have the same name. For example, there is a Colors menu and a Colors item in Windows > Dockable Dialogs. Start the search from the menu or dialog that you opened, not from app.
  • Do not use screen positions. Use do_action_named("click"). To set a value, set .value on a spin button or .text on a text field.
  • Give useful fail messages. Tell what you expected and what you got. Add dump_tree() when a widget was not found.
  • One task for each test. A small test is easy to fix when it fails.

Special cases⁠

  • File dialogs have the role file chooser, not dialog. Start GIMP with GTK_USE_PORTAL=0, so that GIMP uses its own file dialog:

    Python
    with gimp_session(GIMP, MUTTER,
                  extra_environment={"GTK_USE_PORTAL": "0"}) as (session, app, proc):

  • Plug-in dialogs (for example the PNG export options) are in a different process. They are not below app. They show as a different application in the tree. See export-image.py.

  • Restart GIMP. If your test must start GIMP more than once, do it inside one session. See welcome-dialog.py.

Mouse drags on the canvas do not work reliably at this time. A single click works (see tool-selection.py), but a paint stroke does not.

When a test fails⁠

Exit codes⁠

Code Meaning
0 The test passed.
1 The test failed. GIMP did not do what the test expected.
77 The test was skipped. Meson shows it as SKIP, not as a failure.
99 Hard error. The test system is broken (for example, mutter did not start). This is not a GIMP bug.

Read the output⁠

A failed test writes a FAIL: line. Read it first. It tells you which step failed. Often a tree dump follows:

FAIL: menu item 'New Layer...' not found under 'Layer'

Look in the tree dump for the widget. Usually one of these is the cause:

  • The name changed in the GIMP interface. Update the test.
  • The widget did not show in time. Look at the timeout.
  • The test found a different widget with the same name. Start the search from a nearer parent.
  • GIMP has a real bug. Then the test did its job!

The full output of all tests is in _build/meson-logs/testlog.txt.

Failures in CI⁠

The UI tests run on each merge request, in the gimp-debian-nonreloc job. If a test fails:

  1. Open the merge request. The Test summary shows the names of the failed tests and their output.
  2. For more detail, open the gimp-debian-nonreloc job log. Expand the section Testing GIMP execution.
  3. Run the same test on your computer as CI does (see the next section).

Run as CI does⁠

On your computer, the test can use parts of your real desktop. CI has no desktop. So, a test can pass on your computer and fail in CI. To get the same conditions as CI, remove the desktop variables before you run the test:

env -u DISPLAY -u WAYLAND_DISPLAY -u XDG_SESSION_TYPE -u DBUS_SESSION_BUS_ADDRESS \
  meson test -C _build --verbose layer-add

If the test fails only like this, the problem is in the test, not in GIMP.

Last updated on
Debugging TipsAlgorithms