Action Search (Text-based Intent driven Tool)

Table of Contents

TODO: this spec has been implemented years ago and various features changed since then. The spec should be updated accordingly.

Text-based Intent driven Tool (TITo)

This is a first draft of specifications for an Action Search dialog box. The feature has been originally contributed by Srihari and improved from there. The current code is available in the branch “tito” on the GNOME git repository of GIMP.

Unless bugs are found, the following specifications is implemented in this branch.

Author of the Specs: Jehan

Testing, feedback, comments and suggestions: Alexandre Prokoudine and Téo Mazars (did I forget anyone?).

Background

GIMP provides a huge list of individual commands (actions in GTK terminology): generic actions (open/close/save an image…), pixel processing (GEGL operations), plugins, etc. Currently you can run any of these actions either through scripts, shortcuts or the UI if a widget has been created to run it (usually a menu item).

Advanced users want to work fast. Searching through menus can be painfully slow in many cases, in particular when you need to do it very often (users who use a lot of filters). Creating shortcuts is usually a good solution, but only for a limited set of actions, the ones you really use the most often, if not all the time. Indeed shortcuts are limited by the number of possibility in key associations, but also because you may forget the shortcuts if you don’t use them often enough. By default, GIMP is provided with hundreds of actions, and you can create an infinity (some very spread plugins, the ones providing many additional filters in particular, would add dozens of them).

On the other hand, if you may forget an association of meaningless keys, you won’t usually forget natural language keywords, like parts of the name of the filter, the type of the filter, what it does, etc. This feature is proposing to be able to search any action at any time by such keywords.

Overview

The “tito” feature would be named more generically: “Action Search”.

The correspondent action is named “dialogs-action-search”. It would pop-up a dialog box, displaying a simple text entry widget where a user can type keywords, thus automatically updating a list of proposed actions. The proposed actions can be browsed and activated either by mouse or keyboard.

Similar Feature in other Software

Blender has a similar feature called the “space-menu” (its shortcut being “Space”). It can be run from everywhere through Blender’s UI and would provide a list of all available commands.

This is a very practical feature in Blender.

User Interaction

Activating

The default shortcut would be ‘/’, common shortcut for quick-search in many software (text editors, web browsers, etc.), and not assigned yet in GIMP. ‘/’ is a 1-letter shortcut which makes it indeed a very nice default for a “quick” search.
Note: ‘ctrl-f’ was also considered (being also a very common shortcut for a “search” feature in most software) but it was already used for “plug-in-repeat” (Repeat Last Filter).
The “action-search” is also available as a menu item in “Help” > “Search and Run a Command”.

Quitting

The action-search dialog can be closed either by hitting Esc, or by closing the dialog with any usual way provided by the user’s windows manager (usually hitting a window cross on top-right, with common Alt-F4 shortcut, or any alternative which depends on the desktop and is not controlled by GIMP).

The dialog can be called back up again at any time by the previous methods (shortcut, menu…).

Proposed Action List

List Content

The proposed action list order is relevant. The top item is computed as being the most relevant action to the search, the second slightly less, and so on, until the last, which is still considered relevant, but less than all the others. See the section Action Search Algorithm for details on how relevance is computed.
By default, the proposed action list shows only the actions available in the current context (the actions which can actually be run at this time). For instance if no file is currently opened, the “view-close” action won’t appear in the list of actions because there are no views to close. Nevertheless you can set up the action-search to also display the unavailable actions (see the “Preferences” section). But even by doing so, the unavailable actions will be listed, but will be grayed and not selectable.

Single Action Item

Here is a list of values relevant to an action:

Action Name: the action name is the actual GTK naming, usually derived from the English language, though it should not be considered as “Natural Language”, but as “Machine Language”. This information is not displayed in the list.
Action Label: each action in the proposed action list shows a human-language label (localized to user’s chosen locale). This information is the minimum available for all actions.

For instance, the label for the action named view-close is: “Close View”. This label will likely be different if GIMP is localized in another language.

Action Stock Icon: if an action has an associated stock icon, it will be displayed in small version to the left of the label.

As a special exception, a toggle action (for instance “windows-use-single-window-mode”) would display a OK/NO icon to characterize its current state.

Action Shortcut/Accelerator: if the action has a shortcut assigned, it is displayed to the right of the label, in a smaller font size, after a pipe sign. This allows a user to assimilate and learn shortcuts of actions commonly run.
Action Description/Tooltip: if the action has a longer description, it is displayed below the Action Name, in a smaller and lighter font style.

Example: our action view-close will display like this in the proposed action list:

X Close View | Ctrl + W
  Close the active image view

Update

The proposed action list would appear as soon as you type a sign in the text entry. It is automatically updated at each change of the entered text.

When the proposed action list is updated, the top item of the list (judged the most relevant, see below) is selected by default, making it easy to immediately run it with Enter.

The proposed action list disappears when the text entry gets empty again (by hitting backspace repetively, or selection/delete, or any other mean).

When the proposed action list disappears, any previous selection is unselected, avoiding accidental and unintuitive action run when pressing Enter on an empty keyword.

When the text entry is empty, typing the “down arrow” would force-show the full list of all actions (this full list is different whether or not you set up to show unavailable actions) and focus and select the first item. The top items in the list are all the actions in history, sorted by most used first.
When the text entry has the focus, hitting the down arrow would focus and select the top proposition (most relevant one). When a proposition is highlighted, hitting the down arrow again, would highlight the next one, and so on, until the last proposition is highlighted, at which point hitting the down arrow again won’t do anything. Alternative Proposition: since when the text entry has focus, the top proposition is already selected by default, it has been proposed by Téo Mazars that hitting the down key would select directly the second-to-the-top proposition. This may indeed be a better idea. Thoughts?
When the first action is highlighted, hitting the up arrow gives focus back to the text entry.
When an action is highlighted, other than the first one, hitting the up arrow highlights the immediately upper action.
When an action is highlighted, hitting Enter would run this action and close the action-search dialog.
When no action is highlighted, hitting Enter would do nothing.
At any time, hitting Esc would close the action-search dialog without any action being run.
Hitting any other key would give focus back to the text entry, with the pointer and text selection being the exact same as they were before the text entry lost focus. This means that you can always easily update your keywords without having to repetitively use the arrow keys, or the mouse.

The proposed action list will show a scroll bar if the dialog is not big enough to show the whole list of proposed actions on screen. You can scroll through the whole list with your mouse with the usual usage of scroll bar interface.
Clicking an action in the list would select this action.
Double-clicking an action in the list would run this action and close the action-search dialog.

Action History

The feature comes alongside a separate feature: action calls are logged. Only the relative frequency they have been run is memorized. They are logged however the action is run, whether through menu, shortcut, script or action-search itself.

Moreover only a limited number of actions are memorized. This default of “action history length” is 100 actions. A value of 100 means that the 100 most used actions will be kept in memory (and not the last 100 run actions).

Some actions are not logged:

“dialogs-action-search” itself to avoid circular calling;
any action “*-menu” because they are run each time you open a menu, which is very informative;
any action “*-popup”;
any action “context-*” because they happen too often in normal usage (select a tool, etc.);
“plug-in-repeat”, “plug-in-reshow” and any action “plug-in-recent-*” because they are only redirection of actual actions.

NOTE: the number associated to an action does not necessarily correspond to the number of time. I call it a “relative frequency”. The algorithm is the following:

When an action is called, if it was not previously logged, its frequency is 1 and is listed at the top of the 1-actions.
When an action is called, if it was already logged and if other actions are associated to the same frequency number, this action’s frequency is incremented by 1 and at the top of the actions with the same new frequency.
When an action is called, it it was already logged and if no other actions are associated to the same frequency number, but there is at least one action with the next frequency number, then this action’s frequency is incremented by 1 and at the top of the actions with the same new frequency; and the last of the actions with the next frequency has its number decremented by 1.
In any other cases, nothing is changed.

This allows to always have actions with frequency 1, thus we would never end up in a blocking case where the history is full but there are no 1-actions (thus a new action could not be entered). It also prevents highly used actions from ending up with huge frequency and rendering results inadequate, since there would never be a number gap in the frequency list.

Note: the dialog box has been quite modified from the original proposition, but it had several interesting concepts, like the proposition list automatically updated while typing (similarly to a web browser URL bar). So Mitch proposed that the finale version could be made a generic GIMP widget, which may be reused for other purpose in the future (other search tools?). Below are additional features of this dialog widget.

Position and Dimensions

If you often use the search-action dialog, you will likely appreciate not having to displace and resize it each time on the screen (for instance if you want to see your image, or any other reason). For this reason, the dialog remembers where it was positioned last, as well as the width of the text entry, and the height of the proposed action list. This uses GIMP session management.

Opacity

The search dialog may be used over other windows, that we may still want to see. Thus the dialog displays an opacity button (icon GIMP_STOCK_TRANSPARENCY), which shows/hide an opacity spin scale. The chosen opacity is also remembered from one run to another.

Counter-Proposition: the possibility to make the dialog transparent was in the original contribution by Srihari (it was set through a separate settings window, now removed). The developer who took over the integration (Jehan) thought the idea appealing, so made the current evolved Opacity button/spin UI, though is unsure if that is really that useful or even desirable. Téo Mazars also wonders if that would not be more interesting as a feature in other kind of dialogs, like in filter dialogs where you get a live feedback in the image window, but maybe not on this one. Thus this opacity button/spin may be removed. Or maybe we could make them a separate widgets of their own, easy to add to other kind of dialogs.

NOTE: the “opacity” feature has been removed from the finale version. It may reappear later as a generic widget that we may want to use on various windows.

Preferences

The preferences propose 2 preferences in “Help System” tab, “Action Search” frame:

the “Maximum History Size” spin button (see “Action History” section), integer, default 100;
the “Show Unavailable Actions” checkbox (see “Proposed Action List” section), default unchecked.

It also proposes a button to “Clear Action History”.

Action Search Algorithm

Note 1: in the following algorithm, all text (keyword, label, tooltip) is normalized prior to searching: case-insensitive, stripped of leading and trailing spaces, and all duplicated spaces crushed to 1 space, which means that searching “WoRd”, “word” or " WORD " would get the same result.

Note 2: depending on the show-unavailable preference value, the meaning of “any action” differs. If show unavailable is unchecked, it actually means “any action which is available”.

Note 3: the entry text is stripped out of any leading and trailing spaces before searching.

If the entry text has a content (a “keyword”), the algorithm shows first any action in the history, for which, in this order:
1. if and only if the entry text is exactly 2 letters, the first letter of the keyword is the first letter of the first word of the Action Label; and the second letter of the keyword is the first letter of the second word of the Action Label. Example: “gb” selects “Gaussian Blur”.
2. the searched keyword appears at the start of the Action Label; Example: “gauss” selects “Gaussian Blur”.
3. the searched keyword appears anywhere else in the Action Label; Example: “lur” selects “Gaussian Blur”
4. if and only if the entry text is more than 2 letters, the search keyword appears anywhere in the Action Description. Example: “averag” displays “Gaussian Blur” (because its tooltip is: “Performs an averaging of neighboring pixels with the normal distribution as weighting”).
5. if and only if the entry text is composed of several words (defined by separating spaces), each of the word appears somewhere in the label or description, whatever their order of occurrence. Example: “blur gaussian” selects “Gaussian Blur” even though the order of the words is reversed.

Note: when several results from the history are in the same “section”, their memorized number of calls is further used to weight the upper action. For instance the search “blur” would select both “Gaussian Blur”, “Selective Gaussian Blur” and “Tileable Blur…” in the same section “the searched keyword appears anywhere else in the Action Name”. Nevertheless if the history recorded that “Selective Gaussian Blur” had been run 20 times whereas “Tileable Blur” had been run only 10 times and “Gaussian Blur” 5 times, then “Selective Gaussian Blur” would be on top, followed by “Tileable Blur”, and finally “Gaussian Blur”. Additionally if 2 actions have the exact same number of calls, the last one called is considered more relevant.

If the entry text has a content (a “keyword”), after any action from (1) has been shown, the algorithm shows any action not in the history, for which, in this order:
1. if and only if the entry text is exactly 2 letters, the first letter of the keyword is the first letter of the first word of the Action Name; and the second letter of the keyword is the first letter of the second word of the Action Label.
2. the search keyword appears at the start of the Action Label;
3. the searched keyword appears anywhere else in the Action Label;
4. if and only if the entry text is more than 2 letters, the search keyword appears anywhere in the Action Description;
5. if and only if the entry text is composed of several words (defined by separating spaces), each of the word appears somewhere in the label or description, whatever their order of occurrence.
If the entry text has a content (a “keyword”), after any action from (1) and (2) has been shown, the algorithm shows any action (in history or not), for which:
1. individual letters in the search keywords appear in the same order, but with optional intermediate letters, in the Action Name.

This is called fuzzy search. It allows powerful research when you search some action whose name is well known. For instance if you were to convert your image to 16 bit floating point (gamma), you could search ‘1bfpg + Enter’ which would return only this response, because you know exactly what you search, and is a much faster process. Or searching “cdrgb” would return “Convert to Default RGB” as only response.

But it may also display totally irrelevant responses on common searches. This is why its result are always at the end of the list. For instance, when you are searching for color balance actions, you may begin to type “bala”, and you certainly don’t want to get “Set Brush Size to Default Value” being considered more relevant than “White Balance” or “Color Balance” actions.