Mindstorm: n-point deformation tool

NOTE: this is a “mindstorm” document salvaged from our old wiki for a tool which is still in the “Playground” section of GIMP. It should eventually be transformed into a proper specification with final UX choices.

Introduction

This is a description of planned features of n-point deformation tool. It should lead to UI specification.

At the beginning, an image is opened and user selects n-point deformation tool. When he clicks on the image, mesh is automatically created above it. User can add points onto image and manipulate with them. He can also manipulate with mesh and control the deformation process – more below.

Mesh remarks

Mesh above the image is created automatically with use of alpha channel. When there isn’t any alpha channel, mesh is created above the whole image.

The deformation method uses two meshes. One is called ‘source pose’ mesh, the other is called ‘destination pose’ mesh. Source pose mesh is always hidden and destination pose mesh can be visible. Only destination pose mesh is deformed.

Mesh (lattice) for this method usually consists of squares. The deformation method is not limited to square mesh but square mesh can be constructed in an easier way, usually with less elements (than triangle mesh), and with small tweaks we can achieve similar precision as with triangular mesh.

For this method it holds that with decreasing lattice square size (i.e. increasing number of mesh elements), precision of result is increasing, but inefficiency of the algorithm is increasing as well. User should have an option to adjust the balance.

Control points manipulation

Add control points

With a click at some position on image, control point is added. User can add control points at arbitrary positions on image (except position some radius around it which depends on mesh square size or where a point is already placed) – mesh will be adjusted according to that.

Selection, deselection, multi-selection

More or less the same as sections with same names in Cage tool specification.

Delete control points

Selected control points are deleted by <delete> key. Last added point can be deleted by <backspace> key.

Deformation of the image

When user moves (click – drag – release) a control point (or multi-selected control points), the image is deformed.

Depth of control point

There should be an option to adjust which part of overlapping mesh/image is visible – e.g. we have an image of a person (as in the picture below) and we want to move her arm so that it is in front of (or behind) her body.

Every control point has its depth which affects a part of mesh under the control point.

User can change the depth using two buttons – one increases the depth, the other decreases it – or using two keys - which?.

When more control points are selected, every one’s depth is increased/decreased.

Mesh manipulation

Mesh visibility

In some situations it’s useful to see how the mesh looks like. User can toggle mesh visibility using a check button (or switch).

Mesh element size

User can choose the size of mesh element (square) using a slider – but only before he starts adding control points. It is because of the fact that when changing mesh square size, we have to recreate ‘source pose’ and ‘destination pose’ meshes. It’s often impossible to recreate ‘destination pose’ mesh so that it looks similarly to previous ‘destination pose’ mesh.

What would be a suitable name for this option? Density or precision?

Deformation control

Deformation algorithm control

The deformation algorithm iteratively solves a least squares problem described in the paper. During that process we periodically display current state of the solution and user can see that the deformation is evolving somehow. Usually hundreds of iterations are performed during 1 FPS. We don’t know how much time the solving will take and at the same time we often don’t want to find the final solution – we’re interested just in the process.

Hence there should be an option to temporary stop the process at some state or let it continue with solving further. How to do that in the best way? It could be done for example by <space> key.

Deformation termination

When user is satisfied with result, he presses <enter> key. Deformed image is redrawn using selected interpolation method. The deformation ends. When user wants to cancel the deformation, he presses <escape> key. Deformed image is discarded, original image appears on the canvas. The deformation ends.

Undo/redo

TODO

Settings

Deformation mode

The method allows 3 types of deformation:

• rigid (known as as-rigid-as-possible)
• similar (known as as-similar-as-possible)
• stretch (ARAP combined with weights from Moving Least Squares method)

‘Rigid’ type is very good for bending objects or manipulating articulated objects. It’s not so good in stretching or shrinking of objects. ‘Similar’ type allows changing of scale. Only translation, rotation and change of scale is included in deformation. It can be seen in the example. ‘Stretch’ type allows better stretching of objects. It can be seen in the example.

User can select the deformation type using drop down list or radio buttons.

Number of deformation iterations

User can select using slider, the number of iterations that are performed during 1 FPS. The larger the number the more rigid the deformation behaves.

How to call this option? It can be called for example rigidity or flexibility.

Type of interpolation

User can select the type of interpolation which is used when the deformation is terminated in favor of deformed image.

Types of interpolation:

• nearest neighbor (none)
• linear
• cubic
• sinc