Compositing and layer modes in GIMP
This document describes the process of compositing layers and the layer modes in GIMP.
License
This is free documentation; you can modify and/or redistribute it according to the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the license, or (at your option) any later version.
About this document
This document was originally written by Henning Makholm and part of the XCF file format specification. Because the topics here are more general in the context of GIMP they have been moved into a separate document.
9. COMPOSITING AND LAYER MODES
This section describes the “flattening” process that GIMP applies when a multilayer image is displayed in the editor or exported to other image file formats. It is present for reference only; an XCF consumer may well elect to do something different with pixel data from the layers than flattening them.
Most XCF consumers will need to react to the layer mode property of each layer; such a reaction must be informed by knowledge of how the different layer modes affect the flattening process. In some applications it might be acceptable for an XCF consumer to refuse processing images with layer modes other than “Normal”, but such an application will probably not be considered fully XCF capable by its users.
In this section we consider primary color (or grayscale) intensities and alpha values for pixels to be real numbers ranging from 0.0 to 1.0. This makes many of the formulas easier; the reader is asked to keep in mind that a (linear) conversion from the integral 0..255 scale of the actual XCF scale is implied whenever data from the XCF file is mentioned.
Any practical implementation of the computations specified below may suffer rounding errors; this specification do not detail how these are to be handled. GIMP itself rounds values to an integral number of 255ths at many points in the computation. This specification does not specify exactly which these points are, and authors of XCF renderers that aim to reproduce the effects of GIMP’s flattening down to the least significant bits are referred to studying its source code.
In the description below, the variable letter “a” is used for alpha
values. The variable letter “r”, “g”, “b” are used for primary
intensities, “y” is used for grayscale intensities, and “i” is used
for color map indexed. The letter “c” is used for the complete
color information for a pixel; depending on the color mode of the
image that is either an (r,g,b)
triple, a y
, or a c
.
The flattening process works independently for each pixel in the canvas area. The description of some layer modes in the GIMP manual may give the impression that they involve filters that let pixels influence neighbor pixels, but that is not true.
This description does not attempt to preserve the color information for completely transparent pixels in a layer. If an application uses this color information, it should document explicitly how it behaves when transparent pixels from several different layers cover the same point of the canvas.
Flattening overview
This is how to compute the flattened result for a single pixel position (in theory, that is  efficient implementations will of course follow this procedure or an equivalent one for many pixels in parallel):

Initialize a “working pixel”
(a1,c1)
to be completely transparent (that is,a1=0.0
and the value ofc1
is immaterial). 
Do the following for each visible layer in the image, starting with the one that comes LAST in the master layer list:

Ignore the layer if it is the floating selection, or if it does not overlap the pixel position in question.

Let
(a2,c2)
be the pixel data for the layer at the pixel position in question. If the layer does not have an alpha channel, then seta1
to1.0
. 
If the layer is the one that the floating selection is attached to and the floating selection overlaps the pixel position in question, then do the following:

Let
(a3,c3)
be the pixel data for the floating selection layer at the pixel position in question. 
If there is a selection channel, then let
x
be its value at the pixel position in question, and seta3
toa3*x
. 
Let m3 be the layer mode of the floating selection.

Set
(a2,c2)
toCOMPOSITE(a2,c2, a3,c3,m3)
. TheCOMPOSITE
function is defined below.


If the layer has a layer mask and it is enabled, then let x be the value of the layer mask at the pixel position in question, and set
a2
toa2*x
. 
Let
m2
be the layer mode of the layer. 
If the layer is the bottommost visible layer (i.e., if it is the last visible layer in the master layer list) and m2 is not “Normal” or “Dissolve”, then set m2 to “Normal”.

Set
(a1,c1)
toCOMPOSITE(a1,c1, a2,c2,m2)
. TheCOMPOSITE
function is defined below.


If the flattened image is to be shown against a background of color
c0
, then actually visible pixel isCOMPOSITE(1.0, c0, a1, c1, Normal)
.
Note that unless all layers have mode Normal, it would give the
wrong result to start by initializing (a1,c1)
to (1.0,c0)
.
Helper functions
The following auxiliary functions are used in the definition of
COMPOSITE
below:
MIN(x1,...,xn) is the least value of x1...xn
MAX(x1,...,xn) is the largest value of x1...xn
MID(x1,...,xn) = (MIN(x1,...,xn)+MAX(x1,...,xn))/2
CLAMP(x) = if x < 0 then 0.0 else if x > 1 then 1.0 else x
BLEND(a1,x1, a2,x2) = (1k)*x1 + k*x2
where k = a2/(1(1a1)*(1a2))
Layer modes
This and the following sections define the COMPOSITE function used in the general flattening algorithm.
“Normal” mode for RGB or grayscale images is the usual mode of compositing in computer graphics with alpha channels. In indexed mode, the alpha value gets rounded to either 1.0 or 0.0 such that no colors outside the color map get produced:
COMPOSITE(a1,y1, a2,y2,Normal)
= ( 1(1a1)*(1a2), BLEND(a1,y1, a2,y2) )
COMPOSITE(a1,r1,g1,b1, a2,r2,g2,b2,Normal)
= ( 1(1a1)*(1a2), BLEND(a1,r1, a2,r2),
BLEND(a1,g1, a2,g2),
BLEND(a1,b1, a2,b2) )
COMPOSITE(a1,i1, a2,i2,Normal) = if a2 > 0.5 then (1.0,i2) else (a1,i1)
“Dissolve” mode corresponds to randomly dithering the alpha channel to
the set {0.0, 1.0}
:
COMPOSITE(a1,c1, a2,c2,Dissolve) = chose pseudorandomly between
(1.0,c2) with probability a2
(a1,c1) with probability 1a2
These two modes are the only ones that make sense for all of the RGB, grayscale and indexed color models. In the indexed color model, all layer modes except Dissolve are treated as Normal.
Most layer modes belong to the following group, which makes sense for RGB and grayscale images, but not for indexed ones:
COMPOSITE(a1,y2, a2,y2,m)
= ( a1, BLEND(a1,y1, MIN(a1,a2),f(y1,y2, m)) )
COMPOSITE(a1,r1,g1,b1, a2,r2,g2,b2,m)
= ( a1, BLEND(a1,r2, MIN(a1,a2),f(r1,r2, m)),
BLEND(a1,g1, MIN(a1,a2),f(g1,g2, m)),
BLEND(a1,b1, MIN(a1,a2),f(b1,g2, m)) )
when 3 <= m <= 10 or 15 <= m <= 21
.
The following table defines f(x1,x2,m)
:
Multiply: f(x1,x2, 3) = x1*x2
Screen: f(x1,x2, 4) = 1(1x1)*(1x2)
Overlay: f(x1,x2, 5) = (1x2)*x1^2 + x2*(1(1x2)^2)
Difference: f(x1,x2, 6) = if x1 > x2 then x1x2 else x2x1
Addition: f(x1,x2, 7) = CLAMP(x1+x2)
Subtract: f(x1,x2, 8) = CLAMP(x1x2)
Darken Only: f(x1,x2, 9) = MIN(x1,x2)
Lighten Only: f(x1,x2, 10) = MAX(x1,x2)
Divide: f(x1,x2, 15) = CLAMP(x1/x2)
Dodge: f(x1,x2, 16) = CLAMP(x1/(1x2))
Burn f(x1,x2, 17) = CLAMP(1(1x1)/x2)
Hard Light: f(x1,x2, 18) = if x2 < 0.5 then 2*x1*x2 else 12*(1x1)(1x2)
Soft Light: f(x1,x2, 19) = (1x2)*x1^2 + x2*(1(1x2)^2)
Grain Extract: f(x1,x2, 20) = CLAMP(x1x2+0.5)
Grain Merge: f(x1,x2, 21) = CLAMP(x1+x20.5)
Note that the “Overlay” and “Soft Light” modes have identical effects. In the “Divide”, “Dodge”, and “Burn” modes, division by zero should be considered to produce a number so large that CLAMP(x/0) = 1 unless x=0, in which case CLAMP(0/0) = 0.
The remaining four layer modes only make sense in the RGB color model; if the color mode of the image is grayscale or indexed they will be interpreted as Normal.
COMPOSITE(a1,r1,g1,b1, a2,r2,g2,b2,m)
= ( a1, BLEND(a1,r2, MIN(a1,a2),r0),
BLEND(a1,g1, MIN(a1,a2),g0),
BLEND(a1,b1, MIN(a1,a2),b0) )
where (r0,g0,b0) = h(r1,g1,b1, r2,g2,b2, m)
when 11 <= m <= 14
.
For defining these modes, we say that
(r,g,b) has the _hue_ of (r',g',b')
if r' = g' = b' and r >= g = b
or there exist p and q such that p>=0 and r=p*r'+q and b=p*b'+q and g=p*g'+q
(r,g,b) has the _value_ of (r',g',b')
if MAX(r,g,b) = MAX(r',g',b')
(r,g,b) has the _HSVsaturation_ of (r',g',b')
if r' = g' = b' = 0 and r = g = b
or MIN(r,g,b) = MAX(r,g,b)*MIN(r',g',b')/MAX(r',g',b')
(r,g,b) has the _luminosity_ of (r',g',b')
if MID(r,g,b) = MID(r',g',b')
(r,g,b) has the _HSLsaturation_ of (r',g',b')
if r' = g' = b' and r = g = b
or MAX(r,g,b)MIN(r,g,b) = MIN(MID(r,g,b),1MID(r,g,b)) *
(MAX(r',g',b')MIN(r',g',b'))/MIN(MID(r',g',b'),1MID(r',g',b'))
Mode 11: Hue (H of HSV)
h(r1,g1,b1, r2,g2,b2, 11) is
if r2=g2=b2 then (r1,g1,b1) unchanged
otherwise: the color that has
the hue of (r1,g2,b2)
the value of (r1,g1,b1)
the HSVsaturation of (r1,g1,b1)
Mode 12: Saturation (S of HSV)
h(r1,g1,b1, r2,g2,b2, 12) is the color that has
the hue of (r1,g1,b1)
the value of (r1,g1,b1)
the HSVsaturation of (r2,g2,b2)
Mode 13: Color (H and S of HSL)
h(r1,g1,b1, r2,g2,b2, 13) is the color that has
the hue of (r2,g2,b2)
the luminosity of (r1,g1,b1)
the HSLsaturation of (r2,g2,b2)
Mode 14: Value (V of HSV)
h(r1,g1,b1, r2,g2,b2, 14) is the color that has
the hue of (r1,g1,b1)
the value of (r2,g2,b2)
the HSVsaturation of (r1,g1,b1)