I get why Woynert wanted to still interpolate both alpha and RGB
channels here, avoiding some too harsh transition when using the brush
tool. But the algorithm felt overly complicated (unless I misunderstood
the reason). I think what we want is simply to use the mask value to
interpolate, which allows the alpha and color to smoothly transition in
parts where the brush designer was expecting transitionning, when used
in a paintbrush-type tool.
In particular the proposed algorithm was never using the layer[ALPHA]
value which felt quite weird.
Additionally to the algorithm update, this commit:
* Makes the composite mode/space and blend space explicitly immutable.
* Removes code from other composite modes, which was anyway just
copy-paste remnants from gimp:normal operation and add some
g_return_val_if_reached() call to protect these code paths.
* Fix a code path where the out array was never filled (i.e. it was
filled with random data).
* Fix another code path where we could make a division by zero (in my
case, it didn't crash, but I got nan values).
8dbedcf723