For the same file, which was saving in ~50 secs in RLE (now ~12), and ~80 secs
in zlib. It now saves zlib-encoded XCF in ~35 secs on the same machine with 8
threads.
Contributed code was not wrong, far from it! But the whole memory management was
a bit on the hard-to-read side. For maintainability, I prefer some simpler code,
which contains a bit more allocation but it's reasonable and at least much less
indexes to play with.
Also instead of gegl_parallel_distribute(), then waiting for each set of batches
to entirely finish, I'm using a GThreadPool which I progressively fill with new
batch data. And finally I'm running the gegl_buffer_get() inside the threads (I
know that we have only reads on the buffer at this point; also GEGL has an
internal lock mechanism anyway).
These actually do not provide a huge additional speedup (compared to the initial
speedup by just going multi-threaded), most likely because the I/O are now the
bottleneck anyway (which is quite a good thing!).
- Coding style fixes.
- Removing gimp_gegl_num_processors unknown variable which didn't exist (coming
from another patch from suzu. See discussion in #4985).
- Removing the `cleanup()` attribute to keep our code as generic as possible
without specific compiler extensions. Just free memory manually as is expected
in C.
Testing on a 115MiB XCF file, saving went down (on my machine, as average on
saving a few times) from about 50 seconds to 15 seconds.
This first version is already pretty neat, though we can probably do even
better.
Instead of storing vectors as properties, they have their own structure, which
make them able to store and load all the usual and common properties of other
items. In other words, it makes XCF now able to store locks, color tags and
several selected paths.
Creates two new parasites to save the black point compensation status
and rendering intent simulation settings in GimpImage.
The parasites are saved and loaded as part of the image in the
.xcf file.
More safety checks for detecting broken xcf files, also based on examining
issue #8230.
After reading an offset where layer, channel, etc. data is stored, we
add a check to make sure that offset is not before where we read the
offset value. Because the data is always written after the offset that
points to it.
Improvements in loading broken xcf files, based on examining issue #8230.
Besides checking for a minimum width and height, GIMP also has a maximum
size we can and should check.
In the case of the image itself, we change invalid dimensions to a size of
1 in hope that the individual layers etc will have the correct size.
For layer, we will also try to go on, but for channel and layer mask, we
will give up.
GIMP could crash if the information regarding old path properties read
from XCF was incorrect. It did not check if xcf_old_path succeeded and
kept trying to load more paths even if the last one failed to load.
Instead we now stop loading paths as soon as that function fails.
In case we have a failure here we also try to skip to the next property
based on the size of the path property, in hopes that the only problem
was this property.
GIMP stopped trying to read the XCF as soon as an invalid parasite was
encountered. However, in this specific case only the parasite data is
invalid, while the rest of the image is not corrupt.
Instead of terminating when we see a corrupt parasite, we skip to the
offset after the parasite. This may still be corrupt, but we can handle
that correctly, see e.g. the XCF in bugzilla issue 685086, which was
the reason of some of the previous changes.
Additionally:
- We add some logging to make it easier to handle future issues in this
area.
- We add tests for a NULL parasite name, and for reading a different
amount of parasite data than we expected. In both cases we return
NULL instead of a parasite.
XCF 17 includes the new visibility locks and the ability to add position
and alpha locks on layer groups.
I am going to push the various commits implementing these different
features together which is why we gather them as a single XCF version.
Due to recent changes on master the linked layers concept doesn't exist
anymore. The conversion code checks whether all linked layers are valid.
However, text layers need special handling.
In the past updating the linked_layers wasn't needed when the layer pointer
of a text layer was changed. Now, that has changed because we parse that
list for valid layers. To fix this we update linked_layers in the same
way as already was done for selected_layers.
We now save and load layer and channel item sets. Only missing set types
are path ones, but the whole path item is just its own exception in the
XCF format, and adding support for it, while keeping compatibility with
older XCF seem like a small headache. I could do it, but I actually
wonder if it is worth it. Would people really need to store sets of
paths?
Also this commit finally gets rid of any remnant of the old item "link"
concept (I think), so we are getting close to merging the branch.
I cleaned many remaining places where the concept of linked item still
survived.
On loading an XCF file with linked items, we are now going to create a
named sets for the linked items, allowing people to easily select these
back if the relation was still needed.
We don't remove gimp_item_get_linked() yet and in particular, we don't
save stored items into XCF files. This will come in an upcoming change.
In all 3 cases:
- xcf_read_string allocates a new string into name.
- If an error is detected, we return early - name remains unused.
Therefore we add g_free (name) to these early returns to avoid a leak.
… and s/gimp_parasite_flags/gimp_parasite_get_flags/
It is better to have a consistent API and the fact is that most
getter/setter functions use the _get|set_ naming, but these didn't.
I wondered if this was such a great idea to rename these anyway because
even though we are breaking API in GIMP 3, is it the best idea to rename
when no functional change happened? After discussing with Wormnest, we
still agreed it was still better to go for consistency rather than
regret later (and be stuck with these names for many more years).
Also this fixes these warnings:
> [649/1205] Generating gimp-3.0.vapi with a custom command
> Gimp-3.0.gir:24162.7-24162.56: warning: Field `Gimp.Parasite.flags' conflicts with method of the same name
> Gimp-3.0.gir:24318.7-24318.52: warning: Field `Gimp.Parasite.name' conflicts with method of the same name
Allow guides with off-canvas position since we can now view and work
outside the canvas.
The guide deletion interaction does not change too much, except you
don't delete a guide because it's dropped off-canvas but off the display
viewport area instead.
The XCF format is technically unchanged as PROP_GUIDES was already
expecting an int32 coordinate (with max value at GIMP_MAX_IMAGE_SIZE,
way below uint32 max anyway). Yet our code and XCF docs was explicitly
asking to ignore negative coordinate guides, which makes a change in
behavior for the XCF parser, hence a new version XCF 15. Loading will
still ignore negative position guides (even also bigger than image
dimension guides now) for XCF 14 and below, but will now accept any
position for XCF 15 and above.
XMP metadata saved by GIMP 2.8.x or earlier can have duplicate tags
making the XMP data invalid. There's not much we can do without a
whole lot of processing and complicated code and even then no
guarantee we would catch everything.
Instead let's just try to improve the message to the user so they
will be more likely to understand what's going on.
Although I haven't been able to reproduce it, it is apparently
possible to get a Stack Overflow when loading xcf files with
presumably very large dimensions on Windows. From what
I'm reading Windows normally has a smaller stack size than
Linux, probably why it hasn't surfaced there.
Instead of allocating on the stack let's do a g_malloc0
combined with g_free.
This parameter is not even used because we check again the selected
drawables during the save code. Allowing to set to 0 is simpler, and
also it will allow to not have to allocate an object array for layers
when we want to avoid memory management (during crash handling).
This commit just changes our saving API (i.e. the GimpSaveProcedure
class) to take an array of drawables as argument instead of a single
drawable.
It actually doesn't matter much for exporting as the whole API seems
more or less bogus there and all formats plug-ins mostly care only
whether they will merge/flatten all visible layers (the selected ones
don't really matter) or if the format supports layers of some sort. It
may be worth later strengthening a bit this whole logics, and maybe
allow partial exports for instance.
As for saving, it was not even looking at the passed GimpDrawable either
and was simply re-querying the active layer anyway.
Note that I don't implement the multi-selection saving in XCF yet in
this commit. I only updated the API. The reason is that the current
commit won't be backportable to gimp-2-10 because it is an API break. On
the other hand, the code to save multi-selection can still be backported
even though the save() API will only pass a single drawable (as I said
anyway, this argument was mostly bogus until now, hence it doesn't
matter much for 2.10 logics).
In xcf_load_image(), suspend size updates for layer groups before
adding their sublayers, to avoid unnecessary intermediary size
updates. Resume size updates in reverse order, so that nested
groups are updated before their ancestors, after all layers have
been loaded.
When dropping a corrupted layer while loading an XCF, make sure to
clear the corresponding XcfInfo fields that point to it, so that
it's not erroneously used later, as we now continue loading the
image even after corrupted layers are encountered.
See, for example, issue #4643, for a corrputed XCF file affected by
this.
and in an attack of madness, changes almost all file plug-in
code to use GFile instead of filenames, which means passing
the GFile down to the bottom and get its filename at the very
end where it's actually needed.
It's an ancient concept from ancient times when we didn't have URIs
and only filenames (not to speak of GFile), and actually even from
before the ancient time before that ancient time when we first had
ones and zeros, and only had zeros.
and call it gimp_pdb_set_proc_icon(). Change icon registration code in
libgimp/ and app/ so it's now possible to register icons for temporary
procedures.
Turn all ID param specs into object param specs (e.g. GimpParamImageID
becomes GimpParamImage) and convert between IDs and objects in
gimpgpparams.c directly above the the wire protocol, so all of app/,
libgimp/ and plug-ins/ can deal directly with objects down to the
lowest level and not care about IDs.
Use the actual object param specs for procedure arguments and return
values again instead of a plain g_param_spec_object() and bring back
the none_ok parameter.
This implies changing the PDB type checking functions to work on pure
integers instead of IDs (one can't check whether object creation is
possible if performing that check requires the object to already
exist).
For example gimp_foo_is_valid() becomes gimp_foo_id_is_valid() and is
not involved in automatic object creation magic at the protocol
level. Added wrappers which still say gimp_foo_is_valid() and take the
respective objects.
Adapted all code, and it all becomes nicer and less convoluted, even
the generated PDB wrappers in app/ and libgimp/.
And always pass URIs to all file procedures, the ones what didn't
register as "handles remove" will only ever get local file:// URIs.
Change all file plug-ins (also legacy ones) to expect URIs instead
of filenames, and convert to local paths in the plug-in.
The wire protocol should now be almost 100% clean of non-UTF-8 strings.
on behalf of plug-in authors who have no style or can't type.
Instead, simply reject non-canonical procedure names and remove all
code that keeps aroud the original non-canonical shit just to pass it
back to the plug-in.
- Change the wire protocol's GPProcInstall to transmit the entire
information needed for constructing all GParamSpecs we use, don't
use GimpPDBArgType in GPProcInstall but an enum private to the wire
protocol plus the GParamSpec's GType name. Bump the wire protocol
version.
- Add gimpgpparamspecs.[ch] in both app/plug-in/ and libgimp/ which
take care of converting between GPParamDef and GParamSpec. They
share code as far as possible.
- Change pluginrc writing and parsing to re-use GPParamDef and the
utility functions from gimpgpparamspecs.
- Remove gimp_pdb_compat_param_spec() from app/pdb/gimp-pdb-compat.[ch],
the entire core uses proper GParamSpecs from the wire protocol now,
the whole file will follow down the drain once we use a GValue
representation on the wire too.
- In gimp_plug_in_handle_proc_install(), change the "run-mode"
parameter to a GParamSpecEnum(GIMP_TYPE_RUN_MODE) (if it is not
already an enum). and change all places in app/ to treat it as an
enum value.
- plug-ins: fix cml-explorer to register correctly, a typo in
"run-mode" was never noticed until now.
- Add gimpgpcompat.[ch] in libgimp to deal with all the transforms
between old-style wire communication and using GParamSpec and
GValue, it contains some functions that are subject to change or
even removal in the next steps.
- Change the libgimp GimpProcedure and GimpPlugIn in many ways to be
able to actually install procedures the new way.
- plug-ins: change goat-exercise to completely use the new GimpPlugIn
and GimpProcedure API, look here to see how plug-ins will look in
the future, of course subject to change until this is finished.
- Next: changing GPParam to transmit all information about a GValue.
Whereas normal layers of dimension 0x0 are definitely broken, group and
text layers depend on their contents, which will be able to resize the
layer appropriately and fix whatever rendering. This commit allows to
salvage such layers, hence make XCF loading even more resistant to
certain form of file corruption.
This commit (and the previous one) are not theoretical but the result of
discovering some old corrupted file, with an empty group of size 0x0
(saved by GIMP itself, because of some old bug). Rather than destroying
these layer groups, this just allows us to reopen them without any kind
of loss!
If the problem occurs at one of the first layer, we may end up loading
nearly nothing even though there may be a whole lot of other data in
good state. So instead keep track of the number of failed layer and
channel loading to still display an error popup at the end, because we
still need to alert the user something went wrong. Yet damages are
limited.
Also make sure that layer paths are fixed when some layers could not be
loaded, hence stored paths end up wrong.
Add "gboolean push_undo" parameters to gimp_image_parasite_attach()
and _detach() and use the API also from undo, instead of implementing
attaching/removing manually and forgetting about the signals.
Fixes updating of the image properties color profile page.
Fix xcf-load.c to correct out-of-range item offsets (simply set them
to 0), so XCF files that are broken this way can still be recovered.
This doesn't fix the original bug, just recovering the crash images.
The NULL terminator of the tile-offset array of dummy buffer-levels
is erroneously written as an int32, instead of an offset, even in
version-11+ XCFs, in which offsets are 64-bit.
Since the dummy levels aren't actually used by GIMP, we're going to
keep these fields as int32 as an exception, in order to remain
consistent with existing XCFs, and just add a comment in the code,
and update the docs. If we ever make use of the higher buffer
levels, we should change these fields to offsets, and bump the XCF
version.
We can cancel a file overwrite at the last second when closing the
stream by setting a cancelled cancellable. Current code was simply not
closing the stream, but this was not enough as overwriting was happening
anyway (probably when finalizing).
This will allow much safe saving process since we would not be
overwriting a previously sane XCF file when an error occurred (either in
our code or a memory error, or whatnot).
See also discussion in #2565.
Add xyY color space to the color spaces for sampling colors.
Also add code to xcf-load.c that makes sure the sample point loading
code handles unknown future GimpColorPickMode values (fall back to
PIXEL pick mode).
All babl formats now have a space equivalent to a color profile,
determining the format's primaries and TRCs. This commit makes GIMP
aware of this.
libgimp:
- enum GimpPrecision: rename GAMMA values to NON_LINEAR and keep GAMMA
as deprecated aliases, add PERCEPTUAL values so we now have LINEAR,
NON_LINEAR and PERCPTUAL for each encoding, matching the babl
encoding variants RGB, R'G'B' and R~G~B~.
- gimp_color_transform_can_gegl_copy() now returns TRUE if both
profiles can return a babl space, increasing the amount of fast babl
color conversions significantly.
- TODO: no solution yet for getting libgimp drawable proxy buffers in
the right format with space.
plug-ins:
- follow the GimpPrecision change.
- TODO: everything else unchanged and partly broken or sub-optimal,
like setting a new image's color profile too late.
app:
- add enum GimpTRCType { LINEAR, NON_LINEAR, PERCEPTUAL } as
replacement for all "linear" booleans.
- change gimp-babl functions to take babl spaces and GimpTRCType
parameters and support all sorts of new perceptual ~ formats.
- a lot of places changed in the early days of goat invasion didn't
take advantage of gimp-babl utility functions and constructed
formats manually. They all needed revisiting and many now use much
simpler code calling gimp-babl API.
- change gimp_babl_format_get_color_profile() to really extract a
newly allocated color profile from the format, and add
gimp_babl_get_builtin_color_profile() which does the same as
gimp_babl_format_get_color_profile() did before. Visited all callers
to decide whether they are looking for the format's actual profile,
or for one of the builtin profiles, simplifying code that only needs
builtin profiles.
- drawables have a new get_space_api(), get_linear() is now get_trc().
- images now have a "layer space" and an API to get it,
gimp_image_get_layer_format() returns formats in that space.
- an image's layer space is created from the image's color profile,
change gimpimage-color-profile to deal with that correctly
- change many babl_format() calls to babl_format_with_space() and take
the space from passed formats or drawables
- add function gimp_layer_fix_format_space() which replaces the
layer's buffer with one that has the image's layer format, but
doesn't change pixel values
- use gimp_layer_fix_format_space() to make sure layers loaded from
XCF and created by plug-ins have the right space when added to the
image, because it's impossible to always assign the right space upon
layer creation
- "assign color profile" and "discard color profile" now require use
of gimp_layer_fix_format_space() too because the profile is now
embedded in all formats via the space. Add
gimp_image_assign_color_profile() which does all that and call it
instead of a simple gimp_image_set_color_profile(), also from the
PDB set-color-profile functions, which are essentially "assign" and
"discard" calls.
- generally, make sure a new image's color profile is set before
adding layers to it, gimp_image_set_color_profile() is more than
before considered know-what-you-are-doing API.
- take special precaution in all places that call
gimp_drawable_convert_type(), we now must pass a new_profile from
all callers that convert layers within the same image (such as
image_convert_type, image_convert_precision), because the layer's
new space can't be determined from the image's layer format during
the call.
- change all "linear" properties to "trc", in all config objects like
for levels and curves, in the histogram, in the widgets. This results
in some GUI that now has three choices instead of two.
TODO: we might want to reduce that back to two later.
- keep "linear" boolean properties around as compat if needed for file
pasring, but always convert the parsed parsed boolean to
GimpTRCType.
- TODO: the image's "enable color management" switch is currently
broken, will fix that in another commit.
