Tk Source Code

View Ticket
Login
Ticket UUID: d6e9b4db409b4800b2b01ba40d86168b8d0636be
Title: Image format SVG: memory overflow on big files
Type: Bug Version: trunk
Submitter: oehhar Created on: 2020-08-25 16:06:11
Subsystem: 41. Photo Images Assigned To: nobody
Priority: 5 Medium Severity: Minor
Status: Closed Last Modified: 2020-12-09 12:18:52
Resolution: Fixed Closed By: oehhar
    Closed on: 2020-12-09 12:18:52
Description:

Thanks to Paul Obermeier, the following bugs were observed and fixed:

  • When large files (like a big zip-file) are presented to the match function, a memory (and performance) issue arises, as the file is first completly loaded in the match function. The sense of the match function is to decide with a small number of bytes if it is not the image format of the driver. This is not acheved.

Checkin [79583944] within branch "tip-529-image-metadata" solves both issues.

The following line was at the initial version of the ticket but is a false alarm

User Comments: oehhar added on 2020-12-09 12:18:52:

Thanks, Francois, for the work.

A question to Paul Obermeier if this alternate solution is ok was not answered within 2 weeks.

The alternate Androwish solution is now merged with commit [44d8b4d9].

Thank you all, Harald


fvogel added on 2020-11-21 13:10:17:

I have digged into the code and have understood what's happening and why the error message is unexpected (at first sight) in imgSVGnano-5.2.

In fact this error message is correct. Since the format of the data was not given, each stringMatchProc is tried in turn in MatchStringFormat(), but none of them returns without error. No known image format is recognized. The default format is then tried, and returns the error from its parser.

All this is in fact what is expected.

To return a correct and meaningful error message in the test, I have specified -format svg in the test and everything is fine then. See [b625f93d|test imgSVGnano-5.2].


fvogel added on 2020-11-21 09:53:31:
I have added a non-regression test imgSVGnano-5.2. The important thing this test is ckecking is that image data not containing <svg and > now return an error upon reading as an svg image.

Note that the expected result of the test matches the current output but is not adequate. The returned error text should be improved.

oehhar added on 2020-11-16 13:31:15:

The alternate solution from Androwish is now available for review in the branch [bug-d6e9b4db_svg-match-big-file-from-androwish] starting with commit [f4d783e0].


oehhar added on 2020-11-16 09:09:14:

Francois, thank you for your contribution.

So, my next steps are:

  • Provide the Androwish solution as a branch
  • Ask Paul, if this is feasable for him

I see little possibility, that the upstream project accepts this patch. The project is end of life and only security patches are included.

I am personally very happy with tksvg.

Thank you, Harald


fvogel added on 2020-11-14 21:53:36:

I was prompted on the Tcl Core list as follows:

may I ask your opinion on the patch by Paul for bug "Image format SVG: memory overflow on big files" ?

https://core.tcl-lang.org/tk/tktview/d6e9b4db409b4800b2b0

The patch is in branch bug-d6e9b4db_svg-match-big-file.

The alternate solution by Christian Werner is here:

http://www.androwish.org/index.html/vinfo/a9be9dd0259e47f0?diff=1

Therefore here is my brain dump about this (not sure I understand everything! please bear with me...):

The bottom line is that writing a second svg parser looks like an unexpected idea when there is already one in the codebase. The fact the existing one loads the entire file before basic analysis regarding svg-correct format could (should?) be considered a bug of the existing parser rather than a justification for writing a new parser aside of the existing one.

The fact nanosvg is coming from upstream then would imply that:

  1. the bug should be fixed in that upstream zone
  1. we should implement a guard in our side of the code (in generic/tkImgSVGnano.c) to avoid issues until upstream provides the fix (i.e. use belt and suspenders measures).

So my opinion is that Christian Werner's suggestion should go in our Tk code, and the parser bug be fixed upstream. Paul Obermeier's proposal could be posted as a pull request at nanoSVG after some reformatting and factorization with existing code.


oehhar added on 2020-09-14 12:17:00:

Thank you, Francois, for the post.

The issue is as follows:

image create photo i1 -file big.zip

will fail with big performance impact and a tcl memory error on bin zip files.

The use case is the image browser application created by Paul, where whole directory trees are presented to the match function.

The issue is:

  • the match function of svg loads the whole file in memory and searches in the data for xml tags (<svg>,</svg>).

The proposed solution in branch [bug-d6e9b4db_svg-match-big-file] authored by Paul is:

  • to load a chunks of 4k data
  • to use an alternate parser in the match function, which stops much earlier, for example when there is binary data

I hope, it is clrearer now.

Thank you, Harald


fvogel added on 2020-09-14 11:58:38:
To be completely honest, I'm lost about what is the problem, what is the false alarm, what is the fix and so on. Deferring to those who understand the situation is my best bet, sorry.

oehhar added on 2020-09-14 10:13:25:

It is probably bad style to change the ticket title and text. But the false alarm about "circles1.svg" was removed. The main point is to not examine big (binary) files until the end.

The cherrypicked version of the patch is now in branch [bug-d6e9b4db_svg-match-big-file] starting with checkin [9afe0b7d] (based on trunk).

I am in contact with Paul to find an eventual test for the patch.

Otherwise, I would just merge it, as it is clearly a regession and a solution.

Any comments ?

Thank you, Harald


oehhar added on 2020-09-14 06:54:58:

Added a test case in trunk [a40ea6ef] for an image without any of the attributes "width", "height" or "viewbox". This test does not fail in tk trunk but fails in tksvg 0.3.

So, tk trunk is not vulnerable to the given issue. It already passed in release tk 8.3a3.

Thank you, Harald


oehhar added on 2020-09-13 11:06:48:

I want to summarize some private mails with Paul and Christian.

Why is "circles1.svg" not recognized by nanosvg ?

Paul: As far as I remeber, the reason is that none of the attributes "width", "height" or "viewbox" is defined. His new size parser handles this situation.

----

Is there an advantage of the solution by Christian or Paul:

Christian: I was affraid of the fact that there are now two XML parseres in play. So I just limmitted the file size. It would be ideal to have a pre-scanmode in nanosvg which only outputs the svg yes/no, height and width. But this would imply a big change.

Paul: I did not want to touch the nansvg code. So I wrote an own parser.

A no-go of the nanosvg parser is, that there is no stop in case of binary data. It was not created for the task to detect svg format.

----

My former posting was wrong, that the patch should go to trunk, not core-8-6-branch.


oehhar added on 2020-09-10 16:39:56:

Here is a test with the image "circle1.svg" with tcl8.6.10:

% image create photo i1 -file circles1.svg
couldn't recognize data in image file "circles1.svg"

The same test with current tk branch [tip-529-image-metadata] can successfully load and display the image.

% image create photo i1 -file circles1.svg
i1
% pack [label .l -image i1]

Note that Christian Werner has another solution for the big data issue and eventually also for circles1.svg: http://www.androwish.org/index.html/vinfo/a9be9dd0259e47f0?diff=1

I will try to backport the work of Paul to core-8.6-branch. I think, this is sensible work.

Maybe, Christian has also an advice on this ?

Thank you, Harald