Tk Source Code

View Ticket
Login
Ticket UUID: 21252a15905c0e3f4d7cbbd0153c0fbb744dec9b
Title: textDisp-35.1 and Tcl_SetVar2() usage impacting test performance
Type: Bug Version: trunk
Submitter: chrstphrchvz Created on: 2026-11-14 09:22:27
Subsystem: 86. Test Tools Assigned To: nobody
Priority: 5 Medium Severity: Minor
Status: Open Last Modified: 2026-11-18 16:58:13
Resolution: None Closed By: nobody
    Closed on:
Description:

I notice that textDisp.test, which has had several tests added recently, now takes much longer to run than it did only a few weeks ago. Using -DPURIFY and no sanitizers, textDisp.test from 00ba90d1 took 12 seconds, whereas for recent trunk 646b38c2 it takes about 135 seconds. (With sanitizers, it takes several minutes.) I have identified some issues partly responsible for this.

textDisp-35.1 adds a large amount of text to a text widget. (It is also responsible for the peak in memory usage over the entire test suite, at a few hundred megabytes.) However, that large amount of text is not removed after the test finishes, and that in turn is slowing down subsequent tests. I also notice that the test has always created and destroyed another text widget .t1, but has never used it; to me, that looks like a mistake. I have not found discussion of this test other than acknowledgment of it in [1499165] and at https://sourceforge.net/p/tcl/mailman/tcl-core/thread/56451161.8020100%40free.fr/?page=2#msg34639487, but I believe the test meant to use .t1 rather than .t (note that for turning debug on/off, it doesn't matter which text widget that is done from). By having textDisp-35.1 remove the text widget it uses, textDisp.test speeds up to about 85 seconds.

But I also found why the leftover text widget from textDisp-35.1 caused subsequent tests to slow down. That text widget continued to be redrawn, but with debugging turned back on. So the LOG() macro used Tcl_SetVar2() to append an element to a list for each line of text, and each time it did so, the string representation of that list was regenerated, even though it is not needed. By having LOG() instead use Tcl_SetVar2Ex() to avoid needlessly regenerating string representations, textDisp.test speeds up to about 81 seconds. See branch textdisp-faster-log.

The slowdown in later textDisp tests being due to the combination of these issues means that fixing both of them does not noticeably speed tests up any further versus fixing just one of them.

While trunk is the most affected by these issues due to the number of new tests it has, the causes and fixes described here also apply to 8.6 and 9.0.

User Comments: marc_culler (claiming to be Marc Culler) added on 2026-11-18 16:58:13:
Christopher, would it be OK with you if I merge the snapshot-test branch?

chrstphrchvz added on 2026-11-17 03:30:32:

Yes, I had used leaks -atExit. I also tried using malloc_history, but probably not with the best choice of options as it took a very long time and printed an enormous report. This was for a build without AddressSanitizer; if I remember correctly, the leaks command is incompatible with ASan, and better suited for Cocoa programs anyway than ASan’s own leak check. ASan can also greatly increase memory usage seen in a process monitor.


marc_culler (claiming to be Marc Culler) added on 2026-11-17 02:29:51:
When I build with -DPURIFY the memory usage is much lower.  But I do not
see it decrease after the peak.  It starts at 120MB then jumps to 270 MB
and never goes back down, even when the first run terminates.

Are you using Apple's leaks tool?

chrstphrchvz added on 2026-11-17 00:58:54:

I may have to check again, but what I thought I had observed using Tcl built without -DPURIFY is that it could hoard "freed" memory, and so usage might not decrease after wherever it peaked during tests.

I had found that the memory used at peak usage was still reachable, and matched Serhiy's explanation, where the two most common allocations were for about 50MB of TkTextLine instances and 50MB of TkTextSegment instances. And after running all tests on Aqua with -DPURIFY, the total memory leaked was less than 1MB.


serhiy.storchaka added on 2026-11-16 23:44:55:

It is textDisp-35.1: it leaves 1.5 million lines in .t and turns debug back on. Their heights are computed later in the background, and with debug on every line is appended to tk_textHeightCalc and tk_textInvalidateLine. 41.1 is just the first test that runs the event loop long enough. The peak of textDisp.test on X11 is 332 MB, 325 MB without 41.1, and 57 MB without 35.1. Deleting the lines in the cleanup of 35.1 should fix it.


marc_culler (claiming to be Marc Culler) added on 2026-11-16 23:35:35:
I don't think that textDisp-35.1 is responsible for the peak in memory
usage. I think it is textDisp-41.1.  I see memory usage at about 120MB
through the textDisp.test run until it gets to textDisp-41.1 when it jumps
to over 400MB and stays there through the rest of the run.  The memory size
does not go down even when the first batch of tests ends and the second
one starts up.  If I comment out textDisp-41.1 the memory usage stays
around 120MB for the entire run.

Christopher - do you have any idea why this might be happning?  Is it
a leak? The textDisp-41.1 test packs 900 Frames, each containing a 2x1
Text widget. It destroys the window in the cleanup.  If the 900 widget
pairs account for 280MB that would be about 300K for a frame and a 2x1
Text widget, which seems high to me.  But I don't know what else could
be using that memory.  Also, destroying all of those Frames and Texts
in the cleanup phase does not change the memory usage.

marc_culler (claiming to be Marc Culler) added on 2026-11-16 17:03:16:
Thanks Christopher.  I think I have addressed these issues in the
latest commit.

chrstphrchvz added on 2026-11-16 03:54:13:

I suggest either using Tcl_AttemptAlloc(), or assuming Tcl_Alloc() returns non-NULL for a positive size.

tkLayerBitmapContext uses RGBX pixels (since it is created with kGCImageAlphaNoneSkipLast); X is basically a padding byte. I am not aware of any guarantee of what the value might be for those X bytes and any non-pixel bytes at the end of a row.

linePixels calls update. Some update or update idletasks is still needed at least to avoid testverifysnapshot generating an error on my machine.


marc_culler (claiming to be Marc Culler) added on 2026-11-15 22:48:25:
Christopher: I have committed a branch snapshot-test which includes your
textdisp-faster-log branch and implements a new test command
testverifysnapshot that compares the bitmap data of the tkLayerBitmapContext
from before and after an operation by using memcmp. This is used in
textDisp-42.9. It also switches the font used in textDisp-42.8 to one which
actually has overlapping ligatures (Hoefler Text). It adds a comment and
changes the sentence in that test.

I realize that the comparison code could be optimized to avoid one copy
of the bitmap data, but I opted not to mess with that for now anyway.

What do you think?

marc_culler (claiming to be Marc Culler) added on 2026-11-15 02:58:36:
And "office" contains an ffi ligature.

marc_culler (claiming to be Marc Culler) added on 2026-11-15 02:49:56:
And "flows" contains an fl ligature.

chrstphrchvz added on 2026-11-14 23:12:23:

I neglected to mention that my observations were using Aqua (on an older machine running an unsupported macOS version).

Regarding calling testpixel repeatedly for an unmodified widget: testpixel has to get the window contents and convert them to an XImage each time it is called, so it likely would be much more efficient to instead use a single capture of the entire window (something equivalent to the "window" photo image format in TkImg).

The changes on branch bug-35a71bf4 do not seem to affect performance much; time taken for textDisp.test only changes by a few seconds.


serhiy.storchaka added on 2026-11-14 22:16:42:

On Linux (X11) textDisp.test is much faster: about 3 seconds under Xvfb (also with -DPURIFY) and about 5.5 seconds on a real display. The leftover text of textDisp-35.1 is visible here too: with textDisp-35.1 using .t1, the tests after it take 1.2 instead of 1.8 seconds on a real display. I agree that the test was meant to use .t1.

textDisp-42.1 to 42.9 came with the patches of Nicolas Bats ([7d90e42b80]). textDisp-42.9 is run only on Aqua, so its cost is not visible elsewhere. Yes, "fine office affairs" contains the ligatures fi, ffi and ff; textDisp-42.8 checks that the block cursor is drawn correctly over them.

The space on the left is the frame .f; see the comment before its creation at the start of textDisp.test.


marc_culler (claiming to be Marc Culler) added on 2026-11-14 20:05:02:
I have noticed this too.  Thank you for analyzing this.

Your branch reduces the time for a standard run of textDisp.test
(for both Deployment and Development) to 35.6 seconds from 50.1s
on my macbook Air M3.

Some other things:

textDisp-34.1 creates a window wider than my screen and it remains
visible for a long time.  I would suggest adding
    wm geometry . $ge
in the cleanup section for textDisp-34.1.

I see textDisp-41.1 and textDisp-42.9 taking a very long time.  I think
it is intentional and probably unavoidable for 41.1 but 42.9 seems to
be using a Tcl proc to call testpixel for all 5360 pixels in a text
widget.  It seems like it should be possible to speed up verifying that
the contents of a widget did not change, perhaps by adding a function
to tkText.c.

Also, I have no idea why the text widgets have so much space on the left,
and I am still scratching my head over the meaning of "English text flows
from left to right, of fine office affairs."  Is it just to force fi and
ff ligatures?