Tk Source Code

View Ticket
Login
2026-10-08
12:26 • Pending ticket [685ac307]: Aqua: XCopyArea() broken for non-pixmap source (as used by Tktable) plus 4 other changes artifact: a2b61d84 user: serhiy.storchaka
2025-11-21
01:33 • Ticket [9108ce12] macOS : CoreGraphic improvements status still Open with 3 other changes artifact: 68723260 user: chrstphrchvz
2025-08-24
03:48 • New ticket [3a4310ee] An alternative to using the deprecated currentAppearance property. artifact: eb97bf5e user: chrstphrchvz
2024-07-08
21:58 • Closed ticket [3e9e82bc]: Aqua: draw to alternative backing store plus 8 other changes artifact: 47b0aad2 user: marc_culler
20:46 • Ticket [3e9e82bc]: 3 changes artifact: 4fc4c80b user: fvogel
2023-07-09
04:58 • Ticket [31c3db0c] crash of Tk (trunk) on macOS status still Open with 3 other changes artifact: 636701f5 user: chrstphrchvz
2023-04-04
22:36 • Ticket [310c74ec] ThemeChangedProc() can be called after Tk destroyed status still Open with 3 other changes artifact: c5a784bf user: chrstphrchvz
2022-10-06
20:30 • New ticket [eb80f601] Aqua: Handle file events in _runBackgroundLoop. artifact: 01d19178 user: chrstphrchvz
2022-09-24
03:39 • Ticket [3e9e82bc] Aqua: draw to alternative backing store status still Open with 3 other changes artifact: f312d9a0 user: chrstphrchvz
2022-09-17
20:51 • Ticket [3e9e82bc]: 4 changes artifact: b85c53e1 user: marc_culler
2022-09-16
03:50 • Ticket [3e9e82bc]: 4 changes artifact: d6a689e3 user: marc_culler
2022-09-15
23:52 • Ticket [3e9e82bc]: 4 changes artifact: f2812726 user: marc_culler
15:02 • Ticket [3e9e82bc]: 3 changes artifact: ea332ed0 user: marc_culler
14:33 • Ticket [3e9e82bc]: 4 changes artifact: 27f4411e user: marc_culler
12:30 • Ticket [3e9e82bc]: 3 changes artifact: c2ffc6db user: chrstphrchvz
04:59 • Ticket [3e9e82bc]: 4 changes artifact: 4b1f5d48 user: marc_culler
2022-09-14
15:14 • Ticket [3e9e82bc]: 3 changes artifact: 509ab8eb user: chrstphrchvz
2022-09-09
03:00 • Ticket [3e9e82bc]: 3 changes artifact: 279cfeff user: chrstphrchvz
2022-09-08
16:05 • Ticket [3e9e82bc]: 4 changes artifact: 96bd31bb user: marc_culler
15:19 • Ticket [3e9e82bc]: 4 changes artifact: 2a5b9052 user: marc_culler
07:01 • Ticket [3e9e82bc]: 3 changes artifact: 774bf265 user: chrstphrchvz
2022-09-07
04:12 • Ticket [3e9e82bc]: 3 changes artifact: d3885fb0 user: chrstphrchvz
2022-09-01
07:58 • Ticket [685ac307] Aqua: XCopyArea() broken for non-pixmap source (as used by Tktable) status still Open with 3 other changes artifact: b371f98f user: chrstphrchvz
01:09 • Ticket [de6ef827] imageName configure -gamma - has no effect on macOS status still Open with 3 other changes artifact: 2080a586 user: chrstphrchvz
2022-08-24
13:30 • Ticket [3e9e82bc] Aqua: draw to alternative backing store status still Open with 3 other changes artifact: a0228cd0 user: nab
2022-08-23
21:20 • Ticket [3e9e82bc]: 3 changes artifact: 988a224a user: chrstphrchvz
10:48 • Ticket [3e9e82bc]: 3 changes artifact: b12ff70a user: nab
2022-08-22
15:58 • Ticket [3e9e82bc]: 3 changes artifact: 609ed7e2 user: nab
2022-08-18
17:45 • Ticket [3e9e82bc]: 3 changes artifact: b53bd1d2 user: chrstphrchvz
2022-08-17
11:35 • Ticket [3e9e82bc]: 3 changes artifact: 13293417 user: chrstphrchvz
02:00 • Ticket [3e9e82bc]: 3 changes artifact: ce648f66 user: kevin_walzer
2022-08-16
15:12 • Ticket [3e9e82bc]: 4 changes artifact: cb099d5e user: marc_culler
2022-08-15
15:48 • Ticket [3e9e82bc]: 3 changes artifact: 719a978f user: chrstphrchvz
15:38 • Ticket [3e9e82bc]: 3 changes artifact: 105f36ee user: chrstphrchvz
14:15 • Ticket [3e9e82bc]: 3 changes artifact: 449241a0 user: kevin_walzer
13:41 • Ticket [3e9e82bc]: 4 changes artifact: 039864a5 user: marc_culler
12:14 • Ticket [3e9e82bc]: 3 changes artifact: 8792da4d user: kevin_walzer
2022-08-14
14:11 • Ticket [3e9e82bc]: 4 changes artifact: 5b006d5d user: marc_culler
13:29 • Ticket [3e9e82bc]: 4 changes artifact: dc899e3c user: marc_culler
12:52 • Ticket [3e9e82bc]: 3 changes artifact: 153aa5a7 user: chrstphrchvz
12:45 • Ticket [3e9e82bc]: 4 changes artifact: 5b2ade73 user: marc_culler
12:33 • Ticket [3e9e82bc]: 4 changes artifact: e0af4d57 user: marc_culler
2022-08-10
13:31 • Ticket [3e9e82bc]: 3 changes artifact: 3b82285b user: chrstphrchvz
2022-08-09
22:02 • Ticket [3e9e82bc]: 3 changes artifact: ecd12a46 user: chrstphrchvz
2022-08-08
13:38 • Ticket [3e9e82bc]: 4 changes artifact: 4362adbc user: marc_culler
13:00 • Ticket [3e9e82bc]: 3 changes artifact: 988c6dc6 user: kevin_walzer
2022-08-07
22:48 • Ticket [3e9e82bc]: 4 changes artifact: e25e2406 user: chrstphrchvz
09:01 • Ticket [aad0231f] On macOS, DisplayText is called twice (inside and outside of drawRect) status still Open with 3 other changes artifact: 67ab7473 user: chrstphrchvz
08:56 • New ticket [3e9e82bc] Aqua: draw to alternative backing store. artifact: 8e9f15e1 user: chrstphrchvz

Ticket UUID: 3e9e82bcfe4b3a2b660724f682a3002670b220a5
Title: Aqua: draw to alternative backing store
Type: Patch Version: core-8-6-branch
Submitter: chrstphrchvz Created on: 2022-08-07 08:56:46
Subsystem: 81. Portability Assigned To: marc_culler
Priority: 5 Medium Severity: Important
Status: Closed Last Modified: 2024-07-08 21:58:34
Resolution: Accepted Closed By: marc_culler
    Closed on: 2024-07-08 21:58:34
Description:

(Something I worked on back in February-March 2022 but am just now getting around to refreshing with changes from the less_test_constraints branches and sharing here.)

After noticing issues like [4994f5dba4] (and more recently [aad0231f07]), I revisited seeing whether Tk Aqua can use its own backing stores for drawing, as opposed to ones used internally by Cocoa (which Apple doesn’t want apps reading from or synchronously writing to, because doing so violates Model-View-Controller design philosophy). The goal is to enable drawing behavior more like that of Tk on other platforms: the ability to draw whenever something wants to (rather than only when in drawRect:), and the ability to retrieve backing store contents without redrawing.

I had previously tried using an NSImage for this idea (see comments on [685ac30727] from August 2020), but found performance to be very poor. I’ve since learned that this is due to significant overhead when contents of an NSImage are pushed to the GPU for displaying (see https://sean.voisen.org/blog/2013/04/high-performance-image-loading-os-x/).

I have now tried again instead using CGBitmapContext/CGImage, and performance is much better than with NSImage, and close to that of the current approach. But I still notice instances where it is somewhat slower, at least on a slow machine. Running the pendulum widget demo on my MacBook Pro 2010 with Core 2 Duo underclocked to 0.8GHz running patched macOS 10.15.7: according to Quartz Debug, the current approach runs smoothly at 60fps whereas the CGBitmapContext/CGImage approach runs only at about 40fps. I had hoped that performance would possibly be better than the current approach, as it should avoid layout and redraw routines having to run twice (once outside of drawRect: and then a second time inside drawRect:), but it seems to have to spend even more time doing image copy/colorspace conversion; this is possibly a known problem (there is/was an ongoing investigation into this for the Cairo project’s implementation of surfaces on Quartz). I have also seen conflicting discussions of whether CGLayer is a good idea for improving performance (supposedly by further avoiding transfers to GPU), but I have not used it successfully.

The work I have is at https://github.com/chrstphrchvz/tk/tree/cgimage-drawing and is based on core-8-6-branch. It is currently a cargo-culted proof-of-concept; there are or were issues with widgets not drawing when expected (such as right after a toplevel is created), though I think these could partly be due to leaving in place various tricks needed by the current approach.

Probably the main difference this leaves from drawing on other platforms is that a Mac-specific event loop still needs to be entered periodically so that drawn changes appear onscreen (using updateLayer:). Although I hope this approach might help stabilize/future-proof Tk Aqua, I would understand if there is hesitancy toward adopting it. It may involve a significant amount of code churn and/or break user code with special handling for the current approach.

Any thoughts?

User Comments: marc_culler (claiming to be Marc Culler) added on 2024-07-08 21:58:34:
Yes, I think so.

fvogel added on 2024-07-08 20:46:59:

An implementation based on Christopher's idea and code was merged by Marc in [1dfe4fe9]. Can we close this ticket?


chrstphrchvz added on 2022-09-24 03:39:45:

The Ttk background color is coming from GetBackgroundColor() (ttkMacOSXTheme.c) and does not seem to be cached. In [cc2173797876] there was nothing setting the desired appearance before calls to NSColor methods in this or other functions in ttkMacOSXTheme.c (whereas in [de6a3246176d] the desired appearance is set beforehand in TkMacOSXSetupDrawingContext()).

It appears that the important step requiring the appearance to be properly set is whenever components must be generated for a catalog (e.g. semantic) NSColor, such as when a CGColor is retrieved from it, or when it is converted to a component-based NSColor (e.g. by colorUsingColorSpace:).

I have implemented an alternative approach which replaces any direct calls to appearance-dependent NSColor methods with wrappers that call those methods either within performAsCurrentDrawingAppearance: or with NSAppearance.currentAppearance temporarily set, depending OS version. I find that light/dark appearance works as expected on macOS ≥11 without any reliance on the deprecated NSAppearance.currentAppearance property. See https://github.com/chrstphrchvz/tk/compare/89f4970ecc45...c0ac783dde07


marc_culler (claiming to be Marc Culler) added on 2022-09-17 20:51:12:
The ttk background issue is now resolved in the chavez_cgimage_drawing
branch.  The one line synopsis is that due to an Apple bug, TkMacOSXSetupDrawingContext needs to run this command (on 12.5 at least):

NSAppearance.currentAppearance = view.effectiveAppearance;

That line is responsible for the only deprecation warning that remains
for that branch.  There are more details in the comments.

marc_culler (claiming to be Marc Culler) added on 2022-09-16 03:50:11:
I am wrong.  The empty clipping region is a red herring.  Those do occur,
and they should occur, but they are not the problem. The empty clipping
region belongs to the toplevel, because it is completely filled by its
children.

Ttk frames, labels and labelframes look up the correct color to use for
their background color.  But they do not use that color.  Tk frames, labels
and labelframes look up the same color and they use it correctly.  I still
have no explanation.  The behavior is as if ttk is caching the window
background color and using the cached color even after the appearance
changes.  But I see no evidence that any such caching takes place.  When
a ttk::frame is being displayed there must be some place in the code where
the fill color is set in the drawing context.  But I haven't been able to
find it.

marc_culler (claiming to be Marc Culler) added on 2022-09-15 23:52:50:
The ttk.Frames are not being drawn, although their display proc does
get called.  When the display proc calls XFillRectangles with 1 rectangle
which is the bounding box for the frame, XFillRectangles fails to obtain
a drawing context.  The failure in TkMacOSXSetupDrawingContext occurs in
this snippet:

    if (dc.clipRgn && HIShapeIsEmpty(dc.clipRgn)) {
    	canDraw = false;
    	goto end;
    }

This failure occurs often.  For example, if a widget demo window is moved,
there is one of these empty clipRgns for each redraw.

The appropriate windowBackgroundColor is being looked up correctly, but the
ttk frame does not get filled with it due to the failure of XFillRectangles.

marc_culler added on 2022-09-15 15:02:31:
I think monterey_deprecations is broken for 10.14 and 10.15.  I am
attempting a fix.

[NSApp currentAppearance] is used in TkMacOSXInDarkMode which is used
heavily in ttkMacOSXTheme.c.

marc_culler (claiming to be Marc Culler) added on 2022-09-15 14:33:31:
  • tkMacOSXDialog.c does not compile on 10.14 and 10.15 because UTType is unavailable.

Is this an issue with monterey_deprecations, or just my attempted merge of that branch into chavz_cgimage_drawing?


chrstphrchvz added on 2022-09-15 12:30:34:

Ignore my earlier remarks about rearranging code between TkMacOSXSetupDrawingContext()/TkMacOSXRestoreDrawingContext() for now, as there are cases outside of that to consider.

When the system appearance is changed the ttk frames do not get redrawn and keep the background color for the old appearance.

I don’t think it is a matter of them being redrawn. The widget demo status bar redraws itself with the old background color every time the status text changes. Newly created Ttk widgets will also have the old appearance.

I encountered a very similar issue when creating the earlier fix, and I want to say it was because the Ttk widgets rely on calls to TkpGetColor() outside of Aqua drawing routines which take the else if (SetCGColorComponents(…)) { … case, and that the fix was to move the appearance-setting workaround so that it included the call to SetCGColorComponents() and not just the GetRGBA() call in the if (entry->type == semantic) { … case. This was reverted in the merge check-in, but my quick attempt to restore it has not helped.

I’ve not made sense of all the appearance-related changes from the monterey_deprecations branch. My understanding of performAsCurrentDrawingAppearance: is that it basically replaces the old workaround, such that code for macOS 10.15 and earlier which does:

NSAppearance *savedAppearance = [NSAppearance currentAppearance];
[NSAppearance setCurrentAppearance:desiredAppearance];
 
// appearance-dependent code goes here
 
[NSAppearance setCurrentAppearance:savedAppearance];

can now instead do on macOS 11 and later:

// no need to save current appearance
 
[desiredAppearance performAsCurrentDrawingAppearance:^{
    // appearance-dependent code goes here
}];
 
// no need to restore current appearance

Using both of these where needed, I see no need to retrieve NSAppearance currentDrawingAppearance and so no need for the TKApplication currentAppearance method.

Other issues:

  • In TkpGetColor() it could be a problem that haveValidXColor = True; is missing from the else case after GetRGBA():
    		if (@available(macOS 10.14, *)) {
    		    …
    		} else {
    		    GetRGBA(entry, p.ulong, rgba, nil);
    		}
    
  • rgbaForAppearance() tries to call performAsCurrentDrawingAppearance: on macOS 10.14 and 10.15.

  • tkMacOSXDialog.c does not compile on 10.14 and 10.15 because UTType is unavailable.


marc_culler (claiming to be Marc Culler) added on 2022-09-15 04:59:17:
I think that the only thing that needs to be done inside the block is to
create an NSColor and extract its RGBA coordinates.

This is almost all working in the chavez_cgimage_drawing branch.  There is
still one problem.  When the system appearance is changed the ttk frames do not get redrawn and keep the background color for the old appearance.  I do
not know why this is happening.

chrstphrchvz added on 2022-09-14 15:14:15:

The fixes I used for light/dark mode switching ([513f6b2684b4]) conflict with changes in [92ffd3445939] on the monterey_deprecations branch. I haven’t thought of how performAsCurrentDrawingAppearance: can be used where needed other than by rearranging all existing code between TkMacOSXSetupDrawingContext()/TkMacOSXRestoreDrawingContext() calls into Objective-C blocks or at least separate C callback functions.


chrstphrchvz added on 2022-09-09 03:00:55:

Live resize redrawing issues seem to go away by reusing TKBackgroundLoop during live resize and having _runBackgroundLoop handle window events: https://github.com/chrstphrchvz/tk/commit/2c2e6a9bb2ae. There seems to be no need for it to do TkMacOSXDrawAllViews(NULL); things like animations will keep running, and I would also like file events to be handled here, since some Tk programs redisplay in response to a fileevent.

But crashes are currently possible with this simple approach, e.g. if a toplevel is destroyed while a child menu is posted.


marc_culler (claiming to be Marc Culler) added on 2022-09-08 16:05:27:
Actually, the unexpected behavior I am seeing without the clipping regions
seems to be limited to ttk widgets.

Ttk does lots of redundant drawing since it redraws a widget whenever its
state changes.  So, for example, whenever the mouse enters a ttk widget
the widget gets redrawn, even if its appearance is not affected by the state
change.  This tends to mask a lot of drawing artifacts, and create other
artifacts tied to user interaction.

marc_culler (claiming to be Marc Culler) added on 2022-09-08 15:19:44:
I am working in an orthogonal, but related, direction.  I am trying to
eliminate the clipping regions altogether.  What I am seeing surprises me.

There are two calls in tkMacOSXDraw.c which add the clipping regions to the
current graphics context.  One to CGContextEOClip (for complex regions) or
one to CGContextClipToRect (for rectangular regions).  I commented them out.

My understanding was that the generic Tk code should always draw widgets in
stacking order, and that redrawing a widget (say, a frame) should always be
followed by redrawing its children.  That is not what I am seeing.

For the widget demo, just the main window, I see the entire window being
drawn and then, after that is finished, the frame, scrollbar and status bar
get redrawn but the text widget is not redrawn.  Consequently the text
widget get covered with the frame background color and is not visible until
the window is scrolled.

I see similar things with other demos, e.g. in the ttk button demo I see
the labelframe getting redrawn while the buttons which are its children
do not get redrawn, causing the buttons to be covered with the labelfram
background color.

With the current setup, the children get clipped when drawing the frame
so they do not get covered up and they can be drawn completely independently
from the frame.  That should not be necessary if widgets are drawn in
stacking order, with children being redrawn wnenever the parent is redrawn.
Perhaps I misunderstand how these things work with Windows and X11?

chrstphrchvz added on 2022-09-08 07:01:44:

I did not get to the bottom of the redrawing issues during live resize, it seems to have something to do with TkMacOSXInvalClipRgns(). I think a much better solution might come from reusing TKBackgroundLoop and having it run during live resize, so that is what I am looking into.


chrstphrchvz added on 2022-09-07 04:12:03:

I occasionally encounter window-2.12 failing under CGImage-drawing, but I believe it is due to an existing issue: [310c74ecf440]


nab added on 2022-08-24 13:30:42:
in my code there's no pure update call, only update idletasks.
I think it's the same in tableList code

++

chrstphrchvz added on 2022-08-23 21:20:00:

Hello Nicolas, thanks for trying this.

I’m not completely certain how to interpret the spindump for the tablelist issue, aside from how it is always within XSync() during either update or update idletasks. I’m not sure what useful work might be occurring during XSync(), but this prompts me to notice that timer events are being handled (even during update idletasks—which I’m inclined to think is inconsistent with the documentation and other platforms’ behavior, so I have opened a separate ticket: [bfea11a91604]).

I would also not be surprised if TkDND has issues with this approach, since it is directly using Cocoa APIs.


nab added on 2022-08-23 10:48:56:
CPU results are here... on an Intel machine for a regular process in my app:
-with mac_styles_87 branch CPU (more or less)== 70%
-with tk-cgimage-drawing branch CPU (more or less)== 12%

wow !!

my toplevels are all created with 
::tk::unsupported::MacWindowStyle appearance $win darkaqua

and no issues here (with latest commit in tk-cgimage-drawing), I just seeing several steps when toplevels are deiconified (after beeing withdrawn).

there seems to be an issue (coloured wheel) when tableLIst is involved and should redisplay several rows.
attached what's reported:
 Thread 0x495e2    DispatchQueue "com.apple.main-thread"(1)    26 samples (1-26)    priority 0-47 (base 47)    cpu time 2.487s (9.6G cycles, 12.4G instructions, 0.77c/i)
  <process unclamped, process received importance donation from WindowServer [167], IO tier 0, priority 47 (47)>
  26  <truncated backtrace> 1-26
    26  main + 310 (DLight + 2298342) [0x10101f1e6] 1-26
      26  Tk_MainEx + 1238 (tkMain.c:376,5 in Tk + 174694) [0x10188da66] 1-26
        26  Tk_MainLoop + 43 (tkEvent.c:2109,2 in Tk + 117723) [0x10187fbdb] 1-26
          26  Tcl_DoOneEvent + 398 (Tcl + 924910) [0x1015c9cee] 1-26
            26  TclServiceIdle + 118 (Tcl + 1080710) [0x1015efd86] 1-26
              26  ??? (Tcl + 1083593) [0x1015f08c9] 1-26
                26  TclEvalObjEx + 139 (Tcl + 124363) [0x1015065cb] 1-26
                  26  ??? (Tcl + 131027) [0x101507fd3] 1-26
                    26  Tk_UpdateObjCmd + 168 (tkCmds.c:1213,9 in Tk + 86632) [0x101878268] 1-26
                      26  Tcl_DoOneEvent + 398 (Tcl + 924910) [0x1015c9cee] 1-26
                        26  TclServiceIdle + 118 (Tcl + 1080710) [0x1015efd86] 1-26
                          26  ??? (Tcl + 1083593) [0x1015f08c9] 1-26
                            26  TclEvalObjEx + 139 (Tcl + 124363) [0x1015065cb] 1-26
                              26  ??? (Tcl + 131027) [0x101507fd3] 1-26
                                26  Tk_UpdateObjCmd + 202 (tkCmds.c:1229,6 in Tk + 86666) [0x10187828a] 1-26
                                  26  XSync + 46 (tkMacOSXXStubs.c:716,12 in Tk + 945358) [0x101949cce] 1-26
                                    2   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 1-2
                                      2   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 1-2
                                        1   _pthread_mutex_firstfit_unlock_slow + 201 (libsystem_pthread.dylib + 7876) [0x7ff809ba7ec4] (running) 1
                                        1   _pthread_mutex_unlock_init_slow + 15 (libsystem_pthread.dylib + 7566) [0x7ff809ba7d8e] (running) 2
                                    1   Tcl_DoOneEvent + 376 (Tcl + 924888) [0x1015c9cd8] 3
                                      1   Tcl_ServiceEvent + 156 (Tcl + 924124) [0x1015c99dc] 3
                                        1   WindowEventProc + 37 (tkEvent.c:1702,6 in Tk + 116421) [0x10187f6c5] (running) 3
                                    1   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 4
                                      1   Tcl_ServiceEvent + 127 (Tcl + 924095) [0x1015c99bf] (running) 4
                                    2   Tcl_DoOneEvent + 376 (Tcl + 924888) [0x1015c9cd8] 5-6
                                      1   Tcl_ServiceEvent + 156 (Tcl + 924124) [0x1015c99dc] 5
                                        1   WindowEventProc + 8 (tkEvent.c:1698 in Tk + 116392) [0x10187f6a8] (running) 5
                                      1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 6
                                        1   _pthread_mutex_firstfit_unlock_slow + 74 (libsystem_pthread.dylib + 7749) [0x7ff809ba7e45] (running) 6
                                    1   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 7
                                      1   Tcl_ServiceEvent + 167 (Tcl + 924135) [0x1015c99e7] 7
                                        1   Tcl_MutexLock + 136 (Tcl + 1257736) [0x10161b108] 7
                                          1   _pthread_mutex_firstfit_lock_slow + 147 (libsystem_pthread.dylib + 7297) [0x7ff809ba7c81] (running) 7
                                    1   Tcl_DoOneEvent + 319 (Tcl + 924831) [0x1015c9c9f] 8
                                      1   ??? (Tcl + 1352838) [0x101632486] 8
                                        1   CFRunLoopRunSpecific + 449 (CoreFoundation + 515579) [0x7ff809c70dfb] 8
                                          1   __CFRunLoopDoObservers + 543 (CoreFoundation + 521186) [0x7ff809c723e2] 8
                                            1   __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23 (CoreFoundation + 521552) [0x7ff809c72550] 8
                                              1   write + 10 (libsystem_kernel.dylib + 18270) [0x7ff809b7275e] 8
                                               *1   hndl_unix_scall64 + 22 (kernel + 66102) [0xffffff8000220236] 8
                                                 *1   unix_syscall64 + 507 (kernel + 7836043) [0xffffff800098918b] 8
                                                   *1   write_nocancel + 174 (kernel + 6778670) [0xffffff8000886f2e] 8
                                                     *1   ??? (kernel + 6779095) [0xffffff80008870d7] 8
                                                       *1   ??? (kernel + 6796771) [0xffffff800088b5e3] (running) 8
                                    2   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 9-10
                                      2   Tcl_ServiceEvent + 167 (Tcl + 924135) [0x1015c99e7] 9-10
                                        1   Tcl_MutexLock + 136 (Tcl + 1257736) [0x10161b108] 9
                                          1   _pthread_mutex_firstfit_lock_slow + 147 (libsystem_pthread.dylib + 7297) [0x7ff809ba7c81] (running) 9
                                        1   _pthread_mutex_lock_init_slow + 0 (libsystem_pthread.dylib + 6446) [0x7ff809ba792e] (running) 10
                                    5   Tcl_DoOneEvent + 376 (Tcl + 924888) [0x1015c9cd8] 11-15
                                      2   Tcl_ServiceEvent + 156 (Tcl + 924124) [0x1015c99dc] 11-12
                                        2   WindowEventProc + 43 (tkEvent.c:1702,6 in Tk + 116427) [0x10187f6cb] 11-12
                                          2   Tcl_GetThreadData + 21 (Tcl + 1072309) [0x1015edcb5] 11-12
                                            1   pthread_getspecific + 0 (libsystem_pthread.dylib + 8149) [0x7ff809ba7fd5] (running) 11
                                            1   ??? (Tcl + 1077521) [0x1015ef111] (running) 12
  <priority 0 (47)>
                                      1   Tcl_ServiceEvent + 167 (Tcl + 924135) [0x1015c99e7] 13
                                        1   Tcl_MutexLock + 136 (Tcl + 1257736) [0x10161b108] 13
                                          1   _pthread_mutex_lock_init_slow + 6 (libsystem_pthread.dylib + 6452) [0x7ff809ba7934] (running) 13
  <priority 47 (47)>
                                      1   Tcl_ServiceEvent + 127 (Tcl + 924095) [0x1015c99bf] (running) 14
                                      1   Tcl_ServiceEvent + 167 (Tcl + 924135) [0x1015c99e7] 15
                                        1   Tcl_MutexLock + 136 (Tcl + 1257736) [0x10161b108] 15
                                          1   _pthread_mutex_firstfit_lock_slow + 147 (libsystem_pthread.dylib + 7297) [0x7ff809ba7c81] (running) 15
                                    6   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 16-21
                                      3   Tcl_ServiceEvent + 127 (Tcl + 924095) [0x1015c99bf] (running) 16-18
                                      1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 19
                                        1   _pthread_mutex_firstfit_unlock_slow + 243 (libsystem_pthread.dylib + 7918) [0x7ff809ba7eee] (running) 19
                                      2   Tcl_ServiceEvent + 127 (Tcl + 924095) [0x1015c99bf] (running) 20-21
                                    4   Tcl_DoOneEvent + 376 (Tcl + 924888) [0x1015c9cd8] 22-25
                                      1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 22
                                        1   _pthread_mutex_firstfit_unlock_slow + 190 (libsystem_pthread.dylib + 7865) [0x7ff809ba7eb9] (running) 22
                                      1   Tcl_ServiceEvent + 156 (Tcl + 924124) [0x1015c99dc] 23
                                        1   WindowEventProc + 43 (tkEvent.c:1702,6 in Tk + 116427) [0x10187f6cb] 23
                                          1   Tcl_GetThreadData + 62 (Tcl + 1072350) [0x1015edcde] (running) 23
                                      1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 24
                                        1   _pthread_mutex_firstfit_unlock_slow + 201 (libsystem_pthread.dylib + 7876) [0x7ff809ba7ec4] (running) 24
                                      1   Tcl_ServiceEvent + 204 (Tcl + 924172) [0x1015c9a0c] (running) 25
  <thread QoS unspecified (requested override user initiated)>
                                    1   Tcl_DoOneEvent + 174 (Tcl + 924686) [0x1015c9c0e] 26
                                      1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 26
                                        1   ??? [0] 26
                                          1   _sigtramp + 29 (libsystem_platform.dylib + 15869) [0x7ff809bc1dfd] 26
                                            1   TkMacOSXSignalHandler + 25 (tkMacOSXInit.c:419,5 in Tk + 844489) [0x1019312c9] 26
                                              1   Tcl_Exit + 231 (Tcl + 652231) [0x1015873c7] 26
                                                1   ??? (Tcl + 652741) [0x1015875c5] 26
                                                  1   TkFinalizeThread + 103 (tkEvent.c:2082,6 in Tk + 117527) [0x10187fb17] 26
                                                    1   DeleteWindowsExitProc + 218 (tkWindow.c:2898,8 in Tk + 242282) [0x10189e26a] 26
                                                      1   XSync + 46 (tkMacOSXXStubs.c:716,12 in Tk + 945358) [0x101949cce] 26
                                                        1   Tcl_DoOneEvent + 376 (Tcl + 924888) [0x1015c9cd8] 26
                                                          1   Tcl_ServiceEvent + 147 (Tcl + 924115) [0x1015c99d3] 26
                                                            1   _pthread_mutex_firstfit_unlock_slow + 6 (libsystem_pthread.dylib + 7681) [0x7ff809ba7e01] (running) 26


++

nab added on 2022-08-22 15:58:31:
Hi Christopher,
is there's anything I could test with my app that can interrest you?
I have access to both intel and M1 computer.

Also is there's a lot of tickets/patch from you that are not yet part of core-8-6-branch?

++
Nicolas

chrstphrchvz added on 2022-08-18 17:45:17:

See 1ef0e359494d for light/dark mode switching fixes. The necessary changes seem to be:

  • setting NSAppearance.currentAppearance when drawing, i.e. between TkMacOSXSetupDrawingContext() and TkMacOSXRestoreDrawingContext();
  • setting NSAppearance.currentAppearance when getting non-semantic colors from TkpGetColor() (which is not always called between TkMacOSXSetupDrawingContext() and TkMacOSXRestoreDrawingContext()), otherwise toplevels with ::tk::unsupported::MacWindowStyle appearance not set to auto may still have elements with incorrect colors; and
  • use generateExposeEvents: to force toplevels to redraw immediately when the appearance changes.
So far I do not notice any remaining issues with light/dark mode switching.


chrstphrchvz added on 2022-08-17 11:35:16:

I hadn’t encountered issues with light/dark mode earlier; at least on 10.15 Catalina, every toplevel would use whichever appearance Tk was launched with. I hadn’t tried changing the system appearance or a specific toplevel’s appearance with ::tk::unsupported::MacWindowStyle appearance, but doing so now I notice extensive issues: widgets not redrawing after appearance change, and nothing except for few widget elements (e.g. Ttk buttons) reflecting the new appearance, despite e.g. virtual events occurring.

Apparently this is because of Tk Aqua color functions which were depending on an NSView’s drawRect: method to influence the behavior of NSColor methods, as Apple would want (because in MVC terms, only the view should be concerned with the current appearance, and not anything in the model): “If you make appearance-sensitive changes outside of these methods, your app may not draw its content correctly for the current environment. The solution is to move your code into these methods.” The workaround, which is currently only used in TkpGetColor(), is to rely on the NSAppearance currentAppearance property (whose deprecation was announced in the macOS 11 Big Sur SDK).


kevin_walzer added on 2022-08-17 02:00:25:
Looking at the code and running the widget demo, I'm impressed. I won't pretend to understand every detail here, but I understand that by shifting to an explicit CoreGraphics-based drawing framework rather than Cocoa's built-in methods, we avoid the fragility that comes with hanging everything on Apple's undocumented API's that Marc must painstakingly reverse-engineer with each release of the OS. I note that this approach will allow us stop using the obsolete [NSViewscrollRect] method, which I think is great. 

The new approach thus far does break all of Marc's work on Dark Mode - Tk looks very strange when run in Dark Mode. Not sure how easy a fix there would be. 

Overall, very well done. I can't wait to see how it progresses.

marc_culler (claiming to be Marc Culler) added on 2022-08-16 15:12:00:
> I had disabled the event handling loop in `generateExposeEvents:` to
> avoid a panic in textDisp.test, but doing so has caused live resizing
> to stop working—the existing image just moves around (anchored to bottom
> left) without redrawing.

Strictly speaking, I don't consider that loop to be an event handling loop.
The only events that it handles are Expose events generated within the loop.
Using the Tcl event infrastructure to do this is really just a trick to
force every widget in the window to redraw itself.

Completely redrawing a window is an operation which need to be performed
at certain times, notably during a liveResize.  I don't see any reason
why it should be necessary to use any event handling to do this.  In any
case, I think it would make sense to have a self-contained function to
redraw the entire window in as an efficient manner as possible.

I am baffled by why the status bar sometimes disappears when resizing the
main window.  It would be good to understand this.

I re-enabled the loop in the test branch so we can try to figure out what
is going on.

Also, I have not seen any of the crashes that are referred to in the
comment.

chrstphrchvz added on 2022-08-15 15:48:48:

Marc, thanks for creating the branch for Kevin and others to try. To clarify, the cgimage-drawing branch I had on GitHub is based not directly on core-8-6-branch but rather my c86b-patches branch which has a number of fixes for other tickets which are awaiting review/merging. Unless those changes are merged sometime soon, I would suggest the cgimage drawing branch created here instead use the changes from https://github.com/chrstphrchvz/tk/compare/c86b-patches..cgimage-drawing.


chrstphrchvz added on 2022-08-15 15:38:23:

The other issues Marc described were likely resolved by recent changes I force-pushed (return 0 from TkpWillDrawWidget(), return True from TkTestLogDisplay(), do not handle events in generateExposeEvents:). The puzzle demo issue I believe was because the container frame wasn’t even being told to redraw (via RecomputePlacement() → Tk_MoveResizeWindow() → XMoveResizeWindow → MoveResizeWindow() → TkMacOSXInvalidateWindow() → TkMacOSXInvalidateViewRegion() → InvalViewRect()) when a contained widget moved; I fixed this by having InvalViewRect() manually call generateExposeEvents: instead of addTkDirtyRect: (since the latter now only triggers updateLayer which does not generate Expose events, unlike drawRect: → generateExposeEvents:).

(3) When resizing the Canvas item types demo the text label has black bars on the left and right sides.

This appears to be fixed but I do notice the black area may still appear briefly before the container redraws. I think this resembles what I sometimes see when using macOS’ bundled Tk 8.5.9 (which I only use on occasion for comparison).

(4) When scrolling the "text widget with subwindows" the buttons appear in the wrong place. (This looks like an artifact that I saw many years ago.)

This also appears to work but I do notice (at least on my slower Mac) that the redrawing of embedded windows lags behind the redrawing of text, something I also see (less noticeably) in Tk 8.5.9.

I had disabled the event handling loop in generateExposeEvents: to avoid a panic in textDisp.test, but doing so has caused live resizing to stop working—the existing image just moves around (anchored to bottom left) without redrawing. I’m not sure if it is caused by something incorrect elsewhere in the existing code or if it is a limitation of this approach. I would hope there is a better fix for this than reenabling the loop in generateExposeEvents: when inLiveResize is true; I don’t know if it’s possible to have whatever event loop that normally runs outside of live resizing keep running during live resizing. And even with live resizing there are issues with widgets not redrawing (e.g. the status bar at the bottom of the widget demo main window); maybe this also affects other programmatic resizing.

I would note that I currently do not observe [f16cdb6d04] under this approach.

The idea of whether the mask regions are redundant is interesting and not something I had already considered.


kevin_walzer added on 2022-08-15 14:15:36:
Thanks! I'm very intrigued by what Christopher is doing here and will take a look asap. Christopher, we appreciate your good work.

marc_culler (claiming to be Marc Culler) added on 2022-08-15 13:41:34:
I pushed a branch named chavez_cgimage_drawing which contains Christopher's
code. It always draws to a CGImage backing store that belongs to a
TKContentView.  When UpdateLayer is called the backing store is assigned
to the TKContentView's layer.content.  (And drawRect is never called.)

kevin_walzer added on 2022-08-15 12:14:56:
Are these changes committed to a Fossil branch? I'd love to review them.

marc_culler (claiming to be Marc Culler) added on 2022-08-14 14:11:37:
Christopher, your latest change also fixes the issue with the Text with Subwindows and the image viewer demos.  This is great progress.

marc_culler (claiming to be Marc Culler) added on 2022-08-14 13:29:11:
My current guess is that the problem with the 15-puzzle is caused by
masking. When a button moves into the "space" (which is done using
the "place" geometry manager) the previous location of the button should
get filled with the background color of the 4x4 frame. 

I suspect that the fill is actually happening, but it is happening before
the visRgn mask for the 4x4 frame has been updated.  So the square where the button used to be located is still being masked out by visRgn.  The buttons
are children of the frame, so all 15 of them should be masked out leaving
the "space" square as the only part of the frame where drawing is allowed.

Since this masking should not be necessary with the CGImage backing store,
perhaps the next step should be to disable all of the masking.

chrstphrchvz added on 2022-08-14 12:52:53:

Marc, thanks for trying this out. I will read and respond to your other comments, but want to quickly note that I pushed some changes last night (SHA 5a92b6a91801), which fixes the puzzle demo and allows all (automatic) unit tests to pass.


marc_culler (claiming to be Marc Culler) added on 2022-08-14 12:45:31:
I would like to mention that the idea of drawing into a CGImage backing
store may make it possible to remove two of the ugliest parts of the
current macOS Tk implementation.  They are:

(1) The hack of repurposing the Expose event in drawRect in order to force
a redraw of every widget that meets the dirty rectangle.  (Christopher's
code only calls generateExposeEvents when redrawing the entire window,
mainly during liveResize.

(2) The (expensive) maintenance of the masks visRgn, drawRgn and
aboveVisRgn. I believe that the purpose of these is to make it possible
to draw widgets in any order, as opposed to drawing them in stacking
order as is done for the other platforms.  The visRgn, for example, is
a rectangle covering the entire widget from which small rectangles for
each child widget have been removed.  Drawing through this mask will not
affect any child widget, allowing the children to be drawn before the
parent.  If drawing is done to a backing store managed by Tk then the
widgets should be drawn in stacking order and these masks should not be
necessary.

marc_culler (claiming to be Marc Culler) added on 2022-08-14 12:33:29:
I have been experimenting with Christopher's code and I think it looks very
promising.

There are four artifacts which I see in the widget demo that I think are
worth some effort to understand in order to see what might be missing.
They are:

(1) In the 15-puzzle the dark square for the missing piece does not get
    drawn.
(2) In the image viewer, when the image is changed, parts of the previous
    image frame remain, filled with parts of the new image.
(3) When resizing the Canvas item types demo the text label has black
    bars on the left and right sides.
(4) When scrolling the "text widget with subwindows" the buttons appear in
    the wrong place.  (This looks like an artifact that I saw many years
    ago.)

I am pretty sure that these issues have nothing to do with drawing into a
backing store CGImage.  Rather, they are being exposed because some
redundant drawing is no longer happening.  All of these artifacts
disappear if a redraw of the window is forced, say by a small resize.
I think the current Tk redraws the entire window frequently, when it
should not be necessary, and those redundant redraws erase these artifacts.

chrstphrchvz added on 2022-08-10 13:31:01:

… it seems to have to spend even more time doing image copy/colorspace conversion; this is possibly a known problem (there is/was an ongoing investigation into this for the Cairo project’s implementation of surfaces on Quartz)

See https://gitlab.freedesktop.org/cairo/cairo/-/issues/330. Using the display colorspace approach avoids the colorspace conversions and significantly improves performance: pendulum widget demo now runs at over 50fps on old MacBook Pro, although this is using a debug rather than release build. This approach may introduce slight color inaccuracies (e.g. when saving screen capture to an image), so it would be nice to leave in and document an easy way of disabling it (sacrificing performance for color accuracy).


chrstphrchvz added on 2022-08-09 22:02:39:

1. To my knowledge all windows on Aqua are now rendered with Cocoa-based backing stores - that API change is what has led Marc to the current implementation. Keeping up with Apple's constant API churn is difficult, but for now things seem to have stabilized. That is no small thing on the Mac platform. How will your implementation allow the Aqua version of Tk to avoid the private Cocoa-based backing store? I'm not clear on that.

The alternative approach may not completely avoid Cocoa’s internal backing stores/buffers/etc. (which likely are still present, given things like colorspace conversion happening), but it makes them irrelevant as far as Tk drawing routines are concerned.

Tk assumes it can draw to a backing store whenever it wants to and read from it whenever needed (as on X11), whereas Apple’s backing stores and APIs for them assume that any usage adheres to MVC (as espoused by seemingly every Cocoa programming tutorial, but the important part being that a properly designed app never needs to draw outside of drawRect: or retrieve what was already drawn from a backing store). In MVC terms, Tk must consider its backing store to be part of the model rather than strictly part of the view, so I believe Tk must rely on its own “backing store” for its needs rather than any Cocoa internal backing store.

If I were to think of concerns regarding the alternative approach’s resistance to future API revisions, I might start with how I’m not aware of these APIs being recommended by official documentation specifically for this purpose (particularly with regard to performance). Maybe Apple one day deems such usage unpopular enough (if not outright calls it another example of what programs ”shouldn’t be doing”) to support or keep performant. But I feel it requires a bit more pessimism to say so, and other cross-platform open source projects seem willing to adopt similar usage. Or, maybe the approach is fine, but that I should expect better results using Metal or other APIs not present on older macOS. There may continue to be disruptions with things like how Tk must emulate the AppKit event loop of ordinary Mac apps. The main advantage here in my view is, again, that the alternative drawing approach makes it clear in terms of MVC and API usage that Tk considers whatever it uses as a “backing store” to be part of the model rather than strictly part of the view, and so it shouldn’t be broken by API revisions which impose/enforce MVC more strongly (as happened circa 10.14 Mojave, which I find to Apple to have been within reason for doing rather than for the sake of churn or to break programs).

2. Your own admission that this new approach would likely involve significant re-work of the Aqua code base and could possibly break script-level Tk code does indeed give me pause. Do we want to go back to the stage where Tk under the Cocoa framework was a broken mess? It has taken years to get to this relative point of stability.

The significant rework would mainly involve combing through Aqua (and generic and Tcl syntax) code for anything that is only appropriate for the current approach; I expect the alternative approach to be simpler overall (that it would add relatively little new code and could potentially allow a good portion of existing code to be removed).

As for breaking user code, this is a worst case assumption I make mainly out of how little awareness I have of what workarounds are commonly used on Aqua, aside from needing things like update/update idletasks or a timer event (to wait until outside of drawRect: as done by bgerror) or maybe event generate … <Expose> (see [f16cdb6d04]) when not needed on X11 or Windows. My hope is that if these workarounds aren’t harmful when used on X11 or Windows then they won’t have a problem with this alternative approach. I have yet to bother trying the alternative approach with actual Tk apps, given the number of issues still present using widget demos; but if progress is made, I would suggest it at least be tried with the most popular Tk apps (IDLE, Gitk, Git-Gui, Thonny) and familiar maintained ones (Scid, SnapPy, TkChat, PortAuthority, D::Light).

3. I would strongly suggest than any work on this proof of concept be done only on 8.7. There is no need to introduce such sweeping changes in the Tk version that will be end-of-lifed at some point.

Is 8.6 really expected to be EOL’ed shortly after the release of 8.7.0? Depending how well things go with the alternative approach, I feel there may not be any point to maintaining the existing approach on 8.6 before it is EOL. Although I have libraries I would like to prepare for 8.7 prior to its release, I do not yet regularly use 8.7 on macOS.

I wonder whether it would make sense to use the image-based drawing whenever TkMacOSXSetupDrawingContext is called outside of drawRect. Rather than setting canDraw to False and returning in lines 1316-1319, perhaps it could construct a graphics context for a CGImage and copy that into the TkMacOSXDrawingContext which was passed to it. I am not sure how one would schedule copying the CGImage into the window, but I am guessing that Christopher knows that.

I’m not sure I understand this idea correctly, but it sounds like it is to extend the existing approach by caching/coalescing any drawing outside of drawRect: to a temporary image, do setNeedsDisplayInRect:, and then draw that image to the NSView while in drawRect:. It also sounds like it intends to keep the ability for drawing routines to draw directly while in drawRect:, but I wonder if the need for that is mostly eliminated by drawing routines already successfully drawing to the temporary image, and I think things can go wrong trying to mix drawing outside and inside of drawRect: such as over what’s already been drawn or what clipping regions are used. Even though this doesn’t sound impossible, I would say that I still don’t find using a temporary image this way desirable, because it again means forcing Tk to directly rely on Cocoa for a backing store with no way to retrieve its contents later, and possibly end up being more complex than both the current approach or the alternative approach I’m proposing. (Maybe there is another alternative approach whose only difference with the one I’ve proposed is that instead of setting layer contents to a CGImage in updateLayer, it draws the CGImage to the default layer of an NSView in drawRect:; Tk would not be relying on a Cocoa internal backing store, but I’m not aware why drawRect: should be kept around if all it is doing is drawing a single image unless doing so somehow happens to be far more performant.)


marc_culler (claiming to be Marc Culler) added on 2022-08-08 13:38:23:
This seems very interesting.

I wonder whether it would make sense to use the image-based drawing
whenever TkMacOSXSetupDrawingContext is called outside of drawRect.
Rather than setting canDraw to False and returning in lines 1316-1319,
perhaps it could construct a graphics context for a CGImage and copy
that into the TkMacOSXDrawingContext which was passed to it. I am not
sure how one would schedule copying the CGImage into the window, but
I am guessing that Christopher knows that.

kevin_walzer added on 2022-08-08 13:00:30:

Marc has done all the heavy lifting on the drawing code, so I would appreciate his input here, but I have a few questions/concerns:

1. To my knowledge all windows on Aqua are now rendered with Cocoa-based backing stores - that API change is what has led Marc to the current implementation. Keeping up with Apple's constant API churn is difficult, but for now things seem to have stabilized. That is no small thing on the Mac platform. How will your implementation allow the Aqua version of Tk to avoid the private Cocoa-based backing store? I'm not clear on that.

2. Your own admission that this new approach would likely involve significant re-work of the Aqua code base and could possibly break script-level Tk code does indeed give me pause. Do we want to go back to the stage where Tk under the Cocoa framework was a broken mess? It has taken years to get to this relative point of stability.

3. I would strongly suggest than any work on this proof of concept be done only on 8.7. There is no need to introduce such sweeping changes in the Tk version that will be end-of-lifed at some point.


chrstphrchvz added on 2022-08-07 22:48:54:

Retina support now added.