Tk Source Code

On macOS, with the framework install using the GNUmakefile in the macosx
directory, installing Tcl 9.0 makes tclsh8.6 unable to find Tk and vice
versa.

I don't know if this affects other platforms or other installation methods.

It would be convenient, especially right now, to be able to have all
versions of tclsh usable at the same time.

How about making the Tk 9.X pkgIndex.tcl check whether it is being
called by Tcl 8.Y, and, if so, to not run "package if needed Tk 9.X"?

Wouldn't that make Tk 8.Y and 9.X independent from each other?  Isn't that
what we want?

> Why does Tcl 8.7 exit with an error when it reads a pkgIndex.tcl file for a version of Tk that has an incompatible stubs mechanism?

Tcl reads all pkgIndex.tcl files just fine. The problem occurs later. After reading all pkgIndex.tcl files, Tcl determines which is the best version to be loaded. It then inspects the "package ifneeded Tk <version>", and runs this. For tclsh8.7, the "package ifneeded Tk 9.0b4" gives:

    load /Library/Frameworks/Tk.framework/Versions/9.0/Tk

    load /Library/Frameworks/Tk.framework/Versions/9.0/Tk

The solution is to build Tk 9.0 with Tcl 8.7 headers, and install it as a new shared library `tcl8Tk`, in exactly the same way as the debug version of `Tk`, `Tk_debug` is built.

Proposed solution here

Demo:

$ tclsh8.7
% package require Tk 8
8.7b1
% info loaded
{/Library/Frameworks/Tk.framework/Versions/8.7/Tk Tk}
% exit
$ tclsh8.7
% package require Tk 9
version conflict for package "Tcl": have 8.7b1, need 8.7.0
% exit
$ tclsh8.7
% package require Tk 9
version conflict for package "Tcl": have 8.7b1, need 8.7.0
% info loaded
{/Library/Frameworks/Tk.framework/Versions/9.0/tcl8Tk Tk}
% exit
$ tclsh9.0
% package require Tk
9.0b4
% info loaded
{/Library/Frameworks/Tk.framework/Versions/9.0/Tk Tk}

Almost works ... just the "version conflict", which will disappear anyway when Tcl 8.7.0 is final.

Thanks, Jan.  That sounds good.

Here is the thing that I don't understand.  Why does Tcl 8.7 exit with an error
when it reads a pkgIndex.tcl file for a version of Tk that has an incompatible
stubs mechanism?  Why doesn't it just go on and read the next pkgIndex.tcl
file on its list, until it finds one that it can load?

> Yes, but in this commit, I added "8.6" to the path too. That doesn't seem to work .....

I reverted that now.

Now, if you install both Tk 8.7 (compiled with Tcl 8.7) and Tk 9.0 (compiled with Tcl 9.0), this is the effect:

192:tk8.7 jan$ tclsh8.7
% 
% package require Tk
interpreter uses an incompatible stubs mechanism

Since 8.7 is not going to be released anytime soon, there's still time to think how to handle that. For 8.6 and 9.0 everything works now as expected.

This must be what you are worried about:

> When Tk 8.7 goes final, a "package require Tk" will open Tk 8.7.
> I don't think that's intended

Do you mean that tclsh8.6 will open Tk 8.7?  I agree that seems unintended.
But why not just handle that with Tcl version restrictions in the Tk
pkgIndex.tcl?  The Tk 8.6 pkgIndex.tcl could just return if the Tcl
version is at least 8.7.

I assume it would be OK for tclsh8.7 to open Tk 8.7, right?

Is it supposed to be OK for Tcl 9 to open Tk 8.6?  I wouldn't think so.

I see that you were trying to deal with a hypothetical 8.7 issue which
has not arisen yet.  But I don't think I understand what that issue is.
Maybe you could add some details?

I did not notice that.  I would not expect that to work, though.

This is what I see with the commit preceding that one, after completely
removing my build directory and recompiling both versions of Tcl and Tk.

culler@abner ~ % tclsh8.6
% package require Tk
8.6.14
                                                                           culler@abner ~ % tclsh9.0
% package require Tk
9.0b3

That is what I was looking for.

Yes, but in this commit, I added "8.6" to the path too. That doesn't seem to work .....

No, I don't agree.  First of all, it was working for me with 8.6.

Second, if you look at library/auto.tcl you will find:

        foreach d $auto_path {
            lappend dirs [file join $d $basename$version]
            if {$tcl_platform(platform) eq "unix"
                    && $tcl_platform(os) eq "Darwin"} {
                # 4. On MacOSX, check the Resources/Scripts subdir too
                lappend dirs [file join $d $basename$version Resources Scripts]
            }
        }

So there is no need to add "Resources/Scripts" to the path.

My experiments show that if PATH is a path in the auto_path directory
then tclsh will find any pkgIndex.tcl file which matches the glob:

$PATH/*/Resources/Scripts

I don't think it works now, as it is in the core-8-6-branch. this appears to work better. Agreed?

Thanks, Jan.

I knew there was some black magic involved with autoconf!

As I recall, the reason I changed AC_PREREQ was because autoconf
complained about the version not being installed.  I don't remember
the details though.

Thanks!

I see 3 problems left.

1) AC_PREREQ is meant to specify the _minimum_ version of autoconf which is supported. For Tcl 8.6, this is 2.59, for 8.7 and 9.0 this is 2.69. We use a custom version of autoconf-2.59, because Tcl suffers from some bugs (which were all fixed in autoconf-2.69+)

2) When Tk 8.7 goes final, a "package require Tk" will open Tk 8.7. I don't think that's intended. Therefore, in Tcl 8.6 it's better to add "/8.6" to the path.

3) I'll leave that for another day.

1) and 2) are fixed now.

Oh.  I see now.  If exact is not true it uses 9.0b2.  So 9.0b3 should be OK.

Moreover it is OK!

When I recompiled and reinstalled everything both tclsh versions work correctly.
I guess I was just having the usual problem of reusing old build directories.

Sorry for making noise.

Maybe I will go ahead and close this ticket before I make any more noise.

Hi Jan,

OK.  You are right.  So why would Tcl 9.0b3 require exactly Tk 9.0b2
when Tk 9.0b3 exists?  Is that on purpose?

BTW, here is the change to tcl.h that makes everything work for me (after
recompiling both Tcl 9.0 and Tk 9.0):

 #   define Tcl_InitStubs(interp, version, exact) \
-	(Tcl_InitStubs)(interp, (((exact)&1) ? (version) : "9.0b2"), \
+	(Tcl_InitStubs)(interp, version, \
 	    (exact)|(TCL_MAJOR_VERSION<<8)|(TCL_MINOR_VERSION<<16), \
 	    TCL_STUB_MAGIC)

> ... because Tcl_Init_Stubs has been modified in tcl.h to force an exact version match for Tk 9.0b3.

That's not what Tcl_InitStubs() is doing! The current implementation just accepts the compiled extension to run with "9.0b2" and higher.

The change which I committed allows tclsh to find the pkgIndex.tcl files
for all installed versions of Tk.  But it does not work to allow all versions
of tclsh to load Tk.  It was working fine for me, but I forgot that I was
testing with a modified tcl.h file.

With the current tcl.h file for Tcl 9.0, you get this:

% tclsh9.0 
% package require Tk
interpreter uses an incompatible stubs mechanism

That error is being generated by Tcl_Init_Stubs and causes the package
require command to fail, without loading Tk, because Tcl_Init_Stubs has
been modified in tcl.h to force an exact version match for Tk 9.0b3.
The failure occurs when tclsh9.0 reads the pkgIndex.tcl file for Tk 8.6.

I don't understand why this exact match is being required.  But whatever
the reason, it should still be possible for tclsh9.0 to execute a
pkgIndex.tcl script for an incompatible version of Tk without failing
when a compatible version of Tk is, in fact, available.  If one version
of Tk is incompatible it should move on to the next one and not fail
until it has checked that all versions are incompatible.

I think this is a bug.  But a different bug.  The fact that it might go
away when there is a final release of Tcl / Tk 9.0 does not mean that
it is correct.

I said something completely wrong in one of my posts:

  The tclsh executable does not open the Tk library as a
  dynamic library, of course.  The problem is presumably that tclsh cannot
  find the correct pkgIndex.tcl file for its associated Tk.

In fact, tclsh *does* open the Tk library as a dynamic library.  That is
how extensions work, after all.

The load command is in the pkgIndex.tcl file. It attempts to build a path
to the Tk library from the value $dir where the variable dir is set before
running the pkgIndex.tcl script to the path of the directory containing
the pkgIndex.tcl script.  The path constructed by the load command in the
Tk pkgIndex.tcl file is:
 $dir/../../Tk
That means that if the pkgIndex.tcl script has path:
  /Library/Frameworks/Tk.framework/Versions/X.Y/Resources/Scripts/pkgIndex.tcl
then the Tcl load command will use dlopen to load the shared library:
  /Library/Frameworks/Tk.framework/Versions/X.Y/Tk

I found a simple 1-line fix for this.  The fix is to add one more directory
to TCL_PACKAGE_PATH in the Tcl unix/configure script.  The path that
needs to be added is:

 /Library/Frameworks/Tk.framework/Versions

(The fix would only add that for framework builds, which already
have a special TCL_PACKAGE_PATH).

Adding that path causes tclsh to search for the Tk pkgIndex.tcl file
in every directory matching the pattern:

 /Library/Frameworks/Tk.frameworks/Versions/*/Resources/Scripts

so it finds the pkgIndex.tcl for every (framework) installed version of Tk.

That unexpected search behavior is already built-in to framework builds
of Tcl.  The Tcl macosx/README file says:

  Given a potential package directory $pkg, Tcl on OSX checks for the file
  $pkg/Resources/Scripts/pkgIndex.tcl as well as the usual
  $pkg/pkgIndex.tcl. This allows building extensions as frameworks with
  all script files contained in the Resources/Scripts directory of the
  framework.

Two comments about that.  First, it seems that no one is using this feature
to package Tcl extensions as frameworks, but it does seem like a good idea.
Especially given that it solves this problem so neatly.

Second, one should be aware that a "potential package directory" does not
mean a directory in the TCL_PACKAGE_PATH.  It means a *subdirectory* of a
directory in TCL_PACKAGE_PATH.  That is what causes Tcl to check through
all of the installed Tk Versions, which is what is needed to fix this.

Here is one answer to my original question:

 "How can you make both tclsh8.6 and tclsh9.0 work at the same time on the same
  system?"

The answer is:

Create a ~/.tclshrc file containing two lines:
lappend auto_path /Library/Frameworks/Tk.framework/Versions/9.0/Resources/Scripts
lappend auto_path /Library/Frameworks/Tk.framework/Versions/8.6/Resources/Scripts


It seems like there should be a better fix.  But that does work.

I made some progress.

In the case where Tk 8.6 was the most recent, so tclsh9.0 was unable to find
Tk, I tried appending
 /Library/Frameworks/Tk.framework/Versions/9.0/Resources/Scripts
to the autopath list.  Then it was able to find Tk.

So perhaps the problem is that the Tcl auto_path is not being initialized
in a way which allows it to find its matching Tk.

But I don't understand how *any* tclsh is able to find *any* Tk.  The default
auto_path for tclsh9.0 is:

/Library/Frameworks/Tcl.framework/Versions/9.0/Resources/Scripts
/Library/Frameworks/Tcl.framework/Versions/9.0/Resources
/usr/local/lib
/Users/culler/Library/Tcl
/Library/Tcl
/Users/culler/Library/Frameworks
/Library/Frameworks

None of those contain a pkgIndex.tcl for the Tk package.  So tclsh must look
somewhere other than in its auto_path items.  And, wherever those other places
may be, they are not correct.

Here is one hint.  I installed Tk 9.0 first, then Tk 8.6.  That means that
tclsh8.6 can find Tk but tclsh9.0 can not.

I then changed the symlink /Library/Frameworks/Tk.framework/Versions/Current
to make it point to 9.0 instead of 8.6.

That had no effect.  It was still the case that tclsh9.0 could not find Tk.

I think that means that the Versions/Current symlink is not being used by tclsh.

Thanks Kevin.

Looking again, I see that the top level Resources/Scripts directory in the
Tk.framework is actually a symlink:

Resources -> Versions/Current/Resources

So maybe the issue is that all versions of tclsh are looking in the top
level Resources and therefore they all see only the most recently installed
version of Tk, which may not match.  Perhaps, instead of telling us that
they found a Tk which did not match, they just tell us that they could not
find Tk.  And of course they don't tell us where they were looking.  And
it seems that none of us know the answer to that question either.

Unfortunately, it is not as simple as I thought.  (As usual!)  There already
are Scripts directories both in the top level Resources and in
Versions/X.Y/Resources.

So I don't know why tclsh is not able to find the pkgIndex.tcl file for its
associated version of Tk.  Where is it looking?  Could it be trying to use
/Library/Frameworks/Tk.framework/Versions/Current ?
That symlink would change with each install of a different version of Tk.

Marc, go ahead and make the changes to configure.ac and generate a new configure with autoconf. I've done it before.

Here it is:

unix/configure.ac:    TK_LIBRARY="${libdir}/Resources/Scripts"
unix/configure.ac:    TK_PKG_DIR="Resources/Scripts"
unix/configure:    TK_LIBRARY="${libdir}/Resources/Scripts"
unix/configure:    TK_PKG_DIR="Resources/Scripts"

The actual mistake, I think, is this line in unix/configure:

TK_PKG_DIR="Resources/Scripts"

I think that should be

TK_PKG_DIR="Versions/${TK_VERSION}/Resources/Scripts"

I don't know what the rules are regarding changes to unix/configure.  Am I
allowed to edit configure.ac, generate a new configure with autoconf-2.72
and commit both configure.ac and configure?  Or is there some other black
magic involved.

> means that the Tcl shared library on MacOS doesn't have a version number
> attached. That means you cannot install any two of Tcl 8.5, 8.6, 8.7 and 9.0,
> then add the paths to those shared libraries to LD_LIBRARY_PATH, and then
> expect the system to be able to keep those apart

I don't think that is correct.  First of all, the framework install uses
the Version directory of Apple's framework bundle to distinguish the versions.
So the following different versions coexist in my framework:
/Library/Frameworks/Tcl.framework/Versions/8.6/Tcl
/Library/Frameworks/Tcl.framework/Versions/9.0/Tcl

Second, there is no LD_LIBRARY_PATH involved.  Apple does not use
that variable at all.  It does have a DYLD_LIBRARY_PATH but that is also
not involved here.  The Mach-O binary format specifies the library load
paths within the executable file.  In the case of tclsh these paths are:

for tclsh8.6: /Library/Frameworks/Tcl.framework/Versions/8.6/Tcl
for tclsh9.0: /Library/Frameworks/Tcl.framework/Versions/9.0/Tcl

So that is all correct. Moreover, the problem is not that tclsh cannot
find the Tcl library. Both versions of tclsh run fine as long as you do
not attempt to load the Tk package.  However, only one of them is able
to find Tk.  The tclsh executable does not open the Tk library as a
dynamic library, of course.  The problem is presumably that tclsh cannot
find the correct pkgIndex.tcl file for its associated Tk.

While your model of how dynamic libraries work on macOS is incorrect, I
think you have nonetheless explained the problem to me.  What I now see
is that, even though the Tk libraries are correctly separated by version
using the Versions directory in the framework, the same is not true for
the Tk Scripts directories.  There is only one Scripts directory, which
gets over-written each time that a version of Tk is installed.  It is:

/Library/Frameworks/Tk.framework/Resources/Scripts

There should be separate Scripts directories for each version of Tk:

/Library/Frameworks/Tk.framework/Versions/X.Y/Resources/Scripts

It is true that Apple recommends putting text files in the Resources
directory of a framework. But they provide for both version-independent
and version-dependent Resources.  Someone apparently decided that the Tk
Scripts were independent of Tk version and used the top level Resources
instead of having separate Scripts directories for different Tk versions.

So I guess all I have to do is grep for Scripts in the source tree and
figure out where that mistake actually got implemented.  I am betting
that the mistake was made about 20 years ago.

Thanks.

The first problem I see in the framework build of Tcl is here. The line

TCL_LIB_FILE="Tcl"

So, I would recommend to change this line to either

TCL_LIB_FILE="Tcl9"

TCL_LIB_FILE="Tcl9.0"

I don't know enough about how MacOS frameworks are supposed to work.

> I don't know if this affects other platforms or other installation methods.

This problem is MacOS only.