Tcl Source Code

Recent changes (somewhere since 8.6.10) in file  unix/configure.in  lines 871 and 873  cause extra braces to be added to the value of TCL_PACKAGE_PATH

These extra braces spoil installation of tcltls, which attempts to use the value of TCL_PACKAGE_PATH to determine install location.

At this point I cannot yet tell, if this is a regression of tcl, or if tcltls is doing something wrong by using TCL_PACKAGE_PATH from tclConfig.sh .

Tcl can also be rendered 100% dysfunctional by creating a .tclshrc file containing only "exit 1". TCLLIBPATH is documented to be a list, and if a user provides a value that is not a list, Tcl should produce an error, and it does. It would be trivial to make the current error a little more informative, but the interpreter should still exit if a bad value is provided. "Error early" is a thing, and it's a good thing. A syntax/formatting error in TCLLIBPATH (and TCL_PACKAGE_PATH) should be treated the same way that a syntax/formatting error in a script is treated. It is the user's bug when a value is documented to have a certain format but the user provides a value that is not in that format. Anyway, Jan's example really has no bearing on what the format of TCLLIBPATH should be. Likewise, if the format were changed to use delimiters, and there was an error in the implementation of the parser for those delimiters, it wouldn't reflect on the decision to adopt those delimiters as the new format. Also, if the behaviour Jan's example illustrates was actually a problem, it would have been fixed a long, long time ago.

Now try the rfe-3057b6261 branch:

$ export TCLLIBPATH=\{

$ tclsh9.1
% set auto_path
\{ //zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl /usr/lib
%

Just a simple demo, why using Tcl list syntax in environment variable is a bad idea. It means that the TCLLIBPATH environment variable can be used to make tclsh 100% disfunctional. Example:

$ export TCLLIBPATH=\{

$ tclsh
application-specific initialization failed: Can't find a usable init.tcl in the following directories:
    /usr/library /library /tcl8.6.17/library /tcl8.6.17/library

/usr/lib/tcl8.6/init.tcl: unmatched open brace in list
unmatched open brace in list
    while executing
"if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }"
    ("foreach" body line 2)
    invoked from within
"foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }
        }"
    (in namespace eval "::tcl" script line 4)
    invoked from within
"namespace eval tcl {
    if {![interp issafe]} {
        variable Dir
        foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::..."
    (file "/usr/lib/tcl8.6/init.tcl" line 55)
    invoked from within
"source /usr/lib/tcl8.6/init.tcl"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list source $tclfile]"
/usr/lib/tcl8.6/init.tcl: unmatched open brace in list
unmatched open brace in list
    while executing
"if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }"
    ("foreach" body line 2)
    invoked from within
"foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }
        }"
    (in namespace eval "::tcl" script line 4)
    invoked from within
"namespace eval tcl {
    if {![interp issafe]} {
        variable Dir
        foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::..."
    (file "/usr/lib/tcl8.6/init.tcl" line 55)
    invoked from within
"source /usr/lib/tcl8.6/init.tcl"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list source $tclfile]"
/lib/tcl8.6/init.tcl: unmatched open brace in list
unmatched open brace in list
    while executing
"if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }"
    ("foreach" body line 2)
    invoked from within
"foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::auto_path} {
                lappend ::auto_path $Dir
            }
        }"
    (in namespace eval "::tcl" script line 4)
    invoked from within
"namespace eval tcl {
    if {![interp issafe]} {
        variable Dir
        foreach Dir [list $::tcl_library [file dirname $::tcl_library]] {
            if {$Dir ni $::..."
    (file "/lib/tcl8.6/init.tcl" line 55)
    invoked from within
"source /lib/tcl8.6/init.tcl"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list source $tclfile]"

This probably means that Tcl wasn't installed properly.
%

Hi Harald, the main use case is consumption by or for a Tcl interpreter. TCL_PACKAGE_PATH and TCLLIBPATH are intended for consumption by a Tcl interpreter, so they really should be Tcl lists. It's backwards to modify TCL_PACKAGE_PATH, which is intended for consumption by a Tcl intepreter, so that it is in the native format of another language. Doing this complicates the task of reading it for a Tcl interpreter, and there will surely be future bug reports because of this complication. Since these variables happen to be exposed in the environment, non-Tcl programs have access to them, but they were intended to be passed through to a Tcl interpreter, not directly used in another language. Even so, in [3057b6261] I described how the Tcl list format can trivially be consumed by sh-family languages, so not only is the Tcl list format the correct format for values that are intended to be read by Tcl interpreters, but it is also a good standard format for consumption by other languages. The Tcl list format is in many cases actually more convenient for other languages, because these days a shell script like tclConfig.sh which was produced on one platform might for one reason or another be consumed on another platform. In this case, a Tcl list is more convenient than trying to keep track of all the item delimiters one might encounter out in the wild.

This change already "sort of broke" dgp's OOMMF package, and it caused some indigestion for some of my own scripts as well. There are many variables in tclConfig.sh that are intended for use in an sh script, and not intended for consumption by a Tcl interpreter. For many of those variables, the Tcl list format does not make sense, but for TCL_PACKAGE_PATH and TCLLIBPATH, the only format that makes good sense is the Tcl list format. The issue that occurred in the TCLTLS can just as easily be solved by having the related shell scripts treat TCL_PACKAGE_PATH as a Tcl list, which as I have already shown, they can easily do. It's not worth it to keep this broken "fix" in place just because it has been incorporated downstream in a few places.

It's easy to get short-sighted by being too oriented on a particular use case and fixing things for that use case, only for a conflicting use case to appear later. If the principal is that the Tcl list format should be used for structured values intended for consumption by a Tcl interpreter, further implementation complications and user issues will be avoided. A peek at the implementation of the "fix" for this report bears that out. If the "fix" for this issue isn't reverted, then [3057b6261] is probably eventually going to get resolved by applying it to TCLLIBPATH as well, which will probably cause even more complications and user issues.

Ashok wrote: >Tcl should be consistent in the expected syntax - either all should be expected to be in platform-native syntax, or all should be in Tcl syntax

Currently, all environment variables use native syntax, except `TCLLIBPATH`. That's what inspired me on RFE [3057b6261]. So, shall we make everything consistant, finally? This will need a TIP!

Dear Nathan,

thank you for your valuable comment. This was discussed within the ticket and changed and documented on purpose. Upstream packages like TCLTLS got adopted to this change.

Could you please give your use-case why this is in your valuable opinion not the right decission? I tend to be use-case only. The use-case is its use by a make file and other build tools and not by TCL. Due to that, a TCL format is inappropriate and an operating system close format is prefered. The referenced reasoning to get a platform independent format is IMHO not suited for the described use-case. This is platform dependent, what is the aim and ok.

Thank you and take care, Harald

Reopening because this was a bad and unnecessary fix. As explained in [3057b6261], A Tcl list is the best format for representing data in a platform-agnostic manner, and TCL_PACKAGE_PATH is exactly the sort of place where it should be used. The previous implementation of TCL_PACKAGE_PATH got this right.

Related RFE [3057b6261].

Note that in the "unchained" branch TCL_PACKAGE_PATH remains a Tcl list.

See also 0a0fae80cd, which is now fixed

The purpose of TCL_PACKAGE_PATH is to set tcl_pkgPath, which is documented in tclvars.n to be a list. TCL_PACKAGE_PATH is document in tclConfig.sh to be a list. It's been this way for decades. Anything picking it up from tclConfig.sh should take care to process it properly, probably using a Tcl interpreter. It's completely acceptable to require that a Tcl interepreter be on hand when building extensions. Since tcltls is the only package experiencing this issue there is clearly a way to avoid it without changing how TCL_PACKAGE_PATH works.

Let's look at the example given by Harald below:

${MAKE:-make} tcllibdir="${installdir}/lib" TCL_PACKAGE_PATH="${installdir}/lib" "${make_extra[@]}" install || return 1

The problem with TCL_PACKAGE_PATH is its usage in tclConfig.sh, which can be picked up by other package's configure script (as tcltls). Tcl lists don't work well here.

I'm reopening this ticket because I think that TCL_PACKAGE_PATH should remain a Tcl list. Tcl lists for such things makes it easier to deal with them in subsequent code, and Tcl lists are the standard for such values. Other variables like TCLLIBPATH remain a Tcl list.

Some additional commentary on how this incompatible change
(sort of) breaks OOMMF.

The OOMMF project uses Tcl scripts to manage the build of
its executables from source code on multiple platforms.
Essentially a variation on `make`, but written in Tcl and
working cross-platform

Since the executables make use of Tcl, the builds often
want to know something about the installation of Tcl (and Tk)
in order to pass the right options to compilers, linkers, etc.

One thing these scripts want to know is where the C libraries
for Tcl and Tk are.  Here is a stanza of code that determines
that:

  if {[catch {tcl::pkgconfig get libdir,runtime} libdir]} {
    if {[catch {$configuration GetValue TCL_PACKAGE_PATH} pp]} {
       continue
    }
    set libdir [Oc_DirectPathname [lindex $pp 0]]
  }

As you can see this code relies on TCL_PACKAGE_PATH being
in the format of a Tcl list, as it always was.  The new format
breaks this.

The code only makes use of TCL_PACKAGE_PATH as a fallback
when [tcl::pkgconfig] fails.  This has long been in the
code to permit the script to work when the local Tcl install
is a version before Tcl 8.5.  (This has been a project for
a very long time).  Since [tcl::pkgconfig] works in all
active supported Tcl releases now, the broken code is also
code that is never (extremely rarely???) executed.  Otherwise
I might have noticed this sooner.

Anyhow, I will concede no active harm here, but enough of
a near miss for me to raise warnings that people use the
interfaces of Tcl (including things like the format of
environment variables) in more ways than you imagine
and often different from the recommended use you have in mind,
and that changing these things in a patch release risks
damage and disruption, and it really is best if such
changes get discussed beyond comments on one ticket even
when going into a new release sequence.

Hi Jan, we have multiple usage:

• TCL_PACKAGE_PATH is a configure option of TCL
• TCL_PACKAGE_PATH is passed for module compilation via tclConfig.sh

Its "official" use targets a c embedded interpeter.

The only one I know is KitDLL with KitCreator: I scanned the trunk of the distribution https://kitcreator.rkeene.org/fossil/info/bf07147147174383 and found:

function install() {
	local filename newFilename

	mkdir -p "${installdir}/lib" || return 1
	${MAKE:-make} tcllibdir="${installdir}/lib" TCL_PACKAGE_PATH="${installdir}/lib" "${make_extra[@]}" install || return 1

What I understand is, that the Variable TCL_PACKAGE_PATH is used to inform TCL (or other packages) about the package path and it is again equal to ${installdir}/lib.

Nevertheless, within the embedding process, it is not used.

Here is my old howto for reference: https://wiki.tcl-lang.org/page/Embedding+TCL+program+in+DLL

I think in embedding, the libraries are anywahy handled differently.

Long post, not much content. I would favor to remove all this from 9.0 ;-).

If it is used, it is used the wrong way.

Take care, Harald

In this case, I recommend not to document TCL_PACKAGE_PATH. It's meant to change the paths where packages can be found like this:

./configure TCL_PACKAGE_PATH=<xxxx>

I don't know why it's included in tclConfig.sh. We could simply remove it in Tcl 9.0, that would force extension writers to use "${prefix}/lib" in stead, which is what it should be.

Noting that there is a TCL_MODULE_PATH as well, which is specific to the "tm" machinery, adding to the confusion. This one shouldn't be modified either, so better leave it undocumented .....

Any idea were to document this? I have not found anything within the current documentation?

The path separator is now ":" on MacOS/Unix and ";" on Windows. That is the right way to go, but documentation would be great.

Or it is programmer-like: the best documentation is source code ;-).

Take care, Harald

P.S.: I have opened a TCLTLS ticket here: https://core.tcl-lang.org/tcltls/tktview/c9a4f87325d68cf830bf897846b47eff26468ba6

> After `test -z "$TCL_PACKAGE_PATH" && ... ` the value of TCL_PACKAGE_PATH is "probably" empty, so it doesn't really make sense to me to add this empty value afterwards.

Indeed! That's corrected now. In Tcl 8.6.15 there will be no trailing space any more here.

> If we could just set the variable to ${prefix}/lib , then we wouldn't even need to change tcltls, as there wouldn't be any colon in the value, then...

In most cases, $TCL_PACKAGE_PATH is just "${prefix}/lib" now. I also set it to "${prefix}\lib" on Windows now, just in case, even though TCL_PACKAGE_PATH is not actually used in Windows

TCLTLS should be changed to use simply "${prefix}/lib", that's how TEA-compatible extensions are supposed to determine where they should be installed.

Hope this helps, Jan Nijtmans

Andreas, how handles TCLTLS the fact, if there is a space within the path? I suppose, the leading "{" (what caused the bug) should then also be handled.

How is a space to be specified?

• "/first path" "/second path" /nospacepath
• {/first path} {/second path} /nospacepath
• /first\ path /second\ path /nospacepath
• /first path:/second path:/nospacepath

At least, I suppose, that the column-version is the easiest to handle.

I am doubtful to make such a change in 8.6.15. A clear definition is required. TCLTLS should anyway be changed. I think, the ":" version is the must easiest useing for example sed.

Any opinion appreciated, Harald

I prefer the colon to the braces :-)

But...

After `test -z "$TCL_PACKAGE_PATH" && ... ` the value of TCL_PACKAGE_PATH is "probably" empty, so it doesn't really make sense to me to add this empty value afterwards.

If we could just set the variable to  ${prefix}/lib  , then we wouldn't even need to change tcltls, as there wouldn't be any colon in the value, then...

tcltls currently hast code in it to strip away the blank after the value, but it wouldn't yet be prepared to remove a colon, iirc.

Fixed [c2a94d79f639037e|here]. Although - actually - it's not a bug, it's a logical change: on UNIX, it's custom to use ':' as path separator in environment variables so why not do it the same in TCL_PACKAGE_PATH. The Tcl core is capable to convert this to a Tcl list for internal usage.

If TCL_PACKAGE_PATH contains a single path not containing spaces (which will be the usual case) nothing actually changes. Otherwise this is a *** potential incompatibility ***.

Proposed fix [4427b7e16885d94e|here]. Not tested on MacOS yet.

Since this - actually - works as designed (tcltls should _not_ use TCL_PACKAGE_PATH for this purpose), lowered "Severity" to Minor.

Possible solution: we can change TCL_PACKAGE_PATH, not to be a Tcl list any more, but a path separated by ':' (as is usual in UNIX). Then the curly braces can be removed

This is the commit which made the change: [6acb955a47696047]. Comment: "Fix determination of TCL_PACKAGE_PATH if it contains multiple directories, and at least one of them contains a space"

So, this change is made because TCL_PACKAGE_PATH is a Tcl list (multiple entries separated by spaces), and ${libdir} and/or ${prefix} could contain spaces itself. Without the extra braces, paths containing spaces would be split.

I don't think TCL_PACKAGE_PATH is suitable to determine the install location. tcltls should be installed in ${prefix}/lib, and that's determined by "configure --prefix=<prefix>".

Thanks, Andreas, for the ticket. Please allow me to copy the information from the clt thread titled "tcltls install error..." started 3.3.2024 16:39 UTC.

First post

It seemed to compile well, but the last few lines of build-log show:

...
objcopy --discard-all shared-tcltls.so
mv shared-tcltls.so tcltls.so
/usr/bin/install -c -d '{/usr/local/lib}/tcltls1.7.22'
/usr/bin/install -c tcltls.so '{/usr/local/lib}/tcltls1.7.22'
/usr/bin/install -c -m 644    pkgIndex.tcl '{/usr/local/lib}/tcltls1.7.22'

Is it just me...? (I'll go deeper into analysis, if this isn't an already known issue.)

Last time I built tcltls was version 1.7.21 on Solaris, and I didn't have these extra curlies back then.

Andreas addition 1

Well, I did look further into it, and the culprit is not tcltls.

In /usr/local/lib/tclConfig.sh from fresh tcl-8.6.14 install it says literally: TCL_PACKAGE_PATH='{/usr/local/lib} '

All other tclConfig.sh on my system all still have the whitespace, but no curlies in that line.

My last tcl build on that machine is a bit aged, so I cannot yet tell, how "fresh" the regression is - if it isn't caused by some other oddity on my system, anyway.

Andreas addition 2

It seems to be a regression inside tcl-8.6 that happened somewhere between 8.6.10 (correct) and 8.6.14 (wrong)

in 8.6.10 the unix/configure.in had this line 871:

    test -z "$TCL_PACKAGE_PATH" && TCL_PACKAGE_PATH="${prefix}/lib ${TCL_PACKAGE_PATH}"

in 8.6.14, that line (now 873) has changed to:

    test -z "$TCL_PACKAGE_PATH" && TCL_PACKAGE_PATH="{${prefix}/lib} ${TCL_PACKAGE_PATH}"

Note: There is a similar line shortly above, with a similar change and two now-braced items, which may affect other unix-platforms.

Either, there was a wrong assumption about what language code would actually use that tclConfig.sh entry, or tcltls is wrong by using it at all. I don't know.