    Author:        Jan Nijtmans <[email protected]>
    State:         Final
    Type:          Project
    Vote:          Done
    Created:       22-06-2022
    Tcl-Version:   9.0
    Keywords:      Tcl
    Tcl-Branch:    tcl8-compat
    Vote-Summary:  Accepted 3/0/0
    Votes-For:     JN, KBK SL
    Votes-Against: none
    Votes-Present: none

Abstract

Imagine the situation we want to compile an extensions for both Tcl 8.7 and 9.0. That means, two environments need to be set up, with each of them having it's own 'tcl.h' and 'libtclstub<version>.a'.

This TIP proposes bringing this challenge down to a single environment with can compile the extension for both 8.7 (possibly 8.6 also) and 9.0.

Specification

First, the stub library 'libtclstub9.0.a' is renamed to 'libtclstub.a', so without version number. Reason: the source code of 'libtclstub9.0.a' is exactly the same as 'libtclstub8.7.a', there is no reason to have a version-number in the filename. Actually 'libtclstub9.0.a' has some more functions, which are not usable in Tcl 8.7, but that is handled by proper if'def'ing in 'tcl.h'. This makes 'libtclstub.a' usable for both Tcl 8.x and 9.0.

The biggest difference between 'tcl.h' in 8.7 and 9.0 is that in Tcl 8.7 many variables have 'int' arguments while in Tcl 9.0 those are 'ptrdiff_t' arguments. That can be handled in 'tcl.h' as follows:

 #if !defined(TCL_MAJOR_VERSION)
 #   define TCL_MAJOR_VERSION 9
 #endif
 #if TCL_MAJOR_VERSION > 8
 #   define Tcl_Size ptrdiff_t
 #else
 #   define Tcl_Size int
 #endif

All parameters in 'tcl.h', 'tclDecls.h' ... are modified to use type 'Tcl_Size' in stead of 'ptrdiff_t'. This makes 'tcl.h' from 9.0 usable for Tcl 8.7 too with the following additional restrictions:

Anything what was deprecated in Tcl 8.7 (and so, removed in 9.0) is still not usable in this mode.
It only works for stub-enabled extensions, not for static linking.
'TCL_VERSION' and 'TCL_PATCHLEVEL' are not available for extensions.
No guarantee that the extension will load in Tcl 8.6 too (it will work if no Tcl 8.7-specific API is used in the extension)

Demo

So, how should an extension for Tcl 8.7 be compiled in a Tcl 9.0 environment? The only thing to be done is add -DTCL_MAJOR_VERSION=8 to the CFLAGS. Everything else is exactly the same as compiling for Tcl 9.0.

Example. Let's build 'pkga.dll'. This dll can be compiled as follows (for 64-bit Windows) for loading it in tclsh90:

    x86_64-w64-mingw32-gcc -shared -DUSE_TCL_STUBS=1 -o tcl9pkga.dll unix/dltest/pkga.c -ltclstub

Now, compile the same dll for use in Tcl 8.6 or 8.7:

    x86_64-w64-mingw32-gcc -shared -DUSE_TCL_STUBS=1 -DTCL_MAJOR_VERSION=8 -o pkga.dll unix/dltest/pkga.c -ltclstub

You can try it out:

> tclsh90
$ load ./tcl9pkga.dll
$ pkga_eq
wrong # args: should be "pkga_eq string1 string2"
$ exit
> tclsh86
$ load ./pkga.dll
$ pkga_eq
wrong # args: should be "pkga_eq string1 string2"
$ exit
> tclsh90
$ load ./pkga.dll
interpreter uses an incompatible stubs mechanism
$ exit

Implementation

See branch tcl8-compat

win/Makefile is adapted such that the 'registry' and 'dde' dll's are compiled for both Tcl 9.0 and 8.6 (since those 2 dll's fulfill all restrictions mentioned above). And also unix/dltest/Makefile is adapted such that 'pkga', 'pkgb' and 'pkgc' also are compiled for both (8.6 and 9.0) environments.

TEA

For extensions, the TEA files (tcl.m4 and rules.vc) are also extended in order to make it easier to use this feature. For 'rules.vc' there's the additional OPTS=tcl8, for 'configure' there's the additional --with-tcl8. The effect of these options is that -DTCL_MAJOR_VERSION=8 is added to the CFLAGS and that the shared library name changes from libtcl9XXX.so (tcl9XXX.dll) back to libXXX.so (XXX.dll).

This enhancement is implemented in the tcl8-compat branch from the "tclconfig" repository. When this change is merged to trunk, all extensions can use it by just re-generating the "configure" script. No other changes are necessary.

Details

The explanation above that the biggest difference between 8.7 and 9.0 is the int vs. ptrdiff_t is not the whole story. For example, the functions Tcl_UtfNcmp, Tcl_UtfNcasecmp ('tclDecls.h') and TclpUtfNcmp2 ('tclIntDecls.h') have a unsigned long parameter in Tcl8.7 which becomes size_t in Tcl 9.0. On most platforms, sizeof(unsigned long) is equal to sizeof(size_t), so there is no problem, but there is one exception which needs special consideration: Windows 64-bit (unsigned long = 32-bit, size_t = 64-bit). This means that extensions compiled for Tcl 8.7 in a Tcl 9.0 build environment will push 8 bytes on the stack in stead of 4. Since Windows 64 is little-endian, the last 4 bytes will be 0. Conclusion: we are lucky that Windows 64-bit is little-endian and that the parameter is the last one: Pushing 4 extra zero-bytes doesn't hurt at all, everything further works as expected. The 'pkga' demo is changed to use Tcl_UtfNcmp in order to prove that it works.

All other differences between Tcl 8.7 and 9.0 are properly handled with ifdef's, the three mentioned functions were the only 'tricky' ones.

Packaging Tcl distributions

For example, on Ubuntu, the following Tcl 8.6 distribution packages exists:

* tcl8.6
* tcl8.6-dev

When Tcl 8.7 and 9.0 are available, we will see the following new ones:

* tcl8.7
* tcl8.7-dev
* tcl9.0
* tcl9.0-dev

The runtime packages (all without "-dev") can be installed together, so whenever doing:

sudo apt-get install tcl8.7 tcl9.0

This will install both Tcl 8.7 and Tcl 9.0, and everything will run fine independant.

However, with the "-dev" packages it doesn't work that way. Both the "tcl8.7-dev" and "tcl9.0-dev" packages contain (among others) /usr/include/tcl.h, so if we install both it's not clear which "tcl.h" file we have. So, it is impossible to install two different Tcl "-dev" packages. We have to choose which Tcl version we want to develop for.

This TIP is written to resolve exactly this problem. Since the "tcl9.0-dev" package contains a tcl.h file which works for both Tcl 8.7 and Tcl9.0 development, we can develop extensions for both Tcl 8.7 and Tcl 9.0 with a single "-dev" package. And we can immediately run the compiled extension in our environment, whether we want to test it with Tcl 8.7 or 9.0.

Tcl 9.1

Since this TIP adds #if TCL_MAJOR_VERSION ... to 'tcl.h', this will be a maintenance burden if it's kept too long. Therefore, I propose to remove this stuff in Tcl 9.1. So, if you want to build and extension for both the 8.x and the 9.x line, you need a 9.0 development environment.

Addendum

After TIP #660 was accepted, a lot of functions changed from using size_t to ptrdiff_t parameters. In order to prevent confusion, this change has been adapted in the TIP text above as well.

Compatibility

The proposed change is 100% source and binary compatible with Tcl 9.0, but it also makes the 'tcl.h' header file and the stub library compatible with Tcl 8.7.

Copyright

This document has been placed in the public domain.

TIP 628: Shared 8.7/9.0 build environment