Tcl Source Code

(Work in progress...)

Also see Porting extensions to Tcl 9 for another post on the topic.

Init stubs and package require with open version

The list of versions should include TCL/Tk 9. Here is an example for TCL 8.5 and following versions. Remark the "-" after "8.5".

#ifdef USE_TCL_STUBS
    if (Tcl_InitStubs(interp, "8.5-", 0) == NULL)
#else
    if (Tcl_PkgRequire(interp, "Tcl", "8.5-", 0) == NULL)
#endif
    {
        return TCL_ERROR;
    }

64-bit data support

On 64-bit platforms, Tcl 9 no longer limits data size (lengths of strings, lists etc.) and indices to 32-bits. For portability of extensions between 32- and 64-bit platforms, a new typedef Tcl_Size has been defined. This maps to a 64-bit or 32-bit signed integer type on the respective platforms. This affects all public structures and parameter types as appropriate. Variables that hold lengths or indices that were typed as int for Tcl 8 should be changed to be of type Tcl_Size. Care must be taken that related variables, for example loop iterators, are also modified accordingly.

For example, in the following fragment

int len;
int i;
Tcl_ListObjLength(interp, objPtr, &len);
for (i = 0; i < len; ++i) {
   ....
}

the declaration of both i and len should be changed

Tcl_Size len;
Tcl_Size i;

When converting sizes and indices to and from string representation, replace the use of Tcl_GetIntFromObj and Tcl_NewIntObj with the Tcl_Size analogues Tcl_GetSizeIntFromObj and Tcl_NewSizeIntObj.

When checking string limits etc. appropriately for both platform widths, the constant TCL_SIZE_MAX should be used in lieu of INT_MAX.

When formatting as a string through a printf style varargs function, a size modifier appropriate for the platform width needs to be used. This is defined as TCL_SIZE_MODIFIER. For example,

Tcl_Size len;
Tcl_Obj *objP = Tcl_ObjPrintf("The length is %" TCL_SIZE_MODIFIER "d.", len);

If the ability to build against Tcl 8 is desired after making the above changes require for Tcl 9, add the following fragment to the appropriate header for the extension.

#ifndef TCL_SIZE_MAX
# define Tcl_GetSizeIntFromObj Tcl_GetIntFromObj
# define TCL_SIZE_MAX      INT_MAX
# define TCL_SIZE_MODIFIER ""
#endif

"Tcl_Size" itself doesn't need to be provided as long as you update the extension to the latest tclconfig release. TEA (latest tcl.m4/rules.vc files) takes care of providing an appropriate preprocessor #define for Tcl_Size.

Note there is no complementary Tcl_NewSizeIntFromObj. Use Tcl_NewWideIntObj in its stead to wrap Tcl_Size values. Do not use Tcl_NewIntObj as it will truncate values above 4Gb.

Paul Obermeier's very useful post Tcl 9 functions using Tcl_Size enumerates the functions in the Tcl API affected by this change and provides a script to locate the use of these functions within extensions and applications.

See TIP 660

Unicode support

Tcl_UniChar and TCL_UTF_MAX

As part of the changes to support the full range of Unicode code points, the Tcl_UniChar type has changed to a 32-bit integer type. Correspondingly, the default value of TCL_UTF_MAX is now 4.

This means that encoding API's such as Tcl_GetUnicodeFromObj that work with Tcl_UniChar strings can no longer be used to pass values to external API's that expect UTF-16 encoded strings. This is particularly relevant on Windows platforms where it was common in Tcl 8 to use these functions to transform strings to the UTF-16 / UCS-2 form expected by the Win32 API. Instead, use either

• the UTF-16 specific Tcl_Char16ToUtfDString, Tcl_UtfToChar16DString, Tcl_UtfToChar16 functions, or

• the generic functions Tcl_UtfToExternalDStringEx, Tcl_ExternaltoUtfDStringEx, Tcl_UtfToExternal, Tcl_ExternalToUtf with the utf-16 encoding.

Encoding transforms

Tcl 9 supports encoding profiles that allow control of how errors are handled in encoding transforms. At the C API level, profiles are passed by or-ing one of the TCL_PROFILE_* values as part of the flags parameter to encoding transform functions like Tcl_ExternalToUtf and Tcl_UtfToExternal.

Extensions that or-ed TCL_ENCODING_STOPONERROR in the flags parameter to these functions in Tcl 8 should now use TCL_ENCODING_PROFILE_STRICT instead or not specify a profile at all as strict is the default in any case.

Extensions that did not make use of TCL_ENCODING_STOPONERROR will see a change in that the functions default to the strict profile and will return an error on invalid encoding bytes. To revert to Tcl 8 behavior, the TCL_ENCODING_PROFILE_TCL8 must be specified. Alternatively, TCL_ENCODING_PROFILE_REPLACE may be used to which is standard-compliant but does not raise an error on invalid encodings.

See TIP 656

Encoding errors on I/O

In Tcl 8, any invalid byte sequence encountered in channel input were silently handled by treating the invalid bytes as iso8859-1 encoded bytes. Similarly, on output Tcl replaced characters that could not be represented in the channel encoding with an encoding-specific replacement character, usually ?. In contrast, channel I/O in Tcl 9 is by default subject to the strict encoding profile and the above are treated as error conditions. Functions like Tcl_Read, Tcl_Gets etc. on input and Tcl_Write etc. on output will raise an error for these cases.

Applications and extensions that need to match Tcl 8 behavior for compatibility reasons can set the -profile configuration option for the channel to tcl8 via the Tcl_SetChannelOption function. This is however non-conformant with the Unicode standard and is not recommended. A standard-conformant alternative is to set the profile to replace instead.

See TIP 657

Tcl_ObjType versioning

The Tcl value system has been extended. The Tcl_ObjType has additional (optional) fields. Any Tcl extensions, or embedded integration with Tcl, that has a custom Tcl_ObjType value defined, should review these changes. Existing code should compile and execute just fine, unless the particular implementation is using any "creative" programming. In simpler terms, the Tcl_ObjType structure is larger. It should be initialized properly without any changes needed, but YMMV.

Please see the Tcl_RegisterObjType manual page for more details, as well as TIP's 636 and 644.

Abstract lists

An abstract list is a custom Tcl value (custom Tcl_ObjType) that can be treated as a Tcl list at the script level without causing the value to be converted to the internal List type. To make this possible, the custom type must implement a set of accessor functions corresponding to list operations.

See the Tcl_RegisterObjType manual page for more details, as well as TIP's 636 and 644.

Memory management of list elements returned by Tcl_ListObjIndex

When dealing with list elements, a common micro-optimization practice involves skipping Tcl_IncrRefCount and Tcl_DecrRefCount sequences for a Tcl_Obj element returned by Tcl_ListObjIndex when the use is local and temporary. In Tcl 8, this posed no issues because the list implementation internally held a reference to each element, ensuring that the element would be freed (assuming no other references) along with the parent list.

However, with the introduction of Abstract Lists in Tcl 9, this guarantee no longer holds. Abstract lists may or may not retain a reference to Tcl_Obj elements returned from Tcl_ListObjIndex. Specifically, it is possible for elements to be created on demand and returned by the index operation without an internal reference being held. In the absence of a Tcl_DecrRefCount call on the element, the Tcl_Obj would never be freed resulting in a memory leak. The caller needs to explicitly call Tcl_DecrRefCount on the element Tcl_Obj to ensure it is freed.

On the flip side, an abstract list implementation may mimic Tcl 8 behavior by returning an existing internal reference to a Tcl_Obj without incrementing its reference count. In such instances, calling Tcl_DecrRefCount on the element without a corresponding Tcl_IncrRefCount would release an internal reference held by the abstract list implementation and prematurely free the element resulting in use-after-free bugs.

To handle both types of implementations correctly, the caller should always use the Tcl_BounceRefCount function on the returned element Tcl_Obj when finished with it. This function will free the passed Tcl_Obj if, and only if, there are no existing references to it.

If the extension needs to maintain compatibility with Tcl 8.6, an alternative approach is to perform a pair of Tcl_IncrRefCount and Tcl_DecrRefCount calls on the element. While this achieves the same effect as the Tcl_BounceRefCount function, it comes at the cost of a slight efficiency trade-off.

Prefer Tcl_ListObjIndex to Tcl_ListObjGetElements for large lists

The introduction of abstract list implementations has an additional consequence related to iteration over lists. In Tcl 8, it was common practice to iterate over list elements by calling Tcl_ListObjGetElements which returned a pointer to the array of Tcl_Obj * elements stored in the internal representation of a list. In Tcl 9, the internal representation may take a different form and use of this function forces generation of an array with potentially significantly higher memory usage. For example, the lseq command which is used to create a list of integers has an extremely compact internal representation. A list of a million consecutive integers takes up under a hundred bytes of memory. However, if Tcl_ListObjGetElements is called on this list, it is forced to allocate an array of a million pointers each pointing to a Tcl_Obj holding an integer -- more than 50MB of memory.

It is therefore recommended that in cases where an extension may be dealing with large lists, iteration be done using the Tcl_ListObjIndex function instead of Tcl_ListObjGetElements. Note however that there is a performance tradeoff here as iterating using Tcl_ListObjGetElements is faster as the returned array can be directly accessed without a function call for each element.

Tommath

Tcl 9 optionally allows the linking of an external libtommath library for multiple-precision integer arithmetic in lieu of Tcl's internal version. This has some consequences for extensions:

• If including tclTomMath.h, Tcl's stubs interface to libtommath is used and only a subset of libtommath functions are available. If including tommath.h (external library), all libtommath functions are available.

• Extensions using the definitions of mp_digit, mp_word or mp_int need to include either tclTomMath.h or tommath.h in addition tcl.h.

See See TIP 538.

Building

Building with autotools

• Update TEA to the latest tclconfig.

• Within your configure.ac or configure.in file, change the TEA_INIT call to require 3.13 or whatever the latest version happens to be.

• Run autoconf to regenerate configure.

• When running configure, in addition to the --with-tcl option also specify the --with-tclinclude. It's possible this is only required on MingW and seems like a bug in TEA but I am not sufficiently versed in TEA to tell for sure.

Without those options, the load command (directly or via package require) may raise the error

interpreter uses an incompatible stubs mechanism.

This may happen when building against a private Tcl 9 installation with a system installed Tcl 8 version. For example, if you do

% path/to/extension/configure --with-tcl=/path/to/tcl9/lib
% make

To resolve, also specify the --with-tclinclude option to configure.

% path/to/extension/configure --with-tcl=/path/to/tcl9/lib --with-tclinclude=/path/to/tcl9/include
% make

See the ticket for the specific circumstances when this may happen.