D 2024-04-14T11:28:50.405 L Migrating\sC\sextensions\sto\sTcl\s9 N text/x-markdown P 81204fe7210d2b1c29bff5a69f8be3b640cd80081b50440cdcf6cf49317ae6d8 U apnadkarni W 12924 (Work in progress...) Also see [Porting extensions to Tcl 9](https://wiki.tcl-lang.org/page/Porting+extensions+to+Tcl+9) for another post on the topic. ## Init stubs and package require with open version The list of versions should include TCL/Tk 9. Here is an example for TCL 8.5 and following versions. Remark the "-" after "8.5". ~~~ #ifdef USE_TCL_STUBS if (Tcl_InitStubs(interp, "8.5-", 0) == NULL) #else if (Tcl_PkgRequire(interp, "Tcl", "8.5-", 0) == NULL) #endif { return TCL_ERROR; } ~~~ ## 64-bit data support On 64-bit platforms, Tcl 9 no longer limits data size (lengths of strings, lists etc.) and indices to 32-bits. For portability of extensions between 32- and 64-bit platforms, a new typedef `Tcl_Size` has been defined. This maps to a 64-bit or 32-bit signed integer type on the respective platforms. This affects all public structures and parameter types as appropriate. Variables that hold lengths or indices that were typed as `int` for Tcl 8 should be changed to be of type `Tcl_Size`. Care must be taken that related variables, for example loop iterators, are also modified accordingly. For example, in the following fragment ``` int len; int i; Tcl_ListObjLength(interp, objPtr, &len); for (i = 0; i < len; ++i) { .... } ``` the declaration of both `i` and `len` should be changed ``` Tcl_Size len; Tcl_Size i; ``` When converting sizes and indices to and from string representation, replace the use of `Tcl_GetIntFromObj` and `Tcl_NewIntObj` with the `Tcl_Size` analogues `Tcl_GetSizeIntFromObj` and `Tcl_NewSizeIntObj`. When checking string limits etc. appropriately for both platform widths, the constant `TCL_SIZE_MAX` should be used in lieu of `INT_MAX`. When formatting as a string through a `printf` style varargs function, a size modifier appropriate for the platform width needs to be used. This is defined as `TCL_SIZE_MODIFIER`. For example, ``` Tcl_Size len; Tcl_Obj *objP = Tcl_ObjPrintf("The length is %" TCL_SIZE_MODIFIER "d.", len); ``` If the ability to build against Tcl 8 is desired after making the above changes require for Tcl 9, add the following fragment to the appropriate header for the extension. ``` #ifndef TCL_SIZE_MAX # define Tcl_GetSizeIntFromObj Tcl_GetIntFromObj # define TCL_SIZE_MAX INT_MAX # define TCL_SIZE_MODIFIER "" #endif ``` "Tcl_Size" itself doesn't need to be provided as long as you update the extension to the latest tclconfig release. TEA (latest tcl.m4/rules.vc files) takes care of providing an appropriate preprocessor #define for `Tcl_Size`. Note there is no complementary `Tcl_NewSizeIntFromObj`. Use `Tcl_NewWideIntObj` in its stead to wrap `Tcl_Size` values. Do **not** use `Tcl_NewIntObj` as it will truncate values above 4Gb. Paul Obermeier's very useful post [Tcl 9 functions using Tcl_Size](https://wiki.tcl-lang.org/page/Tcl+9+functions+using+Tcl%5FSize) enumerates the functions in the Tcl API affected by this change and provides a script to locate the use of these functions within extensions and applications. See [TIP 660](https://core.tcl-lang.org/tips/doc/trunk/tip/660.md) ## Unicode support ### `Tcl_UniChar` and `TCL_UTF_MAX` As part of the changes to support the full range of Unicode code points, the `Tcl_UniChar` type has changed to a 32-bit integer type. Correspondingly, the default value of `TCL_UTF_MAX` is now 4. This means that encoding API's such as `Tcl_GetUnicodeFromObj` that work with `Tcl_UniChar` strings can no longer be used to pass values to external API's that expect UTF-16 encoded strings. This is particularly relevant on Windows platforms where it was common in Tcl 8 to use these functions to transform strings to the UTF-16 / UCS-2 form expected by the Win32 API. Instead, use either * the UTF-16 specific `Tcl_Char16ToUtfDString`, `Tcl_UtfToChar16DString`, `Tcl_UtfToChar16` functions, or * the generic functions `Tcl_UtfToExternalDStringEx`, `Tcl_ExternaltoUtfDStringEx`, `Tcl_UtfToExternal`, `Tcl_ExternalToUtf` with the `utf-16` encoding. ### Encoding transforms Tcl 9 supports *encoding profiles* that allow control of how errors are handled in encoding transforms. At the C API level, profiles are passed by or-ing one of the `TCL_PROFILE_*` values as part of the `flags` parameter to encoding transform functions like `Tcl_ExternalToUtf` and `Tcl_UtfToExternal`. Extensions that or-ed `TCL_ENCODING_STOPONERROR` in the `flags` parameter to these functions in Tcl 8 should now use `TCL_ENCODING_PROFILE_STRICT` instead or not specify a profile at all as `strict` is the default in any case. Extensions that did not make use of `TCL_ENCODING_STOPONERROR` will see a change in that the functions default to the strict profile and will return an error on invalid encoding bytes. To revert to Tcl 8 behavior, the `TCL_ENCODING_PROFILE_TCL8` must be specified. Alternatively, `TCL_ENCODING_PROFILE_REPLACE` may be used to which is standard-compliant but does not raise an error on invalid encodings. See [TIP 656](https://core.tcl-lang.org/tips/doc/trunk/tip/656.md) ## Encoding errors on I/O In Tcl 8, any invalid byte sequence encountered in channel input were silently handled by treating the invalid bytes as iso8859-1 encoded bytes. Similarly, on output Tcl replaced characters that could not be represented in the channel encoding with an encoding-specific replacement character, usually `?`. In contrast, channel I/O in Tcl 9 is by default subject to the `strict` encoding profile and the above are treated as error conditions. Functions like `Tcl_Read`, `Tcl_Gets` etc. on input and `Tcl_Write` etc. on output will raise an error for these cases. Applications and extensions that need to match Tcl 8 behavior for compatibility reasons can set the `-profile` configuration option for the channel to `tcl8` via the `Tcl_SetChannelOption` function. This is however non-conformant with the Unicode standard and is not recommended. A standard-conformant alternative is to set the profile to `replace` instead. See [TIP 657](https://core.tcl-lang.org/tips/doc/trunk/tip/657.md) ## `Tcl_ObjType` versioning The Tcl value system has been extended. The `Tcl_ObjType` has additional (optional) fields. Any Tcl extensions, or embedded integration with Tcl, that has a custom `Tcl_ObjType` value defined, should review these changes. Existing code should compile and execute just fine, unless the particular implementation is using any "creative" programming. In simpler terms, the `Tcl_ObjType` structure is larger. It should be initialized properly without any changes needed, but YMMV. Please see the `Tcl_RegisterObjType` manual page for more details, as well as TIP's [636](https://core.tcl-lang.org/tips/doc/trunk/tip/636.md) and [644](https://core.tcl-lang.org/tips/doc/trunk/tip/644.md). ## Abstract lists An abstract list is a custom Tcl value (custom `Tcl_ObjType`) that can be treated as a Tcl list at the script level without causing the value to be converted to the internal List type. To make this possible, the custom type must implement a set of accessor functions corresponding to list operations. See the `Tcl_RegisterObjType` manual page for more details, as well as TIP's [636](https://core.tcl-lang.org/tips/doc/trunk/tip/636.md) and [644](https://core.tcl-lang.org/tips/doc/trunk/tip/644.md). ### Memory management of list elements returned by Tcl_ListObjIndex When dealing with list elements, a common micro-optimization practice involves skipping `Tcl_IncrRefCount` and `Tcl_DecrRefCount` sequences for a `Tcl_Obj` element returned by `Tcl_ListObjIndex` when the use is local and temporary. In Tcl 8, this posed no issues because the list implementation internally held a reference to each element, ensuring that the element would be freed (assuming no other references) along with the parent list. However, with the introduction of Abstract Lists in Tcl 9, this guarantee no longer holds. Abstract lists may or may not retain a reference to `Tcl_Obj` elements returned from `Tcl_ListObjIndex`. Specifically, it is possible for elements to be created on demand and returned by the index operation without an internal reference being held. In the absence of a `Tcl_DecrRefCount` call on the element, the `Tcl_Obj` would never be freed resulting in a memory leak. The caller needs to explicitly call `Tcl_DecrRefCount` on the element `Tcl_Obj` to ensure it is freed. On the flip side, an abstract list implementation may mimic Tcl 8 behavior by returning an existing internal reference to a `Tcl_Obj` without incrementing its reference count. In such instances, calling `Tcl_DecrRefCount` on the element without a corresponding `Tcl_IncrRefCount` would release an internal reference held by the abstract list implementation and prematurely free the element resulting in use-after-free bugs. To handle both types of implementations correctly, the caller should always use the `Tcl_BounceRefCount` function on the returned element `Tcl_Obj` when finished with it. This function will free the passed `Tcl_Obj` if, and only if, there are no existing references to it. If the extension needs to maintain compatibility with Tcl 8.6, an alternative approach is to perform a pair of `Tcl_IncrRefCount` and `Tcl_DecrRefCount` calls on the element. While this achieves the same effect as the `Tcl_BounceRefCount` function, it comes at the cost of a slight efficiency trade-off. ### Prefer Tcl_ListObjIndex to Tcl_ListObjGetElements for large lists The introduction of abstract list implementations has an additional consequence related to iteration over lists. In Tcl 8, it was common practice to iterate over list elements by calling `Tcl_ListObjGetElements` which returned a pointer to the array of `Tcl_Obj *` elements stored in the internal representation of a list. In Tcl 9, the internal representation may take a different form and use of this function forces generation of an array with potentially significantly higher memory usage. For example, the `lseq` command which is used to create a list of integers has an extremely compact internal representation. A list of a million consecutive integers takes up under a hundred bytes of memory. However, if `Tcl_ListObjGetElements` is called on this list, it is forced to allocate an array of a million pointers each pointing to a `Tcl_Obj` holding an integer -- more than 50MB of memory. It is therefore recommended that in cases where an extension may be dealing with large lists, iteration be done using the `Tcl_ListObjIndex` function instead of `Tcl_ListObjGetElements`. Note however that there is a performance tradeoff here as iterating using `Tcl_ListObjGetElements` is faster as the returned array can be directly accessed without a function call for each element. ## Tommath Tcl 9 optionally allows the linking of an external `libtommath` library for multiple-precision integer arithmetic in lieu of Tcl's internal version. This has some consequences for extensions: * If including `tclTomMath.h`, Tcl's stubs interface to `libtommath` is used and only a subset of `libtommath` functions are available. If including `tommath.h` (external library), all `libtommath` functions are available. * Extensions using the definitions of `mp_digit`, `mp_word` or `mp_int` need to include either `tclTomMath.h` or `tommath.h` in addition `tcl.h`. See See [TIP 538](https://core.tcl-lang.org/tips/doc/trunk/tip/538.md). ## Building ### Building with autotools * Update TEA to the latest [tclconfig](https://core.tcl-lang.org/tclconfig). * Within your `configure.ac` or `configure.in` file, change the `TEA_INIT` call to require **3.13** or whatever the latest version happens to be. * Run `autoconf` to regenerate configure. * When running `configure`, in addition to the `--with-tcl` option also specify the `--with-tclinclude`. It's possible this is only required on MingW and seems like a [bug](https://core.tcl-lang.org/tcl/tktview/6f22c7a1fc) in TEA but I am not sufficiently versed in TEA to tell for sure. Without those options, the load command (directly or via `package require`) may raise the error *interpreter uses an incompatible stubs mechanism*. This *may* happen when building against a private Tcl 9 installation with a system installed Tcl 8 version. For example, if you do ``` % path/to/extension/configure --with-tcl=/path/to/tcl9/lib % make ``` To resolve, also specify the `--with-tclinclude` option to `configure`. ``` % path/to/extension/configure --with-tcl=/path/to/tcl9/lib --with-tclinclude=/path/to/tcl9/include % make ``` See the [ticket](https://core.tcl-lang.org/tcl/tktview/6f22c7a1fc) for the specific circumstances when this may happen. Z fdb2970f420b7d1bd6a8769a9cf5bca6