D 2024-04-14T11:28:50.405
L Migrating\sC\sextensions\sto\sTcl\s9
N text/x-markdown
P 81204fe7210d2b1c29bff5a69f8be3b640cd80081b50440cdcf6cf49317ae6d8
U apnadkarni
W 12924
(Work in progress...)
Also see [Porting extensions to Tcl 9](https://wiki.tcl-lang.org/page/Porting+extensions+to+Tcl+9)
for another post on the topic.
## Init stubs and package require with open version
The list of versions should include TCL/Tk 9. Here is an example for TCL 8.5 and following versions.
Remark the "-" after "8.5".
~~~
#ifdef USE_TCL_STUBS
if (Tcl_InitStubs(interp, "8.5-", 0) == NULL)
#else
if (Tcl_PkgRequire(interp, "Tcl", "8.5-", 0) == NULL)
#endif
{
return TCL_ERROR;
}
~~~
## 64-bit data support
On 64-bit platforms, Tcl 9 no longer limits data size (lengths of strings, lists
etc.) and indices to 32-bits. For portability of extensions between 32- and
64-bit platforms, a new typedef `Tcl_Size` has been defined. This maps to a
64-bit or 32-bit signed integer type on the respective platforms. This affects
all public structures and parameter types as appropriate. Variables that hold
lengths or indices that were typed as `int` for Tcl 8 should be changed to be of
type `Tcl_Size`. Care must be taken that related variables, for example loop
iterators, are also modified accordingly.
For example, in the following fragment
```
int len;
int i;
Tcl_ListObjLength(interp, objPtr, &len);
for (i = 0; i < len; ++i) {
....
}
```
the declaration of both `i` and `len` should be changed
```
Tcl_Size len;
Tcl_Size i;
```
When converting sizes and indices to and from string representation,
replace the use of `Tcl_GetIntFromObj` and `Tcl_NewIntObj` with the
`Tcl_Size` analogues `Tcl_GetSizeIntFromObj` and `Tcl_NewSizeIntObj`.
When checking string limits etc. appropriately for both platform widths,
the constant `TCL_SIZE_MAX` should be used in lieu of `INT_MAX`.
When formatting as a string through a `printf` style varargs function, a size
modifier appropriate for the platform width needs to be used. This is defined
as `TCL_SIZE_MODIFIER`. For example,
```
Tcl_Size len;
Tcl_Obj *objP = Tcl_ObjPrintf("The length is %" TCL_SIZE_MODIFIER "d.", len);
```
If the ability to build against Tcl 8 is desired after making the above
changes require for Tcl 9, add the following fragment to the appropriate
header for the extension.
```
#ifndef TCL_SIZE_MAX
# define Tcl_GetSizeIntFromObj Tcl_GetIntFromObj
# define TCL_SIZE_MAX INT_MAX
# define TCL_SIZE_MODIFIER ""
#endif
```
"Tcl_Size" itself doesn't need to be provided as long as you update the extension to the latest tclconfig release. TEA (latest tcl.m4/rules.vc files) takes care of providing an appropriate preprocessor #define for `Tcl_Size`.
Note there is no complementary `Tcl_NewSizeIntFromObj`. Use `Tcl_NewWideIntObj` in its stead to wrap `Tcl_Size` values. Do **not** use `Tcl_NewIntObj` as it will truncate values above 4Gb.
Paul Obermeier's very useful post [Tcl 9 functions using Tcl_Size](https://wiki.tcl-lang.org/page/Tcl+9+functions+using+Tcl%5FSize)
enumerates the functions in the Tcl API affected by this change
and provides a script to locate the use of these functions within
extensions and applications.
See [TIP 660](https://core.tcl-lang.org/tips/doc/trunk/tip/660.md)
## Unicode support
### `Tcl_UniChar` and `TCL_UTF_MAX`
As part of the changes to support the full range of Unicode code points, the
`Tcl_UniChar` type has changed to a 32-bit integer type. Correspondingly, the
default value of `TCL_UTF_MAX` is now 4.
This means that encoding API's such as `Tcl_GetUnicodeFromObj` that work with
`Tcl_UniChar` strings can no longer be used to pass values to external API's
that expect UTF-16 encoded strings. This is particularly relevant on Windows
platforms where it was common in Tcl 8 to use these functions to transform
strings to the UTF-16 / UCS-2 form expected by the Win32 API. Instead, use either
* the UTF-16 specific `Tcl_Char16ToUtfDString`, `Tcl_UtfToChar16DString`,
`Tcl_UtfToChar16` functions, or
* the generic functions `Tcl_UtfToExternalDStringEx`, `Tcl_ExternaltoUtfDStringEx`,
`Tcl_UtfToExternal`, `Tcl_ExternalToUtf` with the `utf-16` encoding.
### Encoding transforms
Tcl 9 supports *encoding profiles* that allow control of how errors are handled
in encoding transforms. At the C API level, profiles are passed by or-ing one of
the `TCL_PROFILE_*` values as part of the `flags` parameter to encoding
transform functions like `Tcl_ExternalToUtf` and `Tcl_UtfToExternal`.
Extensions that or-ed `TCL_ENCODING_STOPONERROR` in the `flags` parameter
to these functions in Tcl 8 should now use `TCL_ENCODING_PROFILE_STRICT`
instead or not specify a profile at all as `strict` is the default in any case.
Extensions that did not make use of `TCL_ENCODING_STOPONERROR` will see
a change in that the functions default to the strict profile and will return
an error on invalid encoding bytes. To revert to Tcl 8 behavior, the
`TCL_ENCODING_PROFILE_TCL8` must be specified. Alternatively,
`TCL_ENCODING_PROFILE_REPLACE` may be used to which is standard-compliant
but does not raise an error on invalid encodings.
See [TIP 656](https://core.tcl-lang.org/tips/doc/trunk/tip/656.md)
## Encoding errors on I/O
In Tcl 8, any invalid byte sequence encountered in channel input were silently
handled by treating the invalid bytes as iso8859-1 encoded bytes. Similarly, on
output Tcl replaced characters that could not be represented in the channel
encoding with an encoding-specific replacement character, usually `?`. In
contrast, channel I/O in Tcl 9 is by default subject to the `strict` encoding
profile and the above are treated as error conditions. Functions like
`Tcl_Read`, `Tcl_Gets` etc. on input and `Tcl_Write` etc. on output will raise
an error for these cases.
Applications and extensions that need to match Tcl 8 behavior for compatibility
reasons can set the `-profile` configuration option for the channel to `tcl8` via
the `Tcl_SetChannelOption` function. This is however non-conformant with the
Unicode standard and is not recommended. A standard-conformant alternative is
to set the profile to `replace` instead.
See [TIP 657](https://core.tcl-lang.org/tips/doc/trunk/tip/657.md)
## `Tcl_ObjType` versioning
The Tcl value system has been extended. The `Tcl_ObjType` has additional (optional) fields. Any Tcl extensions, or embedded integration with Tcl, that has a custom `Tcl_ObjType` value defined, should review these changes. Existing code should compile and execute just fine, unless the particular implementation is using any "creative" programming. In simpler terms, the `Tcl_ObjType` structure is larger. It should be initialized properly without any changes needed, but YMMV.
Please see the `Tcl_RegisterObjType` manual page for more details, as well as TIP's [636](https://core.tcl-lang.org/tips/doc/trunk/tip/636.md) and [644](https://core.tcl-lang.org/tips/doc/trunk/tip/644.md).
## Abstract lists
An abstract list is a custom Tcl value (custom `Tcl_ObjType`) that can be treated as a Tcl list at the script level without causing the value to be converted to the internal List type. To make this possible, the custom type must implement a set of accessor functions corresponding to list operations.
See the `Tcl_RegisterObjType` manual page for more details, as well as TIP's [636](https://core.tcl-lang.org/tips/doc/trunk/tip/636.md) and [644](https://core.tcl-lang.org/tips/doc/trunk/tip/644.md).
### Memory management of list elements returned by Tcl_ListObjIndex
When dealing with list elements, a common micro-optimization practice involves
skipping `Tcl_IncrRefCount` and `Tcl_DecrRefCount` sequences for a `Tcl_Obj`
element returned by `Tcl_ListObjIndex` when the use is local and temporary. In
Tcl 8, this posed no issues because the list implementation internally held a
reference to each element, ensuring that the element would be freed (assuming no
other references) along with the parent list.
However, with the introduction of Abstract Lists in Tcl 9, this guarantee no
longer holds. Abstract lists may or may not retain a reference to `Tcl_Obj`
elements returned from `Tcl_ListObjIndex`. Specifically, it is possible for
elements to be created on demand and returned by the index operation without an
internal reference being held. In the absence of a `Tcl_DecrRefCount` call
on the element, the `Tcl_Obj` would never be freed resulting in a memory leak.
The caller needs to explicitly call `Tcl_DecrRefCount` on the element
`Tcl_Obj` to ensure it is freed.
On the flip side, an abstract list implementation may mimic Tcl 8 behavior by
returning an existing internal reference to a `Tcl_Obj` without incrementing its
reference count. In such instances, calling `Tcl_DecrRefCount` on the element
without a corresponding `Tcl_IncrRefCount` would release an internal reference
held by the abstract list implementation and prematurely free the element resulting
in use-after-free bugs.
To handle both types of implementations correctly, the caller should always use
the `Tcl_BounceRefCount` function on the returned element `Tcl_Obj` when
finished with it. This function will free the passed `Tcl_Obj` if, and only if, there are
no existing references to it.
If the extension needs to maintain compatibility with Tcl 8.6, an alternative
approach is to perform a pair of `Tcl_IncrRefCount` and `Tcl_DecrRefCount` calls
on the element. While this achieves the same effect as the `Tcl_BounceRefCount`
function, it comes at the cost of a slight efficiency trade-off.
### Prefer Tcl_ListObjIndex to Tcl_ListObjGetElements for large lists
The introduction of abstract list implementations has an additional consequence
related to iteration over lists. In Tcl 8, it was common practice to iterate
over list elements by calling `Tcl_ListObjGetElements` which returned a pointer
to the array of `Tcl_Obj *` elements stored in the internal representation of a
list. In Tcl 9, the internal representation may take a different form and use of
this function forces generation of an array with potentially significantly
higher memory usage. For example, the `lseq` command which is used to create a
list of integers has an extremely compact internal representation. A list of a
million consecutive integers takes up under a hundred bytes of memory. However,
if `Tcl_ListObjGetElements` is called on this list, it is forced to allocate an
array of a million pointers each pointing to a `Tcl_Obj` holding an integer --
more than 50MB of memory.
It is therefore recommended that in cases where an extension may be dealing with
large lists, iteration be done using the `Tcl_ListObjIndex` function instead
of `Tcl_ListObjGetElements`. Note however that there is a performance tradeoff here
as iterating using `Tcl_ListObjGetElements` is faster as the returned array can
be directly accessed without a function call for each element.
## Tommath
Tcl 9 optionally allows the linking of an external `libtommath` library for multiple-precision
integer arithmetic in lieu of Tcl's internal version. This has some consequences for extensions:
* If including `tclTomMath.h`, Tcl's stubs interface to `libtommath` is used and only a subset of
`libtommath` functions are available. If including `tommath.h` (external library), all `libtommath`
functions are available.
* Extensions using the definitions of `mp_digit`, `mp_word` or `mp_int` need to include either
`tclTomMath.h` or `tommath.h` in addition `tcl.h`.
See See [TIP 538](https://core.tcl-lang.org/tips/doc/trunk/tip/538.md).
## Building
### Building with autotools
* Update TEA to the latest [tclconfig](https://core.tcl-lang.org/tclconfig).
* Within your `configure.ac` or `configure.in` file, change the `TEA_INIT` call to require **3.13** or whatever the latest version happens to be.
* Run `autoconf` to regenerate configure.
* When running `configure`, in addition to the `--with-tcl` option also specify the `--with-tclinclude`. It's possible this is only required on MingW and seems like a [bug](https://core.tcl-lang.org/tcl/tktview/6f22c7a1fc) in TEA but I am not sufficiently versed in TEA to tell for sure.
Without those options, the load command (directly or via `package require`) may raise the error
*interpreter uses an incompatible stubs mechanism*.
This *may* happen when building against a private Tcl 9 installation with a system installed Tcl 8 version. For example, if you do
```
% path/to/extension/configure --with-tcl=/path/to/tcl9/lib
% make
```
To resolve, also specify the `--with-tclinclude` option to `configure`.
```
% path/to/extension/configure --with-tcl=/path/to/tcl9/lib --with-tclinclude=/path/to/tcl9/include
% make
```
See the [ticket](https://core.tcl-lang.org/tcl/tktview/6f22c7a1fc) for the specific circumstances when this may happen.
Z fdb2970f420b7d1bd6a8769a9cf5bca6
