Author:		Rolf Ade <[email protected]>
Author:     Harald Oehlmann <[email protected]>
State:		Final
Type:		Project
Vote:		Done
Created:	25-04-2023
Tcl-Version:	9.0
Tcl-Branch:	disabletcl8api
Vote-Summary:	3 / 0 / 0
Votes-For:	SL, AK, KW
Votes-Against:	none
Votes-Present:	none

Abstract

The TIPs 481 and 616 introduced macro constructs which hide the C API changes from Tcl 8.6 to Tcl 9. This tip proposes to disabling these macros by default.

Specification

The compile time define TCL_8_API activates the compatibility macro layer. The compatibility macro layer is switched off, if not given.

The compatibility macros are off by default if this TIP is accepted. They are on by default if this TIP is not accepted.

If the compatibility macros are switched off, a compile attempt with an int * (instead a Tcl_Size *) argument should give the usual compile error (or warning) for this case, like "wrong pointer type 'int' instead 'Tcl_Size'".

Rationale

Practical impact of compatibility macros

The compatibility macros serve to compile with 32 and 64 bit type with the same function call:

int len;
Tcl_Size biglen;
char *string;
string = Tcl_GetStringFromObj(ObjPtr,&len);
string = Tcl_GetStringFromObj(ObjPtr,&biglen);

This will work and has a compiler warning in the 3rd line that the compatibility layer was used.

The drawbacks of the macros are:

Any NULL Argument must be changed to (Tcl_Size *)NULL on Windows compilers, as sizeof(NULL) triggers a compile failure with miss-leading information "error C2100: illegal indirection". The following very common code will not compile any more.

char *string;
string = Tcl_GetStringFromObj(ObjPtr,NULL);

A full type checking of the arguments are not possible any more. Real program errors may be hidden. Ashok has passed a struct pointer instead of a pointer to a struct element. This error was not found.
Users may not understand the magic behind, as the user code is changed, which may not be correctly reflected in eventual error messages. The called functions are not in the source code.
Typically macro side effects may lead to obscure issues, like double increment when ++ is used. This is a theoretical concern about macros, not special to this implementation.
If the string has more than 31 bit in length, a runtime panic of TCL happens. This is called "late failure". Without compatibility macros, the compilation show a correct and real warning. It is much more probably to see this failure "early" in tests.

A compile run without macros should throw an error or warning (depends on used compiler and its settings), like this code snippet:

void test(Tcl_Size *len);

void fun() {
    int len;
    test(&len);
}

If a compiler or settings are used which allow to ignore this warning, this leads to potential memory corruption, as a 64 bit value is written to a 32 bit int value. This is normal C behaviour and is intentional.

With compatibility macros (current main branch)

Using gcc, the following warnings are thrown. Using MS-VC the behaviour is similar.

../generic/tclsample.c:344:39: warning: passing argument 2 of ‘tclStubsPtr->tcl_GetStringFromObj’ from incompatible pointer type [-Wincompatible-pointer-types]
  344 |     p = Tcl_GetStringFromObj(objv[1], &len);
/home/username/tcl/main/include/tclDecls.h:4278:48: note: in definition of macro ‘Tcl_GetStringFromObj’
 4278 |   tclStubsPtr->tcl_GetStringFromObj((objPtr), (sizePtr)))
      |                                                ^~~~~~~
../generic/tclsample.c:344:39: note: expected ‘Tcl_Size *’ {aka ‘long int *’} but argument is of type ‘int *’
  344 |     p = Tcl_GetStringFromObj(objv[1], &len);
/home/username/tcl/main/include/tclDecls.h:4278:48: note: in definition of macro ‘Tcl_GetStringFromObj’
 4278 |   tclStubsPtr->tcl_GetStringFromObj((objPtr), (sizePtr)))
      |                                                ^~~~~~~

Without compatibility macros (current TIP661 branch)

Using gcc, only warnings are thrown, no errors. Using MS-VC the behaviour is similar. The warnings are as follows:

../generic/tclsample.c:344:39: warning: passing argument 2 of ‘tclStubsPtr->tcl_GetStringFromObj’ from incompatible pointer type [-Wincompatible-pointer-types]
  344 |     p = Tcl_GetStringFromObj(objv[1], &len);
      |                                       ^~~~
      |                                       |
      |                                       int *
../generic/tclsample.c:344:39: note: expected ‘Tcl_Size *’ {aka ‘long int *’} but argument is of type ‘int *’

Remark, that (at least), the warnings are clearer, as the expected type "Tcl_Size *" is mentioned.

Verbose description

All the functions named in the TIPs 481 and 616 changed their prototype from Tcl 8.6 to Tcl 9. This includes often used functions like Tcl_GetStringFromObj. Typically a pointer changed from int to Tcl_Size (a signed 64-bit type).

Those TIPs include a macro compatibility layer. This layer observes the size of the target of the passed pointer and finally call a function with an intargument or a 64 bit type.

TIP 664 changed the macros to trigger warnings, when no 64 bit type is used. This allows extension authors to be aware of any errors or porting issues. In addition, it introduces the extension configure flag -Wno-incompatible-pointer-types.

Extension writer should be aware, that action is required. With Tcl 9 it is possible to have Tcl_Objs with a string representation greater 2 GByte and lists up to 2^31 elements. If an unchanged extension call one of these functions with such a Tcl_Obj, the application will abort.

Without the compatibility macros in place the compiler warning is about a real thing: a 32-bit pointer may be written with a 64-bit value, overwriting memory. This means: much earlier fail than with the compatibility macros.

Ideally the compatibility macros- which had his role in migrating the Tcl core to Tcl 9 - could be removed completely. But there are reports that some parties have binary extensions in the million lines of code size. To adapt such amount of code to the Tcl 9 API may be a bigger task.

Since Tcl 9 has some not backward compatible changes also on script level such parties may have a legitimate interest to see, what that script level changes mean for their test suite. Therefor if build with the compiler flag -DTCL_8_API the compatibility macros are activated to relieve such educational tests.

Implementation

Implementation started in Tcl branch "disabletcl8api".

Copyright

This document has been placed in the public domain.

TIP 661: Disable the Tcl 8 compatibility macros in Tcl 9 by default