Author:         Jan Nijtmans <[email protected]>
State:          Final
Type:           Project
Vote:           Done
Created:        29-Dec-2017
Post-History:
Keywords:       tcl
Tcl-Version:    9.0
Tcl-Branch:     memory-API

Abstract

This TIP describes the non-controversial part of the Tcl 9 changes: Make Tcl 9 ready for the 64-bit era.

Rationale

int is the type of many Tcl API function arguments and return values, and also of many struct fields, but it doesn't provide sufficient room on 64-bit platforms.

Proposal

• For hash functions, replace unsigned int with size_t as the type of the hash value, which allows hash tables to grow beyond 4Gb on 64-bit platforms.

• Make size_t the type of all struct fields representing reference counts or epochs.

• Change the type of the size parameter of each memory-related function, e.g. Tcl_Alloc(), from int to size_t and change the char * argument to void *.

• Change the type of the size parameter of many functions which from int (but NOT int *) to size_t.

• Change the type of all ClientData arguments void *. This is actually the same type, but it prevents the need for type casts in some situations.

• Provide a compilation option, -D_TCL_8_COMPAT, which provides full source compatibility. Using this option, extensions compile and run fine using either Tcl 8 or Tcl 9 headers. See explanation below.

• Provide two new macros, TCL_IO_FAILURE and TCL_AUTO_LENGTH, each defined as ((size_t)-1), to help extensions use the full 64-bit range with Tcl 9, but still compile with Tcl 8 as well (see below). Provide those macros in Tcl 8.7 too, but defined as (-1).

• Modify Tcl_Alloc(), Tcl_Free(), and related functions to redirect to their debugging variants if TCL_MEM_DEBUG is defined: Tcl_Alloc() becomes the same as ckalloc(), ending the general confusion regarding the difference between those two groups of functions: Starting with Tcl 9.0 there no longer any difference. ckalloc() and related functions are deprecated starting with 9.0, but there are no plans to remove them, and no deprecation warning will be given if extensions use them.

On 32-bit platforms this is all 100% upwards binary compatible provided no internal API is used, since some internal structs might have incompatible but externally invisible changes.

On 64-bit platforms, those changes cause binary incompatibility. Therefore the TCL_STUB_MAGIC value needs to change, so extensions compiled using Tcl 9 headers will not load in Tcl 8 and reverse.

Implications

Although those changes are binary-compatible on 32-bit platforms, there is a small potential source incompatibility. Ten functions whose return type was previously int are changed to size_t (for now, see below). This signed/unsigned change might lead to subtle difference in behavior.

The 10 functions are:

• Tcl_Gets()

• Tcl_GetsObj()

• Tcl_Read()

• Tcl_ReadChars()

• Tcl_ReadRaw()

• Tcl_Write()

• Tcl_WriteChars()

• Tcl_WriteObj()

• Tcl_WriteRaw()

• Tcl_Ungets()

If the return value of such a function is directly used for comparison, this could lead cause the operands to be cast to unsigned int instead of int for the comparison. If an extension is compiled extension with -D_TCL_8_COMPAT, those ten functions become wrapper macros that make everything behave as if those functions return Tcl_WideInt. That's the easiest way to resolve this potential problem.

There are two other ways to make this change, without using the TCL_8_COMPAT macro:

• Add 1 to the left and right hand side of the comparison. E.g. like this

• Don't compare to -1 using < or >, but always to TCL_IO_FAILURE using == or != . E.g. like this

Tk and sqlite are already converted to make full use of TIP #494. Other extensions included with Tcl, e.g. tdbc, are unaffected.

Open issues

There has been discussion regarding whether the return type of these ten functions, and possibly other functions, should instead size_t or maybe some other type. A Tcl-specific type Tcl_Size could also be defined for this. Actually, this TIP provides 2 possibilities: Without -D_TCL_8_COMPAT_ the type is size_t, and with -D_TCL_8_COMPAT_ it is Tcl_WideInt. This gives enough room for experimenting which one is best, maybe it would be better to make it ssize_t or ptrdiff_t.

There will certainly be more TIPs in the future which propose additional API changes for 9.0. This one is definitely not the last one. Therefore, this TIP doesn't make a final decision yet what will be the final type returned by those ten functions.

Implementation

The implementation of this TIP can be found in the memory-API branch.

Copyright

This document has been placed in the public domain.