# TIP 494: More use of size_t in Tcl 9
	Author:         Jan Nijtmans <jan.nijtmans@gmail.com>
	State:          Final
	Type:           Project
	Vote:           Done
	Created:        29-Dec-2017
	Post-History:
	Keywords:       tcl
	Tcl-Version:    9.0
	Tcl-Branch:     memory-API
-----

# Abstract

This TIP describes the non-controversial part of the Tcl 9 changes: Make Tcl 9 ready for the 64-bit era.

# Rationale

Many Tcl API functions and struct fields have _int_ parameters, which don't provide sufficient room on 64-bit platforms.

# Proposal

  * Enhance the hash functions, such that the hash value is stored in a size\_t in stead of unsigned int. This allows hash tables to grow beyond 4Gb on 64-bit platforms

  * Enhance all struct fields representing refCounts or epochs, make them of type size_t

  * All memory-related functions, such as Tcl\_Alloc\(\), will change its size argument from _int_ to _size\_t_, and its "char *" arguments to "void *"
  
  * Many functions, which have size parameters of type _int_ (but NOT _int *_) will change to type _size\_t_

  * All _ClientData_ type arguments will be changed to _void *_ arguments. This is actually the same type, but it prevents the need for type-casts in some situations.

  * Provide a compilation option -D_TCL\_8\_COMPAT_, which provides fully source compatibility. Using this option, extensions compile and run fine using either Tcl 8 or Tcl 9 headers.
    More explanation below.

  * Provide 2 new macro's TCL\_IO\_FAILURE and TCL\_AUTO\_LENGTH, both equal to ((size_t)-1). They can help making extensions use the full 64-bit range with Tcl 9, which still compile with Tcl 8 as well (see below). 
    Tcl 8.7 will get those macro's too, but the value there is (-1)

  * The functions Tcl\_Alloc(), Tcl\_Free() (and friends) change to redirect to their debugging variants if the TCL\_MEM\_DEBUG is defined. So, Tcl\_Alloc() becomes the same as ckalloc(), ending the general confusion regarding the difference between those two groups of functions: Starting with Tcl 9.0 there is no difference anymore.
    ckalloc() and friends are deprecated starting with 9.0, but there are no plans to actually remove those and no deprecation warning will be given if extensions use it.


On 32-bit platforms, this is all 100% upwards binary compatible, provided no _internal_ API is used (since some internal structs might have incompatible but externally invisible changes)

On 64-bit platforms, those changes cause binary incompatibility. Therefore the TCL\_STUB\_MAGIC value needs to change, so extensions compiled using Tcl 9 headers will not load in Tcl 8 and reverse.


# Implications

Although those changes are binary compatible on 32-bit platforms, there is a small potential source incompatibility.
There are 10 functions which previously had an _int_ return argument, which is changed to _size\_t_ (for now, see below).
This signed/unsigned change might lead to subtle difference in behavior.

The 10 functions are:

  * Tcl_Gets

  * Tcl_GetsObj

  * Tcl_Read

  * Tcl_ReadChars

  * Tcl_ReadRaw

  * Tcl_Write

  * Tcl_WriteChars

  * Tcl_WriteObj

  * Tcl_WriteRaw

  * Tcl_Ungets

If the return value of such function is directly used in a compare, this could lead to the use of an unsigned compare in stead of a signed compare.
If you compile your extension with -D_TCL\_8\_COMPAT_, those 10 functions will be changed to wrapper macro's which makes everything behave as if those functions return Tcl_WideInt. That's the easiest way to resolve this potential problem.

There are 2 other ways to make this change, which can do it without the use of the _TCL\_8\_COMPAT_ macro:

  * Add _1_ to the left and right hand side of the comparison. E.g. [as here](https://core.tcl-lang.org/tk/fdiff?v1=100235897e9cf359&v2=9cf86629040df0d3)

  * Don't compare to -1 using < or >, but always to TCL\_IO\_FAILURE using == or != . E.g. [as here](https://core.tcl-lang.org/tk/info/abe0d3b121cbb12d)

Tk and [sqlite](http://cyqlite.sourceforge.net/cgi-bin/sqlite/info/17c148b94008df81) are already converted to make full use of TIP #494. Other extensions included with Tcl (e.g. tdbc) are unaffected.

# Open issues

There has been discussion, whether the return-type of these 10 functions (and possibly other functions) should actually be size\_t or maybe some other type.
A Tcl-specific type *Tcl\_Size* could also be defined for this. Actually, this TIP provides 2 possibilities: Without -D_TCL\_8\_COMPAT_, the type is size\_t, with -D_TCL\_8\_COMPAT_ it is Tcl_WideInt.
This gives enough room for experimenting which one is best, maybe it would be better to make it ssize\_t or ptrdiff\_t.

There - for sure - will be more future TIP's, which propose more API changes for 9.0.
This one - definitely - is not the last one.
So, this TIP doesn't make a final decision yet what will be the final type returned by those 10 functions.

# Implementation

The implementation of this TIP can be found in the [memory-API branch]
(https://core.tcl-lang.org/tcl/timeline?r=memory-API).

# Copyright

This document has been placed in the public domain.
