# TIP 497: Full support for Unicode planes 1-16.
Author: Jan Nijtmans <[email protected]>
State: Final
Type: Project
Vote: Done
Created: 23-Jan-2018
Post-History:
Discussions-To: Tcl Core list
Keywords: Tcl
Tcl-Version: 9.0
Tcl-Branch: tip-497
Vote-Summary: Accepted 6/0/1
Votes-For: DKF, FV, JN, KBK, KW, SL
Votes-Against: none
Votes-Present: DGP
-----
# Abstract
This TIP proposes to add full support for Unicode planes 1-16, which
contain characters >= **U\+010000**, inclusive in the regexp
engine. Also, the caveats remaining from TIP #389 will be handled here.
# Rationale
In Tcl 8.7, running `[length "😊"]` gives the result 2. The reason for
this is that - internally - Tcl 8.x splits the character in two
surrogate characters, so they fit in a 16-bit character array.
This TIP (for Tcl 9.0) is meant to put an end to this workaround,
providing a real solution.
# Specification
This document proposes:
* Change the internal "string" objType, such that it will store 32-bit
characters in stead of 16-bit characters internally.
* No longer consider Emoji to consist of 2 surrogate characters. So
`[length "😊"] == 1`
* Modify the Tcl_UniChar type, such that - by default -
`sizeof(Tcl_UniChar)` == 4 (was: 2). All extensions assuming
`sizeof(Tcl_UniChar)` == 2 will be affected by this.
It would be possible to introduce an additional 16-bit character objType
internally, this TIP doesn't implement or propose that. That could always
be added later, but at this moment it is questionable whether that would
give a serious advantage or not. Anyway, out of scope for this TIP.
# Compatibility
For Tcl 8.x, `sizeof(Tcl_UniChar)` == 2. All extensions making this
assumption will be affected, since for Tcl 9.0 that will not be true
any more. There are two ways extensions can be modified:
* Either fix the places in the code where Tcl_UniChar is used in
a non-portable way. The API introduced by TIP #548 can be used for that.
* An alternative is compile the extension using `-DTCL_UTF_MAX=3`. This
will restore `sizeof(Tcl_UniChar) == 2` as visible for the extension.
Almost all (non-deprecated) Tcl API's can still be used by the extension,
even the ones using `Tcl_UniChar`, except: `Tcl_NewUnicodeObj()`,
`Tcl_SetUnicodeObj()`, `Tcl_GetUnicodeFromObj()`, `Tcl_GetUnicode()`.
There is no need to re-compile Tcl using `-DTCL_UTF_MAX=3` for the
extension to work.
This change introduces a potential binary incompatibility compared to Tcl 9.0a3
and earlier: All extensions using any of the 4 following functions need to be recompiled:
* `Tcl_NewUnicodeObj()`
* `Tcl_SetUnicodeObj()`
* `Tcl_GetUnicodeFromObj()`
* `Tcl_GetUnicode()`
None of the (battery-included) extensions included by Tcl (e.g. Thread, tdbc ...)
are affected. Extensions need to be aware that those 4 functions interface
directly with the internal "string" objType, which stores 16-bit characters
in Tcl 8.x and 32-bit characters in Tcl 9.x.
# Reference Implementation
A reference implementation is available in the **tip-497** branch.
<https://core.tcl-lang.org/tcl/timeline?r=tip-497>
# Copyright
This document has been placed in the public domain.