Author: Nathan Coulter <[email protected]> State: Draft Type: Project Vote: Pending Created: 26-Dec-2022 Tcl-Version: 8.7 Tcl-branch: Vote-Summary: Votes-For: Votes-Against: Votes-Present:
Abstract and Specification
string is Unicode introduced in
597 and add
encodable? ..., which returns
1 if the corresponding
command would succeed, and
The only use of
[string is unicode] is to determine whether a string can be
encoded into a unicode transformation format, either utf-8, utf-16, or utf-32.
. Tcl has never needed a
[string is big5],
[string is shiftjs] or any
[string is someencodinghere]. There is also no need for
unicode]. To determine whether a given string can be encoded into a given
encoding, it is sufficient to attempt to perform the encoding without doing the
extra work to return the encoded value.
[string is unicode] fails in its stated purpose. According to TIP 597,
The string is unicode command can be used to check if the "utf-8"/"utf-16" encodings would deliver valid output, ...
. This is not true:
set text \U03fffe string is unicode $text;# -> 0 binary scan [encoding convertto utf-16 $text] H* hex set hex ;# -> fdff
The problem is that according to TIP 597, in addition to the surrogate
characters, the return value is also
0 for the 66 noncharacters, U+??FFFE -
U+??FFFF and U+FDD0 - U+FDEF. This means that
string is unicode and
Tcl_UniCharIsUnicode can not be used to check whether the data could be
encoded into one of the Unicode encoding forms.
The Unicode specification makes it clear that noncharacters may be encoded into an encoding form. First, there is definition 79:
A Unicode encoding form assigns each Unicode scalar value to a unique code unit sequence.
Th specification then declares:
To ensure that the mapping for a Unicode encoding form is one-to-one, all Unicode scalar values, including those corresponding to noncharacter code points and unassigned code points, must be mapped to unique code unit sequences. Note that this requirement does not extend to high-surrogate and low-surrogate code points, which are excluded by definition from the set of Unicode scalar values.
Q: Are noncharacters invalid in Unicode strings and UTFs?
Absolutely not. Noncharacters do not cause a Unicode string to be ill-formed in any UTF. This can be seen explicitly in the table above, where every noncharacter code point has a well-formed representation in UTF-32, in UTF-16, and in UTF-8. An implementation which converts noncharacter code points between one UTF representation and another must preserve these values correctly. The fact that they are called "noncharacters" and are not intended for open interchange does not mean that they are somehow illegal or invalid code points which make strings containing them invalid.
As a response to issue Tcl 9: "illegal byte
sequence" ?! all checks
for noncharacters were removed in commit
[string is unicode] is already obsolete, having fallen behind the
reality of the implementation. The only thing
string is unicode now does is
check for surrogate code points.
Tcl_UniCharIsUnicode() is also not useful. If some encoding functionality is
to be exposed at the C level, the equivalent of
encoding convertto could be
Implementation will be provided as Unicode capabilities of Tcl are further refined.
Copyright © 2023, Nathan Coulter. All rights reserved.
The author of this TIP requests financial support for this and other free software works. Contact and payment information available at: