Tcl Source Code

View Ticket
Login
Ticket UUID: b69e00ecf6b9c3ab7e6a99d731a6f22d82e7b7d8
Title: Tcl does not raise an error even when it detects an invalid internal UTF-8 sequence
Type: Bug Created on: 2025-07-16 06:40:21
Submitter: apnadkarni Assigned to: nobody
Subsystem: 11. Conversions from String Severity: Important
Priority: 5 Medium Last modified: 2025-07-16 06:40:41
Status: Open Closed by: nobody
Resolution: None Closed on:
Version:
Description:

Logging this bug as a reminder to revisit Tcl internal handling of invalid strings.

As an example of the broader issue, invalid UTF-8 sequences raise an error on encoding convertfrom but not on encoding convertto.

% encoding convertfrom utf-8  \xEF\xBF\x7F
unexpected byte sequence starting at index 0: '\xEF'
% binary encode hex [encoding convertto utf-8 [testbytestring \xEF\xBF\x7F]]
c3afc2bf7f

Both the above are actually UTF8->UTF8 conversions, the first converting from standard UTF-8 to Tcl's internal UTF-8 and the latter in the other direction.

(The testbytestring command is part of tcltest and a means to get a specific byte sequence, including invalid utf8 as above, into a Tcl_Obj.bytes field. Essentially it emulates Tcl_NewStringObj behavior from script.)

convertfrom behaves correctly, raising an error, as that UTF-8 sequence is invalid. convertto on the other hand, instead of being strict and raising an error, behaves as Tcl8 despite the default strict profile.

In general this behavior seems to be by design as internally there are many places where Tcl converts invalid sequences without raising an error but instead either mapping to some character, either U+FFFD, or cp1252 equivalent or assuming iso8859-1 encoding (there is no consistency in this mapping either!).

While this situation only occurs when invalid data is passed in via Tcl_NewStringObj, Tcl_AppendToObj etc., I feel Tcl should raise an error (if possible, since some functions do not have an error reporting mechanism other than panicing) when this is detected. In particular,

  • the encoding routines should definitely raise an error under the strict policy irrespective of convertto or convertfrom

  • when there is no possibility of raising an error because of the API, there should be a standard character mapping used, preferably U+FFFD as neither cp1252 or iso8859-1 make any logical sense.

This is a reminder to revisit this rather thorny issue of handling invalid internal string values.