Tcl Source Code

There are multiple instances of signed integer overflow in Tcl_DString functions. But I am not merely proposing that the overflowing operations be performed as unsigned int, because there would still be problematic behavior, such as an infinite loop in Tcl_ExternalToUtfDString() during encoding convertfrom … in this example (assuming it doesn’t first panic due to insufficient memory):

set f [open /dev/zero [list RDONLY BINARY]]
fconfigure $f -buffersize [expr {2**20}]
puts [time {set z [read $f [expr {2**30-1}]]} 1]
puts [::tcl::unsupported::representation $z]
puts [time {set a [string cat [encoding convertto ascii {a}] $z]} 1]
puts [::tcl::unsupported::representation $a]
puts [time {set s [encoding convertfrom ascii $a]} 1]
puts [::tcl::unsupported::representation $s]

The infinite loop is due to never being able to grow the destination string large enough for TableToUtfProc() to stop returning TCL_CONVERT_NOSPACE. There is also overflow wraparound which causes the destination string to be unexpectedly truncated, losing part of the result.

If these sorts of issues are to be resolved (rather than considered won’t-fix for Tcl 8), it seems there would first need to be a decision made on what the intended behaviors are; I’m not sure if I can identify all of them. But this would include deciding what the maximum allowed length should be: it seems possible to support length == INT_MAX by having Tcl internally treat spaceAvl as if it were unsigned int (since clients aren’t supposed to use this field anyway) and accommodate spaceAvl == INT_MAX + 1u; alternatively, it seems possible to only support up to length == INT_MAX - 1 through additional Tcl_Panic() usage (since the affected functions cannot indicate an error to the caller, and panicking during these functions is already possible when ckalloc() or ckrealloc() fail). It would also include deciding what to do when a Tcl_DString of maximum possible length still isn’t large enough to accommodate a result (panicking again seems like an option, since that can already happen due to memory allocation failure).

On panicking: I personally want to avoid encouraging further Tcl_Panic() proliferation, because some situations where Tcl panics instead of generating an error seem to be only because panicking was much easier than implementing Tcl error handling. Unfortunately the lack of error handling is already baked into the Tcl_DString API.