Tcl Source Code

View Ticket
Login
Ticket UUID: 56ab601b7dfd649035e808fb85e024f60e02e280
Title: Canonical sign becomes wrong after list duplication resp. botched after build of string representation of the copy.
Type: Bug Created on: 2018-03-14 21:11:45
Submitter: sebres Assigned to: nobody
Subsystem: 14. List Object Severity: Important
Priority: 5 Medium Last modified: 2018-03-15 14:08:49
Status: Open Closed by: nobody
Resolution: None Closed on:
Version: all
Description:
If we've an object with definitely not canonical form of string representation, 
for example `originPtr` with string content like `a   b`, so any in-place 
conversion of this object to the list set the canonicalFlag of the new created
internal list representation to 0.
So far, everything is OK.

Now something like TclListObjCopy, DupListInternalRep, etc, that just copies 
the internal representation of this list, will be executed, so we have another 
object `copyPtr` with the same internal list representation as `originPtr` (with 
List::refCount == 2 now).

The canonicalFlag is still correct, but evil is already happened.

Because after call of something expected string representation for this duplicate
`copyPtr` (that calls UpdateStringOfList), the canonicalFlag of the list of this
object gets 1 (actually of both objects)!

The problem is - this list representation is "shared" between `originPtr` and 
`copyPtr`, thus the check using ListObjIsCanonical(originPtr) return 1 now.

The issue is basically present because the member canonicalFlag must belong not 
to the List structure, but to the Tcl_Obj (e. g. as second int-rep like flags in 
internalRep.ptrAndLongRep.value, so can be different across `originPtr` and 
`copyPtr` with the same "shared" internal list representation.

Unfortunately it changes the structure of the list (and macros), so not really backwards compatible (at least not for compiled binaries). 
But IMHO this is the proper way only, to fix this issue.
User Comments:
dgp added on 2018-03-14 21:36:08:
Sound analysis.  I'll have to source dive to see if and where it applies
to the existing code base, and whether we can invent test cases that
illustrate the flaw.  Probably will be days to weeks before I
can take that up.

There was a similar problem with bytearrays and the solution in
place on 8.7 branch is to make two Tcl_ObjTypes, one for known
canonical values, and another for values not so constrained. This
puts the discriminator in the Tcl_Obj struct where it belongs,
in the typePtr field. The two Tcl_ObjTypes end up sharing much
of their implementations.

sebres added on 2018-03-15 14:08:49:

As regards "illustrate the flaw" - I don't know how one can do it easy with pure tcl (e. g. to write a tcl test case).

But it is pretty easy using C-API:

Tcl_Obj *originPtr = Tcl_NewStringObj("a   b   c", -1);
Tcl_Obj *copyPtr = TclListObjCopy(NULL, originPtr); /* evil happened here */
printf("%p: %-10s (%d)\n", originPtr, Tcl_GetString(originPtr), ListObjIsCanonical(originPtr));
printf("%p: %-10s (%d)\n", copyPtr, Tcl_GetString(copyPtr), ListObjIsCanonical(copyPtr)); /* it appears from here */
printf("%p: %-10s (%d)\n", originPtr, Tcl_GetString(originPtr), ListObjIsCanonical(originPtr));
You'll see something like:
0068DAA0: a   b   c  (0)
0068DC20: a b c      (1)
0068DAA0: a   b   c  (1)

As regards the possible solution, I've a simple branch fixes this, will try to back-port it to fossil later (too busy currently)...