Tcl Source Code

NOTE: this is a claude fix, I can't be certain this is correct, sorry but he has built and then tested this with and without the patch. This is not something I understand, but the fix is small and hopefully someone will benefit from this and the tests. Where it says I, it means Claude


Follow-up with a tested fix: a patch (9 lines added, 2 removed) and six tests, both below. Against trunk of 2026-10-15, commit 102a512.

Compiled lappend in a proc turns an abstract-list value into an ordinary list in every form I tried, on 9.1.1 and on trunk:

    lappend v x
    lappend v {*}$list
    lappend v {*}{a b}
    lappend A(k) x
    lappend A(k) {*}$list
    lappend $name x

The causes are two calls that convert the object in place and never reach the type's Replace hook:

* ListAppendInVar in tclVar.c calls Tcl_ListObjAppendElement. This handles lappend v x, array elements and lappend $name x.
* lappendListDirect in tclExecute.c calls TclListObjAppendElements. This handles lappend v {*}list, including array elements.

The patch changes both to Tcl_ListObjReplace (insert at the end). The branch of INST_LAPPEND_LIST that already calls Tcl_ListObjReplace keeps the type, as do uncompiled lappend and lappend at the top level of a script.

Checked, with the test type "lstring" from tclTestABSList.c (it has a Replace hook):
* The six tests below fail on unpatched trunk (the value becomes a "list") and pass with the patch.
* With the patch, abstractlist.test, listTypes.test, list.test, listObj.test and listRep.test: 11526 tests, 0 failed.

Not checked: whether any caller relies on the type being lost, whether the extra length lookup costs anything measurable for plain lists, and the other callers of TclListObjAppendElements.

Patch (apply with patch -p1):

--- a/generic/tclExecute.c
+++ b/generic/tclExecute.c
@@ -3624,7 +3624,7 @@ TEBCresume(
 	    varPtr->value.objPtr = objResultPtr = newValue;
 	    Tcl_IncrRefCount(newValue);
 	}
-	if (TclListObjAppendElements(interp, objResultPtr, objc, objv)
+	if (Tcl_ListObjReplace(interp, objResultPtr, len, 0, objc, objv)
 		!= TCL_OK) {
 	    TRACE_ERROR(interp);
 	    goto gotError;
--- a/generic/tclVar.c
+++ b/generic/tclVar.c
@@ -1872,7 +1872,14 @@ ListAppendInVar(
 	Tcl_IncrRefCount(oldValuePtr);	/* Since var is referenced. */
     }
 
-    return Tcl_ListObjAppendElement(interp, oldValuePtr, newValuePtr);
+    {
+	Tcl_Size len;
+
+	if (TclListObjLength(interp, oldValuePtr, &len) != TCL_OK) {
+	    return TCL_ERROR;
+	}
+	return Tcl_ListObjReplace(interp, oldValuePtr, len, 0, 1, &newValuePtr);
+    }
 }
 
 static inline void

Tests (they could go at the end of tests/abstractlist.test; they use its constraints testobj and lstring):

# lappend must keep an abstract list's type (the type has a Replace hook).  The tests use apply so the
# lappend is compiled inside a proc body, which is where the type used to be lost.
test abstractlist-lappend-1.1 {lappend of one element keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set l [lstring abc]; lappend l d; list [testobj objtype $l] $l}}
} -result {lstring {a b c d}}
test abstractlist-lappend-1.2 {lappend {*}$list keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set l [lstring abc]; set x {d e}; lappend l {*}$x; list [testobj objtype $l] $l}}
} -result {lstring {a b c d e}}
test abstractlist-lappend-1.3 {lappend {*}{literal list} keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set l [lstring abc]; lappend l {*}{d e}; list [testobj objtype $l] $l}}
} -result {lstring {a b c d e}}
test abstractlist-lappend-1.4 {lappend to an array element keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set A(x) [lstring abc]; lappend A(x) d; list [testobj objtype $A(x)] $A(x)}}
} -result {lstring {a b c d}}
test abstractlist-lappend-1.5 {lappend {*}$list to an array element keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set A(x) [lstring abc]; set x {d e}; lappend A(x) {*}$x; list [testobj objtype $A(x)] $A(x)}}
} -result {lstring {a b c d e}}
test abstractlist-lappend-1.6 {lappend $name keeps the abstract list type} -constraints {testobj lstring} -body {
    apply {{} {set l [lstring abc]; set n l; lappend $n d; list [testobj objtype $l] $l}}
} -result {lstring {a b c d}}

Thanks for looking at this, Ashok.

Some context on who is hitting it. The bit-array type I mentioned is a real extension that implements all eight abstract-list hooks of TCL_OBJTYPE_V2 (length, index, slice, reverse, getElements, setElement, replace and the in/ni operator). lappend, and linsert in the uncompiled case, are the only commands I have found where the core converts the value to an ordinary list itself instead of calling one of the hooks.

On your point that there is no lappend hook: the Replace hook is indeed enough for us, since an insert at the end is cheap for this type, so routing lappend through it (as in the patch above) is all we need. I can see types where a dedicated hook would be faster, and if one is added we would be happy to implement it as well. Note that the patch I posted only covers the lappend paths; the uncompiled single-element linsert at the end goes through Tcl_LinsertObjCmd, which I have not patched, so that case still converts.

An update on the "not tested" note in the ticket: I have since tried Tcl 9.0.4, and it differs a little from 9.1. A plain lappend of one element keeps the type at the top level of a script but loses it under eval and uplevel (9.1 keeps it there) as well as in a proc; lappend {*}$list keeps the type everywhere on 9.0.4, including in a proc, where 9.1 loses it. If a fix goes in, it may be worth backporting to 9.0.x.

one quick note, having claude do builds (binary search he says) between 9.0.4 and 9.1 to pinpoint when the behavior changed.

While looking at why linsert behaves differently from lappend for us, I believe I found the cause of the linsert case (claude is more certain, and he's the I below in this comment).

In Tcl_LinsertObjCmd (tclCmdIL.c) there is a special case for inserting exactly one element at the end of the list:

    if ((objc == 4) && (index == len)) {
        result = Tcl_ListObjAppendElement(NULL, listPtr, objv[3]);

Tcl_ListObjAppendElement does not call the Replace hook of an abstract list, so the value is converted to an ordinary list. Every other form of linsert (two or more elements, middle, start) calls Tcl_ListObjReplace, which does call the hook, and keeps the type. The code is the same in 9.0.4 and 9.1.1, apart from the argument declarations.

Changing that one call to the following made all the linsert forms I tried keep the type, on both a 9.0.4 and a 9.1.1 build (compiled at top level, command name in a variable, via eval, in a proc). The core's cmdIL, listObj, list and linsert test files also pass with it (324 tests, 0 failed); I have not run the rest of the core test suite:

    result = Tcl_ListObjReplace(NULL, listPtr, len, 0, 1, &objv[3]);

Separately, for lappend: I believe the places where 9.1 already keeps the type (eval, uplevel #0) are where lappendListPtr in tclExecute.c calls Tcl_ListObjReplace instead of TclListObjAppendElements, which 9.0.4 still does:

    9.0.4:  TclListObjAppendElements(interp, valueToAssign, objc, objv)
    9.1:    Tcl_ListObjReplace(interp, valueToAssign, len, 0, objc, objv)

Changing just that call in a 9.0.4 build gave the 9.1.1 results for those two cases. I think the change may come from the fix for INST_LAPPEND_LIST with zero-length lists (dkf, trunk, May 2025), but I could not confirm that from the repository history. The case that still loses the type on both versions is a plain lappend inside a proc, which is what this ticket covers.

Thanks Ashok!

What is lstring? Is is an abstract list, which presents a string as a list where each character is a list element?

Then, you can not add "xyz", as it violates this abstract list. Ok, I understnd the example.

There are two possible solutions:

• error out
• transform to tradional list

Take care, Harald

Harald,

lstring is only part of the test suite used to test abstract lists.

Erroring out is not an option IMO as a command must treat two string representations ("a b c" in both cases) exactly the same. It cannot accept one and not the other. For something like lstring, only alternative is the second solution you proposed.

I did not mean to imply lstring needs to be fixed, it's only used in tests. Rather, my point was more that it cannot be used to test all code paths to test for shimmering of abstract lists.

Thanks for the explanation, I appreciate! Take care, Harald

Fixed for 9.1 in [54a03368ce] and 9.0 in [d27dbcd1b3].

Just a note: not all shimmers have been eliminated. In particular, nested indices and the lset implementation not tackled. Partly because not worth fixing in practice imo and partly because of complexity. lset for instance works on an unshared value while the corresponding abstract list method returns a new object. (Shimmer will occur with the C API, not with the script level command.)

Great work, thank you!

Perhaps the documentation of lseq may talk a bit about conversion to normal list. I suppose, that is always the case if a list element is written (lset).

Thanks for all, Harald

Or a document on the tcl core wiki?

I see the next command (concat) being handled...

Thanks for all, Harald