Tcl Source Code

View Ticket
Login
Ticket UUID: 409a4e5f2853644f6e78dce8f8d0d68a0d393dde
Title: Abstract-list hooks: lpop and join/lset crash, lindex reads before buffer with a negative index
Type: Bug Created on: 2026-10-18 10:23:43
Submitter: rocketship98 Assigned to: apnadkarni
Subsystem: 14. List Object Severity: Important
Priority: 5 Medium Last modified: 2026-11-05 10:49:43
Status: Closed Closed by: apnadkarni
Resolution: Fixed Closed on: 2026-11-05 10:49:43
Version: 9.1.1 trunk (commit 2fa2d8bbd9)
Description:

These were found by Claude doing mutation testing of the Tcl_ObjType v2 hooks. I tested each and got either seg faults or the wrong info as detailed below. The diagnosis is Claudes.

They use the test-only lstring type (tclTestABSList.c), so they need a tcltest shell built from the Tcl sources (make tcltest, then ./tcltest). Start a fresh shell for each snippet, because F1 to F3 crash the shell. No shipped type has a setElementProc, so F1 and F3 only affect extensions that supply that hook. F1 to F4 were each run on a tcltest 9.1.1 build and reproduce as described.

F1. lpop segfaults (tclCmdIL.c ~2670-2682).

set l [lstring abcd]; lpop l 1; set r [list $l]; set l 5; lappend r x; puts $r

Result: segfault. (On another build it printed {} x instead of the correct {a c d} x, so the symptom depends on memory reuse.) Cause (from reading the source, not confirmed in a debugger): the setElementProc branch calls TclUndoRefCount(listPtr) without a prior Tcl_IncrRefCount (tclObj.c: "The caller must have previously incremented the refCount"), so the reference count ends up one too low. lset (tclCmdIL.c ~4295) does increment.

F2. join of a one-element list segfaults (tclCmdIL.c ~2169-2196). For a type with getElementsProc but no indexProc (the manual requires only lengthProc):

join [lstring -not INDEX a] :

Result: segfault. The listLen > 1 test skips getElementsProc for length 1, and the single-element branch then calls TclObjTypeIndex, which has no proc.

F3. lset with a list of indices segfaults, compiled and uncompiled.

set l [lstring abc]; lset l {1 0} X

Result: segfault. Path: Tcl_LsetObjCmd -> TclLsetList -> TclLsetFlat -> Tcl_ListObjGetElements -> SetListFromAny -> Tcl_GetStringFromObj. lset l 1 0 X is refused politely; the one-argument index list is not. (Possibly related to ticket 1, as both go through TclLsetFlat.)

F4. lindex with a non-literal negative index (INST_LIST_INDEX, tclExecute.c ~5037-5047). The negative index is passed to the type's indexProc, unlike the literal-index instruction and Tcl_ListObjIndex. lstring's indexProc does not check it either and reads the byte before its buffer:

proc idx {l i} {lindex $l $i}
string length [idx [lstring abcde] -1]
binary encode hex [encoding convertto utf-8 [idx [lstring abcde] -1]]

Expected: 0 (empty string). Actual: 1, and the hex is 00, a NUL byte read from before the buffer. The value of that byte depends on memory contents.

A test file covering these (47 tests) is available on request.

User Comments:
serhiy.storchaka added on 2026-10-18 18:49:47:

All four reproduced on trunk. Proposed fix in branch abstract-list-hook-fixes:

  • F1: [lpop] now takes a reference to the result of setElementProc, like [lset]. The compiled multi-index [lset] (INST_LSET_FLAT) had the same missing reference.
  • F2: [join] uses getElementsProc for any non-empty list.
  • F3: the same use-after-free as [526b01c0b9]; its fix in branch lset-abstract-list-uaf makes lset l {1 0} X give a X c.
  • F4: ObjectType.3 requires indexProc to accept out-of-range indices, so the core may pass -1; the bug is in lstring's indexProc, which now checks index < 0.

Also fixed: lset l X without indices called setElementProc with no indices (lstring then fails with "bad index"). Tests abstractlist-5.1 to 5.4.


rocketship98 added on 2026-10-18 19:08:01:
Thanks, this caused a crash in my own extension that claude helped me write and he says there's a documention gap:

Addition to F1: doc/ObjectType.3 (SETELEMENT section) is silent on two points, and the core’s callers disagree on the second.

lpop calls setElementProc with valueObj == NULL, meaning “remove this element”. A type written from the manual crashes on lpop l 1. A real extension (a bit-array type) segfaulted this way. The only hint is a comment in TclLsetFlat (“NULL to ‘lpop’”).
The manual does not say who owns the reference count of the returned Tcl_Obj. lset (tclCmdIL.c ~4295) increments it, while lpop (~2670-2682) calls TclUndoRefCount on it with no prior increment, which is the bug in F1.

Suggest documenting both: that valueObj may be NULL for lpop, and whether the returned object comes with a reference. The fix for F1 then follows from the stated rule.

apnadkarni added on 2026-11-01 06:45:06:

@rocketship98, @serhiy.storchaka, I would prefer that rather than having setElementProc be inconsistent with TclLsetFlat, we document that it should return a Tcl_Obj with the reference count incremented and owned by the caller (as is the case for TclLsetFlat). Do you see any issues with that approach?

The documentation is silent but there is a direct mapping between TclLsetFlat and setElementProc and I suspect the original intent would have been for the latter to be consistent with the former.

Eric, yes the abstract list manpage is sorely lacking. It basically just references the script level commands which is not sufficient for a C API. Fixing it will need some source diving.


apnadkarni added on 2026-11-05 10:49:43:
Fixed for 9.0 in [92c75c9bbc] and for 9.1 in [71ed030c94].

The changes proposed in this branch were applied as is so the assumption moving  forward is that setElementProc will return a Tcl_Obj without its reference count incremented (so potentially 0).