Tcl Source Code

View Ticket
Login
Ticket UUID: 89fbf1f86e4935a146f30dcf3829f44e90a564d5
Title: Unclean memory handling
Type: Bug Created on: 2023-08-31 00:27:18
Submitter: pointsman Assigned to: nobody
Subsystem: 41. Memory Allocation Severity: Severe
Priority: 5 Medium Last modified: 2023-10-16 22:13:09
Status: Open Closed by: nobody
Resolution: None Closed on:
Version: 8.6, trunk
Description:
Recipt:

- Build a Tcl 8.6.13 with CFLAGS -DPURIFY -g

- Clone the tcllib fossil repository and check out trunk

- Run in the tcllib check-out directoriy

valgrind path/to/your/tclsh/from/above/tclsh8.6 sak.tcl test run foo

This returns:

==2742752== LEAK SUMMARY:
==2742752==    definitely lost: 128 bytes in 4 blocks
==2742752==    indirectly lost: 956 bytes in 15 blocks
==2742752==      possibly lost: 0 bytes in 0 blocks
==2742752==    still reachable: 124,221 bytes in 28 blocks
==2742752==         suppressed: 0 bytes in 0 blocks


Tcl 9 trunk with the same CFLAGS and with the tcllib branch
tcl9-testarea-rde (tcllib trunk does not run with Tcl 9) there
is a similiar result:

==2742854== LEAK SUMMARY:
==2742854==    definitely lost: 192 bytes in 5 blocks
==2742854==    indirectly lost: 1,230 bytes in 19 blocks
==2742854==      possibly lost: 0 bytes in 0 blocks
==2742854==    still reachable: 68,871 bytes in 25 blocks

This must not be this way. With an according build Tcl 8.5.19:

==2742893==    definitely lost: 0 bytes in 0 blocks
==2742893==    indirectly lost: 0 bytes in 0 blocks
==2742893==      possibly lost: 447,970 bytes in 4,355 blocks
==2742893==    still reachable: 1,685,505 bytes in 20,355 blocks

Even if what valgrind marks here as definitely lost is "harmless" in
sense that it can't be exploited to an ever growing process this is
nasty noise and hinders CI wrt memory checking.
User Comments:
chrstphrchvz added on 2023-09-01 06:58:08:

I agree that “noisy” leaks—even those occurring only once per Tcl process—should be minimized to make it easier for those familiar with widely-used leak detection tools like Valgrind/LeakSanitizer/macOS’ leaks to investigate potentially more serious leaks, as well as to avoid receiving bug reports for them. However, I would not be surprised if Tcl disagrees that these leaks are a severe issue, let alone will all be fixed; see also the valgrind-suppress file. I would suggest submitting separate and more detailed reports for identifiable leaks, with e.g. debug symbol stack traces of leaked allocations; that is what I have tried doing recently.


fvogel added on 2023-09-01 20:05:34:

Not using tcllib but simply the current tip of core-8-6-branch of Tcl, launching tclsh8.6 under valgrind and only typing 'exit' then shows leaks:

francois@debian:~/Documents/tcltk/fossil/tk$ valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all /home/francois/Documents/tcltk/fossil/tcltk/bin/tclsh8.6
==19552== Memcheck, a memory error detector
==19552== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==19552== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==19552== Command: /home/francois/Documents/tcltk/fossil/tcltk/bin/tclsh8.6
==19552== 
% exit
==19552== 
==19552== HEAP SUMMARY:
==19552==     in use at exit: 673 bytes in 12 blocks
==19552==   total heap usage: 20,969 allocs, 20,957 frees, 1,871,744 bytes allocated
==19552== 
==19552== 5 bytes in 1 blocks are still reachable in loss record 1 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x49B369A: TclCreateLiteral (tclLiteral.c:252)
==19552==    by 0x49B39C2: TclRegisterLiteral (tclLiteral.c:456)
==19552==    by 0x493BBCD: CompileCmdLiteral (tclCompile.c:1799)
==19552==    by 0x493BE88: TclCompileInvocation (tclCompile.c:1819)
==19552==    by 0x493D1E4: CompileCommandTokens (tclCompile.c:2080)
==19552==    by 0x493D685: TclCompileScript (tclCompile.c:2228)
==19552==    by 0x493A53B: TclSetByteCodeFromAny (tclCompile.c:814)
==19552==    by 0x496457A: TclCompileObj (tclExecute.c:1837)
==19552==    by 0x489359E: TclNREvalObjEx (tclBasic.c:6189)
==19552== 
==19552== 16 bytes in 1 blocks are still reachable in loss record 2 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x49D2A71: TclHandleCreate (tclPreserve.c:330)
==19552==    by 0x488AD6F: Tcl_CreateInterp (tclBasic.c:522)
==19552==    by 0x109188: main (tclAppInit.c:91)
==19552== 
==19552== 16 bytes in 1 blocks are still reachable in loss record 3 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x49DE4A2: Tcl_AttemptSetObjLength (tclStringObj.c:1067)
==19552==    by 0x4994F96: FilterInputBytes (tclIO.c:5342)
==19552==    by 0x4993EAE: Tcl_GetsObj (tclIO.c:4623)
==19552==    by 0x49B6BA5: Tcl_MainEx (tclMain.c:473)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 40 bytes in 1 blocks are still reachable in loss record 4 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x49C238E: TclSetCmdNameObj (tclObj.c:4241)
==19552==    by 0x493BBFE: CompileCmdLiteral (tclCompile.c:1802)
==19552==    by 0x493BE88: TclCompileInvocation (tclCompile.c:1819)
==19552==    by 0x493D1E4: CompileCommandTokens (tclCompile.c:2080)
==19552==    by 0x493D685: TclCompileScript (tclCompile.c:2228)
==19552==    by 0x493A53B: TclSetByteCodeFromAny (tclCompile.c:814)
==19552==    by 0x496457A: TclCompileObj (tclExecute.c:1837)
==19552==    by 0x489359E: TclNREvalObjEx (tclBasic.c:6189)
==19552==    by 0x489324D: TclEvalObjEx (tclBasic.c:6067)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 5 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x49B3615: TclCreateLiteral (tclLiteral.c:247)
==19552==    by 0x49B39C2: TclRegisterLiteral (tclLiteral.c:456)
==19552==    by 0x493BBCD: CompileCmdLiteral (tclCompile.c:1799)
==19552==    by 0x493BE88: TclCompileInvocation (tclCompile.c:1819)
==19552==    by 0x493D1E4: CompileCommandTokens (tclCompile.c:2080)
==19552==    by 0x493D685: TclCompileScript (tclCompile.c:2228)
==19552==    by 0x493A53B: TclSetByteCodeFromAny (tclCompile.c:814)
==19552==    by 0x496457A: TclCompileObj (tclExecute.c:1837)
==19552==    by 0x489359E: TclNREvalObjEx (tclBasic.c:6189)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 6 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x49B6474: Tcl_MainEx (tclMain.c:309)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 7 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x48935AC: TclNREvalObjEx (tclBasic.c:6191)
==19552==    by 0x489324D: TclEvalObjEx (tclBasic.c:6067)
==19552==    by 0x48931F9: Tcl_EvalObjEx (tclBasic.c:6049)
==19552==    by 0x498545A: Tcl_RecordAndEvalObj (tclHistory.c:191)
==19552==    by 0x49B6CD2: Tcl_MainEx (tclMain.c:520)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 8 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x496790E: TEBCresume (tclExecute.c:3096)
==19552==    by 0x48908C0: TclNRRunCallbacks (tclBasic.c:4503)
==19552==    by 0x4893265: TclEvalObjEx (tclBasic.c:6068)
==19552==    by 0x48931F9: Tcl_EvalObjEx (tclBasic.c:6049)
==19552==    by 0x498545A: Tcl_RecordAndEvalObj (tclHistory.c:191)
==19552==    by 0x49B6CD2: Tcl_MainEx (tclMain.c:520)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 9 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x48901DE: TclNREvalObjv (tclBasic.c:4256)
==19552==    by 0x49679D7: TEBCresume (tclExecute.c:3097)
==19552==    by 0x48908C0: TclNRRunCallbacks (tclBasic.c:4503)
==19552==    by 0x4893265: TclEvalObjEx (tclBasic.c:6068)
==19552==    by 0x48931F9: Tcl_EvalObjEx (tclBasic.c:6049)
==19552==    by 0x498545A: Tcl_RecordAndEvalObj (tclHistory.c:191)
==19552==    by 0x49B6CD2: Tcl_MainEx (tclMain.c:520)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 48 bytes in 1 blocks are still reachable in loss record 10 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x489071A: EvalObjvCore (tclBasic.c:4419)
==19552==    by 0x48908C0: TclNRRunCallbacks (tclBasic.c:4503)
==19552==    by 0x4893265: TclEvalObjEx (tclBasic.c:6068)
==19552==    by 0x48931F9: Tcl_EvalObjEx (tclBasic.c:6049)
==19552==    by 0x498545A: Tcl_RecordAndEvalObj (tclHistory.c:191)
==19552==    by 0x49B6CD2: Tcl_MainEx (tclMain.c:520)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== 112 bytes in 1 blocks are still reachable in loss record 11 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x488B80D: Tcl_CreateInterp (tclBasic.c:775)
==19552==    by 0x109188: main (tclAppInit.c:91)
==19552== 
==19552== 196 bytes in 1 blocks are still reachable in loss record 12 of 12
==19552==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==19552==    by 0x4884F8A: TclpAlloc (tclAlloc.c:699)
==19552==    by 0x489DAB3: Tcl_Alloc (tclCkalloc.c:1059)
==19552==    by 0x493FFA1: TclInitByteCodeObj (tclCompile.c:2797)
==19552==    by 0x493A810: TclSetByteCodeFromAny (tclCompile.c:874)
==19552==    by 0x496457A: TclCompileObj (tclExecute.c:1837)
==19552==    by 0x489359E: TclNREvalObjEx (tclBasic.c:6189)
==19552==    by 0x489324D: TclEvalObjEx (tclBasic.c:6067)
==19552==    by 0x48931F9: Tcl_EvalObjEx (tclBasic.c:6049)
==19552==    by 0x498545A: Tcl_RecordAndEvalObj (tclHistory.c:191)
==19552==    by 0x49B6CD2: Tcl_MainEx (tclMain.c:520)
==19552==    by 0x1091A3: main (tclAppInit.c:91)
==19552== 
==19552== LEAK SUMMARY:
==19552==    definitely lost: 0 bytes in 0 blocks
==19552==    indirectly lost: 0 bytes in 0 blocks
==19552==      possibly lost: 0 bytes in 0 blocks
==19552==    still reachable: 673 bytes in 12 blocks
==19552==         suppressed: 0 bytes in 0 blocks
==19552== 
==19552== For lists of detected and suppressed errors, rerun with: -s
==19552== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


apnadkarni added on 2023-10-09 13:38:00:

I'm not sure the "reachable" warnings on an explicit exit can be treated as real leaks since the memory is still reachable as opposed to "definitely lost".

Anyways, a little more detail on running Rolf's test:

==192368== HEAP SUMMARY:
==192368==     in use at exit: 70,279 bytes in 49 blocks
==192368==   total heap usage: 291,376 allocs, 291,327 frees, 30,779,604 bytes allocated
==192368==
==192368== 48 bytes in 2 blocks are definitely lost in loss record 26 of 38
==192368==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==192368==    by 0x489210F: TclpAlloc (tclAlloc.c:699)
==192368==    by 0x48B0192: Tcl_Alloc (tclCkalloc.c:1038)
==192368==    by 0x48A31B9: TclArgumentEnter (tclBasic.c:5720)
==192368==    by 0x48A2B91: TclEvalEx (tclBasic.c:5454)
==192368==    by 0x49D7D92: Tcl_FSEvalFileEx (tclIOUtil.c:1783)
==192368==    by 0x49E9564: Tcl_MainEx (tclMain.c:412)
==192368==    by 0x4011C4: main (tclAppInit.c:94)
==192368==
==192368== 1,140 (96 direct, 1,044 indirect) bytes in 2 blocks are definitely lost in loss record 37 of 38
==192368==    at 0x484182F: malloc (vg_replace_malloc.c:431)
==192368==    by 0x489210F: TclpAlloc (tclAlloc.c:699)
==192368==    by 0x48B0192: Tcl_Alloc (tclCkalloc.c:1038)
==192368==    by 0x4A14360: Tcl_NewStringObj (tclStringObj.c:255)
==192368==    by 0x49FB4F9: TclSubstTokens (tclParse.c:2329)
==192368==    by 0x48A23FF: TclEvalEx (tclBasic.c:5332)
==192368==    by 0x49D7D92: Tcl_FSEvalFileEx (tclIOUtil.c:1783)
==192368==    by 0x49E9564: Tcl_MainEx (tclMain.c:412)
==192368==    by 0x4011C4: main (tclAppInit.c:94)
==192368==
==192368== LEAK SUMMARY:
==192368==    definitely lost: 144 bytes in 4 blocks
==192368==    indirectly lost: 1,044 bytes in 15 blocks
==192368==      possibly lost: 0 bytes in 0 blocks
==192368==    still reachable: 69,091 bytes in 30 blocks
==192368==         suppressed: 0 bytes in 0 blocks
==192368== Reachable blocks (those to which a pointer was found) are not shown.
==192368== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==192368==
==192368== For lists of detected and suppressed errors, rerun with: -s
==192368== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

chrstphrchvz added on 2023-10-10 07:58:29:

I'm not sure the "reachable" warnings on an explicit exit can be treated as real leaks since the memory is still reachable as opposed to "definitely lost".

The CFWord structures allocated by TclArgumentEnter() are definitely lost, because they become unreachable when Tcl_DeleteHashTable(iPtr->lineLAPtr) is called (for a PURIFY build) from:

DeleteInterpProc (tclBasic.c:1697)
Tcl_Release (tclPreserve.c:229)
FreeMainInterp (tclMain.c:937)
InvokeExitHandlers (tclEvent.c:911)
Tcl_Finalize (tclEvent.c:1107)
Tcl_Exit (tclEvent.c:968)
Tcl_ExitObjCmd (tclCmdAH.c:1035)
⋮
Tcl_EvalObjv
Tcl_EvalEx (tclBasic.c:5408)
⋮


apnadkarni added on 2023-10-15 11:21:38:

Those CFWord structures are supposed to be released from TclArgumentRelease as called from TclEvalEx but since it never returns there from Tcl_ExitObjCmd, they remain allocated. Therefore still unsure this is a real issue.


pointsman added on 2023-10-16 22:13:09:
I agree with apnadkarni and chrstphrchvz that both spots reported by
valgrind here as 'definitely lost' are harmless. The code inspection
is backed up by this script:

    set interp [interp create]
    $interp eval [list proc exit args {return -code return}]
    $interp eval [list set argv0 $argv0]
    $interp eval [list set argv $argv]
    $interp eval [list set argc $argc]
    $interp eval source sak.tcl
    interp delete $interp

Running this under valgrind as sibling to sak.tcl returns:

==488001== LEAK SUMMARY:
==488001==    definitely lost: 0 bytes in 0 blocks
==488001==    indirectly lost: 0 bytes in 0 blocks
==488001==      possibly lost: 0 bytes in 0 blocks
==488001==    still reachable: 780 bytes in 14 blocks
==488001==         suppressed: 0 bytes in 0 blocks

So Tcl is able to cleanly turn down an interpeter which runs this
code.

But it does not in the example given in the ticket. This is obviously
a problem no matter (as I wrote in the ticket) if this can be
exploited for an ever growing process.

We have explict code to handle this case (start with
[https://core.tcl-lang.org/tcl/info?udc=1&ln=692-726&name=aedf4a0025345ada1b844749ff2fc854ba880153257829320e2c84f150004d19]).

Clear memory shutdown is obviously a good thing to have for memory
leaks hunting. The current state makes automatic checks more
difficult. For sure nobody mandates a list of "harmless 'definitely
lost reports'". Every serious contributor will stumble over this with
a wtf moment at some point. There is no other way than to fix this.