Tcl Source Code

View Ticket
Login
Ticket UUID: fdcc860df5b2ab01e5bbff0f83bb227e8dec54f7
Title: Test suite tests not running independent
Type: Bug Created on: 2014-11-13 00:14:25
Submitter: pointsman Assigned to: nobody
Subsystem: 69. Other Severity: Minor
Priority: 3 Low Last modified: 2026-10-22 17:19:32
Status: Closed Closed by: serhiy.storchaka
Resolution: Fixed Closed on: 2026-10-21 10:56:31
Version: 8.6.18, 9.0.5, 9.1.0
Description:
The long list below names 202 tests out of the tcl core test suite, that pass, if they run in the context of a full test suite run with e.g make test, but fail, if they run standalone, selected by the tcltest option -match, e.g.

./tcltest ../tests/all.tcl -match <unique_test_name>

(In practice, one would restrict the sourced test files with the help of the tcltest -file option to the one test file, that includes the test of interest, but that aside.)

In other words, the below named tests doesn't run independed from other tests in their test file.

The usual advise is: tests should be written in a way, that they run independed from other tests. This is not only a matter of taste. 

The - admitted small - impact comes into the picture, if someone tries to track down a memory leak, reported by a mem-checker (like valgrind) after a test suite run to a single test, that triggers the mem leak. 

One simple brute-force attempt for that would be, to bisec the tests in an affected test file with an appropriate -match option, look which part of the tests still show the leak, repeate the bisec on this part and so on until a single test is found, that demonstrates the problem. (Either 'by hand' on the cmd line or with a helper script.)

Having 'false' test failure and (at least in the case of tests with script error) other executed code paths depending on how you range the tests to run makes this process even more tedious, as it already is and harder to write a reliable script, to automate this task.

The following list was generaded and checked on a linux system, using 8.6.3, with just standard ./configure 

The tests not running sucessfully (either wrong result or script error in the body) when run independ from others are: 

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
cmdMZ-return-3.0
cmdMZ-return-3.1
cmdMZ-return-3.3
cmdMZ-return-3.8
cmdMZ-return-3.9
cmdMZ-return-3.10
cmdMZ-return-3.11
cmdMZ-return-3.12
cmdMZ-return-3.13
cmdMZ-return-3.14
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
safe-14.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1
User Comments:
dgp added on 2014-11-13 14:55:04:
A great contribution.  Thank you!

I'm sure the "tool" to build this list is simple,
but it still might be worth sharing around.
This test suite hygiene task is one many
projects could benefit from.

pointsman added on 2014-11-13 23:20:16:
This results are collected by one of a few test suite tool scripts of mine. You'll find them hosted at

http://46.163.78.80/cgi-bin/repros/test-suite-tools/dir?ci=tip

You'll need singletests.tcl, utils.tcl and test-names.tcl. Since that url (although stable since years) may not last forever, I'll attatch them in the current state.

Or what other way of sharing around did you have in mind for such exotic stuff?

pointsman added on 2015-08-06 18:45:52:
This is the current list of not independend tests as of 8931dc163c0e 2015-08-06 04:25:50 UTC. Compared to the original report: A few tests are fixed (mostly in cmdMZ.test), a few got added, mostly the status is unchanged.

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
http-4.15
http11-2.4
http11-2.11
http11-4.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
interp-14.1
interp-36.7
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
msgcat-14.1
msgcat-14.2
msgcat-14.3
msgcat-14.4
msgcat-14.5
msgcat-14.6
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1

serhiy.storchaka added on 2026-10-20 13:39:16:

Rechecked on main by running every test from the 2015 list alone. Only cmdMZ-return-3.* still depend on earlier tests: they read ::errorCode/::errorInfo, which don't exist until some earlier test has raised an error. Proposed fix in branch [acd6e13723|cmdMZ-return-standalone]: set both before the loop.

The tcltest-21.x/24.x/25.3 entries are not dependencies: these tests define nested tests named e.g. tcltest-21.4.0, which an exact -match filters out; with -match 'tcltest-21.4*' they pass. Everything else in the list passes alone now, so this can be closed with the cmdMZ fix.


jan.nijtmans added on 2026-10-21 10:50:15:

+1

Many thanks!


serhiy.storchaka added on 2026-10-21 10:58:57:

Merged into trunk [03c91ff6ac], backported to core-9-0-branch [8f4fc33974] and core-8-6-branch [eeeec49664].


Attachments: