Tcl Source Code

View Ticket
Login
2026-10-22
17:19 • Ticket [fdcc860df5] Test suite tests not running independent status still Cl... artifact: d7df3b6311 user: serhiy.storchaka
2026-10-21
10:58 • Ticket [fdcc860df5]: 3 changes artifact: b9f6b4c7d4 user: serhiy.storchaka
10:57
Fix [fdcc860df5]: Test suite tests not running independent check-in: eeeec49664 user: serhiy.storchaka tags: core-8-6-branch
10:57
Fix [fdcc860df5]: Test suite tests not running independent check-in: 8f4fc33974 user: serhiy.storchaka tags: core-9-0-branch
10:56 • Closed ticket [fdcc860df5]: Test suite tests not running independent plus 2 ... artifact: 55d7a69482 user: serhiy.storchaka
10:55
Fix [fdcc860df5]: Test suite tests not running independent check-in: 03c91ff6ac user: serhiy.storchaka tags: trunk, main
10:50 • Ticket [fdcc860df5] Test suite tests not running independent status still Op... artifact: a2db953751 user: jan.nijtmans
2026-10-20
13:39 • Ticket [fdcc860df5]: 4 changes artifact: 517db79f93 user: serhiy.storchaka
13:38
Fix [fdcc860df5]: cmdMZ-return-3.* set ::errorCode and ::errorInfo first, so they pass when run alon... closed check-in: acd6e13723 user: serhiy.storchaka tags: cmdMZ-return-standalone
2016-07-29
19:10 • Ticket [77d58e3a7a] Tests doesn't run in isolation status still Closed w... artifact: 41b13fbb6d user: fvogel
2015-08-06
18:45 • Ticket [fdcc860df5] Test suite tests not running independent status still Op... artifact: 5c88266db7 user: pointsman
2014-11-13
23:21 • Add attachment utils.tcl to ticket [fdcc860df5] artifact: 1a2524630e user: pointsman
23:20 • Add attachment test-names.tcl to ticket [fdcc860df5] artifact: f6233617df user: pointsman
23:20 • Add attachment singletests.tcl to ticket [fdcc860df5] artifact: 3f4893b7a9 user: pointsman
23:20 • Ticket [fdcc860df5] Test suite tests not running independent status still Op... artifact: 44bf09c275 user: pointsman
14:55 • Ticket [fdcc860df5]: 3 changes artifact: e14c9982f3 user: dgp
00:15 • Ticket [fdcc860df5]: 4 changes artifact: 25599afcb0 user: pointsman
00:14 • New ticket [fdcc860df5]. artifact: c06beba8b4 user: pointsman

Ticket UUID: fdcc860df5b2ab01e5bbff0f83bb227e8dec54f7
Title: Test suite tests not running independent
Type: Bug Created on: 2014-11-13 00:14:25
Submitter: pointsman Assigned to: nobody
Subsystem: 69. Other Severity: Minor
Priority: 3 Low Last modified: 2026-10-22 17:19:32
Status: Closed Closed by: serhiy.storchaka
Resolution: Fixed Closed on: 2026-10-21 10:56:31
Version: 8.6.18, 9.0.5, 9.1.0
Description:
The long list below names 202 tests out of the tcl core test suite, that pass, if they run in the context of a full test suite run with e.g make test, but fail, if they run standalone, selected by the tcltest option -match, e.g.

./tcltest ../tests/all.tcl -match <unique_test_name>

(In practice, one would restrict the sourced test files with the help of the tcltest -file option to the one test file, that includes the test of interest, but that aside.)

In other words, the below named tests doesn't run independed from other tests in their test file.

The usual advise is: tests should be written in a way, that they run independed from other tests. This is not only a matter of taste. 

The - admitted small - impact comes into the picture, if someone tries to track down a memory leak, reported by a mem-checker (like valgrind) after a test suite run to a single test, that triggers the mem leak. 

One simple brute-force attempt for that would be, to bisec the tests in an affected test file with an appropriate -match option, look which part of the tests still show the leak, repeate the bisec on this part and so on until a single test is found, that demonstrates the problem. (Either 'by hand' on the cmd line or with a helper script.)

Having 'false' test failure and (at least in the case of tests with script error) other executed code paths depending on how you range the tests to run makes this process even more tedious, as it already is and harder to write a reliable script, to automate this task.

The following list was generaded and checked on a linux system, using 8.6.3, with just standard ./configure 

The tests not running sucessfully (either wrong result or script error in the body) when run independ from others are: 

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
cmdMZ-return-3.0
cmdMZ-return-3.1
cmdMZ-return-3.3
cmdMZ-return-3.8
cmdMZ-return-3.9
cmdMZ-return-3.10
cmdMZ-return-3.11
cmdMZ-return-3.12
cmdMZ-return-3.13
cmdMZ-return-3.14
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
safe-14.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1
User Comments:
dgp added on 2014-11-13 14:55:04:
A great contribution.  Thank you!

I'm sure the "tool" to build this list is simple,
but it still might be worth sharing around.
This test suite hygiene task is one many
projects could benefit from.

pointsman added on 2014-11-13 23:20:16:
This results are collected by one of a few test suite tool scripts of mine. You'll find them hosted at

http://46.163.78.80/cgi-bin/repros/test-suite-tools/dir?ci=tip

You'll need singletests.tcl, utils.tcl and test-names.tcl. Since that url (although stable since years) may not last forever, I'll attatch them in the current state.

Or what other way of sharing around did you have in mind for such exotic stuff?

pointsman added on 2015-08-06 18:45:52:
This is the current list of not independend tests as of 8931dc163c0e 2015-08-06 04:25:50 UTC. Compared to the original report: A few tests are fixed (mostly in cmdMZ.test), a few got added, mostly the status is unchanged.

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
http-4.15
http11-2.4
http11-2.11
http11-4.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
interp-14.1
interp-36.7
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
msgcat-14.1
msgcat-14.2
msgcat-14.3
msgcat-14.4
msgcat-14.5
msgcat-14.6
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1

serhiy.storchaka added on 2026-10-20 13:39:16:

Rechecked on main by running every test from the 2015 list alone. Only cmdMZ-return-3.* still depend on earlier tests: they read ::errorCode/::errorInfo, which don't exist until some earlier test has raised an error. Proposed fix in branch [acd6e13723|cmdMZ-return-standalone]: set both before the loop.

The tcltest-21.x/24.x/25.3 entries are not dependencies: these tests define nested tests named e.g. tcltest-21.4.0, which an exact -match filters out; with -match 'tcltest-21.4*' they pass. Everything else in the list passes alone now, so this can be closed with the cmdMZ fix.


jan.nijtmans added on 2026-10-21 10:50:15:

+1

Many thanks!


serhiy.storchaka added on 2026-10-21 10:58:57:

Merged into trunk [03c91ff6ac], backported to core-9-0-branch [8f4fc33974] and core-8-6-branch [eeeec49664].


Attachments: