Tk Source Code

View Ticket
Login
2026-11-17
19:59
Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: ca075b43 user: jan.nijtmans tags: core-8-6-branch
17:19
Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: 4593d12d user: jan.nijtmans tags: core-9-0-branch
17:15
Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: d18552b3 user: jan.nijtmans tags: trunk, main
15:19 • Ticket [ead4782b] The test suite exits with status 0 when tests fail, so CI reports success status still Open with 3 other changes artifact: 530eba20 user: serhiy.storchaka
15:15 • Ticket [ead4782b]: 4 changes artifact: 2a92f8a6 user: jan.nijtmans
14:53 • Ticket [ead4782b]: 4 changes artifact: 402eac56 user: jan.nijtmans
14:50 • Ticket [ead4782b]: 3 changes artifact: 4c17773c user: serhiy.storchaka
14:43 • Ticket [ead4782b]: 3 changes artifact: bc9330da user: serhiy.storchaka
2026-11-16
10:27 • Ticket [d0bceae2] Wayland branch: CI failures hidden by the constraints.tcl error status still Open with 3 other changes artifact: dd45ac4e user: serhiy.storchaka
09:58
Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran. Under Tk, tcltest exits with status 0 at the end of runAllTests, before it reports test files with errors and returns its result (Tcl ticket 7abc1460c0). all.tcl now prevents that exit and exits with the result of runAllTests when ERROR_ON_FAILURES is set; tests/all.tcl skips the ttk directory explicitly, since runAllTests now reaches it. The Linux and macOS workflows run both the classic and the ttk tests and fail if either exited with an error. On Windows, where the exit status is lost in the pipe through cat32, the workflow requires a summary with at least one test, no failures and no test file errors. (user: serhiy.storchaka) check-in: 6f50976f user: serhiy.storchaka tags: ci-test-exit-status
09:58 • New ticket [ead4782b] The test suite exits with status 0 when tests fail, so CI reports success. artifact: 66ed8bdb user: serhiy.storchaka

Ticket UUID: ead4782b08b9e9cb013b70b51fdad4959ec5d567
Title: The test suite exits with status 0 when tests fail, so CI reports success
Type: Bug Version: trunk, 9.0, 8.6
Submitter: serhiy.storchaka Created on: 2026-11-16 09:58:05
Subsystem: 86. Test Tools Assigned To: jan.nijtmans
Priority: 5 Medium Severity: Important
Status: Open Last Modified: 2026-11-17 15:19:04
Resolution: None Closed By: nobody
    Closed on:
Description:

tests/all.tcl and tests/ttk/all.tcl end with if {[tcltest::runAllTests] && [info exists env(ERROR_ON_FAILURES)]} {exit 1}, but under Tk tcltest exits with status 0 inside runAllTests, after the summary and before it returns (Tcl ticket 7abc1460c0). So the test suite always ends with status 0, and the CI workflows only check that "Failed<tab>0" appears in the output. That also matches Total 0 Passed 0 Skipped 0 Failed 0, so when every test file failed at startup, as on the wayland branch until 2026-11-15 ([d0bceae242]), the jobs reported success. On Windows the exit status is lost anyway in the pipe through cat32.

Proposed fix in branch ci-test-exit-status:

  • all.tcl keeps tcltest from exiting after the last test file (with the "interactive" constraint, which works with old and new tcltest), so that runAllTests reports the test files with errors and returns its result, and all.tcl exits with status 1 on failures when ERROR_ON_FAILURES is set. The ttk directory is skipped explicitly, since runAllTests now reaches its subdirectories.
  • The Linux and macOS workflows run both the classic and the ttk tests and fail if either exited with an error.
  • The Windows workflow requires a summary with at least one test, no failures and no test file errors.

The workaround can be removed when Tk requires a Tcl with the fix of 7abc1460c0.

User Comments: serhiy.storchaka added on 2026-11-17 15:19:04:

Yes, [1f9358e23b] is as it should be. A check of the tcltest version is not needed: the change is harmless with the fixed tcltest, and it is what makes tests/all.tcl decide the exit status (1 only with ERROR_ON_FAILURES), with any tcltest.


jan.nijtmans added on 2026-11-17 15:15:24:

Ok, I restored it. It it now how it should be?


jan.nijtmans added on 2026-11-17 14:53:53:

Can we check for the "tcltest" version, deciding whether to use the workaround or not?


serhiy.storchaka added on 2026-11-17 14:50:37:

Jan, the change to tests/all.tcl is needed with the fixed tcltest too: without it a local make test fails whenever a test fails ([854cc390a7]). With it, tests/all.tcl exits with 1 only if ERROR_ON_FAILURES is set, as in Tcl, and it also works with older tcltest.


serhiy.storchaka added on 2026-11-17 14:43:43:

Jan, without the change in tests/all.tcl this does not work on Linux and macOS: under Tk, tcltest still exits with status 0 (7abc1460c0), and grep "Failed 0" also matches Total 0 ... Failed 0. Either keep the workaround until the fixed tcltest is used everywhere, or check the output on Linux and macOS as on Windows.