Tk Source Code

View Ticket
Login
Ticket UUID: ead4782b08b9e9cb013b70b51fdad4959ec5d567
Title: The test suite exits with status 0 when tests fail, so CI reports success
Type: Bug Version: trunk, 9.0, 8.6
Submitter: serhiy.storchaka Created on: 2026-11-16 09:58:05
Subsystem: 86. Test Tools Assigned To: jan.nijtmans
Priority: 5 Medium Severity: Important
Status: Open Last Modified: 2026-11-17 15:19:04
Resolution: None Closed By: nobody
    Closed on:
Description:

tests/all.tcl and tests/ttk/all.tcl end with if {[tcltest::runAllTests] && [info exists env(ERROR_ON_FAILURES)]} {exit 1}, but under Tk tcltest exits with status 0 inside runAllTests, after the summary and before it returns (Tcl ticket 7abc1460c0). So the test suite always ends with status 0, and the CI workflows only check that "Failed<tab>0" appears in the output. That also matches Total 0 Passed 0 Skipped 0 Failed 0, so when every test file failed at startup, as on the wayland branch until 2026-11-15 ([d0bceae242]), the jobs reported success. On Windows the exit status is lost anyway in the pipe through cat32.

Proposed fix in branch ci-test-exit-status:

  • all.tcl keeps tcltest from exiting after the last test file (with the "interactive" constraint, which works with old and new tcltest), so that runAllTests reports the test files with errors and returns its result, and all.tcl exits with status 1 on failures when ERROR_ON_FAILURES is set. The ttk directory is skipped explicitly, since runAllTests now reaches its subdirectories.
  • The Linux and macOS workflows run both the classic and the ttk tests and fail if either exited with an error.
  • The Windows workflow requires a summary with at least one test, no failures and no test file errors.

The workaround can be removed when Tk requires a Tcl with the fix of 7abc1460c0.

User Comments: serhiy.storchaka added on 2026-11-17 15:19:04:

Yes, [1f9358e23b] is as it should be. A check of the tcltest version is not needed: the change is harmless with the fixed tcltest, and it is what makes tests/all.tcl decide the exit status (1 only with ERROR_ON_FAILURES), with any tcltest.


jan.nijtmans added on 2026-11-17 15:15:24:

Ok, I restored it. It it now how it should be?


jan.nijtmans added on 2026-11-17 14:53:53:

Can we check for the "tcltest" version, deciding whether to use the workaround or not?


serhiy.storchaka added on 2026-11-17 14:50:37:

Jan, the change to tests/all.tcl is needed with the fixed tcltest too: without it a local make test fails whenever a test fails ([854cc390a7]). With it, tests/all.tcl exits with 1 only if ERROR_ON_FAILURES is set, as in Tcl, and it also works with older tcltest.


serhiy.storchaka added on 2026-11-17 14:43:43:

Jan, without the change in tests/all.tcl this does not work on Linux and macOS: under Tk, tcltest still exits with status 0 (7abc1460c0), and grep "Failed 0" also matches Total 0 ... Failed 0. Either keep the workaround until the fixed tcltest is used everywhere, or check the output on Linux and macOS as on Windows.