| Ticket UUID: | ead4782b08b9e9cb013b70b51fdad4959ec5d567 | |||
| Title: | The test suite exits with status 0 when tests fail, so CI reports success | |||
| Type: | Bug | Version: | trunk, 9.0, 8.6 | |
| Submitter: | serhiy.storchaka | Created on: | 2026-11-16 09:58:05 | |
| Subsystem: | 86. Test Tools | Assigned To: | jan.nijtmans | |
| Priority: | 5 Medium | Severity: | Important | |
| Status: | Open | Last Modified: | 2026-11-17 15:19:04 | |
| Resolution: | None | Closed By: | nobody | |
| Closed on: | ||||
| Description: |
tests/all.tcl and tests/ttk/all.tcl end with if {[tcltest::runAllTests] && [info exists env(ERROR_ON_FAILURES)]} {exit 1}, but under Tk tcltest exits with status 0 inside runAllTests, after the summary and before it returns (Tcl ticket 7abc1460c0). So the test suite always ends with status 0, and the CI workflows only check that "Failed<tab>0" appears in the output. That also matches Total 0 Passed 0 Skipped 0 Failed 0, so when every test file failed at startup, as on the wayland branch until 2026-11-15 ([d0bceae242]), the jobs reported success. On Windows the exit status is lost anyway in the pipe through cat32. Proposed fix in branch ci-test-exit-status:
The workaround can be removed when Tk requires a Tcl with the fix of 7abc1460c0. | |||
| User Comments: |
serhiy.storchaka added on 2026-11-17 15:19:04:
Yes, [1f9358e23b] is as it should be. A check of the tcltest version is not needed: the change is harmless with the fixed tcltest, and it is what makes tests/all.tcl decide the exit status (1 only with ERROR_ON_FAILURES), with any tcltest. jan.nijtmans added on 2026-11-17 15:15:24: Ok, I restored it. It it now how it should be? jan.nijtmans added on 2026-11-17 14:53:53: Can we check for the "tcltest" version, deciding whether to use the workaround or not? serhiy.storchaka added on 2026-11-17 14:50:37: Jan, the change to tests/all.tcl is needed with the fixed tcltest too: without it a local make test fails whenever a test fails ([854cc390a7]). With it, tests/all.tcl exits with 1 only if ERROR_ON_FAILURES is set, as in Tcl, and it also works with older tcltest. serhiy.storchaka added on 2026-11-17 14:43:43: Jan, without the change in tests/all.tcl this does not work on Linux and macOS: under Tk, tcltest still exits with status 0 (7abc1460c0), and grep "Failed 0" also matches Total 0 ... Failed 0. Either keep the workaround until the fixed tcltest is used everywhere, or check the output on Linux and macOS as on Windows. | |||
Home
Timeline
Branches
Tags
Forum
Tickets
Download
Wiki