|
2026-11-17
| ||
| 19:59 | Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: ca075b43 user: jan.nijtmans tags: core-8-6-branch | |
| 17:19 | Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: 4593d12d user: jan.nijtmans tags: core-9-0-branch | |
| 17:15 | Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran check-in: d18552b3 user: jan.nijtmans tags: trunk, main | |
| 15:19 | • Ticket [ead4782b] The test suite exits with status 0 when tests fail, so CI reports success status still Open with 3 other changes artifact: 530eba20 user: serhiy.storchaka | |
| 15:15 | • Ticket [ead4782b]: 4 changes artifact: 2a92f8a6 user: jan.nijtmans | |
| 14:53 | • Ticket [ead4782b]: 4 changes artifact: 402eac56 user: jan.nijtmans | |
| 14:50 | • Ticket [ead4782b]: 3 changes artifact: 4c17773c user: serhiy.storchaka | |
| 14:43 | • Ticket [ead4782b]: 3 changes artifact: bc9330da user: serhiy.storchaka | |
|
2026-11-16
| ||
| 10:27 | • Ticket [d0bceae2] Wayland branch: CI failures hidden by the constraints.tcl error status still Open with 3 other changes artifact: dd45ac4e user: serhiy.storchaka | |
| 09:58 | Fix [ead4782b08]: the test suite exited with status 0 when tests failed, so CI reported success, even when no test ran. Under Tk, tcltest exits with status 0 at the end of runAllTests, before it reports test files with errors and returns its result (Tcl ticket 7abc1460c0). all.tcl now prevents that exit and exits with the result of runAllTests when ERROR_ON_FAILURES is set; tests/all.tcl skips the ttk directory explicitly, since runAllTests now reaches it. The Linux and macOS workflows run both the classic and the ttk tests and fail if either exited with an error. On Windows, where the exit status is lost in the pipe through cat32, the workflow requires a summary with at least one test, no failures and no test file errors. (user: serhiy.storchaka) check-in: 6f50976f user: serhiy.storchaka tags: ci-test-exit-status | |
| 09:58 | • New ticket [ead4782b] The test suite exits with status 0 when tests fail, so CI reports success. artifact: 66ed8bdb user: serhiy.storchaka | |
| Ticket UUID: | ead4782b08b9e9cb013b70b51fdad4959ec5d567 | |||
| Title: | The test suite exits with status 0 when tests fail, so CI reports success | |||
| Type: | Bug | Version: | trunk, 9.0, 8.6 | |
| Submitter: | serhiy.storchaka | Created on: | 2026-11-16 09:58:05 | |
| Subsystem: | 86. Test Tools | Assigned To: | jan.nijtmans | |
| Priority: | 5 Medium | Severity: | Important | |
| Status: | Open | Last Modified: | 2026-11-17 15:19:04 | |
| Resolution: | None | Closed By: | nobody | |
| Closed on: | ||||
| Description: |
tests/all.tcl and tests/ttk/all.tcl end with if {[tcltest::runAllTests] && [info exists env(ERROR_ON_FAILURES)]} {exit 1}, but under Tk tcltest exits with status 0 inside runAllTests, after the summary and before it returns (Tcl ticket 7abc1460c0). So the test suite always ends with status 0, and the CI workflows only check that "Failed<tab>0" appears in the output. That also matches Total 0 Passed 0 Skipped 0 Failed 0, so when every test file failed at startup, as on the wayland branch until 2026-11-15 ([d0bceae242]), the jobs reported success. On Windows the exit status is lost anyway in the pipe through cat32. Proposed fix in branch ci-test-exit-status:
The workaround can be removed when Tk requires a Tcl with the fix of 7abc1460c0. | |||
| User Comments: |
serhiy.storchaka added on 2026-11-17 15:19:04:
Yes, [1f9358e23b] is as it should be. A check of the tcltest version is not needed: the change is harmless with the fixed tcltest, and it is what makes tests/all.tcl decide the exit status (1 only with ERROR_ON_FAILURES), with any tcltest. jan.nijtmans added on 2026-11-17 15:15:24: Ok, I restored it. It it now how it should be? jan.nijtmans added on 2026-11-17 14:53:53: Can we check for the "tcltest" version, deciding whether to use the workaround or not? serhiy.storchaka added on 2026-11-17 14:50:37: Jan, the change to tests/all.tcl is needed with the fixed tcltest too: without it a local make test fails whenever a test fails ([854cc390a7]). With it, tests/all.tcl exits with 1 only if ERROR_ON_FAILURES is set, as in Tcl, and it also works with older tcltest. serhiy.storchaka added on 2026-11-17 14:43:43: Jan, without the change in tests/all.tcl this does not work on Linux and macOS: under Tk, tcltest still exits with status 0 (7abc1460c0), and grep "Failed 0" also matches Total 0 ... Failed 0. Either keep the workaround until the fixed tcltest is used everywhere, or check the output on Linux and macOS as on Windows. | |||
