Author:         Donal K. Fellows <[email protected]>
State:          Final
Type:           Project
Vote:           Done
Created:        24-Oct-2018
Post-History:
Keywords:       Tcl, testing, continuous integration
Tcl-Version:    8.5
Vote-Results:	8/0/0 accepted
Votes-For:	KBK, SL, AF, DGP, FV, DKF, AK, JN
Votes-Against:	none
Votes-Present:	none

---

Abstract

This TIP makes it easier to use tcltest in a continuous integration environment (such as Travis or Appveyor) by making tcltest::runAllTests return a boolean indicator of success or failure.

Rationale and Specification

The Tcl test suite is great, but when a failure happens it reports success to its environment (by effectively doing an exit 0) which means that there's no way to tell at the wider level whether the tests passed or failed except by manually inspecting the logs, which is a manual process that prevents the Tcl test suite from being used in an automated environment such as a continuous integration system.

This TIP changes this by making tcltest::runAllTests return a boolean (instead of the empty string) that reports whether the executed tests all passed or not; the boolean is a true value if any test file fails, and false if all tests pass.

Note that because we want to use this in continuous integration environments with all main branches for which releases could be done in the future, the change must be applied to Tcl 8.5 and 8.6 as well as our main development targets.

Usage

This is sufficient to allow the execution environment to report an overall success or failure. This is done inside Tcl's test suite by changing:

# require packages and configure...
tcltest::runAllTests
exit

to:

# require packages and configure...
set ErrorOnFailures [info exists env(ERROR_ON_FAILURES)]
unset -nocomplain env(ERROR_ON_FAILURES)
if {[tcltest::runAllTests] && $ErrorOnFailures} {
    exit 1
}

so that existing code and tests are not affected by the change. (A system has to set the ERROR_ON_FAILURES environment variable in order to get this additional behaviour at the outer level, so only an environment that is prepared to deal with the consequences will notice.)

It is expected that new code with little need for very strict backward compatibility will instead just do:

# require packages and configure...
exit [tcltest::runAllTests]

Implementation

See the branches travis-8.5, travis-8.6, travis-8.7 and travis-9.0; this TIP is specifically altering the procedure runAllTests to include exactly one more line at the end:

return [expr {[info exists testFileFailures] || [llength $failFiles]}]

This is exactly the condition for Tcltest to report a failing test file (the existence of testFileFailures) or a file with a failing test ($failFiles being non-empty).

Copyright

This document has been placed in the public domain.