Check-in [10c01a7f93]

{"tip": {
	"525":{"url":"./tip/525.md","keywords":"Tcl, testing, continuous integration","created":"24-Oct-2018","post-history":"","state":"Draft","tcl-version":"8.5","vote":"Pending","type":"Project","title":"# TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way","author":["Donal K. Fellows <[email protected]>"],"is-jest":false},
	"524":{"url":"./tip/524.md","keywords":"Tcl, object orientation, customization","created":"23-Oct-2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 524: Custom Definition Dialects for TclOO","author":["Donal K. Fellows <[email protected]>"],"is-jest":false},
	"523":{"url":"./tip/523.md","keywords":"queue, stack, pop","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-523","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 523: New lpop command","author":["Peter Spjuth <[email protected]>"],"is-jest":false},
	"522":{"url":"./tip/522.md","keywords":"tcltest, error codes","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-522","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 522: Test error codes with Tcltest","author":["Peter Spjuth <[email protected]>"],"is-jest":false},
	"521":{"url":"./tip/521.md","keywords":"Tcl, floating point, NaN, not a number","created":"21 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 521: Floating Point Classification Functions","author":["Kevin B. Kenny <[email protected]>"],"is-jest":false},
	"520":{"url":"./tip/520.md","keywords":"Tcl, floating point, NaN, not a number","created":"18 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 520: Make NaN Quiet","author":["Kevin B. Kenny <[email protected]>"],"is-jest":false},
	"519":{"url":"./tip/519.md","keywords":"TclOO","":"","state":"Draft","this tip proposes to extend the `method` subcommand of the `oo":":define` command","type":"Project","author":["Pietro Cerutti <[email protected]>"],"follows":"","oo":":define RPCClient {","created":"18-Oct-2018","title":"# TIP 519: Inline export/unexport option to TclOO method definition","the class definition in the previous example would then be as follows":"","post-history":"","vote":"Pending","tcl-version":"8.7","tcl-branch":"tip-519","is-jest":false},
	"518":{"url":"./tip/518.md","keywords":"Tk","created":"22-Sep-2018","post-history":"","state":"Draft","tk-branch":"tip518-event-last-child-unmanaged","tcl-version":"8.7","vote":"Done","type":"Project","title":"# TIP 518: Virtual Event when Last Child is not Managed any more","author":["Harald Oehlmann <[email protected]>"],"is-jest":false},

	"5":{"url":"./tip/5.md","created":"17-Oct-2000","post-history":"","state":"Final","vote":"Done","tcl-version":"8.4","type":"Project","title":"# TIP 5: Make TkClassProcs and TkSetClassProcs Public and Extensible","author":["Eric Melski <[email protected]>"],"is-jest":false},
	"4":{"url":"./tip/4.md","created":"26-Oct-2000","post-history":"","state":"Draft","vote":"Pending","type":"Informative","title":"# TIP 4: Tcl Release and Distribution Philosophy","discussions-to":"news:comp.lang.tcl","author":["Brent Welch <[email protected]>","Donal K. Fellows <[email protected]>","Larry W. Virden <[email protected]>","Larry W. Virden <[email protected]>"],"is-jest":false},
	"3":{"url":"./tip/3.md","created":"14-Sep-2000","post-history":"","state":"Accepted","vote":"Done","type":"Process","title":"# TIP 3: TIP Format","author":["Andreas Kupries <[email protected]>","Donal K. Fellows <[email protected]>"],"is-jest":false},
	"2":{"url":"./tip/2.md","created":"12-Sep-2000","post-history":"","state":"Draft","vote":"Pending","type":"Process","title":"# TIP 2: TIP Guidelines","author":["Andreas Kupries <[email protected]>","Donal K. Fellows <[email protected]>","Don Porter <[email protected]>","Mo DeJong <[email protected]>","Larry W. Virden <[email protected]>","Kevin Kenny <[email protected]>"],"is-jest":false},
	"1":{"url":"./tip/1.md","created":"14-Sep-2000","post-history":"","state":"Active","vote":"No voting","type":"Informational","title":"# TIP 1: TIP Index","author":["TIP Editor <[email protected]>"],"is-jest":false},
	"0":{"url":"./tip/0.md","created":"11-Dec-2000","post-history":"","state":"Final","vote":"Done","type":"Process","title":"# TIP 0: Tcl Core Team Basic Rules","author":["John Ousterhout <[email protected]>"],"is-jest":false},
	"@min": 0,
	"@max": 525
}, "@timestamp": 1540385393}

<th>Type</th>
<th>Tcl Version</th>
<th>Status</th>
<th>Title</th>
<th>Impl.</th>
</tr></thead><tbody>

<tr class='project projectdraft projectdraft85 project85'>
<td valign='top'><a href='./tip/525.md'>525</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.5</td>
<td valign='top'>Draft</td>
<td valign='top'># TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way</td>
<td></td>
</tr>
<tr class='project projectdraft projectdraft87 project87'>
<td valign='top'><a href='./tip/524.md'>524</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.7</td>
<td valign='top'>Draft</td>
<td valign='top'># TIP 524: Custom Definition Dialects for TclOO</td>
<td></td>

# TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way
	Author:         Donal K. Fellows <[email protected]>
	State:          Draft
	Type:           Project
	Vote:           Pending
	Created:        24-Oct-2018
	Post-History:
	Keywords:       Tcl, testing, continuous integration
	Tcl-Version:    8.5
-----

# Abstract

This TIP makes it easier to use tcltest in a continuous integration
environment (such as [Travis](https://travis-ci.org/) or
[Appveyor](https://www.appveyor.com/)) by making `tcltest::runAllTests` return
a boolean indicator of success or failure.

# Rationale and Specification

The Tcl test suite is great, but when a failure happens it reports success to
its environment (by effectively doing an `exit 0`) which mans that there's no
way to tell at the wider level whether the tests passed or failed except by
manually inspecting the logs, which is a manual process that prevents the Tcl
test suite from being used in an automated environment such as a continuous
integration system.

This TIP changes this by making `tcltest::runAllTests` return a boolean
(instead of the empty string) that reports whether the executed tests all
passed or not; the boolean is a true value if any test file fails, and false
if all tests pass.

Note that because we want to use this in continuous integration environments
with all main branches for which releases could be done in the future, the
change must be applied to Tcl 8.5 and 8.6 as well as our main development
targets.

## Usage

This is sufficient to allow the execution environment to report an overall
success or failure. This is done inside Tcl's test suite by changing:

    # require packages and configure...
    tcltest::runAllTests
	exit

to:

    # require packages and configure...
	set ErrorOnFailures [info exists env(ERROR_ON_FAILURES)]
	unset -nocomplain env(ERROR_ON_FAILURES)
	if {[tcltest::runAllTests] && $ErrorOnFailures} {
	    exit 1
	}

so that existing code and tests are not affected by the change. (A system has
to set the `ERROR_ON_FAILURES` environment variable in order to get this
additional behaviour at the outer level, so only an environment that is
prepared to deal with the consequences will notice.)

# Implementation

See the branches `travis-8.5`, `travis-8.6`, `travis-8.7` and `travis-9.0`;
this TIP is specifically altering the procdure `runAllTests` to include
exactly one more line at the end:

    return [expr {[info exists testFileFailures] || [llength $failFiles]}]

This is exactly the condition for Tcltest to report a failing test file (the
existence of `testFileFailures`) or a file with a failing test (`$failFiles`
being non-empty).

# Copyright

This document has been placed in the public domain.