TIP:            85
Title:          Custom Comparisons in Tcltest
Version:        $Revision: 1.14 $
Author:         Arjen Markus <arjen.markus@wldelft.nl>
Author:         Don Porter <dgp@users.sf.net>
State:          Final
Type:           Project
Vote:           Done
Created:        31-Jan-2002
Post-History:   
Keywords:       test,string comparison,floating-point
Tcl-Version:    8.4

~ Abstract

This TIP proposes a simple mechanism to make the ''tcltest'' package
an even more flexible package than it already is by allowing the
programmer to define his or her own comparison procedures.  Such
procedures can deal with issues like allowing a (small) tolerance in
floating-point results.

~ Rationale

The ''test'' command of the package ''tcltest 2.0'' supports the
comparison of the actual result with the expected result by a number
of methods: exact matching, glob-style matching and matching via a
regular expression, according to the ''-match'' option.  The flexibility
is indeed enhanced over the package ''tcltest 1.0,'' as it is now much
easier to allow for small variations in ''string'' results.  But
it is nearly impossible to define an accurate test that checks if
floating-point results are the "same" - exact matching will seldom
suffice due to platform-specific round-off errors or differences in
formatting a floating-point number (''0.12'' versus ''.12'' for
instance).

It is also impossible to compare results that are not easily expressed
as strings, for instance an application that produces binary files
that need to be compared or simply very long strings - these could
easily be stored in an external file, but would be awkward in a file
with a large number of such tests.

~ Proposal

The package ''tcltest 2.0.2'' defines an internal comparison procedure,
''CompareStrings'' that performs matching according to the three built-in
''-match'' options of ''test''.  This
procedure can easily be replaced by one that invokes registered 
commands or procedures. Such a command or procedure takes two 
arguments and returns 1 for a match and a 0 for failure, 
just as ''CompareStrings'' does in the current implementation:

| proc myMatchProc { expected actual } { 
|   if { $expected (is somehow equal) $actual } {
|      return 1
|   } else
|      return 0
|   }
| }

A new public command ''customMatch'' is proposed for the purpose
of registering these matching commands.  It can register a procedure,
such as ''myMatchProc'' defined above:

| ::tcltest::customMatch mytype myMatchProc

or, as in the sample implementation, an incomplete command:

| ::tcltest::customMatch exact [list ::string equal]

When the ''test'' command is called with the ''-match mytype'' option,
the command ''myMatchProc'' will be completed with two arguments,
the expected and actual results, and will be evaluated in the global
namespace to determine whether the test result matches the expected
result.  Likewise, the ''test'' option ''-match exact'' will
cause matching to be tested by the command ''::string equal''.
The default value of the ''-match'' option will continue to be ''exact''.

Allowing procedures to be invoked by their type names gives us the 
flexibility to register as many such procedures or commands as required.

Because this proposal adds a new public command to the ''tcltest''
package, the version will be incremented to 2.1.

A patch to the current HEAD that implements this proposal is
available as Tcl Patch 521362 at the Tcl project at SourceForge.
http://sf.net/tracker/?func=detail&aid=521362&group_id=10894&atid=310894

~ Two Examples

To show how this works, we include two simple examples:

 * Testing a package for calculating mathematical functions like
   Bessel functions.

 * Testing for negative results, as when providing an alternative, but
   incompatible implementation of a feature.

First, suppose you have defined a package for calculating the value of
a general Bessel function, just the sort of function that returns
floating-point numbers.  Then the results may be imprecise due to
rounding-off errors, different values of ''tcl_precision'' or, even
more banally, differences in the formatting of floating-point numbers
(''0.12'' versus ''.12'' for instance). 

The following shows how to do this:

| #
| # Test implementation of Bessel functions
| # (Table only provides 4 decimals)
| #
| customMatch 4decimals matchFloat4Decimals
|
| proc matchFloat4Decimals { expected actual } {
|    return [expr {abs($expected-$actual) <= 0.5e-4}]
| }
|
| test "J0-1.1" "J0 for x=1.0" -match 4decimals -body {
|    J0 1.0
| } -result 0.7652
|
| test "J1-1.1" "J0 for x=1.0" -match 4decimals -body {
|    J1 1.0
| } -result 0.4401

The second example occurs for instance when testing alternative
implementations: you want to check that the original standard feature
is failing whereas the new but incompatible alternative gets it right.
Then:

| proc matchNegative { expected actual } {
|    set match 0
|    foreach a $actual e $expected {
|       if { $a != $e } {
|          set match 1
|          break
|       }
|    }
|    return $match
| }
|
| customMatch negative matchNegative
|
| #
| # Floating-point comparisons are imprecise. The following
| # test returns typically such a list as {643 1357 1921 79 781 1219}
| # so nothing even close to the expected values.
| # 
| test "ManyCompares-1.2" "Compare fails - naive comparison" \
|    -match negative -body {
|    set naiv_eq 0
|    set naiv_ne 0
|    set naiv_ge 0
|    set naiv_gt 0
|    set naiv_le 0
|    set naiv_lt 0
|
|    for { set i -1000 } { $i <= 1000 } { incr i } {
|       if { $i == 0 } continue
|
|       set x [expr {1.01/double($i)}]
|       set y [expr {(2.1*$x)*(double($i)/2.1)}]
|
|       if { $y == 1.01 } { incr naiv_eq }
|       if { $y != 1.01 } { incr naiv_ne }
|       if { $y >= 1.01 } { incr naiv_ge }
|       if { $y >  1.01 } { incr naiv_gt }
|       if { $y <= 1.01 } { incr naiv_le }
|       if { $y <  1.01 } { incr naiv_lt }
|    }
|    set result [list $naiv_eq $naiv_ne $naiv_ge $naiv_gt $naiv_le $naiv_lt]
| } -result {2000 0 2000 0 2000 0}

makes sure that a mismatch is treated as the expected outcome.

~ Alternatives and objections

Of course, it is possible to achieve these effects within the current
framework of ''tcltest'', by putting these match procedures inside the
body of the test case. No extra user command would be necessary then.

There are at least two drawbacks to this approach:

 * The result against which we want to match is hidden in the code

 * If the test fails, the actual result is not printed (at least not
   by the ''tcltest'' framework).

As a matter of fact, the proposed mechanism actually simplifies the 
current implementation of the three match types to a certain degree by 
turning a switch between the three types into an array index.

~ See Also

Tcl Feature Request 490298.
http://sf.net/tracker/?func=detail&aid=490298&group_id=10894&atid=360894

~ History

''Cameron Laird'' was quite enthousiastic about the idea of providing 
custom match procedures.

''Mo DeJong'' requested the explicit examples (the second is actually 
the situation that triggered this TIP in the first place).

''Don Porter <dgp@users.sf.net>'' revised the registration mechanism 
such that an arbitrary set of matching commands or procedures can be supported. His suggestions led to a revision of the TIP. He also 
revised the draft implementation.

~ Copyright

This document is placed in the public domain.
