Author:         Colin Macleod <[email protected]>
State:          Rejected
Type:           Project
Tcl-Version:    9.1
Created:	23-Jun-2023
Vote:           Done
    Vote-Summary:   Rejected 3/3/2
    Votes-For:      HO, KW, SL
    Votes-Against:  AK, DF, RA
    Votes-Present:  AN, MC

---

Abstract

This TIP proposes adding a new command = which will allow numeric and boolean calculations to be written in a more compact form than with expr. The = command will evaluate expressions similarly to expr but will treat bare words as variable references. Since expressions can be written without $ there is no need to brace the expression to prevent double substitution. This will permit numeric computations to be written within other commands in a more compact and natural form than at present, with no modification to Tcl's parsing rules.

Rationale

Most newcomers to Tcl, and some oldies, find expr awkward. The requirement to brace expressions for safety and performance leads to e.g. a canvas command with computed coordinates looking like:

.canvas addtag enclosed [expr {$x-20}] [expr {$x+20}] \
            [expr {$y-20}] [expr {$y+20}]

The wiki page https://wiki.tcl-lang.org/page/expr+shorthand+for+Tcl9 records many suggestions for a more compact syntax, and various related TIPs have been created. This shows that the issue has been a concern for many years. However all of these proposals involve changing the basic Tcl parsing rules (the dodekalogue), which has a major impact in terms of extra complexity and backward compatibility. This TIP aims to allow such inline expressions to be as concise as possible without changing Tcl's parsing rules. The effect will be to allow the canvas command above to be written as:

.canvas addtag enclosed [= x-20] [= x+20] [= y-20] [= y+20]

Here the = commands will treat identifiers x and y as variable names and use their variable values in the calculation.

It is not practical to write string literals in unbraced expressions, so string operations would not be supported.
But many uses of expr are for simple numeric calculations where strings are not involved but brevity is desirable. The standard expr would still be available for use in other cases.

An alternative which already exists for inline calculations is to use operations from the mathop namespace in prefix form. However this is rather obscure to people who are not Tcl experts, and becomes awkward if several different operators need to be combined.

Specification

The = command will have syntax:

• = arg ?arg arg ...?

It will evaluate expressions in a way which is compatible with expr with the following differences:

• Only numerical values can be written literally in the expression.

• Any substring of the expression which starts with an alphabetic character and continues with alphanumeric characters, underscores or colons and does not terminate with ( will be treated as a variable name and its value will be substituted into the expression.

• If such a substring does terminate with ( it will be treated as the name of a function in the tcl::mathfunc namespace and should be followed by any arguments it takes, separated by commas and terminated by ). If such a function does not exist an error will be reported.

• No other variable or command substitutions will be performed by the = command itself. Normally the expression passed to = would be unbraced, so $ and [] can be used to have such substitutions done before = is invoked. Note that array elements can only be used by being pre-substituted in this way as they would otherwise be interpreted as mathfunc calls.

• The same operators as expr can be used except for the list and pure-string operators in ni eq ne lt gt le ge as they would be interpreted as variable names.

Options

Eric Taylor has suggested that a subexpression like some(thing) could be interpreted as either a mathfunc call or an array reference depending on whether that mathfunc or array exists. I feel this could create confusion and strange bugs, with the meaning of the expression changing depending on what mathfunc or array definitions may have been made elsewhere in the program.

In expr boolean operations accept operands in various non-numeric forms like true, off, y, no. In = these could be confused with variable names, so I feel it's safer to disallow them. However they could be permitted, but with a variable of the same name taking priority.

In expr the logical operations && || ?: perform lazy evaluation, skipping arguments which can be seen to be not needed. There is perhaps less need for this with = as it will not do its own command substitution, but since user-defined mathfuncs could perform complex operations with side-effects I have implemented this lazy evaluation.

Discussion

Previous drafts of this TIP proposed having all variables pre-substituted by writing them with $ and having no substitutions done within the = command. Kevin Kenny on tcl-core mentioned the idea of using barewords as variable references, and I then realised that this enables a neater, more compact form of expression while also avoiding double-substitution.

There may be some existing code which defines its own = command. Since Tcl allows any command to be redefined, such code will continue to work, it will just not have access to the new = command defined here.

Some proposals such as TIP 282 suggest extending expr to support making assignments and/or doing multiple calculations possibly returned as a list of values. This proposal does not include such functionality, but it could be added as a future enhancement. Personally I think that if an individual calculation can be expressed concisely then there is less need for such complications.

Some of the string and list-related commands already support a restricted form of numeric expressions for indices, e.g. [lindex $list end-2]. This could be generalised in future by having them temporarily define end as the appropriate value and run the = processing on index arguments.

René Zaumseil asked how a newline in the arguments would be treated. As with any other Tcl command, an unquoted and unescaped newline will terminate the command, with the following line treated as a separate command. A quoted or escaped newline will just be treated as whitespace by = and ignored, except that individual operators and operands in the expression cannot be split by whitespace.

Examples

Setting a variable:

    set bright [= red*0.3 + green*0.59 + blue*0.11]
    set x [= radius * cos(angle)]

Use with an image command:

    my_img put $shade -to [= left+i] $top [= left+i+1] $bottom

For a more extensive set of examples, see https://cmacleod.me.uk/tcl/tip676.example.ng2.pdf. This is a diff -u between versions of the main script in my "Newsgrouper" project before and after converting all the suitable uses of expr to =, i.e. those not involving string values.

Implementation

Code for this functionality is on branch cgm-equals-command in the Tcl core repository. This calls the existing expr code for lexical analysis of the expression, then uses a simple "Pratt" parser which directly generates Tcl bytecode. Most expressions can be fully byte-compiled, falling back to the uncompiled implementation for the exceptions. This gives performance similar to expr.

Array variables and command substitutions can only be used in = expressions by being pre-substituted before = runs. However it is still possible to generate efficient code for these cases by assuming that they will produce numeric values and then wrapping this in a runtime check that this is correct, falling back to the non-compiled implementation when this fails.

I had previously written a prototype in Tcl, which was very helpful in working out the design: https://cmacleod.me.uk/tcl/expr_ng2.

There is a good set of tests (471 cases), many of which were contributed by Eric Taylor.

Copyright

This document has been placed in the public domain.