TIP 759: Arithmetic Expression Substitution and Compilation

Login
Core Home Timeline Branches Tags Forum Help 
Author:  Florent MERLET <[email protected]>
State:          Draft
Type:           Project
Vote:           Pending
Created:        18-Aug-2026
Tcl-Version:    9.2
Tcl-Branch:     tip-759_tcl90
Keywords:       tcl, expr, operator, assignment, arithmetic, substitution
Version:  $Revision: 2.2 $

Abstract

    This project aims to facilitate arithmetic calculations in Tcl.

Rationale

    Improving the readability of arithmetic calculations is a long term demand in Tcl. One of this first demand was TIP 282. Periodically, expansive discussions about it occur on the wiki. There were many suggestions to make things more practical : TIP 676, TIP 674, TIP 672.

A. Specifications

    The features offered fall into three categories:

  1. Extensions to the expr language.
    1. An ability to assign values ​​to variables (TIP 282 Assignment operator)
    2. An ability to chain calculations (TIP 282 Separating operator)
    3. Simplifying the definition of lists of numbers (Parentheses-comma list operator)
    4. Recognizing arithmetic substitution (Arithmetic TOKEN)
    5. Eventually (see part E below), recognizing some commands to control the flow (for, while, foreach,lmap,...)
  2. More direct access from the command line (Arithmetic substitution)
  3. An ability to define arithmetic scripts : (Arithmetic compilation)

A.1 Extensions to the expr language

A.2 Arithmetic substitution

    At the Tcl parser level, as well as into the expr parser, any word begining by [( and finishing by )] will be subject to an arithmetic substitution : Tcl will produce a value according to the arithmetic language of the expr command.

Synopsis :

        set A [( *expression* )]

This corresponds to this new rule of substitution :

  • Arithmetic substitution : If the first caracter of a word is an opening bracket immediately followed by an opening parenthesis, then Tcl performs an Arithmetic substitution. The expression has to follow the rules of the expr mini-language. The word must finish at the next balanced closing-parenthesis wich is immediately followed by a closing-bracket.

Corner cases :

It is very important to note that not only the expression must begin and end with parentheses,
but that end is at the next balanced closing-parenthesis immediately followed by a closing bracket.
Ex :
# This is correct but adds a superfluous level of parentheses :
set A [((3*4)+(2*5))]
# These are correct too :
# -> even if * or && or ; are not enclosed by parentheses since the last closing parenthesis is balanced and followed by a closing bracket, it works.
set A [(3+4)*(2+5)]; # 49
set B [(a=1);(b=0);(c=0);(d=0)];# 0
set C [($a || $b) && ($c || $d)];# 0

# But this is not OK :
set A [(3*4),(2*5)]; # unexpected "," outside function argument list in expression "(3*4),(2*5)]"

A.3 Arithmetic scripts

    Any string subject to compilation that begins with an opening parenthesis ( and ends with a closing parenthesis ) will be bytecompiled according to the arithmetic language of the expr command.

This new feature shall be mentioned in the documentation for each of the Tcl commands that compiles some of their arguments (like after, apply, catch, dict filter, dict for, dict map, dict update, dict with, eval, for, foreach, if, interp eval, lmap, method, namespace code, namespace eval, namespace inscope, namespace unknown, proc, switch, try, while,... ), as well as for Tk commands (bind, canvas bind, console eval, consoleinterp eval, entry -vcmd, ttk::entry -validatecommand,...)

Synopsis :

        eval {( *expression* )}

Corner cases :

Refer to Arithmetic Substitution above.

B. Examples of use

B.1 Extensions to the expr language examples

B.2 Arithmetic substitution examples

B.3 Arithmetic script examples

C Implementation details

Current state

The code volume is currently approximately 250 lines.

C.1 Extensions to the expr language implementation details

C.2 Arithmetic substitution implementation details

C.3 Arithmetic script examples implementation details

D. Justification of the choices made

D.1 Extensions to expr language

Why allow the left hand side of assignement operator to be a bareword ?

Why choose a list operator of the form ( , )?

The reason lies in the following analogy:

Why not simply use a list function?

The reason is partly aesthetic and partly logical: an infix language based on binary operators is naturally more inclined toward symmetry than a prefix language.

{{1 2 3} {4 5 6} {7 8 9}};# symetric
[((1,2,3),(4,5,6),(7,8,9))]; # symetric
[(list((1,2,3),(4,5,6),(7,8,9)))];# assymetric

That said, the list func can also be used, since it has been included into the table of math function. But this will be longer, and, IMHO, less clear.

To make a comparaison, let's consider this example, taken from tcl lib math exact package. In the Object class TanWorker, we have this (long) command :

next [list \
	  [list \
	       [list [expr {2*$n + 1}] [expr {2*$n + 3}]] \
	       [list [expr {2*$n - 1}] [expr {2*$n + 1}]]] \
	  [list \
	       [list [expr {2*$n + 1}] [expr {2*$n - 1}]] \
	       [list [expr {2*$n + 3}] [expr {2*$n + 1}]]]] 0

With this TIP, using list function, we can write it : 

next [(list(
   list(list(2*$n + 1, 2*$n + 3), list(2*$n - 1, 2*$n + 1)),
   list(list(2*$n + 1, 2*$n - 1), list(2*$n + 3,2*$n + 1))
   ))] 0
Whereas, using parentheses-comma operator, we can write : 
next [(
    ((2*$n + 1, 2*$n + 3), (2*$n - 1, 2*$n + 1)),
    ((2*$n + 1, 2*$n - 1 ), (2*$n + 3, 2*$n + 1))
)] 0

To define clearly the data, in usual Tcl, we need to add all these 140 chars : 

[list \ [list \ [list [expr {}] [expr {}]] \ [list [expr {}] [expr {}]]] \ [list \ [list [expr {}] [expr {}]] \ [list [expr {}] [expr {}]]]]
With this TIP, using the list function, we'll only need to add these 62 chars (NB : some syntactic unnecessary extra-spaces have been added in it solely for readability). 
[(list( list(list(, ), list(, )), list(list(, ), list(, )) ))]
Whereas, with this TIP, but using the parentheses-comma operator instead, we'll only need to add these 32 chars (NB : some syntactic unnecessary extra-spaces have been added in it solely for readability). 
[( ((, ), (, )), ((, ), (, )) )]

This makes it possible to measure the gain in conciseness offered by the parentheses-comma operator. In this specific case drawn from a Tcl library, the saving amounts to 108 characters compared to code using command syntax, and 30 characters compared to code using arithmetic syntax, but with the list function.

Why not having choosen to allow a comma to follow another comma ?

In expr language extension, a sequence of semi-colons is made to not be a syntax error :

Indeed, expr {;;;;} should not complain, since it is just a sequence of null operations.

I was thinking about allowing the same for comma, in the list operator context, so that expr {(,,)} would set the list {{} {} {}}

But I didn't do it. First, I was hesitating : should it set the list {0.0 0.0 0.0} instead ?

Moreover, I don't really like it when a program makes that kind of automatic assumption. It could mask genuine errors, which would then end up buried deep within the data—making them difficult to fix in practice.

D.2 Arithmetic substitution discussion and justifications

Arithmetic substitution appears in a context where the length of the arithmetic expression to parse is still unknown.

As you know, in Tcl, a command is parsed as a collection of words, each of which may contain many substitutions :

Once the command is parsed, Tcl will substitute a value for each sub-TOKEN, all of it being appended to one Tcl_Obj per Word.

To create an arithmetic substitution, I choose to follow the same logic, that is, to define a new kind of token :

The problem in this substitution context is to know the char-length of the TOKEN. For this, in ParseToken, I simply used the procedure Tcl_ParseExpr, and made it to report the length of the expression when its parsing is succesfull.

Why not having choosed a prefix?

Ex : set a {=}1+1

As the name implies, a prefix precedes a word. For this reason, a prefix is ​​detected within Tcl_ParseCommand, the procedure that breaks a command down into individual words. However, a word can be composed of several distinct tokens (its components), which are concatenated into a single Tcl_Obj to serve as the command's argument.

It follows that, if arithmetic substitution were defined by a prefix, it would have been impossible to detect in the middle of a word. Consequently, the introductory syntax must be detected in the parseToken procedure.

set A value{=}1+1 ; # this can't work
set A value[(1+1)]; # this will work

Why wasn't the $(...) shortcut chosen?

Aside from the fact that this syntax makes it impossible to use the {} array—a usage that is well-established—the main reason is that, since the expr command is now multiline, an arithmetic substitution is more akin to calling a nested script than to referencing a variable. The $ sign would inevitably evoke the concept of a variable, thereby causing confusion.

Why wasn't the (...) shortcut choosen?

The use of parentheses—even though they are not significant to the Tcl_parser—remains very common, as they allow for the capture of an array index in variable assignment procedures (such as set, lappend, etc.). No one would want writing set a(1+2) 2 to assign a value to the scalar variable a2 instead of the key 1+1 in array a.

Why was an unicode pair of chars not choosen ?

A shorthand like ⟦...⟧ would be an excellent choice. The fact is, I lack the skills to implement it, hence the decision to stick to the ASCII character set. Furthermore, it is difficult for me to assess whether such a choice would be practical for embedded systems. However, if there is a demand for such a shorthand, a significant part of the implementation would be similar to the one proposed here.

Why not introduce a special command—like the one in TIP 676?

Tcl already possesses an excellent, battle-tested arithmetic expression compiler that integrates seamlessly with the language. I felt that introducing a new parser would be redundant and—given the subtly different syntax—a source of confusion.

Above all, however, I wanted to handle arithmetic expressions using purely syntactic means, as arithmetic is a fundamental element of any programming language.

That is why I proposed the concept of arithmetic substitution to reconcile arithmetic with Tcl's art of substitution.

Why was the double symbol [( ... )] chosen?

The fact is that parentheses are not significant to the Tcl_Parser, whereas they are significant to the expr parser due to the requirement that they be balanced.

Another point is that, while the opening bracket is recognized by the expr parser, the closing bracket is invalid there. This prevents any overflow within expr.

Consequently, in arithmetic substitution mode, one simply needs to count the parentheses until the count reaches zero. If the character following that final parenthesis is a closing bracket, one can infer that the arithmetic expression is well-formed. If, however, a closing bracket is encountered before that point, an "invalid character" error occurs, indicating a parenthesis imbalance.

Admittedly, using symbols that count as two characters makes the syntax more cumbersome. However, since the [()] syntax requires 4 characters, whereas nesting an [expr {}] command requires 9, there is still a gain in conciseness.

Another advantage relates to defining lists. With the expr command, 12 characters would be required—[expr {(,)}]—whereas the [(,)] shortcut requires only 5.

In short, this choice was driven by the simplicity and robustness of the implementation, while ensuring compliance with the specifications.

D.3 Arithmetic script :

The context of an arithmetic script differs from that of a substitution. It is contained within a Tcl_Obj resulting from prior parsing; consequently, the start and end of the string are known. There is no need to determine the string's length—as is required for substitutions—so there is no need to use Tcl_parseExpr for that purpose.

Since the code forms a single, contiguous block and cannot appear as a component of another token, syntax detection can be performed at the very beginning of Tcl_ParseCommand. A simple check of the first and last characters is sufficient to identify it as an arithmetic script. A "Word" token containing a "sub-expression" token is then returned.

The Tcl_CompileScript procedure can then compile the expression by calling Tcl_CompileExpr.

Why was it chosen that an arithmetic script should begin with ( and end with )?

Mostly by analogy with the choice of [(...)], previously made in the context of substitution. The form is similar, which will help the programmer remember it.

It helps also to shorten the usage of list operator in some situation.

% proc init {} ((0,0,0),(0,0,0),(0,0,0))
% init
{0 0 0} {0 0 0} {0 0 0}

This analogy in the definition thus allows for an analogy in the implementation.

E. Mixing command syntax and arithmetic syntax

As we can substitute commands into arithmetic,
and, conversely, arithmetic into commands,
that implies that it exists a lot of distinct ways to write the same program.
To illustrate this point, let's consider the following examples :

A command proc body :

proc calculate_edge_distance_Cmd {px py t1 t2 t3} {
       # Command Syntax proc
       [(  # Arithmetic Syntax
           edges = (($t1,$t2),($t2,$t3),($t3,$t1));
           min_distance = 999999999
        )];
       # Command Syntax
        foreach edge $edges {
             # Command Syntax
             lassign $edge p1 p2
             lassign $p1 x1 y1
             lassign $p2 x2 y2
             [(# Arithmetic Syntax
                 A = $y2-$y1; B = $x1-$x2;
                 C = $x2*$y1 - $x1*$y2;
                 distance = abs($A * $px + $B * $py + $C) / hypot($A, $B);
                 $distance < $min_distance ? (min_distance = $distance) :;
              )]
       }; # Command Syntax
       [(# Arithmetic Syntax
           [point_in_triangle $px $py $t1 $t2 $t3] ? -$min_distance : $min_distance
       )]
};
% calculate_edge_distance_Cmd .5 .3 {0.0 0.0} {1.0 0.0} {0.0 1.0} #-0.14142135623730948

% timerate {calculate_edge_distance_Cmd .5 .3 {0.0 0.0} {1.0 0.0} {0.0 1.0}} # 6.482985 µs/# 154249 # 154249 #/sec 999.994 net-ms

An arithmetic proc body

proc calculate_edge_distance_Arith {px py t1 t2 t3} {(
       # Arithmetic Syntax
        edges = (($t1,$t2),($t2,$t3),($t3,$t1));
        min_distance = 999999999;
       [# Command Syntax
           foreach edge $edges {(
              # Arithmetic Syntax
              [ # Command Syntax
                 lassign $edge p1 p2
                 lassign $p1 x1 y1
                 lassign $p2 x2 y2      ] ;
              A = $y2-$y1; B = $x1-$x2;
              C = $x2*$y1 - $x1*$y2;
              distance = abs($A * $px + $B * $py + $C) / hypot($A, $B);
              $distance < $min_distance ? (min_distance = $distance) :;
         )}];
       [point_in_triangle $px $py $t1 $t2 $t3] ? -$min_distance : $min_distance
)}

% calculate_edge_distance_Arith .5 .3 {0.0 0.0} {1.0 0.0} {0.0 1.0} # -0.14142135623730948

% timerate {calculate_edge_distance_Arith .5 .3 {0.0 0.0} {1.0 0.0} {0.0 1.0}} # 6.567391 µs/# 152267 # 152267 #/sec 999.997 net-ms

Firstly, we can notice that the results are identical, and that the speed is similar.

Secondly, these two simple examples already show that a very long program could be written in a multitude of different ways to achieve exactly the same result.

That's why I think, if ever this TIP was accepted, a developer guideline should be produced to normalize the coding practices of mixing arithmetic syntax and command syntax.

Including commands to control the flow of computing into expr :

When, in arithmetic syntax context, we want control the flow, it is allways possible, thanks to the command substitution behaviour. But, if we need a body in arithmetic syntax, we'll have to use the script shorthand again. The consequence is then that we sometimes need 4 chars )}]; to close an arithmetic body if it is ending a command syntax context. Consider this example with a foreach and a while :


# Arithmetic Syntax
[foreach e $L f $M {(
       ...
)}];
...
# Arithmetic Syntax
[while {$test} {(
       ...
)}]; # Arithmetic Syntax
....

We can't say it is very beautifull. Can we make it lighter ?

Trying to get a lighter syntax with mathfunc

We could decide to add these control flow commands as mathfunc (or mathflows ?), in a manner, as they belong to the arithmetic context, that they would ask for an arithmetic body. For instance :

# notice that the bodies are enclosed in parentheses

proc ::tcl::mathfunc::foreach {args} {
    set body [lpop args]
    uplevel [list ::foreach {*}$args ($body)]
}

proc ::tcl::mathfunc::while {test body} { uplevel [list ::while $test ($body)] }

Then, the previous example can be rewritten :


# Arithmetic Syntax
foreach("e", $L, "f", $M, {
    # Arithmetic Syntax
    ...
});
...
while({$test}, {
    # Arithmetic Syntax
    ...
});

But, the experiment shows that it is slow, mostly because of uplevel, but also because of the need to include the body into parentheses. Moreover, we have to quote every argument, and to separate them with commas.

Conclusion : There is no gain, neither in speed, neither in code concision. This is not the good way to go.

Trying to get a lighter syntax with modifying expr parser and tcl compiler

A prototype, which can be found in the branch tip-759b_tcl90, shows it is possible to get more concision without speed defect, by small modifications of the expr parser and the tcl compiler. Some command names can be recognized in ParseLexeme which will then return a new kind of leaf lexeme type : COMMAND. This leaf lexeme type can be used in ParseExpr, to parse the command, and its arguments till ... the next semi-colon ';', instead of till the next CLOSE_BRACKET ']'.

Finally, to compile this correctly, a new token type TCL_TOKEN_CMD_IN_EXPR has to be created (defined as TCL_TOKEN_SUB_EXPR | TCL_TOKEN_COMMAND), so that Tcl_CompileTokens can call Tcl_CompileScript with the good string boundaries. Here is an illustration of the result, taken from tcllib math package :

Original proc to multiply polynomials

proc multPolyn_cmd {polyn1 polyn2} {
    if { [llength $polyn1] == 1 && [string is double -strict $polyn1] } {
        set polyn1 [polynomial $polyn1]
    }
    if { [llength $polyn2] == 1 && [string is double -strict $polyn2] } {
        set polyn2 [polynomial $polyn2]
    }
    if { [lindex $polyn1 0] != "POLYNOMIAL" ||
         [lindex $polyn2 0] != "POLYNOMIAL"    } {
        return -code error "Both arguments must be polynomials or a real number"
    }

    set coeffs1 [lindex $polyn1 1]
    set coeffs2 [lindex $polyn2 1]

    #
    # Take care of the null polynomial
    #
    if { $coeffs1 == {} || $coeffs2 == {} } {
        return [polynomial {}]
    }

    set zeros {}
    foreach c $coeffs1 {
        lappend zeros 0.0
    }

    set new_coeffs [lrange $zeros 1 end]
    foreach c $coeffs2 {
        lappend new_coeffs 0.0
    }

    set idx        0
    foreach c $coeffs1 {
        set term_coeffs {}
        foreach c2 $coeffs2 {
            lappend term_coeffs [expr {$c*$c2}]
        }
        set term_coeffs [concat [lrange $zeros 0 [expr {$idx-1}]] \
                                $term_coeffs \
                                [lrange $zeros [expr {$idx+1}] end]]

        set sum_coeffs {}
        foreach t $term_coeffs n $new_coeffs {
            lappend sum_coeffs [expr {$t+$n}]
        }
        set new_coeffs $sum_coeffs
        incr idx
    }

    return [list POLYNOMIAL $new_coeffs]
}

Modified proc with TIP759(b), including lmap, foreach and return as known commands in expr

proc multPolyn_ari {polyn1 polyn2} {(

    [llength $polyn1] == 1 && [string is double -strict $polyn1] ? ( polyn1 = [polynomial $polyn1] ):;
    [llength $polyn2] == 1 && [string is double -strict $polyn2] ? ( polyn2 = [polynomial $polyn2] ):;

    [lindex $polyn1 0] != "POLYNOMIAL" || [lindex $polyn2 0] != "POLYNOMIAL"  ?
    [return -code error "Both arguments must be polynomials or a real number"] :;

    coeffs1 = [lindex $polyn1 1];
    coeffs2 = [lindex $polyn2 1];

    #
    # Take care of the null polynomial
    #
    $coeffs1 == {} || $coeffs2 == {} ? [ return [polynomial ""] ]:;

    zeros = lmap c $coeffs1 {(0.0)};
    
    new_coeffs = [lrepeat [( [llength $zeros]+[llength $coeffs2]-1 )] 0.0];

    idx = 0;
    foreach c $coeffs1 {(
        term_coeffs = lmap c2 $coeffs2 {($c*$c2)};
        term_coeffs = [lreplace $zeros $idx $idx {*}$term_coeffs];	
        sum_coeffs = lmap t $term_coeffs n $new_coeffs {($t+$n)};
        new_coeffs=$sum_coeffs;
        [incr idx];
    )};

    return [list POLYNOMIAL $new_coeffs];
)}

What the measurements shows, with P1 = {POLYNOMIAL {2 4 3}} and P2 = {POLYNOMIAL {3 2 4 5}}

proc version length length diff speed speed diff
multPolyn_cmd 1322 +31.02 % 10.19 µs +17.13%
multPolyn_ari 1009 -23.67 % 08.70 µs -14.62%

Conclusion : There is a gain, either in speed, either in code concision. This is the good way to go.

Incompatibilities :

Comments / Questions

Copyright

This document has been placed in the public domain.