Author: Eric Taylor
State: Voting
Type: Project
Created: 2023-06-06
Tcl-Version: 9.1
Tcl-Branch: tip-672-v2
Forum: 1b5db3989c
Abstract
This TIP extends the $ substitution to include expressions as $((expression)).
The expression is parsed once, by the Tcl parser itself, as if it were the
braced argument of expr, so it is never substituted twice. The reference
implementation can also be built to use the spelling $(expression).
Rationale and Discussion
The current method for expressions using the [expr] command has 2
problems. First, it is difficult to BOTH read and write. Some, as in
TIP 282, go so far as to say it is ugly.
Secondly, as indicated in TIP 526, expressions should be braced, or there can be problems with both security and performance.
Consider the following computation (from TIP 282):
set x [expr {5*$y + 6*$z}]
set w [expr {$x**2 + $y**2}]
set v [expr {$w**2 + $y**2}]
The [expr {...}] constructs make the code considerably harder to
read. This TIP would propose that there be a shorthand as such:
set x $((5*$y + 6*$z))
set w $(($x**2 + $y**2))
set v $(($w**2 + $y**2))
The implementation of this should also eliminate the requirement for
the braces from above, as braces would be assumed and code would be
compiled as though the parentheses were braces.
Since this is a $ substitution, it would continue to work inside
double quotes with the expected result.
There would also be an advantage in some text editors, which already have parenthesis balancing and syntax coloring features.
Another Tcl implementation, JimTcl uses this $(...) substitution and it
seems to work without problems.
Version and Incompatibility
This TIP would apply to version 9.2. of Tcl.
One incompatibility for the $(expression) form is that it would require that
the use of the null string "" or {} as an array name would require a
change from using $ substituion to using the 2 argument form of the set
command. With the $((expression)) spelling there is no such incompatibility:
an unescaped ( inside an array index is already an error in Tcl 9.1, so $((
cannot occur in a script that works today.)
Thus, to assign the variable "" to a value, would still work for both scalars
and array variables, as in:
set {} foobar ;# scalar variable
set (index) "array value" ;# array variable
But the use of $(index) would have to change by using the set command:
set var [set (index)]
One other possible incompatibility for the $(expression) form would be with
the use of the subst command.
The issue here is whether $(...) would be a command or variable substitution
that subst has options to suppress. The reference implementation solves this
with two new options of the subst command, -expressions and
-noexpressions (the shorter -expr and -noexpr work as unique
abbreviations). Because the body of an expression can contain [commands]
and $variables, -nocommands and -novariables also turn expression
substitution off. Code that uses them to make text safe therefore behaves as
before.
Migration
The $((expression)) spelling is also a safe way to start using the new
substitution today. If Tcl later changes to the $(expression) spelling, and
deprecates the use of an array with no name, scripts that already use
$((expression)) continue to work with the same value: the $( spelling
accepts $((expression)) as $( followed by a parenthesised expression.
The test suite runs $((...)) in both builds. Only scripts that use the empty
array name, as in $(index), would need the change described above.
Examples
The examples use the proposed $((expression)) spelling.
% lassign {3.0 4.0} a b
% set c $((sqrt($a**2 + $b**2))) ;# 3 4 5 triangle
5.0
% lassign {1000 3} total number
% puts "ave = $((double($total)/$number))" ;# demo usage inside quotes
ave = 333.333333333
.canvas addtag enclosed $(($x - 20)) $(($x + 20)) $(($y - 20)) $(($y + 20))
With the alternative spelling, the first example reads:
% set c $(sqrt($a**2 + $b**2))
5.0
Inside subst:
% set n 4
% subst {n squared is $(($n**2))}
n squared is 16
% subst -noexpressions {n squared is $(($n**2))}
n squared is $((4**2))
With -noexpressions the $( is left as ordinary text, and the $n inside it is
still substituted as a variable.
Inside an expression, including a braced one such as the condition of if
or while:
% set x 5
% expr {1 + $(($x*2))}
11
% if {$(($x*2)) > 9} {puts big}
big
% expr {$(($x + $(2*3)))} ;# nested
11
This works because the expression parser uses the same $ substitution
parsing as the script parser. No new code was needed in expr itself beyond
accepting the new token as an operand (a one-line change in
generic/tclCompExpr.c).
Specification
Syntax
The new substitution is recognised wherever $ substitution is: in words,
inside double quotes, in subst, in Tcl_ParseVar, and inside expressions
(for example expr {1 + $((2*3))} or the condition of an if), where it can
also be nested.
It has the value of [expr {body}], evaluated in the current scope, with the
same result and the same error and return-code behaviour (including return,
break and error inside a [command] in the body).
There are two spellings, chosen when Tcl is built (see Build switch):
$((body)). Valid in both builds. The two outer parentheses play the role of the braces in[expr {body}], which makes the body easy to see.$(body). The alternative spelling.
The body ends at the first ) that is not nested inside parentheses opened
in the body, a braced word, a double-quoted string, or a [command]. A
backslash protects the next character. In the $(( spelling the closing )
must be followed immediately by a second ), otherwise the text is a syntax
error with the message missing second ")". If the input ends right there,
the command is incomplete, so an interactive tclsh waits for more input.
The body is passed to the expression compiler exactly as the argument of a
braced expr would be. The word parser does not substitute anything in it,
so nothing is substituted twice.
The subst command
subst gets -expressions and -noexpressions. -nocommands and
-novariables also switch expression substitution off (see above).
The subst command in current trunk (TIP 712) also has positive forms:
-backslashes, -commands, -variables and -expressions turn on only the
substitutions named and leave every other kind off. For example,
subst -variables $s substitutes variables and nothing else. This makes
-expressions an explicit opt-in: a list of positive options that does not
name it does not substitute expressions. Positive and negative options cannot
be mixed in one call; subst -expressions -novariables $s is the error
cannot combine positive and negative options. With no options at all,
everything is substituted, including expressions, as before.
C interface
Tcl_ParseVarName (and so Tcl_ParseCommand and Tcl_ParseVar) can not
return a token of a new type, TCL_TOKEN_EXPR_SUBST (value 512), an
error is raised in this case. A new flag, TCL_SUBST_EXPRESSIONS, is
used by Tcl_SubstObj. TCL_SUBST_ALL grows from 7 to
15 so that existing callers passing TCL_SUBST_ALL get the new
substitution. Code that walks the tokens of a parse with a switch and no
default case needs a new case. In the default build only text containing
$(( can produce the new token, so a script that does not contain it is
unaffected.
Build switch
TCL_EXPR_SUBST_PARENS is a compile-time constant: 2 (the default) means the
trigger is $((, 1 means $(. Only one source file, tclParse.c, depends
on it. The core team can choose the spelling by changing one number.
Implementation
See branch tip-672-v2
The shortcut is implemented in the parser (Tcl_ParseVarName in
generic/tclParse.c, with the compile-time and run-time handling in
tclCompile.c and TclSubstTokens), not by rewriting text. For a constant
expression $((1+2)) compiles to the same bytecode as [expr {1+2}]. In
instructions executed, running through eval is about 3% cheaper and
compiling in a proc about 4% cheaper.
The change comes with documentation (Tcl.n, subst.n, ParseCmd.3,
SubstObj.3) and 111 new tests in parse.test, subst.test,
compile.test and parseOld.test. The full test suite passes in both
spellings on Linux and Windows. The new code was also run under Valgrind,
AddressSanitizer and the Tcl memory-debug build with no errors, and checked
by mutation testing. The nesting check described below was added later and
has been run under AddressSanitizer and the memory-debug build only.
Expression substitutions nested inside one another are limited by the same
nesting check that limits nested commands (interp recursionlimit): very deep
nesting gives the error "too many nested compilations (infinite loop?)" with
the error code TCL LIMIT STACK, instead of overflowing the C stack.
Copyright
This document has been placed in the public domain.
