TIP 764: Annotated values and typed variables for Tcl 9.0

Login
Core Home Timeline Branches Tags Forum Help 
Author:  Florent MERLET <[email protected]>
Author: Claude
State:          Draft
Type:           Project
Vote:           Pending
Created:        13-Oct-2026
Tcl-Version:    9.2
Tcl-Branch:     tip-764_tcl90
Keywords:       tcl, annotation, type, variable
Version:  $Revision 5.0 $

Revision 5 describes what is implemented and tested in a patched Tcl 9.0 tclsh (core-9-0 branch and TIP 759 branch): {type}word annotations, types with operators, parameterized types, computed defaults, var links, and an index language built like expr. It supersedes revision 3.

1. Principles

Out of scope: typed Tcl arrays (array is reserved without semantics), untagged unions, mutual recursion between types, per-namespace or per-procedure type tables (there is one global table per interpreter), typed expr operators, strictness of nested elements (only the top-level object is strict), typed procedure parameters.

2. Annotation in Tcl words

A word of the form {expr}suffix is an annotated word: the opening brace at the start of the word is balanced and a non-empty suffix immediately follows. Tcl currently reports "extra characters after close-brace" for such words, so extending it breaks no working script. A {int} without suffix remains an ordinary braced word, and {*} keeps priority (expansion).

The parser is neutral: it delivers a (type, suffix) pair without interpreting either. The consumer decides the role of the suffix.

Context Role of the suffix Example
First argument of set variable name set {int}i 0
Value passed to set content to annotate set {list}L {list}{1 2 3}
Root of type def constructor operand type def poly {list}point
Product field field name {double}x
Substitution extracted value {int}$P(x)

Parser representation. For {type}word, Tcl_ParseCommand produces a TCL_TOKEN_WORD whose first component is a TCL_TOKEN_ANNOT token (value 512) carrying the type text, followed by the tokens of the suffix (TEXT, BS, VARIABLE, COMMAND). The suffix may be bare, braced or quoted. The type cannot contain a backslash-newline.

Raw and typed annotation.

Compilation. Every command with an annotated word is compiled as an ordinary call, with no specialized compiler (TclHasAnnotatedWord: set, append, incr, lappend...). The annotated words themselves are compiled by TclCompileTokens.

Empty type. {} means no type: {}$x removes the type of a value, {}name releases the contract of a variable (§6). A {} word without suffix remains Tcl's empty word.

3. The type language

The body of a definition is a braced word for Tcl; type def parses it with the algorithm of ParseExpr (ParseType, ParseTypeLexeme, TclCompileTypeTree). As in expr, the closing parenthesis (here the closing brace) is integrated into the tree.

Operators, from weakest to strongest:

Operator Role Example
; product (fields), binary {double}x; {double}y
, sum (variants), binary {int}i, {double}d
= default value of a field {int}n = 0
. nesting, right-associative list.dict.int (list of dicts of integers)
{ }name grouping; the optional name is attached to the closing brace {double}x: field x; {list.int}: plain grouping
( ) arity of a collection, attached to list or dict list(2), dict(x, y?)

, binds tighter than ;: {int}a; {double}d, {int}i is a product of two fields whose second, unnamed, is the sum {double}d, {int}i. A leading, trailing or repeated ; is ignored, like the empty statements of a script. As soon as a definition contains ;, the outer braces are mandatory, since ; separates Tcl commands.

Operands. A type name (int, double, string, list, dict, var, a defined type, a parameter $T), a field name (attached to }), a value: a double-quoted word (enumeration literal), a bare or braced word after =, or [script] / $variable after = (§4). The words int, double, string, list, dict, var, array are reserved: no user type can shadow them.

Collections.

Products, sums, enumerations.

type def point {{double}x; {double}y}           ;# product
type def polyline {list.point}                  ;# list of point
type def color {"red", "green", "blue"}         ;# enumeration = sum of literals
type def number {{int}i, {double}d}             ;# sum of two variants
type def tree {{int}v; {list.tree}kids}          ;# recursive, through a list

A product field needs a name; a sum variant may be unnamed (enumeration literal).

Recursive types. A type may refer to itself provided every reference goes through a sum variant or a collection; otherwise no finite value exists and the definition is refused. Values are immutable and built bottom-up: no cycles. Mutual recursion remains out of scope.

Variants of a sum. The system orders them by proven inclusion: int ⊂ double ⊂ list ⊂ string, dict ⊂ list, an enumeration literal ⊂ any type that accepts it, a product ⊂ list, list.A ⊂ list.B if A ⊂ B. Writing order is therefore irrelevant. If two variants overlap without inclusion (list of integers and dict of strings, for 1 2), the type is accepted but building the sum from a bare string is the error ambiguous variant. An already built value carries its tag.

Explicit tag. set {number}U {d}1.5 annotates the value with the variant name; on a variable already of type number, set U(d) 5 replaces the value by the d variant. When the target is a sum, {name} is first read as a variant name, then as a type; a variant name identical to a reserved or defined type is therefore refused at definition.

var is a type of links to variables (§7).

4. Parameterized types, computed defaults and {cmd} parameters

Parameterized types. type def accepts parameters in the name; the whole name is braced if it contains blanks:

type def {stack({type}T)}  {list.$T}
type def {matrix({int}i, {int}j = 3)}  {list($i).list($j).int}
type def {tree({type}T)}   {{$T}value; {list.tree($T)}children}

set {stack(int)}s 1 2 3
set {matrix(2,3)}m {1 2 3} {4 5 6}
set {tree(int)}t value 1 children {{2 {}} {3 {}}}

A parameterized type is a template: it denotes no value. Each instantiation (name, arguments) produces an internal nominal type named matrix(2,3), whose body is compiled once with the parameters bound. The instance is registered before its body is compiled, which allows recursion (tree($T) inside tree); if compilation fails, the instances created since the start of the call are removed. A parameter may have a default value ({int}j = 3).

Kinds of parameters:

Computed defaults. After =, the default of a field or of a dict key may be:

Form Meaning
= 0, = "red", = {a b} literal, validated at definition
= $p value of a type parameter; otherwise a global variable ($::level)
= [script] script evaluated at each construction

The script is evaluated at global level (TCL_EVAL_GLOBAL), its result is validated against the field type and then stored: the script is never called again for that value. A supplied value bypasses the script. A variable default is read at each construction (::subst on $name). Parsing uses Tcl's parser APIs (Tcl_ParseCommand in nested mode, Tcl_ParseVarName), like ParseExpr for [...] and $x; the text is kept as is and never executed at definition. A script error is reported as: cannot compute the default of field "e" of journal: ....

type def journal {
    {int}epoch = [clock seconds];
    {string}message;
    {string}author = [info hostname]
}
type def label {dict(name, created?=[clock seconds], level?=$::level).string}

{cmd} parameter. A command in a default is useful even without parameters: {cmd}clock = ::now makes the source of the value an argument of the type.

type def {timestamped({type}T, {cmd}clock = ::now)} {
    {int}date = $clock;
    {$T}value;
    {int}k = 0
}
set {timestamped(string, ::test::fakeclock)}h value x     ;# $h(date) = 1000

The command is a single word, fully qualified (::ns::cmd). Instances differ per command: timestamped(string, ::a) and timestamped(string, ::b) are two distinct types.

5. The type commands, annotated objects, strict semantics

type def name definition creates a nominal type: celsius and fahrenheit, both defined on double, are distinct, and assigning one to the other is an error without conversion. The definition comes in two equivalent forms: an annotated object (type def poly {list}point, read directly) or a plain object (type def poly {list.point}, parsed with the §3 grammar). There is a single global type table per interpreter; a type name can be neither redefined nor shadowed.

type alias name definition creates a transparent alias instead. Constructors (list.point, dict(x).int) stay structural: only a defined name is nominal.

type convert source target body declares the only allowed conversion between two nominal types:

type def pixel int
type def mm double
type convert pixel mm {x {expr {$x * 0.2646}}}
set {pixel}len 10
set {mm}L {mm}$len                  ;# declared conversion: 2.646

The function receives the raw content and its result is validated against the target type. It runs only at cast time, never during validation. No automatic composition (pixel→mm then mm→cm does not allow pixel→cm). Between a nominal type and its underlying type (pixel, int) the direct cast exists without declaration.

Descriptors. They are interned by their canonical text: two identical annotations share the same descriptor. They live as long as the interpreter (no reference counting); an annotated value pins its own, so its type stays valid as long as the interpreter exists.

The annotated object A is a Tcl_Obj of type annotated whose internal representation has two pointers:

updateString copies the string of V (the annotation is invisible); setFromAny always fails (a string never becomes an annotation by shimmering); freeIntRep and dupIntRep count references of V and T. A is never modified in place; it is validated when consumed, before any storing in a variable. Pitfall: Tcl_IsShared(V) ignores the sharing of A, so every modifier rebuilds a fresh object. The compiler adds literal annotated objects to the bytecode literals without hashing.

Strict semantics. A Tcl_Obj has a single internal-representation slot and the kernel cannot refuse a conversion: forbidding shimmering therefore requires the type to travel with the value. The conversion entry points (numbers, booleans, lists, dicts, strings) recognize an annotated object (TclAnnotUnwrap):

Situation Behavior
Content of the requested type used directly (expr {$i + 1}, llength $L, lindex $L 1, incr x work)
String view (puts, string length) always allowed, without replacing the representation
Widening int → double allowed, without touching the representation
Any other requested type error such as can't use int value as list

Cost: typed values leave the typePtr == &tclIntType fast path of the bytecode.

Cast. Annotating an already typed value ({int}$P(x)) is the only type crossing; it builds a fresh object, the original value does not change. Conversion table:

Truncation is silent; expr {round(...)} remains available for rounding. Known limit: expr {$len + $m} accepts a pixel and an mm, because operands are read as their underlying type.

6. Typed variables and annotated set

Storage. The value stored in the variable is the annotated object: the type is read from the value, there is no Var*-to-type table, and the contract disappears with the value on unset. A VAR_TYPED bit in Var.flags flags a typed variable; it joins the masks of the direct-access macros (TclIsVarDirectReadable, ...Writable, ...Modifyable, ...Unsettable) so bytecode takes the slow path. Every writer goes through TclPtrSetVarIdx, where two hooks (TclTypedVarPrepareWrite, TclTypedVarAfterStore) validate the write; for append and lappend, the result is validated, not the delta. Scope: scalar variables; Tcl arrays and their elements are refused at declaration. Links (upvar, global, variable) are followed to the target variable.

Typing rules.

set {int}x 5                ;# x is an integer
set x abc                   ;# error: expected integer but got "abc"
set j $len                  ;# j adopts pixel
set k {}$len                ;# k is an ordinary copy

Contagion. A variable inherits the strictness of the first typed value it receives; {} lets it out voluntarily. A typed argument stays typed inside a procedure; declared parameters (proc f {{int}x}) are an open point.

Annotated set. set without annotation keeps its ordinary arity. When its first argument is annotated, the following ones are read according to the target type:

Target type Arguments Example
scalar exactly 1 value set {int}i 0
collection (list, dict) 0 to n elements set {list.int}L 1 2 3
n-field product n values, or up to n name/value pairs set {point}P 200 100, set {point}P y 100 x 200
enumeration one literal set {color}C red
sum exactly one value, annotated with the variant or recognized set {number}U {d}1.5

Products: positional or name/value. If the number of arguments equals the number of fields, the form is positional; otherwise, if the number is even and at most 2n, it is a list of name/value pairs, possibly partial: omitted fields take their default (=), or, with no default, the error is field "x" of T has no value and no default. Known ambiguity: for a 2-field type, two arguments are always read as positional, so a partial default is inexpressible as name/value; at least 3 fields, or the complete form, are needed.

Unfolding rule. An unannotated argument is always one element. An annotated argument that is a collection of the same kind as the target is unfolded into elements (set {list}L {list}{1 2 3}: three elements); an annotated argument of another type is a single element validated against its type.

Order of operations: build the value, validate it before touching the variable, replace the type, write, restore the old type on failure. Tcl_SetObjCmd dispatches to TclSetAnnotatedCmd (annotated first argument) or TclSetPathCmd (set P(x) ...).

A field or variable of type var is a link to another variable: it holds no value, it designates an existing variable.

Rules:

8. Index language and element access

Element access on a typed variable uses the syntax V(index). The content of the parentheses is an index language parsed the way expr is: ParseIndexLexeme (equivalent of ParseLexeme), ParseIndex (same algorithm as ParseExpr, with operator and operand stacks) then CompileIndexTree (same walk as CompileExprTree). Substitutions use Tcl's parser APIs directly (Tcl_ParseVarName, Tcl_ParseCommand, Tcl_ParseQuotedString).

8.1 Operands

8.2 Operators

From lowest to highest precedence: , (union) < . (descent, cross product) < : (range). Parentheses group.

8.3 Compilation into a set of paths

CompileIndex produces a PathSet: a set of paths, each being a sequence of steps. . takes the cross product of its operands, , the union, : a range whose bounds are single indices. All substitutions are evaluated when the path is compiled.

8.4 Examples

S(b.y)           ;# field y of field b
S((a,b).x)       ;# S(a.x) and S(b.x)
L(1:3)           ;# elements 1 to 3 of the list
D('end')         ;# literal key "end" of the dict
L([(1+1)])       ;# element 2
L($i)            ;# index held in a variable
L([llength $m])  ;# result of a script

8.5 Writing

8.6 Errors

Index out of range, unknown field, missing tag, non-simple path in a write, number of values different from the number of paths: standard Tcl error message, the variable is left unchanged.

8.7 Known limits

Tag only alone; multiple results are flattened; no arithmetic in the index (except through [( … )]); no range or tag writes; path writes do not reorder a keyed dict; append, lappend and incr on an element of a scalar typed variable are not supported (the variable is not an array).

9. Kernel integration

9.1 Files

The wiki's demonstration get command is not kept: access is through $V(index) and set V(index) value.

9.2 Hook points

9.3 Two patches

The index language assumes TIP 759 for [( … )] and the (1,2,3) lists; without it, the other operand forms remain usable.

10. Status, tests and open items

10.1 Tests

10.2 Compatibility

10.3 Open items