TIP 764: Annotated Values, Typed Variables and Generalized Paths

Login
Core Home Timeline Branches Tags Forum Help 
Author:         Florent Merlet <[email protected]>
Author:         Claude (Anthropic)
State:          Draft
Type:           Project
Vote:           Pending
Created:        19-Oct-2026
Tcl-Version:    9.2
Keywords:       annotation, type, typed variable, path, struct
Post-History:

Abstract

This TIP adds annotations to Tcl: a prefix {text}word that attaches metadata to a word without changing its value. An annotation resolved by a variable constructor is a type, and gives the variable a contract checked on every write; an unresolved annotation is a note that travels with the value. A small declarative type language (type def) describes enumerations, sums, products (structs), typed lists, typed dicts, variable links and parameterized types. Parts of a typed value are read and written with a path in the array index: $R(A.x), set R((A,B).x) 10 30.

Both new syntaxes take over forms that are syntax errors in Tcl 9.0 ({x}word) or fall back exactly to the Tcl 9.0 rule ($name(...)). No valid script changes meaning, and code that uses no annotation sees no difference.

Binary representations (type binary, chan binary) are built on this TIP but are left to a separate TIP.

Rationale

"Everything is a string" keeps Tcl simple, but the developer always knows more about a value than its string says: that 12 is in millimetres, that a list holds only integers, that a dict must have a name key. Today this knowledge lives in comments and in hand-written checks scattered through the code. Errors surface far from where the bad value was stored.

An annotation puts this knowledge beside the value, as metadata, without touching the string. The word comes from Latin ad notare, "to put a note beside"; the Greek protokollon was likewise the sheet glued to the head of a papyrus roll, describing it without being part of it. An annotated value has a raw part, the string every command sees, and an added part, read only by commands that ask for it.

The proposal is driven by four needs:

The design rule throughout: annotations live outside the plane of raw values. A computation (expr, string toupper) returns a bare value; only copying, list access and argument passing keep the annotation. Scripts that use no annotation run exactly as before.

Specification

1. Annotated values

Syntax. A word of the form {text}rest, where rest is not empty, is an annotated word. text is the annotation; rest is parsed as an ordinary word (literal, $var, [cmd], "...", {...}). {*} keeps its meaning. The value of the word is the value of rest: the annotation never appears in the string.

puts {mm}12                  ;# 12
string length {mm}12         ;# 2

Glued annotations {a}{b}$x are not a syntax: the form would be ambiguous with {int}{300}, an annotation on a braced literal. Annotations stack by annotating a value that is already annotated.

Notes. An annotation that no constructor has resolved is a note. It follows the value through copies, arguments, return values and list elements. Notes stack, from the oldest to the most recent.

set x {mm}5
set y {px}$x
note get $y                  ;# mm.px

Types. A variable constructor resolves annotations against the types the interpreter knows (section 2). The constructors are set {T}v ... and set v {T}$x, binding of proc arguments, lassign, foreach, and in expr the assignments x = v and x is T = v. Outside a constructor, {T}$x checks and converts nothing, even when T is a known type: T is pushed like a note, at no cost.

Two stacks and unstacking. An annotated value carries at most one resolved type and a stack of pending notes; they are displayed as [email protected]. A constructor:

  1. takes the most recent note and tries to resolve it;
  2. continues while notes resolve; the first that does not stops the process and stays on the stack;
  3. checks that the unstacked types are consistent, else contradictory annotations A and B;
  4. casts the value to the most recent type;
  5. lets the variable's contract make the final check.

A note set before its type def is therefore resolved by the next constructor that sees it.

Context first. A constructor resolves an annotation in its context before the global types: the variant names of a sum ({i}5), the field names of a product ({age}12). A global type of the same name changes nothing.

Local annotations. An annotation whose text contains a character that type names refuse (-, ?, :...) can never be a type. It is local: it passes from a caller to the called command and is not stored. Two families are intended, with no possible collision with a type name: options {-name} and questions {name?}. Storing or declaring with a local annotation is an error (local annotation {-v} cannot be stored by "set").

The empty annotation {}. On a value in a constructor (set v {}$x), it removes the resolved type and keeps the notes. On a variable name (set {}v $x), it lifts the variable's contract. On a value outside a constructor, the reader sees the bare content. The original value is never modified.

The note command.

note get VALUE               the notes, oldest first, joined by "."
note has VALUE NAME          1 if NAME is one of the notes, else 0
note delete VALUE ?NAME?     the value without the note NAME, or without any note

note delete returns a new value; Tcl values stay immutable.

2. The type command and the type language

type def NAME DEFINITION       define a nominal type, or a parameterized family
type alias NAME DEFINITION     give a structural name to a definition
type get VALUE                 the resolved type of a value (empty if none)
type cast TYPE VALUE           a constructor's resolution, without a variable
type convert FROM TO LAMBDA    declare an explicit conversion
type deref VALUE               a snapshot: each var link replaced by its value

type def creates a nominal type: type def port int(u,16) is distinct from int(u,16). type alias only names a definition. In expr, v is T tests a variable's type; x is T = v declares and assigns.

Base types. int (unbounded), double, string, list, dict (the ordinary Tcl list and dict), var (a link to a variable, by name), any (an element of a heterogeneous list.any). Refinements restrict a base type: int(u|s, n), double(32|64), string(ENC, N|z), bitarray(n). Their binary representation is outside the scope of this TIP.

Composition language. A definition is parsed on the model of expr. Operands are types (bare words), names (a word glued after }: {int}age) and values (quoted, braced, or after =). Operators, from the strongest:

operator role example
. nesting (right-associative) list.dict.int
( ) arguments, glued to the name list(3).int, dict(x, y?)
= or ?= default value of a field {int}age ?= 0
; product {double}x; {double}y
, sum {int}i, {string}s
{ }name grouping, named by the word glued after } {{int}x; {int}y}A

; and , have the same precedence; mixing them requires braces. A type name contains only ASCII letters, digits, _ and non-ASCII bytes, and does not start with a digit. Any other ASCII sign outside the grammar is refused with TCL PARSE TYPE BADCHAR; this is what keeps local annotations safe. A definition is declarative: nothing is substituted when it is read, except in default values.

Containers.

type def color {"red", "green", "blue"}           ;# enumeration
type def S {{int}i, {string}s}                     ;# sum: one variant at a time
type def pt {{string}name ?= "?"; {int}age ?= 0}   ;# product
set {list.int}L 10 20 30                           ;# typed list
set {list(2).int}c 3 4                             ;# list of exactly 2
type def record {dict(name, age?=0).string}        ;# dict: name required, age optional
set A 1; set {list.var.int}V A                     ;# link: V(0) reads ::A

The elements of a list.T are atomic: lindex, foreach and $L(i) return typed elements, never split again. Dict keys are always text; only values are typed.

Default values. A default is a literal, a $variable or a [script], evaluated at each construction. A field with no default must be given: field "x" of pt has no value and no default.

Parameterized types. The name in type def may declare parameters, written like fields and used with $ in the body. Types may be recursive.

type def {pair({type}T)} {{$T}x; {$T}y ?= 0}
set {pair(double)}p {x}1.5                         ;# 1.5 0
type def {tree({type}T)} {{$T}value; {list.tree($T)}children}

Phantom parameters. A parameter absent from the body distinguishes instances without changing values. Units of measure follow:

type def {quantity({string}dim, {double}f = 1.0, {double}o = 0.0)} double
type def mm {quantity(L, 0.001)}
type def m  {quantity(L)}
type def kg {quantity(M)}
type convert quantity quantity {{x from to} {
    if {$from(dim) ne $to(dim)} {return -code continue}
    expr {($x * $from(f) + $from(o) - $to(o)) / $to(f)}
}}
set {mm}a 1500
set b {m}$a                  ;# 1.5
set {m}c $a                  ;# type mismatch: expected m but got mm (use an explicit cast)
set d {kg}$a                 ;# unknown way to cast the value from mm to kg

A type convert rule targets a type or a family; return -code continue means "not applicable". Conversions between nominal types are always explicit.

3. Typed variables and constructors

Contract. A variable declared with set {T}v ... checks every later write, whatever its origin: set, lappend, lset, incr, dict set, a path write, or expr {v = ...}.

set {list.int}M 1 2
lappend M 3                  ;# 1 2 3
lappend M x                  ;# expected integer but got "x"

Widening and casts. A refinement goes where a wider type is expected (int(u,8) into int); narrowing needs an explicit cast. A nominal type gives way to its base, but two nominal types with no base link are never equivalent. set v {T}$x and type cast T $x try, in order: a type convert rule, the chain of bases, a sum's variant, a family rule, the built-in scalar conversions. Otherwise: unknown way to cast the value from A to B.

What keeps the annotation.

operation type notes
copy, argument, return value kept kept
lindex, foreach, $L(i) on a list.T element typed T -
string cat $x (one argument) kept kept
computation (expr, string toupper, dict set on a value) lost lost
cast {T}$x in a constructor changed kept

Value types and contracts. A value keeps its type wherever it goes, but only an annotation on a variable's name gives the variable a contract. In set A $B or set A {T}$B, A holds a value of type T (type get $A), and any later write to A is still accepted. Procedure arguments follow the same rule: an argument receives typed values without becoming strict. foreach, lmap, dict for and dict map forget the type of their loop variable before each assignment.

set {list.int}L 1 2 3
set e [lindex $L 1]          ;# e holds a value of type int
type get $e                  ;# int
set e hello                  ;# accepted: e has no contract
set {int}n [lindex $L 1]     ;# the contract comes from the name
set n hello                  ;# expected integer but got "hello"

Declaring in any writing command. A type annotation on the variable name declares the variable in every command that writes it, not only in set: lappend {list}M x is set {list}M $M followed by lappend M x. The new contract is built from the current content and replaces any previous one; {}M lifts it. This is done once, where every access by name converges (TclObjLookupVarEx), so lappend, lset, incr, append and dict set behave alike.

set {list.int}L 1 2 3
set M $L                     ;# M holds a list<int> value, without a contract
lappend M x                  ;# 1 2 3 x: a modification is a computation, M is now a plain list
lappend L x                  ;# expected integer but got "x": L has a contract
lappend {list}L x            ;# 1 2 3 x, L is now declared list
incr {int}n                  ;# n receives the contract int
dict set {dict.int}D b 2     ;# D is now dict<int>

A modification is a computation. Without a contract, lappend, lset, incr, append or dict set on a typed value return a plain value, as expr does; they never fail because of the value's type. Under a contract, the result is checked when it is stored, and the variable keeps its old value if the contract refuses it. Seen from a command that ignores annotations, a value is its string; only declarations, visible in the source on a name, add checks.

Collections and elements. A declaration says what an element is. Only collections (list, dict) show their elements to the list view (llength, lindex, foreach, lassign, {*}); a scalar, a product or a link is one element, and a sum takes the nature of its active variant. The parts of an element are reached by path: $P(x), or $P(*) for all of them in order.

type def Point {{int}x = 0; {int}y = 0}
set {Point}P 100 7
llength $P                   ;# 1: a point is one element
set {list.string}L {hello world} b
llength [lindex $L 0]        ;# 1: "hello world" is one string
deux {*}$P(*)                ;# the fields as two arguments
expr {(x,y) = $P}            ;# tuple assignment reads the fields

This separates two entities that untyped Tcl lists conflate, and removes a family of bugs (a string split into words, an element dug into, a misplaced {*}). Undeclared values keep the Tcl rule: their list view is the list view of their string.

Building a product. A product of n fields is built in one of these forms; nothing is guessed from the data.

form example rule
positional set {pt}a Jean 12 exactly n values, in field order
field tags set {pt}a {age}12 {name}Jean every value tagged, any order; omitted fields take their default
one field set {pt}a {pt.name}Jean the other fields take their default
path + positional set {pt}a(age,name) 12 Jean one value per leaf of the path, or per group
no argument set {pt}a every field at its default
copy set {pt}a $b $b already of type pt

Anything else is wrong # args: pt needs 2 values; to name fields (the others take their default), write {field}value or set {pt}var(field,...) value....

Field tags. The tag of an argument is its most recent annotation, read in the context of the product, so a field name wins over a global type of the same name. A tag carried by a variable counts. Tags are all or nothing:

type def couple {{string}name; {int}age ?= 0}
set {couple}p {name}jean {age}12        ;# jean 12
set {couple}p {name}zoe                 ;# zoe 0
set {couple}p {name}jean 12             ;# can't mix values named by a field annotation ...
set {couple}p {name}a {size}3           ;# unknown field "size" in couple
set {couple}p {name}a {name}b           ;# duplicate field "name" in couple

Path and positional values. set {T}v(path) v1 v2 ... builds the product and its nested products. The number of values chooses the distribution: one per leaf, one per element of the head group, or one for the whole path. Each value has the shape its read would return.

type def rect {{{int}x; {int}y}A; {{int}x; {int}y}B}
set {rect}R(A.x,A.y,B.x,B.y) 1 2 3 4
set {rect}R((A,B).(x,y))     1 2 3 4
set {rect}R(A.(x,y),B.(x,y)) {1 2} {3 4}

Sums and lists. A sum takes its variant as a tag or in the type's name (set {S}v {i}5, set {S.s}w abc). Without a tag, the first variant whose shape fits is chosen. A list takes its elements one by one (set {list.int}L 1 2 3); a single argument is one element.

4. Paths

A typed value is read and written with array syntax; for a typed scalar variable, the index is a path. The language is the one proposed on the Tcl wiki page A better way to do indexation, parsed on the model of expr.

set {rect}R(A.x,A.y,B.x,B.y) 1 2 3 4
$R(A.x)                      ;# 1
set R(B) {5 6}               ;# write a whole field
set R((A,B).x) 10 30         ;# multiple write
${R(A.x, B.y)}               ;# read; spaces allowed

Operands. A bare word is a field, a dict key, a variant name or a list index (3, end, end-1): the container's type decides. 'key' is always a key ('end' is the key "end"). "tag" is literal text copied into the result.

Operators, from the weakest:

operator role example
, collection (a group) A,B
* (unary) unfolding: the group merges into its parent *0:2,4
. nesting; distributes over a group A.(x,y) = A.x,A.y
: list range 0:end-1
( ) grouping (A,B).x

A multiple read returns lists nested like the groups: $R((A,B).(x,y)) gives {1 2} {3 4}; $L(0:2,4) gives {10 20 30} 50; $L(*0:2,4) gives 10 20 30 50.

Per container. Product: $a(name), $R(A.(x,y)); sum: $v(i) reads the current variant; list: $L(2), $L(1:3), $L(0,2); dict: $D(a), $D('end'); bitarray: $b(3), $b(1:4). Writes go through the variable's contract and are revalidated. Reading then writing back the same path always works: set R(c) [set R(c)].

A path substitutes nothing. Tcl substitutes $x, [cmd] and [(expr)] in $R(...) and in bare words; each substituted value is one leaf, never parsed again as a path. In braces and in ${...}, $ and [ are errors. Data is therefore never executed by the path parser.

set k file.txt ; $D($k)      ;# the key "file.txt", not D.file.txt
set idx {0 2}  ; $L(*$idx)   ;# * unfolds a list value
set p {[exec ls]} ; $L($p)   ;# nothing runs: bad index "[exec ls]"

A computed path is read by naming the variable with it: set ref R($path); set $ref.

Spaces. Spaces are allowed in $R(...), ${R(...)} and in an annotated word {T}R(...). In a bare word (set R(A.x, A.y) ...) Tcl splits at spaces as it always has.

Arrays are unchanged. For an array, or for an index that is not a well-formed path, the Tcl 9.0 rule applies exactly. In particular a parenthesized index is not an expression: set A((1+1)) 3 keeps the key (1+1); a computed index is written A([(1+1)]) (TIP 759).

5. C level

No new stub functions. The public C API (tcl.decls) is unchanged. The additions visible from C are two token types and one object type.

Parser tokens (tcl.h):

token value produced by meaning
TCL_TOKEN_ANNOT 512 Tcl_ParseCommand, ParseTokens the annotation of a word {text}rest; the tokens of rest follow
TCL_TOKEN_INDEX 1024 Tcl_ParseVarName, only when the extended index syntax is used wraps the components of an index $name(...); an index valid in Tcl 9.0 keeps exactly its Tcl 9.0 tokens

Tools that walk token arrays (testparser, static checkers) must accept these two types.

The annotated object. An annotated value is a Tcl_Obj of type annotated. Its string is the string of its content, so any code that ignores annotations sees the content.

Tcl_Obj (typePtr = &annotatedObjType)
  bytes/length     the content's string, computed on demand
  twoPtrValue.ptr1 the content V: an ordinary, shared Tcl_Obj
  twoPtrValue.ptr2 the annotation, one of:
                   - a resolved type reference
                   - the raw text of an unresolved annotation
                   - "" (the empty annotation {})
                   - a state {T, N}: resolved type T (or NULL) and notes N

The object is immutable and shared: Tcl_DuplicateObj shares the state, and annotating, unstacking or removing a note makes a new object. setFromAnyProc refuses: an annotated value is born only from the parser or a typed assignment (TCL VALUE ANNOTATED). The type is a Tcl 9 abstract list (TCL_OBJTYPE_V2), so lindex, llength, lrange, foreach and lset return typed elements of a list.T. Core conversions (Tcl_GetIntFromObj, dict and string access) read the content and never convert an annotated value in place.

Compilation.

Error codes. Type errors use TCL TYPE ... (INT, SUM, DICT, LIST, FIELD, PATH, MISMATCH, CONTRADICT, UNKNOWN, LOCALANNOT, STALE, DEREF) and TCL PARSE TYPE BADCHAR. Path parsing errors use TCL PARSE INDEX ...; wrong argument counts use TCL WRONGARGS.

Compatibility

No valid script changes meaning. Both syntaxes take over forms that Tcl 9.0 rejects or reads the same way:

Checks on the reference implementation.

Dependencies. The expr assignments x = v and (a,b) = $L, and the [(expr)] substitution, come from TIP 759. This TIP adds v is T and the declaring assignment x is T = v. Without TIP 759, typed variables are declared with set only.

Performance. The cost for unannotated code is one test on the object type in the hot paths (INST_STORE_SCALAR, conversions). It has not yet been measured against stock Tcl 9.0; this is an open issue.

Reference Implementation

A prototype on core-9-0-branch with the TIP 759 branch, by Florent Merlet with Claude. It adds seven files (about 15,000 lines of C) and hooks in about twenty core files.

file content
tclAnnotObj.c the annotated object, the {T, N} state, resolution, note
tclTypes.c typology, the type command, validation, widening, casts
tclTypeParse.c parser and compiler of type expressions
tclTypeGeneric.c parameterized types, instances, phantom parameters
tclTypedVar.c typed variables and constructors
tclTypePath.c path language: read, write, construction, path index object
tclVarRef.c var links, including links to variables that are gone

The main core hooks are in tclParse.c (tokens), tclCompile.c and tclCompCmdsSZ.c (compilation), tclExecute.c (instructions), tclVar.c (the contract), tclProc.c (typed arguments) and tclCompExpr.c (is).

Rejected Alternatives

alternative why not
a new quoting rule for annotations {x}word takes over a syntax error and needs none
glued stacking {a}{b}$x ambiguous with {int}{300}
resolving types wherever {T}$x appears checking belongs where the value is stored; elsewhere the annotation is free and never fails
the bare name/value form set {pt}a name Jean age 12 read two ways when n arguments match n fields; names are now written in the source
mixing field tags and positions a disguised positional form
a path that substitutes $ and [ data would be executed by the path parser
(expr) as an index breaks the braces rule, clashes with path grouping, and changes Tcl 9.0 keys such as (1+1)
path + field tags (set {rect}R(A) {y}2 {x}1) no proven need yet

Open Issues

Copyright

This document has been placed in the public domain.