Author:	        Ashok P. Nadkarni <[email protected]>
State:          Draft
Tcl-Version:    9.2
Type:           Project
Created:        2026-06-18
Tcl-Branch:     tip-756
Forum:          5abd17b8ab
Keywords:       Tcl, strings, Unicode

Abstract

This TIP proposes commands and a C API for operations on extended grapheme clusters, a term referred to simply as grapheme in this TIP. See Unicode® Standard Annex #29 for a formal definition.

The TIP is intentionally limited to operations that are defined to be locale-independent. Operations such as movement across word boundaries are therefore omitted. These may conceivably be built on top of the operations defined here as part of a future TIP.

While this TIP is targetted towards 9.2, the commands will also be available in 9.1 under the ::tcl::unsupported namespace to gain experience in real-world use.

Rationale

Tcl's string related commands operate on Unicode code points, which do not always correspond to what the user perceives as a character. Examples include

the character "é" may be represented by the sequence of Unicode code points U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT.
the family Emoji consists of multiple Unicode code points MAN, GIRL, WOMAN, BOY separated by ZERO WIDTH JOINER.
the CR LF sequence used to represent line endings on Windows and in many network protocols.

In each case, a single code point is only one component of the internal representation of what constitutes a character from the user perspective. Operating on code points therefore results in incongruous or downright incorrect results. A few examples:

Indexing or range operations may return fragments, for example just the combining mark, not the entire user-perceived character
Cursor movement based on code points will incorrectly position the cursor
String lengths will not match the lengths observed by the user
Operations such as reversing or splitting will return corrupted results such as combining marks applied to the wrong character.

Fixing these issues requires commands to operate on graphemes, not code points. This TIP proposes the requisite commands.

Specification

Commands

The following new commands are proposed.

grapheme index STRING GRINDEX
grapheme length STRING
grapheme next STRING ?STRINDEX? ?VARNAME?
grapheme offset STRING GRINDEX
grapheme prev STRING ?STRINDEX? ?VARNAME?
grapheme range STRING GRFIRST GRLAST
grapheme reverse STRING
grapheme split STRING

In the above GRINDEX, GRFIRST and GRLAST refer to grapheme indices. STRINDEX refers to string (code point) indices as used in the string command. All may be in the simple expression form accepted by existing Tcl string commands.

The length, index, range and reverse commands are analogous to their string counterparts except that they operate on g bbraphemes, not code points.

The split command returns a list of the graphemes that comprise the string. Note: perhaps a different name may be chosen to avoid confusion with the split command.

The next command returns grapheme at string index STRINDEX which defaults to 0. If VARNAME is passed, the string index of the following grapheme is stored in it. If STRINDEX is negative, it is treated as 0. If equal to the string length or greater an empty string is returned and VARNAME contains the length of the string. The token end in the STRINDEX expression is treated as the length of the string.

The prev command returns grapheme preceding string index STRINDEX which defaults to the length of the string. If VARNAME is passed, the string index of the returned grapheme is stored in it. If STRINDEX is not positive, an empty string is returned and VARNAME is set to 0. A value greater than length of the string is treated as length of the string. The token end in the STRINDEX expression is treated as the length of the string.

The offset maps a grapheme index into STRING to the corresponding string index. Returns -1 if the grapheme index is out of range.

C API

TBD

Implementation notes

Implementation is in the tip-756 branch.

Implementation (in the tcl::unsupported namespace) is in the tip-756-91 branch.

Copyright

This document has been placed in the public domain.