Tcl Source Code

View Ticket
Login
Ticket UUID: 1ecea01196a19b73226d5c822e5fdaf22111aa54
Title: string is lower, upper not consistent with Unicode database and other languages
Type: Bug Created on: 2025-07-23 05:13:00
Submitter: apnadkarni Assigned to: nobody
Subsystem: 18. Commands M-Z Severity: Important
Priority: 5 Medium Last modified: 2026-09-20 11:31:32
Status: Open Closed by: nobody
Resolution: None Closed on:
Version:
Description:

For example,

% string is lower \u00aa
0
% regexp {[[:lower:]]} \u00aa
0

On the other hand,

% tcl::unsupported::icu is lower \u00aa; # But see caveat below
1
% unicode is lower \u00aa; # Built off the UCD database
1

FWIW, Perl, Python, Ruby, Java all report it as lower case.

I estimate over a hundred such mismatches.

The issue is that although Tcl also builds its tables using the Unicode data files, for determining lower or upper case, it only looks at the General_Category property of the code point to make the determination. The problem is that the General_Category is a disjoint partition so its value only indicates the "most common" usage, not all possible categorizations. That is a problem for code points that can be placed in multiple categories. The standard therefore defines separate Lowercase and Uppercase properties for this purpose and that is what all other languages look at to characterize a code point as lower or upper case.

I believe Tcl should do the same. If there is agreement on this, the next question is whether changing this can be treated as a bug fix (hopefully!) or needs a TIP.

Opinions on both counts?

User Comments:
apnadkarni added on 2025-07-23 05:52:47:
As a reminder, this is also pertinent to Tcl regular expressions.

oehhar added on 2025-07-23 06:29:04:
Great catch!
I vote for bugfix, as there is IMHO no discussion if this is senseful.
It is just wrong.

Thanks,
Harald

oehhar added on 2025-07-23 07:37:08:
And no bugfix for 8.6 and an entry to the migration page ;-).
We are getting better each day!
Thanks,
Harald

apnadkarni added on 2025-07-23 09:57:28:
I was not planning on making a change even for 9.0.x, only 9.1 onward. I think it is substantial enough to need a minor release, not just a patchlevel.

sebres added on 2025-07-23 17:08:27:

At least regarding RE, I am unsure it shall be always true...

For instance PCRE and PCRE2 don't consider it as lowercase:
(\p{Ll} token is the same as [[:lower:]] in tcls NFA)

% regexp -type pcre {^\p{Ll}+$} \u0061\u00e4\u0430
1
% regexp -type pcre {^\p{Ll}+$} \u00aa
0

or PoC from bash:

$ printf "\\u0061\\u00e4\\u0430\n" | pcre2grep -q '\p{Ll}' && echo 1 || echo 0
1
$ printf "\\u00aa\n" | pcre2grep -q '\p{Ll}' && echo 1 || echo 0
0

Also JS agrees with that:

> /^\p{Lowercase_Letter}+$/u.exec("\u0061\u00e4\u0430") && true
true
> /^\p{Lowercase_Letter}+$/u.exec("\u00aa") && true
null

It can be simply the bugs in both RE-engines, but the root of issue may be different...

Apropos other langs like Python does report it as lowercase. They do indeed, but how about alternative simply logic - could it be called lowercase if there is simply no uppercase for that char?

With other words, why then this behaves like it behaves here?

>>> "\u00aa".islower()
True
>>> "\u00aa".isupper()
False
>>> "\u00aa".lower() == "\u00aa".upper()
True
>>> "\u00aa" == "\u00aa".lower() and "\u00aa" != "\u00aa".upper()
False

Using the logic (detect lowercase only if it has uppercase too), may be the explanation for the issues with RE.

Yes, there are also regular lowercase characters that do not have corresponding uppercase representations, like the German sharp s (ß), that do not have a direct uppercase equivalent. Just take a look what python does for that char:

>>> "ß".lower() == "ß".upper()
False
>>> "ß" == "ß".lower() and "ß" != "ß".upper()
True
>>> "ß".lower()
'ß'
>>> "ß".upper()
'SS'

Another cause (why REs don't recognize them) may be simply the decision to ignore all chars from other letters, modifier letters and other similar categories (for unknown but maybe justified reasons).

However I really don't understand how ordinal indicators (which class basically represents indicators following ordinal numbers), or even such as modifier letter like ʶ ("u02B6"), which is capital inverted R, may be lowercase at all...

Not trying to convince or propagate what is right here, but to show the subject may be more complex as it is.


apnadkarni added on 2025-07-24 05:07:44:

In response to Sergey, of course the subject of case is complex, particularly conversion or folding, which is why I only addressed the islower behavior and not the tolower,toupper which is a bigger can of worms and needs more thought.

For instance PCRE and PCRE2 don't consider it as lowercase

The issue with PCRE is the same as with Tcl except that PCRE2 clearly documents the semantics that \p{Ll}, \p{Nd} etc. refer to the Unicode general category properties "Ll", "Nd" etc.

Also JS agrees with that:

For the same reason. It is documented as general category lookup.

My contention is that it is not desired behavior for categorizing lower case characters because in my opinion lower case characters should be exactly those to which the Unicode standard assigns the Lowercase property. I would not say PCRE or JS have a bug because they specifically document the semantics of \p{} as membership in a specific general category. Still, I don't think it is desirable behavior. Tcl on the other hand does not document is lower in terms of Unicode general categories at all.

"u00aa" == "u00aa".lower() and "u00aa" != "u00aa".upper() False

Characters that do have a lowercase/uppercase property do not necessarily have a upper case, lower case equivalent. The mapping function then maps it to itself (just as string toupper 1 -> 1) so the second clause above is false.

>>> "ß".upper() > 'SS'

That seems correct to me. Python seems to use the Uppercase_Mapping property as opposed to Simple_Uppercase_Mapping. As I said mapping is a different kettle of fish than classification and needs a more complex discussion.


apnadkarni added on 2025-07-24 05:13:03:

There is another semantic difference even for classification that I did not realize when the string contains multiple characters. Consider Tcl's

% string is lower 2a
0

with Python's

>>> "2a".islower()
True

Python subscribes to the Unicode definition of isLowercase where a string is lower case if each character in the string maps to itself under tolower operation. Thus 2 passes the test. Tcl on the other hand uses a definition that each character in the string must be classified as lowercase which is false for 2.

Aside from the fact that some clarity in the Tcl manpage would be helpful, I don't consider this a bug in Tcl but simply different defined semantics. I would not change Tcl's behaviour in this regard but instead (only if deemed useful) analogous commands reflecting the Unicode standard definitions should be added to the unicode command. That would of course require a TIP (not 726, because I do not want to conflate this issue with normalization).

(Note this issue is orthogonal to Tcl's lowercase classification of individual code points like \u00aa which was the original ticket I logged. Even if that was "fixed" it would not affect this difference)


serhiy.storchaka added on 2026-09-20 11:31:32:

Some precision on Python, since it was cited: for a single character str.islower() tests the Unicode derived property Lowercase (general category Ll plus Other_Lowercase) and str.isupper() the Uppercase property, which is what this ticket proposes. For a string the rule is "all cased characters are lowercase and there is at least one cased character", so "2a".islower() is true, but "2".islower() and "".islower() are false. It is not the definition D139 of the standard (toLowercase(X) = X), which would accept "2" and "".

Note that Tcl is also inconsistent with itself here: string toupper \u2170 gives \u2160 and string toupper \u24d0 gives \u24b6 (both are Other_Lowercase, with categories Nl and So), but string is lower returns 0 for them. Using the Lowercase property would make string is lower agree with string toupper.