| Ticket UUID: | 1ecea01196a19b73226d5c822e5fdaf22111aa54 | ||
| Title: | string is lower, upper not consistent with Unicode database and other languages | ||
| Type: | Bug | Created on: | 2025-07-23 05:13:00 |
| Submitter: | apnadkarni | Assigned to: | nobody |
| Subsystem: | 18. Commands M-Z | Severity: | Important |
| Priority: | 5 Medium | Last modified: | 2026-09-20 11:31:32 |
| Status: | Open | Closed by: | nobody |
| Resolution: | None | Closed on: | |
| Version: | |||
| Description: | ||||
|
For example,
On the other hand,
FWIW, Perl, Python, Ruby, Java all report it as lower case. I estimate over a hundred such mismatches. The issue is that although Tcl also builds its tables using the Unicode data files, for determining lower or upper case, it only looks at the I believe Tcl should do the same. If there is agreement on this, the next question is whether changing this can be treated as a bug fix (hopefully!) or needs a TIP. Opinions on both counts? | ||||
| User Comments: | ||||
apnadkarni added on 2025-07-23 05:52:47:
As a reminder, this is also pertinent to Tcl regular expressions. oehhar added on 2025-07-23 06:29:04:
Great catch! I vote for bugfix, as there is IMHO no discussion if this is senseful. It is just wrong. Thanks, Harald oehhar added on 2025-07-23 07:37:08:
And no bugfix for 8.6 and an entry to the migration page ;-). We are getting better each day! Thanks, Harald apnadkarni added on 2025-07-23 09:57:28:
I was not planning on making a change even for 9.0.x, only 9.1 onward. I think it is substantial enough to need a minor release, not just a patchlevel. sebres added on 2025-07-23 17:08:27:
At least regarding RE, I am unsure it shall be always true... For instance PCRE and PCRE2 don't consider it as lowercase:
or PoC from bash:
Also JS agrees with that:
It can be simply the bugs in both RE-engines, but the root of issue may be different... Apropos other langs like Python does report it as lowercase. They do indeed, but how about alternative simply logic - could it be called lowercase if there is simply no uppercase for that char? With other words, why then this behaves like it behaves here?
Using the logic (detect lowercase only if it has uppercase too), may be the explanation for the issues with RE. Yes, there are also regular lowercase characters that do not have corresponding uppercase representations, like the German sharp s (ß), that do not have a direct uppercase equivalent. Just take a look what python does for that char:
Another cause (why REs don't recognize them) may be simply the decision to ignore all chars from other letters, modifier letters and other similar categories (for unknown but maybe justified reasons). However I really don't understand how ordinal indicators (which class basically represents indicators following ordinal numbers), or even such as modifier letter like ʶ ("u02B6"), which is capital inverted R, may be lowercase at all... Not trying to convince or propagate what is right here, but to show the subject may be more complex as it is. apnadkarni added on 2025-07-24 05:07:44:
In response to Sergey, of course the subject of case is complex, particularly conversion or folding, which is why I only addressed the
The issue with PCRE is the same as with Tcl except that PCRE2 clearly documents the semantics that
For the same reason. It is documented as general category lookup. My contention is that it is not desired behavior for categorizing lower case characters because in my opinion lower case characters should be exactly those to which the Unicode standard assigns the
Characters that do have a lowercase/uppercase property do not necessarily have a upper case, lower case equivalent. The mapping function then maps it to itself (just as
That seems correct to me. Python seems to use the Uppercase_Mapping property as opposed to Simple_Uppercase_Mapping. As I said mapping is a different kettle of fish than classification and needs a more complex discussion. apnadkarni added on 2025-07-24 05:13:03:
There is another semantic difference even for classification that I did not realize when the string contains multiple characters. Consider Tcl's
with Python's
Python subscribes to the Unicode definition of isLowercase where a string is lower case if each character in the string maps to itself under tolower operation. Thus Aside from the fact that some clarity in the Tcl manpage would be helpful, I don't consider this a bug in Tcl but simply different defined semantics. I would not change Tcl's behaviour in this regard but instead (only if deemed useful) analogous commands reflecting the Unicode standard definitions should be added to the (Note this issue is orthogonal to Tcl's lowercase classification of individual code points like serhiy.storchaka added on 2026-09-20 11:31:32:
Some precision on Python, since it was cited: for a single character str.islower() tests the Unicode derived property Lowercase (general category Ll plus Other_Lowercase) and str.isupper() the Uppercase property, which is what this ticket proposes. For a string the rule is "all cased characters are lowercase and there is at least one cased character", so "2a".islower() is true, but "2".islower() and "".islower() are false. It is not the definition D139 of the standard (toLowercase(X) = X), which would accept "2" and "". Note that Tcl is also inconsistent with itself here: string toupper \u2170 gives \u2160 and string toupper \u24d0 gives \u24b6 (both are Other_Lowercase, with categories Nl and So), but string is lower returns 0 for them. Using the Lowercase property would make string is lower agree with string toupper. | ||||
