| Ticket UUID: | d63061a1ac4003efe490a216757df4fdd36d61aa | ||
| Title: | PRIVATE != CONTROL in Unicode | ||
| Type: | Bug | Created on: | 2024-01-15 10:44:01 |
| Submitter: | jan.nijtmans | Assigned to: | nobody |
| Subsystem: | 44. UTF-8 Strings | Severity: | Minor |
| Priority: | 5 Medium | Last modified: | 2026-10-22 17:19:29 |
| Status: | Closed | Closed by: | serhiy.storchaka |
| Resolution: | Fixed | Closed on: | 2026-10-21 10:03:35 |
| Version: | 8.6 | ||
| Description: | ||||
|
Example: % string is control \u000A # Unicode category "Cc" 1 % string is control \uE000 # Unicode category "Co" 1 % string is control \uFFFF # Unicode category "Cn" 0 % string is control \uFC00 # Unicode category "Cs" 0 Currently, "string is control" gives "1" if the character is of one of the Unicode categories "Cc" (a C0 or C1 control code), "Cf" (a format control character), "Co" (a private-use character), but not for "Cs" (a surrogate code point) or "Cn" (a reserved unassigned code point or a noncharacter). Why is "Co" (a private-use character) included? It's not logical: This range is normally used for Glyphs (in e.g. China/Japan) which didn't have an entry assigned yet in an official Unicode version. Normally those are not Control codes. Therfore, it would be more logical to leave out characters from the "Co" category from the "string is control" implementation. For Tcl 8.6, this only affects the range from U+E000 - U+F8FF. For Tcl 8.7, it affects the ranges U+F0000 - U+FFFFD and U+100000 - U+10FFFD as well. | ||||
| User Comments: | ||||
jan.nijtmans added on 2024-01-18 07:45:33:
This ticket is not important enough to rush into Tcl 8.6.14. I will start dicussion about it after 8.6.14 is released. But ... if someone (Ashok ....) want to comment on it already, here is the place to do it. apnadkarni added on 2024-01-19 06:14:11:
I would agree that Co being classified as a control character does not make sense and should be removed. Additionally, I have mixed feelings about Cf characters also being classified as control characters. On one hand, I can see why if they are referenced as "format controls". On the other hand, I think most programmers' expectation is that the string is matched against the "traditional" Cc category members. I suppose we can punt that distinction to the future when general Unicode categorization of code points is added to Tcl. jan.nijtmans added on 2024-01-19 12:41:57:
> I would agree that Co being classified as a control character does not make sense and should be removed.
Thanks!
> I suppose we can punt that distinction to the future
I'm thinking about "string is Cc", "string is Cf" for this. But it would also
fit into the regexp syntax as \p{Cc} \p{Cf}. See [https://unicode.org/reports/tr18/].
Of course, this would be a hell of a job to implement completely. But starting
with "1.2.5 General Category Property", short form only, would be nice.
jan.nijtmans added on 2024-01-21 16:16:05:
See: [3464428]. Looks like Ruby got it 'wrong' too ... serhiy.storchaka added on 2026-10-21 10:03:35:
Fixed by [0480bdc823] (in 8.6.15 and 9.0.0): private-use characters (category Co) are no longer string is control. Cc and Cf are unchanged. | ||||
