| Ticket UUID: | 27944a3661a5e409dead945dbc8f9e4a50033e62 | ||
| Title: | Taming test utf-6.88 | ||
| Type: | Bug | Created on: | 2020-04-17 15:32:44 |
| Submitter: | dgp | Assigned to: | dgp |
| Subsystem: | 44. UTF-8 Strings | Severity: | Important |
| Priority: | 5 Medium | Last modified: | 2020-04-29 13:00:19 |
| Status: | Open | Closed by: | nobody |
| Resolution: | None | Closed on: | |
| Version: | 8.5 | ||
| Description: | ||||
Fresh ticket to sort out what we expect from test utf-6.88 with body:
testutfnext \xE8\xA0\xA01
The bytes are the proper 3-byte UTF-8 sequence for U+8821.
The test checks what happens when Tcl_UtfNext is called with the
argument a pointer to the \xA0 byte.
This ticket is to sort out what is expected to happen, and how to
reconcile that with related needs.
| ||||
| User Comments: | ||||
jan.nijtmans added on 2020-04-19 10:02:40:
Here is the documentation from Utf.n: Given \fIsrc\fR, a pointer to some location in a UTF-8 string, \fBTcl_UtfNext\fR returns a pointer to the next UTF-8 character in the string. The caller must not ask for the next character after the last character in the string if the string is not terminated by a null character. The wording in the source code of tclUtf.c is less clear, so I propose to copy the wording from Utf.n to tclUtf.c In the given example, the \xA0 byte is a valid 'some location in a UTF-8 string'. The next character starts at index 3, so that should be the answer. Proposed fix in commit: [74d3f929c887755e]. I leave it to you to implement the same in the optimized implementation you are working on. This is a long-standing bug. Since Tcl never calls Tcl_UtfNext() to random locations in a string, there were never test-cases covering this situation. Now there are: Thanks for that, Don! jan.nijtmans added on 2020-04-19 10:05:48:
The misleading text in tclUtf.c was: move forward one character What the implementation always did was: move forward one byte That's - I think - what caused the confusion. jan.nijtmans added on 2020-04-22 10:36:28:
Fix committed now to core-8-branch and trunk. I would like to have it backported to 8.5 and 8.6 too. dgp added on 2020-04-22 19:21:54:
Placed first-draft implementation of the proposed change to handling of trail bytes by Tcl_UtfNext on branch dgp-27944a3661 for examination against the 8.5 branch. dgp added on 2020-04-22 19:26:22:
The tests that illustrate the change in behavior in the default build are
utf-6. {11, 88, 89, 92, 93}
dgp added on 2020-04-22 19:38:24:
In an unsupported (TCL_UTF_MAX=4) build, exactly the same tests change their behavior in exactly the same way. dgp added on 2020-04-22 19:40:14:
Added tests to demonstrate that the tolerance of Tcl_UtfNext for a sequence of trail bytes has a length limit. dgp added on 2020-04-22 19:52:24:
If it is generally correct to limit the acceptance of trail bytes to a length of 3 (because in a valid, well-formed UTF-8 string made up of complete characters that's the longest trail byte sequence that may appear), the in the default (TCL_UTF_MAX = 3) build, the limit ought to be 2. dgp added on 2020-04-22 20:20:04:
At this point there is a problem that when p points to a trail byte, Tcl_UtfCharComplete(p, 1) returns true, indicating that a call to Tcl_UtfNext(p) will examine only the byte (*p). That is no longer true with this patch. jan.nijtmans added on 2020-04-22 21:07:47:
Indeed, there still is a problem, but there is progress in your understanding. Great! Looks like you are starting to see it. jan.nijtmans added on 2020-04-22 21:09:09:
Now, look at the documentation for Tcl_UtfMext(): <pre> Given \fIsrc\fR, a pointer to some location in a UTF-8 string, \fBTcl_UtfNext\fR returns a pointer to the next UTF-8 character in the string. The caller must not ask for the next character after the last character in the string if the string is not terminated by a null character. </pre> jan.nijtmans added on 2020-04-22 21:09:32:
Now, look at the documentation for Tcl_UtfNext(): Given \fIsrc\fR, a pointer to some location in a UTF-8 string, \fBTcl_UtfNext\fR returns a pointer to the next UTF-8 character in the string. The caller must not ask for the next character after the last character in the string if the string is not terminated by a null character. jan.nijtmans added on 2020-04-22 22:31:10:
Other part of the documentation: \fBTcl_UtfCharComplete\fR returns 1 if the source UTF-8 string \fIsrc\fR of \fIlength\fR bytes is long enough to be decoded by \fBTcl_UtfToUniChar\fR, or 0 otherwise The function Tcl_UtfNext() is - and always has been - documented to require the string to be 0-terminated. There never has been a relation between Tcl_UtfCharComplete() and Tcl_UtfNext(). There never has been a loop in Tcl where Tcl_UtfCharComplete() was used for checking whether Tcl_UtfNext() can be called. There was even never documented how many bytes Tcl_UtfNext() is allowed to read in the case of absence of the terminating 0-byte. But since Tcl_UtfPrev() is documented to read a maximum of TCL_UTF_MAX bytes, it's logical to assume the same for Tcl_UtfNext(). This means, care is required replacing Tcl_UtfToUniChar() calls with Tcl_UtfNext(). Tcl_UtfNext() is not simply an optimized replacement for Tcl_UtfToUniChar(). Those two functions behave differently when encountering invalid byte sequences. dgp added on 2020-04-22 22:54:11:
The implementation of Tcl_UtfNext has ***always*** been return src + Tcl_UtfToUniChar(src, ...); Logically any constraint on calling Tcl_UtfToUniChar safely transfers the same constraint on calling Tcl_UtfNext() safely. This implementation has also tied together the functionality of the two routines in ways callers may expect. The original Tcl_UtfNext() documentation did not mention NUL termination. That wording was added in checkin https://core.tcl-lang.org/tcl/info/5154e4e97b390103 at the time when the behavior of Tcl_UtfPrev when passed a trail byte pointer was trying to be clarified. [769895] The original Tcl_UtfNext() docs did state "The caller must not ask for the next character after the last character in the string" with no explicit direction on how to accomplish that. The reader could reasonably infer the same safety tool for Tcl_UtfToUniChar was appropriate for the job. Someone source diving would confirm that inference was correct. Because of the implementation Tcl_UtfNext() always had, it never had the function of acting on trail bytes in the way being sought here. No caller expects it, because it's never been that way. I'm still willing to follow this exploration trail further to see if there's a reasonable way to add this extension of the functionality consistent with the other expectations. jan.nijtmans added on 2020-04-23 07:54:42:
> I'm still willing to follow this exploration trail further to see if there's a reasonable > way to add this extension of the functionality consistent with the other expectations. Thanks! I see two possible solutions, both consistent with the current documentation and respecting the Tcl_UtfCharComplete() contract. Your latest commit on core-8-5-branch is close, but not 100% there yet: A few test-cases still show inconsistancies. And there is a regression in Tcl_UtfAtIndex(), not covered by any test-case yet. I'll come back on that. dgp added on 2020-04-23 13:11:29:
I want to mention this related dispute in the Unicode world over how invalid byte sequences found in UTF-8 ought to be converted into replacement characters. We're not the only ones struggling a bit to find the right compromises between inconsistent aims. https://hsivonen.fi/broken-utf-8/unicode-submission.html dgp added on 2020-04-23 15:33:38:
Good catch on the macro masking the impact of the changes. I've incorporated that fix into the dev branch. Note the impact on how 4-byte sequences get interpreted. Previously, the byte sequence \xF0\x9F\x92\xA9 would be seen as 4 units \xF0 \x9F \x92 \xA9 But now gets seen as \xF0 \x9f\x92 \xA9 That changes the string length from 4 to 3. dgp added on 2020-04-23 15:51:09:
Even without delving into 4-byte sequences, this change to Tcl_UtfNext induces a change in handling of overlongs. % testnumutfchars [testbytestring \xE0\x80\x80] 2 The answer has been 3. But the new view of skipping to the end when found on a trail byte joins the two trail bytes together as a unit. dgp added on 2020-04-23 15:54:36:
Well, no. In Tcl 8.5 releases, the answer to testnumutfchars [testbytestring \xE0\x80\x80] has been 1 because overlongs were accepted, but we've decided that was a bug. jan.nijtmans added on 2020-04-23 19:24:08:
I'm veeeeery sorry, but I believe that changing Tcl_UfToUniChar() to Tcl_UtfNext() in random places in the source-code is too dangerous. A regression in Tcl_NumUtfChars() is already discovered, I found 3 more places where such a change was done. Since we know that Tcl_UtfNext() doesn't work the same as Tcl_UtfToUniChar(), we know it's not the same when handling invalid byte sequences. The Tcl test suite doesn't have enough test-cases to find such places. So, such a change can only done after preparing a proper test-case, showing that the change is harmless. I recommend to undo the same changes in Tcl 8.5 too. Again, Sorry! jan.nijtmans added on 2020-04-23 19:26:34:
> ... has been 1 because overlongs were accepted, but we've decided > that was a bug. I agree that 3 should be the correct answer. jan.nijtmans added on 2020-04-23 19:33:02:
Don, why are you moving my commits to branch [utf-next-regresions] ???? dgp added on 2020-04-23 19:43:00:
I opened this ticket and put together the work on the dgp-27844a3661 branch "to sort out what is expected to happen, and how to reconcile that with related needs." I think I've now taken that branch to its limit. It demonstrates that the following two goals are not reconcilable. 1) Extend the function of Tcl_UtfNext() so that it can (as much as possible) jump to the valid start of the next character when given an argument pointer to the middle of a multi-byte sequence. 2) Retain the coupling between Tcl_UtfNext() and Tcl_UtfToUniChar() so that both of them advance the same amount with one call. I think we've convinced each other that it is not possible to achieve both aims. It appears we disagree about which aim should take priority. Fair summary? jan.nijtmans added on 2020-04-23 20:06:15:
Fair summary! jan.nijtmans added on 2020-04-23 20:17:28:
It also means that in Tcl8.7, given TIP #389 and #542, the function Tcl_UtfToUniChar() got additional functionality to handle surrogates. It's impossible to keep the tight coupling, as in 8.5 and 8.6 currently. So, forwarding the improvements, handling overflow/underflow checks in Tcl_UtfNext()/Tcl_UtfPrev() has to be done very carefully. Replacing Tcl_UtfToUniChar() with Tcl_UtfNext() there in random places, is really opening a can of worms. For 8.5 and 8.6 I see no other way that keep the tight coupling as you describe. dgp added on 2020-04-23 20:30:58:
Did those changes get merged into 8.7? I left them in dgp-fixme, I thought. Set that aside. If there's trouble along those lines, it should show up in a TCL_UTF_MAX=4 build of Tcl 8.6 just as well, right? dgp added on 2020-04-23 23:44:37:
For any const char string[LEN] = "...."; and any in-range value of int index = ...; Given this sequence: Tcl_UniChar one=0; Tcl_UtfToUniChar(Tcl_UtfAtIndex(string, index), &one); and this sequence: Tcl_UniChar two = Tcl_GetUniChar(Tcl_NewStringObj(string, LEN), index); I expect to always end with the result: (one == two) The issue here is consistency of indexing results via two separate paths. If there's any release of 8.5 or 8.6 where that's not true in a default build, that would be useful to know. For now limiting our attention to the default builds of Tcl 8.5 and 8.6 releases, I would expect future releases to continue making that guarantee. dgp added on 2020-04-24 00:01:00:
Combine that expectation with the documentation: "Tcl_UtfAtIndex ... is equivalent to calling Tcl_UtfNext index times." And I believe the conclusion is that Tcl_UtfNext and Tcl_UtfToUniChar must remain coupled in their increments at least in default builds of patch releases of Tcl 8.5 and 8.6. dgp added on 2020-04-24 00:10:53:
Since we've demonstrated that the revised behavior requested in this ticket is irreconcilable with keeping the increments of Tcl_UtfToUniChar and Tcl_UtfNext in agreement, it follows that the request of this ticket must be denied in (the default builds of) Tcl 8.5 and 8.6. It might be considered at the earliest for Tcl 8.7. jan.nijtmans added on 2020-04-24 10:48:54:
> Did those changes get merged into 8.7? I left them in dgp-fixme, I thought. I merged them to 8.7 and trunk now. But without the Tcl_UtfToUniChar() -> Tcl_UtfNext() changes, since Tcl_UtfNext() steps over the full 4 bytes (since 4-byte UTF-8 sequences are now valid) while Tcl_UtfToUniChar() handles 4-byte sequences as if they are 2 surrogates (in a TCL_UTF_MAX=3 build). In a TCL_UTF_MAX=4 build, Tcl_UtfToUniChar() is able to step 4 bytes. > Set that aside. If there's trouble along those lines, it should show up > in a TCL_UTF_MAX=4 build of Tcl 8.6 just as well, right? In a Tcl 8.6 TCL_UTF_MAX=4 build (which is - actually - unsupported anyway), Tcl_UtfNext() is able to step 4 bytes, while Tcl_UtfToUniChar() will need two steps for this. So using Tcl_UtfNext() with an index correction (as you - apparently - prefer, I don't ....) behaves the same as using Tcl_UtfToUniChar(). I'm not aware of any trouble. dgp added on 2020-04-24 12:38:42:
I agree that the plainest reading of the existing docs for Tcl_UtNext suggests that the behavior requested in this ticket should be expected. But it has never been present. No existing callers expect it, and no ticket has ever been filed asking for it out of a programming need. We are only doing it now out of a desire to make the code and documentation match. I think the better option is to change the documentation so it more clearly describes what the routine does. (If a programmer has a pointer he suspects points to the middle of a sequence, he can first pass it to Tcl_UtfPrev, which will return a new pointer known to point to a character start. Calling Tcl_UtfNext on that will deliver the overall functionality sought.) Contrast with the coupling of Tcl_UtfNext and Tcl_UtfToUniChar which has been in place in the most direct way -- implementing one in terms of the other -- and which underlies the consistency of indexing that it is much more plausible to imagine existing programs relying on. The need for Tcl_UtfToUniChar to change to satisfy TIP 389 creates a new situation in Tcl 8.7. The incompatibilities have been approved by the TIP at least implicitly. The clarity of calling attention to them might need some improvement. In the current implementation of 8.7, the documented connection between Tcl_UtfNext and Tcl_UtfAtIndex has been broken. I intend to explore some alternatives in a branch that might re-couple UtfNext and UtfToUniChar. I'll open a new ticket to track that, and keep it off the 8.7 branch until it gets some review. Meanwhile I think the collection of new tests has been helpful in uncovering bugs, and we can bring those tests forward, adjust where appropriate, and make fixes for them. dgp added on 2020-04-24 13:02:10:
I have an idea how to resolve this conflict. dgp added on 2020-04-24 13:04:49:
It is now clear to me that merging forward the Tcl_UniToUniChar -> Tcl_UtfNext transformation to Tcl 8.7 will at least require significant work, if not be impossible. It would not be a good end state to have the 8.5, 8.6, and 8.7 branches different in this way. If the change is a dead end by 8.7, better to keep the branches consistent. dgp added on 2020-04-24 13:10:14:
So I propose I go back to my work on 8.5, and backout those disputed changes on the main core-8-5-branch. I'll place them on a feature branch instead for now. Same on 8.6. And so on. Later if I can demonstrate a plan that can bring this reform successfully to all branches, I can bring them all in then. What I need in return for this is to hold off on the change in this ticket. Keep this proposed patch also on standby in a feature branch. If the change in this ticket is accepted, any attempt to continue the UtfNext reform effort is blocked. Can we agree on that? If so, I think the task will go easiest from 8.5 forward. jan.nijtmans added on 2020-04-29 13:00:19:
> Combine that expectation with the documentation: > "Tcl_UtfAtIndex ... is equivalent to calling Tcl_UtfNext index times." Just wondering .... did it ever occur to you that this documentation has been wrong always? I mean, when this documentation was written, no-one knew about overlong byte sequences or surrogates. It has been correct then, since Tcl_UtfNext() was always built upon Tcl_UtfToUniChar(). Now Tcl_UtfNext has a new implementation. If the documentation had been: "Tcl_UtfAtIndex ... is equivalent to calling Tcl_UtfToUniChar() index times.", that would be more logical, although not 100% correct either. | ||||
