Tcl Source Code

Since -strict can now be combined with -failindex, the discussion lead to a useful implementation, this ticket can now be closed.

There are a lot of comments regarding -nocomplain, they don't belong in this ticket, so are irrelevant here. Also discussion which TIP preceded which other TIP doesn't make sense here.

Only one remark below, I want to point out:

> Why not choose either usint "--" or making "encoding" mandatory, and proceed that way?

Now closing!

Thank you, Harald, for pointing that out. I didn't realize that TIP 601 preceded TIP 346, although I also wasn't sure based on prior discussions. The creation date of TIP 346 is 02-Feb-2009, and there is no creation date, no vote date, and no implementation date on TIP 601. Did the implementation of TIP 601 also precede the implementation of TIP 346? Why did TIP 346 not reference TIP 601 and make its relationship to that TIP clear? Why did TIP 346 advertise itself as "Error on Failed String Encodings" when TIP 601 already proposed to error on failed string encodings? Why does TIP 346 propose to "raise an explicit error when non-mappable characters are met", when TIP 601 already specified behaviour for invalid characters/bytes? Why does TIP 346 specify optional behaviour for "\xC0\x80" when TIP 601 already specified behaviour for that character sequence?

No need to respond to all these questions. If TIP 601 really preceded TIP 346, that is yet another argument in favor of withdrawing TIP 346. The two TIPs together are incoherent.

Dear Nathan, thank you for the contribution. May I add something probably unrelated: The text of TIP346 is more recent that TIP 601 and TIP607, despite of the lower number. This may change the way or agumenting "what came first". That is also why TIP346 invalidated big parts (like the rationale) of TIP607 and 601 - in came later.

Thanks again, Harald

"... the constraint was already there, as side-effect of TIP #601."

But that was not the intent of TIP 601. It was not discussed in the TIP, and was only an inadvertent side effect. Tcl shouldn't be a hack job.

TIP 601 itself is a side effect of the mistake made in TIP 346: specifying an option ("-strict") that not only determines whether an error will be thrown, but also determines what constitutes an error. Because the control of control of two different functionalities was lumped together, TIP 601 came along to provide an option to separate them, Now we have the unfortunate situation that if one wants errors to be raised on encoding problems, but doesn't want the additional behaviour specified in TIP 346, there's no combination of options for that. Instead, what Tcl has is two options, "-strict" and "-nocomplain", that work together in a confusing way to provide functionality no one wants (or understands).

There was never a TIP proposed to specify that encoding names should not start with a "-", and therefore was never a vote on that change. Constraining the set of values in situation-specific ways is the type of booby trap that bites programmers years down the road after a language has evolved in unexpected ways. This is one example of how recent TIPs have not been properly vetted.

As Harald said, TIP 346 combined disparate functionality into one option. This is bad. TIPs that have so far only affected unreleased version of Tcl should not be sacred cows. Better sometimes to go back to the drawing board before the damage takes root.

The proposed fix, "Allow "-strict" immediately before or after "-failindex var"", has some disadvantages:

It requires a new constraint on encoding names that wasn't there before.

It requires prohibiting "-nocomplain" in when "-failindex" is given. Otherwise, the argument parser breaks. This seems stem from a desire to make minimal changes to the existing argument parser, which is already not so complex.

By very-specifically ordering the allowed options, it closes the door on future evolution of [encoding] options, which I believe will be needed.

Reworking the parser to accept options in any order, as I have done in [6e15578c88cf5750], is not so hard. Why not choose either usint "--" or making "encoding" mandatory, and proceed that way?

TIP 346 can be paraphrased as, "raise an error when there is an encoding
problem, but also redefine what constitutes an encoding problem".  That's just
not very workable.  Would be better to withdraw it and go back to a clean
drawing board.

TIP 601 is basically a "fix" for TIP 346: It proposes the functionality that TIP 346 should have proposed, but a new name had to be chosen because TIP 601 already occupied "-strict".

    encoding convertfrom ?-nocomplain? ?encoding? data
    encoding convertto ?-nocomplain? ?encoding? data

Dear Nathan,

about the ambiguity, you are right, and this is not beautiful. Possible solutions (mandatory encoding, end of options option "--", no encoding with same name as an option) may be discussed further.

Other commands (regexp, switch) also have this ambiguity and choose the "end of options option "--"). I personally think, this would be the best solution.

Thank you for your constant work, Harald

Another point: Jan argues that "-failindex" implies "-strict", but TIP 346 purports to raise an error when there is an encoding problem. "-failindex" does not raise an error, so it implies "not -strict"! "-failindex" implies both X and the opposite of X because TIP 346 was improperly formulated and doesn't make sense. That TIP should should be withdrawn. It's causing far too much trouble.

In the second case, is the first occurrence of "-strict" an option or an encoding name? If it is known, for example that an encoding name never starts with "-", the ambiguity can be resolved, but that's not the case, and therefore its unresolvable.

In [6e15578c88cf5750], on the tcl-encodingdefaultstrict branch, I've modified [encoding ...] to make the "encoding" argument mandatory so it can handle any number of options in any order. On this branch, "strict" doesn't mean anything more than "raise an error on encoding failures". I think it should mean exactly that, and nothing more.

Well, as long as there are no encoding names equal to an option name or its abreviation, it can be sorted out (data is "-strict"):

• encoding convertfrom -nocomplain -strict
• encoding convertfrom -nocomplain -strict -strict
• encoding convertfrom -nocomplain utf8 -strict
• encoding convertfrom -failindex abc -strict
• encoding convertfrom -failindex abc utf8 -strict
• encoding convertfrom -failindex abc -strict -strict
• encoding convertfrom -failindex abc -strict utf8 -strict

So, case 2 and 3 are ambiguous on the positioning, requiring to check, that the encoding name is "-strict". This is not beautiful but doable.

Am I missing something? Harald

Yes, "-nocomplain" was implemented as an option, but when it is provided, no other option can be provided. Allowing multiple options would make the ?encoding? argument ambiguous.

Dear Nathan, please allow me to add my personal opioion:

Well, I have implemented the "-nocomplain var" in the encoding command. I don't see any issues to add additional options.

My main concern was the byte compiler, as those commands are "partly" byte-compiled. But I tried it using special test cases for this and it worked.

Take care, Harald

Normally one wouldn't make an unnecessary argument a syntax error. It would just be a no-op. The reason that Jan is pushing to have one option imply another is that there is only room for [encoding convert...] to take one option before it becomes ambiguous. I think trying to keep the option count at only one is a lost cause, and [encoding convert] should make the "encoding" option mandatory. That would free up syntactic space and allow for more than one option.

Jan wrote, "'-strict' is just a flag, every encoding can do with it whatever it likes (so-to-say)."

In that case, what does "-strict" mean?

> apnadkarni added on 2023-01-18 10:50:14: > > For "encoding convertfrom", using -failindex without -strict makes no sense. > Not the case. As I wrote on the mailing list,

Indeed, sorry. I meant "encoding convertto" here. But .... even then my statement was not correct in general. "-strict" is just a flag, every encoding can do with it whatever it likes (so-to-say).

There is no need to provide an option to configure the encoder/decoder to handle surrogate characters in some special way. The Unicode specification is clear enough on their use, and where they may and may not occur. There has never been an example provided in any TIP or message of when "-strict" in its current incarnation ("disallow surrogate characters in utf encoding forms") might be needed, and there never will be. Surrogate characters are already disallowed in those encoding forms by the Unicode specification. The encoder/decoder can just follow the specification on this count.

"-strict" is a useless option, and "-nocomplain" was introduced to provide the functionality that should have been provided by "-strict". "-nocomplain" is the wrong name for the option that controls raising an error on an encoding error, because there are too many ways not to complain. There should be one option that specifically says "raise an error", and other options that configures the encoder not to complain in different useful ways.

@pouryorick wrote:
> "-strict" should not be a Unicode or utf-specific setting
I was specifally refering to utf-8, immediately before the "-should" you refered to. Nothing is mixed up if you read the full sentence and consider the context of the conversation.

@Harald. Thank you for agreeing on the approach.

Jan wrote, "...should only check for invalid byte sequences, nothing more. Adding "-strict", then it should check for surrogates too."

"-strict" should not be a Unicode or utf-specific setting. It should be general option for handling encodings in general. Tcl is getting things all mixed up.

Jan, thank you for your text:

I see a way out here: without "-strict" the "encoding convertto -failindex pos utf-8" should only check for invalid byte sequences, nothing more. Adding "-strict", then it should check for surrogates too. Then the examples in the TIP #607 will work again as described in the TIP, and -strict will work as expected as well. 

I totally agree, thank you!

Harald

> For "encoding convertfrom", using -failindex without -strict makes no sense.

Not the case. As I wrote on the mailing list,

% encoding convertfrom -failindex i utf-32 \x41\x00\x00\x00\x7f\xff\xff\x7f
A
% set i
4

As expected. Thus cannot have -failindex imply -strict. In particular, when you want the default semantics (allow invalid bytes, surrogates etc.) but not "out of range" code points, and want the failing index, not an exception.

Whether this is useful or not is a different question but the way semantics are currently defined, I don't think -failindex can imply -strict.

/Ashok

Harald wrote: > I have no way out here. But now defining, that (the IMHO toxic "-strict") is implied in "-failindex" makes everything worse and not better

I see a way out here: without "-strict" the "encoding convertto -failindex pos utf-8" should only check for invalid byte sequences, nothing more. Adding "-strict", then it should check for surrogates too. Then the examples in the TIP #607 will work again as described in the TIP, and -strict will work as expected as well.

It's just a matter of implementation .....

For "encoding convertfrom", using -failindex without -strict makes no sense.

I personally would prefer to remove -strict and keep -.nocomplain. "-strict" is only for UTF-8 (switches to alternate version of the encoding), "-nocomplain" is for all encodings.

Thank you, Harald

This sure is confusing. On the trunk-encodingdefaultstrict branch, I removed at the C all flags and code segments related to -nocomplain, and the entire test suite passes there with only minor adjustments regarding no -nocomplain. I would suggest that unless someone can add a test to that branch showing a need for -nocomplain, that it is really nothing more than the nonstrict case.

My definition of "strict" on that branch is, "raises an error when there is an encoding error", and I believe that is what the actual function of strict has evolved to be. Prove me wrong with a failing test added to trunk-encodingdefaultstrict.

Dear Jan, about the examples:

"TIP 346" came after TIP 601 and 607, making the examples of TIP607 invalid.

TIP601 already solved the point, that "enocding convertfrom utf-8" does add invalides bytes as verbatim.

Then, TIP 346 came and changed this again and mixed it all up. TIP346 defined a new (so far unknown) encoding "utf8 without -strict" which is in the result in big parts identical to "encoding convertfrom -nocomplain utf8".

The difference is:

• the new encoding "utf8 without -strict" is only with utf-8.
• "-nocomplain" covers all encodings.

So, in consequence, the following are equivalent, expect the codepoint "\xC0\x80":

• encoding converfrom -nocomplain -strict utf8 $data
• encoding convertfrom utf8 $data

This is not the case for other encodings. where -strict just has no effect:

• encoding converfrom -nocomplain utf16 $data
• encoding convertfrom utf16 $data

The first will include any invalid byte verbatim. The 2nd will throw an error on invalid bytes.

I have no way out here. But now defining, that (the IMHO toxic "-strict") is implied in "-failindex" makes everything worse and not better.

Sorry, this is my personal opinion, not important.

Take care, Harald

Dear Jan, thanks for the comment.

For me, the "-strict" option is an encoding variant. So, it is together with the encoding itself.

The only "-strict" relevant TIP on my radar is TIP 346 The TIP says, that:

• for utf-8 encoding, the byte sequence "\xC0\x80" is accepted without "-strict" and not accepted with it.
• invalid byte sequence handling in UTF-8. Replace any not encodable and incomplete byte by the the byte itself

So, the TIP (without -strict) defines a UTF-8 encoding variant, where not encodable bytes are replaced by the data itself. IMHO, this is just a broken by design utf-8 variant, but ok, the TIP is as it is.

The 2nd bullet of the TIP sould be handled by the error reporting mode, as with all other multi-byte encodings.

Unfortunately, utf-8 is different. Here we have the situation, that we have two different encodings, one with an interesting replacement method of invalid bytes, one without.

I hope, this interesting special handling will not go to other multi-byte encodings like shift-jis, big5, UTF-16....

Thank you again, Harald

I'm OK with changing the implementation, making -strict independant from -nocomplain/-failindex. It's just that -failindex doesn't make much sense without -strict, so the example in TIP #607 (below) won't work without -strict.

> Example for 'Incomplete multi-byte sequence'
>  encoding convertfrom -failindex Pos utf-8 A\xC4
> The byte 'xC4' announces a multi byte sequence. Then, nothing follows.
>
> The command will return 'A' with value 1 in variable 'Pos'.

Of course, we can post-edit the TIP, adding "-strict" to this example. Otherwise:

$ tclsh8.7
% encoding convertfrom -failindex Pos utf-8 A\xC4
AÄ
% set Pos
-1

This is different from what the TIP specifies.

Sorry, Jan, no. "Encoding error reporting mode" (-nocomplain, -failindex) are different to "-strict". "-strict" is an encoding variant switch. There are two encodings with the same -encoding name, one with "-strict", one without "-strict". This has nothing to do with reporting mode.

Sorry, Harald

Proposed fix [7cd1856ca4a693e5|here]

The -failindex option always was intended to imply -strict (otherwise invalid byte-sequences would never be reported), but the documentation and the error-message was not so clear about it. Therefore - also - updated the documentation.

Hope it's clearer now.