Tcl Source Code

To restate, "binary" data is nothing more than Unicode text containing only characters defined in the ASCII and Latin-1 code charts.

There is no "binary" encoding. At the script level it was always just iso8859-1, and always will be. This is not an implementation detail leaked to the script level, just the fact that at the script level there is no "binary" mode, and that "binary" was always just an ill-advised synonym for iso8859-1. At the script level there is only Unicode text, and if one wishes to manipulate "binary" data, one must in principle translate each incoming byte into the corresponding Unicode code point (even if internal optimizations like bytearray elide this step). If each character in the text has a code point of 255 or less, it is amenable to interpretation as bytewise numerical values (binary).

In Tcl 8.6, configuring the translation of a channel as binary and then checking configuration yields a translation of "lf", not "binary", and this has always worked just fine. The same should be true for encoding: Programmers should understand that "binary" means iso8859-1 (assuming binary isn't removed as an encoding).

If anyone is writing code that is relying on whether the encoding for a channel is "binary", their code is probably buggy, and it probably would be less buggy if Tcl had never provided "binary" as an encoding option in the first place. Binary data can come in through just about any encoding. It doesn't make much sense to condition code on whether the channel encoding is "binary".

There is sufficient introspection available already. There is no need for a [chan isbinary], and it would be a bad idea to provide it. It generally makes more sense not to introspect the encoding, and instead to just set the encoding to what it is expected to be. I would like to see the use case for introspecting encoding.

The best course of action would be to eliminate "binary" as an encoding, since it never really was one, and leave "binary" only as an argument to -translation.

How about adding a [41900f4c4fcae34a|chan isbinary] command, which - for the first time - gives the right answer?

How about adding a [chan isbinary|41900f4c4fcae34a] command, which - for the first time - gives the right answer?

So let's go back to the 'introspection' part. I already brought forward that this ticket actually handles the possibility to introspect whether a channel is binary or not. Checking for "chan configure -encoding binary" is not it: Still, the channel could use -translation or -eofchar, which means that the channel is not binary.

How about adding a [chan isbinary|41900f4c4fcae34a] command, which - for the first time - gives the right answer?

typo correction: "if you want to, you can treat it as a list"

For those struggling to understand this concept, consider a similar situation with lists: Every list is a string, i.e. Unicode text. Internally, for performance sake, a structured representation of a list may be attached to the Tcl_Obj for that value, but that's just an optimization. A list is Unicode text, and if you want to, you cant treat it as a list. The same is true with binary data in Tcl: Binary data is Unicode text, and if you want to, you can treat the characters in that text as values representing something other than characters. That doesn't change the fact that the value is Unicode text. In Tcl there is no bifurcation between text and binary values. It's all text. It's just that some text can be interpreted as binary if that's what suits the application.

Every value is a string. It's that simple. bytearray is strictly an internal optimization. In Tcl, binary data is simply Unicode text. Don't let the internal optimizations fool you.

> Also I don't understand how it can be comparable with iso8859-1 (where, strictly seen the standard, it definitely misses chars 0x00-0x1F and 0x7F-0x9F, so I guess it need to fail with profile strict or replace with profile replace, and even if it doesn't right now, it may be changed later).

Allow me to correct this. The ranges 0x00-0x1F and 0x7F-0x9F in iso8859-1 are control codes, definitely a valid range. Therefore "iso8859-1" is one of the few encodings with a unique property: Reading a file with encoding set to "iso8859-1", there are no invalid bytes, so it doesn't matter which -profile is set: No encoding errors are thrown during file reading, whether -profile is "strict", "tcl8" or "replace", or whatever future value. That should never change.

In Tcl 8.x, "binary" and "iso8859-1" are not 100% aliases: "binary" has additional optimizations when using byte arrays. But at script level, those two behave the same. "binary" has some shortcuts which makes things faster, which "iso8859-1" doesn't have. In Tcl 9.0, "iso8859-1" inherits the same optimizations, which means that "binary" now is a real alias to "iso8859-1".

Hope this helps

Also, if this new behaviour remains, "won't fix" is the wrong resolution, because it means "this is a bug we're keeping". "invalid" is the right resolution because it means, "this is not a bug".

It's important that people understand what "binary" really is in this text-oriented language we call Tcl. Sugarcoating names with the term "binary" leads to more subtle confusion among Tcl programmers. The documentation should explain that in Tcl "binary" data *is* text, and that working with binary data means constraining oneself to using only the first 256 Unicode characters. If anything, "binary" should be removed as an encoding (but should remain for -translation). That would have the benefit that people would no longer make the mistake of using "-encoding binary" when they meant "-translation binary", which is a pretty common silent-corruption-producting footgun.

Programmers should take the time to read the docs and understand why iso-8859-1 is used to read and write text that can be interpreted as "binary" data.

I see the discussion going into two directions. One is about the introspection, which is this ticket. "Invalid" is the wrong resolution here (sorry, Nathan). We need to decide whether we decide to roll-back the behavior to be the same as in 8.6 or keep it the way it is now.

I'll discuss this with the other TCT members, and whether "Fixed" or "Won't Fix" is the best here. My personal preference is "Won't fix", but I don't have the power alone to make that decision. In case of "Won't fix" the documentation needs to be improved, that's - at least - my conclusion from this ticket. In case of "Fixed" it shouldn't be a just reversion of Nathan's work: generally, I like it, because it gives "iso8859-1" the same optimization as "binary" had!

Stay tuned. It make take a while, not high prio.

Keeping this ticket open, until a decision has been made. At least, documentation should be improved.

If the user sets an encoding of binary (whether using -encoding or -translation), then for fconfigure to return something different is unexpected and surprising:

It's undocumented in the man pages.

It's not at all obvious to a user that encoding binary and encoding iso8859-1 are the same thing.  As a user, setting encoding binary means that I want byte values to be transfered unchanged.  Setting character set encoding iso8859-1 means that I want character values to be transformed between that encoding and Tcl's internal representation.  If they happen to be implemented in the same way then I'd expect Tcl to manage that internally, rather than exposing it to the user.

Changing the value which fconfigure returns causes needless problems to scripts or GUIs which perform introspection of channel settings.

As Jan said, this is correct behaviour. "binary" is an alias for "iso8859-1".

Actually, since "-encoding binary" is an alias for "-encoding iso8859-1", this new behavior is not strange at all. You most likely want to use "-translation binary". That's also the same as "-encoding iso8859-1", but also sets -translation to lf and -eofchar to {}. See: https://core.tcl-lang.org/tcl/info/f375dcda79?ln=255-265

This change - most likely - comes from [b66d50b4d45f6f36|this] commit. Ticket: [fa3d9fd818fa0072]

Assigning to Nathan Coulter.

The second issue you mention, the system encoding change on Windows Server 2019 is totally unrelated. Better file another ticket for that.