Tcl Source Code

Even if two channels have he same encoding [fcopy $chan1 $chan2 -size] might not be the same as [fcopy $chan1 $chan2 -size]. The makes [fcopy -size] difficult to use correctly.

In the following script, copying with a size of 1 between to channels, both using utf-8 encoding, does not copy a full character. Although the behaviour is currently documented to be what it is, it's still bad, and should be fixed:

set chan [file tempfile]
chan configure $chan -encoding utf-8 -profile strict
puts -nonewline $chan á
flush $chan
chan configure $chan -profile tcl8 -translation lf -eofchar {}

seek $chan 0
set chandata [read $chan]

set chan2 [file tempfile]
chan configure $chan2 -profile tcl8 -encoding utf-8

seek $chan 0
fcopy $chan $chan2 -size 1
flush $chan2

seek $chan2 0
set chan2data [read $chan2]
close $chan2

puts [list fcopy equals original? [expr {$chan2data eq $chandata}]]


set chan3 [file tempfile]
chan configure $chan2 -profile tcl8 -encoding utf-8
seek $chan 0
puts -nonewline $chan3 [read $chan 1]

close $chan

seek $chan3 0
set chan3data [read $chan3]

puts [list read/write equals original? [expr {$chan3data eq $chandata}]]

Fixed in [39a45eb8ff]. I think the proper fix is to stop using NULL to represent the binary encoding in ChannelState.encoding. iso8859-1 is the encoding for handling binary data, and using it brings the implementation more in line with the conceptual design and also reduces the number of different code paths in IO operations. One new aspect of this fix is that setting the encoding to "binary" also sets the profile to strict, which is needed to make some tests pass, but should also help developers avoid silent data corruption for both input and output. For example, in test io-52.10 the previous result was nothing sensible. Now the result is an error that lets the programmer know strange things are afoot in their code. This commit also paves the way for further simplification of the code in tclIO.c.

There's a setting "Beta: Use Unicode UTF-8 for worldwide language support" in Windows. So, if we wait long enough, UTF-8 will eventually become the default encoding in Windows. The problem will automatically disappear. So we could simply ignore it. Not high prio to fix.

> I don't know why the tests fail only with the msvc build though

It's because the gcc environment uses the bash shell, which reports to its clients that the system encoding is utf-8. The Visual Studio build environment uses the normal CMD shell, which has the normal system encoding on Windows. So, yes, this explains the difference.

So, this leads to 3 different solutions.
* Implementation change: Whenever ChannelState.encoding is NULL, make sure that Tcl_UtfToExternal() is explicitly given the iso8859-1 encoding.
* don't set ChannelState.encoding to NULL when we want iso8859-1
* change Tcl_UtfToExternal() so NULL means iso8859-1

I think the last one is not desirable (would need a TIP, but utf-8 would be more reasonable), but the other two should be possible.

I believe the test failures on Windows are related to the fact that Tcl_UtfToExternal() sets the encoding to the system encoding when encoding is NULL. I don't know why the tests fail only with the msvc build though. The design problem is that in ChannelState.encoding, NULL represents the binary encoding, whereas in <Tcl_UtfToExternal(), NULL represents the system encoding. There are probably several ways to fix that, with some of those ways being more hacky than others.

Further info: In the "bug-fa3d9fd818fa0072" branch, the same failure can be seen, so it's sure that this is the trigger.

Unfortunately, with this fix two testcases started failing on Windows, see:

https://github.com/tcltk/tcl/actions/runs/4605104184/jobs/8152506769

For now, I disabled those 2 test-cases, but please take a look.

Fixed in [51d813943bcaf835].