Tcl Source Code

View Ticket
Login
Ticket UUID: 8a13caa31dd8e3a546b8d44c235bb98cd52bd774
Title: Setting of system encoding isn't applied to stdout on certain circumstances
Type: Bug Created on: 2025-08-16 23:02:32
Submitter: sebres Assigned to: nobody
Subsystem: 20. [interp] Severity: Severe
Priority: 5 Medium Last modified: 2025-08-18 21:53:49
Status: Closed Closed by: sebres
Resolution: Fixed Closed on: 2025-08-18 21:53:49
Version: 9.0
Description:

I like it when an experimental branch brings to light problems that essentially have nothing to do with the experiment itself.

My lazy branch (commit [6a997a7f19eaa6fa]) I provided for [effa2e2346f8372a] shows following error in encoding.test now:

==== encoding-24.3 EscapeFreeProc on open channels FAILED
==== Contents of test case:

# Bug #219314 - if we don't free escape encodings correctly on channel # closure, we go boom set file [makeFile { encoding system iso2022-jp set a "乎乞也"; # 3 Japanese Kanji letters puts $a } iso2022.tcl]

set f [open "|[list [interpreter] $file]"] fconfigure $f -encoding iso2022-jp set count [gets $f line] close $f removeFile iso2022.tcl list $count $line

---- Result was: 9 \x00\x00\x00\x00\x00\x00\x00\x00\x00 ---- Result should have been (exact matching): 3 \u4E4E\u4E5E\u4E5F ==== encoding-24.3 FAILED

(this is even valid for single test too, e. g. with -file encoding.test -match encoding-24.3 -singleproc 1)

This particularly means that [encoding system iso2022-jp] doesn't change the encoding of stdout (it remains utf-8 in child process, so the parent reads a "noise").

Changing of stdout encoding directly like here "fixes" the test:

- encoding system iso2022-jp
+ fconfigure stdout -encoding iso2022-jp

Also it is not clear why before the commit [6a997a7f19eaa6fa] it works as expected.
As if moving of some inits (like Tcl_PkgProvideEx) has reached a "critical mass"... Or as something in package mechanisms touched encoding somewhere in different way and it appears now.

Since it looks like an ugly Heisenbug and it is encoding issue (again), I think it were good to document this strange behavior.

User Comments:
apnadkarni added on 2025-08-17 06:33:28:

I am actually confused by that test. When the child process runs the encoding system is2022-jp command, stdout is already open isn't it? Afaik, changing system encoding will not change encoding on existing channels so why does the test expect it will output the characters in iso2022-jp?

And to further the confusion, why would the test fail in your apn-oo-lazy-init-sbmod but not in apn-oo-lazy-init or main?

I'm speculating that the difference is the point at which the stdout channel is opened. If it was already open, the test would fail but it is not obvious looking at the diffs why the two branches would differ in that respect. May be either ObjPrintf or the Pkg* related calls in apn-oo-lazy-init-sbmod cause stdout to be opened earlier? Can't think of what else it might be without further investigation.


apnadkarni added on 2025-08-17 11:23:36:

Curiously, tclsh passes the test fine, tcltest fails. So likely some difference in the initialization sequence. I'm also wondering why the read output is nuls. Definitely worth tracking down.

tclsh:

c:\src\tcltk\wip\tcl\win>Release_AMD64_VC1939\tclsh90.exe ..\tests\all.tcl -file encoding.test -match encoding-24.3 -singleproc 1
Tests running in interp:  c:/src/tcltk/wip/tcl/win/Release_AMD64_VC1939/tclsh90.exe
...blah blah...

Tests ended at 2025-08-17 16:48:47 +0530
all.tcl:        Total   212     Passed  1       Skipped 211     Failed  0

tcltest on the other hand:

c:\src\tcltk\wip\tcl\win>Release_AMD64_VC1939\tcltest90.exe ..\tests\all.tcl -file encoding.test -match encoding-24.3 -singleproc 1
Tests running in interp:  c:/src/tcltk/wip/tcl/win/Release_AMD64_VC1939/tcltest90.exe
...blah blah...


==== encoding-24.3 EscapeFreeProc on open channels FAILED
==== Contents of test case:

    # Bug #219314 - if we don't free escape encodings correctly on channel
    # closure, we go boom
    set file [makeFile {
        encoding system iso2022-jp
        set a "乎乞也"; # 3 Japanese Kanji letters
        puts $a
    } iso2022.tcl]
    set f [open "|[list [interpreter] $file]"]
    fconfigure $f -encoding iso2022-jp
    set count [gets $f line]
    close $f
    removeFile iso2022.tcl
    list $count $line

---- Result was:
9 \x00\x00\x00\x00\x00\x00\x00\x00\x00
---- Result should have been (exact matching):
3 \u4E4E\u4E5E\u4E5F
==== encoding-24.3 FAILED

apnadkarni added on 2025-08-17 11:54:10:

And to further narrow down and eliminate the testing framework:

c:\src\tcltk\wip\tcl\win>cat z.tcl

        encoding system iso2022-jp
        set a "ä1Zä1zä1Y"; # 3 Japanese Kanji letters
        puts $a


c:\src\tcltk\wip\tcl\win>Release_AMD64_VC1939\tcltest90.exe z.tcl | od -t x1
0000000 e4 b9 8e e4 b9 9e e4 b9 9f 0d 0a
0000013

c:\src\tcltk\wip\tcl\win>Release_AMD64_VC1939\tclsh90.exe z.tcl | od -t x1
0000000 1b 24 42 38 43 38 70 4c 69 1b 28 42 0d 0a
0000016

apnadkarni added on 2025-08-17 17:18:36:
@sergey, just so you don't duplicate effort, I think I understand what's going on and it's not an issue with the new code. I'll update tomorrow.

apnadkarni added on 2025-08-18 05:47:52:

So, as suspected, it is not an encoding issue as much as initialization of std channels. The test failure is a result of stdout being already open and configured for utf-8 when the child process in encoding-24.3 calls encoding system. Thus the output from the child is encoded in utf-8, as the encoding system command has no effect on existing channels.

The difference between the main or core-9-0-branch (which do not see the test failure) and the apn-oo-lazy-init-sbmod branch is that tcltest calls Tcl_OOInitStubs as part of its initialization. That in turn does a package require on tcl::oo.

In the case of the apn-oo-lazy-init-sbmod, tcl::oo is only loaded on demand (via oo::_init) so the package loading code runs when Tcl_OOInitStubs is called and at some point loads tclIndex. The call to open this file leads to the initialization of the channel table which as a side effect creates the standard channel including stdout. As stdout already exists, its encoding is not changed by encoding system.

In the main branch, oo is already loaded as a package as part of Tcl initialization. Thus the package require does not go rummaging for tclIndex and stdout therefore remains uninitialized and is created only after the encoding system command and therefore has the encoding specified in encoding system.

As an illustration, if the child process script in encoding-24.3 is modified to add a package require platform at the top, the same failure is seen in the main and core-9-0-branch as well.

First, I think this is a bug in encoding-24.3 as it presumes stdout does not exist when a script is run and Tcl does not guarantee that is the case. Fixing the test by adding a fconfigure (as Sergey originally reported) may be one option but I am not sure if the original bug specifically was intended to the system encoding being attached to the standard channel.

To fix the issue regardless, one possibility is to change Tcl_OOInitStubs to statically load the package instead of doing a package require. I think the package require was only needed for 8.5 where oo was not built in. Doing a load, assuming it is feasible, would bypass the package implementation and leave channels unchanged. However, I don't think this is really mandated or absolutely necessary since scripts should not really assume the state of std channels at startup.

As an aside, I'm surprised the test output prints nuls (x00) as the line read. That is a separate issue that might need to be looked at (perhaps it's the tcltest output ascii-fication getting it wrong).


apnadkarni added on 2025-08-18 11:18:31:

I was wrong about one detail. It is not the package require command (or at least not the only command) that is causing the access to tclIndex resulting in channel stdout initialization. The sequence

	namespace export callback
	namespace eval tmp {namespace import ::oo::Helpers::callback}

in tclOOScript.tcl invokes the Tcl_Import C API which in turn evals auto_import which calls auto_load_index which read tclIndex.

Given it is the oo initialization itself that created stdout, it seems the only option is to fix the test, which in my mind is reasonable to do.


apnadkarni added on 2025-08-18 13:30:53:

@sebres, I am closing this ticket as moving of some of the oo init script to C on the main trunk also fixes this issue and I think this lazy init should be postponed to 9.1 anyways. See the effa2e2346 ticket for more.

Reopen if you think this is still pertinent.


sebres added on 2025-08-18 15:18:51:

Good analyze.

Given it is the oo initialization itself that created stdout ...

Yes, but exactly that is the Heisenbug what we're looking for, isn't it?

I mean why this 3 lines script shall cause that Tcl_OOInitStubs/tclOOScript.tcl/etc gets involved at all? There is nothing OO-related in that script (and a lot of others).

And even if it is caused by tcltest itself, we shall probably rewrite it lazy or on demand too.

Since tcltest is used in a lot of test cases as [interpreter], it'd also make tcltest faster.


apnadkarni added on 2025-08-18 17:10:55:

@sebres, are you suggesting keeping this bug open or that the lazy initializing should also be in 9.0, not just 9.1? Part of the reason for closing this is to track the "lazy oo init" project in a single ticket [effa2e2346].

Indeed the impact of this lazy change on that three line script was mystifying. At least to some extent, I feel this is the result of a rather convoluted initialization sequence (perhaps necessarily so) that I just documented in the wiki. If that could be streamlined and well defined (it currently depends on which of several Tcl initialization functions is called first), many of these "heisenbugs" would go away I think.


sebres added on 2025-08-18 18:56:50:

I mean that probably we shall load tcltest (or invoke Tcltest_Init or its parts) on demand (e. g. in tcltest::loadTestedCommands or somewhere similar).

Or to detach Tcl_OOInitStubs from necessity to do package require tcl::oo etc, so it'd initialize the OO-stubs immediately, but the rest of OO gets loaded on demand.

Basically, the main goal would be to make tcltest executable work more similar to tclsh, because otherwise even in test-suite we mostly test tcltest executable, which is according to new findings slightly different (at least if invoked as a sub process).


sebres added on 2025-08-18 21:53:49:

Branch experimental-load-tcltest-implicitly, particularly [6e6d953e8c2eab8b] illustrate the possibility to circumvent the issues like that and makes tcltest executable to work more similar to tclsh, so would avoid certain side effects and speed-up the interpreter creation from this exe (for cases where tcltest module is unneeded).

I also believe that we can improve it more, e. g. to make calling of `Tcl_OOInitStubs` from there also optional (lazy, on demand, implicitly, however - but only there where OO really needed).