Tcl Source Code

View Ticket
Login
Ticket UUID: 75ecc0e6cb9899bff0fa9e22a10a76b0c244c0a1
Title: TclInitSubsystems issues: the sporadic lack of encodings
Type: Bug Created on: 2018-05-30 18:53:55
Submitter: sebres Assigned to: apnadkarni
Subsystem: 38. Init - Library - Autoload Severity: Severe
Priority: 5 Medium Last modified: 2026-11-07 11:08:42
Status: Closed Closed by: apnadkarni
Resolution: Fixed Closed on: 2026-11-07 11:08:42
Version: 8.6.18, 9.1.0
Description:

Set the system encoding via environment/locale failed sometimes dependent of the system constellation (distribution and/or some env-settings).

Who saw already the test "unixInit-3.2" failed? :)

This cumulates from two actual issues:

1. setlocale (called from TclpInitPlatform) may overwrite environment "LANG" (to "C"), so the try searching of current encoding later in Tcl_GetEncodingNameFromEnvironment will failback to "C" (so the call of "setlocale" takes place too early in some cases, resp. get initial environment via getenv("LANG") too late);

2. second thing which can happen, if 1st issue with "setlocale" gets repaired, would be the fact, that Tcl_GetEncodingSearchPath (called from TclpSetInitialEncodings/Tcl_SetSystemEncoding) may return "" before still any interpreter instance created (and it is so if TclpInitPlatform called).
In this case setting of something other as already available system-encodings going to "C" (iso8859-1), because the load of encoding simply fails.
The reason for this trouble is the wrong value returned by callback TclpInitLibraryPath.
But later (after first interp going created) the init.tcl (as well as the counterpart script hard-coded in Tcl_Init) will indirectly call Tcl_SetEncodingSearchPath to the proper value (after tcl_library correct resolved and TclGetProcessGlobalValue(&libraryPath) returns correct value) and then the load of the same encoding works.

Following code can illustrate this behavior:


$ echo 'puts ==[encoding system]; encoding system euc-jp; puts ==[encoding system]' | LANG=japanese ./tclsh

==iso8859-1 ==euc-jp

But setting of proper tcl_library env-variable resolves it:

$ echo 'puts ==[encoding system]; encoding system euc-jp; puts ==[encoding system]' | LANG=japanese TCL_LIBRARY=../library ./tclsh

==euc-jp ==euc-jp

I saw it already too often, but following cases are affected especially:

- usage of portable-like (not installed) tcl's instance or tcl-ed applications (where TCL_LIBRARY environment variable does not set by start);

- call of other tcl-instance from tcl-app via [open "| some-tcl-app"] inside the tcl-interpterer with changed environment (LC_... resp. LANG are changed);

- call of tcl from development environment (also not istalled), like this:


$ cd ~projects/tcl8.6/unix
$ LD_LIBRARY_PATH=. ./tclsh ../tests/all.tcl -file '*unixInit*' -match '*unixInit-3.2*'

... ==== unixInit-3.2 TclpSetInitialEncodings FAILED ... ---- Result was: 1 ---- Result should have been (exact matching): 0 ==== unixInit-3.2 FAILED ...

I've localized the "wrong" places and will try to fix it soon.
WiP.

User Comments:
sebres added on 2018-05-31 13:50:42:

The branch "fix-initsubsystem-75ecc0e6cb9899bf" provides a fix for this ticket (for 8.6th).

All tests passed on my reference systems (*nix as well as windows).


sebres added on 2018-05-31 14:15:37:

BTW There is a new C-routine "TclpExtendWithRelativeLibraryPaths" now, which could be used later as replacement for some hard-coded script-parts of the tclInit (which belong to parents relative the path of "nameofexecutable").

I mean this scripts:

  file join $parentDir lib tcl[info tclversion]
  file join $grandParentDir lib tcl[info tclversion]
  file join $parentDir library 
  file join $grandParentDir library
  file join $grandParentDir tcl[info patchlevel] library
  file join [file dirname $grandParentDir] tcl[info patchlevel] library
Indirectly TclpExtendWithRelativeLibraryPaths returns all valid (existing) paths of above-mentioned combinations in the same sequence.


jan.nijtmans added on 2022-11-22 19:44:35:

This came up again. See: [fccb9f322]


serhiy.storchaka added on 2026-10-16 20:52:31:

The system-encoding part was fixed in 9.0 by [17c53b7edd]; 8.6 still shows it for an uninstalled or relocated tclsh. On main, TIP 732 dropped the [encoding dirs] addition from init.tcl, so a --disable-zipfs tclsh with a relocated library has no encoding directories at all (9.1.0). Proposed fix in branch init-encoding-dirs, test encoding-26.1.


dgp added on 2026-10-16 22:41:00:
I thought one of the intents of TIP 732 was to simplify and reduce
the search paths.  Before growing them again, someone involved
in TIP 732 should comment.

serhiy.storchaka added on 2026-10-17 07:30:00:

It does not grow the search path, it restores what 9.0 does: init.tcl adds [file join $::tcl_library encoding] to [encoding dirs] if it is not there. TIP 732 ([83e88416ac]) moved that part of init.tcl to InitAutoPathObjCmd(), whose comment still says "The function also adds the encoding subdirectory of tcl_library to the encodings search path if not already present", but the code was not moved. So in 9.1 a tclsh without zipfs whose library is not at the compiled-in place has no encoding directories at all. @apnadkarni, was dropping it intended?


apnadkarni added on 2026-10-17 16:53:24:
I have to retrace but will need some time before I can look at it. I vaguely recall it was supposed to be done elsewhere (which would mean the comment was not fixed) but I could be wrong.

apnadkarni added on 2026-11-06 15:40:08:

Proposed fix in branch apn-bug-75ecc0e6cb.

Implementation differs from the init-encoding-dirs branch. Encoding directory should be added irrespective of whether InitAutoPath is called by a (custom) init.tcl or not so it's called from LocatePreInitScript which sets tcl_library. Implementation is shared with the zipfs enabled case.

Question, in case anyone feels one way or another -

  • new encoding dirs are added to the front, not the back (the new encoding-26.1 test has been modified accordingly). My thinking is directories added later by any mechanism should be searched to earlier ones. Easy enough to change.

  • After an application sets the encoding directories through encoding dirs new interpreters will modify the search path established through that command. I don't know if that is desirable or not but that is the 8.6 behaviour and preserved.


apnadkarni added on 2026-11-07 11:08:42:
Merged into 9.1 in [1bc3a58eac].

Not applicable to 9.0.

Please re-open if you still see an issue.