Tcl Source Code

The ext file system has no concept of "encoding": It names a file exactly the sequence of octets it is handed. In the following script, iso8859-1 is used to create a file, and then that same sequence of octets is obtained using glob. Then an FsPath object created under a different system encoding is created and cached. Later on that FsPath object is used under the original system encoding when it shouldn't be.

set name  登鸛鵲樓

encoding system iso8859-1

# create an iso8859-1-compatible sequence of octets to name the file by.
# They happen to be a string encoded in utf-8, but that's irrelevant.
set utf8 [encoding convertto utf-8 $name]

set chan [open $utf8 w]
puts $chan hello
close $chan

#
lassign [glob -directory [pwd] *$utf8*] fname



encoding system utf-8

# The sequence of octets is now itself converted to utf-8. Such a file does not
# exist, and a FsPath object has been cached.  So far so good.
puts [list utf-8 [file exists $fname]] ;#-> 0
encoding system iso8859-1

# Here the answer here should be 1, but the FsPath object created under a
# previous system encoding is used when it shouldn't be.
puts [list iso8859-1 [file exists $fname]]  ;# -> 0

# Discard the cached FsPath object, causing a new FsPath to be created using
# the (currently correct) system encoding.
puts [list iso8859-1 {string range} [file exists [string range $fname 0 end]]]  ;# -> 1

Well, I put back the test, so is this one fixed now?

> Or ... just change the expected test result for UNIX ...

Of course, I meant Windows here ....

Or ... just change the expected test result for UNIX ...

The bug can not be reproduced on NTFS filesystems in the first place. That's why with an NTFS filesystem the first result of filesystemEncoding-1.0 is 1 rather than 0.

Maybe I'm wrong, but because NTFS has a native encoding whereas common *NIX filsystems do not, this test can not be adapted for NTFS. Therefore it could be constrained to *NIX systems rather than removed.

I just removed the filesystemEncoding test-case now, knowing that this bug is still not solved for Windows. That means, it's not critical any more now.

On UNIX, it appears to work, but all travis Windows builds are failing:

==== filesystemEncoding-1.0 issue bcd100410465 FAILED
==== Contents of test case:
	set dir [tcltests::tempdir]
	set saved [encoding system]
	encoding system iso8859-1
	set fname1a $dir/$fname1
	set utf8name [encoding convertto utf-8 $fname1a]
	makeFile {} $utf8name
	set globbed [lindex [glob -directory $dir *] 0]
	encoding system utf-8
	lappend res [file exists $globbed]
	encoding system iso8859-1 
	lappend res [file exists $globbed]
	return $res

---- Result was:
1 1
---- Result should have been (exact matching):
0 1
==== filesystemEncoding-1.0 FAILED

This way Donald Porter cannot make his RC, so please have a look at this!

Fixed in [440efbd556ce7041] for core-8-6-branch. In contrast with the proposed fix for the core-8-branch, this is a one-line fix. It should probably also be applied to core-8-branch, and additional work done there to refine the record keeping when system encoding changes.

Ok, reproduced.

Unable to reproduce:
  8.6.9 on MX Linux 19
  8.6.5 on Ubuntu 16.04

Unable to reproduce with tcl 8.6.9 on Linux.

The initially-provided example behaves as described in core-8-6-branch, but in core-8-branch only behaves as described prior to commit [165586b224f85f10], " TIP #430 implementation", 2018-09-12. This is because even when built with --disable-zipfs, in core-8-branch there are calls to ZipFSPathInFilesystemProc from Tcl_FSGetFileSystemForPath, and ZipFSPathInFilesystemProc uses Tcl_GetString(pathPtr) instead of working with the fsPathType internal representation.

Hm ...  Added the Tcl_FreeEncoding() now, it doesn't seem to work ...

Anyway, I'll leave the branch to you, in case you want to work on it further. Please, next time, don't merge it to core-8-branch as long as you are not reasonable confident that all test-cases work.

Thanks!

The failue of encoding-2.2 was fixed by adding a corresponding Tcl_FreeEncoding for the newly-added Tcl_GetEncoding.

There's more work to be done on this issue though. There might be undesired interactions between FsPath.nativePathPtr and FsPath.filesystemEpoch. These are older issues, not something introduced by the current changes.

Since Travis already fails for 2 days, I put the work on this ticket in a separate [bug-bcd1004104] branch. Please have a look: Something still wrong with the code, or should the test-case be revised?

Starting with [0f2870649c804dd8]:

encoding.test

==== encoding-2.2 Tcl_FreeEncoding: refcount != 0 FAILED
==== Contents of test case:

encoding system shiftjis		;# incr ref count
    encoding dirs [list [pwd]]
    set x [encoding convertto shiftjis \u4e4e]	;# old one found
    encoding system iso8859-1
    llength shiftjis		;# Shimmer away any cache of Tcl_Encoding
    lappend x [catch {encoding convertto shiftjis \u4e4e} msg] $msg


---- Result was:
ŒÁ 0 ŒÁ
---- Result should have been (exact matching):
ŒÁ 1 {unknown encoding "shiftjis"}
==== encoding-2.2 FAILED