Tcl Source Code

View Ticket
Login
Ticket UUID: 219289
Title: Tcl doesn't read utf-16 files properly
Type: Bug Created on: 2000-10-26 05:10:04
Submitter: nobody Assigned to: hobbs
Subsystem: 10. Objects Severity:
Priority: 3 Low Last modified: 2006-06-21 03:36:47
Status: Open Closed by:
Resolution: None Closed on:
Version: obsolete: 8.1
Description:
OriginalBugID: 2129 Bug
Version: 8.1
SubmitDate: '1999-05-26'
LastModified: '1999-10-21'
Severity: MED
Status: UnAssn
Submitter: pat
ChangedBy: hobbs
RelatedBugIDs: 2128
OS: Other
Machine: NA
FixedDate: '2000-10-25'
ClosedDate: '2000-10-25'


Name:
Victor Wagner

ReproducibleScript:
tclsh8.1 >testfile <<!
    fconfigure stdout -encoding unicode
    puts "\uFEFFSome stuff
    !
    dd if=testfile of=testin conv=swab
    tclsh8.1 <<!
    set f [open testin]
    fconfigure \$f -encoding unicode
    fconfigure stdout -encoding [encoding system]
    puts [read \$f]
    !

ObservedBehavior:
When reading utf-16 file Tcl doesn't recongize byte order
    automatically.
    According to utf-16 specification, each unicode text file should
    contain
    "\uFEFF" as first character, which allows to determine byte order
    automatically.
    But Tcl doesn't pay attention to this characher. So, if proper UTF-16
    file is constructed on MSB-first machine and then read by Tcl on
    LSB-first
    machine of vice versa, it doesn't read properly. On contrary, MS-Word 97
    read
    such files without problems (at least ones with less than 256 chars)

DesiredBehavior:
When reading unicode files, tcl should pay attention to firsh characher
    and if it is FFFE, switch to opposite byte order.
User Comments:
dkf added on 2001-10-07 18:50:14:
Logged In: YES 
user_id=79902

This is not straight-forward to fix, since the BOM should only be read at the start of the file (this gets 
awkward when working with files that are in several encodings.) Perhaps it needs some special kind of 
encoding or a state-flag in the channel structure? (Of course, the encoding system does not know about 
channels at all. Maybe the decision needs to be made there?)

andreas_kupries added on 2001-10-09 22:16:26:
Logged In: YES 
user_id=75003

Also note that there are channels like pipes and sockets 
which might use utf-16 but have no 'start of file' concept
per se.

IMHO this is a problem to be handled by the application and 
not the interpreter.

hobbs added on 2001-10-10 05:21:17:
Logged In: YES 
user_id=72656

The user noted MS Word as an example, and this really is a 
feature for the application, not the language.  The 
developer should test for \uFEFF or \uFFEF, but Tcl does 
need to provide a way to handle utf-16 in MSB or LSB order 
(either as two absolute encodings, or some option).

dgp added on 2006-06-21 03:36:47:
Logged In: YES 
user_id=80530


See 1165752