Tcl Source Code

View Ticket
Login
Ticket UUID: 219297
Title: euc-jp encoding ignores euc-3 extension
Type: Bug Created on: 2000-10-26 05:10:19
Submitter: nobody Assigned to: hobbs
Subsystem: 10. Objects Severity:
Priority: 4 Last modified: 2002-04-20 08:46:33
Status: Open Closed by:
Resolution: None Closed on:
Version: final: 8.1.1
Description:
OriginalBugID: 2609 Bug
Version: 8.1.1
SubmitDate: '1999-08-31'
LastModified: '1999-10-21'
Severity: LOW
Status: UnAssn
Submitter: techsupp
ChangedBy: hobbs
OS: Linux-Red Hat
Machine: Other
FixedDate: '2000-10-25'
ClosedDate: '2000-10-25'


Name:
Roger E Critchlow Jr

ReproducibleScript:
scan [encoding convertfrom euc-jp "\217\260\241 3021 U4e02"] %c%c char1
    char2
    if {$char1 != 0x4e02} {
      puts "tcl euc-jp translated three byte encoding for U+4e02 into
    [format {\\x%02x \\u%04x} $char1 $char2]"
      exit 1
    } else {
      exit 0
    }

ObservedBehavior:
tcl euc-jp encoding ignores the euc-3 extension used in Japanese to
    encode 5800 kanji.

DesiredBehavior:
tcl euc-jp encoding implements the euc-jp encoding.
User Comments:
hobbs added on 2001-10-13 03:14:06:
Logged In: YES 
user_id=72656

We have to verify that this "euc-3 extension" is meant to 
be part of the standard euc-jp or not (since our euc-jp is 
auto-generated from a former standard script from 
Unicode.org).

haible added on 2002-04-20 06:04:07:
Logged In: YES 
user_id=5923

What the submitter meant was the 3-byte area
(0x8F 0x{A1..FE} 0x{A1..FE} of EUC-JP that encodes the
JISX0212
character set. You find it summarized in
http://www.ksky.ne.jp/~smile4me/charcode/japanese.htm

hobbs added on 2002-04-20 08:03:45:
Logged In: YES 
user_id=72656

The above octal is this in hex: \x8f\xb0\xa1

To reiterate the page Bruno mentions, these characters are 
encoded as 3-byte chars (0x8FA1A1-0x8FFEFE) and represent 
the supplemental jisx0212 chars.

A couple other good resources:

http://www.io.com/~kazushi/encoding/ujis.html
http://web.lfw.org/text/jp.html

Basically the problem seems to be that the way that we 
originally did euc-jp was only for two-byte support.  That 
isn't sufficient of course ...

hobbs added on 2002-04-20 08:24:12:
Logged In: YES 
user_id=72656

Perhaps the perl uses of the Tcl encoding stuff have 
provided a possible solution for us:

http://archive.develooper.com/perl-
[email protected]/msg00448.html

haible added on 2002-04-20 08:39:30:
Logged In: YES 
user_id=5923

If Perl has really copied the Tcl .enc files, then yes that
patch should be good for Tcl as well. Except for one thing:
the name of the encoding.

There are not two encodings euc-jp and euc-jp-0212, one
without and one with JISX0212.

There is only one encoding EUC-JP, and it includes JISX0212.

Those who think that EUC-JP does not include JISX0212 are
misled by the standard Japanese encoding on Windows -
Shift_JIS. This one does not contain JISX0212.

See Ken Lunde's book "CJKV Information Processing" if you
want the gory details of all Japanese encodings (probably
more details than you can bear :-)).

hobbs added on 2002-04-20 08:46:33:
Logged In: YES 
user_id=72656

If you look at the person's recommended patch, it appears 
that he uses the euc-jp-0212 part as just a special case 
name.  He turns euc-jp into an escape encoding to handle 
the jis0212 3-byte case.  It's a bit of a hack, but the 
essence is that using "euc-jp" would work correctly.