TIP:            137
Title:          Specifying Script Encodings for [source] and tclsh
Version:        $Revision: 1.6 $
Author:         Anton Kovalenko <a_kovalenko@fromru.com>
State:          Final
Type:           Project
Vote:           Done
Created:        29-May-2003
Post-History:   
Tcl-Version:    8.5

~ Abstract

This TIP proposes a way to specify encoding used to read a script 
with ''source'' command and (tclsh, wish) shell.

~ Rationale

Now all Tcl scripts are expected to be in the system encoding.  An
author of a script or a package can't specify concrete encoding to
source the script or the files of package.  Common practise is to
assume that encoding to be superset of ASCII, and not to use non-ASCII
characters in the scripts that are targeting the wide audience of
potential users.

There's a way to specify UNICODE literals with \uXXXX sequences.  But
it's not an universal and convenient way - this sequences aren't
substituted in {}-quoted strings, and they can't be edited in WYSIWYG
editors without special (and hence uncommon) support.

This TIP proposes to add the ''-encoding'' option support to
''source'' command and to tclsh and wish.  Thus, package authors will be able
to specify encoding of the package files in ''pkgIndex.tcl''; and
script authors will be able to specify the script encoding when
calling tclsh (either in the first line beginning with ''#!'' or in
the line where the ''exec tclsh "$0" "$@"'' is located).

This TIP also proposes to use utf-8 for all the standard system
scripts, for ''pkgIndex.tcl'' and ''tclIndex'' files. Now they all are
supposed to be in the system encoding (message catalogs do not need
this support, as they are already always loaded using the UTF-8
encoding scheme), and it could event prevent Tcl itself from running
when system encoding is not a compatible superset of ASCII.

~ Specification

Tclsh will allow the encoding to be specified on the command-line in
two forms: the first form is ''-encoding name'' as two separate
arguments, and the second is ''-e:name'' (a single argument.)
The second form is intended for when the script begins with ''#!'' and
is because Unix kernels pass extra parameters from the ''#!'' line as
a single argument. This very short notation (-e:) is chosen because
some Unices limit the ''#!'' line to the length of 32.

To implement all these options, this TIP proposes a new C-level public API function Tcl_FSEvalFileEx, which is similar to Tcl_FSEvalFile, but
takes one extra argument that must be an encoding name or NULL (to use
system encoding).

| int Tcl_FSEvalFileEx(Tcl_Interp *interp, Tcl_Obj *pathPtr,
|                      CONST char *encodingName);

Common use of this new options will be like this:

 1. In a script:

| !#/usr/bin/tclsh -e:utf-8
| do something...

 > or it could be:

| #!/bin/sh
| # I wish to use tclsh \
| exec tclsh -encoding utf-8 "$0" "$@"

 2. In a ''pkgIndex.tcl'':

| package ifneeded pkg 1.02 [list source -encoding utf-8 \
|      [file join $dir pkg.tcl]]

~ Implementation

The partial implementation of this TIP can be found at
http://sourceforge.net/projects/tcl as Patch #742683. Some
fixes to source system scripts and pkgIndex'es in utf-8 are not
yet there. But with the existing implementation of "-encoding",
it would be really easy to implement.

~ Copyright

This document is placed in the public domain.
