TIP 579: Improved auto_path for Safe Base interpreters

Login
Home Timeline Branches Tags Help 
Author:         Keith Nash <[email protected]>
State:          Final
Type:           Project
Vote:           Done
Created:        07-Jul-2020
Version:        31-Aug-2022
Post-History:
Tcl-Version:    8.7
Keywords:       safe interp auto_path access_path
Tcl-Branch:     tip-579-8-7
Vote-Summary:   Accepted 3/0/0
Votes-For:      JN, KBK, SL
Votes-Against:  none
Votes-Present:  none

Abstract

This TIP proposes modifying the Safe Base to give users the option of defining the ::auto_path in a Tcl 8.7 Safe Base interpreter to conform to the specification given in pkg_mkIndex(n) and library(n). The TIP's default is to preserve the existing behavior of the Safe Base, in which the ::auto_path includes a large number of additional directories. This choice of default will be changed in Tcl 9.0.

This TIP considers the multiple configuration options of a Safe Base interpreter, notably when these options are changed by safe::interpConfigure. However, in typical use the configuration is defined at interpreter creation and remains unchanged for the lifetime of the interpreter. In this case a much simpler analysis is possible (see section Discussion).

Rationale

A Safe Base interpreter is a safe child interpreter that can apply the commands load and source to files in certain directories specified by its parent interpreter. The list of those directories is called the access path and is recorded in the parent interpreter. In the Safe Base child, each directory of the access path is represented by a token of the form $p(:34:) which includes a literal dollar character and an integer. A filename can be composed as, for example, [file join {$p(:34:)} pkgIndex.tcl]. The Safe Base interpreter uses modified source and load commands that convert a filename rooted at an access path token to a real filename. The tclIndex autoloader, package command, and tcl::tm command in Safe Base interpreters are compatible with such tokenized filenames.

The access path comprises the directories of the requested ::auto_path and all their children, and the directories of the module path [tcl::tm::list] and all their descendants.

In a Safe Base interpreter, the value of ::auto_path is currently set to the tokenized access path, rather than to a tokenized form of the usual definition given by pkg_mkIndex(n) and library(n). This feature, not documented in safe(n), allowed package to work correctly despite the absence of the glob command in a Safe Base interpreter. The value of ::auto_path is synchronized with the tokenized access path whenever the latter is changed by a safe::* command in the parent interpreter.

When tcl::tm facilities were added to Tcl, a restricted form of glob was added to Safe Base interpreters. This new glob command allows package to operate correctly when ::auto_path conforms to the specification in pkg_mkIndex(n) and library(n). However, the Safe Base still defines the ::auto_path to be equal to the tokenized access path, and as a result it lists a large number of directories that are used by tcl::tm, and have nothing to do with the purpose of ::auto_path - seeking pkgIndex.tcl or tclIndex files. This feature is unhelpful when debugging, because the list of directories in the access path can be very long.

Proposed Change

(a) A new exported command safe::setSyncMode

(b) A new option for Safe Base interpreters, -autoPath, defined under certain conditions explained below.

(c) Revised behavior for the commands that can alter the access path, i.e. safe::interpCreate, safe::interpInit, safe::interpConfigure, and safe::interpAddToAccessPath.

The new command is

safe::setSyncMode ?newValue?

When an argument is supplied, the command returns an error in two cases: (a) if the argument is not a boolean value; (b) if any Safe Base interpreters exist.

Otherwise, the command is an accessor for a new namespace variable safe::AutoPathSync. Normal return supplies the value of this variable. The value is initialized when the file tcl8.7/safe.tcl is sourced by the autoloader.

The value [safe::setSyncMode] controls the behavior of all Safe Base interpreters, and can be changed only when no Safe Base interpreters are defined. Typically the value will be set as part of initialization.

When [safe::setSyncMode] is true, the behavior is the same as in current Tcl. The Safe Base interpreter's ::auto_path is defined as a tokenized form of the access path. Operations that change the Safe Base interpreter's access path also change its value of ::auto_path.

When [safe::setSyncMode] is false, ::auto_path is defined to fulfil the requirements of pkg_mkIndex(n) and library(n) only. It is not modified when the access path changes, except by operations that set the paths in the child to values that correspond to those of its parent - in such cases the child's ::auto_path is set to a tokenized form of the parent's ::auto_path.

When [safe::setSyncMode] is false, each Safe Base interpreter is given a new option "-autoPath" which can be used with the commands safe::interpCreate, safe::interpInit, and safe::interpConfigure in the same way as other options. The -autoPath option is undefined when [safe::setSyncMode] is true, permitting 100% compatibility with the existing Tcl Safe Base.

In order to implement this new definition of ::auto_path in a Safe Base interpreter, some changes must be made to the behavior of existing Safe Base commands when [safe::setSyncMode] is false. These changes are specified below.

Specification and Examples

The commands that can alter the access path are safe::interpCreate, safe::interpInit, safe::interpConfigure, and safe::interpAddToAccessPath. The effects of these commands on the access path and ::auto_path are described below, including the changes that are proposed by this TIP.

In this section:

The effects of options -accessPath and -autoPath on the value of the child's ::auto_path can be expressed in four rules, with a fifth rule for the command safe::interpAddToAccessPath.

(1) Calling safe::interpCreate, safe::interpInit, or safe::interpConfigure with an option/value pair for -autoPath.

(2) Calling safe::interpCreate or safe::interpInit without the option -accessPath, or with -accessPath {}; or calling safe::interpConfigure with -accessPath {}.

Examples of calls to commands safe::* that belong to this category are:

        safe::interpCreate foo
        safe::interpCreate foo -accessPath {}
        safe::interpInit bar
        safe::interpInit bar -accessPath {}
        safe::interpConfigure foo -accessPath {}

(3) Calling safe::interpCreate, safe::interpInit, or safe::interpConfigure with a non-empty value of -accessPath.

Each such command:

e.g.

        safe::interpCreate foo -accessPath {
            /usr/local/TclHome/lib/tcl8.7
            /usr/local/TclHome/lib/tcl8.7/opt0.4
            /usr/local/TclHome/lib/tcl8.7/msgs
            /usr/local/TclHome/lib/tcl8.7/encoding
            /usr/local/TclHome/lib
        }

        # The child's ::auto_path must be given a suitable value:

        safe::interpConfigure foo -autoPath {
            /usr/local/TclHome/lib/tcl8.7
            /usr/local/TclHome/lib
        }

        # The two commands can be combined:

        safe::interpCreate foo -accessPath {
            /usr/local/TclHome/lib/tcl8.7
            /usr/local/TclHome/lib/tcl8.7/opt0.4
            /usr/local/TclHome/lib/tcl8.7/msgs
            /usr/local/TclHome/lib/tcl8.7/encoding
            /usr/local/TclHome/lib
        } -autoPath {
            /usr/local/TclHome/lib/tcl8.7
            /usr/local/TclHome/lib
        }

(4) Other calls of command safe::interpConfigure with option/value pairs.

The only case not covered by rules (1) to (3) is a call to safe::interpConfigure with neither option -accessPath nor -autoPath. Such a call does not alter either the access path or the ::auto_path.

(5) Calling safe::interpAddToAccessPath.

e.g.

        safe::interpAddToAccessPath foo /usr/local/TclHome/lib/extras/Img1.4.11

        lassign [safe::interpConfigure foo -autoPath] DUM childAutoPath
        lappend childAutoPath /usr/local/TclHome/lib/extras/Img1.4.11
        safe::interpConfigure foo -autoPath $childAutoPath

        safe::interpAddToAccessPath foo /usr/local/TclHome/lib/sqlite3.30.1.2

        foreach dir {
            tzdata
            tzdata/Africa
            ...
            tzdata/US
        } {
            safe::interpAddToAccessPath foo /usr/local/TclHome/lib/tcl8.7/$path
        }

Summary of Specification

Discussion

Because this TIP adds a new option -autoPath, it must go into detail about the interplay between -autoPath and -accessPath, especially when values are changed by safe::interpConfigure. This leads to a fairly lengthy consideration of multiple cases (above).

When writing tests for the reference implementation, many bugs were discovered in the existing Safe Base code, most especially in handling changes of options by safe::interpConfigure. It is hard to avoid the conclusion that (like the present author) most users create a Safe Base interpreter with the properties that they want, and seldom if ever need to call safe::interpConfigure.

In this usage case, the entirety of the TIP's specification is greatly simplified:

Typical Use

In many cases, the properties of a Safe Base interpreter can be specified when the interpreter is created, and then left unchanged for the lifetime of the interpreter.

If you wish to use Safe Base interpreters with "Sync Mode" off, evaluate the command

       safe::setSyncMode 0

Use safe::interpCreate or safe::interpInit to create an interpreter with the properties that you require. The simplest way is not to specify -accessPath or -autoPath, which means the safe interpreter will use the same paths as the parent interpreter. However, if -accessPath is specified, then -autoPath must also be specified, or else it will be set to {}.

The value of -autoPath is that required to access tclIndex and pkgIndex.txt files according to the same rules as an unsafe interpreter (see pkg_mkIndex(n) and library(n)).

With "Sync Mode" on (the default), the option -autoPath is undefined, and the Safe Base sets the safe interpreter's ::auto_path to a tokenized form of the access path. In addition to the directories present if "Safe Mode" is off, the ::auto_path includes the numerous subdirectories and module paths that belong to the access path.

Reference Implementation

Branch tip-579-8-7 in the public fossil repository for Tcl.

Modified files

Compared to Tcl 8.7 (core-8-branch), branch tip-579-8-7 modifies the following files:

Compatibility

In Tcl 8.7, the default value of [safe::setSyncMode] is 1. The new interpreter option -autoPath is undefined in this case. All existing code will behave in the same way as before.

In Tcl 9.0, The default value of [safe::setSyncMode] will be changed to 0. The value 1 will remain available to ease the porting of legacy code, but must be set by the script.

Copyright

This document has been placed in the public domain.