TIP 554: Library file system for Tcl and Tk core using an C-based VFS

Login
Home Timeline Branches Tags Help 
Author:         Sean Woods <[email protected]>
Author:         Roy Keene
State:          Draft
Type:           Project
Vote:           Pending
Created:        14-Nov-2019
Tcl-Version:	9.1
Post-History:
Tcl-Branch:     tip-554

Abstract

This tip proposes storing the /library file systems for both the Tcl core and Tk core as C data structures presented as a virtual file system.

Rationale

In TIP #430 we explored the various issues with integrating a Zip based file system into interpreter initialization. While this process is finished and largely working, there are some awkward workarounds to manage the process of Zipfs encoding on exotic platforms. Some of these workaround involve forgoing the TIP entirely.

While the use of Zipfs as the VFS of record seems to be somewhat controversial, performance gains from not having to access (and explor) the local file system for every interpreter startup are measurable.

There is also the problem of having to expose the /library system (as built) to kit builders who have to copy it into a designated location in order for the Tcl core and Tk to start properly.

What would be ideal is if the /library file systems were part and parcel with the core libraries themselves.

Implementation

A script will read the Library file system, end encode it into a bespoke virtual file system implementation implemented in a single C file. That C file (tentatively named tclLibraryCvfs.c) will be placed in the generic/ file system, and compiled and linked along with the rest of the core.

The current process of discovering the location of init.tcl (for tcl) or tk.tcl (for Tk) will be replaced with the process of mounting this VFS, thus exposing it's contents to the interpreter.

Work for this project exists as the tip-544 branches of both Tcl and Tk. The tip-544 branch for Tcl is an offshoot of core-8-6-branch. The Branch for Tk is an offshoot of core-8-branch.

Maintenance

This C file will be maintained in a similar manner to the Stubs table. However, because the data contained is largely a copy, this file will not be tracked in Fossil. Distribution tarballs will contain a copy, but developers who work from fossil checkouts will have to periodically re-run the script. A makefile directive will be included to generate the file if it is missing, and "make genstubs" will replace it.

Specification

This system provides no external function calls on the script level, nor does it require adding new functions or modifying existing functions to either the public or private API of Tcl or Tk.

VFS Mount Point

The VFS implementation will advertise itself as //library:/core/tcl for the Tcl library system and //library:/core/tk for the Tk library system. Packages which need to utilize a similar technology are encouraged to adopt //library:/pkg/PACKAGE_NAME. It is intended that these file systems be small enough that keeping them memory resident will not impact the performance of a general purpose computer.

Installation changes

The Tcl and Tk installers will no longer need to provide a copy of the /library file system during installation. This process will be removed.

Memory Usage

On platforms that Tcl typically runs on, blocks of constants are kept in virtual memory, and are not loaded into working RAM. However, some primitive working environments may try to load these C constants into working RAM. The library file system of Tcl is current 4.3 mb (as determined by running tar cf, and measuring the file.) The library file system of Tk is 1.7 mb. On a computer capable of running Tk, even loaded into working memory will not even be a consideration. For an embedded system where every byte counts, we can leave out encodings, and that reduces the size to 2.7mb.

Backward Compatibility

The main area for concern for Compatibility issues is Tk. Tk can be compiled against Tcl cores of various vintages, including those which are not equipped with the features of this TIP. Because the VFS for Tk is self contained, and distributed by virtue of being compiled into the library itself, for most users there will be no change. (Save, of course, for the improved load time.)

Kit builders and speciality developers who provide their own library systems are still free to populate their own values for tcl_library and tk_library and those contracts will still be honored. The C-based VFS will only load if no other candidate to fill the role of the /library file system has stepped forward by the time the interpreter initializes.

Interactions with TIP #430

This TIP only replaces the portions of TIP #430 dealing with interpreter initialization. In all other respects the goals of this tip and non-startup goals of TIP #430 are parallel to one another.

Inspiration

This technology is influenced by Roy Keene's work with XVFS. It also draws on the lessons learned from implementing TIP #430.

Copyright

This document has been placed in the public domain.