TIP:            100
Title:          Add Support for Unloading Dynamic Libraries Loaded with [load]
Version:        $Revision: 1.10 $
Author:         George Petasis <petasis@iit.demokritos.gr>
State:          Final
Type:           Project
Vote:           Done
Created:        11-Jun-2002
Post-History:   
Discussions-To: news:comp.lang.tcl
Keywords:       load,unload,dynamic library
Tcl-Version:    8.5

~ Abstract

Tcl already provides facilities for loading dynamic libraries, through
the ''load'' command.  However no facilities are currently offered in
order to ''unload'' dynamic libraries already loaded with the ''load''
command.  This TIP tries to add support for unloading libraries, by
introducing a new Tcl command (''unload'') and the guidelines that
dynamic libraries must follow, in order to be unloadable.  Note that
the ''unload'' command will operate only on libraries that are
designed to be unloadable by their developers.  This way backward
compatibility with older extensions is maintained, as unload will
never try to unload libraries unaware of this new functionality.

~ Rationale

Tcl is an ideal language for component-based applications.  Usually
these applications offer a framework in which components developed by
the users of the application can be embedded, in order to extend the
functionality of the framework.  Usually, these extensions are
implemented as C/C++ dynamic libraries that are loaded through the
''load'' Tcl command.

However the development of such components can be a time-consuming
process, as developers have to restart the framework application in
order to be able to reload the library into it and test its altered
functionality.  And this can be quite annoying (depending on the
application of course), as usually processing within the application
is required in order to bring it into a proper state before testing
the library.

The development cycle can be significantly shortened if Tcl provides a
mechanism for unloading a dynamic library.  A new version of the
library can be created, as its object file can now be written, and the
updated library can be re-loaded.

However, this is not the only application of unload.
Services running for long periods of time and want to unload
no longer needed functionality, replacing parts of an applications
(i.e. from an automatic update procedure) or functionality
temporarily needed (i.e. a web browser that loads a plugin
to display a file of a particular file type) are some
additional fields of application.

~ Introduction

Unload functionality has been left out of the Tcl core, mainly because
library unloading was poorly implemented in many operating systems.
But almost all operating systems have been improved in the meantime,
and as a result most modern operating systems now support library
unloading.

The main idea of this TIP is to enable dynamic library unloading at
the Tcl level, in the same sense ''load'' provides dynamic library
loading.  However, library unloading will be provided only when the
underlying operating system support this feature (as is also the case
for ''load'') and only when the library to be unloaded provides a set
of functions that can "undo" the changes the library has made to the
interpreter.  In all other cases, unloading a library will result in
an error.

This TIP proposes the insertion of a new Tcl command named ''unload''
and two functions ''pkg_Unload'' and ''pkg_SafeUnload'', modelled
after ''pkg_Init'' and ''pkg_SafeInit'', that libraries which can be
unloaded should implement.

~ Unloadable Libraries

A main concern is related to when a shared library can be "unloadable".
An unloadable library is a library characterised as such by its
developer. The developer of the library must export a function
from the library, similar to the library's initialisation
function. The unload command will never try to unload a library
that does not provide such a function. This makes old libraries
(before the introduction of the unload functionality) safe.

There is a category of libraries that
can never be unloaded (i.e. libraries that register new
tcl object types). However, the choice is upon the developer:
the developer knows if the library can be unloadable. The simpler
case, libraries that only register new commands are the most
probable libraries to be unloadable.
Libraries that export functions through a stub mechanism cannot be
unloaded, as the were meant for having dependencies with
other libraries that use the exported API. There is no way for
the provider library to know wether it is used or not.

~ Specification

Actually, all the facilities for unloading dynamic libraries already
exist in the Tcl core and simply they are not yet exposed at the Tcl
level.  As a result, the implementation of the unload command should
be fairly easy.

''load'' as it is currently implemented loads a dynamic library only
the first time this library is loaded.  It keeps an internal cache of
all loaded libraries and if an already loaded library is requested
again, only its initialisation function is called.  This cache should
be extended to keep some additional information:

   1.  Two reference counts, counting how many times a specific
       library has been loaded.  This reference count should be
       increased by each ''load'' and decreased for each ''unload''.
       When it reaches 0, the library can be unloaded.  Safe
       interpreters and normal interpreters should have different
       reference counts.  Both should be 0 in order for a library to
       be unloaded.

   2.  The addresses of the ''pkg_Unload'' and ''pkg_SafeUnload''
       functions, if these are implemented by the library.  If both of
       these functions are missing, the library will never be
       unloaded.  If only ''pkg_Unload'' is implemented, the library
       can be unloaded if it never has been loaded in a safe
       interpreter.  Finally, if ''pkg_SafeUnload'' is implemented,
       the library can be unloaded if it has never been loaded in a
       normal interpreter.

The ''unload'' command will always return an error, if the operating
system does not support library unloading.  In case the operating
system supports library unloading:

   1.  ''unload'' will examine the cache of ''load'' to locate the
       entry for the library to be unloaded.  It is an error to unload
       a library that has not been loaded with ''load''.

   2.  If the entry in the cache is found, ''unload'' checks whether
       the corresponding for the interpreter type unload function
       pointer is NULL or not.  If it is NULL, the library cannot be
       unloaded under this interpreter and again an error is returned.

   3.  If the unload function pointer is not NULL, it is executed.  If
       an error is returned by this function, ''unload'' also returns
       an error.

   4.  If the unload function finishes without errors, the reference
       count corresponding to the interpreter type is decreased.  If
       both reference counts reach 0, the library is unloaded.

~ Responsibilities of the Unload Functions

Its up to the developer of the library to decide if its library can be
unloaded or not.  A library can be unloaded if the function
''pkg_Unload'' is implemented and exported as a symbol from the
library, and the library will never be loaded in a safe interpreter.
A library that can be also loaded in safe interpreters is unloadable
if the function ''pkg_SafeUnload'' is also available.  These two
functions will accept two arguments, the interpreter under which the
library is unloaded and an integer, holding various flags.  The flags argument
can be either ''TCL_UNLOAD_DETACH_FROM_INTERPRETER'' or
''TCL_UNLOAD_DETACH_FROM_PROCESS''. In case the library will remain attached to
the process after the unload procedure returns (i.e. because the library is
used by other interpreters), TCL_UNLOAD_DETACH_FROM_INTERPRETER will be defined.
However, if the library is used only by the target interpreter and the library
will be detached from the application as soon as the unload procedure returns,
the flags argument will be set to TCL_UNLOAD_DETACH_FROM_PROCESS. 

The main responsibility of these functions is to remove from the
interpreter they are unloaded under any reference to a function
residing inside the library.  For example, such a function must:

   1.  Unregister any commands that have been registered by the
       ''Init()'' function to the ''interpreter'' given by the first
       argument.  In order to do this, the library should keep
       internally the tokens returned by each ''Tcl_Create*Command'',
       as command may have been renamed.

   2.  Unregister any other commands that may have been registered to
       the interpreter during the use of the library (usually used to
       represent special special data structures).

If the flag ''TCL_UNLOAD_DETACH_FROM_PROCESS'' is defined, the
developer must do additional task, that are not normally required when
the library gets unloaded from an interpreter:

   3.  Free any memory occupied by the internal structures of
       the library.

   4.  In general, try to remove any references Tcl may have
       to functions provided by the library.

If the developer cannot remove all reference to functions to the
library, its better to not provide at all these two functions, so as
unload to never attempt to unload the library.

~ Dependencies Among Libraries

It is possible that a library A has been loaded that exports some
symbols.  Then a library B is loaded, that has dependencies (i.e. uses
some exported symbols) on A.  What if A gets unloaded?

Actually, most modern operating systems seem to provide a solution to
this problem, as reference counts are hold internally by the operating
system for each library.  Newer Windows, Solaris and Linux seem to
provide similar solutions and in reality they don't unload the library
if such symbols remain, even if the unload system call has been made
for the library.  Both libraries A and B have to be unloaded in order
for A to be really removed from the address space.

~ Reference Implementation

A reference implementation can be found at:
http://sf.net/tracker/?func=detail&aid=823486&group_id=10894&atid=310894

~ Copyright

This document has been placed in the public domain.

~ Appendix: The unload man page.

     NAME
     unload - Unload machine code.
     SYNOPSIS
     unload ?switches? fileName
     unload ?switches? fileName packageName
     unload ?switches? fileName packageName interp
     DESCRIPTION
     -nocomplain
     -keeplibrary
     - -
     PORTABILITY ISSUES
     Unix
     Macintosh
     BUGS
     SEE ALSO
     KEYWORDS

NAME

unload - Unload machine code.

SYNOPSIS

unload ?switches? fileName

unload ?switches? fileName packageName

unload ?switches? fileName packageName interp

DESCRIPTION

This command tries to unload shared libraries previously
loaded with load from the application's address space.
fileName is the name of the file containing the library
file to be unload; it must be the same as the filename
provided to load for loading the library. packageName is
the name of the package, and is used to compute the name
of the unload procedure. interp is the path name of the
interpreter from which to unload the package (see the
interp manual entry for details); if interp is omitted,
it defaults to the interpreter in which the unload
command was invoked.

If the initial arguments to unload start with - then they
are treated as switches. The following switches are
currently supported:

-nocomplain
     Supresses all error messages. If this switch is
     given unload will never report an error.

-keeplibrary
     This switch will prevent unload from issuing the
     operating system call that will unload the library
     from the process.

--
     Marks the end of switches. The argument following
     this one will be treated as a fileName even if it
     starts with a -.

When a file containing a shared library is loaded through
the load command, Tcl associates two reference counts to
the library file. The first counter shows how many times
the library has been loaded into normal (trusted)
interpreters while the second describes how many times
the library has been loaded into safe interpreters. As a
file containing a shared library can be loaded only once
by Tcl (with the first load call on the file), these
counters track how many interpreters use the library.
Each subsequent call to load after the first, simply
increaments the proper reference count.

unload works in the opposite direction. As a first step,
unload will check whether the library is unloadable: an
unloadable library exports a special unload procedure.
The name of the unload procedure is determined by
packageName and whether or not the target interpreter is
a safe one. For normal interpreters the name of the
initialization procedure will have the form pkg_Unload,
where pkg is the same as packageName except that the
first letter is converted to upper case and all other
letters are converted to lower case. For example, if
packageName is foo or FOo, the initialization procedure's
name will be Foo_Unload. If the target interpreter is a
safe interpreter, then the name of the initialization
procedure will be pkg_SafeUnload instead of pkg_Unload.

If unload determines that a library is not unloadable (or
unload functionality has been disabled during
compilation), an error will be returned. If the library
is unloadable, then unload will call the unload
procedure. If the unload procedure returns TCL_OK, unload
will proceed and decrease the proper reference count
(depending on the target interpreter type). When both
reference counts have reached 0, the library will be
detached from the process.

The unload procedure must match the following prototype:
typedef int Tcl_PackageUnloadProc(Tcl_Interp *interp, int
flags);

The interp argument identifies the interpreter from which
the library is to be unloaded. The unload procedure must
return TCL_OK or TCL_ERROR to indicate whether or not it
completed successfully; in the event of an error it
should set the interpreter's result to point to an error
message. In this case, the result of the unload command
will be the result returned by the unload procedure. The
flags argument can be either
TCL_UNLOAD_DETACH_FROM_INTERPRETER or
TCL_UNLOAD_DETACH_FROM_PROCESS. In case the library will
remain attached to the process after the unload procedure
returns (i.e. because the library is used by other
interpreters), TCL_UNLOAD_DETACH_FROM_INTERPRETER will be
defined. However, if the library is used only by the
target interpreter and the library will be detached from
the application as soon as the unload procedure returns,
the flags argument will be set to
TCL_UNLOAD_DETACH_FROM_PROCESS.

The unload command cannot unload libraries that are
statically linked with the application. If fileName is an
empty string, then packageName must be specified.
If packageName is omitted or specified as an empty
string, Tcl tries to guess the name of the package. This
may be done differently on different platforms. The
default guess, which is used on most UNIX platforms, is
to take the last element of fileName, strip off the first
three characters if they are lib, and use any following
alphabetic and underline characters as the module name.
For example, the command unload libxyz4.2.so uses the
module name xyz and the command unload bin/last.so {}
uses the module name last.

PORTABILITY ISSUES

Unix
     Not all unix operating systems support library
     unloading. Under such an operating system unload
     returns an error (unless -nocomplain has been
     specified).

Macintosh
     <Somebody to comment on this?>

BUGS

If the same file is loaded by different fileNames, it
will be loaded into the process's address space multiple
times. The behavior of this varies from system to system
(some systems may detect the redundant loads, others may
not). In case a library has been silently detached by the
operating system (and as a result Tcl thinks the library
is still loaded), it may be dangerous to use unload on
such a library (as the library will be completely
detached from the application while some interpreters
will continue to use it).

SEE ALSO

info sharedlibextension, load, safe

KEYWORDS

binary code, unloading, safe interpreter, shared library
