TIP 240: An Ensemble Command to Manage Processes

Login
Author:         Steve Bold <[email protected]>
State:          Draft
Type:           Project
Vote:           Pending
Created:        22-Feb-2005
Post-History:   
Keywords:       Tcl
Obsoletes:      88
Tcl-Version:    9.1
Tcl-Ticket:     1315115

Abstract

This TIP proposes some new commands through which Tcl scripts can create and monitor child processes.

Rationale

This TIP is intended to overcome the following limitations of the existing exec and open commands:

  1. While the stderr stream of a child process can be redirected to a file, it cannot be directed to a pipe and so cannot be captured progressively as the process runs. TIP 202 has partially addressed this issue but only for the case where the child's stderr stream is directed to the same pipe as its stdout stream. Independent progressive capture of both stdout and stderr is still not possible.

  2. In the (admittedly rare) case that a program has a significant delay between closing its standard streams and the process itself terminating, a Tcl script running that program as a background process cannot determine the exit status without blocking until the process terminates.

  3. The existing exec and open commands impose a special interpretation on the characters <>|&. This causes two kinds of problems:

    • scripts wishing to invoke a command on a remote computer using an rsh or similar command will sometimes wish to have characters such as <>|& interpreted on the remote machine

    • scripts may pass a user entered string as an argument to exec. Such scripts may break unexpectedly if the comment string contains one of the special characters. Such problems could be considered a security weakness in Tcl.

  4. Multiple child processes can be launched together with pipes used to link the streams of adjacent processes. However, little flexibility is provided in such cases, for example you can only capture the exit status of the last process in the pipeline.

A more general problem is that each process related command is a separate top-level command. This is inconsistent with much else in Tcl, makes it harder to find the related commands in some forms of documentation and increases the risk of name clashes as new process related commands are introduced.

The BLT toolkit contains the command bgexec which addresses items (1) and (2) in the above list. However, the resulting implementation is complex and does not appear easy to transfer to the Tcl core. In addition, it is not clear to the author how bgexec could be extended to address items (3) and (4).

A variety of other approaches to addressing these problems are listed on the Wiki. This suggests that it may be difficult to achieve a consensus on what the ideal command(s) for launching processes should look like. This TIP provides a basis through which many of these approaches could be implemented in pure Tcl. The commands specified in this TIP map easily onto the existing low level process related functions in the Tcl core, so the implementation cost is low.

Specification

There shall be a new ensemble command, process, with at least four subcommands.

  1. The sub-command invoke takes 4 arguments and invokes a sub-process, returning the process id of the child process. The arguments are (in order):

    * a list containing the program name invoke and its arguments

    * a channel to be connected to the stdin stream of the child process (or an empty string if the channel is to be disconnected in the child process).

    * a channel to be connected to the stdout stream of the child process (or an empty string if the channel is to be disconnected in the child process).

    * a channel to be connected to the stderr stream of the child process (or an empty string if the channel is to be disconnected in the child process).

  2. The sub-command status takes a single argument which is a process id and returns a two element list. The first element is either running or completed The second element is the exit status of the process.

    A process will report an arbitrary exit status of zero while it is running.

  3. The sub-command wait is similar to status but blocks until the child process has completed.

Examples

The following shows how the commands proposed here can be used to produce a bgexec like command in pure Tcl. Not all the bgexec options are included and the implementation lacks the error handling needed for a robust implementation.

proc bgExecCloseHandler {pid cmd} {
   lassign [process status $pid] status exitCode
   if {$status eq "running"} {
      puts "... deferring close handling for $pid"
      after 1000 [list bgExecCloseHandler $pid $cmd]
   } else {
      if {$cmd ne ""} {
         {expand}$cmd $pid $exitCode
      }
   }
}

proc bgExecReadHandler {chan cmd} {
   if {[gets $chan line] == -1} {
      close $chan

      if {[info exists ::bgExecCloseInfo($chan)]} {
         lassign $::bgExecCloseInfo($chan) pid cmd
         after 0 bgExecCloseHandler $pid $cmd
         unset ::bgExecCloseInfo($chan)
      }
   } else {
      {expand}$cmd $line
   }
}

proc bgExecLike {args} {
   set outChan ""; set errChan ""
   set i 0
   set exitCmd ""; set parentOutChan ""
   while {$i != [llength $args]} {
      set arg [lindex $args $i]
      switch -glob -- $arg {

      	-onoutput {
      	   incr i
      	   set cmd [lindex $args $i]
      	   lassign [chan pipe] parentChan outChan
      	   fileevent $parentChan readable [list \
                  bgExecReadHandler $parentChan $cmd]
      	   set outCmd $cmd
      	   set parentOutChan $parentChan
      	}

      	-onerror {
      	   incr i
      	   set cmd [lindex $args $i]
      	   lassign [chan pipe] parentChan errChan
      	   fileevent $parentChan readable [list \
                  bgExecReadHandler $parentChan $cmd]
      	}

      	-onexit {
      	   incr i
      	   set exitCmd [lindex $args $i]
      	}

      	-* {
      	   error "Unknown switch $arg"
      	}

      	* {
      	   break
      	}
      }
      incr i
   }

   set cmdLine [lrange $args $i end]

   # puts [list process invoke $cmdLine "" $outChan $errChan]
   set pid [process invoke $cmdLine "" $outChan $errChan]

   # Close the child end of the pipes - if we opened them.
   foreach var {outChan errChan} {
      if {[set $var] ne ""} {
         close [set $var]
      }
   }

   if {$parentOutChan eq ""} {
      # Poll for child process exit then notify client, or at least
      # clean up the zombie.
      after 0 bgExecCloseHandler $pid $exitCmd
   } else {
      # We copy BLT's trick of deferring polling till the stdout pipe
      # closes. This is marginally more efficient, more importantly
      # it stops clients being notified of their process until at stdout
      # channel has closed.
      set ::bgExecCloseInfo($parentOutChan) [list $pid $exitCmd]
   }   
}


# now show bgExecLike in action ...

proc showExit {pid code} {
   puts "$pid terminated with code $code"
}

proc showLine {channel line} {
   puts "$channel: $line"
}

proc runLs {args} {
   puts "invoking ls on $args"
   bgExecLike -onoutput "showLine stdout" -onerror "showLine stderr" \
           -onexit showExit ls {expand}$args
}

# Sample invocations: note when running under tclsh, there is no event loop,
# use 'update' to see the output to see what's happening.

# successful listing
runLs .

# Unsuccessful listing
runLs not-found

# Listing of (non existent) files containing exec/open meta characters
runLs < > | &

Limitations

  1. For convenient use, the functionality proposed here needs to be supplemented with additional commands providing a higher level interface, perhaps one of them being similar to the bgExecLike example given previously. The author has decided to omit this feature from the TIP because:

    * such commands can be implemented in pure Tcl using the commands described here

    * the exact nature of the high level commands may produce lengthy discussions

    * it could even be argued that such commands are more appropriate in tcllib rather than the Tcl core

  2. As with the current implementation of exec, each channel passed to process invoke must have a valid underlying OS file handle. Consequently when running on Windows:

    * use of a wish standard channel will be immediately rejected

    * use of a socket will be accepted but will trigger an error in the child process.

  3. Efficiency - The author has not yet attempted a detailed performance study, but this proposal does have some theoretical inefficiencies when compared to a pure C implementation, such as bgexec:

    * an intermediate Tcl procedure is used to capture output from a pipe

    * each end of each pipe has to be wrapped in a CommandChannel before it can be passed back to the calling script, even if the pipe is just going to be used to link together two processes in a pipeline.

Related Possibilities for Future Enhancements

  1. For Windows, an important limitation that is not addressed by this TIP, is the lack of control over the console window settings when invoking a process. This will require changes to TclpCreateProcess().

  2. A kill subcommand would be a useful addition to the process ensemble. On Windows, the ability to kill a child console process cleanly is related to the choice of console mode, so this issue would ideally be addressed in conjunction with item (1) above.

  3. A command to categorise an exit status obtained from process status or process wait along similar lines to the data placed in $errorCode by TclCleanupChildrens().

  4. Some aspects of the existing exec command depend on use of temporary files. Since this TIP transfers the high level implementation of process launching into Tcl scripts, support for creation of uniquely named temporary files, as proposed in TIP 210, would be useful.

  5. The wish console on Windows could be improved, using this mechanism, so that program names typed interactively will run in the background, allowing output to be seen before the process completes.

  6. The ability to define argv[0], independently from the program name, would occasionally be useful. For example, some UNIX shells run as login shells when argv[0] begins with a dash.

  7. Public C functions for invoking a process and creating a pipe wrapped in command channels.

  8. Support for detaching process and for reaping detached processes.

  9. The existing exit command could be duplicated in the process ensemble.

  10. The basic form of the existing pid command, which obtains the process id of the current process, could be added to the process ensemble.

  11. In some cases, it is more appropriate to run a child process with its streams connected to a null file rather than disconnected. Since the name of the null file is platform specific, it would be helpful to have a platform independent way of accessing the name.

  12. An option to obtain full status information. On Windows, process exit codes are 32 bit. On UNIX, higher bits of a waitpid() status value distinguish termination via exit() from termination via an uncaught signal.

Reference Implementation

Submitted as patch 1315115.

Copyright

This document has been placed in the public domain.