Tcl Library Source Code

# Meta description  This application is a simple processor
# Meta description  for documents written in the doctools
# Meta description  markup language. It covers the most
# Meta description  common use cases, but is not as
# Meta description  configurable as its big brother dtp.
# Meta category     Processing doctools documents
# Meta subject      doctools doctoc docidx
# Meta require      {doctools 1}
# Meta require      {doctools::idx 1}
# Meta require      {doctools::toc 1}
# Meta require      fileutil
# Meta require      textutil::repeat
# Meta author       Andreas Kupries
# Meta license      BSD
# @@ Meta End

package provide dtplite 1.0.4

# dtp lite - Lightweight DocTools Processor
# ======== = ==============================
#
# Use cases
# ---------
#
# (1)	Validation of a single manpage, i.e. checking that it is valid
#	doctools format.
#
# (1a)	Getting a preliminary version of the formatted output, for
#	display in a browser, nroff, etc., proofreading the
#	formatting.
#
# (2)	Generate documentation for a single package, i.e. all the
#	manpages, plus index and table of contents.
#
# (3)	Generation of unified documentation for several
#	packages. Especially unified keyword index and table of
#	contents. This may additionally generate per-package TOCs
#	as well (Per-package indices don't make sense IMHO).
#
# Command syntax
# --------------
#
# Ad 1)	dtplite -o output format file
#
#	The option -o specifies where to write the output to. Using
#	the string "-" as name of the output file causes the tool to
#	write the generated data to stdout. If $output is a directory
#	then a file named [[file rootname $file].$format] is written
#	to the directory.

# Ad 1a)	dtplite validate file
#
#	The "validate" format does not generate output at all, only
#	syntax checking is performed.
#
# Ad 2)	dtplite -o output format directory
#
#	I.e. we distinguish (2) from (1) by the type of the input,
#	file, or directory. In this situation output has to be a
#	directory. Use the path "." to place the results into the
#	current directory.
#
#	We locate _all_ files under directory, i.e. all subdirectories
#	are scanned as well. We replicate the found directory
#	structure in the output (See example below). The index and
#	table of contents are written to the toplevel directory in the
#	output. The names are hardwired to "toc.$format" and
#	"index.$format".
#
# Ad 3)	dtplite -merge -o output format directory
#
#	This can be treated as special case of (2). The -merge option
#	tells it that the output is nested one level deeper, to keep a
#	global toc and index in the toplevel and to merge the package
#	toc and index into them.
#
#	This way the global documents are built up incrementally. This
#	can help us in a future extended installer as well!, extending
#	a global documentation tree of all installed packages.
#
# Additional features.
#
# *	As described above the format name is used as the extension
#	for the generated files. Does it make sense to introduce an
#	option with which we can overide this, or should we simply
#	extect that a calling script does a proper renaming of all the
#	files ?  ... The option is better. In HTML output we have
#	links between the files, and renaming from the outside just
#	breaks the links. This option is '-ext'. It is ignored if the
#	output is a single file (fully specified via -o), or stdout.
#
#	-ext extension
#
# *	Most of the formats don't need much/none of customizability.
#	I.e. text, nroff, wiki, tmml, ...  For HTML however some
#	degree of customizability is required for good output.  What
#	should we given to the user ?
#
#	- Allow setting of a stylesheet.
#	- Allow integration of custom body header and footer html.
#	- Allow additional links for the navigation bar.
#
#	Note: The tool generates standard navigation bars to link the
#	all tocs, indices, and pages together.
#
#	-style file
#	-header file
#	-footer file
#	-nav label url
#
# *	The application may mis-detect files as doctools input.
#	And we cannot always mark them as non-doctools because
#	they may be such. Test cases, for example. To exclude
#	these we have the option '-exclude' taking a glob pattern.
#	Multiple uses of the option accumulate.
#
#	-exclude glob
#
# *	For tcllib itself we have external tools generating a nicer
#	TOC. Use option -toc to specify the doctoc file to use instead
#	of generating our own.
#
#	-toc path
#
# That should be enough to allow the creation of good looking formatted
# documentation without getting overly complex in both implementation
# and use.

package require doctools      1.4.11 ; # 'image' support, -ibase support
package require doctools::idx 1.0.4 ;
package require doctools::toc 1.1.3 ;
package require fileutil
package require textutil::repeat

# ### ### ### ######### ######### #########
## Internal data and status

namespace eval ::dtplite {

    # Path to where the output goes to. This is a file in case of mode
    # 'file', irrelevant for mode 'file.stdout', and the directory for
    # all the generated files for the two directory modes. Specified
    # through the mandatory option '-o'.

    variable  output ""

    # Path to where the documents to convert come from. This is a
    # single file in the case of the two file modes, and a directory
    # for the directory modes. In the later case all files under that
    # directory are significant, including links, if identifiable as
    # in doctools format (fileutil::fileType). Specified through the
    # last argument on the command line. The relative path of a file
    # under 'input' also becomes its relative path under 'output'.

    variable  input  ""

    # The extension to use for the generated files. Ignored by the
    # file modes, as for them they either don't generate a file, or
    # know its full name already, i.e. including any wanted
    # extension. Set via option '-ext'. Defaults to the format name if
    # '-ext' was not used.

    variable  ext    ""

    # Optional. HTML specific, requires engine parameter 'meta'. Path
    # to a stylesheet file to use in the output. The file modes link
    # to it using the original location, but the directory modes copy
    # the file into the 'output' and link it there (to make the
    # 'output' more selfcontained). Initially set via option '-style'.

    variable  style  ""

    # Optional. Path to a file. Contents of the file are assigned to
    # engine parameter 'header', if present. If navigation buttons
    # were defined their HTML will be appended to the file contents
    # before doing the assignment. A specification is ignored if the
    # engine does not support the parameter 'header'. Set via option
    # '-header'.

    variable  header ""

    # Like header, but for the document footer, and no navigation bar
    # insert. Set via option '-footer', requires engine parameter
    # 'footer'.

    variable  footer ""

    # List of buttons/links for a navigation bar. No navigation bar is
    # created if this is empty. HTML specific, requires engine
    # parameter 'header' (The navigation bar is merged with the
    # 'header' data, see above). Each element of the list is a
    # 2-element list, containing the button label and url, in this
    # order. Initial data comes from the command line, via option
    # '-nav'. The commands 'Navbutton(Push|Pop)' then allow the
    # programmatic addition and removal of buttons at the left (stack
    # like, top at index 0). This is used for the insertion of links
    # to TOC and Index into each document, if applicable.

    variable  nav    {}

    # An array caching the result of merging header and navbar data,
    # keyed by the navbar definition (list). This allows us to quickly
    # access the complete header for a navbar, without having to
    # generate it over and over again. Its usefulness is a bit limited
    # by the fact that the navbar itself can be generated on a
    # file-by-file basis (required to get the relative links
    # correct. It helps only if the generated navbars are identical to
    # each other.

    variable  navcache
    array set navcache {}

    # The name of the format to convert the doctools documents
    # into. Set via the next-to-last argument on the command
    # line. Used as extension for the generated files as well by the
    # directory modes, and if not overridden via '-ext'. See 'ext'
    # above.

    variable  format ""

    # Boolean flag. Set by the option '-merge'. Ignored when a file
    # mode is detected, but for a directory it determines the
    # difference between the two directory modes, i.e. plain
    # generation, or incremental merging of many inputs into one
    # output.

    variable  merge  0

    # Boolean flag. Automatically set by code distinguishing between
    # file and directory modes. Set for a the file modes, unset for
    # the directory modes.

    variable  single 1

    # Boolean flag. Automatically set by the code processing the '-o'
    # option. Set if output is '-', unset otherwise. Ignored for the
    # directory modes. Distinguished between the two file modes, i.e.
    # writing to a file (unset), or stdout (set).

    variable  stdout 0

    # Name of the found processing mode. Derived from the values of
    # the three boolean flags (merge, single, stdout). This value is
    # used during the dispatch to the command implementing the mode,
    # after processing the command line.
    #
    # Possible/Legal values:	Meaning
    # ---------------------	-------
    # File			File mode. Write result to a file.
    # File.Stdout		File mode. Write result to stdout.
    # Directory			Directory mode. Plain processing of one set.
    # Directory.Merge		Directory mode. Merging of multiple sets into
    #				one output.
    # ---------------------	-------

    variable  mode   ""

    # Name of the module currently processed. Derived from the 'input'
    # (last element of this path, without extension).

    variable  module ""

    # Crossreference data. Extracted from the processed documents, a
    # rearrangement and filtration of the full meta data (See 'meta'
    # below). Relevant only to the directory modes. I.e. the file
    # modes don't bother with its extraction and use.

    variable  xref
    array set xref   {}

    # Index data. Mapping from keyword (label) to the name of its
    # anchor in the index output. Requires support for the engine
    # parameter 'kwid' in the index engine.

    variable  kwid
    array set kwid {}

    # Cache. This array maps from the path of an input file/document
    # (relative to 'input'), to the paths of the file to generate
    # (relative to 'output', including extension and such). In other
    # words we derive the output paths from the inputs only once and
    # then simply get them here.

    variable  out
    array set out  {}

    # Meta data cache. Stores the meta data extracted from the input
    # files/documents, per input. The meta data is a dictionary and
    # processed several ways to get: Crossreferences (See 'xref'
    # above), Table Of Contents, and Keyword Index. The last two are
    # not cached, but ephemeral.

    variable  meta
    array set meta {}

    # Cache of input documents. When we read an input file we store
    # its contents here, keyed by path (relative to 'input') so that
    # we don't have to go to the disk when we we need the file again.
    # The directory modes need each input twice, for metadata
    # extraction, and the actual conversion.

    variable  data
    array set data {}

    # Database of image files for use by dt_imap.

    variable  imap
    array set imap {}

    # Database of exclusion patterns. Files matching these are not
    # manpages. For example, test files for doctools itself may fall
    # under this.

    variable excl {}

    # Path of a user specified table of contents (doctoc format).

    variable utoc {}
}

# ### ### ### ######### ######### #########
## External data and status
#
## Only the directory merge mode uses external data, saving the
## internal representations of current toc, index. and xref
## information for use by future mergers. It uses three files,
## described below. The files are created if they don't exist.
## Remove them when the merging is complete.
#
## .toc
## Contains the current full toc in form of a dictionary.
#  Keys are division labels, values the lists of toc items.
#
## .idx
## Contains the current full index, plus keyword id map.  Is a list of
#  three elements, index, start id for new kwid entries, and the
#  keyword id map (kwid). Index and Kwid are both dictionaries, keyed
#  by keywords. Index value is a list of 2-tuples containing symbolic
#  file plus label, in this order. Kwid value is the id of the anchor
#  for that keyword in the index.
#
## .xrf
## Contains the current cross reference database, a dictionary. Keys
#  are tags the formatter can search for (keywords, keywrds with
#  prefixes, keywords with suffixces), values a list containing either
#  the file to refer to to, or both file and an anchor in that
#  file. The latter is for references into the index.

# ### ### ### ######### ######### #########
## Option processing.
## Validate command line.
## Full command line syntax.
##
# dtplite	-o outputpath	\
#		?-merge?	\
#		?-ext ext?	\
#		?-style file?	\
#		?-header file?	\
#		?-footer file?	\
#		?-nav label url?... \
#		?-exclude glob?... \
#		?-toc path? \
#		format inputpath
##

proc ::dtplite::processCmdline {} {
    global argv

    variable output ; variable style  ; variable stdout
    variable format ; variable header ; variable single
    variable input  ; variable footer ; variable mode
    variable ext    ; variable nav    ; variable merge
    variable module ; variable excl   ; variable utoc

    # Process the options, perform basic validation.

    while {[llength $argv]} {
	set opt [lindex $argv 0]
	if {![string match "-*" $opt]} break

	if {[string equal $opt "-o"]} {
	    if {[llength $argv] < 2} Usage
	    set output [lindex $argv 1]
	    set argv   [lrange $argv 2 end]
	} elseif {[string equal $opt "-merge"]} {
	    set merge 1
	    set argv [lrange $argv 1 end]
	} elseif {[string equal $opt "-ext"]} {
	    if {[llength $argv] < 2} Usage
	    set ext  [lindex $argv 1]
	    set argv [lrange $argv 2 end]
	} elseif {[string equal $opt "-toc"]} {
	    if {[llength $argv] < 2} Usage
	    set utoc [lindex $argv 1]
	    set argv [lrange $argv 2 end]
	} elseif {[string equal $opt "-exclude"]} {
	    if {[llength $argv] < 2} Usage
	    lappend excl [lindex $argv 1]
	    set argv [lrange $argv 2 end]
	} elseif {[string equal $opt "-style"]} {
	    if {[llength $argv] < 2} Usage
	    set style [lindex $argv 1]
	    set argv  [lrange $argv 2 end]
	} elseif {[string equal $opt "-header"]} {
	    if {[llength $argv] < 2} Usage
	    set header [lindex $argv 1]
	    set argv   [lrange $argv 2 end]
	} elseif {[string equal $opt "-footer"]} {
	    if {[llength $argv] < 2} Usage
	    set footer [lindex $argv 1]
	    set argv   [lrange $argv 2 end]
	} elseif {[string equal $opt "-nav"]} {
	    if {[llength $argv] < 3} Usage
	    lappend nav [lrange $argv 1 2]
	    set argv    [lrange $argv 3 end]
	} else {
	    Usage
	}
    }

    # Additional validation, and extraction of the non-option
    # arguments.

    if {[llength $argv] != 2} Usage

    set format [lindex $argv 0]
    set input  [lindex $argv 1]

    if {[string equal $format validate]} {
	set format null
    }

    # Final validation across the whole configuration.

    if {[string equal $format ""]} {
	ArgError "Illegal empty format specification"

    } else {
	# Early check: Is the chosen format ok ? For this we have
	# create and configure a doctools object.

	doctools::new dt
	if {[catch {dt configure -format $format}]} {
	    ArgError "Unknown format \"$format\""
	}
	dt configure -deprecated 1

	# Check style, header, and footer options, if present.

	CheckInsert header {Header file}
	CheckInsert footer {Footer file}

	if {[llength $nav] && ![in [dt parameters] header]} {
	    ArgError "-nav not supported by format \"$format\""
	}
	if {![string equal $style ""]} {
	    if {![in [dt parameters] meta]} {
		ArgError "-style not supported by format \"$format\""
	    } elseif {![file exists $style]} {
		ArgError "Unable to find style file \"$style\""
	    }
	}
    }

    # Set up an extension based on the format, if no extension was
    # specified.  also compute the name of the module, based on the
    # input. [SF Tcllib Bug 1111364]. Has to come before the line
    # marked with a [*], or a filename without extension is created.

    if {[string equal $ext ""]} {
	set ext $format
    }

    CheckInput $input {Input path}
    if {[file isfile $input]} {
	# Input file. Merge mode is not possible. Output can be file
	# or directory, or "-" for stdout. The output may exist, but
	# does not have to. The directory it is in however does have
	# to exist, and has to be writable (if the output does not
	# exist yet). An existing output has to be writable.

	if {$merge} {
	    ArgError "-merge illegal when processing a single input file."
	}
	if {![string equal $output "-"]} {
	    CheckTheOutput

	    # If the output is an existing directory then we have to
	    # ensure that the actual output is a file in that
	    # directory, and we derive its name from the name of the
	    # input file (and -ext, if present).

	    if {[file isdirectory $output]} {
		# [*] [SF Tcllib Bug 1111364]
		set output [file join $output [file tail [Output $input]]]
	    }
	} else {
	    set stdout 1
	}
    } else {
	# Input directory. Merge mode is possible. Output has to be a
	# directory. The output may exist, but does not have to. The
	# directory it is in however does have to exist. An existing
	# output has to be writable.

	set single 0
	CheckTheOutput 1
    }

    # Determine the operation mode from the flags

    if {$single} {
	if {$stdout} {
	    set mode File.Stdout
	} else {
	    set mode File
	}
    } elseif {$merge} {
	set mode Directory.Merge
    } else {
	set mode Directory
    }

    set module [file rootname [file tail [file normalize $input]]]
    return
}

# ### ### ### ######### ######### #########
## Option processing.
## Helpers: Generation of error messages.
## I.  General usage/help message.
## II. Specific messages.
#
# Both write their messages to stderr and then
# exit the application with status 1.
##

proc ::dtplite::Usage {} {
    global argv0
    puts stderr "$argv0 wrong#args, expected:\
	    -o outputpath ?-merge? ?-ext ext?\
	    ?-style file? ?-header file?\
	    ?-footer file? ?-nav label url?...\
	    format inputpath"
    exit 1
}

proc ::dtplite::ArgError {text} {
    global argv0
    puts stderr "$argv0: $text"
    exit 1
}

proc in {list item} {
    expr {([lsearch -exact $list $item] >= 0)}
}

# ### ### ### ######### ######### #########
## Helper commands. File paths.
## Conversion of relative paths
## to absolute ones for input
## and output. Derivation of
## output file name from input.

proc ::dtplite::Pick {f} {
    variable input
    return [file join $input $f]
}

proc ::dtplite::Output {f} {
    variable ext
    return [file rootname $f].$ext
}

proc ::dtplite::At {f} {
    variable output
    set of     [file normalize [file join $output $f]]
    file mkdir [file dirname $of]
    return $of
}

# ### ### ### ######### ######### #########
## Check existence and permissions of an input/output file or
## directory.

proc ::dtplite::CheckInput {f label} {
    if {![file exists $f]} {
	ArgError "Unable to find $label \"$f\""
    } elseif {![file readable $f]} {
	ArgError "$label \"$f\" not readable (permission denied)"
    }
    return
}

proc ::dtplite::CheckTheOutput {{needdir 0}} {
    variable output
    variable format

    if {[string equal $format null]} {
	# The format does not generate output, so not specifying an
	# output file is ok for that case.
	return
    }

    if {[string equal $output ""]} {
	ArgError "No output path specified"
    }

    set base [file dirname $output]
    if {[string equal $base ""]} {set base [pwd]}

    if {![file exists $output]} {
	if {![file exists $base]} {
	    ArgError "Output base path \"$base\" not found"
	}
	if {![file writable $base]} {
	    ArgError "Output base path \"$base\" not writable (permission denied)"
	}
    } else {
	if {![file writable $output]} {
	    ArgError "Output path \"$output\" not writable (permission denied)"
	}
	if {$needdir && ![file isdirectory $output]} {
	    ArgError "Output path \"$output\" not a directory"
	}
    }
    return
}

proc ::dtplite::CheckInsert {option label} {
    variable format
    variable $option
    upvar 0  $option opt

    if {![string equal $opt ""]} {
	if {![in [dt parameters] $option]} {
	    ArgError "-$option not supported by format \"$format\""
	}
	CheckInput $opt $label
	set opt [Get $opt]
    }
    return
}

# ### ### ### ######### ######### #########
## Helper commands. File reading and writing.

proc ::dtplite::Get {f} {
    variable data
    if {[info exists data($f)]} {return $data($f)}
    return [set data($f) [fileutil::cat $f]]
}

proc ::dtplite::Write {f data} {
    # An empty filename is acceptable, the format will be 'null'
    if {[string equal $f ""]} return
    fileutil::writeFile $f $data
    return
}

# ### ### ### ######### ######### #########
## Dump accumulated warnings.

proc ::dtplite::Warnings {} {
    set warnings [dt warnings]
    if {[llength $warnings] > 0} {
	puts stderr [join $warnings \n]
    }
    return
}

# ### ### ### ######### ######### #########
## Configuation phase, validate command line.

::dtplite::processCmdline

# ### ### ### ######### ######### #########
## We can assume that we have from here on a command 'dt', which is a
## doctools object command, and already configured for the format to
## generate.
# ### ### ### ######### ######### #########

# ### ### ### ######### ######### #########
## Commands implementing the main functionality.

proc ::dtplite::Do.File {} {
    # Process a single input file, write the result to a single outut file.

    variable input
    variable output

    SinglePrep
    Write $output [dt format [Get $input]]
    Warnings
    return
}

proc ::dtplite::Do.File.Stdout {} {
    # Process a single input file, write the result to stdout.

    variable input

    SinglePrep
    puts  stdout [dt format [Get $input]]
    close stdout
    Warnings
    return
}

proc ::dtplite::Do.Directory {} {
    # Process a directory of input files, through all subdirectories.
    # Generate index and toc, but no merging with an existing index
    # and toc. I.e. any existing index and toc files are overwritten.

    variable input
    variable out
    variable module
    variable meta
    variable format
    variable utoc

    # Phase 0. Find the documents to convert.
    # Phase I. Collect meta data, and compute the map from input to
    # ........ output files. This is also the map for the symbolic
    # ........ references. We extend an existing map (required for use
    # ........ in merge op.
    # Phase II. Build index and toc information from the meta data.
    # Phase III. Convert each file, using index, toc and meta
    # .......... information.

    MapImages
    set files [LocateManpages $input]
    if {![llength $files]} {
	ArgError "Module \"$module\" has no files to process."
    }

    MetadataGet $files
    StyleMakeLocal

    if {$utoc ne {}} {
	if {[file exists $utoc]} {
	    set utoc [Get $utoc]
	}
	TocWrite toc index $utoc
    } else {
	TocWrite toc index [TocGenerate [TocGet $module toc]]
    }
    IdxWrite index toc [IdxGenerate $module [IdxGet]]

    dt configure -module $module
    XrefGet
    XrefSetup   dt
    FooterSetup dt
    MapSetup    dt

    foreach f [lsort -dict $files] {
	puts stdout \t$f

	set o $out($f)
	dt configure -file [At $o] -ibase $input/$f

	NavbuttonPush {Keyword Index}     [Output index] $o
	NavbuttonPush {Table Of Contents} [Output toc]   $o
	HeaderSetup dt
	NavbuttonPop
	NavbuttonPop
	StyleSetup dt $o

	if {[string equal $format null]} {
	    dt format [Get [Pick $f]]
	} else {
	    Write [At $o] [dt format [Get [Pick $f]]]
	}
	Warnings
    }
    return
}

proc ::dtplite::Do.Directory.Merge {} {
    # See Do.Directory, but merge the TOC/Index information from this
    # set of input files into an existing TOC/Index.

    variable input
    variable out
    variable module
    variable meta
    variable output
    variable format
    variable utoc

    # Phase 0. Find the documents to process.
    # Phase I. Collect meta data, and compute the map from input to
    # ........ output files. This is also the map for the symbolic
    # ........ references. We extend an existing map (required for use
    # ........ in merge op.
    # Phase II. Build module local toc from the meta data, insert it
    # ......... into the main toc as well, and generate a global
    # ......... index.
    # Phase III. Process each file, using cross references, and links
    # .......... to boths tocs and the index.

    MapImages
    set files [LocateManpages $input]
    if {![llength $files]} {
	ArgError "Module \"$module\" has no files to process."
    }

    MetadataGet $files $module
    StyleMakeLocal     $module

    set localtoc [TocGet $module $module/toc]
    TocWrite $module/toc index [TocGenerate $localtoc] [TocMap $localtoc]
    if {$utoc ne {}} {
	if {[file exists $utoc]} {
	    set utoc [Get $utoc]
	}
	TocWrite toc index $utoc
    } else {
	TocWrite toc index [TocGenerate [TocMergeSaved $localtoc]]
    }
    IdxWrite index toc [IdxGenerate {} [IdxGetSaved index]]

    dt configure -module $module
    XrefGetSaved
    XrefSetup   dt
    FooterSetup dt
    MapSetup    dt

    foreach f [lsort -dict $files] {
	puts stdout \t$f

	set o $out($f)
	dt configure -file [At $o] -ibase $input/$f

	NavbuttonPush {Keyword Index}          [Output index]       $o
	NavbuttonPush {Table Of Contents}      [Output $module/toc] $o
	NavbuttonPush {Main Table Of Contents} [Output toc]         $o
	HeaderSetup dt
	NavbuttonPop
	NavbuttonPop
	NavbuttonPop
	StyleSetup dt $o

	if {[string equal $format null]} {
	    dt format [Get [Pick $f]]
	} else {
	    Write [At $o] [dt format [Get [Pick $f]]]
	}
	Warnings
    }
    return
}

# ### ### ### ######### ######### #########
## Helper commands. Preparations shared between the two file modes.

proc ::dtplite::SinglePrep {} {
    variable input
    variable module

    MapImages
    StyleSetup  dt
    HeaderSetup dt
    FooterSetup dt
    MapSetup    dt

    dt configure -module $module -file $input
    return
}

# ### ### ### ######### ######### #########
## Get the base meta data out of the listed documents.

proc ::dtplite::MetadataGet {files {floc {}}} {
    # meta :: map (symbolicfile -> metadata)
    # metadata = dict (key -> value)
    # key      = set { desc, fid, file, keywords,
    #                  module, section, see_also,
    #                  shortdesc, title, version }
    # desc      :: string 'document title'
    # fid       :: string           'file name, without path/extension'
    # file      :: string           'file name, without path'
    # keywords  :: list (string...) 'key phrases'
    # module    :: string           'module the file is in'
    # section   :: string           'manpage section'
    # see_also  :: list (string...) 'related files'
    # shortdesc :: string           'module description'
    # title     :: string           'manpage file name intended'
    # version   :: string           'file/package version'
    variable meta
    variable input
    variable out

    doctools::new meta -format list -deprecated 1
    foreach f $files {
	meta configure -file $input/$f
	set o [Output [file join $floc files $f]]
	set out($f)  $o
	set meta($o) [lindex [string trim [meta format [Get [Pick $f]]]] 1]
    }
    meta destroy
    return
}

# ### ### ### ######### ######### #########
## Handling Tables of Contents:
## - Get them out of the base meta data.
## - As above, and merging them with global toc.
## - Conversion of internals into doctoc.
## - Processing doctoc into final formatting.

proc ::dtplite::TocGet {desc {f toc}} {
    # Generate the intermediate form of a TOC for the current document
    # set. This generates a single division.

    # Get toc out of the meta data.
    variable meta
    set res {}
    foreach {k item} [array get meta] {
	lappend res [TocItem $k $item]
    }
    return [list $desc [list $f $res]]
}

proc ::dtplite::TocMap {toc {base {}}} {
    if {$base == {}} {
	set base  [lindex [lindex $toc 1] 0]
    }
    set items [lindex [lindex $toc 1] 1]

    set res {}
    foreach i $items {
	foreach {f label desc} $i break
	lappend res $f [fileutil::relativeUrl $base $f]
    }
    return $res
}

proc ::dtplite::TocItem {f meta} {
    array set md $meta
    set desc    $md(desc)
    set label   $md(title)
    return [list $f $label $desc]
}

proc ::dtplite::TocMergeSaved {sub} {
    # sub is the TOC of the current doc set (local toc). Merge this
    # into the main toc (as read from the saved global state), and
    # return the resulting internal rep for further processing.

    set fqn [At .toc]
    if {[file exists $fqn]} {
	array set _ [Get $fqn]
    }
    array set _ $sub
    set thetoc [array get _]

    # Save extended toc for next merge.
    Write $fqn $thetoc

    return $thetoc
}

proc ::dtplite::TocGenerate {data} {
    # Handling single and multiple divisions.
    # single div => div is full toc
    # multi div  => place divs into the toc in alpha order.
    #
    # Sort toc (each division) by label (index 1).
    # Write as doctoc.

    array set toc $data

    TagsBegin
    if {[array size toc] < 2} {
	# Empty, or single division. The division is the TOC, toplevel.

	unset toc
	set desc [lindex $data 0]
	set data [lindex [lindex $data 1] 1]
	TocAlign mxf mxl $data

	Tag+ toc_begin [list {Table Of Contents} $desc]
	foreach item [lsort -dict -index 1 $data] {
	    foreach {symfile label desc} $item break
	    Tag+ item \
		    [FmtR mxf $symfile] \
		    [FmtR mxl $label] \
		    [list $desc]
	}
    } else {
	Tag+ toc_begin [list {Table Of Contents} Modules]
	foreach desc [lsort -dict [array names toc]] {
	    foreach {ref div} $toc($desc) break
	    TocAlign mxf mxl $div

	    Tag+ division_start [list $desc [Output $ref]]
	    foreach item [lsort -dict -index 1 $div] {
		foreach {symfile label desc} $item break
		Tag+ item \
			[FmtR mxf $symfile] \
			[FmtR mxl $label] \
			[list $desc]
	    }
	    Tag+ division_end
	}
    }

    Tag+ toc_end

    #puts ____________________\n[join $lines \n]\n_________________________
    return [join $lines \n]\n
}

proc ::dtplite::TocWrite {ftoc findex text {map {}}} {
    variable format

    if {[string equal $format null]} return
    Write [At .tocdoc] $text

    set ft [Output $ftoc]

    doctools::toc::new toc -format $format -file $ft

    NavbuttonPush {Keyword Index} [Output $findex] $ftoc
    HeaderSetup  toc
    NavbuttonPop
    FooterSetup  toc
    StyleSetup   toc $ftoc

    foreach {k v} $map {toc map $k $v}

    Write [At $ft] [toc format $text]
    toc destroy
    return
}

proc ::dtplite::TocAlign {fv lv div} {
    upvar 1 $fv mxf $lv mxl
    set mxf 0
    set mxl 0
    foreach item $div {
	foreach {symfile label desc} $item break
	Max mxf $symfile
	Max mxl $label
    }
    return
}

# ### ### ### ######### ######### #########
## Handling Keyword Indices:
## - Get them out of the base meta data.
## - As above, and merging them with global index.
## - Conversion of internals into docidx.
## - Processing docidx into final formatting.

proc ::dtplite::IdxGet {{f index}} {
    # Get index out of the meta data.
    array set keys {}
    array set kdup {}
    return [lindex [IdxExtractMeta] 1]
}

proc ::dtplite::IdxGetSaved {{f index}} {
    # Get index out of the meta data, merge into global state.
    variable meta
    variable kwid

    array set keys {}
    array set kwid {}
    array set kdup {}
    set start 0

    set fqn [At .idx]
    if {[file exists $fqn]} {
	foreach {kw kd start ki} [Get $fqn] break
	array set keys $kw
	array set kwid $ki
	array set kdup $kd
    }

    foreach {start theindex} [IdxExtractMeta $start] break

    # Save extended index for next merge.
    Write $fqn [list $theindex [array get kdup] $start [array get kwid]]

    return $theindex
}

proc ::dtplite::IdxExtractMeta {{start 0}} {
    # Get index out of the meta data.
    variable meta
    variable kwid

    upvar keys keys kdup kdup
    foreach {k item} [array get meta] {
	foreach {symfile keywords label} [IdxItem $k $item] break
	# Store inverted file - keyword relationship
	# Kdup is used to prevent entering of duplicates.
	# Checks full (keyword file label).
	foreach k $keywords {
	    set kx [list $k $symfile $label]
	    if {![info exists kdup($kx)]} {
		lappend keys($k) [list $symfile $label]
		set kdup($kx) .
	    }
	    if {[info exist kwid($k)]} continue
	    set kwid($k) key$start
	    incr start
	}
    }
    return [list $start [array get keys]]
}

proc ::dtplite::IdxItem {f meta} {
    array set md $meta
    set keywords $md(keywords)
    set title    $md(title)
    return [list $f $keywords $title]
}

proc ::dtplite::IdxGenerate {desc data} {
    # Sort by keyword label.
    # Write as docidx.

    array set keys $data

    TagsBegin
    Tag+ index_begin [list {Keyword Index} $desc]

    foreach k [lsort -dict [array names keys]] {
	IdxAlign mxf $keys($k)

	Tag+ key [list $k]
	foreach v [lsort -dict -index 1 $keys($k)] {
	    foreach {file label} $v break
	    Tag+ manpage [FmtR mxf $file] [list $label]
	}
    }

    Tag+ index_end
    #puts ____________________\n[join $lines \n]\n_________________________
    return [join $lines \n]\n
}

proc ::dtplite::IdxWrite {findex ftoc text} {
    variable format

    if {[string equal $format null]} return
    Write [At .idxdoc] $text

    set fi [Output $findex]

    doctools::idx::new idx -format $format -file $fi

    NavbuttonPush {Table Of Contents} [Output $ftoc] $findex
    HeaderSetup   idx
    NavbuttonPop
    FooterSetup   idx
    StyleSetup    idx $findex
    XrefSetupKwid idx

    Write [At $fi] [idx format $text]
    idx destroy
    return
}

proc ::dtplite::IdxAlign {v keys} {
    upvar 1 $v mxf
    set mxf 0
    foreach item $keys {
	foreach {symfile label} $item break
	Max mxf $symfile
    }
    return
}

# ### ### ### ######### ######### #########
## Column sizing

proc ::dtplite::Max {v str} {
    upvar 1 $v max
    set l [string length [list $str]]
    if {$max < $l} {set max $l}
    return
}

proc ::dtplite::FmtR {v str} {
    upvar 1 $v max
    return [list $str][textutil::repeat::blank \
	    [expr {$max - [string length [list $str]]}]]
}

# ### ### ### ######### ######### #########
## Code generation.

proc ::dtplite::Tag {n args} {
    if {[llength $args]} {
	return "\[$n [join $args]\]"
    } else {
	return "\[$n\]"
    }
    #return \[[linsert $args 0 $n]\]
}

proc ::dtplite::Tag+ {n args} {
    upvar 1 lines lines
    lappend lines [eval [linsert $args 0 ::dtplite::Tag $n]]
    return
}

proc ::dtplite::TagsBegin {} {
    upvar 1 lines lines
    set lines {}
    return
}

# ### ### ### ######### ######### #########
## Collect all files for possible use as image

proc ::dtplite::MapImages {} {
    variable input
    variable output
    variable single
    variable stdout

    # Ignore images when writing results to a pipe.
    if {$stdout} return

    set out  [file normalize $output]
    set path [file normalize $input]
    set res  {}

    if {$single} {
	# output is file, image directory is sibling to it.
	set imgbase [file join [file dirname $output] image]
	# input to search is director the input file is in, and below
	set path    [file dirname $path]
    } else {
	# output is directory, image directory is inside.
	set imgbase [file join $out image]
    }

    set n [llength [file split $path]]

    foreach f [::fileutil::find $path] {
	MapImage \
	    [::fileutil::stripN $f $n] \
	    $f [file join $imgbase [file tail $f]]
    }
    return
}

proc ::dtplite::MapImage {path orig dest} {
    # A file a/b/x.y is stored under
    # a/b/x.y, b/x.y, and x.y

    variable imap
    set plist [file split $path]
    while {[llength $plist]} {
	set imap([join $plist /]) [list $orig $dest]
	set plist [lrange $plist 1 end]
    }
    return
}

proc ::dtplite::MapSetup {dt} {
    # imap :: map (symbolicfile -> list (originpath,destpath)))
    variable imap
    # Skip if no data available

    #puts MIS|[array size imap]|
    if {![array size imap]} return

    foreach sf [array names imap] {
	foreach {origin destination} $imap($sf) break
	$dt img $sf $origin $destination
    }
    return
}

# ### ### ### ######### ######### #########
## Find the documents to process.

proc ::dtplite::LocateManpages {path} {
    set path [file normalize $path]
    set n    [llength [file split $path]]
    set res  {}
    foreach f [::fileutil::find $path ::dtplite::IsDoctools] {
	lappend res [::fileutil::stripN $f $n]
    }
    return $res
}

proc ::dtplite::IsDoctools {f} {
    set res [expr {[in [::fileutil::fileType $f] doctools] && ![Excluded [file normalize $f]]}]
    #puts ...$f\t$res\t|[fileutil::fileType $f]|\texcluded=[Excluded [file normalize $f]]\tin.[pwd]
    return $res
}

proc ::dtplite::Excluded {f} {
    variable excl
    foreach p $excl {
	if {[string match $p $f]} {return 1}
    }
    return 0
}

# ### ### ### ######### ######### #########
## Handling a style sheet
## - Decoupling output from input location.
## - Generate HTML to insert into a generated document.

proc ::dtplite::StyleMakeLocal {{pfx {}}} {
    variable style
    if {[string equal $style ""]} return
    set base [file join $pfx [file tail $style]]

    # TODO input == output does what here ?

    file copy -force $style [At $base]
    set style $base
    return
}

proc ::dtplite::StyleSetup {o {f {}}} {
    variable style
    if {[string equal $style ""]}   return
    if {![in [$o parameters] meta]} return

    if {![string equal $f ""]} {
	set dst [fileutil::relativeUrl $f $style]
    } else {
	set dst $style
    }
    set value "<link\
	    rel=\"stylesheet\"\
	    href=\"$dst\"\
	    type=\"text/css\">"

    $o setparam meta $value
    return
}

# ### ### ### ######### ######### #########
## Handling the cross references
## - Getting them out of the base meta data.
## - ditto, plus merging with saved xref information.
## - Insertion into processor, cached list.
## - Setting up the keyword-2-anchor map.

proc ::dtplite::XrefGet {} {
    variable meta
    variable xref
    variable kwid

    array set keys {}
    foreach {symfile item} [array get meta] {
	array set md $item
	# Cross-references ... File based, see-also

	set t  $md(title)
	set ts ${t}($md(section))
	set td $md(desc)

	set xref(sa,$t)  [set _ [list $symfile]]
	set xref(sa,$ts) $_
	set xref($t)     $_ ; # index on manpage file name
	set xref($ts)    $_ ; # ditto, with section added
	set xref($td)    $_ ; # index on document title

	# Store an inverted file - keyword relationship, for the index
	foreach kw $md(keywords) {
	    lappend keys($kw) $symfile
	}
    }

    set if [Output index]
    foreach k [array names keys] {
	if {[info exists xref(kw,$k)]} continue

	set frag $kwid($k)
	set xref(kw,$k) [set _ [list $if $frag]]
	set xref($k)    $_
    }
    return
}

proc ::dtplite::XrefGetSaved {} {
    # xref :: map (xrefid -> list (symbolicfile))
    variable  xref
    array set xref {}

    # Load old cross references, from a previous run
    set fqn [At .xrf]
    if {[file exists $fqn]} {
	array set xref [set s [Get $fqn]]
    }

    # Add any new cross references ...
    XrefGet
    Write $fqn [array get xref]
    return
}

proc ::dtplite::XrefSetup {o} {
    # xref :: map (xrefid -> list (symbolicfile))
    variable xref
    # Skip if no data available
    if {![array size xref]}         return
    # Skip if backend doesn't support an index
    if {![in [$o parameters] xref]} return

    # Transfer index data to the backend. The data we keep has to be
    # re-formatted from a dict into a list of tuples with leading
    # xrefid.

    # xrefl :: list (list (xrefid symbolicfile...)...)
    variable xrefl
    if {![info exist xrefl]} {
	set xrefl {}
	foreach k [array names xref] {
	    lappend xrefl [linsert $xref($k) 0 $k]
	    set f [lindex $xref($k) 0]
	    dt map $f [At $f]
	}
    }
    $o setparam xref $xrefl
    return
}

proc ::dtplite::XrefSetupKwid {o} {
    # kwid :: map (label -> anchorname)
    variable kwid
    # Skip if no data available
    if {![array size kwid]}         return
    # Skip if backend doesn't support an index
    if {![in [$o parameters] kwid]} return
    # Transfer index data to the backend
    $o setparam kwid [array get kwid]
    return
}

# ### ### ### ######### ######### #########
## Extending and shrinking the navigation bar.

proc ::dtplite::NavbuttonPush {label file ref} {
    # nav = list (list (label reference) ...)
    variable nav
    set      nav [linsert $nav 0 [list $label [fileutil::relativeUrl $ref $file]]]
    return
}

proc ::dtplite::NavbuttonPop {} {
    # nav = list (list (label reference) ...)
    variable nav
    set      nav [lrange $nav 1 end]
    return
}

# ### ### ### ######### ######### #########
## Header/Footer mgmt
## Header is merged from regular header, plus nav bar.
## Caching the merge result for quicker future access.

proc ::dtplite::HeaderSetup {o} {
    variable header
    variable nav
    variable navcache

    if {[string equal $header ""] && ![llength $nav]} return
    if {![in [$o parameters] header]}                 return

    if {![info exists navcache($nav)]} {
	set sep 0
	set hdr ""
	if {![string equal $header ""]} {
	    append hdr $header
	    set sep 1
	}
	if {[llength $nav]} {
	    if {$sep} {append hdr <br>\n}
	    append hdr <hr>\ \[\n

	    set first 1
	    foreach item $nav {
		if {!$first} {append hdr "| "} else {append hdr "  "}
		set first 0
		foreach {label url} $item break
		append hdr "<a href=\"" $url "\">" $label "</a>\n"
	    }
	    append hdr \]\ <hr>\n
	}
	set navcache($nav) $hdr
    } else {
	set hdr $navcache($nav)
    }

    $o setparam header $hdr
    return
}

proc ::dtplite::FooterSetup {o} {
    variable footer
    if {[string equal $footer ""]}    return
    if {![in [$o parameters] footer]} return
    $o setparam footer $footer
    return
}

# ### ### ### ######### ######### #########
## Invoking the functionality.

if {[catch {
    set mode $::dtplite::mode
    ::dtplite::Do.$mode
} msg]} {
    ## puts $::errorInfo
    ::dtplite::ArgError $msg
}

# ### ### ### ######### ######### #########
exit

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.TP
[2]
The following directory structure is created when processing a single
set of input documents.  The file extension used is for output in
HTML, but that is not relevant to the structure and was just used to
have proper file names.
.CS


    output/
        toc.html
        index.html
        files/
            path/to/FOO.html

.CE
.IP
The last line in the example shows the document
generated for a file FOO located at
.CS


    inputdirectory/path/to/FOO

.CE
.TP
[3]
When merging many packages into a unified set of documents the
generated directory structure is a bit deeper:
.CS


    output
        .toc
        .idx
        .tocdoc
        .idxdoc
        .xrf
        toc.html
        index.html
        FOO1/
            ...
        FOO2/
            toc.html
            files/
                path/to/BAR.html

.CE
.IP
Each of the directories FOO1, ... contains the documents generated for
the package FOO1, ... and follows the structure shown for use case
[2]. The only exception is that there is no per-package index.
.sp
The files "\fI.toc\fR", "\fI.idx\fR", and "\fI.xrf\fR" contain the

.SH KEYWORDS
HTML, TMML, conversion, docidx, doctoc, doctools, manpage, markup, nroff
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2004 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
application, client, name service
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2007-2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
application, name service, server
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2007-2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
application, client, name service
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

[file dirname $output] has to exist and must be a writable
directory, as the application will create the fileto write to.
.sp
If the argument is missing \fB-\fR is assumed.
.RE
.PP
.SS OPERATION
... reading ... transforming ... writing - plugins - pipeline ...
.SS OPTIONS
This section describes all the options available to the user of the
application. Options are always processed in order. I.e. of both
\fB--help\fR and \fB--version\fR are specified the option
encountered first has precedence.
.PP
Unknown options specified before any of the options \fB-rd\fR,

them, and was not superceded by a plugin option coming after.
.PP
Default options are used if and only if the command line did not
contain any options at all. They will set the application up as a
PEG-based parser generator. The exact list of options is
.PP
.CS

-c peg
.CE
.PP
And now the recognized options and their arguments, if they have any:
.PP
.TP
\fB--help\fR

.TP
\fIpeg\fR
It sets the application up as a parser generator accepting parsing
expression grammars and writing a packrat parser in Tcl. The actual
arguments it specifies are:
.sp
.CS


	--reset
	--append
	--reader    peg
	--transform reach
	--transform use
	--writer    me

.CE
.sp
.RE
.TP
\fB-r\fR \fIname\fR
Readers. The name of the package for the plugin \fIname\fR is
"page::reader::\fIname\fR".

.SH KEYWORDS
parser generator, text processing
.SH CATEGORY
Page Parser Generator
.SH COPYRIGHT
.nf
Copyright (c) 2005 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

Please report such in the category \fIdocstrip\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
application and/or documentation.
.SH "SEE ALSO"
docstrip
.SH KEYWORDS
.dtx, LaTeX, conversion, docstrip, documentation, literate programming, markup, source
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2005 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

the output. Errors in encryption affect the current block and the next
block after which the cipher will correct itself. CBC is the most
commonly used mode in software encryption. This is the default mode
of operation for this module.
.PP
.SH EXAMPLES
.CS


% set nil_block [string repeat \\\\0 16]
% aes::aes -hex -mode cbc -dir encrypt -key $nil_block $nil_block
66e94bd4ef8a2c3b884cfa59ca342b2e

.CE
.CS


set Key [aes::Init cbc $sixteen_bytes_key_data $sixteen_byte_iv]
append ciphertext [aes::Encrypt $Key $plaintext]
append ciphertext [aes::Encrypt $Key $additional_plaintext]
aes::Final $Key

.CE
.SH REFERENCES
.IP [1]
"Advanced Encryption Standard",
Federal Information Processing Standards Publication 197, 2001
(\fIhttp://csrc.nist.gov/publications/fips/fips197/fips-197.pdf\fR)
.PP

aes, block cipher, data integrity, encryption, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2005, Pat Thoyts <[email protected]>
Copyright (c) 2012, Andreas Kupries <[email protected]>

.fi

'\"
'\" Generated from file '/net/nas/data/andreask/Dev/Tcllib/tcllib/embedded/man/files/modules/amazon-s3/S3.n' by tcllib/doctools with format 'nroff'
'\" Copyright (c) Copyright 2006,2008 Darren New. All Rights Reserved. See LICENSE.TXT for terms.
'\"
'\" The definitions below are for supplemental macros used in Tcl/Tk
'\" manual entries.
'\"
'\" .AP type name in/out ?indent?
'\"	Start paragraph describing an argument to a library procedure.
'\"	type is type of argument (int, etc.), in/out is either "in", "out",

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH COMMANDS
This package provides several separate levels of complexity.
.IP \(bu
The lowest level simply takes arguments to be sent to the service,
sends them, retrieves the result, and provides it to the caller.
\fINote:\fR This layer allows both synchronous and event-driven
processing. It depends on the MD5 and SHA1 and base64 packages
from Tcllib (available at \fIhttp://tcllib.sourceforge.net/\fR).
Note that \fBS3::Configure\fR is required for \fBS3::REST\fR to
work due to the authentication portion, so we put that in the "lowest level."
.IP \(bu
The next layer parses the results of calls, allowing for functionality
such as uploading only changed files, synchronizing directories,
and so on.  This layer depends on the \fBTclXML\fR package as well as the
included \fBxsxp\fR package. These packages are package required when

.TP
\fB-prefix\fR
This names the prefix that will be added to all resources.
That is, it is the remote equivalent of \fB-directory\fR.
If it is not specified, the root of the bucket will be treated
as the remote directory. An example may clarify.
.CS


S3::Push -bucket test -directory /tmp/xyz -prefix hello/world

.CE
.IP
In this example, /tmp/xyz/pdq.html will be stored as
http://s3.amazonaws.com/test/hello/world/pdq.html in Amazon's servers. Also,
/tmp/xyz/abc/def/Hello will be stored as
http://s3.amazonaws.com/test/hello/world/abc/def/Hello in Amazon's servers.
Without the \fB-prefix\fR option, /tmp/xyz/pdq.html would be stored

\fB-delete\fR \fItrue\fR to delete files that have been
deleted from one place but not the other yet not copying
changed files is untested.
.PP
.SH "USAGE SUGGESTIONS"
To fetch a "directory" out of a bucket, make changes, and store it back:
.CS


file mkdir ./tempfiles
S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \\
  -timestamp aws
do_my_process ./tempfiles other arguments
S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \\
  -compare newer -delete true

.CE
.PP
To delete files locally that were deleted off of S3 but not otherwise
update files:
.CS


S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \\
  -compare never -delete true

.CE
.SH "FUTURE DEVELOPMENTS"
The author intends to work on several additional projects related to
this package, in addition to finishing the unfinished features.
.PP
First, a command-line program allowing browsing of buckets and
transfer of files from shell scripts and command prompts is useful.

package and/or documentation.
.SH KEYWORDS
amazon, cloud, s3
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) Copyright 2006,2008 Darren New. All Rights Reserved. See LICENSE.TXT for terms.

.fi

'\"
'\" Generated from file '/net/nas/data/andreask/Dev/Tcllib/tcllib/embedded/man/files/modules/amazon-s3/xsxp.n' by tcllib/doctools with format 'nroff'
'\" Copyright (c) Copyright 2006 Darren New. All Rights Reserved.
'\"
'\" The definitions below are for supplemental macros used in Tcl/Tk
'\" manual entries.
'\"
'\" .AP type name in/out ?indent?
'\"	Start paragraph describing an argument to a library procedure.
'\"	type is type of argument (int, etc.), in/out is either "in", "out",

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

%PCDATA?
is like %PCDATA, but returns an empty string if
no PCDATA is found.
.RE
.sp
For example, to fetch the first bold text from the fifth paragraph of the body of your HTML file,
.CS

xsxp::fetch $pxml {html body p#4 b} %PCDATA
.CE
.sp
.TP
\fBxsxp::fetchall\fR \fIpxml_list\fR \fIpath\fR ?\fIpart\fR?
This iterates over each PXML in \fIpxml_list\fR (which must be a list
of pxmls) selecting the indicated path from it, building a new list

package and/or documentation.
.SH KEYWORDS
dom, parser, xml
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) Copyright 2006 Darren New. All Rights Reserved.

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2004 Andreas Kupries <[email protected]>
Copyright (c) 2004 Jochen Loewer <[email protected]>
Copyright (c) 2004-2011 Michael Schlenker <[email protected]>

.fi

'\"
'\" Generated from file '/net/nas/data/andreask/Dev/Tcllib/tcllib/embedded/man/files/modules/base32/base32.n' by tcllib/doctools with format 'nroff'
'\" Copyright (c) Public domain
'\"
'\" The definitions below are for supplemental macros used in Tcl/Tk
'\" manual entries.
'\"
'\" .AP type name in/out ?indent?
'\"	Start paragraph describing an argument to a library procedure.
'\"	type is type of argument (int, etc.), in/out is either "in", "out",

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH "CODE MAP"
The code map used to convert 5-bit sequences is shown below, with the
numeric id of the bit sequences to the left and the character used to
encode it to the right. It should be noted that the characters "0" and
"1" are not used by the encoding. This is done as these characters can
be easily confused with "O", "o" and "l" (L).
.CS


	0 A    9 J   18 S   27 3
	1 B   10 K   19 T   28 4
	2 C   11 L   20 U   29 5
	3 D   12 M   21 V   30 6
	4 E   13 N   22 W   31 7
	5 F   14 O   23 X
	6 G   15 P   24 Y
	7 H   16 Q   25 Z
	8 I   17 R   26 2

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIbase32\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
base32, rfc3548
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) Public domain

.fi

'\"
'\" Generated from file '/net/nas/data/andreask/Dev/Tcllib/tcllib/embedded/man/files/modules/base32/base32core.n' by tcllib/doctools with format 'nroff'
'\" Copyright (c) Public domain
'\"
'\" The definitions below are for supplemental macros used in Tcl/Tk
'\" manual entries.
'\"
'\" .AP type name in/out ?indent?
'\"	Start paragraph describing an argument to a library procedure.
'\"	type is type of argument (int, etc.), in/out is either "in", "out",

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

package and/or documentation.
.SH KEYWORDS
base32
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) Public domain

.fi

'\"
'\" Generated from file '/net/nas/data/andreask/Dev/Tcllib/tcllib/embedded/man/files/modules/base32/base32hex.n' by tcllib/doctools with format 'nroff'
'\" Copyright (c) Public domain
'\"
'\" The definitions below are for supplemental macros used in Tcl/Tk
'\" manual entries.
'\"
'\" .AP type name in/out ?indent?
'\"	Start paragraph describing an argument to a library procedure.
'\"	type is type of argument (int, etc.), in/out is either "in", "out",

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH "CODE MAP"
The code map used to convert 5-bit sequences is shown below, with the
numeric id of the bit sequences to the left and the character used to
encode it to the right. The important feature of the extended hex
mapping is that the first 16 codes map to the digits and hex
characters.
.CS


	0 0    9 9        18 I   27 R
	1 1   10 A        19 J   28 S
	2 2   11 B        20 K   29 T
	3 3   12 C        21 L   30 U
	4 4   13 D        22 M   31 V
	5 5   14 E        23 N
	6 6   15 F        24 O
	7 7        16 G   25 P
	8 8        17 H   26 Q

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIbase32\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
base32, hex, rfc3548
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) Public domain

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

\fB::ascii85::decode\fR \fIstring\fR
Ascii85 decodes the given \fIstring\fR and returns the binary data.
The decoder ignores whitespace in the string, as well as tabs and
newlines.
.PP
.SH EXAMPLES
.CS


% ascii85::encode "Hello, world"
87cURD_*#TDfTZ)

.CE
.CS


% ascii85::encode [string repeat xyz 24]
G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G
^4U[H$X^\\H?a^]
% ascii85::encode -wrapchar "" [string repeat xyz 24]
G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]G^4U[H$X^\\H?a^]

.CE
.CS


# NOTE: ascii85 encodes BINARY strings.
% set chemical [encoding convertto utf-8 "C\\u2088H\\u2081\\u2080N\\u2084O\\u2082"]
% set encoded [ascii85::encode $chemical]
6fN]R8E,5Pidu\\UiduhZidua
% set caffeine [encoding convertfrom utf-8 [ascii85::decode $encoded]]

.CE
.SH REFERENCES
.IP [1]
\fIhttp://en.wikipedia.org/wiki/Ascii85\fR
.IP [2]
Postscript Language Reference Manual, 3rd Edition, page 131.
\fIhttp://www.adobe.com/devnet/postscript/pdfs/PLRM.pdf\fR

.SH KEYWORDS
ascii85, encoding
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2010, Emiliano Gavilán

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.TP
\fB::base64::decode\fR \fIstring\fR
Base64 decodes the given \fIstring\fR and returns the binary data.
The decoder ignores whitespace in the string.
.PP
.SH EXAMPLES
.CS


% base64::encode "Hello, world"
SGVsbG8sIHdvcmxk

.CE
.CS


% base64::encode [string repeat xyz 20]
eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6
eHl6eHl6eHl6
% base64::encode -wrapchar "" [string repeat xyz 20]
eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6

.CE
.CS


# NOTE: base64 encodes BINARY strings.
% set chemical [encoding convertto utf-8 "C\\u2088H\\u2081\\u2080N\\u2084O\\u2082"]
% set encoded [base64::encode $chemical]
Q+KCiEjigoHigoBO4oKET+KCgg==
% set caffeine [encoding convertfrom utf-8 [base64::decode $encoded]]

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIbase64\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
base64, encoding
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2000, Eric Melski
Copyright (c) 2001, Miguel Sofer

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

pattern expressed as an octal string. To change the default of 0644
you can set this option. For instance, 0755 would be suitable for an
executable. See \fBchmod(1)\fR.
.PP
.SH EXAMPLES
.PP
.CS


% set d [uuencode::encode "Hello World!"]
2&5L;&\\\\@5V]R;&0A





.CE
.PP
.CS


% uuencode::uudecode $d
Hello World!

.CE
.PP
.CS


% set d [uuencode::uuencode -name hello.txt "Hello World"]
begin 644 hello.txt
+2&5L;&\\@5V]R;&0`
`
end

.CE
.PP
.CS


% uuencode::uudecode $d
{hello.txt 644 {Hello World}}

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIbase64\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
encoding, uuencode
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

-crc32 boolean
The yEnc specification recommends the inclusion of a cyclic redundancy
check value in the footer. Use this option to change the default from
\fItrue\fR to \fIfalse\fR.
.PP
.PP
.CS


% set d [yencode::yencode -file testfile.txt]
=ybegin line=128 size=584 name=testfile.txt
 -o- data not shown -o-
=yend size=584 crc32=ded29f4f

.CE
.SH REFERENCES
.IP [1]
\fIhttp://www.yenc.org/yenc-draft.1.3.txt\fR
.PP
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIbase64\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
encoding, yEnc, ydecode, yencode
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
BitTorrent, bee, bittorrent, serialization, torrent
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2004 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
benchmark, merging, normalization, performance, testing
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
bench language, benchmark, performance, testing
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

of benchmarks.
A document written in this language is a Tcl script and has the same
syntax.
.PP
.SS BASICS
One of the most simplest benchmarks which can be written in bench is
.CS


bench -desc LABEL -body {
    set a b
}

.CE
This code declares a benchmark named \fBLABEL\fR which measures the
time it takes to assign a value to a variable. The Tcl code doing this
assignment is the \fB-body\fR of the benchmark.
.SS "PRE- AND POSTPROCESSING"
Our next example demonstrates how to declare \fIinitialization\fR and
\fIcleanup\fR code, i.e. code computing information for the use of
the \fB-body\fR, and for releasing such resources after the
measurement is done.
They are the \fB-pre\fR- and the \fB-post\fR-body, respectively.
.PP
In our example, directly drawn from the benchmark suite of Tcllib's
\fBaes\fR package, the concrete initialization code constructs the
key schedule used by the encryption command whose speed we measure,
and the cleanup code releases any resources bound to that schedule.
.CS


bench -desc "AES-${len} ECB encryption core" \fB-pre\fR {
    set key [aes::Init ecb $k $i]
} -body {
    aes::Encrypt $key $p
} \fB-post\fR {
    aes::Final $key
}

.CE
.SS "ADVANCED PRE- AND POSTPROCESSING"
Our last example again deals with initialization and cleanup code. To
see the difference to the regular initialization and cleanup discussed
in the last section it is necessary to know a bit more about how bench
actually measures the speed of the the \fB-body\fR.
.PP

above to demonstrate the necessity for the advanced initialization and
cleanup. Its concrete initialization code constructs a variable
refering to a set with specific properties (The set has a string
representation, which is shared) affecting the speed of the inclusion
command, and the cleanup code releases the temporary variables created
by this initialization.
.CS


bench -desc "set include, missing <SC> x$times $n" \fB-ipre\fR {
    set A $sx($times,$n)
    set B $A
} -body {
    struct::set include A x
} \fB-ipost\fR {
    unset A B
}

.CE
.SH "FURTHER READING"
Now that this document has been digested the reader, assumed to be a
\fIwriter\fR of benchmarks, he should be fortified enough to be able
to understand the formal \fIbench language specfication\fR. It will
also serve as the detailed specification and cheat sheet for all
available commands and their syntax.

.SH KEYWORDS
bench language, benchmark, examples, performance, testing
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
bench language, benchmark, performance, specification, testing
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
benchmark, csv, formatting, human readable, parsing, performance, reading, testing, text
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
benchmark, csv, formatting, performance, testing
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
benchmark, formatting, human readable, performance, testing, text
.SH CATEGORY
Benchmark tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
bibliography, bibtex, parsing, text processing
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2005 for documentation, Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

vector should be chosen randomly and transmitted as the first block of
the output. Errors in encryption affect the current block and the next
block after which the cipher will correct itself. CBC is the most
commonly used mode in software encryption.
.PP
.SH EXAMPLES
.CS


% blowfish::blowfish -hex -mode ecb -dir encrypt -key secret01 "hello, world!"
d0d8f27e7a374b9e2dbd9938dd04195a

.CE
.CS


 set Key [blowfish::Init cbc $eight_bytes_key_data $eight_byte_iv]
 append ciphertext [blowfish::Encrypt $Key $plaintext]
 append ciphertext [blowfish::Encrypt $Key $additional_plaintext]
 blowfish::Final $Key

.CE
.SH REFERENCES
.IP [1]
Schneier, B. "Applied Cryptography, 2nd edition", 1996,
ISBN 0-471-11709-9, pub. John Wiley & Sons.
.PP
.SH AUTHORS

.SH KEYWORDS
block cipher, blowfish, cryptography, encryption, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2003, Pat Thoyts <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH KEYWORDS
asynchronous, cache, callback, synchronous
.SH COPYRIGHT
.nf
Copyright (c) 2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

This command returns the "sanitized" version of \fIargv0\fR.  It will
strip off the leading path and removes the extension ".bin". The
latter is used by the pro-apps because they must be wrapped by a shell
script.
.PP
.SH EXAMPLES
.CS


        set options {
            {a          "set the atime only"}
            {m          "set the mtime only"}
            {c          "do not create non-existent files"}
            {r.arg  ""  "use time from ref_file"}
            {t.arg  -1  "use specified time"}
        }
        set usage ": MyCommandName \\[options] filename ...\\noptions:"
        array set params [::cmdline::getoptions argv $options $usage]

        if {  $params(a) } { set set_atime "true" }
        set has_t [expr {$params(t) != -1}]
        set has_r [expr {[string length $params(r)] > 0}]
        if {$has_t && $has_r} {
            return -code error "Cannot specify both -r and -t"
        } elseif {$has_t} {
	    ...
        }

.CE
.PP
This example, taken (and slightly modified) from the package
\fBfileutil\fR, shows how to use cmdline.  First, a list of
options is created, then the 'args' list is passed to cmdline for
processing.  Subsequently, different options are checked to see if
they have been passed to the script, and what their value is.

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

result, \fBcomm\fR works with multiple interpreters, works on
Windows and Macintosh systems, and provides control over the remote
execution path.
.PP
These commands work just like \fBsend\fR and \fBwinfo interps\fR :
.PP
.CS


    ::comm::comm send ?-async? id cmd ?arg arg ...?
    ::comm::comm interps

.CE
.PP
This is all that is really needed to know in order to use
\fBcomm\fR
.SS COMMANDS
The package initializes \fB::comm::comm\fR as the default \fIchan\fR.
.PP

particular command, try the same thing with Tk's send and see if the
result is different.  If there is a problem, please report it.  For
instance, there was had one report that this command produced an
error.  Note that the equivalent \fBsend\fR command also produces the
same error.
.PP
.CS


    % ::comm::comm send id llength {a b c}
    wrong # args: should be "llength list"
    % send name llength {a b c}
    wrong # args: should be "llength list"

.CE
.PP
The \fBeval\fR hook (described below) can be used to change from
\fBsend\fR's double eval semantics to single eval semantics.
.SS "MULTIPLE CHANNELS"
.PP
More than one \fBcomm\fR channel (or \fIlistener\fR) can be created

\fB::comm::comm channels\fR
This lists all the channels allocated in this Tcl interpreter.
.PP
.PP
The default configuration parameters for a new channel are:
.PP
.CS


    "-port 0 -local 1 -listen 0 -silent 0"

.CE
.PP
The default channel \fB::comm::comm\fR is created with:
.PP
.CS


    "::comm::comm new ::comm::comm -port 0 -local 1 -listen 1 -silent 0"

.CE
.SS "CHANNEL CONFIGURATION"
.PP
The \fBconfig\fR method acts similar to \fBfconfigure\fR in that it
sets or queries configuration variables associated with a channel.
.TP
\fB::comm::comm config\fR

itself, including closing the listening socket.  Special code allows
the default \fB::comm::comm\fR channel to be closed such that the
\fB::comm::comm\fR command it is not destroyed.  Doing so closes the
listening socket, preventing both incoming and outgoing commands on
the channel.  This sequence reinitializes the default channel:
.sp
.CS


    "::comm::comm destroy; ::comm::comm new ::comm::comm"

.CE
.PP
.PP
When a remote connection is lost (because the remote exited or called
\fBshutdown\fR), \fBcomm\fR can invoke an application callback.
This can be used to cleanup or restart an ancillary process, for
instance.  See the \fIlost\fR callback below.

\fBchan\fR, \fBid\fR
.sp
This hook is invoked before making a connection to the remote named in
\fIid\fR.  An error return (via \fBerror\fR) will abort the connection
attempt with the error.  Example:
.sp
.CS


    % ::comm::comm hook connecting {
        if {[string match {*[02468]} $id]} {
            error "Can't connect to even ids"
        }
    }
    % ::comm::comm send 10000 puts ok
    Connect to remote failed: Can't connect to even ids
    %

.CE
.TP
\fBconnected\fR
Variables:
\fBchan\fR, \fBfid\fR, \fBid\fR, \fBhost\fR, and \fBport\fR.
.sp
This hook is invoked immediately after making a remote connection to

Hook invoked when receiving an incoming connection, allowing arbitrary
authentication over socket named by \fIfid\fR.  An error return (via
\fBerror\fR) will close the connection with the error.  Note that the
peer is named by \fIremport\fR and \fIaddr\fR but that the remote
\fIid\fR is still unknown.  Example:
.sp
.CS


    ::comm::comm hook incoming {
        if {[string match 127.0.0.1 $addr]} {
            error "I don't talk to myself"
        }
    }

.CE
.TP
\fBeval\fR
Variables:
\fBchan\fR, \fBid\fR, \fBcmd\fR, and \fBbuffer\fR.
.sp
This hook is invoked after collecting a complete script from a remote

.sp
Examples:
.RS
.IP [1]
augmenting a command
.sp
.CS


    % ::comm::comm send [::comm::comm self] pid
    5013
    % ::comm::comm hook eval {puts "going to execute $buffer"}
    % ::comm::comm send [::comm::comm self] pid
    going to execute pid
    5013

.CE
.IP [2]
short circuiting a command
.sp
.CS


    % ::comm::comm hook eval {puts "would have executed $buffer"; return 0}
    % ::comm::comm send [::comm::comm self] pid
    would have executed pid
    0

.CE
.IP [3]
Replacing double eval semantics
.sp
.CS


    % ::comm::comm send [::comm::comm self] llength {a b c}
    wrong # args: should be "llength list"
    % ::comm::comm hook eval {return [uplevel #0 $buffer]}
    return [uplevel #0 $buffer]
    % ::comm::comm send [::comm::comm self] llength {a b c}
    3

.CE
.IP [4]
Using a slave interpreter
.sp
.CS


    % interp create foo
    % ::comm::comm hook eval {return [foo eval $buffer]}
    % ::comm::comm send [::comm::comm self] set myvar 123
    123
    % set myvar
    can't read "myvar": no such variable
    % foo eval set myvar
    123

.CE
.IP [5]
Using a slave interpreter (double eval)
.sp
.CS


    % ::comm::comm hook eval {return [eval foo eval $buffer]}

.CE
.IP [6]
Subverting the script to execute
.sp
.CS


    % ::comm::comm hook eval {
        switch -- $buffer {
            a {return A-OK}
            b {return B-OK}
            default {error "$buffer is a no-no"}
        }
    }
    % ::comm::comm send [::comm::comm self] pid
    pid is a no-no
    % ::comm::comm send [::comm::comm self] a
    A-OK

.CE
.RE
.TP
\fBreply\fR
Variables:
\fBchan\fR, \fBid\fR, \fBbuffer\fR, \fBret\fR, and \fBreturn()\fR.
.sp

\fBchan\fR, \fBid\fR, and \fBreason\fR.
.sp
This hook is invoked when the connection to \fBid\fR is lost.  Return
value (or thrown error) is ignored.  \fIreason\fR is an explanatory
string indicating why the connection was lost.  Example:
.sp
.CS


    ::comm::comm hook lost {
        global myvar
        if {$myvar(id) == $id} {
            myfunc
            return
        }
    }

.CE
.PP
.SS UNSUPPORTED
.PP
These interfaces may change or go away in subsequence releases.
.TP
\fB::comm::comm remoteid\fR
Returns the \fIid\fR of the sender of the last remote command
executed on this channel.  If used by a proc being invoked remotely,
it must be called before any events are processed.  Otherwise, another
command may get invoked and change the value.
.TP
\fB::comm::comm_send\fR
Invoking this procedure will substitute the Tk \fBsend\fR and
\fBwinfo interps\fR commands with these equivalents that use
\fB::comm::comm\fR.
.sp
.CS


    proc send {args} {
        eval ::comm::comm send $args
    }
    rename winfo tk_winfo
    proc winfo {cmd args} {
        if {![string match in* $cmd]} {
            return [eval [list tk_winfo $cmd] $args]
        }
        return [::comm::comm interps]
    }

.CE
.PP
.SS SECURITY
Starting with version 4.6 of the package an option \fB-socketcmd\fR
is supported, allowing the user of a comm channel to specify which
command to use when opening a socket. Anything which is API-compatible
with the builtin \fB::socket\fR (the default) can be used.
.PP
The envisioned main use is the specification of the \fBtls::socket\fR
command, see package \fBtls\fR, to secure the communication.
.PP
.CS


	# Load and initialize tls
	package require tls
	tls::init  -cafile /path/to/ca/cert -keyfile ...

	# Create secured comm channel
	::comm::comm new SECURE -socketcmd tls::socket -listen 1
	...

.CE
.PP
The sections \fBExecution Environment\fR and \fBCallbacks\fR
are also relevant to the security of the system, providing means to
restrict the execution to a specific environment, perform additional
authentication, and the like.
.SS "BLOCKING SEMANTICS"

deliver the result it got, but just destroy itself. The future can be
configured with a command to call when the invoker is lost. This
enables the user to implement an early abort of the long-running
operation, should this be supported by it.
.sp
An example:
.CS


# Procedure invoked by remote clients to run database operations.
proc select {sql} {
    # Signal the async generation of the result

    set future [::comm::comm return_async]

    # Generate an async db operation and tell it where to deliver the result.

    set query [db query -command [list $future return] $sql]

    # Tell the database system which query to cancel if the connection
    # goes away while it is running.

    $future configure -command [list db cancel $query]

    # Note: The above will work without problem only if the async
    # query will nover run its completion callback immediately, but
    # only from the eventloop. Because otherwise the future we wish to
    # configure may already be gone. If that is possible use 'catch'
    # to prevent the error from propagating.
    return
}

.CE
.sp
The API of a future object is:
.RS
.TP
\fB$future\fR \fBreturn\fR ?\fB-code\fR \fIcode\fR? ?\fIvalue\fR?
Use this method to tell the future that long-running operation has

\fBcomm\fR exports itself as a package.  The package version number
is in the form \fImajor . minor\fR, where the major version will
only change when a non-compatible change happens to the API or
protocol.  Minor bug fixes and changes will only affect the minor
version.  To load \fBcomm\fR this command is usually used:
.PP
.CS


    package require comm 3

.CE
.PP
Note that requiring no version (or a specific version) can also be done.
.PP
The revision history of \fBcomm\fR includes these releases:
.TP
4.6.2

conditions were not being returned correctly from \fBcomm send\fR.
This has been fixed by removing the extra level of indirection into
the internal procedure \fBcommSend\fR.  Also added propagation of
the \fIerrorCode\fR variable.  This means that these commands return
exactly as they would with \fBsend\fR:
.sp
.CS


    comm send id break
    catch {comm send id break}
    comm send id expr 1 / 0

.CE
.sp
Added a new hook for reply messages.  Reworked method invocation to
avoid the use of comm:* procedures; this also cut the invocation time
down by 40%.  Documented \fBcomm config\fR (as this manual page
still listed the defunct \fBcomm init\fR!)
.TP

.SH "ON USING OLD VERSIONS OF TCL"
.PP
Tcl7.5 under Windows contains a bug that causes the interpreter to
hang when EOF is reached on non-blocking sockets.  This can be
triggered with a command such as this:
.PP
.CS


    "comm send $other exit"

.CE
.PP
Always make sure the channel is quiescent before closing/exiting or
use at least Tcl7.6 under Windows.
.PP
Tcl7.6 on the Mac contains several bugs.  It is recommended you use
at least Tcl7.6p2.

.SH CATEGORY
Programming tools
.SH COPYRIGHT
.nf
Copyright (c) 1995-1998 The Open Group. All Rights Reserved.
Copyright (c) 2003-2004 ActiveState Corporation.
Copyright (c) 2006-2009 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

This emulates the Tcl "eval" semantics.
In most cases it is best to have only one word in the list, a list
containing the exact command.
.sp
Examples:
.sp
.CS


    (a)     {send 1 {{array get tcl_platform}}}
    (b)     {send 1 {array get tcl_platform}}
    (c)     {send 1 {array {get tcl_platform}}}

    are all valid representations of the same command. They are
    generated via

    (a')    send {array get tcl_platform}
    (b')    send array get tcl_platform
    (c')    send array {get tcl_platform}

    respectively

.CE
.sp
Note that (a), generated by (a'), is the usual form, if only single
commands are sent by the client.
For example constructed using \fBlist\fR, if the command contains
variable arguments. Like
.sp
.CS


    send [list array get $the_variable]

.CE
.sp
These three instructions all invoke the script on the server
side. Their difference is in the treatment of result values, and thus
determines if a reply is expected.
.RS
.TP

is highly restricted.
It has to be a syntactically valid Tcl \fBreturn\fR command. This
contains result code, value, error code, and error info.
.sp
Examples:
.sp
.CS


    {reply 1 {return -code 0 {}}}
    {reply 1 {return -code 0 {osVersion 2.4.21-99-default byteOrder littleEndian machine i686 platform unix os Linux user andreask wordSize 4}}}

.CE
.PP
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcomm\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
comm
.SH KEYWORDS
comm, communication, ipc, message, remote communication, remote execution, rpc, socket
.SH CATEGORY
Programming tools
.SH COPYRIGHT
.nf
Copyright (c) 2005 Docs. Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

combination with [\fBnamespace import\fR], it is possible to
use enabled \fBassert\fR commands in some namespaces and disabled
\fBassert\fR commands in other namespaces at the same time.  This
capability is useful so that debugging efforts can be independently
controlled module by module.
.sp
.CS


% package require control
% control::control assert enabled 1
% namespace eval one namespace import ::control::assert
% control::control assert enabled 0
% namespace eval two namespace import ::control::assert
% one::assert {1 == 0}
assertion failed: 1 == 0
% two::assert {1 == 0}

.CE
.TP
\fBcontrol::do\fR \fIbody\fR ?\fIoption test\fR?
The \fBdo\fR command evaluates the script \fIbody\fR repeatedly
\fIuntil\fR the expression \fItest\fR becomes true or as long as
(\fIwhile\fR) \fItest\fR is true, depending on the value of
\fIoption\fR being \fBuntil\fR or \fBwhile\fR. If

arguments for any value of \fI$code\fR other than \fIok\fR.  In this
way, the commands of the \fBcontrol\fR package are limited as compared
to Tcl's built-in control flow commands (such as \fBif\fR,
\fBwhile\fR, etc.) and those control flow commands that can be
provided by packages coded in C.  An example of this difference:
.PP
.CS


% package require control
% proc a {} {while 1 {return -code error a}}
% proc b {} {control::do {return -code error b} while 1}
% catch a
1
% catch b
0

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcontrol\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
after, channel, coroutine, events, exit, gets, global, green threads, read, threads, update, vwait
.SH CATEGORY
Coroutine
.SH COPYRIGHT
.nf
Copyright (c) 2010-2011 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
after, channel, coroutine, events, exit, gets, global, green threads, read, threads, update, vwait
.SH CATEGORY
Coroutine
.SH COPYRIGHT
.nf
Copyright (c) 2010-2011 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

Returns the checksum value and releases any resources held by this
token. Once this command completes the token will be invalid. The
result is a 32 bit integer value.
.PP
.SH EXAMPLES
.PP
.CS


% crc::cksum "Hello, World!"
2609532967

.CE
.PP
.CS


% crc::cksum -format 0x%X "Hello, World!"
0x9B8A5027

.CE
.PP
.CS


% crc::cksum -file cksum.tcl
1828321145

.CE
.PP
.CS


% set tok [crc::CksumInit]
% crc::CksumUpdate $tok "Hello, "
% crc::CksumUpdate $tok "World!"
% crc::CksumFinal $tok
2609532967

.CE
.SH AUTHORS
Pat Thoyts
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcrc\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
crc32(n), sum(n)
.SH KEYWORDS
checksum, cksum, crc, crc32, cyclic redundancy check, data integrity, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.TP
--
Terminate option processing.
.PP
.SH EXAMPLES
.PP
.CS


% crc::crc16 "Hello, World!"
64077

.CE
.PP
.CS


% crc::crc-ccitt "Hello, World!"
26586

.CE
.PP
.CS


% crc::crc16 -format 0x%X "Hello, World!"
0xFA4D

.CE
.PP
.CS


% crc::crc16 -file crc16.tcl
51675

.CE
.SH AUTHORS
Pat Thoyts
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcrc\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
cksum(n), crc32(n), sum(n)
.SH KEYWORDS
checksum, cksum, crc, crc16, crc32, cyclic redundancy check, data integrity, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

Returns the checksum value and releases any resources held by this
token. Once this command completes the token will be invalid. The
result is a 32 bit integer value.
.PP
.SH EXAMPLES
.PP
.CS


% crc::crc32 "Hello, World!"
3964322768

.CE
.PP
.CS


% crc::crc32 -format 0x%X "Hello, World!"
0xEC4AC3D0

.CE
.PP
.CS


% crc::crc32 -file crc32.tcl
483919716

.CE
.PP
.CS


% set tok [crc::Crc32Init]
% crc::Crc32Update $tok "Hello, "
% crc::Crc32Update $tok "World!"
% crc::Crc32Final $tok
3964322768

.CE
.SH AUTHORS
Pat Thoyts
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcrc\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
cksum(n), crc16(n), sum(n)
.SH KEYWORDS
checksum, cksum, crc, crc32, cyclic redundancy check, data integrity, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.TP
-format \fIstring\fR
Return the checksum using an alternative format template.
.PP
.SH EXAMPLES
.PP
.CS


% crc::sum "Hello, World!"
37287





.CE
.PP
.CS


% crc::sum -format 0x%X "Hello, World!"
0x91A7

.CE
.PP
.CS


% crc::sum -file sum.tcl
13392

.CE
.SH AUTHORS
Pat Thoyts
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcrc\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
cksum(n), crc32(n), sum(1)
.SH KEYWORDS
checksum, cksum, crc, crc32, cyclic redundancy check, data integrity, security, sum
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.PP
The alternate format is activated through specification of the option
\fB-alternate\fR to the various split commands.
.SH EXAMPLE
Using the regular format the record
.PP
.CS


123,"123,521.2","Mary says ""Hello, I am Mary""",""

.CE
.PP
is parsed into the items
.PP
.CS


a) 123
b) 123,521.2
c) Mary says "Hello, I am Mary"
d) "

.CE
.PP
Using the alternate format the result is
.PP
.CS


a) 123
b) 123,521.2
c) Mary says "Hello, I am Mary"
d) (the empty string)

.CE
instead. As can be seen only item (d) is different, now the empty string
instead of a ".
.SH "BUGS, IDEAS, FEEDBACK"
This document, and the package it describes, will undoubtedly contain
bugs and other problems.
Please report such in the category \fIcsv\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have for either
package and/or documentation.
.SH "SEE ALSO"
matrix, queue
.SH KEYWORDS
csv, matrix, package, queue, tcllib
.SH CATEGORY
Text processing
.SH COPYRIGHT
.nf
Copyright (c) 2002-2013 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

OFB is similar to CFB except that the output of the cipher is fed back
into the next round and not the xor'd plain text. This means that
errors only affect a single block but the cipher is more vulnerable to
attack.
.PP
.SH EXAMPLES
.CS


% set ciphertext [DES::des -mode cbc -dir encrypt -key $secret $plaintext]
% set plaintext [DES::des -mode cbc -dir decrypt -key $secret $ciphertext]

.CE
.CS


set iv [string repeat \\\\0 8]
set Key [DES::Init cbc \\\\0\\\\1\\\\2\\\\3\\\\4\\\\5\\\\6\\\\7 $iv]
set ciphertext [DES::Encrypt $Key "somedata"]
append ciphertext [DES::Encrypt $Key "moredata"]
DES::Reset $Key $iv
set plaintext [DES::Decrypt $Key $ciphertext]
DES::Final $Key

.CE
.SH REFERENCES
.IP [1]
"Data Encryption Standard",
Federal Information Processing Standards Publication 46-3, 1999,
(\fIhttp://csrc.nist.gov/publications/fips/fips46-3/fips46-3.pdf\fR)
.IP [2]

.SH KEYWORDS
3DES, DES, block cipher, data integrity, encryption, security
.SH CATEGORY
Hashes, checksums, and encryption
.SH COPYRIGHT
.nf
Copyright (c) 2005, Pat Thoyts <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

/etc/resolv.conf file for nameservers (if it exists) and on Windows
systems we examine certain parts of the registry. If no nameserver can
be found then the loopback address (127.0.0.1) is used as a default.
.PP
.SH EXAMPLES
.PP
.CS


% set tok [dns::resolve www.tcl.tk]
::dns::1
% dns::status $tok
ok
% dns::address $tok
199.175.6.239
% dns::name $tok
www.tcl.tk
% dns::cleanup $tok

.CE
.PP
Using DNS URIs as queries:
.CS


% set tok [dns::resolve "dns:tcl.tk;type=MX"]
% set tok [dns::resolve "dns://l.root-servers.net/www.tcl.tk"]

.CE
.PP
Reverse address lookup:
.CS


% set tok [dns::resolve 127.0.0.1]
::dns::1
% dns::name $tok
localhost
% dns::cleanup $tok

.CE
.SH REFERENCES
.IP [1]
Mockapetris, P., "Domain Names - Concepts and Facilities",
RFC 1034, November 1987.
(\fIhttp://www.ietf.org/rfc/rfc1034.txt\fR)
.IP [2]

.SH KEYWORDS
DNS, domain name service, resolver, rfc 1034, rfc 1035, rfc 1886
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2002, Pat Thoyts

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

\fB::ip::prefixToNative\fR \fIprefix\fR
This command converts the string \fIprefix\fR from dotted form
(<ipaddr>/<mask> format) to native (hex) form. Returns a list
containing two elements, ipaddress and mask, in this order, in
hexadecimal notation.
.sp
.CS


   % ip::prefixToNative 1.1.1.0/24
   0x01010100 0xffffff00

.CE
.TP
\fB::ip::nativeToPrefix\fR \fInativeList\fR|\fInative\fR ?\fB-ipv4\fR?
This command converts from native (hex) form to dotted form.
It is the complement of \fB::ip::prefixToNative\fR.
.sp
.RS

.RE
.sp
The command returns a list of addresses in dotted form if it was
called with a list of addresses. Otherwise a single address in dotted
form is returned.
.sp
.CS


   % ip::nativeToPrefix {0x01010100 0xffffff00} -ipv4
   1.1.1.0/24

.CE
.TP
\fB::ip::intToString\fR \fInumber\fR ?\fB-ipv4\fR?
This command converts from an ip address specified as integer number
to dotted form.
.sp
.CS


       ip::intToString 4294967295
       255.255.255.255

.CE
.TP
\fB::ip::toInteger\fR \fIipaddr\fR
This command converts a dotted form ip into an integer number.
.sp
.CS


   % ::ip::toInteger 1.1.1.0
   16843008

.CE
.TP
\fB::ip::toHex\fR \fIipaddr\fR
This command converts dotted form ip into a hexadecimal number.
.sp
.CS


   % ::ip::toHex 1.1.1.0
   0x01010100

.CE
.TP
\fB::ip::maskToInt\fR \fIipmask\fR
This command convert an ipmask in either dotted (255.255.255.0) form
or mask length form (24) into an integer number.
.sp
.CS


   ::ip::maskToInt 24
   4294967040

.CE
.TP
\fB::ip::broadcastAddress\fR \fIprefix\fR ?\fB-ipv4\fR?
This commands returns a broadcast address in dotted form for the given
route \fIprefix\fR, either in the form "addr/mask", or in native
form. The result is in dotted form.
.sp
.CS


   ::ip::broadcastAddress 1.1.1.0/24
   1.1.1.255

   ::ip::broadcastAddress {0x01010100 0xffffff00}
   0x010101ff

.CE
.TP
\fB::ip::maskToLength\fR \fIdottedMask\fR|\fIintegerMask\fR|\fIhexMask\fR ?\fB-ipv4\fR?
This command converts the dotted or integer form of an ipmask to
the mask length form.
.sp
.CS


   ::ip::maskToLength 0xffffff00 -ipv4
   24

   % ::ip::maskToLength 255.255.255.0
   24

.CE
.TP
\fB::ip::lengthToMask\fR \fImaskLength\fR ?\fB-ipv4\fR?
This command converts an ipmask in mask length form to its dotted
form.
.sp
.CS


   ::ip::lengthToMask 24
   255.255.255.0

.CE
.TP
\fB::ip::nextNet\fR \fIipaddr\fR \fIipmask\fR ?\fIcount\fR? ?\fB-ipv4\fR?
This command returns an ipaddress in the same position in the
\fIcount\fR next network. The default value for \fIcount\fR is
\fB1\fR.
.sp
The address can be specified as either integer number or in dotted
form. The mask can be specified as either integer number, dotted form,
or mask length form.
.sp
The result is in hex form.
.TP
\fB::ip::isOverlap\fR \fIprefix\fR \fIprefix\fR...
This command checks if the given ip prefixes overlap.  All arguments
are in dotted "addr/mask" form. All arguments after the first prefix
are compared against the first prefix. The result is a boolean
value. It is true if an overlap was found for any of the prefixes.
.sp
.CS


  % ::ip::isOverlap 1.1.1.0/24 2.1.0.1/32
  0

  ::ip::isOverlap 1.1.1.0/24 2.1.0.1/32 1.1.1.1/32
  1

.CE
.TP
\fB::ip::isOverlapNative\fR ?\fB-all\fR? ?\fB-inline\fR? ?\fB-ipv4\fR? \fIhexipaddr\fR \fIhexipmask\fR \fIhexiplist\fR
This command is similar to \fB::ip::isOverlap\fR, however the
arguments are in the native form, and the form of the result is under
greater control of the caller.
If the option \fB-all\fR is specified it checks all addresses for

.TP
-all -inline
A list containing the prefixes of all overlaps found, or an empty list
if there are none.
.RE
.sp
.CS


  % ::ip::isOverlapNative 0x01010100 0xffffff00 {{0x02010001 0xffffffff}}
  0

  % ::ip::isOverlapNative 0x01010100 0xffffff00 {{0x02010001 0xffffffff} {0x01010101 0xffffffff}}
  2

.CE
.TP
\fB::ip::ipToLayer2Multicast\fR \fIipaddr\fR
This command an converts ipv4 address in dotted form into a layer 2
multicast address, also in dotted form.
.sp
.CS


  % ::ip::ipToLayer2Multicast 224.0.0.2
  01.00.5e.00.00.02

.CE
.TP
\fB::ip::ipHostFromPrefix\fR \fIprefix\fR ?\fB-exclude\fR \fIprefixExcludeList\fR?
This command returns a host address from a prefix in the form
"ipaddr/masklen", also making sure that the result is not an address
found in the \fIprefixExcludeList\fR.
The result is an ip address in dotted form.
.sp
.CS


  %::ip::ipHostFromPrefix  1.1.1.5/24
  1.1.1.1

  %::ip::ipHostFromPrefix  1.1.1.1/32
  1.1.1.1

.CE
.TP
\fB::ip::reduceToAggregates\fR \fIprefixlist\fR
This command finds nets that overlap and filters out the more specific
nets. The prefixes are in either addr/mask form or in native format.
The result is a list containing the non-overlapping ip prefixes from
the input.
.sp
.CS


  % ::ip::reduceToAggregates {1.1.1.0/24 1.1.0.0/8  2.1.1.0/24 1.1.1.1/32 }
  1.0.0.0/8 2.1.1.0/24

.CE
.TP
\fB::ip::longestPrefixMatch\fR \fIipaddr\fR \fIprefixlist\fR ?\fB-ipv4\fR?
This command finds longest prefix match from set of prefixes, given a
specific host address. The prefixes in the list are in either native
or dotted form, whereas the host address is in either ipprefix format,
dotted form, or integer form.
The result is the prefix which is the most specific match to the host
address.
.sp
.CS


  % ::ip::longestPrefixMatch 1.1.1.1 {1.1.1.0/24 1.0.0.0/8  2.1.1.0/24 1.1.1.0/28 }
  1.1.1.0/28

.CE
.TP
\fB::ip::collapse\fR \fIprefixlist\fR
This commands takes a list of prefixes and returns a list prefixes
with the largest possible subnet masks covering the input, in this
manner collapsing adjacent prefixes into larger ranges.
.sp
This is different from \fB::ip::reduceToAggregates\fR in that
the latter only removes specific nets from a list when they are
covered by other elements of the input whereas this command actively
merges nets into larger ranges when they are adjacent to each other.
.sp
.CS


% ::ip::collapse {1.2.2.0/24 1.2.3.0/24}
1.2.2.0/23

.CE
.TP
\fB::ip::subtract\fR \fIprefixlist\fR
This command takes a list of prefixes, some of which are prefixed by a
dash. These latter \fInegative\fR prefixes are used to punch holes
into the ranges described by the other, \fIpositive\fR,
prefixes. I.e. the negative prefixes are subtracted frrom the positive
ones, resulting in a larger list of describes describing the covered
ranges only as positives.
.PP
.SH EXAMPLES
.PP
.CS


% ip::version ::1
6
% ip::version 127.0.0.1
4

.CE
.CS


% ip::normalize 127/8
127.0.0.0/8
% ip::contract 192.168.0.0
192.168
%
% ip::normalize fec0::1
fec0:0000:0000:0000:0000:0000:0000:0001
% ip::contract fec0:0000:0000:0000:0000:0000:0000:0001
fec0::1

.CE
.CS


% ip::equal 192.168.0.4/16 192.168.0.0/16
1
% ip::equal fec0::1/10 fec0::fe01/10
1

.CE
.SH REFERENCES
.IP [1]
Postel, J. "Internet Protocol." RFC 791,  September 1981,
(\fIhttp://www.ietf.org/rfc/rfc791.txt\fR)
.IP [2]
Hinden, R. and Deering, S.,

internet address, ip, ipv4, ipv6, rfc 3513
.SH CATEGORY
Networking
.SH COPYRIGHT
.nf
Copyright (c) 2004, Pat Thoyts
Copyright (c) 2005 Aamer Akhter <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

The basic unit \fBdocstrip\fR operates on are the \fIlines\fR of
a master source file. Extraction consists of selecting some of these
lines to be copied from input text to output text. The basic
distinction is that between \fIcode lines\fR (which are copied and
do not begin with a percent character) and \fIcomment lines\fR
(which begin with a percent character and are not copied).
.CS


   docstrip::extract [join {
     {% comment}
     {% more comment !"#$%&/(}
     {some command}
     { % blah $blah "Not a comment."}
     {% abc; this is comment}
     {# def; this is code}
     {ghi}
     {% jkl}
   } \\n] {}

.CE
returns the same sequence of lines as
.CS


   join {
     {some command}
     { % blah $blah "Not a comment."}
     {# def; this is code}
     {ghi} ""
   } \\n

.CE
It does not matter to \fBdocstrip\fR what format is used for the
documentation in the comment lines, but in order to do better than
plain text comments, one typically uses some markup language. Most
commonly LaTeX is used, as that is a very established standard and
also provides the best support for mathematical formulae, but the
\fBdocstrip::util\fR package also gives some support for

category is by far the most common.
.PP
Ordinary guard lines conditions extraction of the code line(s) they
guard by the value of a boolean expression; the guarded block of
code lines will only be included if the expression evaluates to true.
The syntax of an ordinary guard line is one of
.CS


    '%' '<' STARSLASH EXPRESSION '>'
    '%' '<' PLUSMINUS EXPRESSION '>' CODE

.CE
where
.CS


    STARSLASH  ::=  '*' | '/'
    PLUSMINUS  ::=  | '+' | '-'
    EXPRESSION ::= SECONDARY | SECONDARY ',' EXPRESSION
                 | SECONDARY '|' EXPRESSION
    SECONDARY  ::= PRIMARY | PRIMARY '&' SECONDARY
    PRIMARY    ::= TERMINAL | '!' PRIMARY | '(' EXPRESSION ')'
    CODE       ::= { any character except end-of-line }

.CE
Comma and vertical bar both denote 'or'. Ampersand denotes 'and'.
Exclamation mark denotes 'not'. A TERMINAL can be any nonempty string
of characters not containing '>', '&', '|', comma, '(', or ')',
although the \fBdocstrip\fR manual is a bit restrictive and only
guarantees proper operation for strings of letters (although even
the LaTeX core sources make heavy use also of digits in TERMINALs).
The second argument of \fBdocstrip::extract\fR is the list of those
TERMINALs that should count as having the value 'true'; all other
TERMINALs count as being 'false' when guard expressions are evaluated.
.PP
In the case of a '%<*\fIEXPRESSION\fR>' guard, the lines guarded are
all lines up to the next '%</\fIEXPRESSION\fR>' guard with the same
\fIEXPRESSION\fR (compared as strings). The blocks of code delimited
by such '*' and '/' guard lines must be properly nested.
.CS


   set text [join {
      {begin}
      {%<*foo>}
      {1}
      {%<*bar>}
      {2}
      {%</bar>}
      {%<*!bar>}
      {3}
      {%</!bar>}
      {4}
      {%</foo>}
      {5}
      {%<*bar>}
      {6}
      {%</bar>}
      {end}
   } \\n]
   set res [docstrip::extract $text foo]
   append res [docstrip::extract $text {foo bar}]
   append res [docstrip::extract $text bar]

.CE
sets $res to the result of
.CS


   join {
      {begin}
      {1}
      {3}
      {4}
      {5}
      {end}
      {begin}
      {1}
      {2}
      {4}
      {5}
      {6}
      {end}
      {begin}
      {5}
      {6}
      {end} ""
   } \\n

.CE
In guard lines without a '*', '/', '+', or '-' modifier after the
\'%<', the guard applies only to the CODE following the '>' on that
single line. A '+' modifier is equivalent to no modifier. A '-'
modifier is like the case with no modifier, but the expression is
implicitly negated, i.e., the CODE of a '%<-' guard line is only
included if the expression evaluates to false.
.PP
Metacomment lines are "comment lines which should not be stripped
away", but be extracted like code lines; these are sometimes used for
copyright notices and similar material. The '%%' prefix is however
not kept, but substituted by the current \fB-metaprefix\fR, which
is customarily set to some "comment until end of line" character (or
character sequence) of the language of the code being extracted.
.CS


   set text [join {
      {begin}
      {%<foo> foo}
      {%<+foo>plusfoo}
      {%<-foo>minusfoo}
      {middle}
      {%% some metacomment}
      {%<*foo>}
      {%%another metacomment}
      {%</foo>}
      {end}
   } \\n]
   set res [docstrip::extract $text foo -metaprefix {# }]
   append res [docstrip::extract $text bar -metaprefix {#}]

.CE
sets $res to the result of
.CS


   join {
      {begin}
      { foo}
      {plusfoo}
      {middle}
      {#  some metacomment}
      {# another metacomment}
      {end}
      {begin}
      {minusfoo}
      {middle}
      {# some metacomment}
      {end} ""
   } \\n

.CE
Verbatim guards can be used to force code line
interpretation of a block of lines even if some of them happen to look
like any other type of lines to docstrip. A verbatim guard has the
form '%<<\fIEND-TAG\fR' and the verbatim block is terminated by the
first line that is exactly '%\fIEND-TAG\fR'.
.CS


   set text [join {
      {begin}
      {%<*myblock>}
      {some stupid()}
      {   #computer<program>}
      {%<<QQQ-98765}
      {% These three lines are copied verbatim (including percents}
      {%% even if -metaprefix is something different than %%).}
      {%</myblock>}
      {%QQQ-98765}
      {   using*strange@programming<language>}
      {%</myblock>}
      {end}
   } \\n]
   set res [docstrip::extract $text myblock -metaprefix {# }]
   append res [docstrip::extract $text {}]

.CE
sets $res to the result of
.CS


   join {
      {begin}
      {some stupid()}
      {   #computer<program>}
      {% These three lines are copied verbatim (including percents}
      {%% even if -metaprefix is something different than %%).}
      {%</myblock>}
      {   using*strange@programming<language>}
      {end}
      {begin}
      {end} ""
   } \\n

.CE
The processing of verbatim guards takes place also inside blocks of
lines which due to some outer block guard will not be copied.
.PP
The final piece of \fBdocstrip\fR syntax is that extraction
stops at a line that is exactly "\\endinput"; this is often used to
avoid copying random whitespace at the end of a file. In the unlikely

LaTeX-based "\fI.dtx\fR" files.
.PP
Master source files with "\fI.dtx\fR" extension are usually set up so
that they can be typeset directly by \fBlatex\fR without any
support from other files. This is achieved by beginning the file
with the lines
.CS


   % \\iffalse
   %<*driver>
   \\documentclass{tclldoc}
   \\begin{document}
   \\DocInput{\fIfilename.dtx\fR}
   \\end{document}
   %</driver>
   % \\fi

.CE
or some variation thereof. The trick is that the file gets read twice.
With normal LaTeX reading rules, the first two lines are comments and
therefore ignored. The third line is the document preamble, the fourth
line begins the document body, and the sixth line ends the document,
so LaTeX stops there — non-comments below that point in
the file are never subjected to the normal LaTeX reading rules. Before

It is not necessary to use the tclldoc document class, but that does
provide a number of features that are convenient for "\fI.dtx\fR"
files containing Tcl code. More information on this matter can be
found in the references above.
.SH "SEE ALSO"
docstrip_util
.SH KEYWORDS
.dtx, LaTeX, docstrip, documentation, literate programming, source
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003–2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

This supports both the style of collecting all catalogue lines in one
place and the style of putting each catalogue line in close proximity
of the code that it declares.
.PP
Putting catalogue entries next to the code they declare may look as
follows
.CS


%    First there's the catalogue entry
%    \\begin{tcl}
%<docstrip.tcl::catalogue>pkgProvide foo::bar 1.0 {foobar load}
%    \\end{tcl}
%    second a metacomment used to include a copyright message
%    \\begin{macrocode}
%<*foobar>
%% This file is placed in the public domain.
%    \\end{macrocode}
%    third the package implementation
%    \\begin{tcl}
namespace eval foo::bar {
   # ... some clever piece of Tcl code elided ...
%    \\end{tcl}
%    which at some point may have variant code to make use of a
%    |load|able extension
%    \\begin{tcl}
%<*load>
   load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
   # ... even more clever scripted counterpart of the extension
   # also elided ...
%</!load>
}
%</foobar>
%    \\end{tcl}
%    and that's it!

.CE
The corresponding set-up with \fBpkgIndex\fR would be
.CS


%    First there's the catalogue entry
%    \\begin{tcl}
%<docstrip.tcl::catalogue>pkgIndex foobar load
%    \\end{tcl}
%    second a metacomment used to include a copyright message
%    \\begin{tcl}
%<*foobar>
%% This file is placed in the public domain.
%    \\end{tcl}
%    third the package implementation
%    \\begin{tcl}
package provide foo::bar 1.0
namespace eval foo::bar {
   # ... some clever piece of Tcl code elided ...
%    \\end{tcl}
%    which at some point may have variant code to make use of a
%    |load|able extension
%    \\begin{tcl}
%<*load>
   load [file rootname [info script]][info sharedlibextension]
%</load>
%<*!load>
   # ... even more clever scripted counterpart of the extension
   # also elided ...
%</!load>
}
%</foobar>
%    \\end{tcl}
%    and that's it!

.CE
.TP
\fBdocstrip::util::index_from_catalogue\fR \fIdir\fR \fIpattern\fR ?\fIoption\fR \fIvalue\fR ...?
This command is a sibling of the standard \fBpkg_mkIndex\fR
command, in that it adds package entries to "\fIpkgIndex.tcl\fR"
files. The difference is that it indexes \fBdocstrip\fR-style
source files rather than raw "\fI.tcl\fR" or loadable library files.

An existing file of the same name as one to be created will be
overwritten.
.TP
\fBdocstrip::util::classical_preamble\fR \fImetaprefix\fR \fImessage\fR \fItarget\fR ?\fIsource\fR \fIterminals\fR ...?
This command returns a preamble in the classical
\fBdocstrip\fR style
.CS


##
## This is `TARGET',
## generated by the docstrip::util package.
##
## The original source files were:
##
## SOURCE (with options: `foo,bar')
##
## Some message line 1
## line2
## line3

.CE
.IP
if called as
.CS


docstrip::util::classical_preamble {##}\\
  "\\nSome message line 1\\nline2\\nline3" TARGET SOURCE {foo bar}

.CE
.IP
The command supports preambles for files generated from multiple
sources, even though \fBmodules_from_catalogue\fR at present does
not need that.
.TP
\fBdocstrip::util::classical_postamble\fR \fImetaprefix\fR \fImessage\fR \fItarget\fR ?\fIsource\fR \fIterminals\fR ...?
This command returns a postamble in the classical
\fBdocstrip\fR style
.CS


## Some message line 1
## line2
## line3
##
## End of file `TARGET'.

.CE
.IP
if called as
.CS


docstrip::util::classical_postamble {##}\\
  "Some message line 1\\nline2\\nline3" TARGET SOURCE {foo bar}

.CE
.IP
In other words, the \fIsource\fR and \fIterminals\fR arguments are
ignored, but supported for symmetry with \fBclassical_preamble\fR.
.TP
\fBdocstrip::util::packages_provided\fR \fItext\fR ?\fIsetup-script\fR?
This command returns a list where every even index element is the

context of the \fBpackages_provided\fR procedure just before the
\fItext\fR is processed. At that time, the name of the slave
command for the safe interpreter that will do this processing is
kept in the local variable \fBc\fR. To for example copy the
contents of the \fB::env\fR array to the safe interpreter, one
might use a \fIsetup-script\fR of
.CS

  $c eval [list array set env [array get ::env]]
.CE
.PP
.SH "SOURCE PROCESSING COMMANDS"
Unlike the previous group of commands, which would use
\fBdocstrip::extract\fR to extract some code lines and then process
those further, the following commands operate on text consisting of
all types of lines.

\fBemph\fRasised.
.RE
.IP
At the time of writing, no project has employed \fBdoctools\fR
markup in master source files, so experience of what works well is
not available. A source file could however look as follows
.CS


% [manpage_begin gcd n 1.0]
% [moddesc {Greatest Common Divisor}]
% [require gcd [opt 1.0]]
% [description]
%
% [list_begin definitions]
% [call [cmd gcd] [arg a] [arg b]]
%   The [cmd gcd] procedure takes two arguments [arg a] and [arg b] which
%   must be integers and returns their greatest common divisor.
proc gcd {a b} {
%   The first step is to take the absolute values of the arguments.
%   This relieves us of having to worry about how signs will be treated
%   by the remainder operation.
   set a [expr {abs($a)}]
   set b [expr {abs($b)}]
%   The next line does all of Euclid's algorithm! We can make do
%   without a temporary variable, since $a is substituted before the
%   [lb]set a $b[rb] and thus continues to hold a reference to the
%   "old" value of [var a].
   while {$b>0} { set b [expr { $a % [set a $b] }] }
%   In Tcl 8.3 we might want to use [cmd set] instead of [cmd return]
%   to get the slight advantage of byte-compilation.
%<tcl83>  set a
%<!tcl83>   return $a
}
% [list_end]
%
% [manpage_end]

.CE
.IP
If the above text is fed through \fBdocstrip::util::ddt2man\fR then
the result will be a syntactically correct \fBdoctools\fR
manpage, even though its purpose is a bit different.
.sp
It is suggested that master source code files with \fBdoctools\fR

a comment in the header of each hunk specifies which case is at
hand. It is normally necessary to manually review both the return
value from \fBpatch\fR and the patched text itself, as this command
cannot adjust comment lines to match new content.
.sp
An example use would look like
.CS


set sourceL [split [docstrip::util::thefile from.dtx] \\n]
set terminals {foo bar baz}
set fromtext [docstrip::util::thefile from.tcl]
set difftext [exec diff --unified from.tcl to.tcl]
set leftover [docstrip::util::patch sourceL $terminals $fromtext\\
  [docstrip::util::import_unidiff $difftext] -metaprefix {#}]
set F [open to.dtx w]; puts $F [join $sourceL \\n]; close $F
return $leftover

.CE
.IP
Here, "\fIfrom.dtx\fR" was used as source for "\fIfrom.tcl\fR", which
someone modified into "\fIto.tcl\fR". We're trying to construct a
"\fIto.dtx\fR" which can be used as source for "\fIto.tcl\fR".
.TP
\fBdocstrip::util::thefile\fR \fIfilename\fR ?\fIoption\fR \fIvalue\fR ...?

is \fB-\fR for lines only in the "from" file, \fB+\fR for lines
that are only in the "to" file, and \fB0\fR for lines that are
in both.
.PP
.SH "SEE ALSO"
docstrip, doctools, doctools_fmt
.SH KEYWORDS
.ddt, Tcl module, catalogue, diff, docstrip, doctools, documentation, literate programming, module, package indexing, patch, source
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003–2010 Lars Hellström <Lars dot Hellstrom at residenset dot net>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

describing the date of the entry, its author, and the comments made,
in this order. The last item in each element/entry, the comments, is a
list of sections. Each section is described by a list containing two
elements, a list of file names, and a string containing the true
comment associated with the files of the section.
.sp
.CS


    {
	{
	    date
	    author
	    {
		{
		    {file ...}
		    commenttext
		}
		...
	    }
	}
	{...}
    }

.CE
.TP
\fB::doctools::changelog::flatten\fR \fIentries\fR
This command converts a list of entries as generated by
\fBchange::scan\fR above into a simpler list of plain
text blocks each containing all the information of a
single entry.

.SH KEYWORDS
changelog, doctools, emacs
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003-2013 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
changelog, cvs, cvs log, emacs, log
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003-2008 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
HTML, TMML, conversion, docidx, documentation, index, keyword index, latex, manpage, markup, nroff, wiki
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003-2010 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
index, keyword index, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
docidx commands, docidx language, docidx markup, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
docidx commands, docidx language, docidx markup, docidx syntax, examples, faq, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.PP
Each markup command is a Tcl command surrounded by a matching pair of
\fB[\fR and \fB]\fR. Inside of these delimiters the usual
rules for a Tcl command apply with regard to word quotation, nested
commands, continuation lines, etc. I.e.
.PP
.CS


    ... [key {markup language}] ...

.CE
.CS


  ... [manpage thefile \\\\
          {file description}] ...

.CE
.SS "BASIC STRUCTURE"
The most simple document which can be written in docidx is
.CS


    [index_begin GROUPTITLE TITLE]
    [index_end]

.CE
.PP
Not very useful, but valid. This also shows us that all docidx
documents consist of only one part where we will list all keys and
their references.
.PP
A more useful index will contain at least keywords, or short 'keys',
i.e. the phrases which were indexed. So:
.CS


[index_begin GROUPTITLE TITLE]
[\fBkey markup\fR]
[\fBkey {semantic markup}]\fR]
[\fBkey {docidx markup}\fR]
[\fBkey {docidx language}\fR]
[\fBkey {docidx commands}\fR]
[index_end]

.CE
.PP
In the above example the command \fBkey\fR is used to declare the
keyword phrases we wish to be part of the index.
.PP
However a truly useful index does not only list the keyword phrases,
but will also contain references to documents associated with the
keywords. Here is a made-up index for all the manpages in the module
\fIbase64\fR:
.CS


[index_begin tcllib/base64 {De- & Encoding}]
[key base64]
[\fBmanpage base64\fR]
[key encoding]
[\fBmanpage base64\fR]
[\fBmanpage uuencode\fR]
[\fBmanpage yencode\fR]
[key uuencode]
[\fBmanpage uuencode\fR]
[key yEnc]
[\fBmanpage yencode\fR]
[key ydecode]
[\fBmanpage yencode\fR]
[key yencode]
[\fBmanpage yencode\fR]
[index_end]

.CE
.PP
In the above example the command \fBmanpage\fR is used to insert
references to documents, using symbolic file names, with each command
belonging to the last \fBkey\fR command coming before it.
.PP
The other command to insert references is \fBurl\fR. In contrast to

document.
.PP
Instead of only whitespace the two templating commands \fBinclude\fR
and \fBvset\fR are also allowed, to enable the writer to either set
and/or import configuration settings relevant to the table of
contents. I.e. it is possible to write
.CS


[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
[index_begin GROUPTITLE TITLE]
...
[index_end]

.CE
Even more important, these two commands are allowed anywhere where a
markup command is allowed, without regard for any other
structure.
.CS


[index_begin GROUPTITLE TITLE]
[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
...
[index_end]

.CE
The only restriction \fBinclude\fR has to obey is that the contents of
the included file must be valid at the place of the inclusion. I.e. a
file included before \fBindex_begin\fR may contain only the templating
commands \fBvset\fR and \fBinclude\fR, a file included after a key
may contain only manape or url references, and other keys, etc.
.SS ESCAPES
Beyond the 6 commands shown so far we have two more available.
However their function is not the marking up of index structure, but
the insertion of characters, namely \fB[\fR and \fB]\fR.
These commands, \fBlb\fR and \fBrb\fR respectively, are required
because our use of [ and ] to bracket markup commands makes it
impossible to directly use [ and ] within the text.
.PP
Our example of their use are the sources of the last sentence in the
previous paragraph, with some highlighting added.
.CS


  ...
  These commands, [cmd lb] and [cmd lb] respectively, are required
  because our use of [\fBlb\fR] and [\fBrb\fR] to bracket markup commands makes it
  impossible to directly use [\fBlb\fR] and [\fBrb\fR] within the text.
  ...

.CE
.SH "FURTHER READING"
Now that this document has been digested the reader, assumed to be a
\fIwriter\fR of documentation should be fortified enough to be able
to understand the formal \fIdocidx language syntax\fR
specification as well. From here on out the
\fIdocidx language command reference\fR will also serve as the

.SH KEYWORDS
docidx commands, docidx language, docidx markup, docidx syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2009 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.IP [1]
The construct { X } stands for zero or more occurrences of X.
.IP [2]
The construct [ X ] stands for zero or one occurrence of X.
.PP
The syntax:
.CS


index     = defs
            INDEX_BEGIN
            [ contents ]
            INDEX_END
            { <WHITE> }

defs      = { INCLUDE | VSET | <WHITE> }
contents  = keyword { keyword }

keyword   = defs KEY ref { ref }
ref       = MANPAGE | URL | defs

.CE
At last a rule we were unable to capture in the EBNF syntax, as it is
about the arguments of the markup commands, something which is not
modeled here.
.IP [1]
The arguments of all markup commands have to be plain text, and/or text
markup commands, i.e. one of

.SH KEYWORDS
docidx commands, docidx language, docidx markup, docidx syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2009 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.IP [4]
query and initialize engine parameters
.PP
.PP
After the plugin has been loaded and the frontend commands are
established the commands will be called in the following sequence:
.CS


    idx_numpasses -> n
    idx_listvariables -> vars

    idx_varset var1 value1
    idx_varset var2 value2
    ...
    idx_varset varK valueK
    idx_initialize
    idx_setup 1
    ...
    idx_setup 2
    ...
    ...
    idx_setup n
    ...
    idx_postprocess
    idx_shutdown
    ...

.CE
I.e. first the number of passes and the set of available engine
parameters is established, followed by calls setting the
parameters. That second part is optional.
.PP
After that the plugin is initialized, the specified number of passes
executed, the final result run through a global post processing step

.SH KEYWORDS
formatting engine, index, index formatter, keywords, markup, plugin, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
HTML, TMML, conversion, doctoc, documentation, latex, manpage, markup, nroff, table of contents, toc, wiki
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003-2010 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
markup, semantic markup, table of contents, toc
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
doctoc commands, doctoc language, doctoc markup, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
doctoc commands, doctoc language, doctoc markup, doctoc syntax, examples, faq, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.PP
Each markup command is a Tcl command surrounded by a matching pair of
\fB[\fR and \fB]\fR. Inside of these delimiters the usual
rules for a Tcl command apply with regard to word quotation, nested
commands, continuation lines, etc. I.e.
.PP
.CS


    ... [division_start {Appendix 1}] ...

.CE
.CS


  ... [item thefile \\\\
          label {file description}] ...

.CE
.SS "BASIC STRUCTURE"
The most simple document which can be written in doctoc is
.CS


    [toc_begin GROUPTITLE TITLE]
    [toc_end]

.CE
This also shows us that all doctoc documents consist of only one
part where we will list \fIitems\fR and \fIdivisions\fR.
.PP
The user is free to mix these as she sees fit. This is a change from
version 1 of the language, which did not allow this mixing, but only
the use of either a series of items or a series of divisions.

Symbolic names are used to preserve the convertibility of this format
to any output format. The actual name of any file will be inserted by
the chosen formatting engine when converting the input, based on a
mapping from symbolic to actual names given to the engine.
.PP
Here a made up example for a table of contents of this document:
.CS


[toc_begin Doctoc {Language Introduction}]
[\fBitem 1 DESCRIPTION\fR]
[\fBitem 1.1 {Basic structure}\fR]
[\fBitem 1.2 Items\fR]
[\fBitem 1.3 Divisions\fR]
[\fBitem 2 {FURTHER READING}\fR]
[toc_end]

.CE
.SS DIVISIONS
One thing of notice in the last example in the previous section is
that the referenced sections actually had a nested structure,
something which was expressed in the item labels, by using a common
prefix for all the sections nested under section 1.
.PP

.TP
\fBdivision_end\fR
This command closes the last opened and not yet closed division.
.PP
.PP
Using this we can recast the last example like this
.CS


[toc_begin Doctoc {Language Introduction}]
[\fBdivision_start DESCRIPTION\fR]
[item 1 {Basic structure}]
[item 2 Items]
[item 3 Divisions]
[\fBdivision_end\fR]
[\fBdivision_start {FURTHER READING}\fR]
[\fBdivision_end\fR]
[toc_end]

.CE
.PP
Or, to demonstrate deeper nesting
.CS


[toc_begin Doctoc {Language Introduction}]
[\fBdivision_start DESCRIPTION\fR]
[\fBdivision_start {Basic structure}\fR]
[item 1 Do]
[item 2 Re]
[\fBdivision_end\fR]
[\fBdivision_start Items\fR]
[item a Fi]
[item b Fo]
[item c Fa]
[\fBdivision_end\fR]
[\fBdivision_start Divisions\fR]
[item 1 Sub]
[item 1 Zero]
[\fBdivision_end\fR]
[\fBdivision_end\fR]
[\fBdivision_start {FURTHER READING}\fR]
[\fBdivision_end\fR]
[toc_end]

.CE
And do not forget, it is possible to freely mix items and divisions,
and to have empty divisions.
.CS


[toc_begin Doctoc {Language Introduction}]
[item 1 Do]
[\fBdivision_start DESCRIPTION\fR]
[\fBdivision_start {Basic structure}\fR]
[item 2 Re]
[\fBdivision_end\fR]
[item a Fi]
[\fBdivision_start Items\fR]
[item b Fo]
[item c Fa]
[\fBdivision_end\fR]
[\fBdivision_start Divisions\fR]
[\fBdivision_end\fR]
[\fBdivision_end\fR]
[\fBdivision_start {FURTHER READING}\fR]
[\fBdivision_end\fR]
[toc_end]

.CE
.SS "ADVANCED STRUCTURE"
In all previous examples we fudged a bit regarding the markup actually
allowed to be used before the \fBtoc_begin\fR command opening the
document.
.PP
Instead of only whitespace the two templating commands \fBinclude\fR
and \fBvset\fR are also allowed, to enable the writer to either set
and/or import configuration settings relevant to the table of
contents. I.e. it is possible to write
.CS


[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
[toc_begin GROUPTITLE TITLE]
...
[toc_end]

.CE
Even more important, these two commands are allowed anywhere where a
markup command is allowed, without regard for any other
structure.
.CS


[toc_begin GROUPTITLE TITLE]
[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
...
[toc_end]

.CE
The only restriction \fBinclude\fR has to obey is that the contents of
the included file must be valid at the place of the inclusion. I.e. a
file included before \fBtoc_begin\fR may contain only the templating
commands \fBvset\fR and \fBinclude\fR, a file included in a division
may contain only items or divisions commands, etc.
.SS ESCAPES
Beyond the 6 commands shown so far we have two more available.
However their function is not the marking up of toc structure, but the
insertion of characters, namely \fB[\fR and \fB]\fR.
These commands, \fBlb\fR and \fBrb\fR respectively, are required
because our use of [ and ] to bracket markup commands makes it
impossible to directly use [ and ] within the text.
.PP
Our example of their use are the sources of the last sentence in the
previous paragraph, with some highlighting added.
.CS


  ...
  These commands, [cmd lb] and [cmd lb] respectively, are required
  because our use of [\fBlb\fR] and [\fBrb\fR] to bracket markup commands makes it
  impossible to directly use [\fBlb\fR] and [\fBrb\fR] within the text.
  ...

.CE
.SH "FURTHER READING"
Now that this document has been digested the reader, assumed to be a
\fIwriter\fR of documentation should be fortified enough to be able
to understand the formal \fIdoctoc language syntax\fR
specification as well. From here on out the
\fIdoctoc language command reference\fR will also serve as the

.SH KEYWORDS
doctoc commands, doctoc language, doctoc markup, doctoc syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.IP [1]
The construct { X } stands for zero or more occurrences of X.
.IP [2]
The construct [ X ] stands for zero or one occurrence of X.
.PP
The syntax:
.CS


toc       = defs
            TOC_BEGIN
            contents
            TOC_END
            { <WHITE> }

defs      = { INCLUDE | VSET | <WHITE> }
contents  = { defs entry } [ defs ]

entry     = ITEM | division

division  = DIVISION_START
            contents
            DIVISION_END

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, will undoubtedly contain bugs and other problems.
Please report such in the category \fIdoctools\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have.
.SH "SEE ALSO"
doctoc_intro, doctoc_lang_cmdref, doctoc_lang_faq, doctoc_lang_intro
.SH KEYWORDS
doctoc commands, doctoc language, doctoc markup, doctoc syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2009 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.IP [4]
query and initialize engine parameters
.PP
.PP
After the plugin has been loaded and the frontend commands are
established the commands will be called in the following sequence:
.CS


    toc_numpasses -> n
    toc_listvariables -> vars

    toc_varset var1 value1
    toc_varset var2 value2
    ...
    toc_varset varK valueK
    toc_initialize
    toc_setup 1
    ...
    toc_setup 2
    ...
    ...
    toc_setup n
    ...
    toc_postprocess
    toc_shutdown
    ...

.CE
I.e. first the number of passes and the set of available engine
parameters is established, followed by calls setting the
parameters. That second part is optional.
.PP
After that the plugin is initialized, the specified number of passes
executed, the final result run through a global post processing step

.SH KEYWORDS
formatting engine, markup, plugin, semantic markup, table of contents, toc, toc formatter
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
HTML, TMML, conversion, documentation, manpage, markup, nroff
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2003-2013 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
doctools commands, doctools language, doctools markup, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2010 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.SH KEYWORDS
doctools commands, doctools language, doctools markup, doctools syntax, examples, faq, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.PP
Each markup command is a Tcl command surrounded by a matching pair of
\fB[\fR and \fB]\fR. Inside of these delimiters the usual
rules for a Tcl command apply with regard to word quotation, nested
commands, continuation lines, etc. I.e.
.PP
.CS


  ... [list_begin enumerated] ...

.CE
.CS


  ... [call [cmd foo] \\\\
          [arg bar]] ...

.CE
.CS


  ... [term {complex concept}] ...

.CE
.CS


  ... [opt "[arg key] [arg value]"] ...

.CE
.SS "BASIC STRUCTURE"
The most simple document which can be written in doctools is
.CS


    [manpage_begin NAME SECTION VERSION]
    [description]
    [manpage_end]

.CE
This also shows us that all doctools documents are split into two
parts, the \fIheader\fR and the \fIbody\fR. Everything coming before
[\fBdescription\fR] belongs to the header, and everything coming
after belongs to the body, with the whole document bracketed by the
two \fBmanpage_*\fR commands. Before and after these opening and
closing commands we have only \fIwhitespace\fR.

any order.
However for \fBtitledesc\fR and \fBmoddesc\fR only the last occurrence
is taken. For the other two the specified information is accumulated,
in the given order. Regular text is not allowed within the header.
.PP
Given the above a less minimal example of a document is
.CS


[manpage_begin NAME SECTION VERSION]
[\fBcopyright {YEAR AUTHOR}\fR]
[\fBtitledesc TITLE\fR]
[\fBmoddesc   MODULE_TITLE\fR]
[\fBrequire   PACKAGE VERSION\fR]
[\fBrequire   PACKAGE\fR]
[description]
[manpage_end]

.CE
Remember that the whitespace is optional. The document
.CS


    [manpage_begin NAME SECTION VERSION]
    [copyright {YEAR AUTHOR}][titledesc TITLE][moddesc MODULE_TITLE]
    [require PACKAGE VERSION][require PACKAGE][description]
    [manpage_end]

.CE
has the same meaning as the example before.
.PP
On the other hand, if \fIwhitespace\fR is present it consists not
only of any sequence of characters containing the space character,
horizontal and vertical tabs, carriage return, and newline, but it may
contain comment markup as well, in the form of the \fBcomment\fR
command.
.CS


[\fBcomment { ... }\fR]
[manpage_begin NAME SECTION VERSION]
[copyright {YEAR AUTHOR}]
[titledesc TITLE]
[moddesc   MODULE_TITLE][\fBcomment { ... }\fR]
[require   PACKAGE VERSION]
[require   PACKAGE]
[description]
[manpage_end]
[\fBcomment { ... }\fR]

.CE
.SS "ADVANCED STRUCTURE"
In the simple examples of the last section we fudged a bit regarding
the markup actually allowed to be used before the \fBmanpage_begin\fR
command opening the document.
.PP
Instead of only whitespace the two templating commands \fBinclude\fR
and \fBvset\fR are also allowed, to enable the writer to either set
and/or import configuration settings relevant to the document. I.e. it
is possible to write
.CS


[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
[manpage_begin NAME SECTION VERSION]
[description]
[manpage_end]

.CE
Even more important, these two commands are allowed anywhere where a
markup command is allowed, without regard for any other
structure. I.e. for example in the header as well.
.CS


[manpage_begin NAME SECTION VERSION]
[\fBinclude FILE\fR]
[\fBvset VAR VALUE\fR]
[description]
[manpage_end]

.CE
The only restriction \fBinclude\fR has to obey is that the contents of
the included file must be valid at the place of the inclusion. I.e. a
file included before \fBmanpage_begin\fR may contain only the
templating commands \fBvset\fR and \fBinclude\fR, a file included in
the header may contain only header commands, etc.
.SS "TEXT STRUCTURE"

The simplest way of structuring the body is through the introduction
of paragraphs. The command for doing so is \fBpara\fR. Each occurrence
of this command closes the previous paragraph and automatically opens
the next. The first paragraph is automatically opened at the beginning
of the body, by \fBdescription\fR. In the same manner the last
paragraph automatically ends at \fBmanpage_end\fR.
.CS


[manpage_begin NAME SECTION VERSION]
[description]
 ...
[\fBpara\fR]
 ...
[\fBpara\fR]
 ...
[manpage_end]

.CE
Empty paragraphs are ignored.
.PP
A structure coarser than paragraphs are sections, which allow the
writer to split a document into larger, and labeled, pieces. The
command for doing so is \fBsection\fR. Each occurrence of this command
closes the previous section and automatically opens the next,
including its first paragraph. The first section is automatically
opened at the beginning of the body, by \fBdescription\fR (This
section is labeled "DESCRIPTION"). In the same manner the last section
automatically ends at \fBmanpage_end\fR.
.PP
Empty sections are \fInot\fR ignored. We are free to (not) use
paragraphs within sections.
.CS


[manpage_begin NAME SECTION VERSION]
[description]
 ...
[\fBsection {Section A}\fR]
 ...
[para]
 ...
[\fBsection {Section B}\fR]
 ...
[manpage_end]

.CE
Between sections and paragraphs we have subsections, to split sections.
The command for doing so is \fBsubsection\fR. Each occurrence of this
command closes the previous subsection and automatically opens the
next, including its first paragraph. A subsection is automatically
opened at the beginning of the body, by \fBdescription\fR, and at the
beginning of each section. In the same manner the last subsection
automatically ends at \fBmanpage_end\fR.
.PP
Empty subsections are \fInot\fR ignored. We are free to (not) use
paragraphs within subsections.
.CS


[manpage_begin NAME SECTION VERSION]
[description]
 ...
[section {Section A}]
 ...
[\fBsubsection {Sub 1}\fR]
 ...
[para]
 ...
[\fBsubsection {Sub 2}\fR]
 ...
[section {Section B}]
 ...
[manpage_end]

.CE
.SS "TEXT MARKUP"
Having handled the overall structure a writer can impose on the
document we now take a closer at the text in a paragraph.
.PP
While most often this is just the unadorned content of the document we
do have situations where we wish to highlight parts of it as some type

.PP
The example demonstrating the use of text markup is an excerpt from
the \fIdoctools language command reference\fR, with some
highlighting added.
It shows their use within a block of text, as the arguments of a list
item command (\fBcall\fR), and our ability to nest them.
.CS


  ...
  [call [\fBcmd arg_def\fR] [\fBarg type\fR] [\fBarg name\fR]] [\fBopt\fR [\fBarg mode\fR]]]

  Text structure. List element. Argument list. Automatically closes the
  previous list element. Specifies the data-[\fBarg type\fR] of the described
  argument of a command, its [\fBarg name\fR] and its i/o-[\fBarg mode\fR]. The
  latter is optional.
  ...

.CE
.SS ESCAPES
Beyond the 20 commands for simple markup shown in the previous section
we have two more available which are technically simple markup.
However their function is not the marking up of phrases as specific
types of things, but the insertion of characters, namely \fB[\fR
and \fB]\fR.
These commands, \fBlb\fR and \fBrb\fR respectively, are required
because our use of [ and ] to bracket markup commands makes it
impossible to directly use [ and ] within the text.
.PP
Our example of their use are the sources of the last sentence in the
previous paragraph, with some highlighting added.
.CS


  ...
  These commands, [cmd lb] and [cmd lb] respectively, are required
  because our use of [\fBlb\fR] and [\fBrb\fR] to bracket markup commands makes it
  impossible to directly use [\fBlb\fR] and [\fBrb\fR] within the text.
  ...

.CE
.SS CROSS-REFERENCES
The last two commands we have to discuss are for the declaration of
cross-references between documents, explicit and implicit. They are
\fBkeywords\fR and \fBsee_also\fR. Both take an arbitrary number of
arguments, all of which have to be plain unmarked text. I.e. it is not
allowed to use markup on them. Both commands can be used multiple

defined by the main content, or considers them as meta data which
should be in the header, etc.
.PP
Our example shows the sources for the cross-references of this
document, with some highlighting added. Incidentally they are found
at the end of the body.
.CS


  ...
  [\fBsee_also doctools_intro\fR]
  [\fBsee_also doctools_lang_syntax\fR]
  [\fBsee_also doctools_lang_cmdref\fR]
  [\fBkeywords markup {semantic markup}\fR]
  [\fBkeywords {doctools markup} {doctools language}\fR]
  [\fBkeywords {doctools syntax} {doctools commands}\fR]
  [manpage_end]

.CE
.SS EXAMPLES
Where ever we can write plain text we can write examples too. For
simple examples we have the command \fBexample\fR which takes a single
argument, the text of the argument. The example text must not contain
markup. If we wish to have markup within an example we have to use the
2-command combination \fBexample_begin\fR / \fBexample_end\fR instead.
.PP
The first opens an example block, the other closes it, and in between
we can write plain text and use all the regular text markup commands.
Note that text structure commands are not allowed. This also means
that it is not possible to embed examples and lists within an example.
On the other hand, we \fIcan\fR use templating commands within
example blocks to read their contents from a file (Remember section
\fBAdvanced structure\fR).
.PP
The source for the very first example in this document (see section
\fBFundamentals\fR), with some highlighting added, is
.CS


  [\fBexample\fR {
    ... [list_begin enumerated] ...
  }]

.CE
Using \fBexample_begin\fR / \fBexample_end\fR this would look like
.CS


  [\fBexample_begin\fR]
    ... [list_begin enumerated] ...
  [\fBexample_end\fR]

.CE
.SS LISTS
Where ever we can write plain text we can write lists too. The main
commands are \fBlist_begin\fR to start a list, and \fBlist_end\fR to
close one. The opening command takes an argument specifying the type
of list started it, and this in turn determines which of the eight
existing list item commands are allowed within the list to start list

\fIwidget option (declaration) list\fR. It is a specialized form of
a term definition list where the term is the name of a configuration
option for a widget, with its name and class in the option database.
.PP
Our example is the source of the definition list in the previous
paragraph, with most of the content in the middle removed.
.CS


  ...
  [\fBlist_begin\fR definitions]
  [\fBdef\fR [const arg]]

  ([cmd arg_def]) This opens an argument (declaration) list. It is a
  specialized form of a definition list where the term is an argument
  name, with its type and i/o-mode.

  [\fBdef\fR [const itemized]]

  ([cmd item])
  This opens a general itemized list.

  ...
  [\fBdef\fR [const tkoption]]

  ([cmd tkoption_def]) This opens a widget option (declaration) list. It
  is a specialized form of a definition list where the term is the name
  of a configuration option for a widget, with its name and class in the
  option database.

  [\fBlist_end\fR]
  ...

.CE
Note that a list cannot begin in one (sub)section and end in
another. Differently said, (sub)section breaks are not allowed within
lists and list items. An example of this \fIillegal\fR construct is
.CS


  ...
  [list_begin itemized]
  [item]
  ...
  [\fBsection {ILLEGAL WITHIN THE LIST}\fR]
  ...
  [list_end]
  ...

.CE
.SH "FURTHER READING"
Now that this document has been digested the reader, assumed to be a
\fIwriter\fR of documentation should be fortified enough to be able
to understand the formal \fIdoctools language syntax\fR
specification as well. From here on out the
\fIdoctools language command reference\fR will also serve as the

.SH KEYWORDS
doctools commands, doctools language, doctools markup, doctools syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

The construct [ X ] stands for zero or one occurrence of X.
.IP [3]
The construct LIST_BEGIN<X> stands for the markup command
\fBlist_begin\fR with \fBX\fR as its type argument.
.PP
The syntax:
.CS


manpage = defs
          MANPAGE_BEGIN
          header
          DESCRIPTION
          body
          MANPAGE_END
          { <WHITE> }

defs    = { INCLUDE | VSET | <WHITE> }

header  = { TITLEDESC | MODDESC | COPYRIGHT | REQUIRE | defs | xref }

xref    = KEYWORDS | SEE_ALSO | CATEGORY

body    = paras { SECTION    sbody  }
sbody   = paras { SUBSECTION ssbody }
ssbody  = paras

paras   = tblock { (PARA | NL) tblock }

tblock  = { <TEXT> | defs | markup | xref | an_example | a_list }

markup  = ARG     | CLASS | CMD     | CONST     | EMPH   | FILE
        | FUN     | LB    | METHOD  | NAMESPACE | OPT    | OPTION
        | PACKAGE | RB    | SECTREF | STRONG    | SYSCMD | TERM
        | TYPE    | URI   | USAGE   | VAR       | WIDGET

example = EXAMPLE
        | EXAMPLE_BEGIN extext EXAMPLE_END

extext  = { <TEXT> | defs | markup }

a_list  = LIST_BEGIN<arguments>   argd_list   LIST_END
        | LIST_BEGIN<commands>    cmdd_list   LIST_END
        | LIST_BEGIN<definitions> def_list    LIST_END
        | LIST_BEGIN<enumerated>  enum_list   LIST_END
        | LIST_BEGIN<itemized>    item_list   LIST_END
        | LIST_BEGIN<options>     optd_list   LIST_END
        | LIST_BEGIN<tkoptions>   tkoptd_list LIST_END

argd_list   = [ <WHITE> ] { ARG_DEF      paras }
cmdd_list   = [ <WHITE> ] { CMD_DEF      paras }
def_list    = [ <WHITE> ] { (DEF|CALL)   paras }
enum_list   = [ <WHITE> ] { ENUM         paras }
item_list   = [ <WHITE> ] { ITEM         paras }
optd_list   = [ <WHITE> ] { OPT_DEF      paras }
tkoptd_list = [ <WHITE> ] { TKOPTION_DEF paras }

.CE
.SH "BUGS, IDEAS, FEEDBACK"
This document, will undoubtedly contain bugs and other problems.
Please report such in the category \fIdoctools\fR of the
\fITcllib SF Trackers\fR [http://sourceforge.net/tracker/?group_id=12883].
Please also report any ideas for enhancements you may have.
.SH "SEE ALSO"
doctools_intro, doctools_lang_cmdref, doctools_lang_faq, doctools_lang_intro
.SH KEYWORDS
doctools commands, doctools language, doctools markup, doctools syntax, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

.IP [4]
query and initialize engine parameters
.PP
.PP
After the plugin has been loaded and the frontend commands are
established the commands will be called in the following sequence:
.CS


    fmt_numpasses -> n
    fmt_listvariables -> vars

    fmt_varset var1 value1
    fmt_varset var2 value2
    ...
    fmt_varset varK valueK
    fmt_initialize
    fmt_setup 1
    ...
    fmt_setup 2
    ...
    ...
    fmt_setup n
    ...
    fmt_postprocess
    fmt_shutdown
    ...

.CE
I.e. first the number of passes and the set of available engine
parameters is established, followed by calls setting the
parameters. That second part is optional.
.PP
After that the plugin is initialized, the specified number of passes
executed, the final result run through a global post processing step

.SH KEYWORDS
document, formatter, formatting engine, manpage, markup, semantic markup
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2007-2010 Andreas Kupries <[email protected]>

.fi

.LP
.nf
.ta 4c
Command-Line Name:	\\fB\\$1\\fR
Database Name:	\\fB\\$2\\fR
Database Class:	\\fB\\$3\\fR
.fi
.IP
..
'\"	# CS - begin code excerpt
.de CS
.RS
.nf
.ta .25i .5i .75i 1i
..

HTML, TMML, conversion, manpage, markup, nroff
.SH CATEGORY
Documentation tools
.SH COPYRIGHT
.nf
Copyright (c) 2002 Andreas Kupries <[email protected]>
Copyright (c) 2003 Andreas Kupries <[email protected]>

.fi