Tcl Library Source Code

  - \[2\]

    The following directory structure is created when processing a single set of
    input documents\. The file extension used is for output in HTML, but that is
    not relevant to the structure and was just used to have proper file names\.

        output/
            toc\.html
            index\.html
            files/
                path/to/FOO\.html

    The last line in the example shows the document generated for a file FOO
    located at

        inputdirectory/path/to/FOO

  - \[3\]

    When merging many packages into a unified set of documents the generated
    directory structure is a bit deeper:

        output
            \.toc
            \.idx
            \.tocdoc
            \.idxdoc
            \.xrf
            toc\.html
            index\.html
            FOO1/
                \.\.\.
            FOO2/
                toc\.html
                files/
                    path/to/BAR\.html

    Each of the directories FOO1, \.\.\. contains the documents generated for the
    package FOO1, \.\.\. and follows the structure shown for use case \[2\]\. The only
    exception is that there is no per\-package index\.

    The files "\.toc", "\.idx", and "\.xrf" contain the internal status of the
    whole output and will be read and updated by the next invokation\. Their

the plugin option they are associated with does not understand them, and was not
superceded by a plugin option coming after\.

Default options are used if and only if the command line did not contain any
options at all\. They will set the application up as a PEG\-based parser
generator\. The exact list of options is

    \-c peg

And now the recognized options and their arguments, if they have any:

  - __\-\-help__

  - __\-h__

      * *peg*

        It sets the application up as a parser generator accepting parsing
        expression grammars and writing a packrat parser in Tcl\. The actual
        arguments it specifies are:

    \-\-reset
    \-\-append
    \-\-reader    peg
    \-\-transform reach
    \-\-transform use
    \-\-writer    me

  - __\-r__ *name*

    Readers\. The name of the package for the plugin *name* is
    "page::reader::*name*"\.

    We have five predefined plugins:

In this section we are working a complete example, starting with a PEG grammar
and ending with running the parser generated from it over some input, following
the outline shown in the figure below:

![](\.\./\.\./\.\./image/flow\.png) Our grammar, assumed to the stored in the file
"calculator\.peg" is

    PEG calculator \(Expression\)
        Digit      <\- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
        Sign       <\- '\-' / '\+'                                     ;
        Number     <\- Sign? Digit\+                                  ;
        Expression <\- Term \(AddOp Term\)\*                            ;
        MulOp      <\- '\*' / '/'                                     ;
        Term       <\- Factor \(MulOp Factor\)\*                        ;
        AddOp      <\- '\+'/'\-'                                       ;
        Factor     <\- '\(' Expression '\)' / Number                   ;
    END;

From this we create a snit\-based parser via

    pt generate snit calculator\.tcl \-class calculator \-name calculator peg calculator\.peg

which leaves us with the parser package and class written to the file
"calculator\.tcl"\. Assuming that this package is then properly installed in a
place where Tcl can find it we can now use this class via a script like

    package require calculator

    lassign $argv input
    set channel \[open $input r\]

    set parser \[calculator\]
    set ast \[$parser parse $channel\]
    $parser destroy
    close $channel

    \.\.\. now process the returned abstract syntax tree \.\.\.

where the abstract syntax tree stored in the variable will look like

    set ast \{Expression 0 4
        \{Factor 0 4
            \{Term 0 2
                \{Number 0 2
                    \{Digit 0 0\}
                    \{Digit 1 1\}
                    \{Digit 2 2\}
                \}
            \}


            \{AddOp 3 3\}
            \{Term 4 4
                \{Number 4 4
                    \{Digit 4 4\}
                \}
            \}
        \}
    \}





assuming that the input file and channel contained the text

    120\+5

A more graphical representation of the tree would be

![](\.\./\.\./\.\./image/expr\_ast\.png) Regardless, at this point it is the user's
responsibility to work with the tree to reach whatever goal she desires\. I\.e\.
analyze it, transform it, etc\. The package
__[pt::ast](\.\./modules/pt/pt\_astree\.md)__ should be of help here,

    If some other branch is desired as the starting point for the coming work
    replace *trunk* in the commands above with the name of that branch\.

    With the base line established we now have two ways of creating the new
    branch, with differing \(dis\)advantages\. The simpler way is to

        fossil branch new NAME\_OF\_NEW\_BRANCH

    and start developing\. The advantage here is that you cannot forget to create
    the branch\. The disadvantages are that we have a branch commit unchanged
    from where we branched from, and that we have to use high\-handed techniques
    like hiding or shunning to get rid of the commit should we decide to abandon
    the work before the first actual commit on the branch\.

    The other way of creating the branch is to start developing, and then on the
    first commit use the option __\-\-branch__ to tell __fossil__ that we
    are starting a branch now\. I\.e\. run

        fossil commit \-\-branch NAME\_OF\_NEW\_BRANCH \.\.\.

    where *\.\.\.* are any other options used to supply the commit message, files
    to commit, etc\.

    The \(dis\)advantages are now reversed\.

    We have no superflous commit, only what is actually developed\. The work is

      1. Validate the checkout again\. The incoming trunk changes may have broken
         something now\. Do any required fixes\.

      1. Now merge to the trunk using

             fossil update trunk
             fossil merge \-\-integrate YOU\_BRANCH

      1. At this point the checkout should be in the same state as at the end of
         point \(3\) above, because we resolved any issues with the trunk already\.
         Thus a simple

             fossil commit \.\.\.

         should be sufficient now to commit the merge back and close the branch
         \(due to the __\-\-integrate__ we used on the merge\)\.

         The more paranoid may validate the checkout a third time before
         commiting\.

    Even if __fossil__ does not report any conflicts it is a good idea to
    check that the operation has not broken the new and/or changed functionality
    we are working on\.

    With the establishment of a good merge we then save the state with

        fossil commit \.\.\.

    before continuing development\.

## <a name='subsection7'></a>Version numbers

In Tcllib all changes to a package have to come with an increment of its version
number\. What part is incremented \(patchlevel, minor, major version\) depends on

  1. The package documentation\.

The "sak\.tcl" command __validate version__ helps finding discrepancies
between the first two\. All the other __validate__ methods are also of
interest to any developer\. Invoke it with

    sak\.tcl help validate

to see their documentation\.

# <a name='section4'></a>Structural Overview

## <a name='subsection8'></a>Main Directories

providing management operations, for example setting a list of standard Tcl
shells to use\.

## <a name='subsection12'></a>Invoke the testsuites of a specific module

Invoke either

    \./sak\.tcl test run foo

or

    \./sak\.tcl test run modules/foo

to invoke the testsuites found in a specific module "foo"\.

## <a name='subsection13'></a>Invoke the testsuites of all modules

Invoke the tool without a module name, i\.e\.

    \./sak\.tcl test run

to invoke the testsuites of all modules\.

## <a name='subsection14'></a>Detailed Test Logs

In all the previous examples the test runner will write a combination of
progress display and testsuite log to the standard output, showing for each
module only the tests that passed or failed and how many of each in a summary at
the end\.

To get a detailed log, it is necessary to invoke the test runner with additional
options\.

For one:

    \./sak\.tcl test run \-\-log LOG foo

While this shows the same short log on the terminal as before, it also writes a
detailed log to the file "LOG\.log", and excerpts to other files \("LOG\.summary",
"LOG\.failures", etc\.\)\.

For two:

    \./sak\.tcl test run \-v foo

This writes the detailed log to the standard output, instead of the short log\.

Regardless of form, the detailed log contains a list of all test cases executed,
which failed, and how they failed \(expected versus actual results\)\.

## <a name='subsection15'></a>Shell Selection

By default the test runner will use all the Tcl shells specified via __test
add__ to invoke the specified testsuites, if any\. If no such are specified it
will fall back to the Tcl shell used to run the tool itself\.

Use option __\-\-shell__ to explicitly specify the Tcl shell to use, like

    \./sak\.tcl test run \-\-shell /path/to/tclsh \.\.\.

## <a name='subsection16'></a>Help

Invoke the tool as

    \./sak\.tcl help test

to see the detailed help for all methods of
__[test](\.\./\.\./\.\./index\.md\#test)__, and the associated options\.

# <a name='section6'></a>Documentation Tooling

The standard format used for documentation of packages and other things in
Tcllib is *[doctools](\.\./\.\./\.\./index\.md\#doctools)*\. Its supporting
packages are a part of Tcllib, see the directories "modules/doctools" and
"modules/dtplite"\. The latter is an application package, with the actual
application "apps/dtplite" a light wrapper around it\.

Tcllib developers gain access to these through the __doc__ method of the
"sak\.tcl" tool, another \(internal\) wrapper around the "modules/dtplite"
application package\.

## <a name='subsection17'></a>Generate documentation for a specific module

Invoke either

    \./sak\.tcl doc html foo

or

    \./sak\.tcl doc html modules/foo

to generate HTML for the documentation found in the module "foo"\. Instead of
__html__ any other supported format can be used here, of course\.

The generated formatted documentation will be placed into a directory "doc" in
the current working directory\.

## <a name='subsection18'></a>Generate documentation for all modules

Invoke the tool without a module name, i\.e\.

    \./sak\.tcl doc html

to generate HTML for the documentation found in all modules\. Instead of
__html__ any other supported format can be used here, of course\.

The generated formatted documentation will be placed into a directory "doc" in
the current working directory\.

## <a name='subsection19'></a>Available output formats, help

Invoke the tool as

    \./sak\.tcl help doc

to see the entire set of supported output formats which can be generated\.

## <a name='subsection20'></a>Validation without output

Note the special format __validate__\.

Using this value as the name of the format to generate forces the tool to simply
check that the documentation is syntactically correct, without generating actual
output\.

Invoke it as either

    \./sak\.tcl doc validate \(modules/\)foo

or

    \./sak\.tcl doc validate

to either check the packages of a specific module or check all of them\.

# <a name='section7'></a>Notes On Writing A Testsuite

While previous sections talked about running the testsuites for a module and the
packages therein, this has no meaning if the module in question has no

\(*Windows*\)\.

## <a name='subsection3'></a>Installing on Unix

For *Unix*\-like environments Tcllib comes with the standard set of files to
make

    \./configure
    make install

a suitable way of installing it\. This is a standard non\-interactive install
automatically figuring out where to place everything, i\.e\. packages,
applications, and the manpages\.

To get a graphical installer invoke

    \./installer\.tcl

instead\.

## <a name='subsection4'></a>Installing on Windows

In a Windows environment we have the __installer\.tcl__ script to perform
installation\.

If the desired __tclsh__ is associated "\.tcl" files then double\-clicking /
opening the __installer\.tcl__ is enough to invoke it in graphical mode\. This
assumes that *[Tk](\.\./\.\./\.\./index\.md\#tk)* is installed and available as
well\.

Without *[Tk](\.\./\.\./\.\./index\.md\#tk)* the only way to invoke the installer
are to open a DOS window, i\.e\. __cmd\.exe__, and then to invoke

    \./installer\.tcl

inside it\.

## <a name='subsection5'></a>Critcl & Accelerators

While the majority of Tcllib consists of packages written in pure Tcl a number
of packages also have *accelerators* associated with them\. These are
__critcl__\-based C packages whose use will boost the performance of the
packages using them\. These accelerators are optional, and they are not installed
by default\.

To build the accelerators the normally optional dependency on __critcl__
becomes required\.

To build and install Tcllib with the accelerators in a Unix\-like environment
invoke:

    \./configure
    make critcl \# This builds the shared library holding
                \# the accelerators
    make install

The underlying tool is "sak\.tcl" in the toplevel directory of Tcllib and the
command __make critcl__ is just a wrapper around

    \./sak\.tcl critcl

Therefore in a Windows environment instead invoke

    \./sak\.tcl critcl
    \./installer\.tcl

from within a DOS window, i\.e\. __cmd\.exe__\.

## <a name='subsection6'></a>Tooling

The core of Tcllib's build system is the script "installer\.tcl" found in the
toplevel directory of a checkout or release\.

The

    configure ; make install

setup available to developers on Unix\-like systems is just a wrapper around it\.
To go beyond the standard embodied in the wrapper it is necessary to directly
invoke this script\.

On Windows system using it directly is the only way to invoke it\.

For basic help invoke it as

    \./installer\.tcl \-help

This will print a short list of all the available options to the standard output
channel\.

The commands associated with the various *install* targets in the
*Makefile\.in* for Unix can be used as additional examples on how to use this
tool as well\.

# <a name='section2'></a>Tools

The "sak\.tcl" script in the toplevel directory of a Tcllib checkout is the one
tool used by the release manager to perform its [Tasks](#section3)\.

The main commands to be used are

    sak\.tcl validate
    sak\.tcl test run
    sak\.tcl review
    sak\.tcl readme
    sak\.tcl localdoc
    sak\.tcl release

More detail will be provided in the explanations of the various
[Tasks](#section3)\.

# <a name='section3'></a>Tasks

## <a name='subsection1'></a>Start a release candidate

For the curious \(or a developer\-to\-be\), the sources are managed by the [Fossil
SCM](http://www\.fossil\-scm\.org)\. Binaries for popular platforms can be found
directly at its [download page](http://www\.fossil\-scm\.org/download\.html)\.

With that tool available the full history can be retrieved via:

    fossil clone  http://core\.tcl\-lang\.org/tcllib  tcllib\.fossil

followed by

    mkdir tcllib
    cd tcllib
    fossil open \.\./tcllib\.fossil

to get a checkout of the head of the trunk\.

    randomly and transmitted as the first block of the output\. Errors in
    encryption affect the current block and the next block after which the
    cipher will correct itself\. CBC is the most commonly used mode in software
    encryption\. This is the default mode of operation for this module\.

# <a name='section5'></a>EXAMPLES

    % set nil\_block \[string repeat \\\\0 16\]
    % aes::aes \-hex \-mode cbc \-dir encrypt \-key $nil\_block $nil\_block
    66e94bd4ef8a2c3b884cfa59ca342b2e

    set Key \[aes::Init cbc $sixteen\_bytes\_key\_data $sixteen\_byte\_iv\]
    append ciphertext \[aes::Encrypt $Key $plaintext\]
    append ciphertext \[aes::Encrypt $Key $additional\_plaintext\]
    aes::Final $Key

# <a name='section6'></a>REFERENCES

  1. "Advanced Encryption Standard", Federal Information Processing Standards
     Publication 197, 2001
     \([http://csrc\.nist\.gov/publications/fips/fips197/fips\-197\.pdf](http://csrc\.nist\.gov/publications/fips/fips197/fips\-197\.pdf)\)

      * __\-prefix__

        This names the prefix that will be added to all resources\. That is, it
        is the remote equivalent of __\-directory__\. If it is not specified,
        the root of the bucket will be treated as the remote directory\. An
        example may clarify\.

            S3::Push \-bucket test \-directory /tmp/xyz \-prefix hello/world

        In this example, /tmp/xyz/pdq\.html will be stored as
        http://s3\.amazonaws\.com/test/hello/world/pdq\.html in Amazon's servers\.
        Also, /tmp/xyz/abc/def/Hello will be stored as
        http://s3\.amazonaws\.com/test/hello/world/abc/def/Hello in Amazon's
        servers\. Without the __\-prefix__ option, /tmp/xyz/pdq\.html would be
        stored as http://s3\.amazonaws\.com/test/pdq\.html\.

    delete files that have been deleted from one place but not the other yet not
    copying changed files is untested\.

# <a name='section7'></a>USAGE SUGGESTIONS

To fetch a "directory" out of a bucket, make changes, and store it back:

    file mkdir \./tempfiles
    S3::Pull \-bucket sample \-prefix of/interest \-directory \./tempfiles \\
      \-timestamp aws
    do\_my\_process \./tempfiles other arguments
    S3::Push \-bucket sample \-prefix of/interest \-directory \./tempfiles \\
      \-compare newer \-delete true

To delete files locally that were deleted off of S3 but not otherwise update
files:

    S3::Pull \-bucket sample \-prefix of/interest \-directory \./myfiles \\
      \-compare never \-delete true

# <a name='section8'></a>FUTURE DEVELOPMENTS

The author intends to work on several additional projects related to this
package, in addition to finishing the unfinished features\.

First, a command\-line program allowing browsing of buckets and transfer of files

To handle this change the applications using
__[TLS](\.\./\.\./\.\./\.\./index\.md\#tls)__ must be patched, and not this
package, nor __[TLS](\.\./\.\./\.\./\.\./index\.md\#tls)__ itself\. Such a patch
may be as simple as generally activating __tls1__ support, as shown in the
example below\.

    package require tls
    tls::init \-tls1 1 ;\# forcibly activate support for the TLS1 protocol

    \.\.\. your own application code \.\.\.

# <a name='section10'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *amazon\-s3* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

      * %PCDATA?

        is like %PCDATA, but returns an empty string if no PCDATA is found\.

    For example, to fetch the first bold text from the fifth paragraph of the
    body of your HTML file,

        xsxp::fetch $pxml \{body p\#4 b\} %PCDATA

  - <a name='3'></a>__xsxp::fetchall__ *pxml\_list* *path* ?*part*?

    This iterates over each PXML in *pxml\_list* \(which must be a list of
    pxmls\) selecting the indicated path from it, building a new list with the
    selected data, and returning that new list\.

    Ascii85 decodes the given *string* and returns the binary data\. The
    decoder ignores whitespace in the string, as well as tabs and newlines\.

# <a name='section2'></a>EXAMPLES

    % ascii85::encode "Hello, world"
    87cURD\_\*\#TDfTZ\)

    % ascii85::encode \[string repeat xyz 24\]
    G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G
    ^4U\[H$X^\\H?a^\]
    % ascii85::encode \-wrapchar "" \[string repeat xyz 24\]
    G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]G^4U\[H$X^\\H?a^\]

    \# NOTE: ascii85 encodes BINARY strings\.
    % set chemical \[encoding convertto utf\-8 "C\\u2088H\\u2081\\u2080N\\u2084O\\u2082"\]
    % set encoded \[ascii85::encode $chemical\]
    6fN\]R8E,5Pidu\\UiduhZidua
    % set caffeine \[encoding convertfrom utf\-8 \[ascii85::decode $encoded\]\]

# <a name='section3'></a>References

  1. [http://en\.wikipedia\.org/wiki/Ascii85](http://en\.wikipedia\.org/wiki/Ascii85)

  1. Postscript Language Reference Manual, 3rd Edition, page 131\.
     [http://www\.adobe\.com/devnet/postscript/pdfs/PLRM\.pdf](http://www\.adobe\.com/devnet/postscript/pdfs/PLRM\.pdf)

    ignores whitespace in the string\.

# <a name='section2'></a>EXAMPLES

    % base64::encode "Hello, world"
    SGVsbG8sIHdvcmxk

    % base64::encode \[string repeat xyz 20\]
    eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6
    eHl6eHl6eHl6
    % base64::encode \-wrapchar "" \[string repeat xyz 20\]
    eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6eHl6

    \# NOTE: base64 encodes BINARY strings\.
    % set chemical \[encoding convertto utf\-8 "C\\u2088H\\u2081\\u2080N\\u2084O\\u2082"\]
    % set encoded \[base64::encode $chemical\]
    Q\+KCiEjigoHigoBO4oKET\+KCgg==
    % set caffeine \[encoding convertfrom utf\-8 \[base64::decode $encoded\]\]

# <a name='section3'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *base64* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

    The uuencoded data header line contains a suggested permissions bit pattern
    expressed as an octal string\. To change the default of 0644 you can set this
    option\. For instance, 0755 would be suitable for an executable\. See
    __chmod\(1\)__\.

# <a name='section3'></a>EXAMPLES

    % set d \[uuencode::encode "Hello World\!"\]
    2&5L;&\\\\@5V\]R;&0A

    % uuencode::uudecode $d
    Hello World\!

    % set d \[uuencode::uuencode \-name hello\.txt "Hello World"\]
    begin 644 hello\.txt
    \+2&5L;&\\@5V\]R;&0\`
    \`

    end

    % uuencode::uudecode $d
    \{hello\.txt 644 \{Hello World\}\}

# <a name='section4'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *base64* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

  - \-crc32 boolean

    The yEnc specification recommends the inclusion of a cyclic redundancy check
    value in the footer\. Use this option to change the default from *true* to
    *false*\.

    % set d \[yencode::yencode \-file testfile\.txt\]
    =ybegin line=128 size=584 name=testfile\.txt
     \-o\- data not shown \-o\-
    =yend size=584 crc32=ded29f4f

# <a name='section3'></a>References

  1. [http://www\.yenc\.org/yenc\-draft\.1\.3\.txt](http://www\.yenc\.org/yenc\-draft\.1\.3\.txt)

# <a name='section4'></a>Bugs, Ideas, Feedback

number of commands to support the declaration of benchmarks\. A document written
in this language is a Tcl script and has the same syntax\.

## <a name='subsection2'></a>Basics

One of the most simplest benchmarks which can be written in bench is

    bench \-desc LABEL \-body \{
        set a b
    \}


This code declares a benchmark named __LABEL__ which measures the time it
takes to assign a value to a variable\. The Tcl code doing this assignment is the
__\-body__ of the benchmark\.

## <a name='subsection3'></a>Pre\- and postprocessing

Our next example demonstrates how to declare *initialization* and
*[cleanup](\.\./\.\./\.\./\.\./index\.md\#cleanup)* code, i\.e\. code computing
information for the use of the __\-body__, and for releasing such resources
after the measurement is done\. They are the __\-pre__\- and the
__\-post__\-body, respectively\.

In our example, directly drawn from the benchmark suite of Tcllib's
__[aes](\.\./aes/aes\.md)__ package, the concrete initialization code
constructs the key schedule used by the encryption command whose speed we
measure, and the cleanup code releases any resources bound to that schedule\.

    bench \-desc "AES\-$\{len\} ECB encryption core" __\-pre__ \{
        set key \[aes::Init ecb $k $i\]
    \} \-body \{
        aes::Encrypt $key $p
    \} __\-post__ \{
        aes::Final $key
    \}


## <a name='subsection4'></a>Advanced pre\- and postprocessing

Our last example again deals with initialization and cleanup code\. To see the
difference to the regular initialization and cleanup discussed in the last
section it is necessary to know a bit more about how bench actually measures the
speed of the the __\-body__\.

example we used above to demonstrate the necessity for the advanced
initialization and cleanup\. Its concrete initialization code constructs a
variable refering to a set with specific properties \(The set has a string
representation, which is shared\) affecting the speed of the inclusion command,
and the cleanup code releases the temporary variables created by this
initialization\.

    bench \-desc "set include, missing <SC> x$times $n" __\-ipre__ \{
        set A $sx\($times,$n\)
        set B $A
    \} \-body \{
        struct::set include A x
    \} __\-ipost__ \{
        unset A B
    \}


# <a name='section2'></a>FURTHER READING

Now that this document has been digested the reader, assumed to be a *writer*
of benchmarks, he should be fortified enough to be able to understand the formal
*bench language specfication*\. It will also serve as the detailed
specification and cheat sheet for all available commands and their syntax\.

    randomly and transmitted as the first block of the output\. Errors in
    encryption affect the current block and the next block after which the
    cipher will correct itself\. CBC is the most commonly used mode in software
    encryption\.

# <a name='section5'></a>EXAMPLES

    % blowfish::blowfish \-hex \-mode ecb \-dir encrypt \-key secret01 "hello, world\!"
    d0d8f27e7a374b9e2dbd9938dd04195a

    set Key \[blowfish::Init cbc $eight\_bytes\_key\_data $eight\_byte\_iv\]
    append ciphertext \[blowfish::Encrypt $Key $plaintext\]
    append ciphertext \[blowfish::Encrypt $Key $additional\_plaintext\]
    blowfish::Final $Key

# <a name='section6'></a>REFERENCES

  1. Schneier, B\. "Applied Cryptography, 2nd edition", 1996, ISBN 0\-471\-11709\-9,
     pub\. John Wiley & Sons\.

Starting with version 1\.5 all errors thrown by the package have a proper
__::errorCode__ for use with Tcl's __[try](\.\./try/tcllib\_try\.md)__
command\. This code always has the word __CMDLINE__ as its first element\.

# <a name='section4'></a>EXAMPLES

            package require Tcl 8\.5
            package require try         ;\# Tcllib\.
            package require cmdline 1\.5 ;\# First version with proper error\-codes\.

            \# Notes:
            \# \- Tcl 8\.6\+ has 'try' as a builtin command and therefore does not
            \#   need the 'try' package\.
            \# \- Before Tcl 8\.5 we cannot support 'try' and have to use 'catch'\.
            \#   This then requires a dedicated test \(if\) on the contents of
            \#   ::errorCode to separate the CMDLINE USAGE signal from actual errors\.

            set options \{
                \{a          "set the atime only"\}
                \{m          "set the mtime only"\}
                \{c          "do not create non\-existent files"\}
                \{r\.arg  ""  "use time from ref\_file"\}
                \{t\.arg  \-1  "use specified time"\}
            \}

            set usage ": MyCommandName \\\[options\] filename \.\.\.\\noptions:"

            try \{
                array set params \[::cmdline::getoptions argv $options $usage\]
            \} trap \{CMDLINE USAGE\} \{msg o\} \{
                \# Trap the usage signal, print the message, and exit the application\.
                \# Note: Other errors are not caught and passed through to higher levels\!
    	    puts $msg
    	    exit 1
            \}

            if \{  $params\(a\) \} \{ set set\_atime "true" \}

            set has\_t \[expr \{$params\(t\) \!= \-1\}\]
            set has\_r \[expr \{\[string length $params\(r\)\] > 0\}\]
            if \{$has\_t && $has\_r\} \{
                return \-code error "Cannot specify both \-r and \-t"
            \} elseif \{$has\_t\} \{
    	    \.\.\.
            \}


This example, taken \(and slightly modified\) from the package
__[fileutil](\.\./fileutil/fileutil\.md)__, shows how to use cmdline\.
First, a list of options is created, then the 'args' list is passed to cmdline
for processing\. Subsequently, different options are checked to see if they have
been passed to the script, and what their value is\.

server for the communication path\. As a result, __comm__ works with multiple
interpreters, works on Windows and Macintosh systems, and provides control over
the remote execution path\.

These commands work just like __[send](\.\./\.\./\.\./\.\./index\.md\#send)__ and
__winfo interps__ :

    ::comm::comm send ?\-async? id cmd ?arg arg \.\.\.?
    ::comm::comm interps

This is all that is really needed to know in order to use __comm__

## <a name='subsection1'></a>Commands

The package initializes __::comm::comm__ as the default *chan*\.

If you find that __::comm::comm send__ doesn't work for a particular
command, try the same thing with Tk's send and see if the result is different\.
If there is a problem, please report it\. For instance, there was had one report
that this command produced an error\. Note that the equivalent
__[send](\.\./\.\./\.\./\.\./index\.md\#send)__ command also produces the same
error\.

    % ::comm::comm send id llength \{a b c\}
    wrong \# args: should be "llength list"
    % send name llength \{a b c\}
    wrong \# args: should be "llength list"

The __eval__ hook \(described below\) can be used to change from
__[send](\.\./\.\./\.\./\.\./index\.md\#send)__'s double eval semantics to single
eval semantics\.

## <a name='subsection3'></a>Multiple Channels

  - <a name='6'></a>__::comm::comm channels__

    This lists all the channels allocated in this Tcl interpreter\.

The default configuration parameters for a new channel are:

    "\-port 0 \-local 1 \-listen 0 \-silent 0"

The default channel __::comm::comm__ is created with:

    "::comm::comm new ::comm::comm \-port 0 \-local 1 \-listen 1 \-silent 0"

## <a name='subsection4'></a>Channel Configuration

The __config__ method acts similar to __fconfigure__ in that it sets or
queries configuration variables associated with a channel\.

  - <a name='7'></a>__::comm::comm config__

    Variables: __chan__, __id__

    This hook is invoked before making a connection to the remote named in
    *id*\. An error return \(via
    __[error](\.\./\.\./\.\./\.\./index\.md\#error)__\) will abort the connection
    attempt with the error\. Example:

    % ::comm::comm hook connecting \{
        if \{\[string match \{\*\[02468\]\} $id\]\} \{
            error "Can't connect to even ids"
        \}
    \}


    % ::comm::comm send 10000 puts ok
    Connect to remote failed: Can't connect to even ids
    %

  - __connected__

    Variables: __chan__, __fid__, __id__, __host__, and

    Hook invoked when receiving an incoming connection, allowing arbitrary
    authentication over socket named by *fid*\. An error return \(via
    __[error](\.\./\.\./\.\./\.\./index\.md\#error)__\) will close the connection
    with the error\. Note that the peer is named by *remport* and *addr* but
    that the remote *id* is still unknown\. Example:

    ::comm::comm hook incoming \{
        if \{\[string match 127\.0\.0\.1 $addr\]\} \{
            error "I don't talk to myself"
        \}
    \}



  - __eval__

    Variables: __chan__, __id__, __cmd__, and __buffer__\.

    This hook is invoked after collecting a complete script from a remote but
    *before* evaluating it\. This allows complete control over the processing

    __break__ and __return \-code break__ *result* is supported, acting
    similarly to __return \{\}__ and __return \-code return__ *result*\.

    Examples:

      1. augmenting a command

    % ::comm::comm send \[::comm::comm self\] pid
    5013
    % ::comm::comm hook eval \{puts "going to execute $buffer"\}
    % ::comm::comm send \[::comm::comm self\] pid
    going to execute pid
    5013

      1. short circuiting a command

    % ::comm::comm hook eval \{puts "would have executed $buffer"; return 0\}
    % ::comm::comm send \[::comm::comm self\] pid
    would have executed pid
    0

      1. Replacing double eval semantics

    % ::comm::comm send \[::comm::comm self\] llength \{a b c\}
    wrong \# args: should be "llength list"
    % ::comm::comm hook eval \{return \[uplevel \#0 $buffer\]\}
    return \[uplevel \#0 $buffer\]
    % ::comm::comm send \[::comm::comm self\] llength \{a b c\}
    3

      1. Using a slave interpreter

    % interp create foo
    % ::comm::comm hook eval \{return \[foo eval $buffer\]\}
    % ::comm::comm send \[::comm::comm self\] set myvar 123
    123
    % set myvar
    can't read "myvar": no such variable
    % foo eval set myvar
    123

      1. Using a slave interpreter \(double eval\)

    % ::comm::comm hook eval \{return \[eval foo eval $buffer\]\}

      1. Subverting the script to execute

    % ::comm::comm hook eval \{
        switch \-\- $buffer \{
            a \{return A\-OK\}
            b \{return B\-OK\}
            default \{error "$buffer is a no\-no"\}
        \}
    \}


    % ::comm::comm send \[::comm::comm self\] pid
    pid is a no\-no
    % ::comm::comm send \[::comm::comm self\] a
    A\-OK

  - __reply__

    Variables: __chan__, __id__, __buffer__, __ret__, and
    __return\(\)__\.

    This hook is invoked after collecting a complete reply script from a remote

    Variables: __chan__, __id__, and __reason__\.

    This hook is invoked when the connection to __id__ is lost\. Return value
    \(or thrown error\) is ignored\. *reason* is an explanatory string indicating
    why the connection was lost\. Example:

    ::comm::comm hook lost \{
        global myvar
        if \{$myvar\(id\) == $id\} \{
            myfunc
            return
        \}
    \}



## <a name='subsection10'></a>Unsupported

These interfaces may change or go away in subsequence releases\.

  - <a name='14'></a>__::comm::comm remoteid__

    Returns the *id* of the sender of the last remote command executed on this
    channel\. If used by a proc being invoked remotely, it must be called before
    any events are processed\. Otherwise, another command may get invoked and
    change the value\.

  - <a name='15'></a>__::comm::comm\_send__

    Invoking this procedure will substitute the Tk
    __[send](\.\./\.\./\.\./\.\./index\.md\#send)__ and __winfo interps__
    commands with these equivalents that use __::comm::comm__\.

    proc send \{args\} \{
        eval ::comm::comm send $args
    \}

    rename winfo tk\_winfo
    proc winfo \{cmd args\} \{
        if \{\!\[string match in\* $cmd\]\} \{
            return \[eval \[list tk\_winfo $cmd\] $args\]
        \}

        return \[::comm::comm interps\]
    \}


## <a name='subsection11'></a>Security

Starting with version 4\.6 of the package an option __\-socketcmd__ is
supported, allowing the user of a comm channel to specify which command to use
when opening a socket\. Anything which is API\-compatible with the builtin
__::socket__ \(the default\) can be used\.

The envisioned main use is the specification of the __tls::socket__ command,
see package __[tls](\.\./\.\./\.\./\.\./index\.md\#tls)__, to secure the
communication\.

    \# Load and initialize tls
    package require tls
    tls::init  \-cafile /path/to/ca/cert \-keyfile \.\.\.

    \# Create secured comm channel
    ::comm::comm new SECURE \-socketcmd tls::socket \-listen 1
    \.\.\.

The sections [Execution Environment](#subsection6) and
[Callbacks](#subsection9) are also relevant to the security of the system,
providing means to restrict the execution to a specific environment, perform
additional authentication, and the like\.

## <a name='subsection12'></a>Blocking Semantics

    being computed the future will not try to deliver the result it got, but
    just destroy itself\. The future can be configured with a command to call
    when the invoker is lost\. This enables the user to implement an early abort
    of the long\-running operation, should this be supported by it\.

    An example:

    \# Procedure invoked by remote clients to run database operations\.
    proc select \{sql\} \{
        \# Signal the async generation of the result

        set future \[::comm::comm return\_async\]

        \# Generate an async db operation and tell it where to deliver the result\.

        set query \[db query \-command \[list $future return\] $sql\]

        \# Tell the database system which query to cancel if the connection
        \# goes away while it is running\.

        $future configure \-command \[list db cancel $query\]

        \# Note: The above will work without problem only if the async
        \# query will nover run its completion callback immediately, but
        \# only from the eventloop\. Because otherwise the future we wish to
        \# configure may already be gone\. If that is possible use 'catch'
        \# to prevent the error from propagating\.
        return
    \}


    The API of a future object is:

      * <a name='17'></a>__$future__ __return__ ?__\-code__ *code*? ?*value*?

        Use this method to tell the future that long\-running operation has
        completed\. Arguments are an optional return value \(defaults to the empty

    being returned correctly from __comm send__\. This has been fixed by
    removing the extra level of indirection into the internal procedure
    __commSend__\. Also added propagation of the *errorCode* variable\. This
    means that these commands return exactly as they would with
    __[send](\.\./\.\./\.\./\.\./index\.md\#send)__:

    comm send id break
    catch \{comm send id break\}
    comm send id expr 1 / 0

    Added a new hook for reply messages\. Reworked method invocation to avoid the
    use of comm:\* procedures; this also cut the invocation time down by 40%\.
    Documented __comm config__ \(as this manual page still listed the defunct
    __comm init__\!\)

To handle this change the applications using
__[TLS](\.\./\.\./\.\./\.\./index\.md\#tls)__ must be patched, and not this
package, nor __[TLS](\.\./\.\./\.\./\.\./index\.md\#tls)__ itself\. Such a patch
may be as simple as generally activating __tls1__ support, as shown in the
example below\.

    package require tls
    tls::init \-tls1 1 ;\# forcibly activate support for the TLS1 protocol

    \.\.\. your own application code \.\.\.

# <a name='section3'></a>Author

John LoVerso, John@LoVerso\.Southborough\.MA\.US

*http://www\.opengroup\.org/~loverso/tcl\-tk/\#comm*

    __concat__enated together by the server to form the full script to
    execute on the server side\. This emulates the Tcl "eval" semantics\. In most
    cases it is best to have only one word in the list, a list containing the
    exact command\.

    Examples:

        \(a\)     \{send 1 \{\{array get tcl\_platform\}\}\}
        \(b\)     \{send 1 \{array get tcl\_platform\}\}
        \(c\)     \{send 1 \{array \{get tcl\_platform\}\}\}

        are all valid representations of the same command\. They are
        generated via

        \(a'\)    send \{array get tcl\_platform\}
        \(b'\)    send array get tcl\_platform
        \(c'\)    send array \{get tcl\_platform\}

        respectively

    Note that \(a\), generated by \(a'\), is the usual form, if only single commands
    are sent by the client\. For example constructed using
    __[list](\.\./\.\./\.\./\.\./index\.md\#list)__, if the command contains
    variable arguments\. Like

        send \[list array get $the\_variable\]

    These three instructions all invoke the script on the server side\. Their
    difference is in the treatment of result values, and thus determines if a
    reply is expected\.

      * __send__

    Like the previous three command, however the tcl script in the payload is
    highly restricted\. It has to be a syntactically valid Tcl
    __[return](\.\./\.\./\.\./\.\./index\.md\#return)__ command\. This contains
    result code, value, error code, and error info\.

    Examples:

        \{reply 1 \{return \-code 0 \{\}\}\}
        \{reply 1 \{return \-code 0 \{osVersion 2\.4\.21\-99\-default byteOrder littleEndian machine i686 platform unix os Linux user andreask wordSize 4\}\}\}

# <a name='section3'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *comm* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

    that debugging efforts can be independently controlled module by module\.

        % package require control
        % control::control assert enabled 1
        % namespace eval one namespace import ::control::assert
        % control::control assert enabled 0
        % namespace eval two namespace import ::control::assert
        % one::assert \{1 == 0\}
        assertion failed: 1 == 0
        % two::assert \{1 == 0\}

  - <a name='3'></a>__control::do__ *body* ?*option test*?

    The __[do](\.\./\.\./\.\./\.\./index\.md\#do)__ command evaluates the script
    *body* repeatedly *until* the expression *test* becomes true or as
    long as \(*while*\) *test* is true, depending on the value of *option*
    being __until__ or __while__\. If *option* and *test* are omitted

\-code $code__\] within one of those script arguments for any value of *$code*
other than *ok*\. In this way, the commands of the __control__ package are
limited as compared to Tcl's built\-in control flow commands \(such as __if__,
__while__, etc\.\) and those control flow commands that can be provided by
packages coded in C\. An example of this difference:

    % package require control
    % proc a \{\} \{while 1 \{return \-code error a\}\}
    % proc b \{\} \{control::do \{return \-code error b\} while 1\}
    % catch a
    1
    % catch b
    0

# <a name='section4'></a>Bugs, Ideas, Feedback

    Returns the checksum value and releases any resources held by this token\.
    Once this command completes the token will be invalid\. The result is a 32
    bit integer value\.

# <a name='section5'></a>EXAMPLES

    % crc::cksum "Hello, World\!"
    2609532967

    % crc::cksum \-format 0x%X "Hello, World\!"
    0x9B8A5027

    % crc::cksum \-file cksum\.tcl
    1828321145

    % set tok \[crc::CksumInit\]
    % crc::CksumUpdate $tok "Hello, "
    % crc::CksumUpdate $tok "World\!"
    % crc::CksumFinal $tok
    2609532967

# <a name='section6'></a>AUTHORS

Pat Thoyts

    flag is important when processing data from parameters\. If the binary data
    looks like one of the options given above then the data will be read as an
    option if this marker is not included\. Always use the *\-\-* option
    termination flag before giving the data argument\.

# <a name='section4'></a>EXAMPLES

    % crc::crc16 \-\- "Hello, World\!"
    64077

    % crc::crc\-ccitt \-\- "Hello, World\!"
    26586

    % crc::crc16 \-format 0x%X \-\- "Hello, World\!"
    0xFA4D

    % crc::crc16 \-file crc16\.tcl
    51675

# <a name='section5'></a>AUTHORS

Pat Thoyts

# <a name='section6'></a>Bugs, Ideas, Feedback

    Returns the checksum value and releases any resources held by this token\.
    Once this command completes the token will be invalid\. The result is a 32
    bit integer value\.

# <a name='section5'></a>EXAMPLES

    % crc::crc32 "Hello, World\!"
    3964322768

    % crc::crc32 \-format 0x%X "Hello, World\!"
    0xEC4AC3D0

    % crc::crc32 \-file crc32\.tcl
    483919716

    % set tok \[crc::Crc32Init\]
    % crc::Crc32Update $tok "Hello, "
    % crc::Crc32Update $tok "World\!"
    % crc::Crc32Final $tok
    3964322768

# <a name='section6'></a>AUTHORS

Pat Thoyts

  - \-format *string*

    Return the checksum using an alternative format template\.

# <a name='section4'></a>EXAMPLES

    % crc::sum "Hello, World\!"
    37287

    % crc::sum \-format 0x%X "Hello, World\!"
    0x91A7

    % crc::sum \-file sum\.tcl
    13392

# <a name='section5'></a>AUTHORS

Pat Thoyts

# <a name='section6'></a>Bugs, Ideas, Feedback

    *timecode*\. If *timecode* is expressed as an integer, the timecode is
    assumed to be in unixtime\. All other inputs will be interpreted by __clock
    scan__ and converted to unix time\. This task can be modified by subsequent
    calls to this package's commands by referencing *processname*\. If
    *processname* exists, it will be replaced\. If *processname* is not
    given, one is generated and returned by the command\.

        ::cron::at start\_coffee \{Tomorrow at 9:00am\}  \{remote::exec::coffeepot power on\}
        ::cron::at shutdown\_coffee \{Tomorrow at 12:00pm\}  \{remote::exec::coffeepot power off\}

  - <a name='2'></a>__::cron::cancel__ *processname*

    This command unregisters the process *processname* and cancels any pending
    commands\. Note: processname can be a process created by either
    __::cron::at__ or __::cron::every__\.

        ::cron::cancel check\_mail

  - <a name='3'></a>__::cron::every__ *processname* *frequency* *command*

    This command registers a *command* to be called at the interval of
    *frequency*\. *frequency* is given in seconds\. This task can be modified
    by subsequent calls to this package's commands by referencing
    *processname*\. If *processname* exists, it will be replaced\.

        ::cron::every check\_mail 900  ::imap\_client::check\_mail
        ::cron::every backup\_db  3600 \{::backup\_procedure ::mydb\}

  - <a name='4'></a>__::cron::in__ *?processname?* *timecode* *command*

    This command registers a *command* to be called after a delay of time
    specified by *timecode*\. *timecode* is expressed as an seconds\. This
    task can be modified by subsequent calls to this package's commands by
    referencing *processname*\. If *processname* exists, it will be replaced\.

    If the ::cron::time variable is > 0 this command will advance the internal
    time, 100ms at a time\.

    In all other cases this command will generate a fictious variable, generate
    an after call, and vwait the variable:

        set eventid \[incr ::cron::eventcount\]
        set var ::cron::event\_\#$eventid
        set $var 0
        ::after $ms "set $var 1"
        ::vwait $var
        ::unset $var

    Usage:

        does so\.

  - <a name='11'></a>__::cron::wake__ *?who?*

    Wake up cron, and arrange for its event loop to be run during the next Idle
    cycle\.

        ::cron::wake \{I just did something important\}

Several utility commands are provided that are used internally within cron and
for testing cron, but may or may not be useful in the general cases\.

  - <a name='12'></a>__::cron::clock\_step__ *milliseconds*

    Return a clock time absolute to the epoch which falls on the next border

The alternate format is activated through specification of the option
__\-alternate__ to the various split commands\.

# <a name='section4'></a>EXAMPLE

Using the regular format the record

    123,"123,521\.2","Mary says ""Hello, I am Mary""",""

is parsed into the items

    a\) 123
    b\) 123,521\.2
    c\) Mary says "Hello, I am Mary"
    d\) "

Using the alternate format the result is

    a\) 123
    b\) 123,521\.2
    c\) Mary says "Hello, I am Mary"
    d\) \(the empty string\)

instead\. As can be seen only item \(d\) is different, now the empty string instead
of a "\.

# <a name='section5'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and

    identifier returned by __::defer::defer__, __::defer::with__, or
    __::defer::autowith__\. Any number of arguments may be supplied, and all
    of the IDs supplied will be cancelled\.

# <a name='section3'></a>EXAMPLES

    package require defer 1
    apply \{\{\} \{
    	set fd \[open /dev/null\]
    	defer::defer close $fd
    \}\}

# <a name='section4'></a>REFERENCES

# <a name='section5'></a>AUTHORS

Roy Keene

    OFB is similar to CFB except that the output of the cipher is fed back into
    the next round and not the xor'd plain text\. This means that errors only
    affect a single block but the cipher is more vulnerable to attack\.

# <a name='section5'></a>EXAMPLES

    % set ciphertext \[DES::des \-mode cbc \-dir encrypt \-key $secret $plaintext\]
    % set plaintext \[DES::des \-mode cbc \-dir decrypt \-key $secret $ciphertext\]

    set iv \[string repeat \\\\0 8\]
    set Key \[DES::Init cbc \\\\0\\\\1\\\\2\\\\3\\\\4\\\\5\\\\6\\\\7 $iv\]
    set ciphertext \[DES::Encrypt $Key "somedata"\]
    append ciphertext \[DES::Encrypt $Key "moredata"\]
    DES::Reset $Key $iv
    set plaintext \[DES::Decrypt $Key $ciphertext\]
    DES::Final $Key

# <a name='section6'></a>REFERENCES

  1. "Data Encryption Standard", Federal Information Processing Standards
     Publication 46\-3, 1999,
     \([http://csrc\.nist\.gov/publications/fips/fips46\-3/fips46\-3\.pdf](http://csrc\.nist\.gov/publications/fips/fips46\-3/fips46\-3\.pdf)\)

  - <a name='6'></a>__rmerge__ *args*

    Return a dict which is the product of a recursive merge of all of the
    arguments\. Unlike __dict merge__, this command descends into all of the
    levels of a dict\. Dict keys which end in a : indicate a leaf, which will be
    interpreted as a literal value, and not descended into further\.

        set items \[dict merge \{
          option \{color \{default: green\}\}
        \} \{
          option \{fruit \{default: mango\}\}
        \} \{
          option \{color \{default: blue\} fruit \{widget: select values: \{mango apple cherry grape\}\}\}
        \}\]
        puts \[dict print $items\]

    Prints the following result:

        option \{
          color \{
            default: blue
          \}

          fruit \{
            widget: select
            values: \{mango apple cherry grape\}
          \}
        \}



# <a name='section2'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *dict* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

    users system\. On a unix machine this parses the /etc/resolv\.conf file for
    nameservers \(if it exists\) and on Windows systems we examine certain parts
    of the registry\. If no nameserver can be found then the loopback address
    \(127\.0\.0\.1\) is used as a default\.

# <a name='section3'></a>EXAMPLES

    % set tok \[dns::resolve www\.tcl\.tk\]
    ::dns::1
    % dns::status $tok
    ok
    % dns::address $tok
    199\.175\.6\.239
    % dns::name $tok
    www\.tcl\.tk
    % dns::cleanup $tok

Using DNS URIs as queries:

    % set tok \[dns::resolve "dns:tcl\.tk;type=MX"\]
    % set tok \[dns::resolve "dns://l\.root\-servers\.net/www\.tcl\.tk"\]

Reverse address lookup:

    % set tok \[dns::resolve 127\.0\.0\.1\]
    ::dns::1
    % dns::name $tok
    localhost
    % dns::cleanup $tok

Using DNS over TLS \(RFC 7858\):

    % set tok \[dns::resolve www\.tcl\.tk \-nameserver dns\-tls\.bitwiseshift\.net  \-usetls 1 \-cafile /etc/ssl/certs/ca\-certificates\.crt\]
    ::dns::12
    % dns::wait $tok
    ok
    % dns::address $tok
    104\.25\.119\.118 104\.25\.120\.118

# <a name='section4'></a>REFERENCES

  1. Mockapetris, P\., "Domain Names \- Concepts and Facilities", RFC 1034,
     November 1987\.
     \([http://www\.ietf\.org/rfc/rfc1034\.txt](http://www\.ietf\.org/rfc/rfc1034\.txt)\)

    suitable for displaying to users\.

  - <a name='6'></a>__::ip::distance__ *ipaddr1* *ipaddr2*

    This command computes the \(integer\) distance from IPv4 address *ipaddr1*
    to IPv4 address *ipaddr2*, i\.e\. "ipaddr2 \- ipaddr1"

        % ::ip::distance 1\.1\.1\.1  1\.1\.1\.5
        4

  - <a name='7'></a>__::ip::nextIp__ *ipaddr* ?*offset*?

    This command adds the integer *offset* to the IPv4 address *ipaddr* and
    returns the new IPv4 address\.

        % ::ip::distance 1\.1\.1\.1  4
        1\.1\.1\.5

  - <a name='8'></a>__::ip::prefix__ *address*

    Returns the address prefix generated by masking the address part with the
    mask if provided\. If there is no mask then it is equivalent to calling
    __normalize__

  - <a name='9'></a>__::ip::type__ *address*

  - <a name='10'></a>__::ip::mask__ *address*

    If the address supplied includes a mask then this is returned otherwise
    returns an empty string\.

  - <a name='11'></a>__::ip::prefixToNative__ *prefix*

    This command converts the string *prefix* from dotted form
    \(<ipaddr>/<mask> format\) to native \(hex\) form\. Returns a list containing two
    elements, ipaddress and mask, in this order, in hexadecimal notation\.

        % ip::prefixToNative 1\.1\.1\.0/24
        0x01010100 0xffffff00

  - <a name='12'></a>__::ip::nativeToPrefix__ *nativeList*&#124;*native* ?__\-ipv4__?

    This command converts from native \(hex\) form to dotted form\. It is the
    complement of __::ip::prefixToNative__\.

      * list *nativeList* \(in\)

        List of several ip addresses in native form\. The native form is a list
        as returned by __::ip::prefixToNative__\.

      * list *native* \(in\)

        A list as returned by __::ip::prefixToNative__\.

    The command returns a list of addresses in dotted form if it was called with
    a list of addresses\. Otherwise a single address in dotted form is returned\.

        % ip::nativeToPrefix \{0x01010100 0xffffff00\} \-ipv4
        1\.1\.1\.0/24

  - <a name='13'></a>__::ip::intToString__ *number* ?__\-ipv4__?

    This command converts from an ip address specified as integer number to
    dotted form\.

        ip::intToString 4294967295
        255\.255\.255\.255

  - <a name='14'></a>__::ip::toInteger__ *ipaddr*

    This command converts a dotted form ip into an integer number\.

        % ::ip::toInteger 1\.1\.1\.0
        16843008

  - <a name='15'></a>__::ip::toHex__ *ipaddr*

    This command converts dotted form ip into a hexadecimal number\.

        % ::ip::toHex 1\.1\.1\.0
        0x01010100

  - <a name='16'></a>__::ip::maskToInt__ *ipmask*

    This command convert an ipmask in either dotted \(255\.255\.255\.0\) form or mask
    length form \(24\) into an integer number\.

        ::ip::maskToInt 24
        4294967040

  - <a name='17'></a>__::ip::broadcastAddress__ *prefix* ?__\-ipv4__?

    This commands returns a broadcast address in dotted form for the given route
    *prefix*, either in the form "addr/mask", or in native form\. The result is
    in dotted form\.

        ::ip::broadcastAddress 1\.1\.1\.0/24
        1\.1\.1\.255

        ::ip::broadcastAddress \{0x01010100 0xffffff00\}
        0x010101ff

  - <a name='18'></a>__::ip::maskToLength__ *dottedMask*&#124;*integerMask*&#124;*hexMask* ?__\-ipv4__?

    This command converts the dotted or integer form of an ipmask to the mask
    length form\.

        ::ip::maskToLength 0xffffff00 \-ipv4
        24

        % ::ip::maskToLength 255\.255\.255\.0
        24

  - <a name='19'></a>__::ip::lengthToMask__ *maskLength* ?__\-ipv4__?

    This command converts an ipmask in mask length form to its dotted form\.

        ::ip::lengthToMask 24
        255\.255\.255\.0

  - <a name='20'></a>__::ip::nextNet__ *ipaddr* *ipmask* ?*count*? ?__\-ipv4__?

    This command returns an ipaddress in the same position in the *count* next
    network\. The default value for *count* is __1__\.

    The address can be specified as either integer number or in dotted form\. The
    mask can be specified as either integer number, dotted form, or mask length
    form\.

    The result is in hex form\.

  - <a name='21'></a>__::ip::isOverlap__ *prefix* *prefix*\.\.\.

    This command checks if the given ip prefixes overlap\. All arguments are in
    dotted "addr/mask" form\. All arguments after the first prefix are compared
    against the first prefix\. The result is a boolean value\. It is true if an
    overlap was found for any of the prefixes\.

        % ::ip::isOverlap 1\.1\.1\.0/24 2\.1\.0\.1/32
        0

        ::ip::isOverlap 1\.1\.1\.0/24 2\.1\.0\.1/32 1\.1\.1\.1/32
        1

  - <a name='22'></a>__::ip::isOverlapNative__ ?__\-all__? ?__\-inline__? ?__\-ipv4__? *hexipaddr* *hexipmask* *hexiplist*

    This command is similar to __::ip::isOverlap__, however the arguments
    are in the native form, and the form of the result is under greater control
    of the caller\. If the option __\-all__ is specified it checks all

        The first overlapping prefix, or an empoty string if there is none\.

      * \-all \-inline

        A list containing the prefixes of all overlaps found, or an empty list
        if there are none\.

        % ::ip::isOverlapNative 0x01010100 0xffffff00 \{\{0x02010001 0xffffffff\}\}
        0

        % ::ip::isOverlapNative 0x01010100 0xffffff00 \{\{0x02010001 0xffffffff\} \{0x01010101 0xffffffff\}\}
        2

  - <a name='23'></a>__::ip::ipToLayer2Multicast__ *ipaddr*

    This command an converts ipv4 address in dotted form into a layer 2
    multicast address, also in dotted form\.

        % ::ip::ipToLayer2Multicast 224\.0\.0\.2
        01\.00\.5e\.00\.00\.02

  - <a name='24'></a>__::ip::ipHostFromPrefix__ *prefix* ?__\-exclude__ *prefixExcludeList*?

    This command returns a host address from a prefix in the form
    "ipaddr/masklen", also making sure that the result is not an address found
    in the *prefixExcludeList*\. The result is an ip address in dotted form\.

        %::ip::ipHostFromPrefix  1\.1\.1\.5/24
        1\.1\.1\.1

        %::ip::ipHostFromPrefix  1\.1\.1\.1/32
        1\.1\.1\.1

  - <a name='25'></a>__::ip::reduceToAggregates__ *prefixlist*

    This command finds nets that overlap and filters out the more specific nets\.
    The prefixes are in either addr/mask form or in native format\. The result is
    a list containing the non\-overlapping ip prefixes from the input\.

        % ::ip::reduceToAggregates \{1\.1\.1\.0/24 1\.1\.0\.0/8  2\.1\.1\.0/24 1\.1\.1\.1/32 \}
        1\.0\.0\.0/8 2\.1\.1\.0/24

  - <a name='26'></a>__::ip::longestPrefixMatch__ *ipaddr* *prefixlist* ?__\-ipv4__?

    This command finds longest prefix match from set of prefixes, given a
    specific host address\. The prefixes in the list are in either native or
    dotted form, whereas the host address is in either ipprefix format, dotted
    form, or integer form\. The result is the prefix which is the most specific
    match to the host address\.

        % ::ip::longestPrefixMatch 1\.1\.1\.1 \{1\.1\.1\.0/24 1\.0\.0\.0/8  2\.1\.1\.0/24 1\.1\.1\.0/28 \}
        1\.1\.1\.0/28

  - <a name='27'></a>__::ip::collapse__ *prefixlist*

    This commands takes a list of prefixes and returns a list prefixes with the
    largest possible subnet masks covering the input, in this manner collapsing
    adjacent prefixes into larger ranges\.

    This is different from __::ip::reduceToAggregates__ in that the latter
    only removes specific nets from a list when they are covered by other
    elements of the input whereas this command actively merges nets into larger
    ranges when they are adjacent to each other\.

        % ::ip::collapse \{1\.2\.2\.0/24 1\.2\.3\.0/24\}
        1\.2\.2\.0/23

  - <a name='28'></a>__::ip::subtract__ *prefixlist*

    This command takes a list of prefixes, some of which are prefixed by a dash\.
    These latter *negative* prefixes are used to punch holes into the ranges
    described by the other, *positive*, prefixes\. I\.e\. the negative prefixes
    are subtracted frrom the positive ones, resulting in a larger list of
    describes describing the covered ranges only as positives\.

# <a name='section3'></a>EXAMPLES

    % ip::version ::1
    6
    % ip::version 127\.0\.0\.1
    4

    % ip::normalize 127/8
    127\.0\.0\.0/8
    % ip::contract 192\.168\.0\.0
    192\.168
    %
    % ip::normalize fec0::1
    fec0:0000:0000:0000:0000:0000:0000:0001
    % ip::contract fec0:0000:0000:0000:0000:0000:0000:0001
    fec0::1

    % ip::equal 192\.168\.0\.4/16 192\.168\.0\.0/16
    1
    % ip::equal fec0::1/10 fec0::fe01/10
    1

# <a name='section4'></a>REFERENCES

  1. Postel, J\. "Internet Protocol\." RFC 791, September 1981,

The basic unit __docstrip__ operates on are the *lines* of a master source
file\. Extraction consists of selecting some of these lines to be copied from
input text to output text\. The basic distinction is that between *code lines*
\(which are copied and do not begin with a percent character\) and *comment
lines* \(which begin with a percent character and are not copied\)\.

    docstrip::extract \[join \{
      \{% comment\}
      \{% more comment \!"\#$%&/\(\}
      \{some command\}
      \{ % blah $blah "Not a comment\."\}
      \{% abc; this is comment\}
      \{\# def; this is code\}
      \{ghi\}
      \{% jkl\}
    \} \\n\] \{\}

returns the same sequence of lines as

    join \{
      \{some command\}
      \{ % blah $blah "Not a comment\."\}
      \{\# def; this is code\}
      \{ghi\} ""
    \} \\n

It does not matter to __docstrip__ what format is used for the documentation
in the comment lines, but in order to do better than plain text comments, one
typically uses some markup language\. Most commonly LaTeX is used, as that is a
very established standard and also provides the best support for mathematical
formulae, but the __docstrip::util__ package also gives some support for
*[doctools](\.\./\.\./\.\./\.\./index\.md\#doctools)*\-like markup\.

line is one of

    '%' '<' STARSLASH EXPRESSION '>'
    '%' '<' PLUSMINUS EXPRESSION '>' CODE

where

    STARSLASH  ::=  '\*' &#124; '/'
    PLUSMINUS  ::=  &#124; '\+' &#124; '\-'
    EXPRESSION ::= SECONDARY &#124; SECONDARY ',' EXPRESSION
                 &#124; SECONDARY '&#124;' EXPRESSION
    SECONDARY  ::= PRIMARY &#124; PRIMARY '&' SECONDARY
    PRIMARY    ::= TERMINAL &#124; '\!' PRIMARY &#124; '\(' EXPRESSION '\)'
    CODE       ::= \{ any character except end\-of\-line \}

Comma and vertical bar both denote 'or'\. Ampersand denotes 'and'\. Exclamation
mark denotes 'not'\. A TERMINAL can be any nonempty string of characters not
containing '>', '&', '&#124;', comma, '\(', or '\)', although the __docstrip__
manual is a bit restrictive and only guarantees proper operation for strings of
letters \(although even the LaTeX core sources make heavy use also of digits in
TERMINALs\)\. The second argument of __docstrip::extract__ is the list of
those TERMINALs that should count as having the value 'true'; all other
TERMINALs count as being 'false' when guard expressions are evaluated\.

In the case of a '%<\**EXPRESSION*>' guard, the lines guarded are all lines up
to the next '%</*EXPRESSION*>' guard with the same *EXPRESSION* \(compared as
strings\)\. The blocks of code delimited by such '\*' and '/' guard lines must be
properly nested\.

    set text \[join \{
       \{begin\}
       \{%<\*foo>\}
       \{1\}
       \{%<\*bar>\}
       \{2\}
       \{%</bar>\}
       \{%<\*\!bar>\}
       \{3\}
       \{%</\!bar>\}
       \{4\}
       \{%</foo>\}
       \{5\}
       \{%<\*bar>\}
       \{6\}
       \{%</bar>\}
       \{end\}
    \} \\n\]
    set res \[docstrip::extract $text foo\]
    append res \[docstrip::extract $text \{foo bar\}\]
    append res \[docstrip::extract $text bar\]

sets $res to the result of

    join \{
       \{begin\}
       \{1\}
       \{3\}
       \{4\}
       \{5\}
       \{end\}
       \{begin\}
       \{1\}
       \{2\}
       \{4\}
       \{5\}
       \{6\}
       \{end\}
       \{begin\}
       \{5\}
       \{6\}
       \{end\} ""
    \} \\n

In guard lines without a '\*', '/', '\+', or '\-' modifier after the '%<', the
guard applies only to the CODE following the '>' on that single line\. A '\+'
modifier is equivalent to no modifier\. A '\-' modifier is like the case with no
modifier, but the expression is implicitly negated, i\.e\., the CODE of a '%<\-'
guard line is only included if the expression evaluates to false\.

Metacomment lines are "comment lines which should not be stripped away", but be
extracted like code lines; these are sometimes used for copyright notices and
similar material\. The '%%' prefix is however not kept, but substituted by the
current __\-metaprefix__, which is customarily set to some "comment until end
of line" character \(or character sequence\) of the language of the code being
extracted\.

    set text \[join \{
       \{begin\}
       \{%<foo> foo\}
       \{%<\+foo>plusfoo\}
       \{%<\-foo>minusfoo\}
       \{middle\}
       \{%% some metacomment\}
       \{%<\*foo>\}
       \{%%another metacomment\}
       \{%</foo>\}
       \{end\}
    \} \\n\]
    set res \[docstrip::extract $text foo \-metaprefix \{\# \}\]
    append res \[docstrip::extract $text bar \-metaprefix \{\#\}\]

sets $res to the result of

    join \{
       \{begin\}
       \{ foo\}
       \{plusfoo\}
       \{middle\}
       \{\#  some metacomment\}
       \{\# another metacomment\}
       \{end\}
       \{begin\}
       \{minusfoo\}
       \{middle\}
       \{\# some metacomment\}
       \{end\} ""
    \} \\n

Verbatim guards can be used to force code line interpretation of a block of
lines even if some of them happen to look like any other type of lines to
docstrip\. A verbatim guard has the form '%<<*END\-TAG*' and the verbatim block
is terminated by the first line that is exactly '%*END\-TAG*'\.

    set text \[join \{
       \{begin\}
       \{%<\*myblock>\}
       \{some stupid\(\)\}
       \{   \#computer<program>\}
       \{%<<QQQ\-98765\}
       \{% These three lines are copied verbatim \(including percents\}
       \{%% even if \-metaprefix is something different than %%\)\.\}
       \{%</myblock>\}
       \{%QQQ\-98765\}
       \{   using\*strange@programming<language>\}
       \{%</myblock>\}
       \{end\}
    \} \\n\]
    set res \[docstrip::extract $text myblock \-metaprefix \{\# \}\]
    append res \[docstrip::extract $text \{\}\]

sets $res to the result of

    join \{
       \{begin\}
       \{some stupid\(\)\}
       \{   \#computer<program>\}
       \{% These three lines are copied verbatim \(including percents\}
       \{%% even if \-metaprefix is something different than %%\)\.\}
       \{%</myblock>\}
       \{   using\*strange@programming<language>\}
       \{end\}
       \{begin\}
       \{end\} ""
    \} \\n

The processing of verbatim guards takes place also inside blocks of lines which
due to some outer block guard will not be copied\.

The final piece of __docstrip__ syntax is that extraction stops at a line
that is exactly "\\endinput"; this is often used to avoid copying random
whitespace at the end of a file\. In the unlikely case that one wants such a code

that files employing that document format are given the suffix "\.ddt", to
distinguish them from the more traditional LaTeX\-based "\.dtx" files\.

Master source files with "\.dtx" extension are usually set up so that they can be
typeset directly by __[latex](\.\./\.\./\.\./\.\./index\.md\#latex)__ without any
support from other files\. This is achieved by beginning the file with the lines

    % \\iffalse
    %<\*driver>
    \\documentclass\{tclldoc\}
    \\begin\{document\}
    \\DocInput\{*filename\.dtx*\}
    \\end\{document\}
    %</driver>
    % \\fi

or some variation thereof\. The trick is that the file gets read twice\. With
normal LaTeX reading rules, the first two lines are comments and therefore
ignored\. The third line is the document preamble, the fourth line begins the
document body, and the sixth line ends the document, so LaTeX stops there —
non\-comments below that point in the file are never subjected to the normal
LaTeX reading rules\. Before that, however, the \\DocInput command on the fifth

terminal '__docstrip\.tcl::catalogue__'\. This supports both the style of
collecting all catalogue lines in one place and the style of putting each
catalogue line in close proximity of the code that it declares\.

Putting catalogue entries next to the code they declare may look as follows

    %    First there's the catalogue entry
    %    \\begin\{tcl\}
    %<docstrip\.tcl::catalogue>pkgProvide foo::bar 1\.0 \{foobar load\}
    %    \\end\{tcl\}
    %    second a metacomment used to include a copyright message
    %    \\begin\{macrocode\}
    %<\*foobar>
    %% This file is placed in the public domain\.
    %    \\end\{macrocode\}
    %    third the package implementation
    %    \\begin\{tcl\}
    namespace eval foo::bar \{
       \# \.\.\. some clever piece of Tcl code elided \.\.\.
    %    \\end\{tcl\}
    %    which at some point may have variant code to make use of a
    %    &#124;load&#124;able extension
    %    \\begin\{tcl\}
    %<\*load>
       load \[file rootname \[info script\]\]\[info sharedlibextension\]
    %</load>
    %<\*\!load>
       \# \.\.\. even more clever scripted counterpart of the extension
       \# also elided \.\.\.
    %</\!load>
    \}

    %</foobar>
    %    \\end\{tcl\}
    %    and that's it\!

The corresponding set\-up with __pkgIndex__ would be

    %    First there's the catalogue entry
    %    \\begin\{tcl\}
    %<docstrip\.tcl::catalogue>pkgIndex foobar load
    %    \\end\{tcl\}
    %    second a metacomment used to include a copyright message
    %    \\begin\{tcl\}
    %<\*foobar>
    %% This file is placed in the public domain\.
    %    \\end\{tcl\}
    %    third the package implementation
    %    \\begin\{tcl\}
    package provide foo::bar 1\.0
    namespace eval foo::bar \{
       \# \.\.\. some clever piece of Tcl code elided \.\.\.
    %    \\end\{tcl\}
    %    which at some point may have variant code to make use of a
    %    &#124;load&#124;able extension
    %    \\begin\{tcl\}
    %<\*load>
       load \[file rootname \[info script\]\]\[info sharedlibextension\]
    %</load>
    %<\*\!load>
       \# \.\.\. even more clever scripted counterpart of the extension
       \# also elided \.\.\.
    %</\!load>
    \}

    %</foobar>
    %    \\end\{tcl\}
    %    and that's it\!

  - <a name='4'></a>__docstrip::util::index\_from\_catalogue__ *dir* *pattern* ?*option* *value* \.\.\.?

    This command is a sibling of the standard __pkg\_mkIndex__ command, in
    that it adds package entries to "pkgIndex\.tcl" files\. The difference is that
    it indexes __[docstrip](docstrip\.md)__\-style source files rather
    than raw "\.tcl" or loadable library files\. Only packages listed in the

    An existing file of the same name as one to be created will be overwritten\.

  - <a name='6'></a>__docstrip::util::classical\_preamble__ *metaprefix* *message* *target* ?*source* *terminals* \.\.\.?

    This command returns a preamble in the classical
    __[docstrip](docstrip\.md)__ style

    \#\#
    \#\# This is \`TARGET',
    \#\# generated by the docstrip::util package\.
    \#\#
    \#\# The original source files were:
    \#\#
    \#\# SOURCE \(with options: \`foo,bar'\)
    \#\#
    \#\# Some message line 1
    \#\# line2
    \#\# line3

    if called as

    docstrip::util::classical\_preamble \{\#\#\}\\
      "\\nSome message line 1\\nline2\\nline3" TARGET SOURCE \{foo bar\}

    The command supports preambles for files generated from multiple sources,
    even though __modules\_from\_catalogue__ at present does not need that\.

  - <a name='7'></a>__docstrip::util::classical\_postamble__ *metaprefix* *message* *target* ?*source* *terminals* \.\.\.?

    This command returns a postamble in the classical
    __[docstrip](docstrip\.md)__ style

    \#\# Some message line 1
    \#\# line2
    \#\# line3
    \#\#
    \#\# End of file \`TARGET'\.

    if called as

    docstrip::util::classical\_postamble \{\#\#\}\\
      "Some message line 1\\nline2\\nline3" TARGET SOURCE \{foo bar\}

    In other words, the *source* and *terminals* arguments are ignored, but
    supported for symmetry with __classical\_preamble__\.

  - <a name='8'></a>__docstrip::util::packages\_provided__ *text* ?*setup\-script*?

    This command returns a list where every even index element is the name of a

    *setup\-script* is evaluated in the local context of the
    __packages\_provided__ procedure just before the *text* is processed\.
    At that time, the name of the slave command for the safe interpreter that
    will do this processing is kept in the local variable __c__\. To for
    example copy the contents of the __::env__ array to the safe
    interpreter, one might use a *setup\-script* of

    $c eval \[list array set env \[array get ::env\]\]

# <a name='section3'></a>Source processing commands

Unlike the previous group of commands, which would use __docstrip::extract__
to extract some code lines and then process those further, the following
commands operate on text consisting of all types of lines\.

        __emph__asised\.

    At the time of writing, no project has employed
    __[doctools](\.\./doctools/doctools\.md)__ markup in master source
    files, so experience of what works well is not available\. A source file
    could however look as follows

    % \[manpage\_begin gcd n 1\.0\]
    % \[keywords divisor\]
    % \[keywords math\]
    % \[moddesc \{Greatest Common Divisor\}\]
    % \[require gcd \[opt 1\.0\]\]
    % \[description\]
    %
    % \[list\_begin definitions\]
    % \[call \[cmd gcd\] \[arg a\] \[arg b\]\]
    %   The \[cmd gcd\] procedure takes two arguments \[arg a\] and \[arg b\] which
    %   must be integers and returns their greatest common divisor\.
    proc gcd \{a b\} \{
    %   The first step is to take the absolute values of the arguments\.
    %   This relieves us of having to worry about how signs will be treated
    %   by the remainder operation\.
       set a \[expr \{abs\($a\)\}\]
       set b \[expr \{abs\($b\)\}\]
    %   The next line does all of Euclid's algorithm\! We can make do
    %   without a temporary variable, since $a is substituted before the
    %   \[lb\]set a $b\[rb\] and thus continues to hold a reference to the
    %   "old" value of \[var a\]\.
       while \{$b>0\} \{ set b \[expr \{ $a % \[set a $b\] \}\] \}
    %   In Tcl 8\.3 we might want to use \[cmd set\] instead of \[cmd return\]
    %   to get the slight advantage of byte\-compilation\.
    %<tcl83>  set a
    %<\!tcl83>   return $a
    \}

    % \[list\_end\]
    %
    % \[manpage\_end\]

    If the above text is fed through __docstrip::util::ddt2man__ then the
    result will be a syntactically correct
    __[doctools](\.\./doctools/doctools\.md)__ manpage, even though its
    purpose is a bit different\.

    It is suggested that master source code files with

    the header of each hunk specifies which case is at hand\. It is normally
    necessary to manually review both the return value from
    __[patch](\.\./\.\./\.\./\.\./index\.md\#patch)__ and the patched text itself,
    as this command cannot adjust comment lines to match new content\.

    An example use would look like

    set sourceL \[split \[docstrip::util::thefile from\.dtx\] \\n\]
    set terminals \{foo bar baz\}
    set fromtext \[docstrip::util::thefile from\.tcl\]
    set difftext \[exec diff \-\-unified from\.tcl to\.tcl\]
    set leftover \[docstrip::util::patch sourceL $terminals $fromtext\\
      \[docstrip::util::import\_unidiff $difftext\] \-metaprefix \{\#\}\]
    set F \[open to\.dtx w\]; puts $F \[join $sourceL \\n\]; close $F
    return $leftover

    Here, "from\.dtx" was used as source for "from\.tcl", which someone modified
    into "to\.tcl"\. We're trying to construct a "to\.dtx" which can be used as
    source for "to\.tcl"\.

  - <a name='12'></a>__docstrip::util::thefile__ *filename* ?*option* *value* \.\.\.?

    ChangeLog\. Each element/entry is then a list of three elements describing
    the date of the entry, its author, and the comments made, in this order\. The
    last item in each element/entry, the comments, is a list of sections\. Each
    section is described by a list containing two elements, a list of file
    names, and a string containing the true comment associated with the files of
    the section\.

            \{
        	\{


        	    date
        	    author
        	    \{
        		\{


        		    \{file \.\.\.\}
        		    commenttext
        		\}

        		\.\.\.
        	    \}
        	\}


        	\{\.\.\.\}
            \}


  - <a name='2'></a>__::doctools::changelog::flatten__ *entries*

    This command converts a list of entries as generated by __change::scan__
    above into a simpler list of plain text blocks each containing all the
    information of a single entry\.

interspersed between them, except for whitespace\.

Each markup command is a Tcl command surrounded by a matching pair of __\[__
and __\]__\. Inside of these delimiters the usual rules for a Tcl command
apply with regard to word quotation, nested commands, continuation lines, etc\.
I\.e\.

    \.\.\. \[key \{markup language\}\] \.\.\.

    \.\.\. \[manpage thefile \\\\
            \{file description\}\] \.\.\.

## <a name='subsection2'></a>Basic structure

The most simple document which can be written in docidx is

    \[index\_begin GROUPTITLE TITLE\]
    \[index\_end\]

Not very useful, but valid\. This also shows us that all docidx documents consist
of only one part where we will list all keys and their references\.

A more useful index will contain at least keywords, or short 'keys', i\.e\. the
phrases which were indexed\. So:

    \[index\_begin GROUPTITLE TITLE\]
    \[__key markup__\]
    \[__key \{semantic markup\}\]__\]
    \[__key \{docidx markup\}__\]
    \[__key \{docidx language\}__\]
    \[__key \{docidx commands\}__\]
    \[index\_end\]

In the above example the command __key__ is used to declare the keyword
phrases we wish to be part of the index\.

However a truly useful index does not only list the keyword phrases, but will
also contain references to documents associated with the keywords\. Here is a
made\-up index for all the manpages in the module
*[base64](\.\./\.\./\.\./\.\./index\.md\#base64)*:

    \[index\_begin tcllib/base64 \{De\- & Encoding\}\]
    \[key base64\]
    \[__manpage base64__\]
    \[key encoding\]
    \[__manpage base64__\]
    \[__manpage uuencode__\]
    \[__manpage yencode__\]
    \[key uuencode\]
    \[__manpage uuencode__\]
    \[key yEnc\]
    \[__manpage yencode__\]
    \[key ydecode\]
    \[__manpage yencode__\]
    \[key yencode\]
    \[__manpage yencode__\]
    \[index\_end\]

In the above example the command
__[manpage](\.\./\.\./\.\./\.\./index\.md\#manpage)__ is used to insert references
to documents, using symbolic file names, with each command belonging to the last
__key__ command coming before it\.

The other command to insert references is

to be used before the __index\_begin__ command opening the document\.

Instead of only whitespace the two templating commands __include__ and
__vset__ are also allowed, to enable the writer to either set and/or import
configuration settings relevant to the table of contents\. I\.e\. it is possible to
write

    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \[index\_begin GROUPTITLE TITLE\]
    \.\.\.
    \[index\_end\]

Even more important, these two commands are allowed anywhere where a markup
command is allowed, without regard for any other structure\.

    \[index\_begin GROUPTITLE TITLE\]
    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \.\.\.
    \[index\_end\]

The only restriction __include__ has to obey is that the contents of the
included file must be valid at the place of the inclusion\. I\.e\. a file included
before __index\_begin__ may contain only the templating commands __vset__
and __include__, a file included after a key may contain only manape or url
references, and other keys, etc\.

## <a name='subsection4'></a>Escapes

Beyond the 6 commands shown so far we have two more available\. However their
function is not the marking up of index structure, but the insertion of
characters, namely __\[__ and __\]__\. These commands, __lb__ and
__rb__ respectively, are required because our use of \[ and \] to bracket
markup commands makes it impossible to directly use \[ and \] within the text\.

Our example of their use are the sources of the last sentence in the previous
paragraph, with some highlighting added\.

    \.\.\.
    These commands, \[cmd lb\] and \[cmd lb\] respectively, are required
    because our use of \[__lb__\] and \[__rb__\] to bracket markup commands makes it
    impossible to directly use \[__lb__\] and \[__rb__\] within the text\.
    \.\.\.

# <a name='section2'></a>FURTHER READING

Now that this document has been digested the reader, assumed to be a *writer*
of documentation should be fortified enough to be able to understand the formal
*[docidx language syntax](docidx\_lang\_syntax\.md)* specification as well\.
From here on out the *[docidx language command

  1. The construct \{ X \} stands for zero or more occurrences of X\.

  1. The construct \[ X \] stands for zero or one occurrence of X\.

The syntax:

    index     = defs
                INDEX\_BEGIN
                \[ contents \]
                INDEX\_END
                \{ <WHITE> \}

    defs      = \{ INCLUDE &#124; VSET &#124; <WHITE> \}
    contents  = keyword \{ keyword \}

    keyword   = defs KEY ref \{ ref \}
    ref       = MANPAGE &#124; URL &#124; defs

At last a rule we were unable to capture in the EBNF syntax, as it is about the
arguments of the markup commands, something which is not modeled here\.

  1. The arguments of all markup commands have to be plain text, and/or text
     markup commands, i\.e\. one of

  1. initialize and shutdown each pass

  1. query and initialize engine parameters

After the plugin has been loaded and the frontend commands are established the
commands will be called in the following sequence:

    idx\_numpasses \-> n
    idx\_listvariables \-> vars

    idx\_varset var1 value1
    idx\_varset var2 value2
    \.\.\.
    idx\_varset varK valueK
    idx\_initialize
    idx\_setup 1
    \.\.\.
    idx\_setup 2
    \.\.\.
    \.\.\.
    idx\_setup n
    \.\.\.
    idx\_postprocess
    idx\_shutdown
    \.\.\.

I\.e\. first the number of passes and the set of available engine parameters is
established, followed by calls setting the parameters\. That second part is
optional\.

After that the plugin is initialized, the specified number of passes executed,
the final result run through a global post processing step and at last the

interspersed between them, except for whitespace\.

Each markup command is a Tcl command surrounded by a matching pair of __\[__
and __\]__\. Inside of these delimiters the usual rules for a Tcl command
apply with regard to word quotation, nested commands, continuation lines, etc\.
I\.e\.

    \.\.\. \[division\_start \{Appendix 1\}\] \.\.\.

    \.\.\. \[item thefile \\\\
            label \{file description\}\] \.\.\.

## <a name='subsection2'></a>Basic structure

The most simple document which can be written in doctoc is

    \[toc\_begin GROUPTITLE TITLE\]
    \[toc\_end\]

This also shows us that all doctoc documents consist of only one part where we
will list *items* and *divisions*\.

The user is free to mix these as she sees fit\. This is a change from version 1
of the language, which did not allow this mixing, but only the use of either a
series of items or a series of divisions\.

Symbolic names are used to preserve the convertibility of this format to any
output format\. The actual name of any file will be inserted by the chosen
formatting engine when converting the input, based on a mapping from symbolic to
actual names given to the engine\.

Here a made up example for a table of contents of this document:

    \[toc\_begin Doctoc \{Language Introduction\}\]
    \[__item 1 DESCRIPTION__\]
    \[__item 1\.1 \{Basic structure\}__\]
    \[__item 1\.2 Items__\]
    \[__item 1\.3 Divisions__\]
    \[__item 2 \{FURTHER READING\}__\]
    \[toc\_end\]

## <a name='subsection4'></a>Divisions

One thing of notice in the last example in the previous section is that the
referenced sections actually had a nested structure, something which was
expressed in the item labels, by using a common prefix for all the sections
nested under section 1\.

  - __division\_end__

    This command closes the last opened and not yet closed division\.

Using this we can recast the last example like this

    \[toc\_begin Doctoc \{Language Introduction\}\]
    \[__division\_start DESCRIPTION__\]
    \[item 1 \{Basic structure\}\]
    \[item 2 Items\]
    \[item 3 Divisions\]
    \[__division\_end__\]
    \[__division\_start \{FURTHER READING\}__\]
    \[__division\_end__\]
    \[toc\_end\]

Or, to demonstrate deeper nesting

    \[toc\_begin Doctoc \{Language Introduction\}\]
    \[__division\_start DESCRIPTION__\]
    \[__division\_start \{Basic structure\}__\]
    \[item 1 Do\]
    \[item 2 Re\]
    \[__division\_end__\]
    \[__division\_start Items__\]
    \[item a Fi\]
    \[item b Fo\]
    \[item c Fa\]
    \[__division\_end__\]
    \[__division\_start Divisions__\]
    \[item 1 Sub\]
    \[item 1 Zero\]
    \[__division\_end__\]
    \[__division\_end__\]
    \[__division\_start \{FURTHER READING\}__\]
    \[__division\_end__\]
    \[toc\_end\]

And do not forget, it is possible to freely mix items and divisions, and to have
empty divisions\.

    \[toc\_begin Doctoc \{Language Introduction\}\]
    \[item 1 Do\]
    \[__division\_start DESCRIPTION__\]
    \[__division\_start \{Basic structure\}__\]
    \[item 2 Re\]
    \[__division\_end__\]
    \[item a Fi\]
    \[__division\_start Items__\]
    \[item b Fo\]
    \[item c Fa\]
    \[__division\_end__\]
    \[__division\_start Divisions__\]
    \[__division\_end__\]
    \[__division\_end__\]
    \[__division\_start \{FURTHER READING\}__\]
    \[__division\_end__\]
    \[toc\_end\]

## <a name='subsection5'></a>Advanced structure

In all previous examples we fudged a bit regarding the markup actually allowed
to be used before the __toc\_begin__ command opening the document\.

Instead of only whitespace the two templating commands __include__ and
__vset__ are also allowed, to enable the writer to either set and/or import
configuration settings relevant to the table of contents\. I\.e\. it is possible to
write

    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \[toc\_begin GROUPTITLE TITLE\]
    \.\.\.
    \[toc\_end\]

Even more important, these two commands are allowed anywhere where a markup
command is allowed, without regard for any other structure\.

    \[toc\_begin GROUPTITLE TITLE\]
    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \.\.\.
    \[toc\_end\]

The only restriction __include__ has to obey is that the contents of the
included file must be valid at the place of the inclusion\. I\.e\. a file included
before __toc\_begin__ may contain only the templating commands __vset__
and __include__, a file included in a division may contain only items or
divisions commands, etc\.

## <a name='subsection6'></a>Escapes

Beyond the 6 commands shown so far we have two more available\. However their
function is not the marking up of toc structure, but the insertion of
characters, namely __\[__ and __\]__\. These commands, __lb__ and
__rb__ respectively, are required because our use of \[ and \] to bracket
markup commands makes it impossible to directly use \[ and \] within the text\.

Our example of their use are the sources of the last sentence in the previous
paragraph, with some highlighting added\.

    \.\.\.
    These commands, \[cmd lb\] and \[cmd lb\] respectively, are required
    because our use of \[__lb__\] and \[__rb__\] to bracket markup commands makes it
    impossible to directly use \[__lb__\] and \[__rb__\] within the text\.
    \.\.\.

# <a name='section2'></a>FURTHER READING

Now that this document has been digested the reader, assumed to be a *writer*
of documentation should be fortified enough to be able to understand the formal
*[doctoc language syntax](doctoc\_lang\_syntax\.md)* specification as well\.
From here on out the *[doctoc language command

  1. The construct \{ X \} stands for zero or more occurrences of X\.

  1. The construct \[ X \] stands for zero or one occurrence of X\.

The syntax:

    toc       = defs
                TOC\_BEGIN
                contents
                TOC\_END
                \{ <WHITE> \}

    defs      = \{ INCLUDE &#124; VSET &#124; <WHITE> \}
    contents  = \{ defs entry \} \[ defs \]

    entry     = ITEM &#124; division

    division  = DIVISION\_START
                contents
                DIVISION\_END

# <a name='section5'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *doctools* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

  1. initialize and shutdown each pass

  1. query and initialize engine parameters

After the plugin has been loaded and the frontend commands are established the
commands will be called in the following sequence:

    toc\_numpasses \-> n
    toc\_listvariables \-> vars

    toc\_varset var1 value1
    toc\_varset var2 value2
    \.\.\.
    toc\_varset varK valueK
    toc\_initialize
    toc\_setup 1
    \.\.\.
    toc\_setup 2
    \.\.\.
    \.\.\.
    toc\_setup n
    \.\.\.
    toc\_postprocess
    toc\_shutdown
    \.\.\.

I\.e\. first the number of passes and the set of available engine parameters is
established, followed by calls setting the parameters\. That second part is
optional\.

After that the plugin is initialized, the specified number of passes executed,
the final result run through a global post processing step and at last the

[//000000001]: # (doctools \- Documentation tools)
[//000000002]: # (Generated from file 'doctools\.man' by tcllib/doctools with format 'markdown')
[//000000003]: # (Copyright &copy; 2003\-2019 Andreas Kupries <andreas\_kupries@users\.sourceforge\.net>)
[//000000004]: # (doctools\(n\) 1\.5\.1 tcllib "Documentation tools")

<hr> [ <a href="../../../../toc.md">Main Table Of Contents</a> &#124; <a
href="../../../toc.md">Table Of Contents</a> &#124; <a
href="../../../../index.md">Keyword Index</a> &#124; <a
href="../../../../toc0.md">Categories</a> &#124; <a
href="../../../../toc1.md">Modules</a> &#124; <a
href="../../../../toc2.md">Applications</a> ] <hr>

  - [Category](#category)

  - [Copyright](#copyright)

# <a name='synopsis'></a>SYNOPSIS

package require Tcl 8\.2  
package require doctools ?1\.5\.1?  

[__::doctools::new__ *objectName* ?*option value*\.\.\.?](#1)  
[__::doctools::help__](#2)  
[__::doctools::search__ *path*](#3)  
[__objectName__ __method__ ?*arg arg \.\.\.*?](#4)  
[*objectName* __configure__](#5)  
[*objectName* __configure__ *option*](#6)  

consists primarily of text, with markup commands embedded into it\.

Each markup command is a Tcl command surrounded by a matching pair of __\[__
and __\]__\. Inside of these delimiters the usual rules for a Tcl command
apply with regard to word quotation, nested commands, continuation lines, etc\.
I\.e\.

    \.\.\. \[list\_begin enumerated\] \.\.\.

    \.\.\. \[call \[cmd foo\] \\\\
            \[arg bar\]\] \.\.\.

    \.\.\. \[term \{complex concept\}\] \.\.\.

    \.\.\. \[opt "\[arg key\] \[arg value\]"\] \.\.\.

## <a name='subsection2'></a>Basic structure

The most simple document which can be written in doctools is

        \[manpage\_begin NAME SECTION VERSION\]
    \[see\_also doctools\_intro\]
    \[see\_also doctools\_lang\_cmdref\]
    \[see\_also doctools\_lang\_faq\]
    \[see\_also doctools\_lang\_syntax\]
    \[keywords \{doctools commands\}\]
    \[keywords \{doctools language\}\]
    \[keywords \{doctools markup\}\]
    \[keywords \{doctools syntax\}\]
    \[keywords markup\]
    \[keywords \{semantic markup\}\]
        \[description\]
        \[vset CATEGORY doctools\]
    \[include \.\./doctools2base/include/feedback\.inc\]
    \[manpage\_end\]

This also shows us that all doctools documents are split into two parts, the
*header* and the *body*\. Everything coming before \[__description__\]
belongs to the header, and everything coming after belongs to the body, with the
whole document bracketed by the two __manpage\_\*__ commands\. Before and after
these opening and closing commands we have only *whitespace*\.

and they can be used in any order\. However for __titledesc__ and
__moddesc__ only the last occurrence is taken\. For the other two the
specified information is accumulated, in the given order\. Regular text is not
allowed within the header\.

Given the above a less minimal example of a document is

    \[manpage\_begin NAME SECTION VERSION\]
    \[__copyright \{YEAR AUTHOR\}__\]
    \[__titledesc TITLE__\]
    \[__moddesc   MODULE\_TITLE__\]
    \[__require   PACKAGE VERSION__\]
    \[__require   PACKAGE__\]
    \[description\]
    \[manpage\_end\]

Remember that the whitespace is optional\. The document

        \[manpage\_begin NAME SECTION VERSION\]
        \[copyright \{YEAR AUTHOR\}\]\[titledesc TITLE\]\[moddesc MODULE\_TITLE\]
        \[require PACKAGE VERSION\]\[require PACKAGE\]\[description\]
        \[vset CATEGORY doctools\]
    \[include \.\./doctools2base/include/feedback\.inc\]
    \[manpage\_end\]

has the same meaning as the example before\.

On the other hand, if *whitespace* is present it consists not only of any
sequence of characters containing the space character, horizontal and vertical
tabs, carriage return, and newline, but it may contain comment markup as well,
in the form of the __[comment](\.\./\.\./\.\./\.\./index\.md\#comment)__ command\.

    \[__comment \{ \.\.\. \}__\]
    \[manpage\_begin NAME SECTION VERSION\]
    \[copyright \{YEAR AUTHOR\}\]
    \[titledesc TITLE\]
    \[moddesc   MODULE\_TITLE\]\[__comment \{ \.\.\. \}__\]
    \[require   PACKAGE VERSION\]
    \[require   PACKAGE\]
    \[description\]
    \[manpage\_end\]
    \[__comment \{ \.\.\. \}__\]

## <a name='subsection3'></a>Advanced structure

In the simple examples of the last section we fudged a bit regarding the markup
actually allowed to be used before the __manpage\_begin__ command opening the
document\.

Instead of only whitespace the two templating commands __include__ and
__vset__ are also allowed, to enable the writer to either set and/or import
configuration settings relevant to the document\. I\.e\. it is possible to write

    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \[manpage\_begin NAME SECTION VERSION\]
    \[description\]
    \[manpage\_end\]

Even more important, these two commands are allowed anywhere where a markup
command is allowed, without regard for any other structure\. I\.e\. for example in
the header as well\.

    \[manpage\_begin NAME SECTION VERSION\]
    \[__include FILE__\]
    \[__vset VAR VALUE__\]
    \[description\]
    \[manpage\_end\]

The only restriction __include__ has to obey is that the contents of the
included file must be valid at the place of the inclusion\. I\.e\. a file included
before __manpage\_begin__ may contain only the templating commands
__vset__ and __include__, a file included in the header may contain only
header commands, etc\.

The simplest way of structuring the body is through the introduction of
paragraphs\. The command for doing so is __para__\. Each occurrence of this
command closes the previous paragraph and automatically opens the next\. The
first paragraph is automatically opened at the beginning of the body, by
__description__\. In the same manner the last paragraph automatically ends at
__manpage\_end__\.

    \[manpage\_begin NAME SECTION VERSION\]
    \[description\]
     \.\.\.
    \[__para__\]
     \.\.\.
    \[__para__\]
     \.\.\.
    \[manpage\_end\]

Empty paragraphs are ignored\.

A structure coarser than paragraphs are sections, which allow the writer to
split a document into larger, and labeled, pieces\. The command for doing so is
__section__\. Each occurrence of this command closes the previous section and
automatically opens the next, including its first paragraph\. The first section
is automatically opened at the beginning of the body, by __description__
\(This section is labeled "DESCRIPTION"\)\. In the same manner the last section
automatically ends at __manpage\_end__\.

Empty sections are *not* ignored\. We are free to \(not\) use paragraphs within
sections\.

    \[manpage\_begin NAME SECTION VERSION\]
    \[description\]
     \.\.\.
    \[__section \{Section A\}__\]
     \.\.\.
    \[para\]
     \.\.\.
    \[__section \{Section B\}__\]
     \.\.\.
    \[manpage\_end\]

Between sections and paragraphs we have subsections, to split sections\. The
command for doing so is __subsection__\. Each occurrence of this command
closes the previous subsection and automatically opens the next, including its
first paragraph\. A subsection is automatically opened at the beginning of the
body, by __description__, and at the beginning of each section\. In the same
manner the last subsection automatically ends at __manpage\_end__\.

Empty subsections are *not* ignored\. We are free to \(not\) use paragraphs
within subsections\.

    \[manpage\_begin NAME SECTION VERSION\]
    \[description\]
     \.\.\.
    \[section \{Section A\}\]
     \.\.\.
    \[__subsection \{Sub 1\}__\]
     \.\.\.
    \[para\]
     \.\.\.
    \[__subsection \{Sub 2\}__\]
     \.\.\.
    \[section \{Section B\}\]
     \.\.\.
    \[manpage\_end\]

## <a name='subsection5'></a>Text markup

Having handled the overall structure a writer can impose on the document we now
take a closer at the text in a paragraph\.

While most often this is just the unadorned content of the document we do have

    Its argument is a widget name\.

The example demonstrating the use of text markup is an excerpt from the
*[doctools language command reference](doctools\_lang\_cmdref\.md)*, with
some highlighting added\. It shows their use within a block of text, as the
arguments of a list item command \(__call__\), and our ability to nest them\.

    \.\.\.
    \[call \[__cmd arg\_def__\] \[__arg type__\] \[__arg name__\] \[__opt__ \[__arg mode__\]\]\]

    Text structure\. List element\. Argument list\. Automatically closes the
    previous list element\. Specifies the data\-\[__arg type__\] of the described
    argument of a command, its \[__arg name__\] and its i/o\-\[__arg mode__\]\. The
    latter is optional\.
    \.\.\.

## <a name='subsection6'></a>Escapes

Beyond the 20 commands for simple markup shown in the previous section we have
two more available which are technically simple markup\. However their function
is not the marking up of phrases as specific types of things, but the insertion
of characters, namely __\[__ and __\]__\. These commands, __lb__ and
__rb__ respectively, are required because our use of \[ and \] to bracket
markup commands makes it impossible to directly use \[ and \] within the text\.

Our example of their use are the sources of the last sentence in the previous
paragraph, with some highlighting added\.

    \.\.\.
    These commands, \[cmd lb\] and \[cmd lb\] respectively, are required
    because our use of \[__lb__\] and \[__rb__\] to bracket markup commands makes it
    impossible to directly use \[__lb__\] and \[__rb__\] within the text\.
    \.\.\.

## <a name='subsection7'></a>Cross\-references

The last two commands we have to discuss are for the declaration of
cross\-references between documents, explicit and implicit\. They are
__[keywords](\.\./\.\./\.\./\.\./index\.md\#keywords)__ and __see\_also__\. Both
take an arbitrary number of arguments, all of which have to be plain unmarked

whether she wants to have them at the beginning of the body, or at its end,
maybe near the place a keyword is actually defined by the main content, or
considers them as meta data which should be in the header, etc\.

Our example shows the sources for the cross\-references of this document, with
some highlighting added\. Incidentally they are found at the end of the body\.

    \.\.\.
    \[__see\_also doctools\_intro__\]
    \[__see\_also doctools\_lang\_syntax__\]
    \[__see\_also doctools\_lang\_cmdref__\]
    \[__keywords markup \{semantic markup\}__\]
    \[__keywords \{doctools markup\} \{doctools language\}__\]
    \[__keywords \{doctools syntax\} \{doctools commands\}__\]
    \[manpage\_end\]

## <a name='subsection8'></a>Examples

Where ever we can write plain text we can write examples too\. For simple
examples we have the command __example__ which takes a single argument, the
text of the argument\. The example text must not contain markup\. If we wish to
have markup within an example we have to use the 2\-command combination
__example\_begin__ / __example\_end__ instead\.

The first opens an example block, the other closes it, and in between we can
write plain text and use all the regular text markup commands\. Note that text
structure commands are not allowed\. This also means that it is not possible to
embed examples and lists within an example\. On the other hand, we *can* use
templating commands within example blocks to read their contents from a file
\(Remember section [Advanced structure](#subsection3)\)\.

The source for the very first example in this document \(see section
[Fundamentals](#subsection1)\), with some highlighting added, is

    \[__example__ \{
        \.\.\. \[list\_begin enumerated\] \.\.\.
      \}\]

Using __example\_begin__ / __example\_end__ this would look like

    \[__example\_begin__\]
        \.\.\. \[list\_begin enumerated\] \.\.\.
      \[__example\_end__\]

## <a name='subsection9'></a>Lists

Where ever we can write plain text we can write lists too\. The main commands are
__list\_begin__ to start a list, and __list\_end__ to close one\. The
opening command takes an argument specifying the type of list started it, and
this in turn determines which of the eight existing list item commands are

    is a specialized form of a term definition list where the term is the name
    of a configuration option for a widget, with its name and class in the
    option database\.

Our example is the source of the definition list in the previous paragraph, with
most of the content in the middle removed\.

    \.\.\.
    \[__list\_begin__ definitions\]
    \[__def__ \[const arg\]\]

    \(\[cmd arg\_def\]\) This opens an argument \(declaration\) list\. It is a
    specialized form of a definition list where the term is an argument
    name, with its type and i/o\-mode\.

    \[__def__ \[const itemized\]\]

    \(\[cmd item\]\)
    This opens a general itemized list\.

    \.\.\.
    \[__def__ \[const tkoption\]\]

    \(\[cmd tkoption\_def\]\) This opens a widget option \(declaration\) list\. It
    is a specialized form of a definition list where the term is the name
    of a configuration option for a widget, with its name and class in the
    option database\.

    \[__list\_end__\]
    \.\.\.

Note that a list cannot begin in one \(sub\)section and end in another\.
Differently said, \(sub\)section breaks are not allowed within lists and list
items\. An example of this *illegal* construct is

    \.\.\.
    \[list\_begin itemized\]
    \[item\]
    \.\.\.
    \[__section \{ILLEGAL WITHIN THE LIST\}__\]
    \.\.\.
    \[list\_end\]
    \.\.\.

# <a name='section2'></a>FURTHER READING

Now that this document has been digested the reader, assumed to be a *writer*
of documentation should be fortified enough to be able to understand the formal
*[doctools language syntax](doctools\_lang\_syntax\.md)* specification as
well\. From here on out the *[doctools language command

  1. The construct LIST\_BEGIN<X> stands for the markup command
     __list\_begin__ with __X__ as its type argument\.

The syntax:

    manpage = defs
              MANPAGE\_BEGIN
              header
              DESCRIPTION
              body
              MANPAGE\_END
              \{ <WHITE> \}

    defs    = \{ INCLUDE &#124; VSET &#124; <WHITE> \}

    header  = \{ TITLEDESC &#124; MODDESC &#124; COPYRIGHT &#124; REQUIRE &#124; defs &#124; xref \}

    xref    = KEYWORDS &#124; SEE\_ALSO &#124; CATEGORY

    body    = paras \{ SECTION    sbody  \}
    sbody   = paras \{ SUBSECTION ssbody \}
    ssbody  = paras

    paras   = tblock \{ \(PARA &#124; NL\) tblock \}

    tblock  = \{ <TEXT> &#124; defs &#124; markup &#124; xref &#124; an\_example &#124; a\_list \}

    markup  = ARG     &#124; CLASS &#124; CMD     &#124; CONST     &#124; EMPH   &#124; FILE
            &#124; FUN     &#124; LB    &#124; METHOD  &#124; NAMESPACE &#124; OPT    &#124; OPTION
            &#124; PACKAGE &#124; RB    &#124; SECTREF &#124; STRONG    &#124; SYSCMD &#124; TERM
            &#124; TYPE    &#124; URI   &#124; USAGE   &#124; VAR       &#124; WIDGET

    example = EXAMPLE
            &#124; EXAMPLE\_BEGIN extext EXAMPLE\_END

    extext  = \{ <TEXT> &#124; defs &#124; markup \}

    a\_list  = LIST\_BEGIN<arguments>   argd\_list   LIST\_END
            &#124; LIST\_BEGIN<commands>    cmdd\_list   LIST\_END
            &#124; LIST\_BEGIN<definitions> def\_list    LIST\_END
            &#124; LIST\_BEGIN<enumerated>  enum\_list   LIST\_END
            &#124; LIST\_BEGIN<itemized>    item\_list   LIST\_END
            &#124; LIST\_BEGIN<options>     optd\_list   LIST\_END
            &#124; LIST\_BEGIN<tkoptions>   tkoptd\_list LIST\_END

    argd\_list   = \[ <WHITE> \] \{ ARG\_DEF      paras \}
    cmdd\_list   = \[ <WHITE> \] \{ CMD\_DEF      paras \}
    def\_list    = \[ <WHITE> \] \{ \(DEF&#124;CALL\)   paras \}
    enum\_list   = \[ <WHITE> \] \{ ENUM         paras \}
    item\_list   = \[ <WHITE> \] \{ ITEM         paras \}
    optd\_list   = \[ <WHITE> \] \{ OPT\_DEF      paras \}
    tkoptd\_list = \[ <WHITE> \] \{ TKOPTION\_DEF paras \}

# <a name='section5'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *doctools* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

  1. initialize and shutdown each pass

  1. query and initialize engine parameters

After the plugin has been loaded and the frontend commands are established the
commands will be called in the following sequence:

    fmt\_numpasses \-> n
    fmt\_listvariables \-> vars

    fmt\_varset var1 value1
    fmt\_varset var2 value2
    \.\.\.
    fmt\_varset varK valueK
    fmt\_initialize
    fmt\_setup 1
    \.\.\.
    fmt\_setup 2
    \.\.\.
    \.\.\.
    fmt\_setup n
    \.\.\.
    fmt\_postprocess
    fmt\_shutdown
    \.\.\.

I\.e\. first the number of passes and the set of available engine parameters is
established, followed by calls setting the parameters\. That second part is
optional\.

After that the plugin is initialized, the specified number of passes executed,
the final result run through a global post processing step and at last the

# <a name='section3'></a>JSON notation of keyword indices

The JSON format used for keyword indices is a direct translation of the
[Keyword index serialization format](#section5), mapping Tcl dictionaries
as JSON objects and Tcl lists as JSON arrays\. For example, the Tcl serialization

    doctools::idx \{
    	label \{Keyword Index\}
    	keywords \{
    		changelog  \{changelog\.man cvs\.man\}
    		conversion \{doctools\.man docidx\.man doctoc\.man apps/dtplite\.man mpexpand\.man\}
    		cvs        cvs\.man
    	\}

    	references \{
    		apps/dtplite\.man \{manpage dtplite\}
    		changelog\.man    \{manpage doctools::changelog\}
    		cvs\.man          \{manpage doctools::cvs\}
    		docidx\.man       \{manpage doctools::idx\}
    		doctoc\.man       \{manpage doctools::toc\}
    		doctools\.man     \{manpage doctools\}
    		mpexpand\.man     \{manpage mpexpand\}
    	\}

    	title \{\}
    \}


is equivalent to the JSON string

    \{

        "doctools::idx" : \{
            "label"      : "Keyword Index",
            "keywords"   : \{
                "changelog"  : \["changelog\.man","cvs\.man"\],
                "conversion" : \["doctools\.man","docidx\.man","doctoc\.man","apps\\/dtplite\.man","mpexpand\.man"\],
                "cvs"        : \["cvs\.man"\],
            \},
            "references" : \{
                "apps\\/dtplite\.man" : \["manpage","dtplite"\],
                "changelog\.man"     : \["manpage","doctools::changelog"\],
                "cvs\.man"           : \["manpage","doctools::cvs"\],
                "docidx\.man"        : \["manpage","doctools::idx"\],
                "doctoc\.man"        : \["manpage","doctools::toc"\],
                "doctools\.man"      : \["manpage","doctools"\],
                "mpexpand\.man"      : \["manpage","mpexpand"\]
            \},
            "title"      : ""
        \}
    \}



# <a name='section4'></a>Configuration

The JSON export plugin recognizes the following configuration variables and
changes its behaviour as they specify\.

  - boolean *indented*

# <a name='section3'></a>JSON notation of keyword indices

The JSON format used for keyword indices is a direct translation of the
[Keyword index serialization format](#section4), mapping Tcl dictionaries
as JSON objects and Tcl lists as JSON arrays\. For example, the Tcl serialization

    doctools::idx \{
    	label \{Keyword Index\}
    	keywords \{
    		changelog  \{changelog\.man cvs\.man\}
    		conversion \{doctools\.man docidx\.man doctoc\.man apps/dtplite\.man mpexpand\.man\}
    		cvs        cvs\.man
    	\}

    	references \{
    		apps/dtplite\.man \{manpage dtplite\}
    		changelog\.man    \{manpage doctools::changelog\}
    		cvs\.man          \{manpage doctools::cvs\}
    		docidx\.man       \{manpage doctools::idx\}
    		doctoc\.man       \{manpage doctools::toc\}
    		doctools\.man     \{manpage doctools\}
    		mpexpand\.man     \{manpage mpexpand\}
    	\}

    	title \{\}
    \}


is equivalent to the JSON string

    \{

        "doctools::idx" : \{
            "label"      : "Keyword Index",
            "keywords"   : \{
                "changelog"  : \["changelog\.man","cvs\.man"\],
                "conversion" : \["doctools\.man","docidx\.man","doctoc\.man","apps\\/dtplite\.man","mpexpand\.man"\],
                "cvs"        : \["cvs\.man"\],
            \},
            "references" : \{
                "apps\\/dtplite\.man" : \["manpage","dtplite"\],
                "changelog\.man"     : \["manpage","doctools::changelog"\],
                "cvs\.man"           : \["manpage","doctools::cvs"\],
                "docidx\.man"        : \["manpage","doctools::idx"\],
                "doctoc\.man"        : \["manpage","doctools::toc"\],
                "doctools\.man"      : \["manpage","doctools"\],
                "mpexpand\.man"      : \["manpage","mpexpand"\]
            \},
            "title"      : ""
        \}
    \}



# <a name='section4'></a>Keyword index serialization format

Here we specify the format used by the doctools v2 packages to serialize keyword
indices as immutable values for transport, comparison, etc\.

We distinguish between *regular* and *canonical* serializations\. While a

markup of *tables of contents*, and of general documentation, respectively\.
They are described in their own sets of documents, starting at the *DocTools \-
Tables Of Contents* and the *DocTools \- General*, respectively\.

# <a name='section3'></a>Package Overview

                                        ~~~~~~~~~~~ doctools::idx ~~~~~~~~~~~
                                       ~~                   &#124;               ~~
                    doctools::idx::export ~~~~~~~~~~~~~~~~~ &#124; ~~~~~~~~~~~~~ doctools::idx::import
                            &#124;                               &#124;                       &#124;
            \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+     &#124;    \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+
            &#124;               &#124;                         &#124;     &#124;    &#124;                  &#124;               &#124;                       &#124;               &#124;
    doctools::config        =                         &#124;     &#124;    &#124;                  =       doctools::include       doctools::config doctools::paths
                            &#124;                         &#124;     &#124;    &#124;                  &#124;
                    doctools::idx::export::<\*>        &#124;     &#124;    &#124;          doctools::idx::import::<\*>
                            docidx                    &#124;     &#124;    &#124;                  docidx, json
                            json                      &#124;     &#124;    &#124;                  &#124;           \\\\
                            html                      &#124;     &#124;    &#124;          doctools::idx::parse \\\\
                            nroff                     &#124;     &#124;    &#124;                  &#124;             \\\\
                            wiki                      &#124;     &#124;    &#124;  \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+              json
                            text                      &#124;     &#124;    &#124;  &#124;               &#124;
                                                    doctools::idx::structure        &#124;
                                                                                    &#124;

                                                                            \+\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+
                                                                            &#124;                       &#124;
              doctools::html  doctools::html::cssdefaults           doctools::tcl::parse    doctools::msgcat
                    &#124;                                                                               &#124;
              doctools::text  doctools::nroff::man\_macros                                           =
                                                                                                    &#124;

                                                                                            doctools::msgcat::idx::<\*>
                                                                                                    c, en, de, fr
                                                                                                    \(fr == en for now\)
            ~~      Interoperable objects, without actual package dependencies
            \-\-      Package dependency, higher requires lower package
            =       Dynamic dependency through plugin system
            <\*>     Multiple packages following the given form of naming\.

# <a name='section4'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *doctools* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

# <a name='section3'></a>JSON notation of tables of contents

The JSON format used for tables of contents is a direct translation of the [ToC
serialization format](#section5), mapping Tcl dictionaries as JSON objects
and Tcl lists as JSON arrays\. For example, the Tcl serialization

    doctools::toc \{
        items \{
            \{reference \{
    	    desc \{DocTools \- Tables of Contents\}
    	     id introduction\.man
    	     label doctools::toc::introduction
    	\}\}
    	\{division \{
    	     id processing\.man
    	     items \{
    	         \{reference \{
    		     desc \{doctoc serialization utilities\}
    		     id structure\.man
    		     label doctools::toc::structure
    		 \}\}
    		 \{reference \{
    		     desc \{Parsing text in doctoc format\}
    		     id parse\.man
    		     label doctools::toc::parse
    		 \}\}
    	     \}

                 label Processing
            \}\}
        \}

        label \{Table of Contents\}
        title TOC
    \}


is equivalent to the JSON string

    \{

        "doctools::toc" : \{
            "items" : \[\{
                "reference" : \{
                    "desc"  : "DocTools \- Tables of Contents",
                    "id"    : "introduction\.man",
                    "label" : "doctools::toc::introduction"

                \}
            \},\{
                "division" : \{
                    "id"    : "processing\.man",
                    "items" : \[\{
                        "reference" : \{
                            "desc"  : "doctoc serialization utilities",
                            "id"    : "structure\.man",
                            "label" : "doctools::toc::structure"

                        \}
                    \},\{
                        "reference" : \{
                            "desc"  : "Parsing text in doctoc format",
                            "id"    : "parse\.man",
                            "label" : "doctools::toc::parse"

                        \}
                    \}\],
                    "label" : "Processing"

                \}
            \}\],
            "label" : "Table of Contents",
            "title" : "TOC"
        \}
    \}



# <a name='section4'></a>Configuration

The JSON export plugin recognizes the following configuration variables and
changes its behaviour as they specify\.

  - boolean *indented*

# <a name='section3'></a>JSON notation of tables of contents

The JSON format used for tables of contents is a direct translation of the [ToC
serialization format](#section4), mapping Tcl dictionaries as JSON objects
and Tcl lists as JSON arrays\. For example, the Tcl serialization

    doctools::toc \{
        items \{
            \{reference \{
    	    desc \{DocTools \- Tables of Contents\}
    	     id introduction\.man
    	     label doctools::toc::introduction
    	\}\}
    	\{division \{
    	     id processing\.man
    	     items \{
    	         \{reference \{
    		     desc \{doctoc serialization utilities\}
    		     id structure\.man
    		     label doctools::toc::structure
    		 \}\}
    		 \{reference \{
    		     desc \{Parsing text in doctoc format\}
    		     id parse\.man
    		     label doctools::toc::parse
    		 \}\}
    	     \}

                 label Processing
            \}\}
        \}

        label \{Table of Contents\}
        title TOC
    \}


is equivalent to the JSON string

    \{

        "doctools::toc" : \{
            "items" : \[\{
                "reference" : \{
                    "desc"  : "DocTools \- Tables of Contents",
                    "id"    : "introduction\.man",
                    "label" : "doctools::toc::introduction"

                \}
            \},\{
                "division" : \{
                    "id"    : "processing\.man",
                    "items" : \[\{
                        "reference" : \{
                            "desc"  : "doctoc serialization utilities",
                            "id"    : "structure\.man",
                            "label" : "doctools::toc::structure"

                        \}
                    \},\{
                        "reference" : \{
                            "desc"  : "Parsing text in doctoc format",
                            "id"    : "parse\.man",
                            "label" : "doctools::toc::parse"

                        \}
                    \}\],
                    "label" : "Processing"

                \}
            \}\],
            "label" : "Table of Contents",
            "title" : "TOC"
        \}
    \}



# <a name='section4'></a>ToC serialization format

Here we specify the format used by the doctools v2 packages to serialize tables
of contents as immutable values for transport, comparison, etc\.

We distinguish between *regular* and *canonical* serializations\. While a

markup of *keyword indices*, and of general documentation, respectively\. They
are described in their own sets of documents, starting at the *DocTools \-
Keyword Indices* and the *DocTools \- General*, respectively\.

# <a name='section3'></a>Package Overview

                                        ~~~~~~~~~~~ doctools::toc ~~~~~~~~~~~
                                       ~~                   &#124;               ~~
                    doctools::toc::export ~~~~~~~~~~~~~~~~~ &#124; ~~~~~~~~~~~~~ doctools::toc::import
                            &#124;                               &#124;                       &#124;
            \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+     &#124;    \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+
            &#124;               &#124;                         &#124;     &#124;    &#124;                  &#124;               &#124;                       &#124;               &#124;
    doctools::config        =                         &#124;     &#124;    &#124;                  =       doctools::include       doctools::config doctools::paths
                            &#124;                         &#124;     &#124;    &#124;                  &#124;
                    doctools::toc::export::<\*>        &#124;     &#124;    &#124;          doctools::toc::import::<\*>
                            doctoc                    &#124;     &#124;    &#124;                  doctoc, json
                            json                      &#124;     &#124;    &#124;                  &#124;           \\\\
                            html                      &#124;     &#124;    &#124;          doctools::toc::parse \\\\
                            nroff                     &#124;     &#124;    &#124;                  &#124;             \\\\
                            wiki                      &#124;     &#124;    &#124;  \+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+              json
                            text                      &#124;     &#124;    &#124;  &#124;               &#124;
                                                    doctools::toc::structure        &#124;
                                                                                    &#124;

                                                                            \+\-\-\-\-\-\-\-\+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\+
                                                                            &#124;                       &#124;
              doctools::html  doctools::html::cssdefaults           doctools::tcl::parse    doctools::msgcat
                    &#124;                                                                               &#124;
              doctools::text  doctools::nroff::man\_macros                                           =
                                                                                                    &#124;

                                                                                            doctools::msgcat::toc::<\*>
                                                                                                    c, en, de, fr
                                                                                                    \(fr == en for now\)
            ~~      Interoperable objects, without actual package dependencies
            \-\-      Package dependency, higher requires lower package
            =       Dynamic dependency through plugin system
            <\*>     Multiple packages following the given form of naming\.

# <a name='section4'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *doctools* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

  - \[2\]

    The following directory structure is created when processing a single set of
    input documents\. The file extension used is for output in HTML, but that is
    not relevant to the structure and was just used to have proper file names\.

        output/
            toc\.html
            index\.html
            files/
                path/to/FOO\.html

    The last line in the example shows the document generated for a file FOO
    located at

        inputdirectory/path/to/FOO

  - \[3\]

    When merging many packages into a unified set of documents the generated
    directory structure is a bit deeper:

        output
            \.toc
            \.idx
            \.tocdoc
            \.idxdoc
            \.xrf
            toc\.html
            index\.html
            FOO1/
                \.\.\.
            FOO2/
                toc\.html
                files/
                    path/to/BAR\.html

    Each of the directories FOO1, \.\.\. contains the documents generated for the
    package FOO1, \.\.\. and follows the structure shown for use case \[2\]\. The only
    exception is that there is no per\-package index\.

    The files "\.toc", "\.idx", and "\.xrf" contain the internal status of the
    whole output and will be read and updated by the next invokation\. Their

    This command performs purely lexical normalization on the *path* and
    returns the changed path as its result\. Symbolic links in the path are
    *not* resolved\.

    Examples:

        fileutil::lexnormalize /foo/\./bar
        => /foo/bar

        fileutil::lexnormalize /foo/\.\./bar
        => /bar

  - <a name='2'></a>__::fileutil::fullnormalize__ *path*

    This command resolves all symbolic links in the *path* and returns the
    changed path as its result\. In contrast to the builtin __file
    normalize__ this command resolves a symbolic link in the last element of

    joined it with the result of __pwd__ to get an absolute filename\.

    The result of *filtercmd* is a boolean value that indicates if the current
    file should be included in the list of interesting files\.

    Example:

        \# find \.tcl files
        package require fileutil
        proc is\_tcl \{name\} \{return \[string match \*\.tcl $name\]\}
        set tcl\_files \[fileutil::find \. is\_tcl\]

  - <a name='13'></a>__::fileutil::findByPattern__ *basedir* ?__\-regexp__&#124;__\-glob__? ?__\-\-__? *patterns*

    This command is based upon the __TclX__ command __recursive\_glob__,
    except that it doesn't allow recursion over more than one directory at a
    time\. It uses __::fileutil::find__ internally and is thus able to and
    does follow symbolic links, something the __TclX__ command does not do\.

    A concrete example and extreme case is the "/sys" hierarchy under Linux
    where some hundred devices exist under both "/sys/devices" and "/sys/class"
    with the two sub\-hierarchies linking to the other, generating millions of
    legal paths to enumerate\. The structure, reduced to three devices, roughly
    looks like

        /sys/class/tty/tty0 \-\-> \.\./\.\./dev/tty0
        /sys/class/tty/tty1 \-\-> \.\./\.\./dev/tty1
        /sys/class/tty/tty2 \-\-> \.\./\.\./dev/tty1

        /sys/dev/tty0/bus
        /sys/dev/tty0/subsystem \-\-> \.\./\.\./class/tty
        /sys/dev/tty1/bus
        /sys/dev/tty1/subsystem \-\-> \.\./\.\./class/tty
        /sys/dev/tty2/bus
        /sys/dev/tty2/subsystem \-\-> \.\./\.\./class/tty

    The command __fileutil::find__ currently has no way to escape this\. When
    having to handle such a pathological hierarchy It is recommended to switch
    to package __fileutil::traverse__ and the same\-named command it
    provides, and then use the __\-prefilter__ option to prevent the
    traverser from following symbolic links, like so:

        package require fileutil::traverse

        proc NoLinks \{fileName\} \{
            if \{\[string equal \[file type $fileName\] link\]\} \{
                return 0
            \}

            return 1
        \}


        fileutil::traverse T /sys/devices \-prefilter NoLinks
        T foreach p \{
            puts $p
        \}

        T destroy

# <a name='section3'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *fileutil* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas

    Returns the current path type limiter\.

# <a name='section5'></a>EXAMPLES

The following examples assume that the variable __F__ contains a reference
to a multi\-file operation object\.

        $F do copy                       \\\\
    	the  \*\.dll                    \\\\
    	from c:/TDK/PrivateOpenSSL/bin \\\\
    	to   \[installdir\_of tls\]

        $F do move      \\\\
    	the  \*       \\\\
    	from /sources \\\\
    	into /scratch  \\\\
    	but not \*\.html

        \# Alternatively use 'except for \*\.html'\.

        $F do           \\\\
    	move         \\\\
    	the  index    \\\\
    	from /sources  \\\\
    	into /scratch   \\\\
    	as   pkgIndex\.tcl

        $F do         \\\\
    	remove     \\\\
    	the \*\.txt  \\\\
    	in /scratch

Note that the fact that most commands just modify the object state allows us to
use more off forms as specifications instead of just nearly\-natural language
sentences\. For example the second example in this section can re\-arranged into:

        $F do            \\\\
    	from /sources \\\\
    	into /scratch  \\\\
    	but not \*\.html \\\\
    	move           \\\\
    	the  \*

and the result is not only still a valid specification, but even stays
relatively readable\.

Further note that the information collected by the commands __but__,
__except__, and __as__ is automatically reset after the associated
__the__ was executed\. However no other state is reset in that manner,
allowing the user to avoid repetitions of unchanging information\. For example
the second and third examples of this section can be merged and rewritten into
the equivalent:

    $F do                   \\\\
        move                 \\\\
        the  \*                \\\\
        from /sources          \\\\
        into /scratch           \\\\
        but not \*\.html not index \\\\
        the  index               \\\\
        as   pkgIndex\.tcl

# <a name='section6'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *fileutil* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas
for enhancements you may have for either package and/or documentation\.

    A concrete example and extreme case is the "/sys" hierarchy under Linux
    where some hundred devices exist under both "/sys/devices" and "/sys/class"
    with the two sub\-hierarchies linking to the other, generating millions of
    legal paths to enumerate\. The structure, reduced to three devices, roughly
    looks like

        /sys/class/tty/tty0 \-\-> \.\./\.\./dev/tty0
        /sys/class/tty/tty1 \-\-> \.\./\.\./dev/tty1
        /sys/class/tty/tty2 \-\-> \.\./\.\./dev/tty1

        /sys/dev/tty0/bus
        /sys/dev/tty0/subsystem \-\-> \.\./\.\./class/tty
        /sys/dev/tty1/bus
        /sys/dev/tty1/subsystem \-\-> \.\./\.\./class/tty
        /sys/dev/tty2/bus
        /sys/dev/tty2/subsystem \-\-> \.\./\.\./class/tty

    When having to handle such a pathological hierarchy it is recommended to use
    the __\-prefilter__ option to prevent the traverser from following
    symbolic links, like so:

        package require fileutil::traverse

        proc NoLinks \{fileName\} \{
            if \{\[string equal \[file type $fileName\] link\]\} \{
                return 0
            \}

            return 1
        \}


        fileutil::traverse T /sys/devices \-prefilter NoLinks
        T foreach p \{
            puts $p
        \}

        T destroy

# <a name='section4'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems\. Please report such in the category *fileutil* of the [Tcllib
Trackers](http://core\.tcl\.tk/tcllib/reportlist)\. Please also report any ideas

    This command behaves mostly like __::fileutil::magic::rt::Nv__, except
    that it compares the fetched and masked value against *val* as specified
    with *comp* and returns the result of that comparison\.

    The argument *comp* has to contain one of Tcl's comparison operators, and
    the comparison made will be

        <val> <comp> <fetched\-and\-masked\-value>

    The special comparison operator __x__ signals that no comparison should
    be done, or, in other words, that the fetched value will always match
    *val*\.

  - <a name='12'></a>__::fileutil::magic::rt::Nvx__ *type* *offset* ?*qual*?

multiple return values and looping over multiple generators at once\. Writing a
generator is also a simple task, much like writing a normal procedure: simply
use the __define__ command to define the generator, and then call
__yield__ instead of __[return](\.\./\.\./\.\./\.\./index\.md\#return)__\. For
example, we can define a generator for looping through the integers in a
particular range:

    generator define range \{n m\} \{
        for \{set i $n\} \{$i <= $m\} \{incr i\} \{ generator yield $i \}
    \}

    generator foreach x \[range 1 10\] \{
        puts "x = $x"
    \}


The above example will print the numbers from 1 to 10 in sequence, as you would
expect\. The difference from a normal loop over a list is that the numbers are
only generated as they are needed\. If we insert a break into the loop then any
remaining numbers in the sequence would never be generated\. To illustrate, we
can define a generator that produces the sequence of natural numbers: an
infinite series\. A normal procedure would never return trying to produce this
series as a list\. By using a generator we only have to generate those values
which are actually used:

    generator define nats \{\} \{
        while 1 \{ generator yield \[incr nat\] \}
    \}

    generator foreach n \[nats\] \{
        if \{$n > 100\} \{ break \}
    \}


# <a name='section2'></a>COMMANDS

  - <a name='1'></a>__generator__ __define__ *name* *params* *body*

    Creates a new generator procedure\. The arguments to the command are
    identical to those for __[proc](\.\./\.\./\.\./\.\./index\.md\#proc)__: a

    be used like a __finally__ block in the
    __[try](\.\./try/tcllib\_try\.md)__ command, except that it is tied to
    the life\-cycle of the generator rather than to a particular scope\. For
    example, if we create a generator to iterate over the lines in a text file,
    we can use __finally__ to ensure that the file is closed whenever the
    generator is destroyed:

    generator define lines file \{
        set in \[open $file\]
        \# Ensure file is always closed
        generator finally close $in
        while \{\[gets $in line\] >= 0\} \{
            generator yield $line
        \}
    \}


    generator foreach line \[lines /etc/passwd\] \{
        puts "\[incr count\]: $line"
        if \{$count > 10\} \{ break \}
    \}

    \# File will be closed even on early exit

    If you create a generator that consumes another generator \(such as the
    standard __map__ and __filter__ generators defined later\), then you
    should use a __finally__ command to ensure that this generator is
    destroyed when its parent is\. For example, the __map__ generator is
    defined as follows:

    generator define map \{f xs\} \{
        generator finally generator destroy $xs
        generator foreach x $xs \{ generator yield \[\{\*\}$f $x\] \}
    \}


  - <a name='9'></a>__generator__ __from__ *format* *value*

    Creates a generator from a data structure\. Currently, supported formats are
    __list__, __dict__, or __string__\. The list format yields each
    element in turn\. For dictionaries, each key and value are yielded
    separately\. Finally, strings are yielded a character at a time\.

  - <a name='10'></a>__generator__ __to__ *format* *generator*

    Converts a generator into a data structure\. This is the reverse operation of
    the __from__ command, and supports the same data structures\. The two
    operations obey the following identity laws \(where __=__ is interpreted
    appropriately\):

    \[generator to $fmt \[generator from $fmt $value\]\] = $value
    \[generator from $fmt \[generator to $fmt $gen\]\]   = $gen

# <a name='section3'></a>PRELUDE

The following commands are provided as a standard library of generator
combinators and functions that perform convenience operations on generators\. The
functions in this section are loosely modelled on the equivalent functions from
the Haskell Prelude\. *Warning:* most of the functions in this prelude destroy
any generator arguments they are passed as a side\-effect\. If you want to have
persistent generators, see the streams library\.

  - <a name='11'></a>__generator__ __map__ *function* *generator*

    Apply a function to every element of a generator, returning a new generator
    of the results\. This is the classic map function from functional
    programming, applied to generators\. For example, we can generate all the
    square numbers using the following code \(where __nats__ is defined as
    earlier\):

    proc square x \{ expr \{$x \* $x\} \}
    generator foreach n \[generator map square \[nats\]\] \{
        puts "n = $n"
        if \{$n > 1000\} \{ break \}
    \}


  - <a name='12'></a>__generator__ __filter__ *predicate* *generator*

    Another classic functional programming gem\. This command returns a generator
    that yields only those items from the argument generator that satisfy the
    predicate \(boolean function\)\. For example, if we had a generator
    __employees__ that returned a stream of dictionaries representing
    people, we could filter all those whose salaries are above 100,000 dollars
    \(or whichever currency you prefer\) using a simple filter:

    proc salary> \{amount person\} \{ expr \{\[dict get $person salary\] > $amount\} \}
    set fat\-cats \[generator filter \{salary> 100000\} $employees\]

  - <a name='13'></a>__generator__ __reduce__ *function* *zero* *generator*

    This is the classic left\-fold operation\. This command takes a function, an
    initial value, and a generator of values\. For each element in the generator
    it applies the function to the current accumulator value \(the *zero*
    argument initially\) and that element, and then uses the result as the new
    accumulator value\. This process is repeated through the entire generator
    \(eagerly\) and the final accumulator value is then returned\. If we consider
    the function to be a binary operator, and the zero argument to be the left
    identity element of that operation, then we can consider the __reduce__
    command as *folding* the operator between each successive pair of values
    in the generator in a left\-associative fashion\. For example, the sum of a
    sequence of numbers can be calculated by folding a __\+__ operator
    between them, with 0 as the identity:

    \# sum xs          = reduce \+ 0 xs
    \# sum \[range 1 5\] = reduce \+ 0 \[range 1 5\]
    \#                 = reduce \+ \[\+ 0 1\] \[range 2 5\]
    \#                 = reduce \+ \[\+ 1 2\] \[range 3 5\]
    \#                 = \.\.\.
    \#                 = reduce \+ \[\+ 10 5\] <empty>
    \#                 = \(\(\(\(0\+1\)\+2\)\+3\)\+4\)\+5
    \#                 = 15
    proc \+ \{a b\} \{ expr \{$a \+ $b\} \}
    proc sum gen \{ generator reduce \+ 0 $gen \}
    puts \[sum \[range 1 10\]\]

    The __reduce__ operation is an extremely useful one, and a great variety
    of different operations can be defined using it\. For example, we can define
    a factorial function as the product of a range using generators\. This
    definition is both very clear and also quite efficient \(in both memory and
    running time\):

    proc \* \{x y\} \{ expr \{$x \* $y\} \}
    proc prod gen \{ generator reduce \* 0 $gen \}
    proc fac n \{ prod \[range 1 $n\] \}

    However, while the __reduce__ operation is efficient for finite
    generators, care should be taken not to apply it to an infinite generator,
    as this will result in an infinite loop:

    sum \[nats\]; \# Never returns

  - <a name='14'></a>__generator__ __foldl__ *function* *zero* *generator*

    This is an alias for the __reduce__ command\.

  - <a name='15'></a>__generator__ __foldr__ *function* *zero* *generator*

  - <a name='32'></a>__generator__ __iterate__ *function* *init*

    Returns an infinite generator formed by repeatedly applying the function to
    the initial argument\. For example, the Fibonacci numbers can be defined as
    follows:

    proc fst pair \{ lindex $pair 0 \}
    proc snd pair \{ lindex $pair 1 \}
    proc nextFib ab \{ list \[snd $ab\] \[expr \{\[fst $ab\] \+ \[snd $ab\]\}\] \}
    proc fibs \{\} \{ generator map fst \[generator iterate nextFib \{0 1\}\] \}

  - <a name='33'></a>__generator__ __last__ *generator*

    Returns the last element of the generator \(if it exists\)\.

  - <a name='34'></a>__generator__ __length__ *generator*

  - <a name='40'></a>__generator__ __splitWhen__ *predicate* *generator*

    Splits the generator into lists of elements using the predicate to identify
    delimiters\. The resulting lists are returned as a generator\. Elements
    matching the delimiter predicate are discarded\. For example, to split up a
    generator using the string "&#124;" as a delimiter:

    set xs \[generator from list \{a &#124; b &#124; c\}\]
    generator split \{string equal "&#124;"\} $xs ;\# returns a then b then c

  - <a name='41'></a>__generator__ __scanl__ *function* *zero* *generator*

    Similar to __foldl__, but returns a generator of all of the intermediate
    values for the accumulator argument\. The final element of this generator is
    equivalent to __foldl__ called on the same arguments\.

    elements: *latitude*, *longitude* and *metadata dictionary*\.
    *Latitude* and *longitude* are decimal numbers\. The *metadata
    dictionary* format is described above\. For points in a track, typically
    there will always be ele \(elevation\) and time metadata keys\.

# <a name='section4'></a>EXAMPLE

    % set token \[::gpx::Create myGpxFile\.gpx\]
    % set version \[dict get \[::gpx::GetGPXMetadata $token\] version\]
    % set trackCnt \[::gpx::GetTrackCount $token\]
    % set firstPoint \[lindex \[::gpx::GetTrackPoints $token 1\] 0\]
    % lassign $firstPoint lat lon ptMetadata
    % puts "first point in the first track is at $lat, $lon"
    % if \{\[dict exists $ptMetadata ele\]\} \{
         puts "at elevation \[dict get $ptMetadata ele\] meters"
      \}

    % ::gpx::Cleanup $token

# <a name='section5'></a>REFERENCES

  1. GPX: the GPS Exchange Format
     \([http://www\.topografix\.com/gpx\.asp](http://www\.topografix\.com/gpx\.asp)\)