Tcl Library Source Code

generator](../../../index.md#parser_generator)*. Its intended audience are
people who wish to create a parser for some language of theirs. Should you wish
to modify the application instead, please see the section about the
application's [Internals](#section11) for the basic references.

It resides in the User Application Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_user_app.png)

# <a name='section2'></a>Command Line

  - <a name='1'></a>__pt__ __generate__ *resultformat* ?*options...*? *resultfile* *inputformat* *inputfile*

    This sub-command of the application reads the parsing expression grammar
    stored in the *inputfile* in the format *inputformat*, converts it to the

__oo__, and __snit__), one (__container__) provides code which can be used in
conjunction with a generic parser (also known as a grammar interpreter), and the
last two (__json__ and __peg__) are doing double-duty as input formats, allowing
the transformation of grammars for exchange, reformatting, and the like.

The created parsers fall into three categories:

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/gen_options.png)

  - __Specialized parsers implemented in C__

    The fastest parsers are created when using the result formats __c__ and
    __critcl__. The first returns the raw C code for the parser, while the
    latter wraps it into a Tcl package using *CriTcl*.

# <a name='section10'></a>Example

In this section we are working a complete example, starting with a PEG grammar
and ending with running the parser generated from it over some input, following
the outline shown in the figure below:

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/flow.png) Our grammar,
assumed to the stored in the file "calculator.peg" is

    PEG calculator (Expression)
        Digit      <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
        Sign       <- '-' / '+'                                     ;
        Number     <- Sign? Digit+                                  ;
        Expression <- Term (AddOp Term)*                            ;
        MulOp      <- '*' / '/'                                     ;

assuming that the input file and channel contained the text

    120+5

A more graphical representation of the tree would be

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/expr_ast.png) Regardless,
at this point it is the user's responsibility to work with the tree to reach
whatever goal she desires. I.e. analyze it, transform it, etc. The package
__[pt::ast](../modules/pt/pt_astree.md)__ should be of help here, providing
commands to walk such ASTs structures in various ways.

One important thing to note is that the parsers used here return a data
structure representing the structure of the input per the grammar underlying the
parser. There are *no* callbacks during the parsing process, i.e. no *parsing
actions*, as most other parsers will have.

This package provides commands to work with the serializations of abstract
syntax trees as managed by the Parser Tools, and specified in section [AST
serialization format](#section3).

This is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::ast__ __verify__ *serial* ?*canonvar*?

    This command verifies that the content of *serial* is a valid serialization
    of an abstract syntax tree and will throw an error if that is not the case.

                }
            }
        }
    }

Or, more graphical

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/expr_ast.png)

# <a name='section4'></a>Bugs, Ideas, Feedback

This document, and the package it describes, will undoubtedly contain bugs and
other problems. Please report such in the category *pt* of the [Tcllib
Trackers](http://core.tcl.tk/tcllib/reportlist). Please also report any ideas
for enhancements you may have for either package and/or documentation.

make the use of this highly configurable package easier by providing a canned
configuration. When applied this configuration causes the package
__[pt::peg::to::cparam](pt_peg_to_cparam.md)__ to generate __critcl__-based
parser packages.

It is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::cparam::configuration::critcl__ __def__ *name* *pkg* *version* *cmdprefix*

    The command applies the configuration provided by this package to the
    *cmdprefix*, causing the creation of __critcl__-based parsers whose class is

make the use of this highly configurable package easier by providing a canned
configuration. When applied this configuration causes the package
__[pt::peg::to::cparam](pt_peg_to_cparam.md)__ to generate plain parser code
ready for inclusion into a *TEA*-based C extension.

It is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::cparam::configuration::tea__ __def__ *name* *pkg* *version* *cmdprefix*

    The command applies the configuration provided by this package to the
    *cmdprefix*, causing the creation of __tea__-based parsers whose class is

the conversion packages.

Its intended audience are people who wish to create their own converter for some
type of input, and/or an import plugin for their or some other converter.

It resides in the Import section of the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_import.png)

# <a name='section2'></a>Converter API

Any (grammar) import converter has to follow the rules set out below:

  1. A converter is a package. Its name is arbitrary, however it is recommended
     to put it under the __::pt::peg::from__ namespace.

     See the *[Introduction to Parsing Expression
     Grammars](pt_peg_introduction.md)*.

# <a name='section2'></a>Parser Tools Architecture

The system can be split into roughly three layers, as seen in the figure below

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/architecture.png) These
layers are, from high to low:

  1. At the top we have the application and the packages using the packages of
     the layer below to implement common usecases. One example is the
     aforementioned __[pt::pgen](pt_pgen.md)__ which provides a parser
     generator.

     The list of packages belonging to this layer can be found in section [User

__snit__, and __oo__ regarding access to the actual parsing functionality.

Its intended audience are people who wish to create a parser for some language
of theirs and then use that parser within a Tcl-based package or application.

It resides in the User Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_user_pkg.png)

# <a name='section2'></a>Class API

  - <a name='1'></a>__className__ ?*objectName*?

    The class command constructs parser instances, i.e. objects. The result of
    the command is the fully-qualified name of the instance command.

                }
            }
        }
    }

Or, more graphical

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/expr_ast.png)

# <a name='section6'></a>PE serialization format

Here we specify the format used by the Parser Tools to serialize Parsing
Expressions as immutable values for transport, comparison, etc.

We distinguish between *regular* and *canonical* serializations. While a parsing

This package provides a container class for parsing expression grammars, with
each instance storing a single grammar and allowing the user to manipulate and
query its definition.

It resides in the Storage section of the Core Layer of Parser Tools, and is one
of the three pillars the management of parsing expression grammars resides on.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_container.png)
The other two pillars are, as shown above

  1. *[PEG Import](pt_peg_import.md)*, and

  1. *[PEG Export](pt_peg_export.md)*

Packages related to this are:

instance handling a set of plugins for the export of them to other formats, i.e.
their conversion to, for example *[nroff](../../../../index.md#nroff)*,
*[HTML](../../../../index.md#html)*, etc.

It resides in the Export section of the Core Layer of Parser Tools, and is one
of the three pillars the management of parsing expression grammars resides on.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_export.png) The
other two pillars are, as shown above

  1. *[PEG Import](pt_peg_import.md)*, and

  1. *[PEG Storage](pt_peg_container.md)*

For information about the data structure which is the major input to the manager
objects provided by this package see the section [PEG serialization

generation of CONTAINER markup.

It resides in the Export section of the Core Layer of Parser Tools and is
intended to be used by __[pt::peg::export](pt_peg_export.md)__, the export
manager, sitting between it and the corresponding core conversion functionality
provided by __[pt::peg::to::container](pt_peg_to_container.md)__.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

While the direct use of this package with a regular interpreter is possible,
this is strongly disrecommended and requires a number of contortions to provide
the expected environment. The proper way to use this functionality depends on
the situation:

  1. In an untrusted environment the proper access is through the package

generation of JSON markup.

It resides in the Export section of the Core Layer of Parser Tools and is
intended to be used by __[pt::peg::export](pt_peg_export.md)__, the export
manager, sitting between it and the corresponding core conversion functionality
provided by __[pt::peg::to::json](pt_peg_to_json.md)__.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

While the direct use of this package with a regular interpreter is possible,
this is strongly disrecommended and requires a number of contortions to provide
the expected environment. The proper way to use this functionality depends on
the situation:

  1. In an untrusted environment the proper access is through the package

generation of PEG markup.

It resides in the Export section of the Core Layer of Parser Tools and is
intended to be used by __[pt::peg::export](pt_peg_export.md)__, the export
manager, sitting between it and the corresponding core conversion functionality
provided by __[pt::peg::to::peg](pt_peg_to_peg.md)__.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

While the direct use of this package with a regular interpreter is possible,
this is strongly disrecommended and requires a number of contortions to provide
the expected environment. The proper way to use this functionality depends on
the situation:

  1. In an untrusted environment the proper access is through the package

It resides in the Import section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the import manager provided by __[pt::peg::import](pt_peg_import.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding import plugin __[pt::peg::import::json](pt_peg_import_json.md)__
sitting between converter and import manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_iplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Import API](pt_from_api.md)* specification.

  - <a name='1'></a>__pt::peg::from::json__ __convert__ *text*

It resides in the Import section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the import manager provided by __[pt::peg::import](pt_peg_import.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding import plugin __[pt::peg::import::peg](pt_peg_import_peg.md)__
sitting between converter and import manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_iplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Import API](pt_from_api.md)* specification.

  - <a name='1'></a>__pt::peg::from::peg__ __convert__ *text*

instance handling a set of plugins for the import of them from other formats,
i.e. their conversion from, for example *peg*, *container*,
*[json](../../../../index.md#json)*, etc.

It resides in the Import section of the Core Layer of Parser Tools, and is one
of the three pillars the management of parsing expression grammars resides on.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_import.png) The
other two pillars are, as shown above

  1. *[PEG Export](pt_peg_export.md)*, and

  1. *[PEG Storage](pt_peg_container.md)*

For information about the data structure which is the major output of the
manager objects provided by this package see the section [PEG serialization

JSON markup.

It resides in the Import section of the Core Layer of Parser Tools and is
intended to be used by __[pt::peg::import](pt_peg_import.md)__, the import
manager, sitting between it and the corresponding core conversion functionality
provided by __[pt::peg::from::json](pt_peg_from_json.md)__.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_iplugins.png)

While the direct use of this package with a regular interpreter is possible,
this is strongly disrecommended and requires a number of contortions to provide
the expected environment. The proper way to use this functionality depends on
the situation:

  1. In an untrusted environment the proper access is through the package

PEG markup.

It resides in the Import section of the Core Layer of Parser Tools and is
intended to be used by __[pt::peg::import](pt_peg_import.md)__, the import
manager, sitting between it and the corresponding core conversion functionality
provided by __[pt::peg::from::peg](pt_peg_from_peg.md)__.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_iplugins.png)

While the direct use of this package with a regular interpreter is possible,
this is strongly disrecommended and requires a number of contortions to provide
the expected environment. The proper way to use this functionality depends on
the situation:

  1. In an untrusted environment the proper access is through the package

with a parsing expression grammar. The grammar is executed directly, i.e.
interpreted, with the underlying runtime provided by the package
__[pt::rde](pt_rdengine.md)__, basing everything on the PARAM.

Like the supporting runtime this package resides in the Execution section of the
Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_transform.png)

The interpreted grammar is copied from an instance of
__[pt::peg::container](pt_peg_container.md)__, or anything providing the same
API, like the container classes created by
__[pt::peg::to::container](pt_peg_to_container.md)__ or the associated export
plugin __[pt::peg::export::container](pt_peg_export_container.md)__.

                }
            }
        }
    }

Or, more graphical

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/expr_ast.png)

# <a name='section3'></a>PE serialization format

Here we specify the format used by the Parser Tools to serialize Parsing
Expressions as immutable values for transport, comparison, etc.

We distinguish between *regular* and *canonical* serializations. While a parsing

used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin
__[pt::peg::export::container](pt_peg_export_container.md)__ sitting between
converter and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::container__ __reset__

It resides in the Export section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin __pt::peg::export::cparam__ sitting between
converter and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::cparam__ __reset__

It resides in the Export section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin __[pt::peg::export::json](pt_peg_export_json.md)__
sitting between converter and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::json__ __reset__

It resides in the Export section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin __pt::peg::export::param__ sitting between converter
and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::param__ __reset__

It resides in the Export section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin __[pt::peg::export::peg](pt_peg_export_peg.md)__
sitting between converter and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::peg__ __reset__

It resides in the Export section of the Core Layer of Parser Tools, and can be
used either directly with the other packages of this layer, or indirectly
through the export manager provided by __[pt::peg::export](pt_peg_export.md)__.
The latter is intented for use in untrusted environments and done through the
corresponding export plugin __pt::peg::export::tclparam__ sitting between
converter and export manager.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_eplugins.png)

# <a name='section2'></a>API

The API provided by this package satisfies the specification of the Converter
API found in the *[Parser Tools Export API](pt_to_api.md)* specification.

  - <a name='1'></a>__pt::peg::to::tclparam__ __reset__

This package provides commands to work with the serializations of parsing
expression grammars as managed by the Parser Tools, and specified in section
[PEG serialization format](#section3).

This is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::peg__ __verify__ *serial* ?*canonvar*?

    This command verifies that the content of *serial* is a valid serialization
    of a parsing expression and will throw an error if that is not the case. The

This package provides commands to work with the serializations of parsing
expressions as managed by the Parser Tools, and specified in section [PE
serialization format](#section3).

This is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::pe__ __verify__ *serial* ?*canonvar*?

    This command verifies that the content of *serial* is a valid serialization
    of a parsing expression and will throw an error if that is not the case. The

As such the intended audience of this document are people wishing to modify
and/or extend this part of __[pt](../../apps/pt.md)__'s functionality. Users of
__[pt](../../apps/pt.md)__ on the other hand are hereby refered to the
applications' manpage, i.e. *[Parser Tools Application](../../apps/pt.md)*.

It resides in the User Package Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_user_pkg.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::pgen__ *inputformat* *text* *resultformat* ?*options...*?

    This command takes the parsing expression grammar in *text* (in the format
    specified by *inputformat*), and returns the same grammar in the format

# <a name='section3'></a>Example

In this section we are working a complete example, starting with a PEG grammar
and ending with running the parser generated from it over some input, following
the outline shown in the figure below:

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/flow.png) Our grammar,
assumed to the stored in the file "calculator.peg" is

    PEG calculator (Expression)
        Digit      <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
        Sign       <- '-' / '+'                                     ;
        Number     <- Sign? Digit+                                  ;
        Expression <- Term (AddOp Term)*                            ;
        MulOp      <- '*' / '/'                                     ;

assuming that the input file and channel contained the text

    120+5

A more graphical representation of the tree would be

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/expr_ast.png) Regardless,
at this point it is the user's responsibility to work with the tree to reach
whatever goal she desires. I.e. analyze it, transform it, etc. The package
__[pt::ast](pt_astree.md)__ should be of help here, providing commands to walk
such ASTs structures in various ways.

One important thing to note is that the parsers used here return a data
structure representing the structure of the input per the grammar underlying the
parser. There are *no* callbacks during the parsing process, i.e. no *parsing
actions*, as most other parsers will have.

Going back to the last snippet of code, the execution of the parser for some

Specification](pt_param.md)*, as such that document is *required* reading to
understand both this manpage, and the package itself. The description below does
make numerous shorthand references to the PARAM's instructions and the various
parts of its architectural state.

The package resides in the Execution section of the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_transform.png)

Note: This package not only has the standard Tcl implementation, but also an
accelerator, i.e. a C implementation, based on Critcl.

## <a name='subsection1'></a>Class API

The package exports the API described here.

highly configurable package easier by providing a canned configuration. When
applied this configuration causes the package
__[pt::peg::to::tclparam](pt_peg_to_tclparam.md)__ to generate __NX__-based
parser packages.

It is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::tclparam::configuration::nx__ __def__ *name* *pkg* *version* *cmdprefix*

    The command applies the configuration provided by this package to the
    *cmdprefix*, causing the creation of __NX__-based parsers whose class is

highly configurable package easier by providing a canned configuration. When
applied this configuration causes the package
__[pt::peg::to::tclparam](pt_peg_to_tclparam.md)__ to generate
__[snit](../snit/snit.md)__-based parser packages.

It is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::tclparam::configuration::snit__ __def__ *name* *pkg* *version* *cmdprefix*

    The command applies the configuration provided by this package to the
    *cmdprefix*, causing the creation of __[snit](../snit/snit.md)__-based

highly configurable package easier by providing a canned configuration. When
applied this configuration causes the package
__[pt::peg::to::tclparam](pt_peg_to_tclparam.md)__ to generate __OO__-based
parser packages.

It is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::tclparam::configuration::tcloo__ __def__ *name* *pkg* *version* *cmdprefix*

    The command applies the configuration provided by this package to the
    *cmdprefix*, causing the creation of __OO__-based parsers whose class is

the conversion packages.

Its intended audience are people who wish to create their own converter for some
type of output, and/or an export plugin for their or some other converter.

It resides in the Export section of the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_export.png)

# <a name='section2'></a>Converter API

Any (grammar) export converter has to follow the rules set out below:

  1. A converter is a package. Its name is arbitrary, however it is recommended
     to put it under the __::pt::peg::to__ namespace.

Tools](pt_introduction.md)*. This document is the entrypoint to the whole system
the current package is a part of.

This package provides general utility commands.

This is a supporting package in the Core Layer of Parser Tools.

![](/home/aku/Play/Tcllib/w-scratch/embedded/md/image/arch_core_support.png)

# <a name='section2'></a>API

  - <a name='1'></a>__::pt::util__ __error2readable__ *error* *text*

    This command takes the structured form of a syntax *error* as thrown by
    parser runtimes and the input *text* to the parser which caused that error

proc fmt_image {text {label {}}} {
    # text = symbolic name of the image.

    # formatting based on the available data ...

    set img [dt_imgdst $text {png gif jpg}]
    if {$img != {}} {

	if {$label != {}} {
	    return "!\[\]($img \"$label\")"
	} else {
	    return "!\[\]($img)"
	}
    }