| Description: |
I have noticed some gaps, which ought to be filled when convenient,
in the documentation of the **pt** serialisation formats.
## My context (can be skipped)
I am exploring the idea of using dynamically constructed parsing
expression grammars, for parsing strings denoting (rather complicated)
nested data, where the exact data type is created at runtime by composing
constructions of independent origin. The challenge there is: can we make
this work, when each construction only understands its own little part of
the whole? Parsing expressions just might be up to the task.
A further catch is that (because of Reasons) my whole grammar has to be a
value, so using the Parser Tools as-is — notably storing grammars in
(snit or TclOO) container objects — is a non-starter. Luckily:
* I don't have huge amounts of data to parse (so I don't need C speed),
* writing a packrat parser is fairly simple, and
* the **pt** serialisation formats are clear enough to use directly.
In particular, with a little ensemble magic one can pretty much run the
parser by executing the parsing expression serialisation format directly.
But writing my own implementation (in part to strip out the reflected
channel detour, but mostly because I needed to make some extensions)
requires me to look at all the edge cases, and I notice some of them
aren't convered in the documentation.
(I also happened to notice some enhancements that could be done to the
**pt** packages, even if those would be orthogonal to my actual goals.
I'll return to that in separate tickets.)
## The gaps in documentation
### The range type ..
Already from the PEG grammar, it is obvious that some mechanism for
representing ranges of characters is needed, but none of the forms listed
in the "PE serialization format" sections will do the trick. In the
[pt::pe manpage](https://core.tcl-lang.org/tcllib/doc/tcllib-1-21/embedded/md/tcllib/files/modules/pt/pt_pexpression.md#26)
the `::pt::pe range` subcommand is documented, but not the `..` form of
parsing expression that it returns!
### The class type cl
The `pt::pe class` command returns `cl` expressions, which are not covered
in the documentation, but which appear to be supported by both PARAM
implementations. However, it this not recognised by `pt::pe verify`
```
% pt::pe verify [pt::pe class abc]
Error: error in serialization: invalid operator "cl"
```
Hence `pt::peg::container` objects refuse to `deserialize` grammars making
use of that, and indeed `pt::peg::from::peg` rather produces multiway
choices.
### The string type str
Similarly, the `pt::pe str` command returns `str` expressions, which are
not covered in the documentation, but which appear to be supported by both
PARAM implementations. Again this is not recognised by `pt::pe verify`
```
% pt::pe verify [pt::pe str abc]
Error: error in serialization: invalid operator "str"
```
Hence `pt::peg::container` objects refuse to `deserialize` grammars making
use of that, and indeed `pt::peg::from::peg` rather produces sequences of
individual terminals.
### Length of a terminal
What is said about the atomic parsing expression for terminals `t` is
* The expression `[list t` *x*`]` is an atomic parsing expression.
It matches the terminal string *x*.
What it notably does not claim is that *x* has to be a single character,
nor does `pt::pe verify` complain about using a multicharacter string as
*x*, but it appears the resulting parsing expression will not match
anything — at least not the string supplied as *x*.
### Empty ranges in AST
The [pt::ast manpage](https://core.tcl-lang.org/tcllib/doc/tcllib-1-21/embedded/md/tcllib/files/modules/pt/pt_astree.md)
claims the end location has to be greater than or equal to the start
position, but since the range includes these locations, that won't allow
representing a match of length zero. Consider matching the grammar
```
PEG demo ('a' Capture 'c')
Capture <- 'b'* ;
END;
```
against the string `ac`: the resulting abstract syntax tree should
probably be "`Capture 1 0`".
## Explanations considered useful
These are not gaps as such, but things I believe could do with being
explained.
### Singleton dictionary idiom
That the single first level key in the PEG serialisation format must be
`pt::grammar::peg` is clearly stated, but why things are set up that way
is never explained. I suspect this is an idiom: if the grammar appears in
a context of type-tagged values, then this `pt::grammar::peg` may be
interpreted as its type. Type-tagging is usually done with a list, but
since the value underneath is primarily a dict, it becomes much easier to
access if the first level is also a dict; the string representation is the
same either way.
### What is the point of ddigit?
The `<ddigit>` character class appears to be equivalent to the shorter
`[0-9]`, so what was ever the point of it (besides, by now, backwards
compatibility)??
Also the claim in the manpage that it is implemented using `regexp` seems
to be wrong, since it in fact is implemented using `string match`.
|