Tcl Library Source Code

View Ticket
Login
Ticket UUID: 0fc5d24109625d948233676786654ac9840beb51
Title: Gaps in format documentation
Type: Bug Version: 1.21
Submitter: lars_h_again Created on: 2024-01-21 18:43:53
Subsystem: pt (parsetools) Assigned To: nobody
Priority: 5 Medium Severity: Minor
Status: Open Last Modified: 2024-01-21 18:43:53
Resolution: None Closed By: nobody
    Closed on:
Description:
I have noticed some gaps, which ought to be filled when convenient, 
in the documentation of the **pt** serialisation formats.

## My context (can be skipped)

I am exploring the idea of using dynamically constructed parsing 
expression grammars, for parsing strings denoting (rather complicated) 
nested data, where the exact data type is created at runtime by composing 
constructions of independent origin. The challenge there is: can we make 
this work, when each construction only understands its own little part of 
the whole? Parsing expressions just might be up to the task.

A further catch is that (because of Reasons) my whole grammar has to be a 
value, so using the Parser Tools as-is — notably storing grammars in 
(snit or TclOO) container objects — is a non-starter. Luckily:

* I don't have huge amounts of data to parse (so I don't need C speed),
* writing a packrat parser is fairly simple, and
* the **pt** serialisation formats are clear enough to use directly.

In particular, with a little ensemble magic one can pretty much run the 
parser by executing the parsing expression serialisation format directly. 
But writing my own implementation (in part to strip out the reflected 
channel detour, but mostly because I needed to make some extensions) 
requires me to look at all the edge cases, and I notice some of them 
aren't convered in the documentation.

(I also happened to notice some enhancements that could be done to the 
**pt** packages, even if those would be orthogonal to my actual goals. 
I'll return to that in separate tickets.)

## The gaps in documentation

### The range type ..

Already from the PEG grammar, it is obvious that some mechanism for 
representing ranges of characters is needed, but none of the forms listed 
in the "PE serialization format" sections will do the trick. In the 
[pt::pe manpage](https://core.tcl-lang.org/tcllib/doc/tcllib-1-21/embedded/md/tcllib/files/modules/pt/pt_pexpression.md#26)
the `::pt::pe range` subcommand is documented, but not the `..` form of 
parsing expression that it returns!

### The class type cl

The `pt::pe class` command returns `cl` expressions, which are not covered 
in the documentation, but which appear to be supported by both PARAM 
implementations. However, it this not recognised by `pt::pe verify`

```
% pt::pe verify [pt::pe class abc]
Error: error in serialization: invalid operator "cl"
```

Hence `pt::peg::container` objects refuse to `deserialize` grammars making 
use of that, and indeed `pt::peg::from::peg` rather produces multiway 
choices.

### The string type str

Similarly, the `pt::pe str` command returns `str` expressions, which are 
not covered in the documentation, but which appear to be supported by both 
PARAM implementations. Again this is not recognised by `pt::pe verify`

```
% pt::pe verify [pt::pe str abc]
Error: error in serialization: invalid operator "str"
```

Hence `pt::peg::container` objects refuse to `deserialize` grammars making 
use of that, and indeed `pt::peg::from::peg` rather produces sequences of 
individual terminals.

### Length of a terminal

What is said about the atomic parsing expression for terminals `t` is

* The expression `[list t`  *x*`]` is an atomic parsing expression. 
  It matches the terminal string *x*.

What it notably does not claim is that *x* has to be a single character, 
nor does `pt::pe verify` complain about using a multicharacter string as 
*x*, but it appears the resulting parsing expression will not match 
anything — at least not the string supplied as *x*.

### Empty ranges in AST

The [pt::ast manpage](https://core.tcl-lang.org/tcllib/doc/tcllib-1-21/embedded/md/tcllib/files/modules/pt/pt_astree.md) 
claims the end location has to be greater than or equal to the start 
position, but since the range includes these locations, that won't allow 
representing a match of length zero. Consider matching the grammar

```
PEG demo ('a' Capture 'c')
Capture <- 'b'* ;
END;
```

against the string `ac`: the resulting abstract syntax tree should 
probably be "`Capture 1 0`".


## Explanations considered useful

These are not gaps as such, but things I believe could do with being 
explained.

### Singleton dictionary idiom

That the single first level key in the PEG serialisation format must be 
`pt::grammar::peg` is clearly stated, but why things are set up that way 
is never explained. I suspect this is an idiom: if the grammar appears in 
a context of type-tagged values, then this `pt::grammar::peg` may be 
interpreted as its type. Type-tagging is usually done with a list, but 
since the value underneath is primarily a dict, it becomes much easier to 
access if the first level is also a dict; the string representation is the 
same either way.

### What is the point of ddigit?

The `<ddigit>` character class appears to be equivalent to the shorter 
`[0-9]`, so what was ever the point of it (besides, by now, backwards 
compatibility)??

Also the claim in the manpage that it is implemented using `regexp` seems 
to be wrong, since it in fact is implemented using `string match`.