Check-in [698fcf8c5b]

4. Produce HTML from Markdown using Pandoc and include the HTML files into the Tcl/Tk source distributions (for convenience, so user do not need to install Pandoc as external tool just in order to prodce HTML versions of the manual pages)
5. Produce nroff manual pages from the Markdown files (for people still wanting the usual manual pages) and include them into the Tcl/Tk source distributions (for convenience also)

## The conversion script and process

Apart from directly converting the manual pages from nroff markup to Markdown, the conversion script will try to also detect certain constructs for enhancing the text. As an example, there are no links in the text of the nroff version while the rendered HTML (via the tcltk-man2hml.tcl script) has links. The conversion script will try to detect places that are good candidates for links and place them into the Markdown version directly. Another example is the conversion of section headers and subsection headers from upper case to title case as the formatting of the title should not be hard-coded into the Markdown text but left to the rendering process to decide. Also, it is considered to change the way subcommands and options are represented. Currently they are presented as 'definition lists' (nroff .TP or .IP macros) or using custom macros. It will be beneficial to represent these as individual subsections so every subcommand/option has its own place in the outline of the manual and can easily be added to a table of contents later.

In order to use semantic markup as much as possible and thus gain flexibility in terms of the visual rendering and also for gathering proper indexing information from the manual pages, the synopsis and the other command syntax lines will use a special Markdown construct provided by Pandoc's Markdown and called 'bracketed_spans'. An example:

```
# nroff markup:
\fBarray \fIoption arrayName\fR ?\fIarg arg ...\fR?
\fBarray get\fI arrayName \fR?\fIpattern\fR?

# simple Markdown:
**array** *option arrayName* ?*arg arg ...*?
**array get** *arrayName* ?*pattern*?

# semantic Markdown using 'bracketed_spans':
[array]{.cmd} [option]{.sub} [arrayName]{.arg} [arg arg]{.dotopt}
[array]{.cmd} [get]{.sub} [arrayName]{.arg} [pattern]{.opt}
```

So, the different elements of a command call are supplied with Markdown class attributes (starting with a dot):

- .cmd = the cammand name
- .sub = the subcommand name
- .arg = a mandatory argument
- .opt = an optional argument rendered with question marks around
- .dotopt = an optional argument that can occur multiple times, rendered with question marks around and a series of dots at the end

The class names indicate what the respective text is representing. This information can subsequently be used to e.g. specify CSS for rendering of these syntax specifications. The whole line can optionally be nested in another bracket_span to mark the line as a command syntax line or it can be put into a fenced_div. However, this will decrease readability further.

A similar markup will be used for Tk and ttk widget option specifications and for the C API specifications in the manual section 3 pages.

# Considerations

The alternative format 'doctools' is not considered because it does not support links and tables ('doctools' is widely used for Tcllib) and the syntax is more verbose and thus less 'readable'. Also, 'AsciiDoc' is not considered as it is not as widespread as Markdown, its syntax is not so easy to read as Markdown and it does not offer anything that is not possible with Pandoc's Markdown already (with regard to the features needed for the manual pages).

Markdown is widely used, there are lots of editors for it and the syntax is easy and visually pleasing. This is an advantage of Markdown vs. doctools to get people write good documentation and keeping the hurdle low. Pandoc's Markdown is very comprehensive (and extensible via fenced_divs and brackeded_spans and filters) and serves as a very good generic format from which other formats can be generated.


# Reference implementation

Work on the conversion script (nroff to Markdown) and the HTML generation is in progress. The implementation is currently hosted [here on chiselapp](https://chiselapp.com/user/stevel/repository/Tcl-Tk-Documentation).


# Next steps

The following next steps are logical follow-up actions. They will be the subject of subsequent TIPs when needed.

After the intial conversion, the plan is to manually unify/enhance the Markdown files both with respect to content and with respect to structure/outline. E.g., there is meta information contained within the nroff manual pages such as Tcl version numbers (the exact meaning of them not being clear) and references to TIPs. This can be used to e.g. (in the Markdown files) provide information on when a certain command, subcommand or command option was introduced into the language (that information can be extracted from the changelog). The information on TIPs can be used to provide links to the actual TIPs for people wanting to reading further details on a specific feature of Tcl/tk. See the [Fossil project on chiselapp](https://chiselapp.com/user/stevel/repository/Tcl-Tk-Documentation/wiki?name=ToDo:+changes+to+the+content+of+manual+pages) where some content change proposals are collected.


After the work on the manual pages, the way is open to continue along this path and make an effort to also update other Tcl/Tk resources. The goal is to address all [four quadrants of documentation](:https://documentation.divio.com):

1. Reference (information-oriented) → manual pages (this TIP)
2. Explanation (understanding-oriented) → e.g. add links to TIPs from within the manual pages, make sure every page has at least one piece of example code (this TIP)
3. How-to guides (problem-oriented) → to be written?
4. Tutorials (learning-oriented) → Tcl Wiki tutorial

With this in mind (and wanting to keep the toolchain to Tcl tools shipped with the core), the plan is to

- convert Markdown to nroff and HTML without the need of Pandoc
- update www.tcl.tk (= www.tcl-lang.org) to a similar modern look just like the HTML manual pages
- generate cheatsheets from the manual pages (e.g. using the accessible semantic command syntax markup)
- publish an updated "Tcl/Tk Pocket Reference" (originally a book by Paul Reines, published by O'Reilly)
- perhaps translate the manual pages to other languages ... at least the dodecalogue as it already has translations on the Wiki


# Copyright

This document has been placed in the public domain.