Author:		Torsten Berg <[email protected]>
State:		Draft
Type:		Project
Vote:		Pending
Created:	10-Jun-2024
Tcl-Version:	9.1
Tcl-Branch:	documentation-cleanup-for-transition
Keywords:	manual pages, nroff, Markdown, documentation

Abstract

This TIP proposes to change the markup language of the manual pages within the Tcl and Tk source trees from nroff to Markdown. This will be done by a conversion script that catches as much of the original formatting as necessary and possible. The conversion is supposed to ease maintenance of the manual pages, get more people involved in helping to have proper documentation of the Tcl language and the Tk toolkit and to enable a modern visual representation of the HTML version of the manual pages. The goal is also to improve the whole public perception and image of Tcl/Tk.

Rationale

The current manual pages are written in nroff. This markup language is no longer among the languages typically used for documenting software. It was designed to be used with terminal output and line printers, both end points not typically used for reading documentation anymore. As not many people actively use nroff these days, it is becoming increasingly difficult to find people willing to write and maintain documentation using nroff.

The actual text of the manual pages is often hidden behind some macros making it hard to just read and comprehend the content. One example is (from re_synax.n):

A word character is an \fIalnum\fR character or an underscore
.PQ \fB_\fR "" .
These special bracket expressions ...

Here, it is not easy to see that this will be rendered as "... or an underscore (_). These special ...".

The markup is complex and Tcl/Tk uses 31 different nroff macros. Some of them are custom macros only for the manual pages (see file man.macros). The documentation of these macros in that file is sometimes quite sparse and leaves room for misuse or misunderstanding. Some macros are not documented at all (".QR"), others macro documentations are wrong (".PQ"). Together with the obscure syntax of the macro definitions, this can easily lead to errorneous usage of the markup. nroff also mixes both semantic markup (e.g. ".SH" and ".SS" for sections and subsections) and directly visual markup (e.g. ".RS"/".RE" pairs for relative indentation).

A limitation of nroff is the avialability of only two section levels (".SH" and ".SS"). Every time a third level is needed, the manual pages typically emulate this using a tagged paragraph (".TP"). This is visually not distinguishalbe from any other tagged paragraph and is used extensively for command options. Any further subdivision needs to be emulated with e.g. relative insets and bold font (i.e. no semantic markup).

Further, there is no documentation of how and where exactly the individual macros should be used in the manual pages and what the best practices are (or what to avoid) when writing a manual page.

Therefore, this TIP proposes to take the nroff manual pages and convert them into Markdown. This markup language is very wide-spread, easy to learn (even for non-programmers) and also meant to be easy to read and write by humans. Is largely uses semantic markup. There are many variants of Markdown with partly differing syntax. This TIP proposes to use Pandoc's Markdown. This is probably the most flexible and versatile Markdown variant available and is actively developed. There are lots of editors for Markdown supporting the syntax and highlighting text to make the writing and editing visually pleasing. It is flexible enough to support all formatting and layout needs that the Tcl/Tk manual pages have.

Markdown is already used and supported in Fossil, the SCM where Tcl and Tk are hosted. This also means that the manual pages can be viewed within Fossil (they are automatically rendered as HTML when viewed in Fossil's brower-based interface) with some but not all Markdown features supported by Fossil. The Tcl TIP system also now uses Markdown and the changelog for Tcl releases has been changed from plain text to Markdown.

Specification

Subject of this TIP are all manual pages contained in the doc folders of both Tcl and Tk, i.e. the core Tcl and Tk commands and their C API functions. The current main branches of the source trees are the starting points. Changes on the main branch are continuously merged into the TIP branch. Along the way, errors and difficult to convert nroff structures are changed and unified to lead to a proper conversion. At least one construct, namely having multiple consecutive terms in a definition list (nroff uses ".TP" and ".IP" for this) without term definitions is not possible to represent in Markdown. This is taken care of by the conversion script, inserting placeholder texts, and not involve a change to the nroff files.

The conversion script man2markdown.tcl converts the nroff markup into Pandoc's Markdown. Custom Markdown blocks and inline elements are defined in Pandoc's Markdown by using fenced divs and bracketed spans. This is used together with Pandoc filters during the conversion to the output formats to create special output during the conversion from Markdown.

There are two output formats: nroff and HTML. Both are produed by Pandoc with the help of Lua filters which are an integral part of Pandoc's machinery'. The nroff format is generated by the script markdown2nroff.tcl. It reproduces the current man pages as they are now present in the source code as closely as sensible. The HTML output is generated by the script markdown2html.tcl and replaces the current tcltk-man2html.tcl and tcltk-man2html-utils.tcl scripts.

Certain information of the original nroff sources are stored in Markdown metadata (in a YAML frontmatter block) instead of the main body of the text. This enables us to flexibly handle metadata and to decide where to to put (e.g. into the visual part of HTML or just into the source). One example is the sometimes very long list of copyright statements. Another example is the manual section specifier ("n" or "3", "Tcl" or "Tk").

The use of the external Pandoc program will create another dependency for Tcl/Tk. Therefore, the manual pages should be distributed in all three formats (Markdown, HTML, nroff) together with the source distribution so people do not need to also install Pandoc.

Tasks included in this TIP

Write conversion scripts man2markdown.tcl, markdown2html.tcl, markdown2nroff.tcl and their helper Lua scripts.
Do the conversion from the current nroff files to markdown files, including some adjustments and unifications to the original nroff files to make the automatic conversion possible.
Manually edit the generated Markdown files for structures not captured by the conversion script.
Write a Markdown document for the HTML output explaining how the manual pages are structured and formatted. This will include an explanation of how to read the rendered syntax specifications (e.g. what is the meaning og "..." and "?" in the syntax and what is the difference between bold and italic in the syntax). This text is displayed as the index.html page to guide the user. It will also include important links to e.g. the Tcl.md page explaining Tcl syntax in general.
Remove the nroff man page files from the Tcl/Tk source trees and replace them with the converted Markdown files. Also remove tcltk-man2hml.tcl, connected utility scripts and the corresponding build target.
Change the directory structure: Markdown files go into the doc/markdown directory, nroff files into the doc/nroff directory, HTML files into the doc/html directory. There will be sub-directories named "Tcl", "TclCAPI" (in the Tcl repository) and "Tk", "TkCAPI" (in the Tk repository) which then only contain the corresponding manual pages. In the source distribution for end users, the doc/html directory of the Tcl distribution will contain both the Tcl and the Tk HTML manual pages (and thus all four sub-directories) and no HTML manual pages will be placed into th Tk distribution files.

The Markdown representation of the manual pages

There are no links in the text of the nroff version while the rendered HTML (via the tcltk-man2hml.tcl script) has links. The conversion from nroff to Markdown detects places that are links to other manual pages (cross references) and places links into the Markdown version directly.
Section headers and subsection headers will be changed from upper case to title case as the formatting of the title should not be hard-coded into the Markdown text but left to the rendering process to decide.
The command syntax is represented in nroff as direct formatting with italic and bold text. This is changed to a semantic syntax (see below).

Semantic markup for Tcl command syntax

In order to use semantic markup as much as possible and thus gain flexibility in terms of the visual rendering and also for gathering proper indexing information from the manual pages, the synopsis and the other command syntax lines will use a special Markdown construct provided by Pandoc's Markdown and called 'bracketed_spans'. A few examples:

# nroff markup:
\fBarray \fIoption arrayName\fR ?\fIarg arg ...\fR?
\fBarray get\fI arrayName \fR?\fIpattern\fR?
\fBreturn \fR?\fB\-code \fIcode\fR? ?\fIresult\fR?
\fBputs \fR?\fB\-nonewline\fR? ?\fIchannel\fR? \fIstring\fR
\fIpathName \fBaddtag \fItag searchSpec \fR?\fIarg ...\fR?

# simple Markdown:
**array** *option arrayName* ?*arg arg ...*?
**array get** *arrayName* ?*pattern*?
**return** ?**-code** *code*? ?*result*?
**puts** ?**-nonewline**? ?*channel*? *string*
*pathName* **addtag** *tag searchSpec* ?*arg ...*?

# semantic Markdown using 'bracketed_spans':
[array]{.cmd} [option]{.arg} [arrayName]{.arg} [arg arg]{.optdot}
[array]{.cmd} [get]{.sub} [arrayName]{.arg} [pattern]{.optarg}
[return]{cmd} [[-code]{.lit} [code]{.arg}]{.optarg} [result]{.optarg}
[puts]{.cmd} [-nonewline]{.optlit} [channel]{.optarg} [string]{.arg}
[pathName]{.ins} [addtag]{.sub} [tag]{.arg} [searchSpec]{.arg} [arg]{.optdot}

So, the different elements of a command call are supplied with Markdown class attributes (starting with a dot and enclosed in curly braces). All words of a Tcl command are normally defined/written separately (as opposed to the simple markdown syntax), the only exception being a mixture of words belonging to an optional group (such as the [[-code]{.lit} [code]{.arg}]{.optarg} in the return command).

In the following tables, the term 'literal' means that the word needs to be put into the command exactly as it is written in the syntax line. All words not marked as 'literal' are place holders for the user and be (nearly) arbitrary strings.

The first word of a Tcl command can have two different forms:

syntax element	class	explanation	rendering
command name	.cmd	literal word	bold font
instance command name	.ins	command name created by another command, i.e. an instance of some "object"	italic font

The remaining words of a Tcl command can have these forms:

syntax element	class	explanation	rendering
subcommand name	.sub	literal word, can only be directly after a (instance) command name	bold font
mandatory literal argument	.lit	mandatory literal syntax element that is not a command or subcommand	bold font
optional literal argument	.optlit	optional literal syntax element that is not a command or subcommand	bold font between question marks
mandatory argument	.arg	mandatory argument (placeholder for user input)	italic font
optional argument	.optarg	optional argument (placeholder for user input), can only occur once	italic font between question marks
multiple optional argument	.optdot	optional argument (placeholder for user input) that can occur multiple times	italic font between question marks, the word is followed by exactly one space and three dots

Note, that a {.sub} can not be a word starting with a dash. In this case, it is a {.lit}. An example is the pkg::create command where the second word is mandatory syntax and starts with a dash:

# nroff markup:
::pkg::create -name packageName -version packageVersion ?-load filespec? ... ?-source filespec? ...

Note further, that a {.lit} can have the special form [on|off]{.lit}, i.e. with a vertical line splitting the string into two words. This represents alternative forms of the literal string and is used in some commands such asmemory`. Nonetheless, the additional square brackets used in the nroff format for this case, will be deleted before the conversion.

For this new syntax representation to be consistent, the current usage of "..." for optional repetitive arguments in Tcl syntax will be unified in the nroff files before conversion to markdown. Currently, we have at least four different ways these arguments are represented:

# 1. one instance of the repetitive option before the dots:
foreach varlist1 list1 ?varlist2 list2 ...? body
interp invokehidden path ?-option ...? hiddenCmdName ?arg ...?
switch ?options? string pattern body ?pattern body ...?

# 2. two instances of the repetitive option before the dots:
linsert list index ?element element ...?
binary format formatString ?arg arg ...?
binary decode format ?-option value ...? data

# 3. three instances of the repetitive option before the dots:
append varName ?value value value ...?

# 4. dots outside the '?' pair:
::pkg::create -name packageName -version packageVersion ?-load filespec? ... ?-source filespec? ...
::msgcat::mcpreferences ?locale preference? ...

Despite the different ways of representing this syntax element, they all mean the same: a set of one, two or more words that be can be repeated as a set multiple times (including zero times). Because the words always are treated as a set, they should only be written once followed by one instance of "...". Thus, variant 1 is the only variant this type of command argument should be written in. This will be changed in the original nroff files.

The class names indicate what the respective text is representing within the syntax. The {.opt} and {.dotopt} part may contain other syntax elements. This information can subsequently be used to e.g. specify CSS for rendering of these syntax specifications. Literal words are typically written using a bold style, other parts of the syntax are typically using an italic style.

A similar markup will be used for Tk and ttk widget option specifications and for the C API specifications in the manual section 3 pages.

Another syntax to be changed is the use of bold or italic font curly braces in the syntax descriptions. These are used to group a number of words as a composite argument. There are at least three places where this construct is used:

dict for {keyVariable valueVariable} dictionaryValue body
array for {keyVariable valueVariable} arrayName body
-ttycontrol {signal boolean signal boolean ...} (an option of the open command)

Having the braces defined with "\fB" in the nroff sources marks them as mandatory literal value. This is, however, not true. The braces need not be there. The only requirement for the argument is that it is a list having two elements. It can be constructed in any way possible with Tcl, e.g. by using a list variable or quotes. Consequently, this usage should be removed. The manual page for the foreach command shows how to do this: it just uses "varList" in such a place. In general, every argument of a Tcl command should be represented by one word in the syntax description. The same applies to curly braces inside of a "\fI" construct. In the -ttycontrol option of the open command, it is used to define a list with two or more elements (therefore the "..." at the end). Also in such cases, the argument should be one word in the syntax description.

Further, the naming of the syntax elements usually uses camelCase. In some instances this is not the ccase (global being one example). This will be unified so camelCase is used consistently. In this respect it is also desirable to use long version of the argument (variableList) instead of abbreviated ones (varList) fo enhanced clarity.

The tclsh/wish programs give users a short help when typing a wrong syntax (typically "wrong # args: should be ..."). The syntax and wording used for this help tends to be the same as in the manual pages but this is not guaranteed. This TIP also proposes to change and align the help given when running Tcl with the manual pages synopses. The syntax and wording emitted by Tcl shall be the one of the manual pages. A test should be present in the Tcl test suite to provoke all those messages from Tcl and compare them with the manual pages automatically (implementing these tests is not part of the TIP!).

Semantic markup for C API command syntax

The Markdown syntax for the Tcl and Tk C API functions is done in the same way as for the Tcl commands. The only difference is that the class attributes are different. Instead of the attribute .cmd, the attribute .ccmd (c command) is used. The arguments to the functions use the class attribute .cargs. All arguments go into one Markdown span element with this class, separated by commas.

In addition, the C functions sometimes return a value. This value is marked with the class attribute .ret.

Here a few examples:

[Tcl_AllowExceptions]{.ccmd}[interp]{.cargs}
[Tcl_SetByteArrayObj]{.ccmd}[objPtr, bytes, numBytes]{.cargs}
[unsigned char *]{.ret} [Tcl_GetBytesFromObj]{.ccmd}[interp, objPtr, numBytesPtr]{.cargs}

This distinction between Tcl commands and C API commands is done to allow for different rendering of these two types of commands/functions. Note, that e.g. the parenthesis of the C functions are not included in the definitions. The rendering of these is taken care of by the CSS style when doing the HTML output.

The arguments for those C API commands are contained in a separate section of the manual pages as a table-like structure. This will be converted to a definition list in markdown which renders more nicely. For this, the arguments are wrapped into a fenced div "{.arguments}" and the individual arguments use this syntax within the definition list:

[\*interp]{.carg .in type="Tcl_Interp"}
: Interpreter in which to record information.

[\*statPtr]{.carg .out type="struct stat"}
: The structure that contains the result.

The .carg class marks the content as a C API argument, the additional .in, .out or .inout classes mark the argument as being either input or output or both in the C function. They "type" attribute holds the type of the return value of the C function. The whole .carg line is the definition list term while the explanation is the actual definition text.

Markup for links

The conversion script produces cross references to other manual pages which are referred to in the command descriptions. These links use Pandoc's 'Reference links' syntax together with the 'shortcut_reference_links' extension. This means that a cross reference to another manual page can be written with a single pair of square brackts in the most simple case. Example:

See the documentation for the [encoding] command for more details.

If the referenced file is not the same word as the link text, another pair of square brackts indicate the cross reference. Example:

See the documentation for the [binary scan][binary] command for more details.

In both cases, the actual manual page filename to link to is only specified at the bottom of the Markdown page, using this format:

 [encoding]: encoding.md
 [binary]: binary.md
 [Init]: ../TclCAPI/Init.md

Note that the links are paths. Only when referring to pages from the same directory, the path may be omitted. Otherwise it should be specified as a path relative to the document it occurs in. This will allow to also use Mardown viewers and have the links working across pages.

Markdown attributes

In some cases, the markdown-formatted manual pages use markdown attributes (Pandoc-style) to carry additional metadata convering a specific part of the file. One of these cases is the use of the 'type' attribute in C API function arguments (see above). There are two more attributes:

'version' : This is used to mark additions to the manual pages which where introduced as part of a TIP and is typically tied to the ".info" class. The TIP itself is the value of the attribute. E.g., the span [foo bar]{.info version="TIP123"} is used to mark a specific inline text portion as originating from TIP 123. This attribute can occur in spans and divs and can be used to represent this portion visually different from the remaining text.

'DISPLAY' : This marks a div which should be formatted in a special way to resemble a table or to present a larger amount of items in a compact form.

The conversion to HTML

The conversion script markdown2html.tcl does not only convert the individual Markdown files to HTML but also creates an index.html file which serves as the entry point into the HTML version of the manual pages. This is done when building the documentation so the index.html is not a file managed by Fossil in the core repository. However, this file and the generated manual page files depend on some CSS styling. The two corresponding files index.css and tcl-docs.css are part of the managed files. The index page contains the content described in item 4 of the task list for this TIP. It will also contain a navigation panel on the left with a structured outline of the pages, divided into thematic sections plus a filter/search entry field to easier find stuff users are looking for. The keywords contained in the manual page metadata blocks will be extracted and also put into the index file so these are included into the search. See this Wiki page on the Tcl core repository New Tcl manual page navigation structure for more information.

Considerations

The alternative format 'doctools' is not considered because it does not support links and tables ('doctools' is widely used for Tcllib) and the syntax is more verbose and thus less 'readable'. Also, 'AsciiDoc' is not considered as it is not as widespread as Markdown, its syntax is not so easy to read as Markdown and it does not offer anything that is not possible with Pandoc's Markdown already (with regard to the features needed for the manual pages).

Markdown is widely used, there are lots of editors for it and the syntax is easy and visually pleasing. This is an advantage of Markdown vs. doctools to get people write good documentation and keeping the hurdle low. Pandoc's Markdown is very comprehensive (and extensible via fenced_divs and brackeded_spans and filters) and serves as a very good generic format from which other formats can be generated.

The need for the external 'Pandoc' tool is not considered a problem as the nroff processing now also requires an external tool (namely the nroff processor). It should, nonetheless, be a goal to make the conversion from Markdown to HTML and nroff with Tcl itself (not part of this TIP).

Reference implementation

The implementation is in this branch on the Tcl core repository tip-700. It contains the scripts for both Tcl and Tk manual pages and the script assumes that both the Tcl and Tk source code are in directories side by side of each other.

Next steps

The following next steps are logical follow-up actions. They will be the subject of subsequent TIPs when needed. If a TIP is not needed, these steps will be done directly after TIP 700 is implemented.

Enhance structure, outline and content of the manual pages

After the intial conversion, the plan is to manually further unify/enhance the Markdown files both with respect to content and with respect to structure/outline. E.g., there is meta information contained within the nroff manual pages such as Tcl version numbers (the exact meaning of them not being clear) and references to TIPs. This can be used to e.g. (in the Markdown files) provide information on when a certain command, subcommand or command option was introduced into the language (that information can be extracted from the changelog). The information on TIPs can be used to provide links to the actual TIPs for people wanting to reading further details on a specific feature of Tcl/Tk.

The way subcommands and options are represented will be changed. Currently they are presented as 'definition lists' (nroff .TP or .IP macros) or using custom macros. It will be beneficial to represent these as individual subsections so every subcommand/option has its own place in the outline of the manual and thus can automatically be added to a table of contents when processing to e.g. HTML.

In addition, the contents need to be corrected in general, potential misunderstandings removed and other changes should be made as well.

A full list of the current work proposed is on this tcl-core wiki page: Proposed changes to the Tcl/Tk+manual pages

Code blocks

Code blocks in manual pages are inconsistent in terms of how they are written. Both command lines and result lines are written in different ways. Commands are often on lines starting with "%" to explicitly mark them a command line and distinguish them from the resulting output. Sometimes the "%" is there, sometimes not. The "%" sign should be put in front of all command lines. This is how Tcl looks in a terminal anyway and also makes it possible to display the command with a different visual appearance (colour, font etc.) when rendered as HTML.

The results of a command execution are also inconsistently worded in the manual pages' code blocks. Sometimes they appear below the code block as plain text, sometimes they appear in Tcl comments within the code block (append and lassign) and sometimes within the code block and marked with an "->" (lset or lsearch) or just verbatim (lrange). Sometimes, results even occur in code blocks of their own (binary). This will be unfied to clearly distinguish commands from command results and in a way enabling different rendering in HTML.

To add more value to the code blocks, they will be supplemented with buttons when rendered as HTML. One button will allow the code to be copied (just as in the Tcl wiki) and another button will take the code from the block and open it in a "Tcl playground" so it can be tested live there.

Copyright

This document has been placed in the public domain.

TIP 700: Use Markdown instead of nroff for the Tcl/Tk manual pages