Attachment "expand.diff" to
ticket [471418ffff]
added by
andreas_kupries
2001-11-10 00:11:21.
Index: ChangeLog
===================================================================
RCS file: /cvsroot/tcllib/tcllib/ChangeLog,v
retrieving revision 1.74
diff -u -r1.74 ChangeLog
--- ChangeLog 2001/11/08 06:44:02 1.74
+++ ChangeLog 2001/11/09 17:06:48
@@ -1,3 +1,7 @@
+2001-11-08 Andreas Kupries <[email protected]>
+
+ * textutil: Added 'expander' code by William H. Duquette.
+
2001-11-07 Andreas Kupries <[email protected]>
* mime: Fixed bug #479174.
Index: modules/textutil/ChangeLog
===================================================================
RCS file: /cvsroot/tcllib/tcllib/modules/textutil/ChangeLog,v
retrieving revision 1.9
diff -u -r1.9 ChangeLog
--- modules/textutil/ChangeLog 2001/10/17 17:27:26 1.9
+++ modules/textutil/ChangeLog 2001/11/09 17:06:48
@@ -1,3 +1,15 @@
+2001-11-08 Andreas Kupries <[email protected]>
+
+ * expander.tcl:
+ * expander.test:
+ * expander.ehtml:
+ * expander.html:
+ * expander.n:
+ * expander_license.txt:
+ * expander_notes.txt: Documentation, code and testsuite for
+ expander objects; the heart of the expand macro processor by
+ William H. Duquette <> packaged up as a library.
+
2001-10-16 Andreas Kupries <[email protected]>
* pkgIndex.tcl:
Index: modules/textutil/expander.ehtml
===================================================================
RCS file: expander.ehtml
diff -N expander.ehtml
--- /dev/null Thu May 24 22:33:05 2001
+++ expander.ehtml Fri Nov 9 09:06:48 2001
@@ -0,0 +1,325 @@
+[pageheader "expander"]
+
+[section SYNOPSIS]
+
+<pre>
+ package require expander 1.0
+</pre><p>
+
+[section DESCRIPTION]
+
+The Tcl "subst" command is often used to support a kind of template
+processing. Given a string with embedded variables or function calls,
+"subst" will interpolate the variable and function values, returning
+the new string:<p>
+
+[listing]
+[tclsh {set greeting "Howdy"}]
+[tclsh {proc place {} {return "World"}}]
+[tclsh {subst {$greeting, [place]!}}]
+%
+[/listing]
+
+By defining a suitable set of Tcl commands, "subst" can be used to
+implement a markup language similar to HTML.<p>
+
+The "subst" command is efficient, but it has three drawbacks for this
+kind of template processing:<p>
+
+<ul>
+ <li> There's no way to identify and process the plain text between two
+ embedded Tcl commands; that makes it difficult to handle plain
+ text in a context-sensitive way.<p>
+
+ <li> Embedded commands are necessarily bracketed by "[lb]" and
+ "[rb]"; it's convenient to be able to choose different brackets
+ in special cases. Someone producing web pages that include a
+ large quantity of Tcl code examples might easily prefer to use
+ "<<" and ">>" as the embedded code delimiters instead.<p>
+
+ <li> There's no easy way to handle incremental input, as one might
+ wish to do when reading data from a socket.<p>
+</ul>
+
+At present, expander solves the first two problems; eventually it will
+solve the third problem as well.<p>
+
+To begin, create an expander object:<p>
+
+[listing]
+[tclsh {package require expander}]
+[tclsh {::expander::expander myexp}]
+%
+[/listing]
+
+The created "::myexp" object can be used to expand text strings containing
+embedded Tcl commands. By default, embedded commands are delimited by
+square brackets. Note that expander doesn't attempt to interpolate
+variables, since variables can be referenced by embedded commands:<p>
+
+[listing]
+[tclsh {set greeting "Howdy"}]
+[tclsh {proc place {} {return "World"}}]
+[tclsh {::myexp expand {[set greeting], [place]!}}]
+%
+[/listing]
+
+[subsection "Embedding Macros"]
+
+An expander macro is simply a Tcl script embedded within a text
+string. Expander evaluates the script in the global context, and
+replaces it with its result string. For example,
+
+[listing]
+[tclsh {set greetings {Howdy Hi "What's up"}}]
+[tclsh {::myexp expand {There are many ways to say "Hello, World!":
+[set result {}
+foreach greeting $greetings {
+ append result "$greeting, World!\n"
+}
+set result]
+And that's just a small sample!}}]
+%
+[/listing]
+
+[subsection "Writing Macro Commands"]
+
+More typically, "macro commands" are used to create a markup
+language. A macro command is just a Tcl command that returns an
+output string. For example, expand can be used to implement a generic
+document markup language that can be retargeted to HTML or any other
+output format:
+
+[listing]
+[tclsh {proc bold {} {return "<b>"}}]
+[tclsh {proc /bold {} {return "</b>"}}]
+[tclsh {::myexp expand {Some of this text is in [bold]boldface[/bold]}}]
+%
+[/listing]
+
+The above definition of "bold" and "/bold" returns HTML, but such
+commands can be as complicated as needed; they could, for example,
+decide what to return based on the desired output format.<p>
+
+[subsection "Changing the Expansion Brackets"]
+
+By default, embedded macros are enclosed in square brackets,
+"[lb]" and "[rb]". If square brackets need to be included in the
+output, the input can contain the [command lb] and [command rb]
+commands. Alternatively, or if square brackets are objectionable for
+some other reason, the macro expansion brackets can be changed to any
+pair of non-empty strings.<p>
+
+The [command setbrackets] command changes the brackets permanently.
+For example, you can write pseudo-html by change them to "<" and ">":<p>
+
+[listing]
+[tclsh {::myexp setbrackets < >}]
+[tclsh {::myexp expand {<bold>This is boldface</bold>}}]
+[/listing]
+
+Alternatively, you can change the expansion brackets temporarily by
+passing the desired brackets to the [command expand] command:<p>
+
+[listing]
+[tclsh {::myexp setbrackets "\[" "\]"}]
+[tclsh {::myexp expand {<bold>This is boldface</bold>} {< >}}]
+%
+[/listing]
+
+[subsection "Using the Context Stack"]
+
+ Often it's desirable to define a pair of macros
+which operate in some way on the plain text between them. Consider a
+set of macros for adding footnotes to a web page: one could
+have implement something like this:<p>
+
+[listing]
+ Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[lb]footnote "See Candide, by Voltaire"[rb]
+[/listing]
+
+The <code>footnote</code> macro would, presumably, assign a number to
+this footnote and save the text to be formatted later on. However,
+this solution is ugly if the footnote text is long or should contain
+additional markup. Consider the following instead:<p>
+
+[listing]
+ Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[lb]footnote[rb]See [lb]bookTitle "Candide"[rb], by
+ [lb]authorsName "Voltaire"[rb], for more information.[lb]/footnote[rb]
+[/listing]
+
+Here the footnote text is contained between <code>footnote</code> and
+<code>/footnote</code> macros, continues onto a second line, and
+contains several macros of its own. This is both clearer and more
+flexible; however, with the features presented so far there's no easy
+way to do it. That's the purpose of the context stack.<p>
+
+All macro expansion takes place in a particular context.
+Here, the <code>footnote</code> macro pushes a new
+context onto the context stack. Then, all expanded text gets placed
+in that new context. <code>/footnote</code> retrieves it by popping
+the context. Here's a skeleton implementation of these two macros:<p>
+
+[listing]
+ proc footnote {} {
+ ::myexp cpush footnote
+ }
+
+ proc /footnote {} {
+ set footnoteText [lb]::myexp cpop footnote[rb]
+
+ # Save the footnote text, and return an appropriate footnote
+ # number and link.
+ }
+[/listing]
+
+The [command cpush] command pushes a new context onto the stack; the
+argument is the context's name. It can be any string, but would
+typically be the name of the macro itself. Then, [command cpop]
+verifies that the current context has the expected name, pops it off
+of the stack, and returns the accumulated text.<p>
+
+Expand provides several other tools related to the context stack.
+Suppose the first macro in a context pair takes arguments or computes
+values which the second macro in the pair needs. After calling
+[command cpush], the first macro can define one or more context
+variables; the second macro can retrieve their values any time before
+calling [command cpop]. For example, suppose the document must
+specify the footnote number explicitly:<p>
+
+[listing]
+ proc footnote {footnoteNumber} {
+ ::myexp cpush footnote
+ ::myexp csave num $footnoteNumber
+ # Return an appropriate link
+ }
+
+ proc /footnote {} {
+ set footnoteNumber [lb]::myexp cget num[rb]
+ set footnoteText [lb]::myexp cpop footnote[rb]
+
+ # Save the footnote text and its footnoteNumber for future
+ # output.
+ }
+[/listing]
+
+At times, it might be desirable to define macros that are valid only
+within a particular context pair; such macros should verify that they
+are only called within the correct context using either
+[command cis] or [command cname].<p>
+
+[section "TCL COMMANDS"]
+
+The package defines the following Tcl commands:<p>
+
+<dl>
+ <dt> [commanddef expander <i>name</i>]
+ <dd> This command creates a new expander object;
+ name is the name of the object, and becomes a new
+ command. By default, if the name isn't fully qualified, i.e.,
+ if it doesn't completely specify the namespace in which to
+ create the new command, the command is created in the caller's
+ current namespace.<p>
+</dl>
+
+[section "EXPANDER OBJECT COMMANDS"]
+
+Every expander object will accept the following
+subcommands:<p>
+
+<dl>
+ <dt> [commanddef cappend <i>text</i>]
+ <dd> Appends a string to the output in the current context. This
+ command should rarely be used by macros or application code.<p>
+
+ <dt> [commanddef cget <i>varname</i>]
+ <dd> Retrieves the value of variable <i>varname</i>, defined in the
+ current context.<p>
+
+ <dt> [commanddef cis <i>cname</i>]
+ <dd> Determines whether or not the name of the current context
+ is <i>cname</i>.<p>
+
+ <dt> [commanddef cname]
+ <dd> Returns the name of the current context.<p>
+
+ <dt> [commanddef cpop <i>cname</i>]
+ <dd> Pops a context from the context stack, returning all accumulated
+ output in that context. The context must be named <i>cname</i>, or
+ an error results.<p>
+
+ <dt> [commanddef cpush <i>cname</i>]
+ <dd> Pushes a context named <i>cname</i> onto the context stack.
+ The context must be popped by [command cpop] before expansion
+ ends or an error results.<p>
+
+ <dt> [commanddef cset <i>varname</i> <i>value</i>]
+ <dd> Sets variable <i>varname</i> to <i>value</i> in the current context.<p>
+
+ <dt> [commanddef cvar <i>varname</i>]
+ <dd> Retrieves the internal variable name of context variable
+ <i>varname</i>; this allows the variable to be passed to
+ commands like <b>lappend</b>.<p>
+
+ <dt> [commanddef errmode ?<i>newErrmode</i>?]
+ <dd> Sets the macro expansion error mode to one of "nothing",
+ "macro", "error", or "fail"; the default value is "fail". The
+ value determines what the expander does if an error is detected
+ during expansion of a macro.<p>
+
+ If the error mode is "fail", the error propagates normally and
+ can be caught or ignored by the application.<p>
+
+ If the error mode is "error", the macro expands into a detailed
+ error message, and expansion continues.<p>
+
+ If the error mode is "macro", the macro expands to itself; that
+ is, it is passed along to the output unchanged.<p>
+
+ If the error mode is "nothing", the macro expands to the empty
+ string, and is effectively ignored.<p>
+
+ <dt> [commanddef expand <i>inputString</i> ?<i>brackets</i>?]
+ <dd> Expands the input string, replacing embedded macros with their
+ expanded values, and returns the expanded string.<p>
+
+ If <i>brackets</i> is given, it must be a list of two strings;
+ the items will be used as the left and right macro expansion
+ bracket sequences for this expansion only.<p>
+
+ <dt> [commanddef lb ?<i>newbracket</i>?]
+ <dd> Returns the current value of the right macro expansion
+ bracket; this is for use as or within a macro, when the bracket
+ needs to be included in the output text. If <i>newbracket</i> is
+ specified, it becomes the new bracket, and is returned.<p>
+
+ <dt> [commanddef rb ?<i>newbracket</i>?]
+ <dd> Returns the current value of the right macro expansion
+ bracket; this is for use as or within a macro, when the bracket
+ needs to be included in the output text. If <i>newbracket</i> is
+ specified, it becomes the new bracket, and is returned.<p>
+
+ <dt> [commanddef reset]
+ <dd> Resets all expander settings to their initial values. Unusual
+ results are likely if this command is called from within a call
+ to [command expand].<p>
+
+ <dt> [commanddef setbrackets <i>lbrack</i> <i>rbrack</i>]
+ <dd> Sets the left and right macro expansion brackets. This command
+ is for use as or within a macro, or to permanently change the
+ bracket definitions. By default, the brackets are "[lb]" and
+ "[rb]", but any non-empty string can be used; for example,
+ "<" and ">" or "(*" and "*)" or even "Hello," and "World!".<p>
+<p>
+</dl>
+
+
+[section "HISTORY"]
+
+expander was written by William H. Duquette; it is a repackaging of
+the central algorithm of the
+[link http://www.wjduquette.com/expand "expand"] macro processing tool.<p>
+
+[copyright 2001 "William H. Duquette"]
Index: modules/textutil/expander.html
===================================================================
RCS file: expander.html
diff -N expander.html
--- /dev/null Thu May 24 22:33:05 2001
+++ expander.html Fri Nov 9 09:06:48 2001
@@ -0,0 +1,331 @@
+<html><head>
+ <title>expander</title>
+</head>
+
+<BODY>
+
+<h1>expander</h1>
+
+<h2><a name="synopsis">SYNOPSIS</a></h2>
+
+<pre>
+ package require expander 1.0
+</pre><p>
+
+<h2><a name="description">DESCRIPTION</a></h2>
+
+The Tcl "subst" command is often used to support a kind of template
+processing. Given a string with embedded variables or function calls,
+"subst" will interpolate the variable and function values, returning
+the new string:<p>
+
+<pre>% set greeting "Howdy"
+Howdy
+% proc place {} {return "World"}
+% subst {$greeting, [place]!}
+Howdy, World!
+%</pre>
+
+By defining a suitable set of Tcl commands, "subst" can be used to
+implement a markup language similar to HTML.<p>
+
+The "subst" command is efficient, but it has three drawbacks for this
+kind of template processing:<p>
+
+<ul>
+ <li> There's no way to identify and process the plain text between two
+ embedded Tcl commands; that makes it difficult to handle plain
+ text in a context-sensitive way.<p>
+
+ <li> Embedded commands are necessarily bracketed by "[" and
+ "]"; it's convenient to be able to choose different brackets
+ in special cases. Someone producing web pages that include a
+ large quantity of Tcl code examples might easily prefer to use
+ "<<" and ">>" as the embedded code delimiters instead.<p>
+
+ <li> There's no easy way to handle incremental input, as one might
+ wish to do when reading data from a socket.<p>
+</ul>
+
+At present, expander solves the first two problems; eventually it will
+solve the third problem as well.<p>
+
+To begin, create an expander object:<p>
+
+<pre>% package require expander
+1.0
+% ::expander::expander myexp
+::myexp
+%</pre>
+
+The created "::myexp" object can be used to expand text strings containing
+embedded Tcl commands. By default, embedded commands are delimited by
+square brackets. Note that expander doesn't attempt to interpolate
+variables, since variables can be referenced by embedded commands:<p>
+
+<pre>% set greeting "Howdy"
+Howdy
+% proc place {} {return "World"}
+% ::myexp expand {[set greeting], [place]!}
+Howdy, World!
+%</pre>
+
+<h3><a name="embedding_macros">Embedding Macros</a></h3>
+
+An expander macro is simply a Tcl script embedded within a text
+string. Expander evaluates the script in the global context, and
+replaces it with its result string. For example,
+
+<pre>% set greetings {Howdy Hi "What's up"}
+Howdy Hi "What's up"
+% ::myexp expand {There are many ways to say "Hello, World!":
+[set result {}
+foreach greeting $greetings {
+ append result "$greeting, World!\n"
+}
+set result]
+And that's just a small sample!}
+There are many ways to say "Hello, World!":
+Howdy, World!
+Hi, World!
+What's up, World!
+
+And that's just a small sample!
+%</pre>
+
+<h3><a name="writing_macro_commands">Writing Macro Commands</a></h3>
+
+More typically, "macro commands" are used to create a markup
+language. A macro command is just a Tcl command that returns an
+output string. For example, expand can be used to implement a generic
+document markup language that can be retargeted to HTML or any other
+output format:
+
+<pre>% proc bold {} {return "<b>"}
+% proc /bold {} {return "</b>"}
+% ::myexp expand {Some of this text is in [bold]boldface[/bold]}
+Some of this text is in <b>boldface</b>
+%</pre>
+
+The above definition of "bold" and "/bold" returns HTML, but such
+commands can be as complicated as needed; they could, for example,
+decide what to return based on the desired output format.<p>
+
+<h3><a name="changing_the_expansion_brackets">Changing the Expansion Brackets</a></h3>
+
+By default, embedded macros are enclosed in square brackets,
+"[" and "]". If square brackets need to be included in the
+output, the input can contain the <code><a href="#lb">lb</a></code> and <code><a href="#rb">rb</a></code>
+commands. Alternatively, or if square brackets are objectionable for
+some other reason, the macro expansion brackets can be changed to any
+pair of non-empty strings.<p>
+
+The <code><a href="#setbrackets">setbrackets</a></code> command changes the brackets permanently.
+For example, you can write pseudo-html by change them to "<" and ">":<p>
+
+<pre>% ::myexp setbrackets < >
+% ::myexp expand {<bold>This is boldface</bold>}
+<b>This is boldface</b></pre>
+
+Alternatively, you can change the expansion brackets temporarily by
+passing the desired brackets to the <code><a href="#expand">expand</a></code> command:<p>
+
+<pre>% ::myexp setbrackets "\[" "\]"
+% ::myexp expand {<bold>This is boldface</bold>} {< >}
+<b>This is boldface</b>
+%</pre>
+
+<h3><a name="using_the_context_stack">Using the Context Stack</a></h3>
+
+ Often it's desirable to define a pair of macros
+which operate in some way on the plain text between them. Consider a
+set of macros for adding footnotes to a web page: one could
+have implement something like this:<p>
+
+<pre> Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[footnote "See Candide, by Voltaire"]</pre>
+
+The <code>footnote</code> macro would, presumably, assign a number to
+this footnote and save the text to be formatted later on. However,
+this solution is ugly if the footnote text is long or should contain
+additional markup. Consider the following instead:<p>
+
+<pre> Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[footnote]See [bookTitle "Candide"], by
+ [authorsName "Voltaire"], for more information.[/footnote]</pre>
+
+Here the footnote text is contained between <code>footnote</code> and
+<code>/footnote</code> macros, continues onto a second line, and
+contains several macros of its own. This is both clearer and more
+flexible; however, with the features presented so far there's no easy
+way to do it. That's the purpose of the context stack.<p>
+
+All macro expansion takes place in a particular context.
+Here, the <code>footnote</code> macro pushes a new
+context onto the context stack. Then, all expanded text gets placed
+in that new context. <code>/footnote</code> retrieves it by popping
+the context. Here's a skeleton implementation of these two macros:<p>
+
+<pre> proc footnote {} {
+ ::myexp cpush footnote
+ }
+
+ proc /footnote {} {
+ set footnoteText [::myexp cpop footnote]
+
+ # Save the footnote text, and return an appropriate footnote
+ # number and link.
+ } </pre>
+
+The <code><a href="#cpush">cpush</a></code> command pushes a new context onto the stack; the
+argument is the context's name. It can be any string, but would
+typically be the name of the macro itself. Then, <code><a href="#cpop">cpop</a></code>
+verifies that the current context has the expected name, pops it off
+of the stack, and returns the accumulated text.<p>
+
+Expand provides several other tools related to the context stack.
+Suppose the first macro in a context pair takes arguments or computes
+values which the second macro in the pair needs. After calling
+<code><a href="#cpush">cpush</a></code>, the first macro can define one or more context
+variables; the second macro can retrieve their values any time before
+calling <code><a href="#cpop">cpop</a></code>. For example, suppose the document must
+specify the footnote number explicitly:<p>
+
+<pre> proc footnote {footnoteNumber} {
+ ::myexp cpush footnote
+ ::myexp csave num $footnoteNumber
+ # Return an appropriate link
+ }
+
+ proc /footnote {} {
+ set footnoteNumber [::myexp cget num]
+ set footnoteText [::myexp cpop footnote]
+
+ # Save the footnote text and its footnoteNumber for future
+ # output.
+ } </pre>
+
+At times, it might be desirable to define macros that are valid only
+within a particular context pair; such macros should verify that they
+are only called within the correct context using either
+<code><a href="#cis">cis</a></code> or <code><a href="#cname">cname</a></code>.<p>
+
+<h2><a name="tcl_commands">TCL COMMANDS</a></h2>
+
+The package defines the following Tcl commands:<p>
+
+<dl>
+ <dt> <code><a name="expander">expander <i>name</i></a></code>
+ <dd> This command creates a new expander object;
+ name is the name of the object, and becomes a new
+ command. By default, if the name isn't fully qualified, i.e.,
+ if it doesn't completely specify the namespace in which to
+ create the new command, the command is created in the caller's
+ current namespace.<p>
+</dl>
+
+<h2><a name="expander_object_commands">EXPANDER OBJECT COMMANDS</a></h2>
+
+Every expander object will accept the following
+subcommands:<p>
+
+<dl>
+ <dt> <code><a name="cappend">cappend <i>text</i></a></code>
+ <dd> Appends a string to the output in the current context. This
+ command should rarely be used by macros or application code.<p>
+
+ <dt> <code><a name="cget">cget <i>varname</i></a></code>
+ <dd> Retrieves the value of variable <i>varname</i>, defined in the
+ current context.<p>
+
+ <dt> <code><a name="cis">cis <i>cname</i></a></code>
+ <dd> Determines whether or not the name of the current context
+ is <i>cname</i>.<p>
+
+ <dt> <code><a name="cname">cname</a></code>
+ <dd> Returns the name of the current context.<p>
+
+ <dt> <code><a name="cpop">cpop <i>cname</i></a></code>
+ <dd> Pops a context from the context stack, returning all accumulated
+ output in that context. The context must be named <i>cname</i>, or
+ an error results.<p>
+
+ <dt> <code><a name="cpush">cpush <i>cname</i></a></code>
+ <dd> Pushes a context named <i>cname</i> onto the context stack.
+ The context must be popped by <code><a href="#cpop">cpop</a></code> before expansion
+ ends or an error results.<p>
+
+ <dt> <code><a name="cset">cset <i>varname</i> <i>value</i></a></code>
+ <dd> Sets variable <i>varname</i> to <i>value</i> in the current context.<p>
+
+ <dt> <code><a name="cvar">cvar <i>varname</i></a></code>
+ <dd> Retrieves the internal variable name of context variable
+ <i>varname</i>; this allows the variable to be passed to
+ commands like <b>lappend</b>.<p>
+
+ <dt> <code><a name="errmode">errmode ?<i>newErrmode</i>?</a></code>
+ <dd> Sets the macro expansion error mode to one of "nothing",
+ "macro", "error", or "fail"; the default value is "fail". The
+ value determines what the expander does if an error is detected
+ during expansion of a macro.<p>
+
+ If the error mode is "fail", the error propagates normally and
+ can be caught or ignored by the application.<p>
+
+ If the error mode is "error", the macro expands into a detailed
+ error message, and expansion continues.<p>
+
+ If the error mode is "macro", the macro expands to itself; that
+ is, it is passed along to the output unchanged.<p>
+
+ If the error mode is "nothing", the macro expands to the empty
+ string, and is effectively ignored.<p>
+
+ <dt> <code><a name="expand">expand <i>inputString</i> ?<i>brackets</i>?</a></code>
+ <dd> Expands the input string, replacing embedded macros with their
+ expanded values, and returns the expanded string.<p>
+
+ If <i>brackets</i> is given, it must be a list of two strings;
+ the items will be used as the left and right macro expansion
+ bracket sequences for this expansion only.<p>
+
+ <dt> <code><a name="lb">lb ?<i>newbracket</i>?</a></code>
+ <dd> Returns the current value of the right macro expansion
+ bracket; this is for use as or within a macro, when the bracket
+ needs to be included in the output text. If <i>newbracket</i> is
+ specified, it becomes the new bracket, and is returned.<p>
+
+ <dt> <code><a name="rb">rb ?<i>newbracket</i>?</a></code>
+ <dd> Returns the current value of the right macro expansion
+ bracket; this is for use as or within a macro, when the bracket
+ needs to be included in the output text. If <i>newbracket</i> is
+ specified, it becomes the new bracket, and is returned.<p>
+
+ <dt> <code><a name="reset">reset</a></code>
+ <dd> Resets all expander settings to their initial values. Unusual
+ results are likely if this command is called from within a call
+ to <code><a href="#expand">expand</a></code>.<p>
+
+ <dt> <code><a name="setbrackets">setbrackets <i>lbrack</i> <i>rbrack</i></a></code>
+ <dd> Sets the left and right macro expansion brackets. This command
+ is for use as or within a macro, or to permanently change the
+ bracket definitions. By default, the brackets are "[" and
+ "]", but any non-empty string can be used; for example,
+ "<" and ">" or "(*" and "*)" or even "Hello," and "World!".<p>
+<p>
+</dl>
+
+
+<h2><a name="history">HISTORY</a></h2>
+
+expander was written by William H. Duquette; it is a repackaging of
+the central algorithm of the
+<a href="http://www.wjduquette.com/expand">expand</a> macro processing tool.<p>
+
+
+<p><hr><p>
+Copyright © 2001, by William H. Duquette. All rights reserved.<p>
+
+
+</body>
+</html>
Index: modules/textutil/expander.n
===================================================================
RCS file: expander.n
diff -N expander.n
--- /dev/null Thu May 24 22:33:05 2001
+++ expander.n Fri Nov 9 09:06:48 2001
@@ -0,0 +1,368 @@
+'\"
+'\" Copyright (c) 1998-2000 by nobody :-)
+'\" All rights not reserved.
+'\"
+'\" RCS: @(#) $Id: textutil.n,v 1.7 2001/10/17 17:27:26 andreas_kupries Exp $
+'\"
+.so man.macros
+.TH expander n 1.0 Textutil "Text expansion and template processing"
+.BS
+'\" Note: do not modify the .SH NAME line immediately below!
+.SH NAME
+textutil::expander \- Procedures to process templates and expand text.
+.SH SYNOPSIS
+.BS
+\fBpackage require Tcl 8.2\fR
+.sp
+\fBpackage require textutil::expander ?1.0?\fR
+.sp
+\fB::textutil::expander\fR \fIexpanderName\fR
+.BE
+.SH DESCRIPTION
+.PP
+The Tcl \fBsubst\fR command is often used to support a kind of
+template processing. Given a string with embedded variables or
+function calls, \fBsubst\fR will interpolate the variable and function
+values, returning the new string:
+.PP
+.CS
+ % set greeting "Howdy"
+ Howdy
+ % proc place {} {return "World"}
+ % subst {$greeting, [place]!}
+ Howdy, World!
+ %
+.CE
+.PP
+By defining a suitable set of Tcl commands, \fBsubst\fR can be used to
+implement a markup language similar to HTML.
+.sp
+The \fBsubst\fR command is efficient, but it has three drawbacks for
+this kind of template processing:
+.TP
+*
+There's no way to identify and process the plain text between two
+embedded Tcl commands; that makes it difficult to handle plain text in
+a context-sensitive way.
+.TP
+*
+Embedded commands are necessarily bracketed by \fB[\fR and \fB]\fR;
+it's convenient to be able to choose different brackets in special
+cases. Someone producing web pages that include a large quantity of
+Tcl code examples might easily prefer to use \fB<<\fR and \fB>>\fR as
+the embedded code delimiters instead.
+.TP
+*
+There's no easy way to handle incremental input, as one might wish to
+do when reading data from a socket.
+
+.PP
+At present, expander solves the first two problems; eventually it will
+solve the third problem as well.
+
+The following section describes the command API to the expander; this
+is followed by tutorial section.
+
+.SH EXPANDER API
+.PP
+The \fBtextutil::expander\fR package provides only one command,
+described below. The rest of the section is taken by a description of
+the methods for the exapnder objects created by this command.
+.TP
+\fB::textutil::expander\fR \fIexpanderName\fR
+The \fB::textutil::expander\fR command creates a new expander object
+with an associated Tcl command whose name is \fIexpanderName\fR. This
+command may be used to invoke various operations on the graph. If the
+\fIexpanderName\fR is not fully qualified it is interpreted as
+relative to the current namespace. The command has the following
+general form:
+.CS
+\fIexpanderName option \fR?\fIarg arg ...\fR?
+.CE
+\fIOption\fR and the \fIarg\fRs determine the exact behavior of the
+command.
+.PP
+The following commands are possible for expander objects:
+.TP
+\fIexpanderName\fR \fBcappend\fR \fItext\fR
+Appends a string to the output in the current context. This command
+should rarely be used by macros or application code.
+.TP
+\fIexpanderName\fR \fBcget\fR \fIvarname\fR
+Retrieves the value of variable \fIvarname\fR, defined in the current
+context.
+.TP
+\fIexpanderName\fR \fBcis\fR \fIcname\fR
+Determines whether or not the name of the current context is
+\fIcname\fR.
+.TP
+\fIexpanderName\fR \fBcname\fR
+Returns the name of the current context.
+.TP
+\fIexpanderName\fR \fBcpop\fR \fIcname\fR
+Pops a context from the context stack, returning all accumulated
+output in that context. The context must be named \fIcname\fR, or an
+error results.
+.TP
+\fIexpanderName\fR \fBcpush\fR \fIcname\fR
+Pushes a context named \fIcname\fR onto the context stack. The
+context must be popped by \fBcpop\fR before expansion ends or an error
+results.
+.TP
+\fIexpanderName\fR \fBcset\fR \fIvarname\fR
+Sets variable \fIvarname\fR to \fIvalue\fR in the current context.
+.TP
+\fIexpanderName\fR \fBcvar\fR \fIvarname\fR
+Retrieves the internal variable name of context variable
+\fIvarname\fR; this allows the variable to be passed to commands like
+\fBlappend\fR.
+.TP
+\fIexpanderName\fR \fBerrmode\fR \fInewErrmode\fR
+Sets the macro expansion error mode to one of \fBnothing\fR,
+\fBmacro\fR, \fBerror\fR, or \fBfail\fR; the default value is
+\fBfail\fR. The value determines what the expander does if an error
+is detected during expansion of a macro.
+.RS
+.TP
+*
+If the error mode is \fBfail\fR, the error propagates normally and can
+be caught or ignored by the application.
+.TP
+*
+If the error mode is \fBerror\fR, the macro expands into a detailed
+error message, and expansion continues.
+.TP
+*
+If the error mode is \fBmacro\fR, the macro expands to itself; that
+is, it is passed along to the output unchanged.
+.TP
+*
+If the error mode is \fBnothing\fR, the macro expands to the empty
+string, and is effectively ignored.
+.RE
+.TP
+\fIexpanderName\fR \fBexpand\fR \fIstring\fR ?\fIbrackets\fR?
+Expands the input string, replacing embedded macros with their
+expanded values, and returns the expanded string.
+.sp
+If \fIbrackets\fR is given, it must be a list of two strings; the
+items will be used as the left and right macro expansion bracket
+sequences for this expansion only.
+.TP
+\fIexpanderName\fR \fBlb\fR ?\fInewbracket\fR?
+Returns the current value of the right macro expansion bracket; this
+is for use as or within a macro, when the bracket needs to be included
+in the output text. If \fInewbracket\fR is specified, it becomes the
+new bracket, and is returned.
+.TP
+\fIexpanderName\fR \fBrb\fR ?\fInewbracket\fR?
+Returns the current value of the right macro expansion bracket; this
+is for use as or within a macro, when the bracket needs to be included
+in the output text. If \fInewbracket\fR is specified, it becomes the
+new bracket, and is returned.
+.TP
+\fIexpanderName\fR \fBreset\fR
+Resets all expander settings to their initial values. Unusual results
+are likely if this command is called from within a call to
+\fBexpand\fR.
+.TP
+\fIexpanderName\fR \fBsetbrackets\fR \fIlbrack rbrack\fR
+Sets the left and right macro expansion brackets. This command is for
+use as or within a macro, or to permanently change the bracket
+definitions. By default, the brackets are \fB[\fR and \fB]\fR, but
+any non-empty string can be used; for example, \fB<\fR and \fB>\fR or
+\fB(*\fR and \fB*)\fR or even \fBHello,\fR and \fBWorld!\fR.
+.SH TUTORIAL
+.PP
+To begin, create an expander object:
+.PP
+.CS
+ % package require expander
+ 1.0
+ % ::expander::expander myexp
+ ::myexp
+ %
+.CE
+.PP
+The created \fB::myexp\fR object can be used to expand text strings
+containing embedded Tcl commands. By default, embedded commands are
+delimited by square brackets. Note that expander doesn't attempt to
+interpolate variables, since variables can be referenced by embedded
+commands:
+.PP
+.CS
+ % set greeting "Howdy"
+ Howdy
+ % proc place {} {return "World"}
+ % ::myexp expand {[set greeting], [place]!}
+ Howdy, World!
+ %
+.CE
+.PP
+\fBEmbedding Macros\fR
+.PP
+An expander macro is simply a Tcl script embedded within a text
+string. Expander evaluates the script in the global context, and
+replaces it with its result string. For example,
+.PP
+.CS
+ % set greetings {Howdy Hi "What's up"}
+ Howdy Hi "What's up"
+ % ::myexp expand {There are many ways to say "Hello, World!":
+ [set result {}
+ foreach greeting $greetings {
+ append result "$greeting, World!\n"
+ }
+ set result]
+ And that's just a small sample!}
+ There are many ways to say "Hello, World!":
+ Howdy, World!
+ Hi, World!
+ What's up, World!
+
+ And that's just a small sample!
+ %
+.CE
+.PP
+\fBWriting Macro Commands\fR
+.PP
+More typically, \fImacro commands\fR are used to create a markup
+language. A macro command is just a Tcl command that returns an
+output string. For example, expand can be used to implement a generic
+document markup language that can be retargeted to HTML or any other
+output format:
+.PP
+.CS
+ % proc bold {} {return "<b>"}
+ % proc /bold {} {return "</b>"}
+ % ::myexp expand {Some of this text is in [bold]boldface[/bold]}
+ Some of this text is in <b&>boldface</b>
+ %
+.CE
+.PP
+The above definitions of \fBbold\fR and \fB/bold\fR returns HTML, but
+such commands can be as complicated as needed; they could, for
+example, decide what to return based on the desired output format.
+.PP
+\fBChanging the Expansion Brackets\fR
+.PP
+By default, embedded macros are enclosed in square brackets, \fB[\fR
+and \fB]\fR. If square brackets need to be included in the output,
+the input can contain the \fBlb\fR and \fBrb\fR commands.
+Alternatively, or if square brackets are objectionable for some other
+reason, the macro expansion brackets can be changed to any pair of
+non-empty strings.
+.PP
+The \fBsetbrackets\fR command changes the brackets permanently. For
+example, you can write pseudo-html by change them to \fB<\fR and
+\fB>\fR:
+.PP
+.CS
+ % ::myexp setbrackets < >
+ % ::myexp expand {<bold>This is boldface</bold>}
+ <b>This is boldface</b>
+.CE
+.PP
+Alternatively, you can change the expansion brackets temporarily by
+passing the desired brackets to the \fBexpand\fR command:
+.PP
+.CS
+ % ::myexp setbrackets "\\[" "\\]"
+ % ::myexp expand {<bold>This is boldface</bold>} {< >}
+ <b>This is boldface</b>
+ %
+.CE
+.PP
+\fBUsing the Context Stack\fR
+.PP
+Often it's desirable to define a pair of macros which operate in some
+way on the plain text between them. Consider a set of macros for
+adding footnotes to a web page: one could have implement something
+like this:
+.PP
+.CS
+ Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[footnote "See Candide, by Voltaire"]
+.CE
+.PP
+The \fBfootnote\fR macro would, presumably, assign a number to this
+footnote and save the text to be formatted later on. However, this
+solution is ugly if the footnote text is long or should contain
+additional markup. Consider the following instead:
+.PP
+.CS
+ Dr. Pangloss, however, thinks that this is the best of all
+ possible worlds.[footnote]See [bookTitle "Candide"], by
+ [authorsName "Voltaire"], for more information.[/footnote]
+.CE
+.PP
+Here the footnote text is contained between \fBfootnote\fR and
+\fB/footnote\fR macros, continues onto a second line, and contains
+several macros of its own. This is both clearer and more flexible;
+however, with the features presented so far there's no easy way to do
+it. That's the purpose of the context stack.
+.PP
+All macro expansion takes place in a particular context. Here, the
+\fBfootnote\fR macro pushes a new context onto the context stack.
+Then, all expanded text gets placed in that new context.
+\fB/footnote\fR retrieves it by popping the context. Here's a
+skeleton implementation of these two macros:
+.PP
+.CS
+ proc footnote {} {
+ ::myexp cpush footnote
+ }
+
+ proc /footnote {} {
+ set footnoteText [::myexp cpop footnote]
+
+ # Save the footnote text, and return an appropriate footnote
+ # number and link.
+ }
+.CE
+.PP
+The \fBcpush\fR command pushes a new context onto the stack; the
+argument is the context's name. It can be any string, but would
+typically be the name of the macro itself. Then, \fBcpop\fR verifies
+that the current context has the expected name, pops it off of the
+stack, and returns the accumulated text.
+.PP
+Expand provides several other tools related to the context stack.
+Suppose the first macro in a context pair takes arguments or computes
+values which the second macro in the pair needs. After calling
+\fBcpush\fR, the first macro can define one or more context variables;
+the second macro can retrieve their values any time before calling
+\fBcpop\fR. For example, suppose the document must specify the
+footnote number explicitly:
+.PP
+.CS
+ proc footnote {footnoteNumber} {
+ ::myexp cpush footnote
+ ::myexp csave num $footnoteNumber
+ # Return an appropriate link
+ }
+
+ proc /footnote {} {
+ set footnoteNumber [::myexp cget num]
+ set footnoteText [::myexp cpop footnote]
+
+ # Save the footnote text and its footnoteNumber for future
+ # output.
+ }
+.CE
+.PP
+At times, it might be desirable to define macros that are valid only
+within a particular context pair; such macros should verify that they
+are only called within the correct context using either \fBcis\fR or
+\fBcname\fR.
+
+.SH HISTORY
+.PP
+\fBexpander\fR was written by William H. Duquette; it is a repackaging
+of the central algorithm of the expand macro processing tool.
+
+.SH SEE ALSO
+regexp, split, string, http://www.wjduquette.com/expand
+
+.SH KEYWORDS
+string, template processing, text expansion
Index: modules/textutil/expander.tcl
===================================================================
RCS file: expander.tcl
diff -N expander.tcl
--- /dev/null Thu May 24 22:33:05 2001
+++ expander.tcl Fri Nov 9 09:06:48 2001
@@ -0,0 +1,755 @@
+#---------------------------------------------------------------------
+# TITLE:
+# expander.tcl
+#
+# AUTHOR:
+# Will Duquette
+#
+# DESCRIPTION:
+#
+# An expander is an object that takes as input text with embedded
+# Tcl code and returns text with the embedded code expanded. The
+# text can be provided all at once or incrementally.
+#
+# See expander.[e]html for usage info.
+# Also expander.n
+#
+# LICENSE:
+# Copyright (C) 2001 by William H. Duquette. See expander_license.txt,
+# distributed with this file, for license information.
+#
+# CHANGE LOG:
+#
+# 10/31/01: V1.0 code is complete.
+
+# Provide the package.
+
+# Create the package's namespace.
+
+namespace eval ::textutil {
+ namespace eval expander {
+ # All indices are prefixed by "$exp-".
+ #
+ # lb The left bracket sequence
+ # rb The right bracket sequence
+ # errmode How to handle macro errors:
+ # nothing, macro, error, fail.
+ # level The context level
+ # output-$level The accumulated text at this context level.
+ # name-$level The tag name of this context level
+ # data-$level-$var A variable of this context level
+
+ variable Info
+
+ # In methods, the current object:
+ variable This ""
+
+ # Export public commands
+ namespace export expander
+ }
+
+ #namespace import expander::*
+ namespace export expander
+
+ proc expander {name} {uplevel ::textutil::expander::expander [list $name]}
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# expander name
+#
+# INPUTS:
+# name A proc name for the new object. If not
+# fully-qualified, it is assumed to be relative
+# to the caller's namespace.
+#
+# RETURNS:
+# nothing
+#
+# DESCRIPTION:
+# Creates a new expander object.
+
+proc ::textutil::expander::expander {name} {
+ variable Info
+
+ # FIRST, qualify the name.
+ if {![string match "::*" $name]} {
+ # Get caller's namespace; append :: if not global namespace.
+ set ns [uplevel 1 namespace current]
+ if {"::" != $ns} {
+ append ns "::"
+ }
+
+ set name "$ns$name"
+ }
+
+ # NEXT, Check the name
+ if {"" != [info command $name]} {
+ return -code error "command name \"$name\" already exists"
+ }
+
+ # NEXT, Create the object.
+ proc $name {method args} [format {
+ if {[catch {::textutil::expander::Methods %s $method $args} result]} {
+ return -code error $result
+ } else {
+ return $result
+ }
+ } $name]
+
+ # NEXT, Initialize the object
+ Op_reset $name
+
+ return $name
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Methods name method argList
+#
+# INPUTS:
+# name The object's fully qualified procedure name.
+# This argument is provided by the object command
+# itself.
+# method The method to call.
+# argList Arguments for the specific method.
+#
+# RETURNS:
+# Depends on the method
+#
+# DESCRIPTION:
+# Handles all method dispatch for a expander object.
+# The expander's object command merely passes its arguments to
+# this function, which dispatches the arguments to the
+# appropriate method procedure. If the method raises an error,
+# the method procedure's name in the error message is replaced
+# by the object and method names.
+
+proc ::textutil::expander::Methods {name method argList} {
+ variable Info
+ variable This
+
+ switch -exact -- $method {
+ expand -
+ lb -
+ rb -
+ setbrackets -
+ errmode -
+ cpush -
+ cis -
+ cname -
+ cset -
+ cget -
+ cvar -
+ cpop -
+ cappend -
+ reset {
+ # FIRST, execute the method, first setting This to the object
+ # name; then, after the method has been called, restore the
+ # old object name.
+ set oldThis $This
+ set This $name
+
+ set retval [catch "Op_$method $name $argList" result]
+
+ set This $oldThis
+
+ # NEXT, handle the result based on the retval.
+ if {$retval} {
+ regsub -- "Op_$method" $result "$name $method" result
+ return -code error $result
+ } else {
+ return $result
+ }
+ }
+ default {
+ return -code error "\"$name $method\" is not defined"
+ }
+ }
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Get key
+#
+# INPUTS:
+# key A key into the Info array, excluding the
+# object name. E.g., "lb"
+#
+# RETURNS:
+# The value from the array
+#
+# DESCRIPTION:
+# Gets the value of an entry from Info for This.
+
+proc ::textutil::expander::Get {key} {
+ variable Info
+ variable This
+
+ return $Info($This-$key)
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Set key value
+#
+# INPUTS:
+# key A key into the Info array, excluding the
+# object name. E.g., "lb"
+#
+# value A Tcl value
+#
+# RETURNS:
+# The value
+#
+# DESCRIPTION:
+# Sets the value of an entry in Info for This.
+
+proc ::textutil::expander::Set {key value} {
+ variable Info
+ variable This
+
+ return [set Info($This-$key) $value]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Var key
+#
+# INPUTS:
+# key A key into the Info array, excluding the
+# object name. E.g., "lb"
+#
+# RETURNS:
+# The full variable name, suitable for setting or lappending
+
+proc ::textutil::expander::Var {key} {
+ variable Info
+ variable This
+
+ return ::textutil::expander::Info($This-$key)
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Contains list value
+#
+# INPUTS:
+# list any list
+# value any value
+#
+# RETURNS:
+# TRUE if the list contains the value, and false otherwise.
+
+proc ::textutil::expander::Contains {list value} {
+ if {[lsearch -exact $list $value] == -1} {
+ return 0
+ } else {
+ return 1
+ }
+}
+
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_lb ?newbracket?
+#
+# INPUTS:
+# newbracket If given, the new bracket token.
+#
+# RETURNS:
+# The current left bracket
+#
+# DESCRIPTION:
+# Returns the current left bracket token.
+
+proc ::textutil::expander::Op_lb {name {newbracket ""}} {
+ if {[string length $newbracket] != 0} {
+ Set lb $newbracket
+ }
+ return [Get lb]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_rb ?newbracket?
+#
+# INPUTS:
+# newbracket If given, the new bracket token.
+#
+# RETURNS:
+# The current left bracket
+#
+# DESCRIPTION:
+# Returns the current left bracket token.
+
+proc ::textutil::expander::Op_rb {name {newbracket ""}} {
+ if {[string length $newbracket] != 0} {
+ Set rb $newbracket
+ }
+ return [Get rb]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_setbrackets lbrack rbrack
+#
+# INPUTS:
+# lbrack The new left bracket
+# rbrack The new right bracket
+#
+# RETURNS:
+# nothing
+#
+# DESCRIPTION:
+# Sets the brackets as a pair.
+
+proc ::textutil::expander::Op_setbrackets {name lbrack rbrack} {
+ Set lb $lbrack
+ Set rb $rbrack
+ return
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_errmode ?newErrmode?
+#
+# INPUTS:
+# newErrmode If given, the new error mode.
+#
+# RETURNS:
+# The current error mode
+#
+# DESCRIPTION:
+# Returns the current error mode.
+
+proc ::textutil::expander::Op_errmode {name {newErrmode ""}} {
+ if {[string length $newErrmode] != 0} {
+ if {![Contains "macro nothing error fail" $newErrmode]} {
+ error "$name errmode: Invalid error mode: $newErrmode"
+ }
+
+ Set errmode $newErrmode
+ }
+ return [Get errmode]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_reset
+#
+# INPUTS:
+# none
+#
+# RETURNS:
+# nothing
+#
+# DESCRIPTION:
+# Resets all object values, as though it were brand new.
+
+proc ::textutil::expander::Op_reset {name} {
+ variable Info
+
+ if {[info exists Info($name-lb)]} {
+ array unset Info "$name-*"
+ }
+
+ set Info($name-lb) "\["
+ set Info($name-rb) "\]"
+ set Info($name-errmode) "fail"
+ set Info($name-level) 0
+ set Info($name-output-0) ""
+ set Info($name-name-0) ":0"
+
+ return
+}
+
+#-------------------------------------------------------------------------
+# Context: Every expansion takes place in its own context; however,
+# a macro can push a new context, causing the text it returns and all
+# subsequent text to be saved separately. Later, a matching macro can
+# pop the context, acquiring all text saved since the first command,
+# and use that in its own output.
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cpush cname
+#
+# INPUTS:
+# cname The context name
+#
+# RETURNS:
+# nothing
+#
+# DESCRIPTION:
+# Pushes an empty macro context onto the stack. All expanded text
+# will be added to this context until it is popped.
+
+proc ::textutil::expander::Op_cpush {name cname} {
+ incr [Var level]
+ set [Var output-[Get level]] {}
+ set [Var name-[Get level]] $cname
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cis cname
+#
+# INPUTS:
+# cname A context name
+#
+# RETURNS:
+# true or false
+#
+# DESCRIPTION:
+# Returns true if the current context has the specified name, and
+# false otherwise.
+
+proc ::textutil::expander::Op_cis {name cname} {
+ return [expr [string compare $cname [Op_cname $name]] == 0]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cname
+#
+# INPUTS:
+# none
+#
+# RETURNS:
+# The context name
+#
+# DESCRIPTION:
+# Returns the name of the current context.
+
+proc ::textutil::expander::Op_cname {name} {
+ return [Get name-[Get level]]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cset varname value
+#
+# INPUTS:
+# varname The name of a context variable
+# value The new value for the context variable
+#
+# RETURNS:
+# The value
+#
+# DESCRIPTION:
+# Sets a variable in the current context.
+
+proc ::textutil::expander::Op_cset {name varname value} {
+ Set data-[Get level]-$varname $value
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cget varname
+#
+# INPUTS:
+# varname The name of a context variable
+#
+# RETURNS:
+# The value
+#
+# DESCRIPTION:
+# Returns the value of a context variable. It's an error if
+# the variable doesn't exist.
+
+proc ::textutil::expander::Op_cget {name varname} {
+ if {![info exists [Var data-[Get level]-$varname]]} {
+ error "$name cget: $varname doesn't exist in this context"
+ }
+
+ return [Get data-[Get level]-$varname]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cvar varname
+#
+# INPUTS:
+# varname The name of a context variable
+#
+# RETURNS:
+# The index to the variable
+#
+# DESCRIPTION:
+# Returns the index to a context variable, for use with set,
+# lappend, etc.
+
+proc ::textutil::expander::Op_cvar {name varname} {
+ if {![info exists [Var data-[Get level]-$varname]]} {
+ error "$name cvar: $varname doesn't exist in this context"
+ }
+
+ return [Var data-[Get level]-$varname]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cpop cname
+#
+# INPUTS:
+# cname The expected context name.
+#
+# RETURNS:
+# The accumulated output in this context
+#
+# DESCRIPTION:
+# Returns the accumulated output for the current context, first
+# popping the context from the stack. The expected context name
+# must match the real name, or an error occurs.
+
+proc ::textutil::expander::Op_cpop {name cname} {
+ variable Info
+
+ if {[Get level] == 0} {
+ error "$name cpop underflow on '$cname'"
+ }
+
+ if {[string compare [Op_cname $name] $cname] != 0} {
+ error "$name cpop context mismatch: expected [Op_cname $name], got $cname"
+ }
+
+ set result [Get output-[Get level]]
+ set [Var output-[Get level]] ""
+ set [Var name-[Get level]] ""
+
+ array unset "Info data-[Get level]-*"
+
+ incr [Var level] -1
+
+ return $result
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_cappend text
+#
+# INPUTS:
+# text Text to add to the output
+#
+# RETURNS:
+# The accumulated output
+#
+# DESCRIPTION:
+# Appends the text to the accumulated output in the current context.
+
+proc ::textutil::expander::Op_cappend {name text} {
+ append [Var output-[Get level]] $text
+}
+
+#-------------------------------------------------------------------------
+# Macro-expansion: The following code is the heart of the module.
+# Given a text string, and the current variable settings, this code
+# returns an expanded string, with all macros replaced.
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# Op_expand inputString ?brackets?
+#
+# INPUTS:
+# inputString The text to expand.
+# brackets A list of two bracket tokens.
+#
+# RETURNS:
+# The expanded text.
+#
+# DESCRIPTION:
+# Finds all embedded macros in the input string, and expands them.
+# If ?brackets? is given, it must be list of length 2, containing
+# replacement left and right macro brackets; otherwise the default
+# brackets are used.
+
+proc ::textutil::expander::Op_expand {name inputString {brackets ""}} {
+ # FIRST, push a new context onto the stack, and save the current
+ # brackets.
+ Op_cpush $name expand
+ Op_cset $name lb [Get lb]
+ Op_cset $name rb [Get rb]
+
+ # NEXT, use the user's brackets, if given.
+ if {[llength $brackets] == 2} {
+ Set lb [lindex $brackets 0]
+ Set rb [lindex $brackets 1]
+ }
+
+ # NEXT, loop over the string, finding and expanding macros.
+ while {[string length $inputString] > 0} {
+ set plainText [ExtractToToken inputString [Get lb] exclude]
+
+ # FIRST, If there was plain text, append it to the output, and
+ # continue.
+ if {$plainText != ""} {
+ Op_cappend $name $plainText
+ if {[string length $inputString] == 0} {
+ break
+ }
+ }
+
+ # NEXT, A macro is the next thing; process it.
+ if {[catch "GetMacro inputString" macro]} {
+ error "Error reading macro: $macro"
+ }
+
+ # Expand the macro, and output the result, or
+ # handle an error.
+ if {![catch "uplevel #0 [list $macro]" result]} {
+ Op_cappend $name $result
+ continue
+ }
+
+ switch [Get errmode] {
+ nothing { }
+ macro {
+ Op_cappend $name "[Get lb]$macro[Get rb]"
+ }
+ error {
+ Op_cappend $name "\n=================================\n"
+ Op_cappend $name "*** Error in macro:\n"
+ Op_cappend $name "*** [Get lb]$macro[Get rb]\n--> $result\n"
+ Op_cappend $name "=================================\n"
+ }
+ fail {
+ error "Error in macro:\n[Get lb]$macro[Get rb]\n--> $result"
+ }
+ default {
+ error "Unknown error mode: [Get errmode]"
+ }
+ }
+ }
+
+ Op_lb $name [Op_cget $name lb]
+ Op_rb $name [Op_cget $name rb]
+
+ return [Op_cpop $name expand]
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# ExtractToToken string token mode
+#
+# INPUTS:
+# string The text to process.
+# token The token to look for
+# mode include or exclude
+#
+# RETURNS:
+# The extracted text
+#
+# DESCRIPTION:
+# Extract text from a string, up to or including a particular
+# token. Remove the extracted text from the string.
+# mode determines whether the found token is removed;
+# it should be "include" or "exclude". The string is
+# modified in place, and the extracted text is returned.
+
+proc ::textutil::expander::ExtractToToken {string token mode} {
+ upvar $string theString
+
+ # First, determine the offset
+ switch $mode {
+ include { set offset [expr [string length $token] - 1] }
+ exclude { set offset -1 }
+ default { error "::expander::ExtractToToken: unknown mode $mode" }
+ }
+
+ # Next, find the first occurrence of the token.
+ set tokenPos [string first $token $theString]
+
+ # Next, return the entire string if it wasn't found, or just
+ # the part upto or including the character.
+ if {$tokenPos == -1} {
+ set theText $theString
+ set theString ""
+ } else {
+ set newEnd [expr $tokenPos + $offset]
+ set newBegin [expr $newEnd + 1]
+ set theText [string range $theString 0 $newEnd]
+ set theString [string range $theString $newBegin end]
+ }
+
+ return $theText
+}
+
+#---------------------------------------------------------------------
+# FUNCTION:
+# GetMacro string
+#
+# INPUTS:
+# string The text to process.
+#
+# RETURNS:
+# The macro, stripped of its brackets.
+#
+# DESCRIPTION:
+
+proc ::textutil::expander::GetMacro {string} {
+ upvar $string theString
+
+ # FIRST, it's an error if the string doesn't begin with a
+ # bracket.
+ if {[string first [Get lb] $theString] != 0} {
+ error "::expander::GetMacro: assertion failure, next text isn't a command! '$theString'"
+ }
+
+ # NEXT, extract a full macro
+ set macro [ExtractToToken theString [Get lb] include]
+ while {[string length $theString] > 0} {
+ append macro [ExtractToToken theString [Get rb] include]
+
+ # Verify that the command really ends with the [rb] characters,
+ # whatever they are. If not, break because of unexpected
+ # end of file.
+ if {![IsBracketed $macro]} {
+ break;
+ }
+
+ set strippedMacro [StripBrackets $macro]
+
+ if {[info complete "puts \[$strippedMacro\]"]} {
+ return $strippedMacro
+ }
+ }
+
+ if {[string length $macro] > 40} {
+ set macro "[string range $macro 0 39]...\n"
+ }
+ error "Unexpected EOF in macro:\n$macro"
+}
+
+# Strip left and right bracket tokens from the ends of a macro,
+# provided that it's properly bracketed.
+proc ::textutil::expander::StripBrackets {macro} {
+ set llen [string length [Get lb]]
+ set rlen [string length [Get rb]]
+ set tlen [string length $macro]
+
+ return [string range $macro $llen [expr $tlen - $rlen - 1]]
+}
+
+# Return 1 if the macro is properly bracketed, and 0 otherwise.
+proc ::textutil::expander::IsBracketed {macro} {
+ set llen [string length [Get lb]]
+ set rlen [string length [Get rb]]
+ set tlen [string length $macro]
+
+ set leftEnd [string range $macro 0 [expr $llen - 1]]
+ set rightEnd [string range $macro [expr $tlen - $rlen] end]
+
+ if {$leftEnd != [Get lb]} {
+ return 0
+ } elseif {$rightEnd != [Get rb]} {
+ return 0
+ } else {
+ return 1
+ }
+}
+
+# Provide the package only if the code above was read and executed
+# without error.
+
+package provide textutil::expander 0.9
Index: modules/textutil/expander.test
===================================================================
RCS file: expander.test
diff -N expander.test
--- /dev/null Thu May 24 22:33:05 2001
+++ expander.test Fri Nov 9 09:06:48 2001
@@ -0,0 +1,321 @@
+# -*-Tcl-*-
+#---------------------------------------------------------------------
+# TITLE:
+# expander.test
+#
+# AUTHOR:
+# Will Duquette
+#
+# DESCRIPTION:
+# Test cases for expander.tcl. Uses the ::tcltest:: harness.
+
+
+#---------------------------------------------------------------------
+# Load the tcltest package
+
+if {[lsearch [namespace children] ::tcltest] == -1} {
+ package require tcltest
+ namespace import ::tcltest::*
+}
+
+#---------------------------------------------------------------------
+# Load the expander package
+
+if { [ lsearch [ namespace children ] "::textutil" ] == -1 } then {
+ source [file join [file dirname [info script]] expander.tcl]
+}
+
+#---------------------------------------------------------------------
+# Test cases 1.x: Expander Accessors
+
+test expander-1.1 {initial expander settings} {} {
+ catch {::textutil::expander exp}
+ set result "[exp lb] [exp rb] [exp errmode]"
+} {[ ] fail}
+
+test expander-1.2 {setting/retrieving lbrack} {} {
+ catch {::textutil::expander exp}
+ set result "[exp lb FOO] [exp lb] [exp lb {[}]"
+} {FOO FOO [}
+
+test expander-1.3 {setting/retrieving rbrack} {} {
+ catch {::textutil::expander exp}
+ set result "[exp rb FOO] [exp rb] [exp rb {]}]"
+} {FOO FOO ]}
+
+test expander-1.4 {setting/retrieving errmode fail} {} {
+ catch {::textutil::expander exp}
+ list [exp errmode fail] [exp errmode]
+} {fail fail}
+
+test expander-1.5 {setting/retrieving errmode nothing} {} {
+ catch {::textutil::expander exp}
+ list [exp errmode nothing] [exp errmode]
+} {nothing nothing}
+
+test expander-1.6 {setting/retrieving errmode macro} {} {
+ catch {::textutil::expander exp}
+ list [exp errmode macro] [exp errmode]
+} {macro macro}
+
+test expander-1.7 {setting/retrieving errmode error} {} {
+ catch {::textutil::expander exp}
+ list [exp errmode error] [exp errmode]
+} {error error}
+
+test expander-1.8 {setting/retrieving errmode incorrectly} {} {
+ catch {::textutil::expander exp}
+ exp errmode nothing
+ catch {exp errmode FOO} result
+ list $result [exp errmode]
+} {{::exp errmode: Invalid error mode: FOO} nothing}
+
+test expander-1.9 {resetting the object} {} {
+ catch {::textutil::expander exp}
+ exp errmode macro
+ exp lb FOO
+ exp rb BAR
+ exp reset
+ set result "[exp lb] [exp rb] [exp errmode]"
+} {[ ] fail}
+
+#---------------------------------------------------------------------
+# Test cases 2.x: The Context Stack
+
+test expander-2.1 {initial context stack settings} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ list [exp cname] [exp cis [exp cname]]
+} {:0 1}
+
+test expander-2.2 {context stack underflow} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ catch {exp cpop FOO} result
+ set result
+} {::exp cpop underflow on 'FOO'}
+
+test expander-2.3 {context push} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ list [exp cname] [exp cis [exp cname]]
+} {FOO 1}
+
+test expander-2.4 {cvar error} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ catch {exp cvar BAR} result
+ set result
+} {::exp cvar: BAR doesn't exist in this context}
+
+test expander-2.5 {cget error} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ catch {exp cget BAR} result
+ set result
+} {::exp cget: BAR doesn't exist in this context}
+
+test expander-2.6 {cpop mismatch} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ catch {exp cpop BAR} result
+ set result
+} {::exp cpop context mismatch: expected FOO, got BAR}
+
+test expander-2.7 {cpush, cappend, cpop} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ exp cappend "Hello, "
+ exp cappend "World!"
+ set result [exp cpop FOO]
+ list $result [exp cname]
+} {{Hello, World!} :0}
+
+test expander-2.8 {two-stage cpush, cappend, cpop} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ exp cappend "Goodbye "
+ exp cpush BAR
+ exp cappend "Cruel "
+ exp cappend [exp cpop BAR]
+ exp cappend "World!"
+ set result [exp cpop FOO]
+ list $result [exp cname]
+} {{Goodbye Cruel World!} :0}
+
+test expander-2.9 {cset, cvar, cget} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush FOO
+ exp cset BAR QUUX
+ list [exp cget BAR] [set [exp cvar BAR]]
+} {QUUX QUUX}
+
+test expander-2.10 {two-stage cset, cvar, cget} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp cpush ONE
+ exp cset FOO BAR
+ exp cpush TWO
+ exp cset FOO QUUX
+ set v2 [exp cget FOO]
+ exp cpop TWO
+ set v1 [exp cget FOO]
+ list $v1 $v2
+} {BAR QUUX}
+
+#---------------------------------------------------------------------
+# Test cases 3.x: Successful Macro Expansion
+
+proc howdy {} {return "Howdy"}
+
+test expander-3.1 {expand the empty string} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand ""
+} {}
+
+test expander-3.2 {expand a string with no macros} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {Hello, world!}
+} {Hello, world!}
+
+test expander-3.3 {expand a string consisting of a macro} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {[howdy]}
+} {Howdy}
+
+test expander-3.3 {expand a string beginning with a macro} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {[howdy], world!}
+} {Howdy, world!}
+
+test expander-3.4 {expand a string ending with a macro} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {Well, [howdy]}
+} {Well, Howdy}
+
+test expander-3.5 {expand a string with macro in middle} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {Well, [howdy]!}
+} {Well, Howdy!}
+
+test expander-3.6 {expand macro with changed default brackets} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp lb "<<<"
+ exp rb ">>>"
+ exp expand {Well, <<<howdy>>>!}
+} {Well, Howdy!}
+
+test expander-3.7 {expand macro with changed user brackets} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {Well, <<<howdy>>>!} {<<< >>>}
+} {Well, Howdy!}
+
+test expander-3.8 {expand macro with changed user brackets} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ set a [exp expand {[howdy]}]
+ set b [exp expand {Well, <<<howdy>>>!} {<<< >>>}]
+ list $a $b
+} {Howdy {Well, Howdy!}}
+
+test expander-3.9 {macros change brackets} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ string trim [exp expand {
+ Well, [howdy]!
+ [exp setbrackets <<< >>>]
+ Well, <<<howdy>>>!
+ <<<exp setbrackets "\[" "\]">>>
+ Well, [howdy]!
+ }]
+} {Well, Howdy!
+
+ Well, Howdy!
+
+ Well, Howdy!}
+
+test expander-3.10 {brackets are restored correctly} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ list [exp expand {<howdy>} "< >"] [exp expand {[howdy]}]
+} {Howdy Howdy}
+
+test expander-3.11 {nested expansion: one expander} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp expand {[howdy] [exp expand {*[howdy]*}] [howdy]}
+} {Howdy *Howdy* Howdy}
+
+test expander-3.12 {nested expansion: two expanders} {} {
+ catch {::textutil::expander exp}
+ catch {::textutil::expander exp2}
+ exp reset
+ exp2 reset
+ exp expand {[howdy] [exp2 expand {*[howdy]*}] [howdy]}
+} {Howdy *Howdy* Howdy}
+
+#---------------------------------------------------------------------
+# Test cases 4.x: Failed Macro Expansion
+
+test expander-4.1 {error mode fail} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp errmode fail
+ catch {exp expand {+++[nop]+++}} result
+ set result
+} {Error in macro:
+[nop]
+--> invalid command name "nop"}
+
+test expander-4.2 {error mode error} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp errmode error
+ catch {exp expand {+++[nop]+++}} result
+ set result
+} {+++
+=================================
+*** Error in macro:
+*** [nop]
+--> invalid command name "nop"
+=================================
++++}
+
+test expander-4.3 {error mode macro} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp errmode macro
+ catch {exp expand {+++[nop]+++}} result
+ set result
+} {+++[nop]+++}
+
+test expander-4.4 {error mode nothing} {} {
+ catch {::textutil::expander exp}
+ exp reset
+ exp errmode nothing
+ catch {exp expand {+++[nop]+++}} result
+ set result
+} {++++++}
+
+
+#---------------------------------------------------------------------
+# Clean up
+
+::tcltest::cleanupTests
+return
+
Index: modules/textutil/expander_license.txt
===================================================================
RCS file: expander_license.txt
diff -N expander_license.txt
--- /dev/null Thu May 24 22:33:05 2001
+++ expander_license.txt Fri Nov 9 09:06:48 2001
@@ -0,0 +1,38 @@
+This software is copyrighted by William H. Duquette. The following
+terms apply to all files associated with the software unless
+explicitly disclaimed in individual files.
+
+The authors hereby grant permission to use, copy, modify, distribute,
+and license this software and its documentation for any purpose, provided
+that existing copyright notices are retained in all copies and that this
+notice is included verbatim in any distributions. No written agreement,
+license, or royalty fee is required for any of the authorized uses.
+Modifications to this software may be copyrighted by their authors
+and need not follow the licensing terms described here, provided that
+the new terms are clearly indicated on the first page of each file where
+they apply.
+
+IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY
+FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES
+ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY
+DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES,
+INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. THIS SOFTWARE
+IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE
+NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR
+MODIFICATIONS.
+
+GOVERNMENT USE: If you are acquiring this software on behalf of the
+U.S. government, the Government shall have only "Restricted Rights"
+in the software and related documentation as defined in the Federal
+Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2). If you
+are acquiring the software on behalf of the Department of Defense, the
+software shall be classified as "Commercial Computer Software" and the
+Government shall have only "Restricted Rights" as defined in Clause
+252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the
+authors grant the U.S. Government and others acting in its behalf
+permission to use and distribute the software in accordance with the
+terms specified in this license.
Index: modules/textutil/expander_notes.txt
===================================================================
RCS file: expander_notes.txt
diff -N expander_notes.txt
--- /dev/null Thu May 24 22:33:05 2001
+++ expander_notes.txt Fri Nov 9 09:06:48 2001
@@ -0,0 +1,42 @@
+expander notes:
+o expander.tcl is an objectified version of the expansion algorithm
+ used in expand and Spinster. Goals
+ - Multiple expanders can be used at one time
+ - Handling of batch or incremental input
+ - Support for user-settable brackets
+ - Support for context stack.
+o Next: add and test incremental expansion.
+
+Things done: 11/3/2001
+x Added a couple of more tests for the nested expander problem I
+ fixed the other day.
+x Finished the man page for the current version.
+x Time to zip it up and send it off to Andreas.
+
+Things done: 10/31/2001
+x Updated the list of possible error modes to match the list in
+ Expand 2.1
+x Added tests for each of the error modes to expander.test.
+x Created a modified version of Expand 2.1 that uses expander; it
+ was able to process the Ex Libris website without error.
+x Found an error: if an expander is used to expand text which
+ contains a call to another expander, the two get confused--because
+ of the "::expander::This" variable.
+ - It works for the recordkeeper, because a recordkeeper method
+ can never execute a method for a different recordkeeper.
+ - What if Methods saved the old This, and restored it at the
+ end?
+ - Tried it; it works! All existing tests pass.
+
+Things done: 10/30/2001
+x Wrote more tests; found and fixed bracket restoration bug in expander.
+
+Things done: 10/25/2001
+x Wrote tests for the lb, rb, and errmode commands.
+x Added the reset command.
+x Wrote tests for the reset command.
+x Added the context stack commands, and wrote tests for them.
+
+Things done: 10/24/2001
+x Copied recordkeeper code and docs as a template.
+x Implemented the lb, rb, and errmode accessor methods.
Index: modules/textutil/pkgIndex.tcl
===================================================================
RCS file: /cvsroot/tcllib/tcllib/modules/textutil/pkgIndex.tcl,v
retrieving revision 1.4
diff -u -r1.4 pkgIndex.tcl
--- modules/textutil/pkgIndex.tcl 2001/10/17 17:27:26 1.4
+++ modules/textutil/pkgIndex.tcl 2001/11/09 17:06:48
@@ -9,4 +9,5 @@
# full path name of this file's directory.
if {![package vsatisfies [package provide Tcl] 8.2]} {return}
-package ifneeded textutil 0.3 [list source [file join $dir textutil.tcl]]
+package ifneeded textutil 0.3 [list source [file join $dir textutil.tcl]]
+package ifneeded textutil::expander 0.9 [list source [file join $dir expander.tcl]]
