Tcl Library Source Code

View Ticket
Login
Ticket UUID: 3091244
Title: parsing doesn't respect pre tag & trims trailing whitespaces
Type: Bug Version: None
Submitter: martinlemburg Created on: 2010-10-20 12:09:33
Subsystem: htmlparse Assigned To: andreas_kupries
Priority: 5 Medium Severity:
Status: Open Last Modified: 2010-10-27 15:15:28
Resolution: None Closed By:
    Closed on:
Description:
tcllib 1.11.1 - htmlparse 1.1.3:
========================

Parsing of preformatted text in HTML data enclosed by the HTML tag "pre" leads to the loss of trailing whitespaces.

The procedure 2treeCallback must be fixed for example like this:

    set  tag  [string tolower $tag]
    if {$tag ne "pre"} {
        set  textBehindTheTag [string trim $textBehindTheTag]
    }

Perhaps there are still other tags to think over a specialized handling.
User Comments: martinlemburg added on 2010-10-27 15:15:28:
Thanks to Rolf Ade going into details.

I would prefer that the htmlparse package would preserve all whitespaces, not modifying the content to return really only the logical structure of the given document!

I as user of HTML data have to care for this, knowing that some tags have a special whitespace behavior, knowing that CSS rules modify the whitespace behavior, but NOT the logical structure of the document.

So my proposed change would be to remove all whitespace handling (replacing several whitespaces by a space and trimming trailing whitespaces) and to document this in the man page, to make sure, that the package user knows, that the contents are really not changed!

Regards,

Martin

pointsman added on 2010-10-27 05:55:31:
Well, I also stumbled over 2treeCallback some days ago. And thought: that whole proc isn't an example for good code.

That starts with unusual coding style:

    if {"$slash" == "/"} {

The quotes around $slash aren't wrong in the sense of being a bug, sure. But the experienced tcl programmer get disturbed for an eye blink and unexperienced tcl programmers may get confused and may assume some special magic.

And goes on with superfluous code, within the first non-comment lines of the proc:

    if {$textBehindTheTag != {}} {
        set text [mapEscapes $textBehindTheTag]
    }

Too sad, $text isn't used anywhere later on in the proc.

This all is harmless for the question "is the result of the proc
correct", sure.

The code cited by the originial poster is not, but not only in the case of the PRE element.

An example is pretty easy (here done with an other element, to show that we talk about a more general problem):


package require struct::tree
package require htmlparse

struct::tree resultTree1
htmlparse::2tree "<p>foo <b>bar</b> baz</p>" resultTree1

struct::tree resultTree2
htmlparse::2tree "<p>foo<b>bar</b>baz</p>" resultTree2

if {[resultTree1 serialize] eq [resultTree2 serialize]} {
    puts "Identical HTML fragments"
} else {
    puts "HTML fragments differ."
}

This script returns "Identical HTML fragments", which doesn't look quit right. 

Well, sure, the proc does what it does. In the words of the manual: "[It] takes an HTML string and converts it into a tree containing the logical structure of the parsed document." This is a vague description of what should happen (or, otherwise around, what one should expect to get out, if used).

But as it is now, the user get neither the whole unmodified PCDATA content of the html input string nor the PCDATA with white space condensed in the ususal way by an HTML browser (css white-space value "normal") nor the PCDATA with white space handled in any other of the ways the css attribute white-space knows ("pre", "nowrap").

That result my be meant by "logical structure of the parsed document" but it at least makes 2tree useless for everyone which want to parse an html document to render it in so way (to pdf, rtf, ...).

At a minium a word about the strange white space handling of the current proc within the documentation is necessary.

Adding a special case for the PRE element into this weird white space handling - as the originial poster proposes - may help for a small range of real word problems but in general makes things even worser, in my opinion.

The easiest would be, to remove the whole line: no white space stripping at all of the PCDATA contents. Sure, this burden the consumer of the result tree with the white space handling, but at least it then has a chance to get it right, while it now hasn't any chance to get it right because it gets it already mangled. If nobody will be do more, I would for sure at least do that: remove the line.

A more elaborated white space handling would for sure be convenient in a lot of cases. If someone writes one, it would still be a good feature, to be able to get the whole unmodified PCDATA, if needed (by an option).

A natural candidate for how an elaborated white space handling should work is the css white-space attribute value "normal", that means, all white spaces (spaces, tabs, line breaks) condensed into a single space.

Beside all that, the special handling of some auto-closing HTML tags does just work by accident, if ever.

andreas_kupries added on 2010-10-27 00:58:11:
Do you have a mini example for me ?