第 8 章 HTML 处理
Description: Python from novice to pro
Keywords: Python, Dive Into Python, tutorial, object-oriented, programming, documentation, book, free

# 深入 Python :Dive Into Python 中文版 Python 从新手到专家 [Dip_5.4b_CPyUG_Release] | Find:

## 第 8 章 HTML 处理

## 8.1. 概览

我经常在 comp.lang.python 上看到关于如下的问题： “ 怎么才能从我的 HTML 文档中列出所有的 [头|图像|链接] 呢？” “怎么才能 [分析|解释|munge] 我的 HTML 文档的文本，但是又要保留标记呢？” “怎么才能一次给我所有的 HTML 标记 [增加|删除|加引号] 属性呢？” 本章将回答所有这些问题。

下面给出一个完整的，可工作的 Python 程序，它分为两部分。第一部分，`BaseHTMLProcessor.py` 是一个通用工具，它可以通过扫描标记和文本块来帮助您处理 HTML 文件。第二部分，`dialect.py` 是一个例子，演示了如何使用 `BaseHTMLProcessor.py` 来转化 HTML 文档，保留文本但是去掉了标记。阅读文档字符串 (`doc string`) 和注释来了解将要发生事情的概况。大部分内容看上去像巫术，因为任一个这些类的方法是如何调用的不是很清楚。不要紧，所有内容都会按进度被逐步地展示出来。

### 例 8.1. `BaseHTMLProcessor.py`

如果您还没有下载本书附带的样例程序, 可以 下载本程序和其他样例程序。

    from sgmllib import SGMLParser
    import htmlentitydefs

    class BaseHTMLProcessor(SGMLParser):
        def reset(self):
            # extend (called by SGMLParser.__init__)
            self.pieces = []
            SGMLParser.reset(self)

        def unknown_starttag(self, tag, attrs):
            # called for each start tag
            # attrs is a list of (attr, value) tuples
            # e.g. for <pre class="screen">, tag="pre", attrs=[("class", "screen")]
            # Ideally we would like to reconstruct original tag and attributes, but
            # we may end up quoting attribute values that weren't quoted in the source
            # document, or we may change the type of quotes around the attribute value
            # (single to double quotes).
            # Note that improperly embedded non-HTML code (like client-side Javascript)
            # may be parsed incorrectly by the ancestor, causing runtime script errors.
            # All non-HTML code must be enclosed in HTML comment tags (<!-- code -->)
            # to ensure that it will pass through this parser unaltered (in handle_comment).
            strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
            self.pieces.append("<%(tag)s%(strattrs)s>" % locals())

        def unknown_endtag(self, tag):
            # called for each end tag, e.g. for </pre>, tag will be "pre"
            # Reconstruct the original end tag.
            self.pieces.append("</%(tag)s>" % locals())

        def handle_charref(self, ref):
            # called for each character reference, e.g. for "&#160;", ref will be "160"
            # Reconstruct the original character reference.
            self.pieces.append("&#%(ref)s;" % locals())

        def handle_entityref(self, ref):
            # called for each entity reference, e.g. for "&copy;", ref will be "copy"
            # Reconstruct the original entity reference.
            self.pieces.append("&%(ref)s" % locals())
            # standard HTML entities are closed with a semicolon; other entities are not
            if htmlentitydefs.entitydefs.has_key(ref):
                self.pieces.append(";")

        def handle_data(self, text):
            # called for each block of plain text, i.e. outside of any tag and
            # not containing any character or entity references
            # Store the original text verbatim.
            self.pieces.append(text)

        def handle_comment(self, text):
            # called for each HTML comment, e.g. <!-- insert Javascript code here -->
            # Reconstruct the original comment.
            # It is especially important that the source document enclose client-side
            # code (like Javascript) within comments so it can pass through this
            # processor undisturbed; see comments in unknown_starttag for details.
            self.pieces.append("<!--%(text)s-->" % locals())

        def handle_pi(self, text):
            # called for each processing instruction, e.g. <?instruction>
            # Reconstruct original processing instruction.
            self.pieces.append("<?%(text)s>" % locals())

        def handle_decl(self, text):
            # called for the DOCTYPE, if present, e.g.
            # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
            #     "http://www.w3.org/TR/html4/loose.dtd">
            # Reconstruct original DOCTYPE
            self.pieces.append("<!%(text)s>" % locals())

        def output(self):
            """Return processed HTML as a single string"""
            return "".join(self.pieces)

### 例 8.2. `dialect.py`

    import re
    from BaseHTMLProcessor import BaseHTMLProcessor

    class Dialectizer(BaseHTMLProcessor):
        subs = ()

        def reset(self):
            # extend (called from __init__ in ancestor)
            # Reset all data attributes
            self.verbatim = 0
            BaseHTMLProcessor.reset(self)

        def start_pre(self, attrs):
            # called for every <pre> tag in HTML source
            # Increment verbatim mode count, then handle tag like normal
            self.verbatim += 1
            self.unknown_starttag("pre", attrs)

        def end_pre(self):
            # called for every </pre> tag in HTML source
            # Decrement verbatim mode count
            self.unknown_endtag("pre")
            self.verbatim -= 1

        def handle_data(self, text):
            # override
            # called for every block of text in HTML source
            # If in verbatim mode, save text unaltered;
            # otherwise process the text with a series of substitutions
            self.pieces.append(self.verbatim and text or self.process(text))

        def process(self, text):
            # called from handle_data
            # Process text block by performing series of regular expression
            # substitutions (actual substitions are defined in descendant)
            for fromPattern, toPattern in self.subs:
                text = re.sub(fromPattern, toPattern, text)
            return text

    class ChefDialectizer(Dialectizer):
        """convert HTML to Swedish Chef-speak

        based on the classic chef.x, copyright (c) 1992, 1993 John Hagerman
        """
        subs = ((r'a([nu])', r'u\1'),
                (r'A([nu])', r'U\1'),
                (r'a\B', r'e'),
                (r'A\B', r'E'),
                (r'en\b', r'ee'),
                (r'\Bew', r'oo'),
                (r'\Be\b', r'e-a'),
                (r'\be', r'i'),
                (r'\bE', r'I'),
                (r'\Bf', r'ff'),
                (r'\Bir', r'ur'),
                (r'(\w*?)i(\w*?)$', r'\1ee\2'),
                (r'\bow', r'oo'),
                (r'\bo', r'oo'),
                (r'\bO', r'Oo'),
                (r'the', r'zee'),
                (r'The', r'Zee'),
                (r'th\b', r't'),
                (r'\Btion', r'shun'),
                (r'\Bu', r'oo'),
                (r'\BU', r'Oo'),
                (r'v', r'f'),
                (r'V', r'F'),
                (r'w', r'w'),
                (r'W', r'W'),
                (r'([a-z])[.]', r'\1.  Bork Bork Bork!'))

    class FuddDialectizer(Dialectizer):
        """convert HTML to Elmer Fudd-speak"""
        subs = ((r'[rl]', r'w'),
                (r'qu', r'qw'),
                (r'th\b', r'f'),
                (r'th', r'd'),
                (r'n[.]', r'n, uh-hah-hah-hah.'))

    class OldeDialectizer(Dialectizer):
        """convert HTML to mock Middle English"""
        subs = ((r'i([bcdfghjklmnpqrstvwxyz])e\b', r'y\1'),
                (r'i([bcdfghjklmnpqrstvwxyz])e', r'y\1\1e'),
                (r'ick\b', r'yk'),
                (r'ia([bcdfghjklmnpqrstvwxyz])', r'e\1e'),
                (r'e[ea]([bcdfghjklmnpqrstvwxyz])', r'e\1e'),
                (r'([bcdfghjklmnpqrstvwxyz])y', r'\1ee'),
                (r'([bcdfghjklmnpqrstvwxyz])er', r'\1re'),
                (r'([aeiou])re\b', r'\1r'),
                (r'ia([bcdfghjklmnpqrstvwxyz])', r'i\1e'),
                (r'tion\b', r'cioun'),
                (r'ion\b', r'ioun'),
                (r'aid', r'ayde'),
                (r'ai', r'ey'),
                (r'ay\b', r'y'),
                (r'ay', r'ey'),
                (r'ant', r'aunt'),
                (r'ea', r'ee'),
                (r'oa', r'oo'),
                (r'ue', r'e'),
                (r'oe', r'o'),
                (r'ou', r'ow'),
                (r'ow', r'ou'),
                (r'\bhe', r'hi'),
                (r've\b', r'veth'),
                (r'se\b', r'e'),
                (r"'s\b", r'es'),
                (r'ic\b', r'ick'),
                (r'ics\b', r'icc'),
                (r'ical\b', r'ick'),
                (r'tle\b', r'til'),
                (r'll\b', r'l'),
                (r'ould\b', r'olde'),
                (r'own\b', r'oune'),
                (r'un\b', r'onne'),
                (r'rry\b', r'rye'),
                (r'est\b', r'este'),
                (r'pt\b', r'pte'),
                (r'th\b', r'the'),
                (r'ch\b', r'che'),
                (r'ss\b', r'sse'),
                (r'([wybdp])\b', r'\1e'),
                (r'([rnt])\b', r'\1\1e'),
                (r'from', r'fro'),
                (r'when', r'whan'))

    def translate(url, dialectName="chef"):
        """fetch URL and translate using dialect

        dialect in ("chef", "fudd", "olde")"""
        import urllib
        sock = urllib.urlopen(url)
        htmlSource = sock.read()
        sock.close()
        parserName = "%sDialectizer" % dialectName.capitalize()
        parserClass = globals()[parserName]
        parser = parserClass()
        parser.feed(htmlSource)
        parser.close()
        return parser.output()

    def test(url):
        """test all dialects against URL"""
        for dialect in ("chef", "fudd", "olde"):
            outfile = "%s.html" % dialect
            fsock = open(outfile, "wb")
            fsock.write(translate(url, dialect))
            fsock.close()
            import webbrowser
            webbrowser.open_new(outfile)

    if __name__ == "__main__":
        test("http://diveintopython.org/odbchelper_list.html")

### 例 8.3. `dialect.py` 的输出结果

运行这个脚本会将 第 3.2 节 “List 介绍” 转换成模仿瑞典厨师用语 (mock Swedish Chef-speak) (来自 The Muppets)、模仿埃尔默唠叨者用语 (mock Elmer Fudd-speak) (来自 Bugs Bunny 卡通画) 和模仿中世纪英语 (mock Middle English) (零散地来源于乔叟的《坎特伯雷故事集》)。如果您查看输出页面的 HTML 源代码，您会发现所有的 HTML 标记和属性没有改动，但是在标记之间的文本被转换成模仿语言了。如果您观查得更仔细些，您会发现，实际上，仅有标题和段落被转换了；代码列表和屏幕例子没有改动。

    <div class="abstract">
    <p>Lists awe <span class="application">Pydon</span>'s wowkhowse datatype.
    If youw onwy expewience wif wists is awways in
    <span class="application">Visuaw Basic</span> ow (God fowbid) de datastowe
    in <span class="application">Powewbuiwdew</span>, bwace youwsewf fow
    <span class="application">Pydon</span> wists.</p>
    </div>