1 2:mod:`HTMLParser` --- Simple HTML and XHTML parser 3================================================== 4 5.. module:: HTMLParser 6 :synopsis: A simple parser that can handle HTML and XHTML. 7 8.. note:: 9 10 The :mod:`HTMLParser` module has been renamed to :mod:`html.parser` in Python 11 3. The :term:`2to3` tool will automatically adapt imports when converting 12 your sources to Python 3. 13 14 15.. versionadded:: 2.2 16 17.. index:: 18 single: HTML 19 single: XHTML 20 21**Source code:** :source:`Lib/HTMLParser.py` 22 23-------------- 24 25This module defines a class :class:`.HTMLParser` which serves as the basis for 26parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. 27Unlike the parser in :mod:`htmllib`, this parser is not based on the SGML parser 28in :mod:`sgmllib`. 29 30 31.. class:: HTMLParser() 32 33 An :class:`.HTMLParser` instance is fed HTML data and calls handler methods 34 when start tags, end tags, text, comments, and other markup elements are 35 encountered. The user should subclass :class:`.HTMLParser` and override its 36 methods to implement the desired behavior. 37 38 The :class:`.HTMLParser` class is instantiated without arguments. 39 40 Unlike the parser in :mod:`htmllib`, this parser does not check that end tags 41 match start tags or call the end-tag handler for elements which are closed 42 implicitly by closing an outer element. 43 44An exception is defined as well: 45 46.. exception:: HTMLParseError 47 48 :class:`.HTMLParser` is able to handle broken markup, but in some cases it 49 might raise this exception when it encounters an error while parsing. 50 This exception provides three attributes: :attr:`msg` is a brief 51 message explaining the error, :attr:`lineno` is the number of the line on 52 which the broken construct was detected, and :attr:`offset` is the number of 53 characters into the line at which the construct starts. 54 55 56Example HTML Parser Application 57------------------------------- 58 59As a basic example, below is a simple HTML parser that uses the 60:class:`.HTMLParser` class to print out start tags, end tags and data 61as they are encountered:: 62 63 from HTMLParser import HTMLParser 64 65 # create a subclass and override the handler methods 66 class MyHTMLParser(HTMLParser): 67 def handle_starttag(self, tag, attrs): 68 print "Encountered a start tag:", tag 69 70 def handle_endtag(self, tag): 71 print "Encountered an end tag :", tag 72 73 def handle_data(self, data): 74 print "Encountered some data :", data 75 76 # instantiate the parser and fed it some HTML 77 parser = MyHTMLParser() 78 parser.feed('<html><head><title>Test</title></head>' 79 '<body><h1>Parse me!</h1></body></html>') 80 81The output will then be: 82 83.. code-block:: none 84 85 Encountered a start tag: html 86 Encountered a start tag: head 87 Encountered a start tag: title 88 Encountered some data : Test 89 Encountered an end tag : title 90 Encountered an end tag : head 91 Encountered a start tag: body 92 Encountered a start tag: h1 93 Encountered some data : Parse me! 94 Encountered an end tag : h1 95 Encountered an end tag : body 96 Encountered an end tag : html 97 98 99:class:`.HTMLParser` Methods 100---------------------------- 101 102:class:`.HTMLParser` instances have the following methods: 103 104 105.. method:: HTMLParser.feed(data) 106 107 Feed some text to the parser. It is processed insofar as it consists of 108 complete elements; incomplete data is buffered until more data is fed or 109 :meth:`close` is called. *data* can be either :class:`unicode` or 110 :class:`str`, but passing :class:`unicode` is advised. 111 112 113.. method:: HTMLParser.close() 114 115 Force processing of all buffered data as if it were followed by an end-of-file 116 mark. This method may be redefined by a derived class to define additional 117 processing at the end of the input, but the redefined version should always call 118 the :class:`.HTMLParser` base class method :meth:`close`. 119 120 121.. method:: HTMLParser.reset() 122 123 Reset the instance. Loses all unprocessed data. This is called implicitly at 124 instantiation time. 125 126 127.. method:: HTMLParser.getpos() 128 129 Return current line number and offset. 130 131 132.. method:: HTMLParser.get_starttag_text() 133 134 Return the text of the most recently opened start tag. This should not normally 135 be needed for structured processing, but may be useful in dealing with HTML "as 136 deployed" or for re-generating input with minimal changes (whitespace between 137 attributes can be preserved, etc.). 138 139 140The following methods are called when data or markup elements are encountered 141and they are meant to be overridden in a subclass. The base class 142implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): 143 144 145.. method:: HTMLParser.handle_starttag(tag, attrs) 146 147 This method is called to handle the start of a tag (e.g. ``<div id="main">``). 148 149 The *tag* argument is the name of the tag converted to lower case. The *attrs* 150 argument is a list of ``(name, value)`` pairs containing the attributes found 151 inside the tag's ``<>`` brackets. The *name* will be translated to lower case, 152 and quotes in the *value* have been removed, and character and entity references 153 have been replaced. 154 155 For instance, for the tag ``<A HREF="https://www.cwi.nl/">``, this method 156 would be called as ``handle_starttag('a', [('href', 'https://www.cwi.nl/')])``. 157 158 .. versionchanged:: 2.6 159 All entity references from :mod:`htmlentitydefs` are now replaced in the 160 attribute values. 161 162 163.. method:: HTMLParser.handle_endtag(tag) 164 165 This method is called to handle the end tag of an element (e.g. ``</div>``). 166 167 The *tag* argument is the name of the tag converted to lower case. 168 169 170.. method:: HTMLParser.handle_startendtag(tag, attrs) 171 172 Similar to :meth:`handle_starttag`, but called when the parser encounters an 173 XHTML-style empty tag (``<img ... />``). This method may be overridden by 174 subclasses which require this particular lexical information; the default 175 implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`. 176 177 178.. method:: HTMLParser.handle_data(data) 179 180 This method is called to process arbitrary data (e.g. text nodes and the 181 content of ``<script>...</script>`` and ``<style>...</style>``). 182 183 184.. method:: HTMLParser.handle_entityref(name) 185 186 This method is called to process a named character reference of the form 187 ``&name;`` (e.g. ``>``), where *name* is a general entity reference 188 (e.g. ``'gt'``). 189 190 191.. method:: HTMLParser.handle_charref(name) 192 193 This method is called to process decimal and hexadecimal numeric character 194 references of the form ``&#NNN;`` and ``&#xNNN;``. For example, the decimal 195 equivalent for ``>`` is ``>``, whereas the hexadecimal is ``>``; 196 in this case the method will receive ``'62'`` or ``'x3E'``. 197 198 199.. method:: HTMLParser.handle_comment(data) 200 201 This method is called when a comment is encountered (e.g. ``<!--comment-->``). 202 203 For example, the comment ``<!-- comment -->`` will cause this method to be 204 called with the argument ``' comment '``. 205 206 The content of Internet Explorer conditional comments (condcoms) will also be 207 sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``, 208 this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``. 209 210 211.. method:: HTMLParser.handle_decl(decl) 212 213 This method is called to handle an HTML doctype declaration (e.g. 214 ``<!DOCTYPE html>``). 215 216 The *decl* parameter will be the entire contents of the declaration inside 217 the ``<!...>`` markup (e.g. ``'DOCTYPE html'``). 218 219 220.. method:: HTMLParser.handle_pi(data) 221 222 This method is called when a processing instruction is encountered. The *data* 223 parameter will contain the entire processing instruction. For example, for the 224 processing instruction ``<?proc color='red'>``, this method would be called as 225 ``handle_pi("proc color='red'")``. 226 227 .. note:: 228 229 The :class:`.HTMLParser` class uses the SGML syntactic rules for processing 230 instructions. An XHTML processing instruction using the trailing ``'?'`` will 231 cause the ``'?'`` to be included in *data*. 232 233 234.. method:: HTMLParser.unknown_decl(data) 235 236 This method is called when an unrecognized declaration is read by the parser. 237 238 The *data* parameter will be the entire contents of the declaration inside 239 the ``<![...]>`` markup. It is sometimes useful to be overridden by a 240 derived class. 241 242 243.. _htmlparser-examples: 244 245Examples 246-------- 247 248The following class implements a parser that will be used to illustrate more 249examples:: 250 251 from HTMLParser import HTMLParser 252 from htmlentitydefs import name2codepoint 253 254 class MyHTMLParser(HTMLParser): 255 def handle_starttag(self, tag, attrs): 256 print "Start tag:", tag 257 for attr in attrs: 258 print " attr:", attr 259 260 def handle_endtag(self, tag): 261 print "End tag :", tag 262 263 def handle_data(self, data): 264 print "Data :", data 265 266 def handle_comment(self, data): 267 print "Comment :", data 268 269 def handle_entityref(self, name): 270 c = unichr(name2codepoint[name]) 271 print "Named ent:", c 272 273 def handle_charref(self, name): 274 if name.startswith('x'): 275 c = unichr(int(name[1:], 16)) 276 else: 277 c = unichr(int(name)) 278 print "Num ent :", c 279 280 def handle_decl(self, data): 281 print "Decl :", data 282 283 parser = MyHTMLParser() 284 285Parsing a doctype:: 286 287 >>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" ' 288 ... '"http://www.w3.org/TR/html4/strict.dtd">') 289 Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd" 290 291Parsing an element with a few attributes and a title:: 292 293 >>> parser.feed('<img src="python-logo.png" alt="The Python logo">') 294 Start tag: img 295 attr: ('src', 'python-logo.png') 296 attr: ('alt', 'The Python logo') 297 >>> 298 >>> parser.feed('<h1>Python</h1>') 299 Start tag: h1 300 Data : Python 301 End tag : h1 302 303The content of ``script`` and ``style`` elements is returned as is, without 304further parsing:: 305 306 >>> parser.feed('<style type="text/css">#python { color: green }</style>') 307 Start tag: style 308 attr: ('type', 'text/css') 309 Data : #python { color: green } 310 End tag : style 311 312 >>> parser.feed('<script type="text/javascript">' 313 ... 'alert("<strong>hello!</strong>");</script>') 314 Start tag: script 315 attr: ('type', 'text/javascript') 316 Data : alert("<strong>hello!</strong>"); 317 End tag : script 318 319Parsing comments:: 320 321 >>> parser.feed('<!-- a comment -->' 322 ... '<!--[if IE 9]>IE-specific content<![endif]-->') 323 Comment : a comment 324 Comment : [if IE 9]>IE-specific content<![endif] 325 326Parsing named and numeric character references and converting them to the 327correct char (note: these 3 references are all equivalent to ``'>'``):: 328 329 >>> parser.feed('>>>') 330 Named ent: > 331 Num ent : > 332 Num ent : > 333 334Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but 335:meth:`~HTMLParser.handle_data` might be called more than once:: 336 337 >>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']: 338 ... parser.feed(chunk) 339 ... 340 Start tag: span 341 Data : buff 342 Data : ered 343 Data : text 344 End tag : span 345 346Parsing invalid HTML (e.g. unquoted attributes) also works:: 347 348 >>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>') 349 Start tag: p 350 Start tag: a 351 attr: ('class', 'link') 352 attr: ('href', '#main') 353 Data : tag soup 354 End tag : p 355 End tag : a 356
