1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 2 "http://www.w3.org/TR/html4/loose.dtd"> 3<html> 4<head> 5 <meta http-equiv="Content-Type" content="text/html"> 6 <style type="text/css"></style> 7<!-- 8TD {font-family: Verdana,Arial,Helvetica} 9BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em} 10H1 {font-family: Verdana,Arial,Helvetica} 11H2 {font-family: Verdana,Arial,Helvetica} 12H3 {font-family: Verdana,Arial,Helvetica} 13A:link, A:visited, A:active { text-decoration: underline } 14 </style> 15--> 16 <title>Libxml2 XmlTextReader Interface tutorial</title> 17</head> 18 19<body bgcolor="#fffacd" text="#000000"> 20<h1 align="center">Libxml2 XmlTextReader Interface tutorial</h1> 21 22<p></p> 23 24<p>This document describes the use of the XmlTextReader streaming API added 25to libxml2 in version 2.5.0 . This API is closely modeled after the <a 26href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader</a> 27and <a 28href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlReader.html">XmlReader</a> 29classes of the C# language.</p> 30 31<p>This tutorial will present the key points of this API, and working 32examples using both C and the Python bindings:</p> 33 34<p>Table of content:</p> 35<ul> 36 <li><a href="#Introducti">Introduction: why a new API</a></li> 37 <li><a href="#Walking">Walking a simple tree</a></li> 38 <li><a href="#Extracting">Extracting informations for the current 39 node</a></li> 40 <li><a href="#Extracting1">Extracting informations for the 41 attributes</a></li> 42 <li><a href="#Validating">Validating a document</a></li> 43 <li><a href="#Entities">Entities substitution</a></li> 44 <li><a href="#L1142">Relax-NG Validation</a></li> 45 <li><a href="#Mixing">Mixing the reader and tree or XPath 46 operations</a></li> 47</ul> 48 49<p></p> 50 51<h2><a name="Introducti">Introduction: why a new API</a></h2> 52 53<p>Libxml2 <a href="http://xmlsoft.org/html/libxml-tree.html">main API is 54tree based</a>, where the parsing operation results in a document loaded 55completely in memory, and expose it as a tree of nodes all availble at the 56same time. This is very simple and quite powerful, but has the major 57limitation that the size of the document that can be hamdled is limited by 58the size of the memory available. Libxml2 also provide a <a 59href="http://www.saxproject.org/">SAX</a> based API, but that version was 60designed upon one of the early <a 61href="http://www.jclark.com/xml/expat.html">expat</a> version of SAX, SAX is 62also not formally defined for C. SAX basically work by registering callbacks 63which are called directly by the parser as it progresses through the document 64streams. The problem is that this programming model is relatively complex, 65not well standardized, cannot provide validation directly, makes entity, 66namespace and base processing relatively hard.</p> 67 68<p>The <a 69href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader 70API from C#</a> provides a far simpler programming model. The API acts as a 71cursor going forward on the document stream and stopping at each node in the 72way. The user's code keeps control of the progress and simply calls a 73Read() function repeatedly to progress to each node in sequence in document 74order. There is direct support for namespaces, xml:base, entity handling and 75adding DTD validation on top of it was relatively simple. This API is really 76close to the <a href="http://www.w3.org/TR/DOM-Level-2-Core/">DOM Core 77specification</a> This provides a far more standard, easy to use and powerful 78API than the existing SAX. Moreover integrating extension features based on 79the tree seems relatively easy.</p> 80 81<p>In a nutshell the XmlTextReader API provides a simpler, more standard and 82more extensible interface to handle large documents than the existing SAX 83version.</p> 84 85<h2><a name="Walking">Walking a simple tree</a></h2> 86 87<p>Basically the XmlTextReader API is a forward only tree walking interface. 88The basic steps are:</p> 89<ol> 90 <li>prepare a reader context operating on some input</li> 91 <li>run a loop iterating over all nodes in the document</li> 92 <li>free up the reader context</li> 93</ol> 94 95<p>Here is a basic C sample doing this:</p> 96<pre>#include <libxml/xmlreader.h> 97 98void processNode(xmlTextReaderPtr reader) { 99 /* handling of a node in the tree */ 100} 101 102int streamFile(char *filename) { 103 xmlTextReaderPtr reader; 104 int ret; 105 106 reader = xmlNewTextReaderFilename(filename); 107 if (reader != NULL) { 108 ret = xmlTextReaderRead(reader); 109 while (ret == 1) { 110 processNode(reader); 111 ret = xmlTextReaderRead(reader); 112 } 113 xmlFreeTextReader(reader); 114 if (ret != 0) { 115 printf("%s : failed to parse\n", filename); 116 } 117 } else { 118 printf("Unable to open %s\n", filename); 119 } 120}</pre> 121 122<p>A few things to notice:</p> 123<ul> 124 <li>the include file needed : <code>libxml/xmlreader.h</code></li> 125 <li>the creation of the reader using a filename</li> 126 <li>the repeated call to xmlTextReaderRead() and how any return value 127 different from 1 should stop the loop</li> 128 <li>that a negative return means a parsing error</li> 129 <li>how xmlFreeTextReader() should be used to free up the resources used by 130 the reader.</li> 131</ul> 132 133<p>Here is similar code in python for exactly the same processing:</p> 134<pre>import libxml2 135 136def processNode(reader): 137 pass 138 139def streamFile(filename): 140 try: 141 reader = libxml2.newTextReaderFilename(filename) 142 except: 143 print "unable to open %s" % (filename) 144 return 145 146 ret = reader.Read() 147 while ret == 1: 148 processNode(reader) 149 ret = reader.Read() 150 151 if ret != 0: 152 print "%s : failed to parse" % (filename)</pre> 153 154<p>The only things worth adding are that the <a 155href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">xmlTextReader 156is abstracted as a class like in C#</a> with the same method names (but the 157properties are currently accessed with methods) and that one doesn't need to 158free the reader at the end of the processing. It will get garbage collected 159once all references have disapeared.</p> 160 161<h2><a name="Extracting">Extracting information for the current node</a></h2> 162 163<p>So far the example code did not indicate how information was extracted 164from the reader. It was abstrated as a call to the processNode() routine, 165with the reader as the argument. At each invocation, the parser is stopped on 166a given node and the reader can be used to query those node properties. Each 167<em>Property</em> is available at the C level as a function taking a single 168xmlTextReaderPtr argument whose name is 169<code>xmlTextReader</code><em>Property</em> , if the return type is an 170<code>xmlChar *</code> string then it must be deallocated with 171<code>xmlFree()</code> to avoid leaks. For the Python interface, there is a 172<em>Property</em> method to the reader class that can be called on the 173instance. The list of the properties is based on the <a 174href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">C# 175XmlTextReader class</a> set of properties and methods:</p> 176<ul> 177 <li><em>NodeType</em>: The node type, 1 for start element, 15 for end of 178 element, 2 for attributes, 3 for text nodes, 4 for CData sections, 5 for 179 entity references, 6 for entity declarations, 7 for PIs, 8 for comments, 180 9 for the document nodes, 10 for DTD/Doctype nodes, 11 for document 181 fragment and 12 for notation nodes.</li> 182 <li><em>Name</em>: the <a 183 href="http://www.w3.org/TR/REC-xml-names/#ns-qualnames">qualified 184 name</a> of the node, equal to (<em>Prefix</em>:)<em>LocalName</em>.</li> 185 <li><em>LocalName</em>: the <a 186 href="http://www.w3.org/TR/REC-xml-names/#NT-LocalPart">local name</a> of 187 the node.</li> 188 <li><em>Prefix</em>: a shorthand reference to the <a 189 href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with 190 the node.</li> 191 <li><em>NamespaceUri</em>: the URI defining the <a 192 href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with 193 the node.</li> 194 <li><em>BaseUri:</em> the base URI of the node. See the <a 195 href="http://www.w3.org/TR/xmlbase/">XML Base W3C specification</a>.</li> 196 <li><em>Depth:</em> the depth of the node in the tree, starts at 0 for the 197 root node.</li> 198 <li><em>HasAttributes</em>: whether the node has attributes.</li> 199 <li><em>HasValue</em>: whether the node can have a text value.</li> 200 <li><em>Value</em>: provides the text value of the node if present.</li> 201 <li><em>IsDefault</em>: whether an Attribute node was generated from the 202 default value defined in the DTD or schema (<em>unsupported 203 yet</em>).</li> 204 <li><em>XmlLang</em>: the <a 205 href="http://www.w3.org/TR/REC-xml#sec-lang-tag">xml:lang</a> scope 206 within which the node resides.</li> 207 <li><em>IsEmptyElement</em>: check if the current node is empty, this is a 208 bit bizarre in the sense that <code><a/></code> will be considered 209 empty while <code><a></a></code> will not.</li> 210 <li><em>AttributeCount</em>: provides the number of attributes of the 211 current node.</li> 212</ul> 213 214<p>Let's look first at a small example to get this in practice by redefining 215the processNode() function in the Python example:</p> 216<pre>def processNode(reader): 217 print "%d %d %s %d" % (reader.Depth(), reader.NodeType(), 218 reader.Name(), reader.IsEmptyElement())</pre> 219 220<p>and look at the result of calling streamFile("tst.xml") for various 221content of the XML test file.</p> 222 223<p>For the minimal document "<code><doc/></code>" we get:</p> 224<pre>0 1 doc 1</pre> 225 226<p>Only one node is found, its depth is 0, type 1 indicate an element start, 227of name "doc" and it is empty. Trying now with 228"<code><doc></doc></code>" instead leads to:</p> 229<pre>0 1 doc 0 2300 15 doc 0</pre> 231 232<p>The document root node is not flagged as empty anymore and both a start 233and an end of element are detected. The following document shows how 234character data are reported:</p> 235<pre><doc><a/><b>some text</b> 236<c/></doc></pre> 237 238<p>We modifying the processNode() function to also report the node Value:</p> 239<pre>def processNode(reader): 240 print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(), 241 reader.Name(), reader.IsEmptyElement(), 242 reader.Value())</pre> 243 244<p>The result of the test is:</p> 245<pre>0 1 doc 0 None 2461 1 a 1 None 2471 1 b 0 None 2482 3 #text 0 some text 2491 15 b 0 None 2501 3 #text 0 251 2521 1 c 1 None 2530 15 doc 0 None</pre> 254 255<p>There are a few things to note:</p> 256<ul> 257 <li>the increase of the depth value (first row) as children nodes are 258 explored</li> 259 <li>the text node child of the b element, of type 3 and its content</li> 260 <li>the text node containing the line return between elements b and c</li> 261 <li>that elements have the Value None (or NULL in C)</li> 262</ul> 263 264<p>The equivalent routine for <code>processNode()</code> as used by 265<code>xmllint --stream --debug</code> is the following and can be found in 266the xmllint.c module in the source distribution:</p> 267<pre>static void processNode(xmlTextReaderPtr reader) { 268 xmlChar *name, *value; 269 270 name = xmlTextReaderName(reader); 271 if (name == NULL) 272 name = xmlStrdup(BAD_CAST "--"); 273 value = xmlTextReaderValue(reader); 274 275 printf("%d %d %s %d", 276 xmlTextReaderDepth(reader), 277 xmlTextReaderNodeType(reader), 278 name, 279 xmlTextReaderIsEmptyElement(reader)); 280 xmlFree(name); 281 if (value == NULL) 282 printf("\n"); 283 else { 284 printf(" %s\n", value); 285 xmlFree(value); 286 } 287}</pre> 288 289<h2><a name="Extracting1">Extracting information for the attributes</a></h2> 290 291<p>The previous examples don't indicate how attributes are processed. The 292simple test "<code><doc a="b"/></code>" provides the following 293result:</p> 294<pre>0 1 doc 1 None</pre> 295 296<p>This proves that attribute nodes are not traversed by default. The 297<em>HasAttributes</em> property allow to detect their presence. To check 298their content the API has special instructions. Basically two kinds of operations 299are possible:</p> 300<ol> 301 <li>to move the reader to the attribute nodes of the current element, in 302 that case the cursor is positionned on the attribute node</li> 303 <li>to directly query the element node for the attribute value</li> 304</ol> 305 306<p>In both case the attribute can be designed either by its position in the 307list of attribute (<em>MoveToAttributeNo</em> or <em>GetAttributeNo</em>) or 308by their name (and namespace):</p> 309<ul> 310 <li><em>GetAttributeNo</em>(no): provides the value of the attribute with 311 the specified index no relative to the containing element.</li> 312 <li><em>GetAttribute</em>(name): provides the value of the attribute with 313 the specified qualified name.</li> 314 <li>GetAttributeNs(localName, namespaceURI): provides the value of the 315 attribute with the specified local name and namespace URI.</li> 316 <li><em>MoveToAttributeNo</em>(no): moves the position of the current 317 instance to the attribute with the specified index relative to the 318 containing element.</li> 319 <li><em>MoveToAttribute</em>(name): moves the position of the current 320 instance to the attribute with the specified qualified name.</li> 321 <li><em>MoveToAttributeNs</em>(localName, namespaceURI): moves the position 322 of the current instance to the attribute with the specified local name 323 and namespace URI.</li> 324 <li><em>MoveToFirstAttribute</em>: moves the position of the current 325 instance to the first attribute associated with the current node.</li> 326 <li><em>MoveToNextAttribute</em>: moves the position of the current 327 instance to the next attribute associated with the current node.</li> 328 <li><em>MoveToElement</em>: moves the position of the current instance to 329 the node that contains the current Attribute node.</li> 330</ul> 331 332<p>After modifying the processNode() function to show attributes:</p> 333<pre>def processNode(reader): 334 print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(), 335 reader.Name(), reader.IsEmptyElement(), 336 reader.Value()) 337 if reader.NodeType() == 1: # Element 338 while reader.MoveToNextAttribute(): 339 print "-- %d %d (%s) [%s]" % (reader.Depth(), reader.NodeType(), 340 reader.Name(),reader.Value())</pre> 341 342<p>The output for the same input document reflects the attribute:</p> 343<pre>0 1 doc 1 None 344-- 1 2 (a) [b]</pre> 345 346<p>There are a couple of things to note on the attribute processing:</p> 347<ul> 348 <li>Their depth is the one of the carrying element plus one.</li> 349 <li>Namespace declarations are seen as attributes, as in DOM.</li> 350</ul> 351 352<h2><a name="Validating">Validating a document</a></h2> 353 354<p>Libxml2 implementation adds some extra features on top of the XmlTextReader 355API. The main one is the ability to DTD validate the parsed document 356progressively. This is simply the activation of the associated feature of the 357parser used by the reader structure. There are a few options available 358defined as the enum xmlParserProperties in the libxml/xmlreader.h header 359file:</p> 360<ul> 361 <li>XML_PARSER_LOADDTD: force loading the DTD (without validating)</li> 362 <li>XML_PARSER_DEFAULTATTRS: force attribute defaulting (this also imply 363 loading the DTD)</li> 364 <li>XML_PARSER_VALIDATE: activate DTD validation (this also imply loading 365 the DTD)</li> 366 <li>XML_PARSER_SUBST_ENTITIES: substitute entities on the fly, entity 367 reference nodes are not generated and are replaced by their expanded 368 content.</li> 369 <li>more settings might be added, those were the one available at the 2.5.0 370 release...</li> 371</ul> 372 373<p>The GetParserProp() and SetParserProp() methods can then be used to get 374and set the values of those parser properties of the reader. For example</p> 375<pre>def parseAndValidate(file): 376 reader = libxml2.newTextReaderFilename(file) 377 reader.SetParserProp(libxml2.PARSER_VALIDATE, 1) 378 ret = reader.Read() 379 while ret == 1: 380 ret = reader.Read() 381 if ret != 0: 382 print "Error parsing and validating %s" % (file)</pre> 383 384<p>This routine will parse and validate the file. Error messages can be 385captured by registering an error handler. See python/tests/reader2.py for 386more complete Python examples. At the C level the equivalent call to cativate 387the validation feature is just:</p> 388<pre>ret = xmlTextReaderSetParserProp(reader, XML_PARSER_VALIDATE, 1)</pre> 389 390<p>and a return value of 0 indicates success.</p> 391 392<h2><a name="Entities">Entities substitution</a></h2> 393 394<p>By default the xmlReader will report entities as such and not replace them 395with their content. This default behaviour can however be overriden using:</p> 396 397<p><code>reader.SetParserProp(libxml2.PARSER_SUBST_ENTITIES,1)</code></p> 398 399<h2><a name="L1142">Relax-NG Validation</a></h2> 400 401<p style="font-size: 10pt">Introduced in version 2.5.7</p> 402 403<p>Libxml2 can now validate the document being read using the xmlReader using 404Relax-NG schemas. While the Relax NG validator can't always work in a 405streamable mode, only subsets which cannot be reduced to regular expressions 406need to have their subtree expanded for validation. In practice it means 407that, unless the schemas for the top level element content is not expressable 408as a regexp, only chunk of the document needs to be parsed while 409validating.</p> 410 411<p>The steps to do so are:</p> 412<ul> 413 <li>create a reader working on a document as usual</li> 414 <li>before any call to read associate it to a Relax NG schemas, either the 415 preparsed schemas or the URL to the schemas to use</li> 416 <li>errors will be reported the usual way, and the validity status can be 417 obtained using the IsValid() interface of the reader like for DTDs.</li> 418</ul> 419 420<p>Example, assuming the reader has already being created and that the schema 421string contains the Relax-NG schemas:</p> 422<pre><code>rngp = libxml2.relaxNGNewMemParserCtxt(schema, len(schema))<br> 423rngs = rngp.relaxNGParse()<br> 424reader.RelaxNGSetSchema(rngs)<br> 425ret = reader.Read()<br> 426while ret == 1:<br> 427 ret = reader.Read()<br> 428if ret != 0:<br> 429 print "Error parsing the document"<br> 430if reader.IsValid() != 1:<br> 431 print "Document failed to validate"</code><br> 432</pre> 433 434<p>See <code>reader6.py</code> in the sources or documentation for a complete 435example.</p> 436 437<h2><a name="Mixing">Mixing the reader and tree or XPath operations</a></h2> 438 439<p style="font-size: 10pt">Introduced in version 2.5.7</p> 440 441<p>While the reader is a streaming interface, its underlying implementation 442is based on the DOM builder of libxml2. As a result it is relatively simple 443to mix operations based on both models under some constraints. To do so the 444reader has an Expand() operation allowing to grow the subtree under the 445current node. It returns a pointer to a standard node which can be 446manipulated in the usual ways. The node will get all its ancestors and the 447full subtree available. Usual operations like XPath queries can be used on 448that reduced view of the document. Here is an example extracted from 449reader5.py in the sources which extract and prints the bibliography for the 450"Dragon" compiler book from the XML 1.0 recommendation:</p> 451<pre>f = open('../../test/valid/REC-xml-19980210.xml') 452input = libxml2.inputBuffer(f) 453reader = input.newTextReader("REC") 454res="" 455while reader.Read(): 456 while reader.Name() == 'bibl': 457 node = reader.Expand() # expand the subtree 458 if node.xpathEval("@id = 'Aho'"): # use XPath on it 459 res = res + node.serialize() 460 if reader.Next() != 1: # skip the subtree 461 break;</pre> 462 463<p>Note, however that the node instance returned by the Expand() call is only 464valid until the next Read() operation. The Expand() operation does not 465affects the Read() ones, however usually once processed the full subtree is 466not useful anymore, and the Next() operation allows to skip it completely and 467process to the successor or return 0 if the document end is reached.</p> 468 469<p><a href="mailto:xml@gnome.org">Daniel Veillard</a></p> 470 471<p>$Id$</p> 472 473<p></p> 474</body> 475</html> 476