1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>Quickstart 3 - Counting Words Using a Parser</title> 5<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css"> 6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 7<link rel="home" href="../../../index.html" title="Spirit 2.5.8"> 8<link rel="up" href="../tutorials.html" title="Spirit.Lex Tutorials"> 9<link rel="prev" href="lexer_quickstart2.html" title="Quickstart 2 - A better word counter using Spirit.Lex"> 10<link rel="next" href="../abstracts.html" title="Abstracts"> 11</head> 12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 13<table cellpadding="2" width="100%"><tr> 14<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td> 15<td align="center"><a href="../../../../../../../index.html">Home</a></td> 16<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td> 17<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 18<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 19<td align="center"><a href="../../../../../../../more/index.htm">More</a></td> 20</tr></table> 21<hr> 22<div class="spirit-nav"> 23<a accesskey="p" href="lexer_quickstart2.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../abstracts.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 24</div> 25<div class="section"> 26<div class="titlepage"><div><div><h4 class="title"> 27<a name="spirit.lex.tutorials.lexer_quickstart3"></a><a class="link" href="lexer_quickstart3.html" title="Quickstart 3 - Counting Words Using a Parser">Quickstart 28 3 - Counting Words Using a Parser</a> 29</h4></div></div></div> 30<p> 31 The whole purpose of integrating <span class="emphasis"><em>Spirit.Lex</em></span> as part 32 of the <a href="http://boost-spirit.com" target="_top">Spirit</a> library was 33 to add a library allowing the merger of lexical analysis with the parsing 34 process as defined by a <a href="http://boost-spirit.com" target="_top">Spirit</a> 35 grammar. <a href="http://boost-spirit.com" target="_top">Spirit</a> parsers read 36 their input from an input sequence accessed by iterators. So naturally, 37 we chose iterators to be used as the interface between the lexer and the 38 parser. A second goal of the lexer/parser integration was to enable the 39 usage of different lexical analyzer libraries. The utilization of iterators 40 seemed to be the right choice from this standpoint as well, mainly because 41 these can be used as an abstraction layer hiding implementation specifics 42 of the used lexer library. The <a class="link" href="lexer_quickstart3.html#spirit.lex.flowcontrol" title="Figure 7. The common flow control implemented while parsing combined with lexical analysis">picture</a> 43 below shows the common flow control implemented while parsing combined 44 with lexical analysis. 45 </p> 46<p> 47 </p> 48<div class="figure"> 49<a name="spirit.lex.flowcontrol"></a><p class="title"><b>Figure 7. The common flow control implemented while parsing 50 combined with lexical analysis</b></p> 51<div class="figure-contents"><span class="inlinemediaobject"><img src="../../.././images/flowofcontrol.png" alt="The common flow control implemented while parsing combined with lexical analysis"></span></div> 52</div> 53<p><br class="figure-break"> 54 </p> 55<p> 56 Another problem related to the integration of the lexical analyzer with 57 the parser was to find a way how the defined tokens syntactically could 58 be blended with the grammar definition syntax of <a href="http://boost-spirit.com" target="_top">Spirit</a>. 59 For tokens defined as instances of the <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><></span></code> class the most natural way of integration 60 was to allow to directly use these as parser components. Semantically these 61 parser components succeed matching their input whenever the corresponding 62 token type has been matched by the lexer. This quick start example will 63 demonstrate this (and more) by counting words again, simply by adding up 64 the numbers inside of semantic actions of a parser (for the full example 65 code see here: <a href="../../../../../example/lex/word_count.cpp" target="_top">word_count.cpp</a>). 66 </p> 67<h6> 68<a name="spirit.lex.tutorials.lexer_quickstart3.h0"></a> 69 <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart3.prerequisites"></a></span><a class="link" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.prerequisites">Prerequisites</a> 70 </h6> 71<p> 72 This example uses two of the <a href="http://boost-spirit.com" target="_top">Spirit</a> 73 library components: <span class="emphasis"><em>Spirit.Lex</em></span> and <span class="emphasis"><em>Spirit.Qi</em></span>, 74 consequently we have to <code class="computeroutput"><span class="preprocessor">#include</span></code> 75 the corresponding header files. Again, we need to include a couple of header 76 files from the <a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a> 77 library. This example shows how to attach functors to parser components, 78 which could be done using any type of C++ technique resulting in a callable 79 object. Using <a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a> 80 for this task simplifies things and avoids adding dependencies to other 81 libraries (<a href="../../../../../../../libs/phoenix/doc/html/index.html" target="_top">Boost.Phoenix</a> 82 is already in use for <a href="http://boost-spirit.com" target="_top">Spirit</a> 83 anyway). 84 </p> 85<p> 86</p> 87<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">qi</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 88<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_lexertl</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 89<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_operator</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 90<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_statement</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 91<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">phoenix_container</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 92</pre> 93<p> 94 </p> 95<p> 96 To make all the code below more readable we introduce the following namespaces. 97 </p> 98<p> 99</p> 100<pre class="programlisting"><span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">;</span> 101<span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">ascii</span><span class="special">;</span> 102</pre> 103<p> 104 </p> 105<h6> 106<a name="spirit.lex.tutorials.lexer_quickstart3.h1"></a> 107 <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart3.defining_tokens"></a></span><a class="link" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.defining_tokens">Defining 108 Tokens</a> 109 </h6> 110<p> 111 If compared to the two previous quick start examples (<a class="link" href="lexer_quickstart1.html" title="Quickstart 1 - A word counter using Spirit.Lex">Lex 112 Quickstart 1 - A word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a> 113 and <a class="link" href="lexer_quickstart2.html" title="Quickstart 2 - A better word counter using Spirit.Lex">Lex Quickstart 114 2 - A better word counter using <span class="emphasis"><em>Spirit.Lex</em></span></a>) 115 the token definition class for this example does not reveal any surprises. 116 However, it uses lexer token definition macros to simplify the composition 117 of the regular expressions, which will be described in more detail in the 118 section <span class="bold"><strong>FIXME</strong></span>. Generally, any token definition 119 is usable without modification from either a stand alone lexical analyzer 120 or in conjunction with a parser. 121 </p> 122<p> 123</p> 124<pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">></span> 125<span class="keyword">struct</span> <span class="identifier">word_count_tokens</span> <span class="special">:</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><</span><span class="identifier">Lexer</span><span class="special">></span> 126<span class="special">{</span> 127 <span class="identifier">word_count_tokens</span><span class="special">()</span> 128 <span class="special">{</span> 129 <span class="comment">// define patterns (lexer macros) to be used during token definition </span> 130 <span class="comment">// below</span> 131 <span class="keyword">this</span><span class="special">-></span><span class="identifier">self</span><span class="special">.</span><span class="identifier">add_pattern</span> 132 <span class="special">(</span><span class="string">"WORD"</span><span class="special">,</span> <span class="string">"[^ \t\n]+"</span><span class="special">)</span> 133 <span class="special">;</span> 134 135 <span class="comment">// define tokens and associate them with the lexer</span> 136 <span class="identifier">word</span> <span class="special">=</span> <span class="string">"{WORD}"</span><span class="special">;</span> <span class="comment">// reference the pattern 'WORD' as defined above</span> 137 138 <span class="comment">// this lexer will recognize 3 token types: words, newlines, and </span> 139 <span class="comment">// everything else</span> 140 <span class="keyword">this</span><span class="special">-></span><span class="identifier">self</span><span class="special">.</span><span class="identifier">add</span> 141 <span class="special">(</span><span class="identifier">word</span><span class="special">)</span> <span class="comment">// no token id is needed here</span> 142 <span class="special">(</span><span class="char">'\n'</span><span class="special">)</span> <span class="comment">// characters are usable as tokens as well</span> 143 <span class="special">(</span><span class="string">"."</span><span class="special">,</span> <span class="identifier">IDANY</span><span class="special">)</span> <span class="comment">// string literals will not be escaped by the library</span> 144 <span class="special">;</span> 145 <span class="special">}</span> 146 147 <span class="comment">// the token 'word' exposes the matched string as its parser attribute</span> 148 <span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> <span class="identifier">word</span><span class="special">;</span> 149<span class="special">};</span> 150</pre> 151<p> 152 </p> 153<h6> 154<a name="spirit.lex.tutorials.lexer_quickstart3.h2"></a> 155 <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart3.using_token_definition_instances_as_parsers"></a></span><a class="link" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.using_token_definition_instances_as_parsers">Using 156 Token Definition Instances as Parsers</a> 157 </h6> 158<p> 159 While the integration of lexer and parser in the control flow is achieved 160 by using special iterators wrapping the lexical analyzer, we still need 161 a means of expressing in the grammar what tokens to match and where. The 162 token definition class above uses three different ways of defining a token: 163 </p> 164<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "> 165<li class="listitem"> 166 Using an instance of a <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><></span></code>, which is handy whenever you 167 need to specify a token attribute (for more information about lexer 168 related attributes please look here: Lexer Attributes). 169 </li> 170<li class="listitem"> 171 Using a single character as the token, in this case the character represents 172 itself as a token, where the token id is the ASCII character value. 173 </li> 174<li class="listitem"> 175 Using a regular expression represented as a string, where the token 176 id needs to be specified explicitly to make the token accessible from 177 the grammar level. 178 </li> 179</ul></div> 180<p> 181 All three token definition methods require a different method of grammar 182 integration. But as you can see from the following code snippet, each of 183 these methods are straightforward and blend the corresponding token instances 184 naturally with the surrounding <span class="emphasis"><em>Spirit.Qi</em></span> grammar syntax. 185 </p> 186<div class="informaltable"><table class="table"> 187<colgroup> 188<col> 189<col> 190</colgroup> 191<thead><tr> 192<th> 193 <p> 194 Token definition 195 </p> 196 </th> 197<th> 198 <p> 199 Parser integration 200 </p> 201 </th> 202</tr></thead> 203<tbody> 204<tr> 205<td> 206 <p> 207 <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><></span></code> 208 </p> 209 </td> 210<td> 211 <p> 212 The <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><></span></code> instance is directly 213 usable as a parser component. Parsing of this component will 214 succeed if the regular expression used to define this has been 215 matched successfully. 216 </p> 217 </td> 218</tr> 219<tr> 220<td> 221 <p> 222 single character 223 </p> 224 </td> 225<td> 226 <p> 227 The single character is directly usable in the grammar. However, 228 under certain circumstances it needs to be wrapped by a <code class="computeroutput"><span class="identifier">char_</span><span class="special">()</span></code> 229 parser component. Parsing of this component will succeed if the 230 single character has been matched. 231 </p> 232 </td> 233</tr> 234<tr> 235<td> 236 <p> 237 explicit token id 238 </p> 239 </td> 240<td> 241 <p> 242 To use an explicit token id in a <span class="emphasis"><em>Spirit.Qi</em></span> 243 grammar you are required to wrap it with the special <code class="computeroutput"><span class="identifier">token</span><span class="special">()</span></code> 244 parser component. Parsing of this component will succeed if the 245 current token has the same token id as specified in the expression 246 <code class="computeroutput"><span class="identifier">token</span><span class="special">(<</span><span class="identifier">id</span><span class="special">>)</span></code>. 247 </p> 248 </td> 249</tr> 250</tbody> 251</table></div> 252<p> 253 The grammar definition below uses each of the three types demonstrating 254 their usage. 255 </p> 256<p> 257</p> 258<pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">></span> 259<span class="keyword">struct</span> <span class="identifier">word_count_grammar</span> <span class="special">:</span> <span class="identifier">qi</span><span class="special">::</span><span class="identifier">grammar</span><span class="special"><</span><span class="identifier">Iterator</span><span class="special">></span> 260<span class="special">{</span> 261 <span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">TokenDef</span><span class="special">></span> 262 <span class="identifier">word_count_grammar</span><span class="special">(</span><span class="identifier">TokenDef</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">tok</span><span class="special">)</span> 263 <span class="special">:</span> <span class="identifier">word_count_grammar</span><span class="special">::</span><span class="identifier">base_type</span><span class="special">(</span><span class="identifier">start</span><span class="special">)</span> 264 <span class="special">,</span> <span class="identifier">c</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">w</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">l</span><span class="special">(</span><span class="number">0</span><span class="special">)</span> 265 <span class="special">{</span> 266 <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">phoenix</span><span class="special">::</span><span class="identifier">ref</span><span class="special">;</span> 267 <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">phoenix</span><span class="special">::</span><span class="identifier">size</span><span class="special">;</span> 268 269 <span class="identifier">start</span> <span class="special">=</span> <span class="special">*(</span> <span class="identifier">tok</span><span class="special">.</span><span class="identifier">word</span> <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">w</span><span class="special">),</span> <span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">+=</span> <span class="identifier">size</span><span class="special">(</span><span class="identifier">_1</span><span class="special">)]</span> 270 <span class="special">|</span> <span class="identifier">lit</span><span class="special">(</span><span class="char">'\n'</span><span class="special">)</span> <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">),</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">l</span><span class="special">)]</span> 271 <span class="special">|</span> <span class="identifier">qi</span><span class="special">::</span><span class="identifier">token</span><span class="special">(</span><span class="identifier">IDANY</span><span class="special">)</span> <span class="special">[++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)]</span> 272 <span class="special">)</span> 273 <span class="special">;</span> 274 <span class="special">}</span> 275 276 <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span> <span class="identifier">c</span><span class="special">,</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">l</span><span class="special">;</span> 277 <span class="identifier">qi</span><span class="special">::</span><span class="identifier">rule</span><span class="special"><</span><span class="identifier">Iterator</span><span class="special">></span> <span class="identifier">start</span><span class="special">;</span> 278<span class="special">};</span> 279</pre> 280<p> 281 </p> 282<p> 283 As already described (see: <a class="link" href="../../abstracts/attributes.html" title="Attributes">Attributes</a>), 284 the <span class="emphasis"><em>Spirit.Qi</em></span> parser library builds upon a set of 285 fully attributed parser components. Consequently, all token definitions 286 support this attribute model as well. The most natural way of implementing 287 this was to use the token values as the attributes exposed by the parser 288 component corresponding to the token definition (you can read more about 289 this topic here: <a class="link" href="../abstracts/lexer_primitives/lexer_token_values.html" title="About Tokens and Token Values">About 290 Tokens and Token Values</a>). The example above takes advantage of the 291 full integration of the token values as the <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><></span></code>'s parser attributes: the <code class="computeroutput"><span class="identifier">word</span></code> token definition is declared as 292 a <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span></code>, 293 making every instance of a <code class="computeroutput"><span class="identifier">word</span></code> 294 token carry the string representation of the matched input sequence as 295 its value. The semantic action attached to <code class="computeroutput"><span class="identifier">tok</span><span class="special">.</span><span class="identifier">word</span></code> 296 receives this string (represented by the <code class="computeroutput"><span class="identifier">_1</span></code> 297 placeholder) and uses it to calculate the number of matched characters: 298 <code class="computeroutput"><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">+=</span> 299 <span class="identifier">size</span><span class="special">(</span><span class="identifier">_1</span><span class="special">)</span></code>. 300 </p> 301<h6> 302<a name="spirit.lex.tutorials.lexer_quickstart3.h3"></a> 303 <span class="phrase"><a name="spirit.lex.tutorials.lexer_quickstart3.pulling_everything_together"></a></span><a class="link" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.pulling_everything_together">Pulling 304 Everything Together</a> 305 </h6> 306<p> 307 The main function needs to implement a bit more logic now as we have to 308 initialize and start not only the lexical analysis but the parsing process 309 as well. The three type definitions (<code class="computeroutput"><span class="keyword">typedef</span></code> 310 statements) simplify the creation of the lexical analyzer and the grammar. 311 After reading the contents of the given file into memory it calls the function 312 <code class="computeroutput"><span class="identifier">tokenize_and_parse</span><span class="special">()</span></code> 313 to initialize the lexical analysis and parsing processes. 314 </p> 315<p> 316</p> 317<pre class="programlisting"><span class="keyword">int</span> <span class="identifier">main</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">argc</span><span class="special">,</span> <span class="keyword">char</span><span class="special">*</span> <span class="identifier">argv</span><span class="special">[])</span> 318<span class="special">{</span> 319<a class="co" name="spirit.lex.tutorials.lexer_quickstart3.c0" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.c1"><img src="../../../../../../../doc/src/images/callouts/1.png" alt="1" border="0"></a> <span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><</span> 320 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">vector</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> 321 <span class="special">></span> <span class="identifier">token_type</span><span class="special">;</span> 322 323<a class="co" name="spirit.lex.tutorials.lexer_quickstart3.c2" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.c3"><img src="../../../../../../../doc/src/images/callouts/2.png" alt="2" border="0"></a> <span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><</span><span class="identifier">token_type</span><span class="special">></span> <span class="identifier">lexer_type</span><span class="special">;</span> 324 325<a class="co" name="spirit.lex.tutorials.lexer_quickstart3.c4" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.c5"><img src="../../../../../../../doc/src/images/callouts/3.png" alt="3" border="0"></a> <span class="keyword">typedef</span> <span class="identifier">word_count_tokens</span><span class="special"><</span><span class="identifier">lexer_type</span><span class="special">>::</span><span class="identifier">iterator_type</span> <span class="identifier">iterator_type</span><span class="special">;</span> 326 327 <span class="comment">// now we use the types defined above to create the lexer and grammar</span> 328 <span class="comment">// object instances needed to invoke the parsing process</span> 329 <span class="identifier">word_count_tokens</span><span class="special"><</span><span class="identifier">lexer_type</span><span class="special">></span> <span class="identifier">word_count</span><span class="special">;</span> <span class="comment">// Our lexer</span> 330 <span class="identifier">word_count_grammar</span><span class="special"><</span><span class="identifier">iterator_type</span><span class="special">></span> <span class="identifier">g</span> <span class="special">(</span><span class="identifier">word_count</span><span class="special">);</span> <span class="comment">// Our parser </span> 331 332 <span class="comment">// read in the file int memory</span> 333 <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">str</span> <span class="special">(</span><span class="identifier">read_from_file</span><span class="special">(</span><span class="number">1</span> <span class="special">==</span> <span class="identifier">argc</span> <span class="special">?</span> <span class="string">"word_count.input"</span> <span class="special">:</span> <span class="identifier">argv</span><span class="special">[</span><span class="number">1</span><span class="special">]));</span> 334 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">first</span> <span class="special">=</span> <span class="identifier">str</span><span class="special">.</span><span class="identifier">c_str</span><span class="special">();</span> 335 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">last</span> <span class="special">=</span> <span class="special">&</span><span class="identifier">first</span><span class="special">[</span><span class="identifier">str</span><span class="special">.</span><span class="identifier">size</span><span class="special">()];</span> 336 337<a class="co" name="spirit.lex.tutorials.lexer_quickstart3.c6" href="lexer_quickstart3.html#spirit.lex.tutorials.lexer_quickstart3.c7"><img src="../../../../../../../doc/src/images/callouts/4.png" alt="4" border="0"></a> <span class="keyword">bool</span> <span class="identifier">r</span> <span class="special">=</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_parse</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">,</span> <span class="identifier">word_count</span><span class="special">,</span> <span class="identifier">g</span><span class="special">);</span> 338 339 <span class="keyword">if</span> <span class="special">(</span><span class="identifier">r</span><span class="special">)</span> <span class="special">{</span> 340 <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special"><<</span> <span class="string">"lines: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">l</span> <span class="special"><<</span> <span class="string">", words: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">w</span> 341 <span class="special"><<</span> <span class="string">", characters: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">c</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span> 342 <span class="special">}</span> 343 <span class="keyword">else</span> <span class="special">{</span> 344 <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">rest</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">);</span> 345 <span class="identifier">std</span><span class="special">::</span><span class="identifier">cerr</span> <span class="special"><<</span> <span class="string">"Parsing failed\n"</span> <span class="special"><<</span> <span class="string">"stopped at: \""</span> 346 <span class="special"><<</span> <span class="identifier">rest</span> <span class="special"><<</span> <span class="string">"\"\n"</span><span class="special">;</span> 347 <span class="special">}</span> 348 <span class="keyword">return</span> <span class="number">0</span><span class="special">;</span> 349<span class="special">}</span> 350</pre> 351<p> 352 </p> 353<div class="calloutlist"><table border="0" summary="Callout list"> 354<tr> 355<td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart3.c1"></a><a href="#spirit.lex.tutorials.lexer_quickstart3.c0"><img src="../../../../../../../doc/src/images/callouts/1.png" alt="1" border="0"></a> </p></td> 356<td valign="top" align="left"><p> 357 Define the token type to be used: <code class="computeroutput"><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span></code> 358 is available as the type of the token attribute 359 </p></td> 360</tr> 361<tr> 362<td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart3.c3"></a><a href="#spirit.lex.tutorials.lexer_quickstart3.c2"><img src="../../../../../../../doc/src/images/callouts/2.png" alt="2" border="0"></a> </p></td> 363<td valign="top" align="left"><p> 364 Define the lexer type to use implementing the state machine 365 </p></td> 366</tr> 367<tr> 368<td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart3.c5"></a><a href="#spirit.lex.tutorials.lexer_quickstart3.c4"><img src="../../../../../../../doc/src/images/callouts/3.png" alt="3" border="0"></a> </p></td> 369<td valign="top" align="left"><p> 370 Define the iterator type exposed by the lexer type 371 </p></td> 372</tr> 373<tr> 374<td width="5%" valign="top" align="left"><p><a name="spirit.lex.tutorials.lexer_quickstart3.c7"></a><a href="#spirit.lex.tutorials.lexer_quickstart3.c6"><img src="../../../../../../../doc/src/images/callouts/4.png" alt="4" border="0"></a> </p></td> 375<td valign="top" align="left"><p> 376 Parsing is done based on the token stream, not the character stream 377 read from the input. The function <code class="computeroutput"><span class="identifier">tokenize_and_parse</span><span class="special">()</span></code> wraps the passed iterator range 378 <code class="computeroutput"><span class="special">[</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">)</span></code> by the lexical analyzer and uses its 379 exposed iterators to parse the token stream. 380 </p></td> 381</tr> 382</table></div> 383</div> 384<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 385<td align="left"></td> 386<td align="right"><div class="copyright-footer">Copyright © 2001-2011 Joel de Guzman, Hartmut Kaiser<p> 387 Distributed under the Boost Software License, Version 1.0. (See accompanying 388 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) 389 </p> 390</div></td> 391</tr></table> 392<hr> 393<div class="spirit-nav"> 394<a accesskey="p" href="lexer_quickstart2.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorials.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../abstracts.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 395</div> 396</body> 397</html> 398
