1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>The Static Lexer Model</title> 5<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css"> 6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 7<link rel="home" href="../../../index.html" title="Spirit 2.5.8"> 8<link rel="up" href="../abstracts.html" title="Abstracts"> 9<link rel="prev" href="lexer_semantic_actions.html" title="Lexer Semantic Actions"> 10<link rel="next" href="../quick_reference.html" title="Quick Reference"> 11</head> 12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 13<table cellpadding="2" width="100%"><tr> 14<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td> 15<td align="center"><a href="../../../../../../../index.html">Home</a></td> 16<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td> 17<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 18<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 19<td align="center"><a href="../../../../../../../more/index.htm">More</a></td> 20</tr></table> 21<hr> 22<div class="spirit-nav"> 23<a accesskey="p" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../quick_reference.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 24</div> 25<div class="section"> 26<div class="titlepage"><div><div><h4 class="title"> 27<a name="spirit.lex.abstracts.lexer_static_model"></a><a class="link" href="lexer_static_model.html" title="The Static Lexer Model">The <span class="emphasis"><em>Static</em></span> 28 Lexer Model</a> 29</h4></div></div></div> 30<p> 31 The documentation of <span class="emphasis"><em>Spirit.Lex</em></span> so far mostly was 32 about describing the features of the <span class="emphasis"><em>dynamic</em></span> model, 33 where the tables needed for lexical analysis are generated from the regular 34 expressions at runtime. The big advantage of the dynamic model is its flexibility, 35 and its integration with the <a href="http://boost-spirit.com" target="_top">Spirit</a> 36 library and the C++ host language. Its big disadvantage is the need to 37 spend additional runtime to generate the tables, which especially might 38 be a limitation for larger lexical analyzers. The <span class="emphasis"><em>static</em></span> 39 model strives to build upon the smooth integration with <a href="http://boost-spirit.com" target="_top">Spirit</a> 40 and C++, and reuses large parts of the <span class="emphasis"><em>Spirit.Lex</em></span> 41 library as described so far, while overcoming the additional runtime requirements 42 by using pre-generated tables and tokenizer routines. To make the code 43 generation as simple as possible, the static model reuses the token definition 44 types developed for the <span class="emphasis"><em>dynamic</em></span> model without any 45 changes. As will be shown in this section, building a code generator based 46 on an existing token definition type is a matter of writing 3 lines of 47 code. 48 </p> 49<p> 50 Assuming you already built a dynamic lexer for your problem, there are 51 two more steps needed to create a static lexical analyzer using <span class="emphasis"><em>Spirit.Lex</em></span>: 52 </p> 53<div class="orderedlist"><ol class="orderedlist" type="1"> 54<li class="listitem"> 55 generating the C++ code for the static analyzer (including the tokenization 56 function and corresponding tables), and 57 </li> 58<li class="listitem"> 59 modifying the dynamic lexical analyzer to use the generated code. 60 </li> 61</ol></div> 62<p> 63 Both steps are described in more detail in the two sections below (for 64 the full source code used in this example see the code here: <a href="../../../../../example/lex/static_lexer/word_count_tokens.hpp" target="_top">the 65 common token definition</a>, <a href="../../../../../example/lex/static_lexer/word_count_generate.cpp" target="_top">the 66 code generator</a>, <a href="../../../../../example/lex/static_lexer/word_count_static.hpp" target="_top">the 67 generated code</a>, and <a href="../../../../../example/lex/static_lexer/word_count_static.cpp" target="_top">the 68 static lexical analyzer</a>). 69 </p> 70<p> 71 But first we provide the code snippets needed to further understand the 72 descriptions. Both, the definition of the used token identifier and the 73 of the token definition class in this example are put into a separate header 74 file to make these available to the code generator and the static lexical 75 analyzer. 76 </p> 77<p> 78</p> 79<pre class="programlisting"><span class="keyword">enum</span> <span class="identifier">tokenids</span> 80<span class="special">{</span> 81 <span class="identifier">IDANY</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">min_token_id</span> <span class="special">+</span> <span class="number">1</span><span class="special">,</span> 82<span class="special">};</span> 83</pre> 84<p> 85 </p> 86<p> 87 The important point here is, that the token definition class is not different 88 from a similar class to be used for a dynamic lexical analyzer. The library 89 has been designed in a way, that all components (dynamic lexical analyzer, 90 code generator, and static lexical analyzer) can reuse the very same token 91 definition syntax. 92 </p> 93<p> 94</p> 95<pre class="programlisting"><span class="comment">// This token definition class can be used without any change for all three</span> 96<span class="comment">// possible use cases: a dynamic lexical analyzer, a code generator, and a</span> 97<span class="comment">// static lexical analyzer.</span> 98<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">BaseLexer</span><span class="special">></span> 99<span class="keyword">struct</span> <span class="identifier">word_count_tokens</span> <span class="special">:</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><</span><span class="identifier">BaseLexer</span><span class="special">></span> 100<span class="special">{</span> 101 <span class="identifier">word_count_tokens</span><span class="special">()</span> 102 <span class="special">:</span> <span class="identifier">word_count_tokens</span><span class="special">::</span><span class="identifier">base_type</span><span class="special">(</span> 103 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">match_flags</span><span class="special">::</span><span class="identifier">match_not_dot_newline</span><span class="special">)</span> 104 <span class="special">{</span> 105 <span class="comment">// define tokens and associate them with the lexer</span> 106 <span class="identifier">word</span> <span class="special">=</span> <span class="string">"[^ \t\n]+"</span><span class="special">;</span> 107 <span class="keyword">this</span><span class="special">-></span><span class="identifier">self</span> <span class="special">=</span> <span class="identifier">word</span> <span class="special">|</span> <span class="char">'\n'</span> <span class="special">|</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><>(</span><span class="string">"."</span><span class="special">,</span> <span class="identifier">IDANY</span><span class="special">);</span> 108 <span class="special">}</span> 109 110 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> <span class="identifier">word</span><span class="special">;</span> 111<span class="special">};</span> 112</pre> 113<p> 114 </p> 115<p> 116 The only thing changing between the three different use cases is the template 117 parameter used to instantiate a concrete token definition. For the dynamic 118 model and the code generator you probably will use the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><></span></code> template, where for the static 119 model you will use the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">static_lexer</span><span class="special"><></span></code> type as the template parameter. 120 </p> 121<p> 122 This example not only shows how to build a static lexer, but it additionally 123 demonstrates how such a lexer can be used for parsing in conjunction with 124 a <span class="emphasis"><em>Spirit.Qi</em></span> grammar. For completeness, we provide 125 the simple grammar used in this example. As you can see, this grammar does 126 not have any dependencies on the static lexical analyzer, and for this 127 reason it is not different from a grammar used either without a lexer or 128 using a dynamic lexical analyzer as described before. 129 </p> 130<p> 131</p> 132<pre class="programlisting"><span class="comment">// This is an ordinary grammar definition following the rules defined by </span> 133<span class="comment">// Spirit.Qi. There is nothing specific about it, except it gets the token</span> 134<span class="comment">// definition class instance passed to the constructor to allow accessing the</span> 135<span class="comment">// embedded token_def<> instances.</span> 136<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">></span> 137<span class="keyword">struct</span> <span class="identifier">word_count_grammar</span> <span class="special">:</span> <span class="identifier">qi</span><span class="special">::</span><span class="identifier">grammar</span><span class="special"><</span><span class="identifier">Iterator</span><span class="special">></span> 138<span class="special">{</span> 139 <span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">TokenDef</span><span class="special">></span> 140 <span class="identifier">word_count_grammar</span><span class="special">(</span><span class="identifier">TokenDef</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">tok</span><span class="special">)</span> 141 <span class="special">:</span> <span class="identifier">word_count_grammar</span><span class="special">::</span><span class="identifier">base_type</span><span class="special">(</span><span class="identifier">start</span><span class="special">)</span> 142 <span class="special">,</span> <span class="identifier">c</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">w</span><span class="special">(</span><span class="number">0</span><span class="special">),</span> <span class="identifier">l</span><span class="special">(</span><span class="number">0</span><span class="special">)</span> 143 <span class="special">{</span> 144 <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">phoenix</span><span class="special">::</span><span class="identifier">ref</span><span class="special">;</span> 145 <span class="keyword">using</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">phoenix</span><span class="special">::</span><span class="identifier">size</span><span class="special">;</span> 146 147 <span class="comment">// associate the defined tokens with the lexer, at the same time </span> 148 <span class="comment">// defining the actions to be executed </span> 149 <span class="identifier">start</span> <span class="special">=</span> <span class="special">*(</span> <span class="identifier">tok</span><span class="special">.</span><span class="identifier">word</span> <span class="special">[</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">w</span><span class="special">),</span> <span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">+=</span> <span class="identifier">size</span><span class="special">(</span><span class="identifier">_1</span><span class="special">)</span> <span class="special">]</span> 150 <span class="special">|</span> <span class="identifier">lit</span><span class="special">(</span><span class="char">'\n'</span><span class="special">)</span> <span class="special">[</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">l</span><span class="special">),</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">]</span> 151 <span class="special">|</span> <span class="identifier">qi</span><span class="special">::</span><span class="identifier">token</span><span class="special">(</span><span class="identifier">IDANY</span><span class="special">)</span> <span class="special">[</span> <span class="special">++</span><span class="identifier">ref</span><span class="special">(</span><span class="identifier">c</span><span class="special">)</span> <span class="special">]</span> 152 <span class="special">)</span> 153 <span class="special">;</span> 154 <span class="special">}</span> 155 156 <span class="identifier">std</span><span class="special">::</span><span class="identifier">size_t</span> <span class="identifier">c</span><span class="special">,</span> <span class="identifier">w</span><span class="special">,</span> <span class="identifier">l</span><span class="special">;</span> <span class="comment">// counter for characters, words, and lines</span> 157 <span class="identifier">qi</span><span class="special">::</span><span class="identifier">rule</span><span class="special"><</span><span class="identifier">Iterator</span><span class="special">></span> <span class="identifier">start</span><span class="special">;</span> 158<span class="special">};</span> 159</pre> 160<p> 161 </p> 162<h6> 163<a name="spirit.lex.abstracts.lexer_static_model.h0"></a> 164 <span class="phrase"><a name="spirit.lex.abstracts.lexer_static_model.generating_the_static_analyzer"></a></span><a class="link" href="lexer_static_model.html#spirit.lex.abstracts.lexer_static_model.generating_the_static_analyzer">Generating 165 the Static Analyzer</a> 166 </h6> 167<p> 168 The first additional step to perform in order to create a static lexical 169 analyzer is to create a small stand alone program for creating the lexer 170 tables and the corresponding tokenization function. For this purpose the 171 <span class="emphasis"><em>Spirit.Lex</em></span> library exposes a special API - the function 172 <code class="computeroutput"><span class="identifier">generate_static_dfa</span><span class="special">()</span></code> 173 . It implements the whole code generator, no further code is needed. All 174 what it takes to invoke this function is to supply a token definition instance, 175 an output stream to use to generate the code to, and an optional string 176 to be used as a suffix for the name of the generated function. All in all 177 just a couple lines of code. 178 </p> 179<p> 180</p> 181<pre class="programlisting"><span class="keyword">int</span> <span class="identifier">main</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">argc</span><span class="special">,</span> <span class="keyword">char</span><span class="special">*</span> <span class="identifier">argv</span><span class="special">[])</span> 182<span class="special">{</span> 183 <span class="comment">// create the lexer object instance needed to invoke the generator</span> 184 <span class="identifier">word_count_tokens</span><span class="special"><</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><></span> <span class="special">></span> <span class="identifier">word_count</span><span class="special">;</span> <span class="comment">// the token definition</span> 185 186 <span class="comment">// open the output file, where the generated tokenizer function will be </span> 187 <span class="comment">// written to</span> 188 <span class="identifier">std</span><span class="special">::</span><span class="identifier">ofstream</span> <span class="identifier">out</span><span class="special">(</span><span class="identifier">argc</span> <span class="special"><</span> <span class="number">2</span> <span class="special">?</span> <span class="string">"word_count_static.hpp"</span> <span class="special">:</span> <span class="identifier">argv</span><span class="special">[</span><span class="number">1</span><span class="special">]);</span> 189 190 <span class="comment">// invoke the generator, passing the token definition, the output stream </span> 191 <span class="comment">// and the name suffix of the tables and functions to be generated</span> 192 <span class="comment">//</span> 193 <span class="comment">// The suffix "wc" used below results in a type lexertl::static_::lexer_wc</span> 194 <span class="comment">// to be generated, which needs to be passed as a template parameter to the </span> 195 <span class="comment">// lexertl::static_lexer template (see word_count_static.cpp).</span> 196 <span class="keyword">return</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">generate_static_dfa</span><span class="special">(</span><span class="identifier">word_count</span><span class="special">,</span> <span class="identifier">out</span><span class="special">,</span> <span class="string">"wc"</span><span class="special">)</span> <span class="special">?</span> <span class="number">0</span> <span class="special">:</span> <span class="special">-</span><span class="number">1</span><span class="special">;</span> 197<span class="special">}</span> 198</pre> 199<p> 200 </p> 201<p> 202 The shown code generator will generate output, which should be stored in 203 a file for later inclusion into the static lexical analyzer as shown in 204 the next topic (the full generated code can be viewed <a href="../../../../../example/lex/static_lexer/word_count_static.hpp" target="_top">here</a>). 205 </p> 206<div class="note"><table border="0" summary="Note"> 207<tr> 208<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../images/note.png"></td> 209<th align="left">Note</th> 210</tr> 211<tr><td align="left" valign="top"><p> 212 The generated code will have compiled in the version number of the current 213 <span class="emphasis"><em>Spirit.Lex</em></span> library. This version number is used 214 at compilation time of your static lexer object to ensure this is compiled 215 using exactly the same version of the <span class="emphasis"><em>Spirit.Lex</em></span> 216 library as the lexer tables have been generated with. If the versions 217 do not match you will see an compilation error mentioning an <code class="computeroutput"><span class="identifier">incompatible_static_lexer_version</span></code>. 218 </p></td></tr> 219</table></div> 220<h6> 221<a name="spirit.lex.abstracts.lexer_static_model.h1"></a> 222 <span class="phrase"><a name="spirit.lex.abstracts.lexer_static_model.modifying_the_dynamic_analyzer"></a></span><a class="link" href="lexer_static_model.html#spirit.lex.abstracts.lexer_static_model.modifying_the_dynamic_analyzer">Modifying 223 the Dynamic Analyzer</a> 224 </h6> 225<p> 226 The second required step to convert an existing dynamic lexer into a static 227 one is to change your main program at two places. First, you need to change 228 the type of the used lexer (that is the template parameter used while instantiating 229 your token definition class). While in the dynamic model we have been using 230 the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">lexer</span><span class="special"><></span></code> 231 template, we now need to change that to the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">static_lexer</span><span class="special"><></span></code> type. The second change is tightly 232 related to the first one and involves correcting the corresponding <code class="computeroutput"><span class="preprocessor">#include</span></code> statement to: 233 </p> 234<p> 235</p> 236<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_static_lexertl</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 237</pre> 238<p> 239 </p> 240<p> 241 Otherwise the main program is not different from an equivalent program 242 using the dynamic model. This feature makes it easy to develop the lexer 243 in dynamic mode and to switch to the static mode after the code has been 244 stabilized. The simple generator application shown above enables the integration 245 of the code generator into any existing build process. The following code 246 snippet provides the overall main function, highlighting the code to be 247 changed. 248 </p> 249<p> 250</p> 251<pre class="programlisting"><span class="keyword">int</span> <span class="identifier">main</span><span class="special">(</span><span class="keyword">int</span> <span class="identifier">argc</span><span class="special">,</span> <span class="keyword">char</span><span class="special">*</span> <span class="identifier">argv</span><span class="special">[])</span> 252<span class="special">{</span> 253 <span class="comment">// Define the token type to be used: 'std::string' is available as the type </span> 254 <span class="comment">// of the token value.</span> 255 <span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><</span> 256 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">vector</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> 257 <span class="special">></span> <span class="identifier">token_type</span><span class="special">;</span> 258 259 <span class="comment">// Define the lexer type to be used as the base class for our token </span> 260 <span class="comment">// definition.</span> 261 <span class="comment">//</span> 262 <span class="comment">// This is the only place where the code is different from an equivalent</span> 263 <span class="comment">// dynamic lexical analyzer. We use the `lexertl::static_lexer<>` instead of</span> 264 <span class="comment">// the `lexertl::lexer<>` as the base class for our token definition type.</span> 265 <span class="comment">//</span> 266 <span class="comment">// As we specified the suffix "wc" while generating the static tables we </span> 267 <span class="comment">// need to pass the type lexertl::static_::lexer_wc as the second template</span> 268 <span class="comment">// parameter below (see word_count_generate.cpp).</span> 269 <span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">static_lexer</span><span class="special"><</span> 270 <span class="identifier">token_type</span><span class="special">,</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">static_</span><span class="special">::</span><span class="identifier">lexer_wc</span> 271 <span class="special">></span> <span class="identifier">lexer_type</span><span class="special">;</span> 272 273 <span class="comment">// Define the iterator type exposed by the lexer.</span> 274 <span class="keyword">typedef</span> <span class="identifier">word_count_tokens</span><span class="special"><</span><span class="identifier">lexer_type</span><span class="special">>::</span><span class="identifier">iterator_type</span> <span class="identifier">iterator_type</span><span class="special">;</span> 275 276 <span class="comment">// Now we use the types defined above to create the lexer and grammar</span> 277 <span class="comment">// object instances needed to invoke the parsing process.</span> 278 <span class="identifier">word_count_tokens</span><span class="special"><</span><span class="identifier">lexer_type</span><span class="special">></span> <span class="identifier">word_count</span><span class="special">;</span> <span class="comment">// Our lexer</span> 279 <span class="identifier">word_count_grammar</span><span class="special"><</span><span class="identifier">iterator_type</span><span class="special">></span> <span class="identifier">g</span> <span class="special">(</span><span class="identifier">word_count</span><span class="special">);</span> <span class="comment">// Our parser</span> 280 281 <span class="comment">// Read in the file into memory.</span> 282 <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">str</span> <span class="special">(</span><span class="identifier">read_from_file</span><span class="special">(</span><span class="number">1</span> <span class="special">==</span> <span class="identifier">argc</span> <span class="special">?</span> <span class="string">"word_count.input"</span> <span class="special">:</span> <span class="identifier">argv</span><span class="special">[</span><span class="number">1</span><span class="special">]));</span> 283 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">first</span> <span class="special">=</span> <span class="identifier">str</span><span class="special">.</span><span class="identifier">c_str</span><span class="special">();</span> 284 <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">last</span> <span class="special">=</span> <span class="special">&</span><span class="identifier">first</span><span class="special">[</span><span class="identifier">str</span><span class="special">.</span><span class="identifier">size</span><span class="special">()];</span> 285 286 <span class="comment">// Parsing is done based on the token stream, not the character stream.</span> 287 <span class="keyword">bool</span> <span class="identifier">r</span> <span class="special">=</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_parse</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">,</span> <span class="identifier">word_count</span><span class="special">,</span> <span class="identifier">g</span><span class="special">);</span> 288 289 <span class="keyword">if</span> <span class="special">(</span><span class="identifier">r</span><span class="special">)</span> <span class="special">{</span> <span class="comment">// success</span> 290 <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special"><<</span> <span class="string">"lines: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">l</span> <span class="special"><<</span> <span class="string">", words: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">w</span> 291 <span class="special"><<</span> <span class="string">", characters: "</span> <span class="special"><<</span> <span class="identifier">g</span><span class="special">.</span><span class="identifier">c</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span> 292 <span class="special">}</span> 293 <span class="keyword">else</span> <span class="special">{</span> 294 <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span> <span class="identifier">rest</span><span class="special">(</span><span class="identifier">first</span><span class="special">,</span> <span class="identifier">last</span><span class="special">);</span> 295 <span class="identifier">std</span><span class="special">::</span><span class="identifier">cerr</span> <span class="special"><<</span> <span class="string">"Parsing failed\n"</span> <span class="special"><<</span> <span class="string">"stopped at: \""</span> 296 <span class="special"><<</span> <span class="identifier">rest</span> <span class="special"><<</span> <span class="string">"\"\n"</span><span class="special">;</span> 297 <span class="special">}</span> 298 <span class="keyword">return</span> <span class="number">0</span><span class="special">;</span> 299<span class="special">}</span> 300</pre> 301<p> 302 </p> 303<div class="important"><table border="0" summary="Important"> 304<tr> 305<td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../../images/important.png"></td> 306<th align="left">Important</th> 307</tr> 308<tr><td align="left" valign="top"><p> 309 The generated code for the static lexer contains the token ids as they 310 have been assigned, either explicitly by the programmer or implicitly 311 during lexer construction. It is your responsibility to make sure that 312 all instances of a particular static lexer type use exactly the same 313 token ids. The constructor of the lexer object has a second (default) 314 parameter allowing it to designate a starting token id to be used while 315 assigning the ids to the token definitions. The requirement above is 316 fulfilled by default as long as no <code class="computeroutput"><span class="identifier">first_id</span></code> 317 is specified during construction of the static lexer instances. 318 </p></td></tr> 319</table></div> 320</div> 321<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 322<td align="left"></td> 323<td align="right"><div class="copyright-footer">Copyright © 2001-2011 Joel de Guzman, Hartmut Kaiser<p> 324 Distributed under the Boost Software License, Version 1.0. (See accompanying 325 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) 326 </p> 327</div></td> 328</tr></table> 329<hr> 330<div class="spirit-nav"> 331<a accesskey="p" href="lexer_semantic_actions.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../abstracts.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../quick_reference.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 332</div> 333</body> 334</html> 335
