1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>Lexer</title> 5<link rel="stylesheet" href="../../../../../../../../doc/src/boostbook.css" type="text/css"> 6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 7<link rel="home" href="../../../../index.html" title="Spirit 2.5.8"> 8<link rel="up" href="../lexer_concepts.html" title="Lexer Concepts"> 9<link rel="prev" href="../lexer_concepts.html" title="Lexer Concepts"> 10<link rel="next" href="primitivelexer.html" title="PrimitiveLexer"> 11</head> 12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 13<table cellpadding="2" width="100%"><tr> 14<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../../boost.png"></td> 15<td align="center"><a href="../../../../../../../../index.html">Home</a></td> 16<td align="center"><a href="../../../../../../../../libs/libraries.htm">Libraries</a></td> 17<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 18<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 19<td align="center"><a href="../../../../../../../../more/index.htm">More</a></td> 20</tr></table> 21<hr> 22<div class="spirit-nav"> 23<a accesskey="p" href="../lexer_concepts.html"><img src="../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lexer_concepts.html"><img src="../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../index.html"><img src="../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitivelexer.html"><img src="../../../../../../../../doc/src/images/next.png" alt="Next"></a> 24</div> 25<div class="section"> 26<div class="titlepage"><div><div><h5 class="title"> 27<a name="spirit.lex.reference.lexer_concepts.lexer"></a><a class="link" href="lexer.html" title="Lexer">Lexer</a> 28</h5></div></div></div> 29<h6> 30<a name="spirit.lex.reference.lexer_concepts.lexer.h0"></a> 31 <span class="phrase"><a name="spirit.lex.reference.lexer_concepts.lexer.description"></a></span><a class="link" href="lexer.html#spirit.lex.reference.lexer_concepts.lexer.description">Description</a> 32 </h6> 33<p> 34 The <span class="emphasis"><em>Lexer</em></span> is the most fundamental concept. A Lexer 35 has a member function, <code class="computeroutput"><span class="identifier">collect</span></code>, 36 that accepts a token definition container <code class="computeroutput"><span class="identifier">Def</span></code>, 37 and a the name of the lexer state the token definitions of the lexer 38 component need to be added to (a string). It doesn't return anything 39 (return type is <code class="computeroutput"><span class="keyword">void</span></code>). Each 40 Lexer can represent a specific pattern or algorithm, or it can be a more 41 complex lexer component formed as a composition of other Lexer's. Additionally, 42 a Lexer exposes a member <code class="computeroutput"><span class="identifier">add_actions</span></code>, 43 that accepts the token definition container <code class="computeroutput"><span class="identifier">Def</span></code>, 44 while returning nothing (again, the returned type is <code class="computeroutput"><span class="keyword">void</span></code>). 45 </p> 46<div class="variablelist"> 47<p class="title"><b>Notation</b></p> 48<dl class="variablelist"> 49<dt><span class="term"><code class="computeroutput"><span class="identifier">l</span></code></span></dt> 50<dd><p> 51 A <code class="computeroutput"><span class="identifier">Lexer</span></code>. 52 </p></dd> 53<dt><span class="term"><code class="computeroutput"><span class="identifier">L</span></code></span></dt> 54<dd><p> 55 A <code class="computeroutput"><span class="identifier">Lexer</span></code> type. 56 </p></dd> 57<dt><span class="term"><code class="computeroutput"><span class="identifier">Def</span></code></span></dt> 58<dd><p> 59 A token definition container type. 60 </p></dd> 61<dt><span class="term"><code class="computeroutput"><span class="identifier">State</span></code></span></dt> 62<dd><p> 63 A type used to represent lexer state names. 64 </p></dd> 65</dl> 66</div> 67<h6> 68<a name="spirit.lex.reference.lexer_concepts.lexer.h1"></a> 69 <span class="phrase"><a name="spirit.lex.reference.lexer_concepts.lexer.valid_expressions"></a></span><a class="link" href="lexer.html#spirit.lex.reference.lexer_concepts.lexer.valid_expressions">Valid 70 Expressions</a> 71 </h6> 72<p> 73 In the expressions below, the behavior of the lexer component, <code class="computeroutput"><span class="identifier">l</span></code>, is left unspecified in the base 74 <code class="computeroutput"><span class="identifier">Lexer</span></code> concept. These 75 are specified in subsequent, more refined concepts and by the actual 76 models thereof. 77 </p> 78<p> 79 For any Lexer the following expressions must be valid: 80 </p> 81<div class="informaltable"><table class="table"> 82<colgroup> 83<col> 84<col> 85<col> 86</colgroup> 87<thead><tr> 88<th> 89 <p> 90 Expression 91 </p> 92 </th> 93<th> 94 <p> 95 Semantics 96 </p> 97 </th> 98<th> 99 <p> 100 Return type 101 </p> 102 </th> 103</tr></thead> 104<tbody> 105<tr> 106<td> 107 <p> 108 <code class="computeroutput"><span class="identifier">l</span><span class="special">.</span><span class="identifier">collect</span><span class="special">(</span><span class="identifier">def</span><span class="special">,</span> 109 <span class="identifier">state</span><span class="special">,</span> 110 <span class="identifier">targetstate</span><span class="special">)</span></code> 111 </p> 112 </td> 113<td> 114 <p> 115 Add all token definitions provided by this Lexer instance to 116 the lexer state <code class="computeroutput"><span class="identifier">state</span></code> 117 of the token definition container <code class="computeroutput"><span class="identifier">def</span></code>. 118 After matching this token, the lexer should be switched into 119 the state <code class="computeroutput"><span class="identifier">targetstate</span></code> 120 (optional) 121 </p> 122 </td> 123<td> 124 <p> 125 <code class="computeroutput"><span class="keyword">void</span></code> 126 </p> 127 </td> 128</tr> 129<tr> 130<td> 131 <p> 132 <code class="computeroutput"><span class="identifier">l</span><span class="special">.</span><span class="identifier">add_actions</span><span class="special">(</span><span class="identifier">def</span><span class="special">)</span></code> 133 </p> 134 </td> 135<td> 136 <p> 137 Add all semantic actions provided by this Lexer instance to 138 the token definition container <code class="computeroutput"><span class="identifier">def</span></code>. 139 </p> 140 </td> 141<td> 142 <p> 143 <code class="computeroutput"><span class="keyword">void</span></code> 144 </p> 145 </td> 146</tr> 147</tbody> 148</table></div> 149<h6> 150<a name="spirit.lex.reference.lexer_concepts.lexer.h2"></a> 151 <span class="phrase"><a name="spirit.lex.reference.lexer_concepts.lexer.type_expressions"></a></span><a class="link" href="lexer.html#spirit.lex.reference.lexer_concepts.lexer.type_expressions">Type 152 Expressions</a> 153 </h6> 154<div class="informaltable"><table class="table"> 155<colgroup> 156<col> 157<col> 158</colgroup> 159<thead><tr> 160<th> 161 <p> 162 Expression 163 </p> 164 </th> 165<th> 166 <p> 167 Description 168 </p> 169 </th> 170</tr></thead> 171<tbody><tr> 172<td> 173 <p> 174 <code class="computeroutput"><span class="identifier">traits</span><span class="special">::</span><span class="identifier">is_lexer</span><span class="special"><</span><span class="identifier">L</span><span class="special">>::</span><span class="identifier">type</span></code> 175 </p> 176 </td> 177<td> 178 <p> 179 Metafunction that evaluates to <code class="computeroutput"><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">true_</span></code> 180 if a certain type, <code class="computeroutput"><span class="identifier">L</span></code> 181 is a Lexer, <code class="computeroutput"><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">false_</span></code> 182 otherwise (See <a href="../../../../../../../../libs/mpl/doc/refmanual/integral-constant.html" target="_top">MPL 183 Boolean Constant</a>). 184 </p> 185 </td> 186</tr></tbody> 187</table></div> 188<h6> 189<a name="spirit.lex.reference.lexer_concepts.lexer.h3"></a> 190 <span class="phrase"><a name="spirit.lex.reference.lexer_concepts.lexer.postcondition"></a></span><a class="link" href="lexer.html#spirit.lex.reference.lexer_concepts.lexer.postcondition">Postcondition</a> 191 </h6> 192<p> 193 Upon return from <code class="computeroutput"><span class="identifier">l</span><span class="special">.</span><span class="identifier">collect</span></code> the following post conditions 194 should hold: 195 </p> 196<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"> 197 On return, <code class="computeroutput"><span class="identifier">def</span></code> holds 198 all token definitions defined in the Lexer, <code class="computeroutput"><span class="identifier">l</span></code>. 199 This includes all Lexer's contained inside <code class="computeroutput"><span class="identifier">l</span></code>. 200 </li></ul></div> 201<p> 202 Upon return from <code class="computeroutput"><span class="identifier">l</span><span class="special">.</span><span class="identifier">add_actions</span></code> the following post conditions 203 should hold: 204 </p> 205<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"> 206 On return, <code class="computeroutput"><span class="identifier">def</span></code> holds 207 all semantic actions correctly associated with the corresponding 208 token definitions as defined in the Lexer, <code class="computeroutput"><span class="identifier">l</span></code>. 209 This includes all semantic actions defined by the Lexer's contained 210 inside <code class="computeroutput"><span class="identifier">l</span></code>. 211 </li></ul></div> 212<h6> 213<a name="spirit.lex.reference.lexer_concepts.lexer.h4"></a> 214 <span class="phrase"><a name="spirit.lex.reference.lexer_concepts.lexer.models"></a></span><a class="link" href="lexer.html#spirit.lex.reference.lexer_concepts.lexer.models">Models</a> 215 </h6> 216<p> 217 All lexer components in <span class="emphasis"><em>Spirit.Lex</em></span> are models of 218 the <span class="emphasis"><em>Lexer</em></span> concept. 219 </p> 220</div> 221<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 222<td align="left"></td> 223<td align="right"><div class="copyright-footer">Copyright © 2001-2011 Joel de Guzman, Hartmut Kaiser<p> 224 Distributed under the Boost Software License, Version 1.0. (See accompanying 225 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) 226 </p> 227</div></td> 228</tr></table> 229<hr> 230<div class="spirit-nav"> 231<a accesskey="p" href="../lexer_concepts.html"><img src="../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lexer_concepts.html"><img src="../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../index.html"><img src="../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitivelexer.html"><img src="../../../../../../../../doc/src/images/next.png" alt="Next"></a> 232</div> 233</body> 234</html> 235
