1[/============================================================================== 2 Copyright (C) 2001-2011 Hartmut Kaiser 3 Copyright (C) 2001-2011 Joel de Guzman 4 5 Distributed under the Boost Software License, Version 1.0. (See accompanying 6 file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) 7===============================================================================/] 8 9[section:lexer_concepts Lexer Concepts] 10 11__lex__ components fall into a couple of generalized __concepts__. The 12/Lexer/ is the most fundamental concept. All __lex__ components are 13models of the /Lexer/ concept. /PrimitiveLexer/, /UnaryLexer/, 14and /NaryLexer/ are all refinements of the /Lexer/ concept. 15 16The following sections provide details on these concepts. 17 18[/////////////////////////////////////////////////////////////////////////////] 19[section Lexer] 20 21[heading Description] 22 23The /Lexer/ is the most fundamental concept. A Lexer has a member 24function, `collect`, that accepts a token definition container `Def`, and a 25the name of the lexer state the token definitions of the lexer component need 26to be added to (a string). It doesn't return anything (return type is `void`). 27Each Lexer can represent a specific pattern or algorithm, or it 28can be a more complex lexer component formed as a composition of other Lexer's. 29Additionally, a Lexer exposes a member `add_actions`, that accepts the token 30definition container `Def`, while returning nothing (again, the returned type 31is `void`). 32 33[variablelist Notation 34 [[`l`] [A `Lexer`.]] 35 [[`L`] [A `Lexer` type.]] 36 [[`Def`] [A token definition container type.]] 37 [[`State`] [A type used to represent lexer state names.]] 38] 39 40[heading Valid Expressions] 41 42In the expressions below, the behavior of the lexer component, `l`, is left 43unspecified in the base `Lexer` concept. These are specified in subsequent, 44more refined concepts and by the actual models thereof. 45 46For any Lexer the following expressions must be valid: 47 48[table 49 [[Expression] [Semantics] [Return type]] 50 [[`l.collect(def, state, targetstate)`] 51 [Add all token definitions provided 52 by this Lexer instance to the lexer 53 state `state` of the token definition 54 container `def`. After matching this token, the 55 lexer should be switched into the state 56 `targetstate` (optional)] [`void`]] 57 [[`l.add_actions(def)`] [Add all semantic actions provided 58 by this Lexer instance to the token 59 definition container `def`.] [`void`]] 60] 61 62[heading Type Expressions] 63 64[table 65 [[Expression] [Description]] 66 [[`traits::is_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if 67 a certain type, `L` is a Lexer, `mpl::false_` 68 otherwise (See __mpl_boolean_constant__).]] 69] 70 71[heading Postcondition] 72 73Upon return from `l.collect` the following post conditions should hold: 74 75* On return, `def` holds all token definitions defined in the Lexer, `l`. This 76 includes all Lexer's contained inside `l`. 77 78Upon return from `l.add_actions` the following post conditions should hold: 79 80* On return, `def` holds all semantic actions correctly associated with the 81 corresponding token definitions as defined in the Lexer, `l`. This 82 includes all semantic actions defined by the Lexer's contained inside `l`. 83 84[heading Models] 85 86All lexer components in __lex__ are models of the /Lexer/ concept. 87 88[endsect] [/ Lexer Concept] 89 90[/////////////////////////////////////////////////////////////////////////////] 91[section PrimitiveLexer] 92 93[heading Description] 94 95/PrimitiveLexer/ is the most basic building block that the client uses 96to build more complex lexer components. 97 98[heading Refinement of] 99 100[:__lexer_concept__] 101 102[heading Type Expressions] 103 104[table 105 [[Expression] [Description]] 106 [[`traits::is_primitive_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if 107 a certain type, `L`, is a PrimitiveLexer, `mpl::false_` 108 otherwise (See __mpl_boolean_constant__).]] 109] 110 111[heading Models] 112 113The following lexer components conform to this model: 114 115* character literals (i.e. `'x'`), `char_`, 116* string literals (`"abc"`), `std::basic_string<>`, `string` 117 118__fixme__ Add more links to /PrimitiveLexer/ models here. 119 120[endsect] [/ PrimitiveLexer Concept] 121 122[/////////////////////////////////////////////////////////////////////////////] 123[section UnaryLexer] 124 125[heading Description] 126 127/UnaryLexer/ is a composite lexer component that has a single subject. The 128UnaryLexer may change the behavior of its subject following the 129__delegate_pattern__. 130 131[heading Refinement of] 132 133[:__lexer_concept__] 134 135[variablelist Notation 136 [[`l`] [A UnaryLexer.]] 137 [[`L`] [A UnaryLexer type.]] 138] 139 140[heading Valid Expressions] 141 142In addition to the requirements defined in __lexer_concept__, for any 143UnaryLexer the following must be met: 144 145[table 146 [[Expression] [Semantics] [Return type]] 147 [[`l.subject`] [Subject lexer component.] [__lexer_concept__]] 148] 149 150[heading Type Expressions] 151 152[table 153 [[Expression] [Description]] 154 [[`L::subject_type`] [The subject lexer component type.]] 155 [[`traits::is_unary_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if 156 a certain type, `L` is a UnaryLexer, `mpl::false_` 157 otherwise (See __mpl_boolean_constant__).]] 158] 159 160[heading Invariants] 161 162For any UnaryLexer, `L`, the following invariant always holds: 163 164* `traits::is_lexer<L::subject_type>::type` evaluates to `mpl::true_` 165 166[heading Models] 167 168The following lexer components conform to this model: 169 170* action lexer component (allowing to attach semantic actions) 171 172__fixme__ Add more links to models of UnaryLexer concept 173 174[endsect] [/ UnaryLexer Concept] 175 176[/////////////////////////////////////////////////////////////////////////////] 177[section NaryLexer] 178 179[heading Description] 180 181/NaryLexer/ is a composite lexer component that has one or more subjects. The 182NaryLexer allows its subjects to be treated in the same way as a single 183instance of a __lexer_concept__ following the __composite_pattern__. 184 185[heading Refinement of] 186 187[:__lexer_concept__] 188 189[variablelist Notation 190 [[`l`] [A NaryLexer.]] 191 [[`L`] [A NaryLexer type.]] 192] 193 194[heading Valid Expressions] 195 196In addition to the requirements defined in __lexer_concept__, for any 197NaryLexer the following must be met: 198 199[table 200 [[Expression] [Semantics] [Return type]] 201 [[`l.elements`] [The tuple of elements.] [A __fusion__ Sequence of __lexer_concept__ types.]] 202] 203 204[heading Type Expressions] 205 206[table 207 [[Expression] [Description]] 208 [[`l.elements_type`] [Elements tuple type.]] 209 [[`traits::is_nary_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if 210 a certain type, `L` is a NaryLexer, `mpl::false_` 211 otherwise (See __mpl_boolean_constant__).]] 212] 213 214[heading Invariants] 215 216For each element, `E`, in any NaryLexer, `L`, the following 217invariant always holds: 218 219* `traits::is_lexer<E>::type` evaluates to `mpl::true_` 220 221[heading Models] 222 223The following lexer components conform to this model: 224 225* lexer sequence component 226 227__fixme__ Add more links to models of NaryLexer concept 228 229[endsect] [/ NaryLexer Concept] 230 231[endsect] 232