1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> 2 3<html> 4<head> 5 <meta http-equiv="Content-Language" content="en-us"> 6 <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 7 <meta name="GENERATOR" content="Microsoft FrontPage 6.0"> 8 <meta name="ProgId" content="FrontPage.Editor.Document"> 9 10 <title>TokenizerFunction Concept</title> 11</head> 12 13<body bgcolor="#FFFFFF" text="#000000" link="#0000EE" vlink="#551A8B" alink= 14"#FF0000"> 15 <p><img src="../../../boost.png" alt="C++ Boost" width="277" height="86"></p> 16 17 <h1 align="center">TokenizerFunction Concept</h1> 18 19 <p>A TokenizerFunction is a functor whose purpose is to parse a given 20 sequence until exactly 1 token has been found or the end is reached. It 21 then updates the token, and informs the caller of the location in the 22 sequence of the next element immediately after the last element of the 23 sequence that was parsed for the current token.</p> 24 25 <h2>Refinement of</h2> 26 27 <p>Assignable, CopyConstructable</p> 28 29 <h2>Notation</h2> 30 31 <table border="1" summary=""> 32 <tr> 33 <td valign="top"><tt>X</tt></td> 34 35 <td valign="top">A type that is a model of TokenizerFunction</td> 36 </tr> 37 38 <tr> 39 <td valign="top"><tt>func</tt></td> 40 41 <td valign="top">Object of type <tt>X</tt></td> 42 </tr> 43 44 <tr> 45 <td valign="top"><tt>tok</tt></td> 46 47 <td valign="top">Object of Token</td> 48 </tr> 49 50 <tr> 51 <td>next</td> 52 53 <td>iterator that points to the first unparsed element of the sequence 54 being parsed</td> 55 </tr> 56 57 <tr> 58 <td>end</td> 59 60 <td>iterator that points to the past the end of the sequence being 61 parsed</td> 62 </tr> 63 </table> 64 65 <h2>Definitions</h2> 66 67 <p>A token is the result of parsing a sequence.</p> 68 69 <h2>Valid expressions</h2> 70 71 <p>In addition to the expression in Assignable and CopyConstructable the 72 following expressions are valid</p> 73 74 <table border="1" summary=""> 75 <tr> 76 <th>Name</th> 77 78 <th>Expression</th> 79 80 <th>Return type</th> 81 </tr> 82 83 <tr> 84 <td valign="top">Functor</td> 85 86 <td valign="top"><tt>func(next, end, tok)</tt></td> 87 88 <td valign="top"><tt>bool</tt></td> 89 </tr> 90 91 <tr> 92 <td valign="top">reset</td> 93 94 <td valign="top"><tt>reset()</tt></td> 95 96 <td valign="top"><tt>void</tt></td> 97 </tr> 98 </table> 99 100 <h2>Expression semantics</h2> 101 102 <p>In addition to the expression semantics in Assignable and 103 CopyConstructable, TokenizerFunction has the following expression 104 semantcs</p> 105 106 <table border="1" summary=""> 107 <tr> 108 <th>Name</th> 109 110 <th>Expression</th> 111 112 <th>Precondition</th> 113 114 <th>Semantics</th> 115 116 <th>Postcondition</th> 117 </tr> 118 119 <tr> 120 <td>operator()</td> 121 122 <td><tt>func(next, end, tok)</tt></td> 123 124 <td><tt>next</tt> and <tt>end</tt> are valid iterators to the same 125 sequence. next is a reference the function is free to modify. tok is 126 constructed.</td> 127 128 <td>The return value indicates whether a new token was found in the 129 sequence [next,end)</td> 130 131 <td>If the return value is true, the new token is assigned to tok. next 132 is always updated to the position where parsing should start on the 133 subsequent call.</td> 134 </tr> 135 136 <tr> 137 <td>reset</td> 138 139 <td><tt>reset()</tt></td> 140 141 <td><tt>None</tt></td> 142 143 <td>Clears out all state variables that are used by the object in 144 parsing the current sequence.</td> 145 146 <td>A new sequence to parse can be given.</td> 147 </tr> 148 </table> 149 150 <h2>Complexity guarantees</h2> 151 152 <p>No guarantees. Models of TokenizerFunction are free to define their own 153 complexity</p> 154 155 <h2>Models</h2> 156 157 <p><a href="escaped_list_separator.htm">escaped_list_separator</a></p> 158 159 <p><a href="offset_separator.htm">offset_separator</a></p> 160 161 <p><a href= 162 "char_delimiters_separator.htm">char_delimiters_separator</a></p> 163 164 <p> </p> 165 <hr> 166 167 <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src= 168 "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional" 169 height="31" width="88"></a></p> 170 171 <p>Revised 172 <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 173 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p> 174 175 <p><i>Copyright © 2001 John R. Bandela</i></p> 176 177 <p><i>Distributed under the Boost Software License, Version 1.0. (See 178 accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or 179 copy at <a href= 180 "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p> 181</body> 182</html> 183
