1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> 2 3<html> 4<head> 5 <meta http-equiv="Content-Language" content="en-us"> 6 <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 7 <meta name="GENERATOR" content="Microsoft FrontPage 5.0"> 8 <meta name="ProgId" content="FrontPage.Editor.Document"> 9 10 <title>Boost Tokenizer Class</title> 11</head> 12 13<body bgcolor="#FFFFFF" text="#000000" link="#0000EE" vlink="#551A8B" alink= 14"#FF0000"> 15 <p><img src="../../../boost.png" alt="C++ Boost" width="277" height= 16 "86"><br></p> 17 18 <h1 align="center">Tokenizer Class</h1> 19 <pre> template < 20 class TokenizerFunc = char_delimiters_separator<char>, 21 class Iterator = std::string::const_iterator, 22 class Type = std::string 23 > 24 class tokenizer 25</pre> 26 27 <p>The tokenizer class provides a container view of a series of tokens 28 contained in a sequence. You set the sequence to parse and the 29 TokenizerFunction to use to parse the sequence either upon construction or 30 using the assign member function. Note: No parsing is actually done upon 31 construction. Parsing is done on demand as the tokens are accessed via the 32 iterator provided by begin.</p> 33 34 <h2>Example</h2> 35 <pre>// simple_example_1.cpp 36#include<iostream> 37#include<boost/tokenizer.hpp> 38#include<string> 39 40int main(){ 41 using namespace std; 42 using namespace boost; 43 string s = "This is, a test"; 44 tokenizer<> tok(s); 45 for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){ 46 cout << *beg << "\n"; 47 } 48} 49</pre> 50 51 <p>The output from simple_example_1 is:</p> 52 53 <blockquote> 54 55 <p><code>This<br> 56 is<br> 57 a<br> 58 test</code></p> 59 60 </blockquote> 61 62 <h3>Template Parameters</h3> 63 64 <table border="1" summary=""> 65 <tr> 66 <th>Parameter</th> 67 68 <th>Description</th> 69 </tr> 70 71 <tr> 72 <td><tt>TokenizerFunc</tt></td> 73 74 <td>The TokenizerFunction used to parse the sequence.</td> 75 </tr> 76 77 <tr> 78 <td><tt>Iterator</tt></td> 79 80 <td>The type of the iterator the specifies the sequence.</td> 81 </tr> 82 83 <tr> 84 <td><tt>Type</tt></td> 85 86 <td>The type of the token, typically string.</td> 87 </tr> 88 </table> 89 90 <p> </p> 91 92 <h2>Related Types</h2> 93 94 <table border="1" summary=""> 95 <tr> 96 <td> 97 <p align="center"><strong>Type</strong></p> 98 </td> 99 100 <td> 101 <p align="center"><strong>Remarks</strong></p> 102 </td> 103 </tr> 104 105 <tr> 106 <td>iterator</td> 107 108 <td>The type returned by begin and end. Note: the category of iterator 109 will be at most ForwardIterator. It will be InputIterator if the 110 Iterator template parameter is an InputIterator. For any other 111 category, it will be ForwardIterator.</td> 112 </tr> 113 114 <tr> 115 <td>const_iterator</td> 116 117 <td>Same type as iterator.</td> 118 </tr> 119 120 <tr> 121 <td>value_type</td> 122 123 <td>Same type as the template parameter Type</td> 124 </tr> 125 126 <tr> 127 <td>reference</td> 128 129 <td>Same type as value_type&</td> 130 </tr> 131 132 <tr> 133 <td>const_reference</td> 134 135 <td>Same type as const reference</td> 136 </tr> 137 138 <tr> 139 <td>pointer</td> 140 141 <td>Same type as value_type*</td> 142 </tr> 143 144 <tr> 145 <td>const_pointer</td> 146 147 <td>Same type as const pointer</td> 148 </tr> 149 150 <tr> 151 <td>size_type</td> 152 153 <td>void</td> 154 </tr> 155 156 <tr> 157 <td>difference_type</td> 158 159 <td>void</td> 160 </tr> 161 </table> 162 163 <p> </p> 164 165 <h2>Construction and Member Functions</h2> 166 <pre>tokenizer(Iterator first, Iterator last,const TokenizerFunc& f = TokenizerFunc()) 167 168template<class Container> 169tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc()) 170 171void assign(Iterator first, Iterator last) 172 173void assign(Iterator first, Iterator last, const TokenizerFunc& f) 174 175template<class Container> 176void assign(const Container& c) 177 178template<class Container> 179void assign(const Container& c, const TokenizerFunc& f) 180 181iterator begin() const 182 183iterator end() const 184</pre> 185 186 <table border="1" summary=""> 187 <tr> 188 <td> 189 <p align="center"><strong>Parameter</strong></p> 190 </td> 191 192 <td> 193 <p align="center"><strong>Description</strong></p> 194 </td> 195 </tr> 196 197 <tr> 198 <td>c</td> 199 200 <td>A container that contains the sequence to parse. Note: c.begin() 201 and c.end() must be convertible to the template parameter 202 Iterator.</td> 203 </tr> 204 205 <tr> 206 <td>f</td> 207 208 <td>A functor that is a model of TokenizerFunction that will be used to 209 parse the sequence.</td> 210 </tr> 211 212 <tr> 213 <td>first</td> 214 215 <td>The iterator that represents the beginning position in the sequence 216 to be parsed.</td> 217 </tr> 218 219 <tr> 220 <td>last</td> 221 222 <td>The iterator that represents the past the end position in the 223 sequence to be parsed.</td> 224 </tr> 225 </table> 226 227 <p> </p> 228 <hr> 229 230 <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src= 231 "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional" 232 height="31" width="88"></a></p> 233 234 <p>Revised 235 <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->16 February, 2008<!--webbot bot="Timestamp" endspan i-checksum="40414" --></p> 236 237 <p><i>Copyright © 2001 John R. Bandela</i></p> 238 239 <p><i>Distributed under the Boost Software License, Version 1.0. (See 240 accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or 241 copy at <a href= 242 "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p> 243</body> 244</html> 245
