1<html> 2<head> 3<meta http-equiv="Content-Language" content="en-us"> 4<meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> 5<title>ICU's Unicode Tools Read Me</title> 6 <meta name="COPYRIGHT" content= 7 "Copyright (c) 2004-2006 IBM Corporation and others. All Rights Reserved." /> 8<style> 9<!-- 10li { margin-top: 0.5em; margin-bottom: 0.5em } 11--> 12</style> 13</head> 14 15<body> 16 17<h1>UnicodeTools</h1> 18<p>This file provides instructions for building and running the UnicodeTools, which<br> 19can be used to:</p> 20<ul> 21 <li>build the Derived Unicode files in the UCD (Unicode Character Database),</li> 22 <li>build the transformed UCA (Unicode Collation Algorithm) files needed by ICU.</li> 23 <li>run consistency checks on beta releases of the UCD and the UCA.</li> 24 <li>build 4 chart folders on the unicode site</li> 25</ul> 26<p><font color="#FF0000"><b>WARNING!!</b></font></p> 27<ul> 28 <li>This is NOT production level code, and should never be used in programs.</li> 29 <li>The API is subject to change without notice, and will not be maintained.</li> 30 <li>The source is uncommented, and has many warts; since it is not production code, it has not 31 been worth the time to clean it up.</li> 32 <li>It will probably not work on Unix or Mac without changing the file separator.</li> 33 <li>Currently it uses hard-coded directory names.</li> 34 <li>The contents of multiple versions of the UCD must be copied to a local directory, as described 35 below.</li> 36</ul> 37<h2>Instructions:</h2> 38<h3>0. You will need to get ICU4J on your system, using CVS.</h3> 39<p>The rest of this will assume that you have set up CVS so that you load the ICU4J project into 40C:\ICU4J<br> 41<br> 42You need both the main icu4j and a subproject called unicodetools. See: 43<a href="http://www.ibm.com/software/globalization/icu/repository.jsp"> 44http://www.ibm.com/software/globalization/icu/repository.jsp</a>. Inside unicodetools, look at com/ibm/text. The 45main directories of interest are UCD, UCA and utility.</p> 46<h4>0a. If you are using Eclipse for your IDE, look at the instructions on 47<a href="http://icu.sourceforge.net/docs/eclipse_howto/eclipse_howto.html"> 48http://icu.sourceforge.net/docs/eclipse_howto/eclipse_howto.html</a> </h4> 49<p>Set up Eclipse to build two projects: ICU4J and UnicodeTools:<br> 50<br> 51<b>Project Name: </b>ICU4J<br> 52<b>Directory: </b>C:\ICU4J\icu4j<br> 53<b>Default output folder = </b>ICU4J/classes<br> 54<br> 55<b>Project Name: </b>unicodetools<br> 56<b>Create project from existing source: </b>C:\ICU4J\unicodetools<br> 57<b>Default Output Folder: </b>unicodetools/classes<br> 58<br> 59After Eclipse is set up with these, exclude certain files from unicodetools:<br> 60<br> 61Right-Click UnicodeTools > Properties > Java Build Path > Exclusions<br> 62com/ibm/rbm/<br> 63com/ibm/text/utility/UnicodeMapInt.java<br> 64com/ibm/text/utility/TestUtility.java<br> 65com/ibm/text/UCD/GenerateThaiBreaks-old.java/<br> 66com/ibm/text/UCD/ProcessUnihan.java/<br> 67com/ibm/text/UCA/WriteHTMLCollation.java/<br> 68<br> 69UnicodeTools must also include the ICU4J project, with<br> 70<br> 71Right-Click UnicodeTools > Properties > Java Build Path > Projects</p> 72<h3>1. In UCD, you must edit UCD_Types.java at the top, to set the directories for the build:</h3> 73<p>public static final String DATA_DIR = "C:\\DATA\\";<br> 74public static final String UCD_DIR = BASE_DIR + "UCD\\";<br> 75public static final String BIN_DIR = DATA_DIR + "BIN\\";<br> 76public static final String GEN_DIR = DATA_DIR + "GEN\\";<br> 77<br> 78Make sure that each of these directories exist. Also make sure that the following<br> 79exist:<br> 80<br> 81<GEN_DIR>/DerivedData<br> 82<GEN_DIR>/DerivedData/ExtractedProperties<br> 83<UCD_DIR>/EXTRAS-Update</p> 84<h3>2. Download all of the UnicodeData files for each version into UCD_DIR.</h3> 85<p>The folder names must be of the form: "3.2.0-Update", so rename the folders on the<br> 86Unicode site to this format. I<span style="background-color: #FFFF00">f the 87folder contains ucd, then make the contents of that directory be the contents of 88the x.x.x-Update directory. That is, each directory will directly contain files 89like PropList....txt</span></p> 90<h4>2a Ensure Complete Release</h4> 91<p>If you are downloading any "incomplete" release (one that does not contain a complete set of data 92files for that release, you need to also download the previous complete release). Most of the N.M-Update 93directoriess are complete, *except*:</p> 94<p>4.0-Update, which does not contain a copy of Unihan.txt and some other files<br> 953.1-Update, which does not contain a copy of BidiMirroring.txt</p> 96<p>Also, make the following changes to UnicodeData for 1.1.5:</p> 97<p><b>Delete</b></p> 98<pre>3400;HANGUL SYLLABLE KIYEOK A;Lo;0;L;1100 1161;;;;N;;;;; 99... 1004DFF;HANGUL SYLLABLE MIEUM WEO RIEUL-THIEUTH;Lo;0;L;1106 116F 11B4;;;;N;;;;; 1014E00;<cjk IDEOGRAPH REPRESENTATIVE>;Lo;0;L;;;;;N;;;;;</pre> 102<p><b>Add:</b></p> 103<pre>4E00;<cjk Ideograph, First>;Lo;0;L;;;;;N;;;;; 1049FA5;<cjk Ideograph, Last>;Lo;0;L;;;;;N;;;;; 105E000;<private Use, First>;Co;0;L;;;;;N;;;;; 106F8FF;<private Use, Last>;Co;0;L;;;;;N;;;;;</pre> 107<p><b>And from a late version of Unicode, add:</b></p> 108<pre>F900;CJK COMPATIBILITY IDEOGRAPH-F900;Lo;0;L;8C48;;;;N;;;;; 109... 110FA2D;CJK COMPATIBILITY IDEOGRAPH-FA2D;Lo;0;L;9DB4;;;;N;;;;;</pre> 111<h4>2b. UCA data</h4> 112<p>If you are building any of the UCA tools, you need to get a copy of the UCA data file<br> 113from http://www.unicode.org/reports/tr10/#AllKeys. The default location for this is:<br> 114<br> 115BASE_DIR + "Collation\allkeys" + VERSION + ".txt".<br> 116<br> 117If you have it in a different location, change that value for KEYS in UCA.java, and <br> 118the value for BASE_DIR</p> 119<h4>2c. Here is an example of the default directory structure with files. All of 120the yellow ones should exist</h4> 121<pre>C://DATA/ 122 123 BIN/ 124 125<span style="background-color: #FFFF00"> Collation/ 126 allkeys-3.1.1.txt 127</span> 128 GEN/ 129 DerivedData/ 130<span style="background-color: #FFFF00"> </span><span style="background-color: #FFFF00">UCD/ 131 3.0.0-Update/ 132 Unihan-3.2.0.txt 133 ... 134 3.0.1-Update/ 135 ... 136 3.1.0-Update/ 137 ... 138 3.1.1-Update/ 139 ... 140 3.2.0-Update/ 141 ... 142 4.0.0-Update/ 143 ArabicShaping-4.0.0d14b.txt 144 BidiMirroring-4.0.0d1b.txt 145 ... 146 EXTRAS-Update/</span></pre> 147<h3>3. Versions</h3> 148<p>All of the following have "version X" in the options you give to Java (either on the 149command line, or in the Eclipse 'run' options. If you want a specific version like 3.1.0, then you 150would write "version 3.1.1". If you want the latest version (4.1.0), you can omit the "version X".</p> 151<h3>4. Building Files</h3> 152<ol> 153 <li><b>Setup</b><ol> 154 <li>In Eclipse, open the Package Explorer (Use Window>Show View if you 155 don't see it)</li> 156 <li>Open UnicodeTools<ul> 157 <li>com.ibm.text.UCD<ul> 158 <li>MakeUnicodeFiles.<span style="background-color: #FFFF00">txt</span><p>This file drives the production of 159 the derived Unicode files. The first three lines contain 160 parameters that you may want to modify at some times:</p> 161 <pre>Generate: <b>.*script.*</b> <i>// this is a regular expression. Use .* for all files</i> 162DeltaVersion: <b>10</b> <i> // This gets appended to the file name. Pick 1+ the highest value in Public</i> 163CopyrightYear: <b>2006</b> <i> // Pick the current year</i></pre> 164 </li> 165 </ul> 166 </li> 167 </ul> 168 </li> 169 <li>Open in Package Explorer 170 <ul> 171 <li>com.ibm.text.UCD<ul> 172 <li>Main</li> 173 </ul> 174 </li> 175 </ul> 176 </li> 177 <li>Run>Run As...<ol> 178 <li>Choose Java Application<ul> 179 <li>it will fail, don't worry; you need to set some parameters.</li> 180 </ul> 181 </li> 182 </ol> 183 </li> 184 <li>Run>Run...<ul> 185 <li>Select the Arguments tab, and fill in the following<ul> 186 <li>Program arguments:<pre>build 5.0<span style="background-color: #FFFF00">.0</span> MakeUnicodeFiles</pre> 187 </li> 188 <li>VM arguments: 189 <pre>-Xms512m -Xmx512m</pre> 190 </li> 191 </ul> 192 </li> 193 <li>Close and Save</li> 194 </ul> 195 </li> 196 </ol> 197 </li> 198 <li><b>Run</b><ol> 199 <li>You'll see it build the 5.0 files, with something like the following 200 results:<pre>Writing UCD_Data5.0.0 201Data Size: 109,802 202Wrote Data 109802</pre> 203 </li> 204 <li>For each version, the tools build a set of binary data in BIN that 205 contain the information for that release. This is done automatically, or 206 you can manually do it with the Program Arguments<pre>version X build</pre> 207 <p>This builds an compressed format of all the UCD data (except blocks 208 and Unihan) into the BIN directory. Don't worry about the voluminous 209 console messages, unless one says "FAIL".</p> 210 <p><font color="#FF0000"><i>You have to manually do this if you change 211 any of the data files in that version!</i></font></p> 212 <p>Note: if for any reason you modify the binary format of the BIN files, you also have to bump the 213value in that file:</p> 214 <pre>static final byte BINARY_FORMAT = 8; // bumped if binary format of UCD changes</pre> 215 </li> 216 </ol> 217 </li> 218 <li>Results in <a href="file:///C:/DATA/GEN/DerivedData"> 219 C:\DATA\GEN\DerivedData</a><ol> 220 <li>The files will be in this directory.</li> 221 <li>There are also DIFF folders, that contain BAT files that you can run 222 on Windows with CompareIt. (You can modify the code to build BATs with 223 another Diff program if you want).<ol> 224 <li>For any file with a significant difference, it will build two 225 BAT files, such as the first two below.<pre>Diff_PropList-5.0.0d10.txt.bat 226OLDER-Diff_PropList-5.0.0d10.txt.bat 227 228UNCHANGED-Diff_PropertyValueAliases-5.0.0d10.txt.bat</pre> 229 </li> 230 </ol> 231 </li> 232 <li>Any files without significant changes will have "UNCHANGED" as a 233 prefix: ignore them. The OLDER prefix is the comparison to the 234 last version of Unicode.</li> 235 <li>On Windows you can run these BATs to compare files:</li> 236 </ol> 237 </li> 238 <li><span style="background-color: #FFFF00">NFSkippable</span><ol> 239 <li><span style="background-color: #FFFF00">A file is needed by ICU that is 240 generated with the same tool. Just use the input parameter "NFSkippable" to 241 generate the file NFSafeSets.txt, also in </span> 242 <a href="file:///C:/DATA/GEN"><span style="background-color: #FFFF00"> 243 file:///C:/DATA/GEN</span></a></li> 244</ol> 245 </li> 246</ol> 247<h3>5. Invariant Checking</h3> 248<ol> 249 <li>Setup<ol> 250 <li>Open in Package Explorer<ul> 251 <li>com.ibm.text.UCD<ul> 252 <li>TestUnicodeInvariants.java</li> 253 </ul> 254 </li> 255 </ul> 256 </li> 257 <li>Run>Run As... Java Application<br> 258 Will create the following file of results:<pre><a href="file:///C:/DATA/GEN/UnicodeInvariantResults.txt/">C:\DATA\GEN\UnicodeInvariantResults.txt\</a></pre> 259 <p>And on the console will list whether any problems are found. Thus in 260 the following case there was one failure:</p> 261 <pre>ParseErrorCount=0 262TestFailureCount=1</pre> 263 </li> 264 <li>The header of the result file explains the syntax of the tests.</li> 265 <li>Open that file and search for "**** START Error Info ****". Each such 266 point provides a dump of comparison information.<ol> 267 <li>Failures print a list of differences between two sets being 268 compared. So if A and B are being compared, it prints all the items in 269 A-B, then in B-A, then in A&B.</li> 270 <li>For example, here is a listing of a problem that must be corrected. 271 Note that usually there is a comment that explains what the following 272 line or lines are supposed to test. Then will come false (indicating 273 that the test failed), then the detailed error report.<pre><span style="font-size: 9pt"># Canonical decompositions (minus exclusions) must be identical across releases 274[$Decomposition_Type:Canonical - $Full_Composition_Exclusion] = [$�Decomposition_Type:Canonical - $�Full_Composition_Exclusion] 275 276false 277**** START Error Info **** 278 279In [$�Decomposition_Type:Canonical - $�Full_Composition_Exclusion], but not in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] : 280 281# Total code points: 0 282 283Not in [$�Decomposition_Type:Canonical - $�Full_Composition_Exclusion], but in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] : 2841B06 # Lo BALINESE LETTER AKARA TEDUNG 2851B08 # Lo BALINESE LETTER IKARA TEDUNG 2861B0A # Lo BALINESE LETTER UKARA TEDUNG 2871B0C # Lo BALINESE LETTER RA REPA TEDUNG 2881B0E # Lo BALINESE LETTER LA LENGA TEDUNG 2891B12 # Lo BALINESE LETTER OKARA TEDUNG 2901B3B # Mc BALINESE VOWEL SIGN RA REPA TEDUNG 2911B3D # Mc BALINESE VOWEL SIGN LA LENGA TEDUNG 2921B40..1B41 # Mc [2] BALINESE VOWEL SIGN TALING TEDUNG..BALINESE VOWEL SIGN TALING REPA TEDUNG 2931B43 # Mc BALINESE VOWEL SIGN PEPET TEDUNG 294 295# Total code points: 11 296 297In both [$�Decomposition_Type:Canonical - $�Full_Composition_Exclusion], and in [$Decomposition_Type:Canonical - $Full_Composition_Exclusion] : 29800C0..00C5 # L& [6] LATIN CAPITAL LETTER A WITH GRAVE..LATIN CAPITAL LETTER A WITH RING ABOVE 29900C7..00CF # L& [9] LATIN CAPITAL LETTER C WITH CEDILLA..LATIN CAPITAL LETTER I WITH DIAERESIS 30000D1..00D6 # L& [6] LATIN CAPITAL LETTER N WITH TILDE..LATIN CAPITAL LETTER O WITH DIAERESIS 301... 30230F7..30FA # Lo [4] KATAKANA LETTER VA..KATAKANA LETTER VO 30330FE # Lm KATAKANA VOICED ITERATION MARK 304AC00..D7A3 # Lo [11172] HANGUL SYLLABLE GA..HANGUL SYLLABLE HIH 305 306# Total code points: 12089 307**** END Error Info ****</span></pre> 308 </li> 309 </ol> 310 </li> 311 <li>Options:<ol> 312 <li>-r Print the failures as a range list.</li> 313 <li>-fxxx Use a different input file, such as -fInvariantTest.txt</li> 314 </ol> 315 </li> 316 </ol> 317 </li> 318</ol> 319<h3>6. Options</h3> 320<ol> 321 <li>If you want to see files that are opened while processing, do the 322 following:<ol> 323 <li>Run>Run</li> 324 <li>Select the Arguments tab, and add the following<ol> 325 <li>VM arguments: 326 <pre>-DSHOW_FILES</pre> 327 </li> 328 </ol> 329 </li> 330 </ol> 331 </li> 332</ol> 333<h3>5. UCA</h3> 334<ol> 335 <li>You will use com.ibm.text.UCA.Main as your main class, creating along 336 the same lines as above.</li> 337 <li>To test whether the UCA files are valid, use the 338 <span style="font-weight: 400">options (<i>note: you must also build the ICU 339 files below, since they test other aspects</i>).</span><pre>writeCollationValidityLog</pre> 340 <p>It will create a file:</p> 341 <pre><a href="file:///C:/DATA/GEN/collation/5.0.0/CheckCollationValidity.html">C:\DATA\GEN\collation\5.0.0\CheckCollationValidity.html</a></pre> 342 <ol> 343 <li>Review this file. It will list errors. Some of those are actually 344 warnings, and indicate possible problems (this is indicated in the text, 345 such as by: "These are not necessarily errors, but should be examined for 346 <i>possible</i> errors"). In those cases, the items should be reviewed to make 347 sure that there are no inadvertent problems.</li> 348 <li>If it is not so marked, it is a true error, and must be fixed.</li> 349 <li>At the end, there is section <b>11. Coverage</b>. There are two sections:<ol> 350 <li>In UCDxxx, but not in allkeys. Check this over to make sure that these 351 are all the characters that should get <b><i>implicit</i></b> weights.</li> 352 <li>In allkeys, but not in UCD. These should be <b><i>only</i></b> 353 contractions. Check them over to make sure they look right also.</li> 354 </ol></li> 355 </ol></li> 356 <li> 357 <h4><span style="font-weight: 400">To build all the charts (including for 358 the UCA), use the options: </span></h4> 359 <pre>normalizationChart caseChart scriptChart indexChart</pre> 360 <p>They will be built into</p> 361 <pre><a href="file:///C:/DATA/GEN/charts">C:\DATA\GEN\charts</a></pre> 362 <p><b>Once UCA is released, then copy those files up to the right spots in 363 the Unicode site:</b><ul> 364 <li> 365 <pre><a href="http://www.unicode.org/charts/normalization/">http://www.unicode.org/charts/normalization/</a></pre> 366 </li> 367 <li> 368 <pre><a href="http://www.unicode.org/charts/collation/">http://www.unicode.org/charts/collation/</a> </pre> 369 </li> 370 <li> 371 <pre><a href="http://www.unicode.org/charts/case/">http://www.unicode.org/charts/case/</a> </pre> 372 </li> 373 <li> 374 <pre><a href="http://www.unicode.org/charts/collation/">http://www.unicode.org/charts/collation/</a> </pre> 375 </li> 376 </ul> 377 </li> 378 <li> 379 <h4><span style="font-weight: 400">To build all the UCA files used by ICU, use the 380 option:</span></h4> 381 <pre>ICU</pre> 382 <p>They will be built into:</p> 383 <pre><a href="file:///C:/DATA/GEN/collation/5.0.0">C:\DATA\GEN\collation\5.0.0</a></pre> 384 </li> 385 <li>You should then build a set of the ICU files for the previous version, 386 if you don't have them. Use the options:<pre>version 4.1.0 ICU</pre> 387 <p>Or whatever the last version was.</li> 388 <li>Now, you will want to compare versions. The key file is 389 UCA_Rules_NoCE.txt. It contains the rules expressed in ICU format, which 390 allows for comparison across versions of UCA without spurious variations of 391 the numbers getting in the way.<ol> 392 <li>Do a Diff between the last and current versions of these files, and 393 verify that all the differences are either new characters, or were 394 authorized to be changed by the UTC.</li> 395 </ol></li> 396</ol> 397 398</body> 399 400</html>