| Unit Resource Filter Rules |
Unit Resource Size |
Commentary |
Result |
-/*/*
+/*/digital
-/*/digital/*/dnam
-/durationUnits
-/units
-/unitsNarrow
| 77 KiB |
First, remove all unit types. Then, add back digital units across all unit
widths. Then, remove display names from digital units. Then, remove duration
unit patterns and long and narrow forms.
|
Digital units in short form are included; all other units are removed.
|
-/durationUnits
-/units
-/unitsNarrow
-/*/*
+/*/digital
-/*/digital/*/dnam
| 125 KiB |
First, remove duration unit patterns and long and narrow forms. Then, remove
all unit types. Then, add back digital units across all unit widths. Then,
remove display names from digital units.
|
Digital units are included in all widths; all other units are removed.
|
-/*/*
+/*/digital
-/*/*/*/dnam
-/durationUnits
-/units
-/unitsNarrow
| 191 KiB |
First, remove all unit types. Then, add back digital units across all unit
widths. Then, remove display names from all units. Then, remove duration unit
patterns and long and narrow forms.
|
Digital units in short form are included, as is the tree structure
for all other units, even though the other units have no real data.
|
By design, empty tree structure is retained in the unit bundle. This is
because there are numerous instances in ICU data where the presence of an
empty tree carries meaning. However, it means that you must be careful when
building resource filter rules in order to achieve the optimal data bundle
size.
Using the `-v` option in genrb (described above) is helpful when debugging
these types of issues.
## Other Features of the ICU Data Build Tool
While data filtering is the primary reason the ICU Data Build Tool was
developed, there are there are additional use cases.
### Running Data Build without Configure/Make
You can build the dat file outside of the ICU build system by directly
invoking the Python icutools.databuilder. Run the following command to see the
help text for the CLI tool:
$ PYTHONPATH=path/to/icu4c/source/python python3 -m icutools.databuilder --help
### Collation UCAData
For using collation (sorting and searching) in any language, the "root"
collation data file must be included. It provides the Unicode CLDR default
sort order for all code points, and forms the basis for language-specific
tailorings as well as for custom collators built at runtime.
There are two versions of the root collation data file:
- ucadata-unihan.txt (compiled size: 511 KiB)
- ucadata-implicithan.txt (compiled size: 178 KiB)
The unihan version sorts Han characters in radical-stroke order according to
Unicode, which is a somewhat useful default sort order, especially for use
with non-CJK languages. The implicithan version sorts Han characters in the
order of their Unicode assignment, which is similar to radical-stroke order
for common characters but arbitrary for others. For more information, see
[UTS #10 §10.1.3](https://www.unicode.org/reports/tr10/#Implicit_Weights).
By default, the unihan version is used. The unihan version of the data file
is much larger than that for implicithan, so if you need collation but also
small data, then you may want to select the implicithan version. To use the
implicithan version, put the following setting in your *filters.json* file:
{
"collationUCAData": "implicithan"
}
### Disable Pool Bundle
By default, ICU uses a "pool bundle" to store strings shared between locales.
This saves space and is recommended for most users. However, when developing
a system where locale data files may be added "on the fly" and not included in
the original ICU distribution, those additional data files may not be able to
use a pool bundle due to name collisions with the existing pool bundle.
To disable the pool bundle in the current ICU build, put the following setting
in your *filters.json* file:
{
"usePoolBundle": false
}
### File Substitution
Using the configuration file, you can perform whole-file substitutions. For
example, suppose you want to replace the transliteration rules for
*Zawgyi_my*. You could create a directory called `my_icu_substitutions`
containing your new `Zawgyi_my.txt` rule file, and then put this in your
configuration file:
fileReplacements: {
directory: "/path/to/my_icu_substitutions"
replacements: [
{
src: "Zawgyi_my.txt"
dest: "translit/Zawgyi_my.txt"
},
"misc/dayPeriods.txt"
]
}
`directory` should either be an absolute path, or a path starting with one of
the following, and it should not contain a trailing slash:
- "$SRC" for the *icu4c/source/data* directory in the source tree
- "$FILTERS" for the directory containing filters.json
- "$CWD" for your current working directory
When the entry in the `replacements` array is an object, the `src` and `dest`
fields indicate, for each file in the source directory (`src`), what file in
the ICU hierarchy it should replace (`dest`). When the entry is a string, the
same relative path is used for both `src` and `dest`.
Whole-file substitution happens before all other filters are applied.
The `icudt64l.dat` file is 27 MiB uncompressed and 11 MiB gzipped. This file
size is too large for certain use cases, such as bundling the data file into a
smartphone app or an embedded device. This is something the ICU Data Build
Tool aims to solve.
## ICU Data Configuration File
The ICU Data Build Tool enables you to write a configuration file that
specifies what features and locales to include in a custom data bundle.
The configuration file may be written in either [JSON](http://json.org/) or
[Hjson](https://hjson.org/). To build ICU4C with custom data, set the
`ICU_DATA_FILTER_FILE` environment variable when running `runConfigureICU` on
Unix or when building the data package on Windows. For example:
ICU_DATA_FILTER_FILE=filters.json path/to/icu4c/source/runConfigureICU Linux
**Important:** You *must* have the data sources in order to use the ICU Data
Build Tool. Check for the file icu4c/source/data/locales/root.txt. If that file
is missing, you need to download "icu4c-\*-data.zip", delete the old
icu4c/source/data directory, and replace it with the data directory from the zip
file. If there is a \*.dat file in icu4c/source/data/in, that file will be used
even if you gave ICU custom filter rules.
In order to use Hjson syntax, the `hjson` pip module must be installed on
your system. You should also consider installing the `jsonschema` module to
print messages when errors are found in your config file.
$ pip3 install --user hjson jsonschema
To build ICU4J with custom data, you must first build ICU4C with custom data
and then generate the JAR file. For more information on building ICU4J, read the
[ICU4J Readme](../icu4j/).
### Locale Slicing
The simplest way to slice ICU data is by locale. The ICU Data Build Tool
makes it easy to select your desired locales to suit a number of use cases.
#### Filtering by Language Only
Here is a *filters.json* file that builds ICU data with support for English,
Chinese, and German, including *all* script and regional variants for those
languages:
{
"localeFilter": {
"filterType": "language",
"includelist": [
"en",
"de",
"zh"
]
}
}
The *filterType* "language" only supports slicing by entire languages.
##### Terminology: Includelist, Excludelist, Whitelist, Blacklist
Prior to ICU 68, use `"whitelist"` and `"blacklist"` instead of `"includelist"`
and `"excludelist"`, respectively. ICU 68 allows all four terms.
#### Filtering by Locale
For more control, use *filterType* "locale". Here is a *filters.hjson* file that
includes the same three languages as above, including regional variants, but
only the default script (e.g., Simplified Han for Chinese):
localeFilter: {
filterType: locale
includelist: [
en
de
zh
]
}
*If using ICU 67 or earlier, see note above regarding allowed keywords.*
#### Adding Script Variants (includeScripts = true)
You may set the *includeScripts* option to true to include all scripts for a
language while using *filterType* "locale". This results in behavior similar
to *filterType* "language". In the following JSON example, all scripts for
Chinese are included:
{
"localeFilter": {
"filterType": "locale",
"includeScripts": true,
"includelist": [
"en",
"de",
"zh"
]
}
}
*If using ICU 67 or earlier, see note above regarding allowed keywords.*
If you wish to explicitly list the scripts, you may put the script code in the
locale tag in the whitelist, and you do not need the *includeScripts* option
enabled. For example, in Hjson, to include Han Traditional ***but not Han
Simplified***:
localeFilter: {
filterType: locale
includelist: [
en
de
zh_Hant
]
}
*If using ICU 67 or earlier, see note above regarding allowed keywords.*
**Note:** the option *includeScripts* is only supported at the language level;
i.e., in order to include all scripts for a particular language, you must
specify the language alone, without a region tag.
#### Removing Regional Variants (includeChildren = false)
If you wish to enumerate exactly which regional variants you wish to support,
you may use *filterType* "locale" with the *includeChildren* setting turned to
false. The following *filters.hjson* file includes English (US), English
(UK), German (Germany), and Chinese (China, Han Simplified), as well as their
dependencies, *but not* other regional variants like English (Australia),
German (Switzerland), or Chinese (Taiwan, Han Traditional):
localeFilter: {
filterType: locale
includeChildren: false
includelist: [
en_US
en_GB
de_DE
zh_CN
]
}
*If using ICU 67 or earlier, see note above regarding allowed keywords.*
Including dependencies, the above filter would include the following data files:
- root.txt
- en.txt
- en_US.txt
- en_001.txt
- en_GB.txt
- de.txt
- de_DE.txt
- zh.txt
- zh_Hans.txt
- zh_Hans_CN.txt
- zh_CN.txt
### File Slicing (coarse-grained features)
ICU provides a lot of features, of which you probably need only a small subset
for your application. Feature slicing is a powerful way to prune out data for
any features you are not using.
***CAUTION:*** When slicing by features, you must manually include all
dependencies. For example, if you are formatting dates, you must include not
only the date formatting data but also the number formatting data, since dates
contain numbers. Expect to spend a fair bit of time debugging your feature
filter to get it to work the way you expect it to.
The data for many ICU features live in individual files. The ICU Data Build
Tool puts similar *types* of files into categories. The following table
summarizes the ICU data files and their corresponding features and categories:
| Feature | Category ID(s) | Data Files