--- layout: default title: UText nav_order: 4 parent: Chars and Strings --- # UText ## Overview UText is a text abstraction facility for ICU The intent is to make it possible to extend ICU to work with text data that is in formats above and beyond those that are native to ICU. UText directly supports text in these formats: 1. UTF-8 (`char*`) strings 2. UTF-16 (`UChar*` or `UnicodeString`) strings 3. `Replaceable` The ICU services that can accept UText based input are: 1. Regular Expressions 2. Break Iteration Examples of text formats that UText could be extended to support: 1. UTF-32 format. 2. Text that is stored in discontiguous chunks in memory, or in application-specific representations. 3. Text that is in a non-Unicode code page If ICU does not directly support a desired text format, it is possible for application developers themselves to extend UText, and in that way gain the ability to use their text with ICU. ## Using UText There are three fairly distinct classes of use of UText. These are: 1. **Simple wrapping of existing text.** Application text data exists in a format that is already supported by UText (such as UTF-8). The application opens a UText on the data, and then passes the UText to an ICU service for analysis/processing. Most use of UText from applications will follow this simple pattern. Only a very few UText APIs and only a few lines of code are required. 2. **Accessing the underlying text.** UText provides APIs for iterating over the text in various ways, and for fetching individual code points from the text. These functions will probably be used primarily from within ICU, in the implementation of services that can accept input in the form of a UText. While applications are certainly free to use these text access functions if necessary, there may often be no need. 3. **UText support for new text storage formats.** If an application has text data stored in a format that is not directly supported by ICU, extending UText to support that format will provide the ability to conveniently use those ICU services that support UText. Extending UText to a new format is accomplished by implementing a well defined set of *Text Provider Functions* for that format. ## UText compared with CharacterIterator CharacterIterator is an abstract base class that defines a protocol for accessing characters in a text-storage object. This class has methods for iterating forward and backward over Unicode characters to return either the individual Unicode characters or their corresponding index values. UText and CharacterIterator both provide an abstraction for accessing text while hiding details of the actual storage format. UText is the more flexible of the two, however, with these advantages: 1. UText can conveniently operate on text stored in formats other than UTF-16. 2. UText includes functions for modifying or editing the text. 3. UText is more efficient. When iterating over a range of text using the CharacterIterator API, a function call is required for every character. With UText, iterating to the next character is usually done with small amount of inline code. At this time, more ICU services support CharacterIterator than UText. ICU services that can operate on text represented by a CharacterIterator are 1. Normalizer 2. Break Iteration 3. String Search 4. Collation Element Iteration ## Example: Counting the Words in a UTF-8 String Here is a function that uses UText and an ICU break iterator to count the number of words in a nul-terminated UTF-8 string. The use of UText only adds two lines of code over what a similar function operating on normal UTF-16 strings would require. ```c #include "unicode/utypes.h" #include "unicode/ubrk.h" #include "unicode/utext.h" int countWords(const char *utf8String) { UText *ut = NULL; UBreakIterator *bi = NULL; int wordCount = 0; UErrorCode status = U_ZERO_ERROR; ut = utext_openUTF8(ut, utf8String, -1, &status); bi = ubrk_open(UBRK_WORD, "en_us", NULL, 0, &status); ubrk_setUText(bi, ut, &status); while (ubrk_next(bi) != UBRK_DONE) { if (ubrk_getRuleStatus(bi) != UBRK_WORD_NONE) { /* Count only words and numbers, not spaces or punctuation */ wordCount++; } } utext_close(ut); ubrk_close(bi); assert(U_SUCCESS(status)); return wordCount; } ``` ## UText API Functions The UText API is declared in the ICU header file [utext.h](https://github.com/unicode-org/icu/blob/master/icu4c/source/common/unicode/utext.h) ### Opening and Closing. Normal usage of UText by an application consists of opening a UText to wrap some existing text, then passing the UText to ICU functions for processing. For this kind of usage, all that is needed is the appropriate UText open and close functions. | Function | Description | |--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | `uext_openUChars` | Open a UText over a standard ICU (`UChar *`) string. The string consists of a UTF-16 array in memory, either nul terminated or with an explicit length. | | `utext_openUnicodeString` | Open a UText over an instance of an ICU C++ `UnicodeString`. | | `Utext_openConstUnicodeString` | Open a UText over a read-only `UnicodeString`. Disallows UText APIs that modify the text. | | `utext_openReplaceable` | Open a UText over an instance of an ICU C++ `Replaceable`. | | `utext_openUTF8` | Open a UText over a UTF-8 encoded C string. May be either Nul terminated or have an explicit length. | | `utext_close` | Close an open UText. Frees any allocated memory; required to prevent memory leaks. | Here are some suggestions and techniques for efficient use of UText. #### Minimizing Heap Usage Utext's open functions include features to allow applications to minimize the number of heap memory allocations that will be needed. Specifically, 1. UText structs may declared as local variables, that is, they may be stack allocated rather than heap allocated. 2. Existing UText structs may be reused to refer to new text, avoiding the need to allocate and initialize a new UText instance. Minimizing heap allocations is important in code that has critical performance requirements, and is doubly important for code that must scale well in multithreaded, multiprocessor environments. #### Stack Allocation Here is code for stack-allocating a UText: ```c UText mytext = UTEXT_INITIALIZER; utext_openUChars(&myText, ... ``` The first parameter to all `utext_open` functions is a pointer to a UText. If it is non-null, the supplied UText will be used; if it is null, a new UText will be heap allocated. Stack allocated UText objects *must *be initialized with `UTEXT_INITIALIZER`. An uninitialized instance will fail to open. #### Heap Allocation Here is code for creating a heap allocated UText: ```c UText *mytext = utext_openUChars(NULL, ... ``` This is slightly smaller and more convenient to write than the stack allocated code, and there is no reason not to use heap allocated UText objects in the vast majority of code that does not have extreme performance constraints. #### Reuse To reuse an existing UText, simply pass it as the first parameter to any of the UText open functions. There is no need to close the UText first, and it may actually be more efficient not to close it first. Here is an example of a function that iterates over an array of UTF-8 strings, wrapping each in a UText and passing it off to another function. On the first time through the loop the utext open function will heap allocate a UText. On each subsequent iterations the existing UText will be reused. ```c #include "unicode/utypes.h" #include "unicode/utext.h" void f(char **strings, int numStrings) { UText *ut = NULL; UErrorCode status; int i; for (i=0; ind through the Nth positions of a multi byte or multi-code-unit character, the index will be normalized back to the first or lowest index. 4. An input index that is greater than the length of the text will be set to refer to the end of the string, and will not generate out of bounds error. This is similar to the indexing behavior in the UnicodeString class. 5. Iteration uses post-increment and pre-decrement conventions. That is, `utext_next32()` fetches the code point at the current index, then leaves the index pointing at the next character. Here are the functions for accessing the actual text data represented by a UText. The primary use of these functions will be in the implementation of ICU services that accept input in the form of a UText, although application code may also use them if the need arises. For more detailed descriptions of each, see the API reference. | Function | Description | |-------------------------|------------------------------------------------------------------------------------------------------------| | `utext_nativeLength` | Get the length of the text string in terms of the underlying native storage – bytes for UTF-8, for example | | `utext_isLengthExpensive` | Indicate whether determining the length of the string would require scanning the string. | | `utext_char32At` | Get the code point at the specified index. | | `utext_current32` | Get the code point at the current iteration position. Does not advance the position. | | `utext_next32` | Get the next code point, iterating forwards. | | `utext_previous32` | Get the previous code point, iterating backwards. | | `utext_next32From` | Begin a forwards iteration at a specified index. | | `utext_previous32From` | Begin a reverse iteration at a specified index. | | `utext_getNativeIndex` | Get the current iteration index. | | `utext_setNativeIndex` | Set the iteration index. | | `utext_moveIndex32` | Move the current index forwards or backwards by the specified number of code points. | | `utext_extract` | Retrieve a range of text, placing it into a UTF-16 buffer. | | `UTEXT_NEXT32` | inline (high performance) version of `utext_next32` | | `UTEXT_PREVIOUS32` | inline (high performance) version of `utext_previous32` | ## Modifying the Text UText provides API for modifying or editing the text. | Function | Description | |---------------------|----------------------------------------------------------------------------------------------------| | `utext_replace` | Replace a range of the original text with a replacement string. | | `utext_copy` | Copy or Move a range of the text to a new position. | | `utext_isWritable` | Test whether a UText supports writing operations. | | `utext_hasMetaData` | Test whether the text includes metadata. See the class `Replaceable` for more information on meta data.. | Certain conventions must be followed when modifying text using these functions: 1. Not all types of UText can support modifying the data. Code working with UText instances of unknown origin should check `utext_isWritable()` first, and be prepared to deal with failures. 2. There must be only one UText open onto the underlying string that is being modified. (Strings that are not being modified can be the target of any number of UTexts at the same time) The existence of a second UText that refers to a string that is being modified is not a situation that is detected by the implementation. The application code must be structured to avoid the situation. #### Cloning UText instances may be cloned. The clone function, ```c UText * utext_clone(UText *dest, const UText *src, UBool deep, UBool readOnly, UErrorCode *status) ``` behaves very much like a UText open functions, with the source of the text being another UText rather than some other form of a string. A *shallow* clone creates a new UText that maintains its own iteration state, but does not clone the underlying text itself. A *deep* clone copies the underlying text in addition to the UText state. This would be appropriate if you wished to modify the text without the changes being reflected back to the original source string. Not all text providers support deep clone, so checking for error status returns from `utext_clone()` is importatnt. #### Thread Safety UText follows the usual ICU conventions for thread safety: concurrent calls to functions accessing the same non-const UText is not supported. If concurrent access to the text is required, the UText can be cloned, allowing each thread access via a separate UText. So long as the underlying text is not being modified, a shallow clone is sufficient. ## Text Providers A *text provider* is a set of functions that let UText support a specific text storage format. ICU includes several UText text provider implementations, and applications can provide additional ones if needed. To implement a new UText text provider, it is necessary to have an understanding of how UText is designed. Underneath the covers, UText is a struct that includes: 1. A pointer to a *Text Chunk*, which is a UTF-16 buffer containing a section (or all) of the text being referenced. For text sources whose native format is UTF-16, the chunk description can refer directly to the original text data. For non-UTF-16 sources, the chunk will refer to a side buffer containing some range of the text that has been converted to UTF-16 format. 2. The iteration position, as a UTF-16 offset within the chunk. If a text access function (one of those described above, in the previous section) can do its thing based on the information maintained in the UText struct, it will. If not, it will call out to one of the provider functions (below) to do the work, or to update the UText. The best way to really understand what is required of a UText provider is to study the implementations that are included with ICU, and to borrow as much as possible. Here is the list of text provider functions. | Function | Description | |----------------------------|----------------------------------------------------------------------------------------------------| | `UTextAccess` | Set up the Text Chunk associated with this UText so that it includes a requested index position. | | `UTextNativeLength` | Return the full length of the text. | | `UTextClone` | Clone the UText. | | `UTextExtract` | Extract a range of text into a caller-supplied buffer | | `UTextReplace` | Replace a range of text with a caller-supplied replacement. May expand or shrink the overall text. | | `UTextCopy` | Move or copy a range of text to a new position. | | `UTextMapOffsetToNative` | Within the current text chunk, translate a UTF-16 buffer offset to an absolute native index. | | `UTextMapNativeIndexToUTF16` | Translate an absolute native index to a UTF-16 buffer offset within the current text. | | `UTextClose` | Provider specific close. Free storage as required. | Not every provider type requires all of the functions. If the text type is read-only, no implementation for Replace or Copy is required. If the text is in UTF-16 format, no implementation of the native to UTF-16 index conversions is required. To fully understand what is required to support a new string type with UText, it will be necessary to study both the provider function declarations from [utext.h](https://github.com/unicode-org/icu/blob/master/icu4c/source/common/unicode/utext.h) and the existing text provider implementations in [utext.cpp](https://github.com/unicode-org/icu/blob/master/icu4c/source/common/utext.cpp).