Character

Overview

To calculate the length and position of characters, Watson NLP internally uses the code unit (UTF-16) of character; the API returns that value (e.g., Annotation.length(), Annotation.beginIndex(), Annotation.endIndex()). Most Java libraries use code unit (UTF-16) historically, while Python3 primarily uses code point; they do not match in some cases.

Character, Code Point, and Code Unit

In Unicode,

• Character (or grapheme) is assigned a unique code point (or a unique code point sequence).

• Code point is an identification number. It is 21-bit (U+000000 – U+10FFFF). U+ prefix indicates that it is a code point.

• Code unit is a value that encoded code point to 8/16/32 bits to store/transmit Unicode text efficiently on a computer.

Unicode defines 3 major encoding forms from code point to code unit.

The following table shows an example of that. In this case, the length is 1 for all of them.

Examples

As Unicode grows, the mapping between character (grapheme), code point, and code unit has become complicated.

Python Strings

Strings are stored differently in Java and Python. It needs to be taken care of when creating test cases to estimate their memory consumptions. Python uses code point (21 bits) as a base unit to represent characters. Python strings can be stored by an array of 32 bits (There is no data type of 21 bits in Python). However this is not memory efficient. For example, most Latin characters are in the range of U+00 - U+FF (Only lower 8 bits have non-zero values; the rest of the bits are always zero). To address this issue, Python 3.3 introduced flexible string representation (PEP 393). It stores strings by an array of 8, 16, or 32 bits. The size is determined by the largest code point in the string. The following shows examples.

Empty string consumes 49 bytes. This is baseline.

>>> sys.getsizeof("")
49

Latin character a (U+0061) consumes 1 byte (8 bits) for each code point.

>>> sys.getsizeof("a")
50
>>> sys.getsizeof("aa")
51
>>> sys.getsizeof("aaa")
52

Emoji character 🎉 (U+1F389) consumes 4 bytes (32 bits) for each code point. It consumes 76 bytes as baseline.

>>> sys.getsizeof("🎉")
80
>>> sys.getsizeof("🎉🎉")
84
>>> sys.getsizeof("🎉🎉🎉")
88

Emoji character 👏🏽 (U+1F44F, U+1F3FD) consumes 8 bytes (32 bits * 2) for each. Because it consists of 2 code points.

>>> sys.getsizeof("👏🏽")
84
>>> sys.getsizeof("👏🏽👏🏽")
92
>>> sys.getsizeof("👏🏽👏🏽👏🏽")
100

If both Latin a (U+0061) and Emoji 🎉 (U+1F389), 👏🏽 (U+1F44F, U+1F3FD) characters are used in a string, it consumes 4 bytes (32 bits) for each code point.

>>> sys.getsizeof("a🎉")
84
>>> sys.getsizeof("a🎉👏🏽")
92

References

1. Unicode Codepoints
2. Unicode Emoji
3. Emojipedia
4. Unicode Normalization Forms
5. Unicode Ideographic Variant Database
6. String API in Java SDK
7. International Components for Unicode (Reference implementation of Unicode specification)
8. What's the difference between a character, a code point, a glyph and a grapheme?