1.3. Feature Set

The following table indicates the type of support that RBL-JE provides for each supported language. The RBL-JE Tokenizer provides normalization, tokenization, and sentence boundary detection. The RBL-JE Analyzer provides lemma lookup (including orthographic normalization for Japanese), lemma guessing (when the lookup fails), decompounding, and supports lemma, segmentation, and many-to-one normalization user dictionaries.

^a. With the exception of Hebrew and Arabic, the tokenizer can apply Normalization Form KC (NFKC) to the tokens. For Arabic, Persian, and Urdu, see Arabic, Persian, and Urdu Token Analysis [103]. ↩

^b. For Japanese, Japanese Lemma Normalization [108] is also available. ↩

^c. See Part-of-Speech Tags [112]. ↩

^d. With the exception of Japanese, by default the analyzer returns a disambiguated analysis for the supported languages. For performance, Japanese disambiguation of is turned off by default. ↩

^e. The base form of the token to which affixes may be added. For Finnish, this is the Porter stem. ↩

^f. The Semitic root for the token (an empty string if the root cannot be determined). ↩

^g. Maps one or more tokens to a single token. ↩

^h. Simplified and Traditional Chinese. ↩

ⁱ. See Chinese Script Converter [72]. ↩

^j. For Hebrew, the tokenizer generates a lemma and a Semitic root for each token. ↩

^k. The base linguistics token filter excludes Japanese lemmas for auxiliary verbs, particles, and adverbs from the token stream. ↩

^l. Transcriptions rendered in Hiragana for Japanese tokens. ↩

^m. Bokmål and Nynorsk ↩

ⁿ. Western Farsi and Dari ↩