Package org.apache.lucene.analysis.th

Class ThaiNormalizer

java.lang.Object
org.apache.lucene.analysis.th.ThaiNormalizer
public class ThaiNormalizer extends Object
Normalizer for Thai.

Normalizes Thai text to handle common orthographic variations, typos, and encoding anomalies:

  • Removes zero-width characters (ZWSP U+200B, ZWNJ U+200C).
  • Replaces double Sara E (U+0E40 U+0E40) with Sara Ae (U+0E41).
  • Recomposes decomposed Sara Am (Nikhahit U+0E4D + optional tone mark + Sara Aa U+0E32) into canonical Sara Am (U+0E33).
  • Normalizes Lakkhangyao (U+0E45) to Sara Aa (U+0E32) unless preceded by Ru (U+0E24) or Lu (U+0E26).
  • Reorders misplaced tone marks (U+0E48-U+0E4B) or Thanthakhat (U+0E4C) with above/below vowels (U+0E31, U+0E34-U+0E3A, U+0E47) to canonical Unicode order (vowel before tone mark).
  • Reorders follow vowels (e.g. Sara Aa U+0E32, Sara Am U+0E33) and misplaced trailing tone marks.
  • Deduplicates repeated vowels, tone marks, and diacritics.
  • Removes dangling combining marks at the start of a token.

  • Constructor Details

    • ThaiNormalizer

      public ThaiNormalizer()

  • Method Details

    • normalize

      public int normalize(char[] s, int len)
      Normalize an input buffer of Thai text.
      Parameters:
      s - input buffer
      len - length of input buffer
      Returns:
      length of input buffer after normalization