Package org.apache.lucene.analysis.th

Class ThaiCharFilter

java.lang.Object

java.io.Reader

org.apache.lucene.analysis.CharFilter

org.apache.lucene.analysis.charfilter.BaseCharFilter

org.apache.lucene.analysis.th.ThaiCharFilter

All Implemented Interfaces:

Closeable, AutoCloseable, Readable

public class ThaiCharFilter
extends BaseCharFilter

A CharFilter that normalizes Thai text at the character stream level before tokenization.

Applying normalization before tokenization is critical for Thai, because typographic anomalies (such as two consecutive Sara E characters instead of Sara Ae, or zero-width characters) prevent the dictionary-based BreakIterator from locating word boundaries, causing multiple adjacent words to be mistakenly merged into one token.

Normalizations performed:

• Removes Zero-Width characters (U+200B ZWSP, U+200C ZWNJ, U+200D ZWJ, U+FEFF BOM)
• Replaces double Sara E (เ + เ, U+0E40 U+0E40) with Sara Ae (แ, U+0E41)
• Recomposes decomposed Sara Am (U+0E4D + U+0E32 → U+0E33, and U+0E4D + tone + U+0E32 → tone + U+0E33)
• Deduplicates consecutive repeated diacritics / tone marks (e.g. ดีี → ดี)
• Reorders misplaced tone mark before an above/below vowel to canonical order
• Replaces Lakkhangyao (ๅ, U+0E45) with Sara Aa (า, U+0E32) unless preceded by Ru (ฤ) or Lu (ฦ)

All character deletions and contractions correctly update the character offset map using BaseCharFilter.addOffCorrectMap(int, int).

Since:

10.6.0

Field Summary

Fields inherited from class org.apache.lucene.analysis.CharFilter

input

Fields inherited from class java.io.Reader

lock

Constructor Summary

Constructors

Constructor	Description
ThaiCharFilter(Reader in)	Creates a new ThaiCharFilter wrapping the given Reader.

Method Summary

Modifier and Type	Method	Description
int	read()
int	read(char[] cbuf, int off, int len)

Methods inherited from class org.apache.lucene.analysis.charfilter.BaseCharFilter

addOffCorrectMap, correct, getLastCumulativeDiff

Methods inherited from class org.apache.lucene.analysis.CharFilter

close, correctOffset

Methods inherited from class java.io.Reader

mark, markSupported, nullReader, read, read, ready, reset, skip, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ThaiCharFilter

public ThaiCharFilter(Reader in)

Creates a new ThaiCharFilter wrapping the given Reader.

Parameters:

in - the input reader

Method Details

read

public int read()
         throws IOException

Overrides:

read in class Reader

Throws:

IOException

read

public int read(char[] cbuf,
 int off,
 int len)
         throws IOException

Specified by:

read in class Reader

Throws:

IOException