Package org.apache.lucene.analysis.th

Class ThaiTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.util.SegmentingTokenizerBase

org.apache.lucene.analysis.th.ThaiTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class ThaiTokenizer
extends SegmentingTokenizerBase

Tokenizer that uses BreakIterator to tokenize Thai text.

Supports an optional user dictionary (CharArraySet) for custom or domain-specific words. When a user dictionary word matches, it takes precedence over default segmentation boundaries.

WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer instead, which uses an ICU Thai BreakIterator that will always be available.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
static final boolean	DBBI_AVAILABLE	True if the JRE supports a working dictionary-based breakiterator for Thai.
static final int	DEFAULT_BUFFER_SIZE	Default buffer size in characters for Thai sentence segmentation

Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

buffer, BUFFERMAX, offset

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
ThaiTokenizer()	Creates a new ThaiTokenizer
ThaiTokenizer(int bufferSize)	Creates a new ThaiTokenizer with a custom buffer size.
ThaiTokenizer(CharArraySet userDictionary)	Creates a new ThaiTokenizer with a user dictionary.
ThaiTokenizer(CharArraySet userDictionary, int bufferSize)	Creates a new ThaiTokenizer with a user dictionary and custom buffer size.
ThaiTokenizer(AttributeFactory factory)	Creates a new ThaiTokenizer, supplying the AttributeFactory
ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary)	Creates a new ThaiTokenizer, supplying the AttributeFactory and user dictionary.
ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary, int bufferSize)	Creates a new ThaiTokenizer, supplying the AttributeFactory, user dictionary, and buffer size.

Method Summary

Modifier and Type	Method	Description
protected boolean	incrementWord()	Returns true if another word is available
protected boolean	isSafeEnd(char ch)	For sentence tokenization, these are the unambiguous break positions.
protected void	setNextSentence(int sentenceStart, int sentenceEnd)	Provides the next input sentence for analysis

Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

end, incrementToken, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader, setReaderTestPoint

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

DBBI_AVAILABLE

public static final boolean DBBI_AVAILABLE

True if the JRE supports a working dictionary-based breakiterator for Thai. If this is false, this tokenizer will not work at all!

DEFAULT_BUFFER_SIZE

public static final int DEFAULT_BUFFER_SIZE

Default buffer size in characters for Thai sentence segmentation

See Also:

• Constant Field Values

Constructor Details

ThaiTokenizer

public ThaiTokenizer()

Creates a new ThaiTokenizer

ThaiTokenizer

public ThaiTokenizer(CharArraySet userDictionary)

Creates a new ThaiTokenizer with a user dictionary.

Parameters:

userDictionary - custom dictionary of words, or null for default dictionary only

ThaiTokenizer

public ThaiTokenizer(int bufferSize)

Creates a new ThaiTokenizer with a custom buffer size.

Parameters:

bufferSize - internal buffer size in characters

ThaiTokenizer

public ThaiTokenizer(CharArraySet userDictionary,
 int bufferSize)

Creates a new ThaiTokenizer with a user dictionary and custom buffer size.

Parameters:

userDictionary - custom dictionary of words, or null for default dictionary only

bufferSize - internal buffer size in characters

ThaiTokenizer

public ThaiTokenizer(AttributeFactory factory)

Creates a new ThaiTokenizer, supplying the AttributeFactory

ThaiTokenizer

public ThaiTokenizer(AttributeFactory factory,
 CharArraySet userDictionary)

Creates a new ThaiTokenizer, supplying the AttributeFactory and user dictionary.

Parameters:

factory - AttributeFactory to use

userDictionary - custom dictionary of words, or null for default dictionary only

ThaiTokenizer

public ThaiTokenizer(AttributeFactory factory,
 CharArraySet userDictionary,
 int bufferSize)

Creates a new ThaiTokenizer, supplying the AttributeFactory, user dictionary, and buffer size.

Parameters:

factory - AttributeFactory to use

userDictionary - custom dictionary of words, or null for default dictionary only

bufferSize - internal buffer size in characters

Method Details

isSafeEnd

protected boolean isSafeEnd(char ch)

Description copied from class: SegmentingTokenizerBase

For sentence tokenization, these are the unambiguous break positions.

Overrides:

isSafeEnd in class SegmentingTokenizerBase

setNextSentence

protected void setNextSentence(int sentenceStart,
 int sentenceEnd)

Description copied from class: SegmentingTokenizerBase

Provides the next input sentence for analysis

Specified by:

setNextSentence in class SegmentingTokenizerBase

incrementWord

protected boolean incrementWord()

Description copied from class: SegmentingTokenizerBase

Returns true if another word is available

Specified by:

incrementWord in class SegmentingTokenizerBase