Package com.twitter.common.text.token

Class Summary
LuceneTokenizer2TokenStreamWrapper Converts a Lucene Tokenizer instance into TokenStream instance.
TokenGroupStream A TokenStream used to access the member of a token group created by TokenGrouper.
TokenizedCharSequence Keeps the original text as well as its tokenized tokens.
TokenizedCharSequenceStream Reproduces the result of tokenization if an input text is an instance of TokenizedCharSequence.
TokenProcessor A TokenStream whose input is another TokenStream.
TokenStream Abstraction to enumerate a sequence of tokens.
TokenStream2LuceneTokenizerWrapper Converts a TokenStream instance to a Lucene's Tokenizer instance.
TokenStreamAggregator Aggregates the outputs of multiple TokenStreams into a single TokenStream.
TokenStreamDuplicator Creates a duplicate a TokenStream.