Class TokenGroupStream

  extended by org.apache.lucene.util.AttributeSource
      extended by com.twitter.common.text.token.TokenStream
          extended by com.twitter.common.text.token.TokenGroupStream

public class TokenGroupStream
extends TokenStream

A TokenStream used to access the member of a token group created by TokenGrouper.

Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
Constructor Summary
TokenGroupStream(org.apache.lucene.util.AttributeSource source, List<org.apache.lucene.util.AttributeSource.State> states)
Method Summary
 void accessTokenAt(int index)
 boolean incrementToken()
          Consumers call this method to advance the stream to the next token.
 void reset(CharSequence input)
          Resets this token group stream.
 int size()
Methods inherited from class com.twitter.common.text.token.TokenStream
getInstanceOf, toStringList
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail


public TokenGroupStream(org.apache.lucene.util.AttributeSource source,
                        List<org.apache.lucene.util.AttributeSource.State> states)
Method Detail


public boolean incrementToken()
Description copied from class: TokenStream
Consumers call this method to advance the stream to the next token.

Specified by:
incrementToken in class TokenStream
false for end of stream; true otherwise


public void reset(CharSequence input)
Resets this token group stream. input is discarded.

Specified by:
reset in class TokenStream
input - new text to parse.


public int size()


public void accessTokenAt(int index)