Class TokenFilter

  extended by org.apache.lucene.util.AttributeSource
      extended by com.twitter.common.text.token.TokenStream
          extended by com.twitter.common.text.token.TokenProcessor
              extended by com.twitter.common.text.filter.TokenFilter
Direct Known Subclasses:

public abstract class TokenFilter
extends TokenProcessor

Filters out tokens from a given TokenStream.

Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
Constructor Summary
TokenFilter(TokenStream inputStream)
Method Summary
abstract  boolean acceptToken()
          Overwrite this method to control which tokens are filtered out.
 boolean incrementToken()
          Consumers call this method to advance the stream to the next token.
Methods inherited from class com.twitter.common.text.token.TokenProcessor
getInputStream, getInstanceOf, reset
Methods inherited from class com.twitter.common.text.token.TokenStream
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail


public TokenFilter(TokenStream inputStream)
Method Detail


public boolean incrementToken()
Description copied from class: TokenStream
Consumers call this method to advance the stream to the next token.

Specified by:
incrementToken in class TokenStream
false for end of stream; true otherwise


public abstract boolean acceptToken()
Overwrite this method to control which tokens are filtered out.

true to accept the current token, false otherwise