com.twitter.common.text.filter
Class TokenTypeFilter

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by com.twitter.common.text.token.TokenStream
          extended by com.twitter.common.text.token.TokenProcessor
              extended by com.twitter.common.text.filter.TokenFilter
                  extended by com.twitter.common.text.filter.TokenTypeFilter
Direct Known Subclasses:
PunctuationFilter

public class TokenTypeFilter
extends TokenFilter

Either filters out or retains tokens of a specified type(s). If the mode is Mode.ACCEPT, tokens are retained. If the mode is Mode.REJECT, tokens are filtered out.


Nested Class Summary
static class TokenTypeFilter.Builder
           
static class TokenTypeFilter.Mode
           
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
 
Constructor Summary
protected TokenTypeFilter(TokenStream inputStream)
           
 
Method Summary
 boolean acceptToken()
          Overwrite this method to control which tokens are filtered out.
protected  void setMode(TokenTypeFilter.Mode mode)
           
protected  void setTypesToFilter(TokenType... types)
           
 
Methods inherited from class com.twitter.common.text.filter.TokenFilter
incrementToken
 
Methods inherited from class com.twitter.common.text.token.TokenProcessor
getInputStream, getInstanceOf, reset
 
Methods inherited from class com.twitter.common.text.token.TokenStream
toStringList
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

TokenTypeFilter

protected TokenTypeFilter(TokenStream inputStream)
Method Detail

setTypesToFilter

protected void setTypesToFilter(TokenType... types)

setMode

protected void setMode(TokenTypeFilter.Mode mode)

acceptToken

public boolean acceptToken()
Description copied from class: TokenFilter
Overwrite this method to control which tokens are filtered out.

Specified by:
acceptToken in class TokenFilter
Returns:
true to accept the current token, false otherwise