com.twitter.common.text.token
Class TokenizedCharSequence

java.lang.Object
  extended by com.twitter.common.text.token.TokenizedCharSequence
All Implemented Interfaces:
CharSequence

public class TokenizedCharSequence
extends Object
implements CharSequence

Keeps the original text as well as its tokenized tokens.


Nested Class Summary
static class TokenizedCharSequence.Builder
           
static class TokenizedCharSequence.Token
           
 
Constructor Summary
protected TokenizedCharSequence(CharSequence text, List<TokenizedCharSequence.Token> tokens)
           
 
Method Summary
 char charAt(int index)
           
static TokenizedCharSequence createFrom(CharSequence text, TokenStream tokenizer)
           
static List<TokenizedCharSequence> createFromTokenGroupsIn(TokenStream stream)
           
 boolean equals(Object obj)
           
 List<TokenizedCharSequence.Token> getTokens()
          Returns all tokens.
 List<TokenizedCharSequence.Token> getTokensOf(TokenType... types)
          Returns tokens of one or more specified types.
 List<String> getTokenStrings()
          Returns all tokens as String.
 List<String> getTokenStringsOf(TokenType... types)
          Returns tokens of one or more specified types as Strings.
 int hashCode()
           
 int length()
           
 CharSequence subSequence(int fromIndex, int toIndex)
           
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

TokenizedCharSequence

protected TokenizedCharSequence(CharSequence text,
                                List<TokenizedCharSequence.Token> tokens)
Method Detail

charAt

public char charAt(int index)
Specified by:
charAt in interface CharSequence

length

public int length()
Specified by:
length in interface CharSequence

subSequence

public CharSequence subSequence(int fromIndex,
                                int toIndex)
Specified by:
subSequence in interface CharSequence

toString

public String toString()
Specified by:
toString in interface CharSequence
Overrides:
toString in class Object

equals

public boolean equals(Object obj)
Overrides:
equals in class Object

hashCode

public int hashCode()
Overrides:
hashCode in class Object

getTokens

public List<TokenizedCharSequence.Token> getTokens()
Returns all tokens.

Returns:
a list of tokens as CharBuffer objects

getTokenStrings

public List<String> getTokenStrings()
Returns all tokens as String.

Returns:
a list of tokens as String objects

getTokensOf

public List<TokenizedCharSequence.Token> getTokensOf(TokenType... types)
Returns tokens of one or more specified types.

Parameters:
types - token type(s)
Returns:
tokens of the specified type(s)

getTokenStringsOf

public List<String> getTokenStringsOf(TokenType... types)
Returns tokens of one or more specified types as Strings.

Parameters:
types - token type(s)
Returns:
list of tokens of specified type(s) as String objects

createFrom

public static final TokenizedCharSequence createFrom(CharSequence text,
                                                     TokenStream tokenizer)

createFromTokenGroupsIn

public static final List<TokenizedCharSequence> createFromTokenGroupsIn(TokenStream stream)