Class TextTokenizer
The tokenizer knows about four different character classes: regular word characters, whitespace characters, sentence delimiters and separator characters. Tokens can consist of
- sequences of word characters and sentence delimiters where the last character is a word character,
- sentence delimiter characters (if they do not precede a word character),
- sequences of whitespace characters,
- and individual separator characters.
The character classes are completely user definable. By default, whitespace characters are the Unicode whitespace characters. All other characters are word characters. The two separator classes are empty by default. The different classes may have non-empty intersections. When determining the class of a character, the user defined classes are considered in the following order: end-of-sentence delimiter before other separators before whitespace before word characters. That is, if a character is defined to be both a separator and a whitespace character, it will be considered to be a separator.
By default, the tokenizer will return all tokens, including whitespace. That is, appending the sequence of tokens will recover the original input text. This behavior can be changed so that whitespace and/or separator tokens are skipped.
A tokenizer provides a standard iterator interface similar to StringTokenizer. The validity of the iterator can be queried with hasNext(), and
the next token can be queried with nextToken(). In addition,
getNextTokenType() returns the type of the token as an integer. NB that you need to
call getNextTokenType() before calling nextToken(), since calling
nextToken() will advance the iterator.
- Version:
- $Id: TextTokenizer.java,v 1.1 2002/09/30 19:09:09 goetz Exp $
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final intstatic final intSentence delimiter character/word type.private char[]private booleanprivate intprivate intprivate intprivate intstatic final intSeparator character/word type.private char[]private booleanprivate booleanprivate final char[]static final intWord character/word type.private char[]private char[]static final intWhitespace character/word type. -
Constructor Summary
ConstructorsConstructorDescriptionTextTokenizer(String string) Construct a tokenizer from a Java string.TextTokenizer(CharArrayString string) Construct a tokenizer from a CharArrayString. -
Method Summary
Modifier and TypeMethodDescriptionvoidaddSeparators(String chars) Add to the set of separator characters.voidaddToEndOfSentenceChars(String chars) Add to the set of sentence delimiters.private char[]addToSortedList(String s, char[] list) Add the characters insto the sorted array of characters inlist, returning a new, sorted array.voidaddWhitespaceChars(String chars) Add to the set of whitespace characters.voidaddWordChars(String chars) Add to the set of word characters.private booleanCompute the next token.intgetCharType(char c) Get the type of an individual character.intGet the type of the token returned by the next call tonextToken().booleanhasNext()private char[]voidsetEndOfSentenceChars(String chars) Set the set of sentence delimiters.voidsetSeparators(String chars) Set the set of separator characters.voidsetShowSeparators(boolean b) Set the flag for showing separator tokens.voidsetShowWhitespace(boolean b) Set the flag for showing whitespace tokens.voidsetWhitespaceChars(String chars) Set the set of whitespace characters (in addition to the Unicode whitespace chars).voidsetWordChars(String chars) Set the set of word characters.
-
Field Details
-
EOS
public static final int EOSSentence delimiter character/word type.- See Also:
-
SEP
public static final int SEPSeparator character/word type.- See Also:
-
WSP
public static final int WSPWhitespace character/word type.- See Also:
-
WCH
public static final int WCHWord character/word type.- See Also:
-
text
private final char[] text -
end
private final int end -
pos
private int pos -
eosDels
private char[] eosDels -
separators
private char[] separators -
whitespace
private char[] whitespace -
wordChars
private char[] wordChars -
nextTokenStart
private int nextTokenStart -
nextTokenEnd
private int nextTokenEnd -
nextTokenType
private int nextTokenType -
nextComputed
private boolean nextComputed -
showWhitespace
private boolean showWhitespace -
showSeparators
private boolean showSeparators
-
-
Constructor Details
-
TextTokenizer
Construct a tokenizer from a CharArrayString.- Parameters:
string- The string to tokenize.
-
TextTokenizer
Construct a tokenizer from a Java string.- Parameters:
string- -
-
-
Method Details
-
setShowWhitespace
public void setShowWhitespace(boolean b) Set the flag for showing whitespace tokens.- Parameters:
b- -
-
setShowSeparators
public void setShowSeparators(boolean b) Set the flag for showing separator tokens.- Parameters:
b- -
-
setEndOfSentenceChars
Set the set of sentence delimiters.- Parameters:
chars- -
-
addToEndOfSentenceChars
Add to the set of sentence delimiters.- Parameters:
chars- -
-
setSeparators
Set the set of separator characters.- Parameters:
chars- -
-
addSeparators
Add to the set of separator characters.- Parameters:
chars- -
-
setWhitespaceChars
Set the set of whitespace characters (in addition to the Unicode whitespace chars).- Parameters:
chars- -
-
addWhitespaceChars
Add to the set of whitespace characters.- Parameters:
chars- -
-
setWordChars
Set the set of word characters.- Parameters:
chars- -
-
addWordChars
Add to the set of word characters.- Parameters:
chars- -
-
getNextTokenType
public int getNextTokenType()Get the type of the token returned by the next call tonextToken().- Returns:
- The token type, or
-1if there is no next token.
-
hasNext
public boolean hasNext()- Returns:
trueiff there is a next token.
-
nextToken
- Returns:
- the next token.
-
computeNextToken
private boolean computeNextToken()Compute the next token. -
getCharType
public int getCharType(char c) Get the type of an individual character.- Parameters:
c- -- Returns:
- -
-
addToSortedList
Add the characters insto the sorted array of characters inlist, returning a new, sorted array. -
makeSortedList
-