org.apache.lucene.analysis.standard

Class StandardTokenizer

public class StandardTokenizer extends Tokenizer implements StandardTokenizerConstants

A grammar-based tokenizer constructed with JavaCC.

This should be a good tokenizer for most European-language documents:

Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

Field Summary
Token	jj_nt
Token	token
StandardTokenizerTokenManager	token_source

Constructor Summary
	StandardTokenizer(Reader reader) Constructs a tokenizer for this Reader.
	StandardTokenizer(CharStream stream)
	StandardTokenizer(StandardTokenizerTokenManager tm)

Method Summary
void	disable_tracing()
void	enable_tracing()
ParseException	generateParseException()
Token	getNextToken()
Token	getToken(int index)
Token	next() Returns the next token in the stream, or null at EOS.
void	ReInit(CharStream stream)
void	ReInit(StandardTokenizerTokenManager tm)