StreamTokenizer

所谓token就是一个一个的单词，数字，符号，可以作为整体一次读入。而StreamTokennizer能够识别这些token.

import java.io.IOException;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class tokenNizer {

	/**
	 * @param args
	 * @throws IOException
	 */
	public static void main(String[] args) throws IOException {
		// TODO Auto-generated method stub
		StreamTokenizer tokenizer = new StreamTokenizer(new StringReader(
				"Mary had 1 little lamb..."));

		while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {

			if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
				System.out.println(tokenizer.sval);
			} else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
				System.out.println(tokenizer.nval);
			} else if (tokenizer.ttype == StreamTokenizer.TT_EOL) {
				System.out.println();
			}

		}

	}

}

输出：

Mary
had
1.0
little
lamb...

原文：http://tutorials.jenkov.com/java-io/streamtokenizer.html