本文整理匯總了Java中org.eclipse.jdt.internal.core.util.PublicScanner類的典型用法代碼示例。如果您正苦於以下問題：Java PublicScanner類的具體用法？Java PublicScanner怎麽用？Java PublicScanner使用的例子？那麽恭喜您, 這裏精選的類代碼示例或許可以為您提供幫助。
PublicScanner類屬於org.eclipse.jdt.internal.core.util包，在下文中一共展示了PublicScanner類的18個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於我們的係統推薦出更棒的Java代碼示例。
示例1: getIdentifierType
點讚 3
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
public String getIdentifierType(final PublicScanner scanner) {
final int startPos = scanner.getCurrentTokenStartPosition();
final int endPos = scanner.getCurrentTokenEndPosition();
final Range tokenRange = Range
.closedOpen(startPos, endPos);
final String tokenName = scanner.getCurrentTokenString();
// TODO: Find the tightest of all
if (isInSet(tokenName, tokenRange, variables)) {
return IDENTIFIER_PREFIX + "_VAR";
} else if (isInSet(tokenName, tokenRange, methods)) {
return IDENTIFIER_PREFIX + "_METHOD";
} else if (isInSet(tokenName, tokenRange, types)) {
return IDENTIFIER_PREFIX + "_TYPE";
}
return IDENTIFIER_PREFIX + "_UNK";
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:19，
示例2: getTokenListFromCode
點讚 3
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public List getTokenListFromCode(final char[] code) {
final List tokens = Lists.newArrayList();
tokens.add(new FullToken(SENTENCE_START, SENTENCE_START));
final PublicScanner scanner = prepareScanner(code);
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
for (final String cToken : getConvertedToken(scanner, token)) {
tokens.add(new FullToken(cToken, ""));
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
tokens.add(new FullToken(SENTENCE_END, SENTENCE_END));
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:22，
示例3: tokenListFromCode
點讚 3
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public List tokenListFromCode(final char[] code) {
final List tokens = Lists.newArrayList();
tokens.add(SENTENCE_START);
final PublicScanner scanner = prepareScanner(code);
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
tokens.addAll(getConvertedToken(scanner, token));
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
tokens.add(SENTENCE_END);
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:20，
示例4: tokenListWithPos
點讚 3
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public SortedMap tokenListWithPos(final char[] code) {
final PublicScanner scanner = prepareScanner();
final SortedMap tokens = Maps.newTreeMap();
tokens.put(-1, SENTENCE_START);
tokens.put(Integer.MAX_VALUE, SENTENCE_END);
scanner.setSource(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
final String nxtToken = transformToken(token,
scanner.getCurrentTokenString());
final int position = scanner.getCurrentTokenStartPosition();
tokens.put(position, stripTokenIfNeeded(nxtToken));
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:27，
示例5: getTokenType
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
private final String getTokenType(final int tokenType,
final PublicScanner scanner, final IdentifierTypeRetriever retriever) {
if (tokenType == ITerminalSymbols.TokenNameIdentifier) {
return retriever.getIdentifierType(scanner);
} else if (JavaTokenTypeTokenizer.isLiteralToken(tokenType)) {
return LITERAL;
} else {
return scanner.getCurrentTokenString();
}
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:11，
示例6: fullTokenListWithPos
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public SortedMap fullTokenListWithPos(final char[] code) {
final SortedMap tokens = Maps.newTreeMap();
tokens.put(-1, new FullToken(SENTENCE_START, SENTENCE_START));
tokens.put(Integer.MAX_VALUE, new FullToken(SENTENCE_END, SENTENCE_END));
final PublicScanner scanner = createScanner();
scanner.setSource(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
final int position = scanner.getCurrentTokenStartPosition();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
} else if (token == ITerminalSymbols.TokenNameIdentifier) {
tokens.put(position,
new FullToken(IDENTIFIER_TOKEN, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_BLOCK) {
tokens.put(position, new FullToken(COMMENT_BLOCK, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_JAVADOC) {
tokens.put(position, new FullToken(COMMENT_JAVADOC, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_LINE) {
tokens.put(position, new FullToken(COMMENT_LINE, ""));
} else if (isLiteralToken(token)) {
tokens.put(position, new FullToken(LITERAL_TOKEN, ""));
} else {
tokens.put(position,
new FullToken(scanner.getCurrentTokenString(),
""));
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:41，
示例7: getTokenListFromCode
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public List getTokenListFromCode(final char[] code) {
final List tokens = Lists.newArrayList();
tokens.add(new FullToken(SENTENCE_START, SENTENCE_START));
final PublicScanner scanner = createScanner();
scanner.setSource(code);
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
} else if (token == ITerminalSymbols.TokenNameIdentifier) {
tokens.add(new FullToken(IDENTIFIER_TOKEN, ""));
} else if (isLiteralToken(token)) {
tokens.add(new FullToken(LITERAL_TOKEN, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_BLOCK) {
tokens.add(new FullToken(COMMENT_BLOCK, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_JAVADOC) {
tokens.add(new FullToken(COMMENT_JAVADOC, ""));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_LINE) {
tokens.add(new FullToken(COMMENT_LINE, ""));
} else {
tokens.add(new FullToken(scanner.getCurrentTokenString(),
""));
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
tokens.add(new FullToken(SENTENCE_END, SENTENCE_END));
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:34，
示例8: tokenListFromCode
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public List tokenListFromCode(final char[] code) {
final List tokens = Lists.newArrayList();
tokens.add(SENTENCE_START);
final PublicScanner scanner = createScanner();
scanner.setSource(code);
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
} else if (token == ITerminalSymbols.TokenNameIdentifier) {
tokens.add(IDENTIFIER_TOKEN);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_BLOCK) {
tokens.add(COMMENT_BLOCK);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_LINE) {
tokens.add(COMMENT_LINE);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_JAVADOC) {
tokens.add(COMMENT_JAVADOC);
} else if (isLiteralToken(token)) {
tokens.add(LITERAL_TOKEN);
} else {
tokens.add(scanner.getCurrentTokenString());
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
tokens.add(SENTENCE_END);
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:33，
示例9: tokenListWithPos
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public SortedMap tokenListWithPos(final char[] code) {
final SortedMap tokens = Maps.newTreeMap();
tokens.put(-1, SENTENCE_START);
tokens.put(Integer.MAX_VALUE, SENTENCE_END);
final PublicScanner scanner = createScanner();
scanner.setSource(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
final int position = scanner.getCurrentTokenStartPosition();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
} else if (token == ITerminalSymbols.TokenNameIdentifier) {
tokens.put(position, IDENTIFIER_TOKEN);
} else if (isLiteralToken(token)) {
tokens.put(position, LITERAL_TOKEN);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_BLOCK) {
tokens.put(position, COMMENT_BLOCK);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_JAVADOC) {
tokens.put(position, COMMENT_JAVADOC);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_LINE) {
tokens.put(position, COMMENT_LINE);
} else {
tokens.put(position, scanner.getCurrentTokenString());
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:38，
示例10: getConvertedToken
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
/**
* @param tokens
* @param scanner
* @param token
* @return
* @throws InvalidInputException
*/
private List getConvertedToken(final PublicScanner scanner,
final int token) throws InvalidInputException {
final List tokens = Lists.newArrayList();
if (token == ITerminalSymbols.TokenNameEOF) {
return Collections.emptyList();
}
final String tokenString = scanner.getCurrentTokenString();
if (token == ITerminalSymbols.TokenNameWHITESPACE) {
tokens.add(whitespaceConverter.toWhiteSpaceSymbol(tokenString));
} else if (token == ITerminalSymbols.TokenNameIdentifier) {
tokens.add(toIdentifierSymbol(tokenString));
} else if (JavaTokenTypeTokenizer.isLiteralToken(token)) {
tokens.add(toLiteralSymbol(tokenString));
} else if (token == ITerminalSymbols.TokenNameCOMMENT_BLOCK) {
tokens.add(JavaTokenTypeTokenizer.COMMENT_BLOCK);
} else if (token == ITerminalSymbols.TokenNameCOMMENT_LINE) {
tokens.add(JavaTokenTypeTokenizer.COMMENT_LINE);
final int nextToken = scanner.getNextToken();
if (nextToken == ITerminalSymbols.TokenNameWHITESPACE) {
tokens.add(whitespaceConverter.toWhiteSpaceSymbol("\n"
+ scanner.getCurrentTokenString()));
} else {
tokens.add(whitespaceConverter.toWhiteSpaceSymbol("\n"));
tokens.addAll(getConvertedToken(scanner, nextToken));
}
} else if (token == ITerminalSymbols.TokenNameCOMMENT_JAVADOC) {
tokens.add(JavaTokenTypeTokenizer.COMMENT_JAVADOC);
} else {
tokens.add(tokenString);
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:42，
示例11: getTokensWithWidthData
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
public List getTokensWithWidthData(
final char[] code) {
final List tokens = Lists.newArrayList();
tokens.add(new WhitespaceAnnotatedToken(SENTENCE_START,
SENTENCE_START, 0, 0));
final PublicScanner scanner = prepareScanner(code);
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
for (final String cToken : getConvertedToken(scanner, token)) {
final int currentPosition = scanner
.getCurrentTokenStartPosition();
final int currentLine = scanner
.getLineNumber(currentPosition);
final int lineStart = scanner.getLineStart(currentLine);
tokens.add(new WhitespaceAnnotatedToken(cToken, "",
currentPosition - lineStart, scanner
.getCurrentTokenString().length()));
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
tokens.add(new WhitespaceAnnotatedToken(SENTENCE_END, SENTENCE_END,
0, 0));
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:31，
示例12: prepareScanner
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
/**
* @param code
* @return
*/
private PublicScanner prepareScanner(final char[] code) {
final PublicScanner scanner = new PublicScanner();
scanner.setSource(code);
scanner.tokenizeWhiteSpace = true;
scanner.recordLineSeparator = true;
scanner.tokenizeComments = true;
whitespaceConverter = new WhitespaceToTokenConverter();
return scanner;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:14，
示例13: tokenListWithPos
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public SortedMap tokenListWithPos(final char[] code) {
final SortedMap tokens = Maps.newTreeMap();
tokens.put(-1, SENTENCE_START);
tokens.put(Integer.MAX_VALUE, SENTENCE_END);
final PublicScanner scanner = prepareScanner(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
final int position = scanner
.getCurrentTokenStartPosition();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
int i = 0;
final List cTokens = getConvertedToken(scanner,
token);
for (final String cToken : cTokens) {
tokens.put(position + i, cToken);
i++;
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:31，
示例14: fullTokenListWithPos
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
@Override
public SortedMap fullTokenListWithPos(final char[] code) {
// TODO Duplicate Code
final PublicScanner scanner = prepareScanner();
final SortedMap tokens = Maps.newTreeMap();
tokens.put(-1, new FullToken(SENTENCE_START, SENTENCE_START));
tokens.put(Integer.MAX_VALUE, new FullToken(SENTENCE_END, SENTENCE_END));
scanner.setSource(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
final String nxtToken = transformToken(token,
scanner.getCurrentTokenString());
final int position = scanner.getCurrentTokenStartPosition();
tokens.put(position,
new FullToken(nxtToken, Integer.toString(token)));
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:29，
示例15: createScanner
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
/**
* @return
*/
private PublicScanner createScanner() {
final PublicScanner scanner = new PublicScanner();
scanner.tokenizeComments = tokenizeComments;
return scanner;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:9，
示例16: tokenListWithPosAndWidth
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
public SortedMap tokenListWithPosAndWidth(
final char[] code) {
final SortedMap tokens = Maps
.newTreeMap();
tokens.put(-1, new WhitespaceAnnotatedToken(SENTENCE_START,
SENTENCE_START, 0, 0));
tokens.put(Integer.MAX_VALUE, new WhitespaceAnnotatedToken(
SENTENCE_END, SENTENCE_END, 0, 0));
final PublicScanner scanner = prepareScanner(code);
while (!scanner.atEnd()) {
do {
try {
final int token = scanner.getNextToken();
final int currentPosition = scanner
.getCurrentTokenStartPosition();
final int currentLine = scanner
.getLineNumber(currentPosition);
final int lineStart = scanner.getLineStart(currentLine);
final int position = scanner
.getCurrentTokenStartPosition();
if (token == ITerminalSymbols.TokenNameEOF) {
break;
}
int i = 0;
final List cTokens = getConvertedToken(scanner,
token);
for (final String cToken : cTokens) {
tokens.put(position + i,
new WhitespaceAnnotatedToken(cToken, "",
currentPosition - lineStart,
scanner.getCurrentTokenString()
.length()));
i++;
}
} catch (final InvalidInputException e) {
LOGGER.warning(ExceptionUtils.getFullStackTrace(e));
}
} while (!scanner.atEnd());
}
return tokens;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:43，
示例17: prepareScanner
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
/**
* Create the scanner.
*
* @return
*/
protected PublicScanner prepareScanner() {
final PublicScanner scanner = new PublicScanner();
scanner.tokenizeComments = tokenizeComments;
return scanner;
}
開發者ID:mast-group，項目名稱:tassal，代碼行數:11，
示例18: createScanner
點讚 2
import org.eclipse.jdt.internal.core.util.PublicScanner; //導入依賴的package包/類
/**
* Create a scanner, indicating the level of detail requested for tokenizing. The scanner can then be
* used to tokenize some source in a Java aware way.
* Here is a typical scanning loop:
*
*
*
* IScanner scanner = ToolFactory.createScanner(false, false, false, false);
* scanner.setSource("int i = 0;".toCharArray());
* while (true) {
* int token = scanner.getNextToken();
* if (token == ITerminalSymbols.TokenNameEOF) break;
* System.out.println(token + " : " + new String(scanner.getCurrentTokenSource()));
* }
*
*
*
*
By default the compliance used to create the scanner is the workspace's compliance when running inside the IDE
* or 1.4 if running from outside of a headless eclipse.
*
*
* @param tokenizeComments if set to false, comments will be silently consumed
* @param tokenizeWhiteSpace if set to false, white spaces will be silently consumed,
* @param assertMode if set to false, occurrences of 'assert' will be reported as identifiers
* ({@link ITerminalSymbols#TokenNameIdentifier}), whereas if set to true, it
* would report assert keywords ({@link ITerminalSymbols#TokenNameassert}). Java 1.4 has introduced
* a new 'assert' keyword.
* @param recordLineSeparator if set to true, the scanner will record positions of encountered line
* separator ends. In case of multi-character line separators, the last character position is considered. These positions
* can then be extracted using {@link IScanner#getLineEnds()}. Only non-unicode escape sequences are
* considered as valid line separators.
* @return a scanner
* @see org.eclipse.jdt.core.compiler.IScanner
* @see #createScanner(boolean, boolean, boolean, String, String)
*/
public static IScanner createScanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean assertMode, boolean recordLineSeparator){
// use default workspace compliance
long complianceLevelValue = CompilerOptions.versionToJdkLevel(JavaCore.getOption(JavaCore.COMPILER_COMPLIANCE));
if (complianceLevelValue == 0) complianceLevelValue = ClassFileConstants.JDK1_4; // fault-tolerance
PublicScanner scanner =
new PublicScanner(
tokenizeComments,
tokenizeWhiteSpace,
false/*nls*/,
assertMode ? ClassFileConstants.JDK1_4 : ClassFileConstants.JDK1_3/*sourceLevel*/,
complianceLevelValue,
null/*taskTags*/,
null/*taskPriorities*/,
true/*taskCaseSensitive*/);
scanner.recordLineSeparator = recordLineSeparator;
return scanner;
}
開發者ID:trylimits，項目名稱:Eclipse-Postfix-Code-Completion，代碼行數:53，
注：本文中的org.eclipse.jdt.internal.core.util.PublicScanner類示例整理自Github/MSDocs等源碼及文檔管理平台，相關代碼片段篩選自各路編程大神貢獻的開源項目，源碼版權歸原作者所有，傳播和使用請參考對應項目的License；未經允許，請勿轉載。
614
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
