java停用词,如何删除java中的停用词？

I want to remove stop words in java.

So, I read stop words from text file.

and store Set

Set stopWords = new LinkedHashSet();

BufferedReader br = new BufferedReader(new FileReader("stopwords.txt"));

String words = null;

while( (words = br.readLine()) != null) {

stopWords.add(words.trim());

}

br.close();

And, I read another text file.

So, I wanna remove to duplicate string in text file.

How can I?

解决方案

You want to remove duplicate words from file, below is the high level logic for same.

Read File

Loop through file content(i.e one line at a time)

Have string tokenizer for that line based on space

Add each each token to your set. This will make sure that you have only one entry per word.

Close file

Now you have set that contains all the unique word of file.