一、Introduction
有关R语言对PDF词频统计的博客已很多,但有以下问题未解决:
- 对英文进行词频统计时,“a” “an” "it"等词汇无实际意义,数字的出现也会干扰词频统计。
- 未把相关代码整合成自定义函数,导致使用不方便。
二、代码
hasdigit <- function(str){
if(!is.character(str)){
stop("'str' should be character.")
}
n <- nchar(str)
for(i in 1:n){
ch <- substr(str, i, i)
if(ch>="0"&&ch<="9"){
return(T)
}
}
return(F)
}
wordstat.pdf <- function(file, lo=3, simplify=T, del_num=T){
# lo: minimum word length
# simplify: whether to delete simple words like "a", "an", "we", etc
# del_num: whether to delete numbers
if(!"pdftools" %in% .packages()){
library(pdftools)
}
if(!"jiebaRD" %in% .packages()){
library(jiebaRD)
}
if(!"jiebaR" %in% .packages()){
library(jiebaR)
}
if(!"wordcloud2" %in% .packages()){
library(wordcloud2)
}
text <- pdf_text(file)
seg <- tolower(qseg[text])
seg <- sort(seg, decreasing = TRUE)
seg <- table(seg)
# deplete namelist
if(simplify){
ex <- c("is","are","be","was","were","become","becomes","do","did","does","a","an","the",
"can","will","would","could","should","may","might","have","has",
"and","or","not","but","although","though","no","also","if","against","any",
"for","on","off","from","to","of","in","by","like","as","at","about","up","down",
"below","between","above","with",
"many","more","much","most","better","worse","worst","best","good","bad",
"it","them","its","their","we","you","our","this","that","these","those",
"what","when","where","how","which","whose","why",
"get","some","other","others")
seg[ex] <- -1
}
# deplete digit
if(del_num){
for (i in 1:length(seg)){
if(hasdigit(names(seg)[i])){
seg[i] <- -1
}
}
}
seg <- seg[seg > lo]
return(seg)
}
三、使用案例
- 需要安装包:pdftools、jiebaRD、jiebaR、wordcloud2
以雅思Simon考官撰写的Ideas for IELTS topics为例(从网站ielts-simon.com中可获取PDF),先运行【二、代码】,将两个函数导入R语言环境中,再运行下列代码:
file <- "D:\\Ideas_for_IELTS_topics.pdf"
res <- wordstat.pdf(file)
wordcloud2(res, size = 1, shape = 'circle',color = 'random-light')
结果如下:
可见Simon考官提供的雅思Part2素材中,children一词高频出现。
参考资料:https://blog.csdn.net/BEYONDMA/article/details/85465403