1.用Hive对爬虫大作业产生的文本文件(或者英文词频统计下载的英文长篇小说)词频统计。
1.启动hadoop
2.Hdfs上创建文件夹并查看
3.上传英文词频统计文本至hdfs
1、把英文文章的每个单词放到列表里,并统计列表长度;
2、遍历列表,对每个单词出现的次数进行统计,并将结果存储在字典中;
3、利用步骤1中获得的列表长度,求出每个单词出现的频率,并将结果存储在频率字典中;
3、以字典键值对的“值”为标准,对字典进行排序,输出结果(也可利用切片输出频率最大或最小的特定几个,因为经过排序sorted()函数处理后,单词及其频率信息已经存储在元组中,所有元组再组成列表。)
fin = open('The_Magic_Skin _Honore_de_Balzac.txt') #the txt is up
#to you
lines=fin.readlines()
fin.close()
'''transform the article into word list
'''
def words_list():
chardigit='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 '
all_lines = ''
for line in lines:
one_line=''
for ch in line:
if ch in chardigit:
one_line = one_line + ch
all_lines = all_lines + one_line
return all_lines.split()
'''calculate the total number of article list
s is the article list
'''
def total_num(s):
return len(s)
'''calculate the occurrence times of every word
t is the article list
'''
def word_dic(t):
fre_dic = dict()
for i in range(len(t)):
fre_dic[t[i]] = fre_dic.get(t[i],0) + 1
return fre_dic
'''calculate the occurrence times of every word
w is dictionary of the occurrence times of every word
'''
def word_fre(w):
for key in w:
w[key] = w[key] / total
return w
'''sort the dictionary
v is the frequency of words
'''
def word_sort(v):
sort_dic = sorted(v.items(), key = lambda e:e[1])
return sort_dic
'''This is entrance of functions
output is the ten words with the largest frequency
'''
total = total_num(words_list())
print(word_sort(word_fre(word_dic(words_list())))[-10:])
英文长篇小说TXT如下
词频截图如下
2.用Hive对爬虫大作业产生的csv文件进行数据分析,写一篇博客描述你的分析过程和分析结果。
csv文件,通过 Numbers 打开
"Header1","Header2","Header3"
"Data1","Data2","Data3"
"Data1","Data2","Data3"
所有的数据都读到了 datas 数组中
def main(input_file_path):
input_file = open(input_file_path);
# skip header
input_file.readline();
datas = [];
key_index_table = {
"Header1": 0,
"Header2": 1,
"Header3": 2};
count = 0;
max_count = 10000;
while count < max_count:
count += 1;
line = input_file.readline();
if not line:
break;
values = line.split(",");
data = {};
for key in key_index_table:
value = values[key_index_table[key]];
value = value[1:-1]; # remove quotation mark
data[key] = int(value);
datas.append(data);
drawTable(datas);
分析截图如下
这样就可以对自己感兴趣的方向分析。