大数据实训2 - 法律咨询数据分析和服务推荐

最新推荐文章于 2023-09-14 10:00:57 发布

ＮＥＫＯ

最新推荐文章于 2023-09-14 10:00:57 发布

阅读量1.6k

点赞数

分类专栏：笔记文章标签：大数据机器学习

本文链接：https://blog.csdn.net/qq_37873426/article/details/89419063

版权

原数据(前2行)：

"1",2683657840,140100,"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 1.0","Windows XP","785022225.1422973265","785022225.1422973265",1422973268278,"2015-02-03 
22:21:08","/info/hunyin/hunyinfagui/201404102884290_6.html",20150203,"http://www.lawtime.cn/info/hunyin/hunyinfagui/201404102884290_6.html","107001","www.lawtime.cn","广东省人口与计划生育条例全文2014 - 法律快车婚姻法",31,"故意伤害","计划生育",NA,NA,NA,NA
"2",973705742,140100,"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36","Windows 7","2048326726.1422973286","2048326726.1422973286",1422973268308,"2015-02-03 22:21:08","/ask/exp/17199.html",20150203,"http://www.lawtime.cn/ask/exp/17199.html","1999001","www.lawtime.cn","非广州户籍人员可以在广州申请出入境证件吗？ - 法律快车法律经验",20,"劳资纠纷","出入境","baidu","http://www.baidu.com/s?wd=%E9%9D%9E%E5%B9%BF%E5%B7%9E%E6%88%B7%E7%B1%8D%E4%BA%BA%E5%91%98%E6%80%8E%E4%B9%88%E7%94%B3%E8%AF%B7%E9%A2%84%E7%BA%A6%E5%87%BA%E5%85%A5%E5%A2%83&ie=utf-8","非广州户籍人员怎么申请预约出入境","baidu"

预处理

将原数据中的无用数据及数据中的“，”去除，避免导入数据库时将其分辨为分割符

#!/bin/bash
infile=$1
outfile=$2
#sed 's/"\([^",]*\),\([^"]*\)"/"\1;\2"/g' $infile > $outfile
awk -F "\"" '{
   	 str = $1;
	 for(i=2; i<=3;i++){ str = str$i };
	 for(i=5; i<=NF;i++){ str = str$i };
	 print str;	
     }' $infile > tempfile1

awk -F "," 'BEGIN{
    }
    {
	 str = $1
	 for(i=2; i<=3; i++){ str = str"\t"$i };
	 for(i=5; i<=NF; i++){ str = str"\t"$i };
	 print str;	
     }' tempfile1 > $outfile
#sed 's/\"//g' tempfile2 > $outfile
rm tempfile1
#rm tempfile2