使用pig对钓鱼网站链接url做词频统计【大数据处理与分析技术】

最新推荐文章于 2024-04-27 20:28:20 发布

nabobess

最新推荐文章于 2024-04-27 20:28:20 发布

阅读量451

点赞数 1

分类专栏：金功勋老师课程相关（大数据云计算）文章标签：大数据

本文链接：https://blog.csdn.net/nabobess/article/details/124151854

版权

金功勋老师课程相关（大数据云计算）专栏收录该内容

1 篇文章 1 订阅

订阅专栏

使用pig对钓鱼网站链接url做词频统计（大数据处理与分析技术）
数据展示：（verified_online.csv）
在这里插入图片描述

grunt> A = load 'verified_online.csv' using PigStorage(',') AS (phish_id:chararray, url:chararray, phish_detail_url:chararray, submission_time:chararray, verified:chararray, verification_time:chararray, online:chararray, target:chararray);
grunt> B = FILTER A BY target == 'targetWord';
grunt> C = FOREACH B GENERATE REPLACE(target, '"', ''), $0,REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE($1, ':', ','), '//', ','), ',,', ','), '\\.', ','), '\\/', ','), '\\?', ','), '\\=', ',') as (line);
grunt> D = FOREACH C GENERATE flatten(TOKENIZE(line)) as word;
grunt> E = GROUP D BY word;
grunt> F = FOREACH E GENERATE group, COUNT(D) AS cnt;
grunt> G = order F by cnt DESC;
grunt> DUMP G;