娱乐头条-03spider

最新推荐文章于 2022-08-17 11:27:36 发布

通天云计算

最新推荐文章于 2022-08-17 11:27:36 发布

阅读量2k

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/WYJ____/article/details/103863206

版权

本文介绍了如何使用Java实现娱乐新闻的爬取，从确定首页URL开始，通过发送请求获取参数，解析JSONP数据，存储到数据库。同时，文章还讨论了如何处理数据重复问题，并引入了Java定时任务来定期执行爬虫。

摘要由CSDN通过智能技术生成

1.确定首页的URL

访问这个URL

https://pacaio.match.qq.com/irs/rcd?cid=146&token=49cbb2154853ef1a74ff4e53723372ce&ext=ent&page=1&expIds=20200106A0JR02|20200106A0JQFA|20200106A0JFB9|20200106A0DWAP|20200106A0JGBR|20200106A0JHTK|20200106A0JI0G|20200106A0JIH6|20200106A0JJ9Z|20200106A0JLXW|20200106A0JNFB|20200106A00XQL|20200106A0JVJ3|20200106A0JVL5|20200106A0K03B|20200106A0JN2R|20200106A0JQOE|20200106A0JSOC|20200106A0JSYG|20200106A0JY8S&callback=__jp6

简化URL（page和callback之间的都删除），再次访问

https://pacaio.match.qq.com/irs/rcd?cid=146&token=49cbb2154853ef1a74ff4e53723372ce&ext=ent&page=1&callback=__jp6

这个url就是我们要确定的url

2.发送请求获取参数

先添加依赖

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.6</version>
        </dependency>

    public void main() throws Exception{
        //1.确定url
        String indexUrl = "https://pacaio.match.qq.com/irs/rcd?cid=146&token=49cbb2154853ef1a74ff4e53723372ce&ext=ent&page=1&callback=__jp6";

        //2.发送请求，获取数据
        //创建HttpClients对象
        CloseableHttpClient httpClient = HttpClients.createDefault();
        //创建请求方式对象
        HttpGet get = new HttpGet(indexUrl);
        //执行请求操作
        CloseableHttpResponse response = httpClient.execute(get);

        if(response.getStatusLine().getStatusCode() == 200){
            HttpEntity entity = response.getEntity();
            String res = EntityUtils.toString(entity);
            System.out.println(res);
        }
    }

3.解析数据的思路

什么是跨域？

简单的来说，出于安全方面的考虑，页面中的JS无法访问其他服务器上的数据，即“同源策略”。而跨域就是通过某些手段来绕过同源策略限制，实现不同服务器之间通信的效果。

什么是JSONP？

在同源策略下，在某个服务器下的页面是无法获取到该服务器以外的数据的，但img\iframe\script等标签是个例外，这些标签可以通过src属性请求到其他服务器上的数据。利用script标签的开放策略，我们可以实现跨域请求数据，当然，也需要服务端的配合。当我们正常的请求一个json数据的时候，服务端返回的是一串json类型的数据，而我们使用jsonp模式来请求数据的时候，服务端返回的是一段可执行的js代码。

举个例子，假如需要从服务器（url）获取的数据如下：

{"id":123,"name":张三,"age":12}

那么，使用json方式请求(url?callback=foo)的数据将会是如下:

foo({"id":123,"name":张三,"age":12})

思路：去头尾将剩下的json格式的字符串转换成一个java对象（看简化的url最后callback）

4.解析数据的实现

添加依赖

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.47</version>
        </dependency>

    public void main() throws Exception{
        //1.确定url
        String indexUrl = "https://pacaio.match.qq.com/irs/rcd?cid=146&token=49cbb2154853ef1a74ff4e53723372ce&ext=ent&page=1&callback=__jp6";

        //2.发送请求，获取数据
        //创建HttpClients对象
        CloseableHttpClient httpClient = HttpClients.createDefault();
        //创建请求方式对象
        HttpGet get = new HttpGet(indexUrl);
        //执行请求操作
        CloseableHttpResponse response = httpClient.execute(get);

        if(response.getStatusLine().getStatusCode() == 200){
            HttpEntity entity = response.getEntity();
            String res = EntityUtils.toString(entity);
            res = toJsonString(res);//截取字符串
            //System.out.println(res);
            Map<String, Object> map = jsonToMap(res);//转为map对象
        }
    }

    //截取结果字符串
    public String toJsonString(String src){
        int start = src.indexOf("{");
        int end = src.lastIndexOf("}")+1;
        return src.substring(start,end);
    }

    //json字符串转换为java对象
    public Map<String,Object> jsonToMap(String src){
        return JSON.parseObject(src, HashMap.class);
    }

得到的res

{"code":0,"msg":"","data":[{"app_id":"20200106A0O59K00","article_type":0,"bimg":"https://inews.gtimg.com/newsapp_ls/0/11144808337_640330/0","category":"ent","category1_chn":"娱乐","category1_id":"1","category2_chn":"中国娱乐","category2_id":"112","category_chn":"娱乐","category_id":"1","comment_id":"4643559759","comment_num":0,"duration":0,"ext":{"tonality_score":2,"news_score":3,"img_type":3,"video_count":0},"flag":0,"fm_url":"","from":"cms_pool","id":"20200106A0O59K","img":"https://inews.gtimg.com/newsapp_ls/0/11144808337_640330/0","img_count":28,"img_type":3,"imgs":{"294X195":"https://inews.gtimg.com/newsapp_ls/0/11144808337_294195/0","295X195":"https://inews.gtimg.com/newsapp_ls/0/11144808337_295195/0","354X264":"https://inews.gtimg.com/newsapp_ls/0/11144779611_354264/0","400X320":"https://inews.gtimg.com/newsapp_ls/0/11144808337_400320/0","580X328":"https://inews.gtimg.com/newsapp_ls/0/11144808337_580328/0","640X330":"https://inews.gtimg.com/newsapp_ls/0/11144808337_640330/0","640X470":"https://inews.gtimg.com/newsapp_ls/0/11144808337_640470/0","870X492":"https://inews.gtimg.com/newsapp_ls/0/11144808337_870492/0"},"intro":"1月6日，阅文风云盛典在上海开启录制，很多知名演员来到现场助阵，除此之外也有不少备受大家期待的当红编剧和作家到场，而他们在红毯上的动向也是最让人关注的。关晓彤以一袭白色水晶裙登场，裙摆与长发间都有闪……","irs_imgs":{"294X195":["https://inews.gtimg.com/newsapp_ls/0/11144808337_294195/0"],"295X195":["https://inews.gtimg.com/newsapp_ls/0/11144808337_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144808338_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144808339_295195/0"],"354X264":["https://inews.gtimg.com/newsapp_ls/0/11144779611_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144784672_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144779608_354264/0"],"400X320":["https://inews.gtimg.com/newsapp_ls/0/11144808337_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144808338_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144808339_400320/0"],"580X328":["https://inews.gtimg.com/newsapp_ls/0/11144808337_580328/0"],"640X330":["https://inews.gtimg.com/newsapp_ls/0/11144808337_640330/0"],"640X470":["https://inews.gtimg.com/newsapp_ls/0/11144808337_640470/0"],"870X492":["https://inews.gtimg.com/newsapp_ls/0/11144808337_870492/0"]},"keywords":"关晓彤;阅文;肖战","media_icon":"https://inews.gtimg.com/newsapp_ls/0/5489913596_200200/0","mini_img":"","multi_imgs":["https://inews.gtimg.com/newsapp_bt/0/11144778372/641","https://inews.gtimg.com/newsapp_bt/0/11144778373/641","https://inews.gtimg.com/newsapp_bt/0/11144778374/641"],"news_level":3,"play_url_high":"","play_url_medium":"","play_url_small":"","pool_type":"pc:article","publish_time":"2020-01-06 19:11:11","report":"{\"ext\": \"44+0\"+ \"all\": \";;;72+7\"+ \"3h\": \"5590530+383246;2778593+201973;ent:1204926+93453#tv:0+0;72+7;6409+453\"+ \"app\": \";;;15916+552;;0;0;0;25081;45.43659;0.03468;1.57584\"+ \"day\": \"33013074+2591064;16178876+1361251;ent:6489037+524317#tv:0+0;72+7;70533+5385\"}","s_group":2,"showtype":"article","source":"萌神木木","source_fans":0,"source_id":"5113868","source_logo":"","strategy":2,"surl":"http://kuaibao.qq.com/s/20200106A0O59K00","tag_label":[["关晓彤","80996"],["阅文","271346"],["肖战","237936"]],"tags":"关晓彤;阅文;肖战","title":"阅文盛典红毯高清图：关晓彤水晶裙闪亮登场，肖战化身暗黑系王子","title_len":29.11111,"ts":1578309071,"update_time":"2020-01-06 19:11:11","url":"https://xw.qq.com/cmsid/20200106A0O59K00","view_count":44,"vurl":"https://new.qq.com/omn/20200106/20200106A0O59K00.html"},{"app_id":"20200106A0MGE700","article_type":1,"bimg":"https://inews.gtimg.com/newsapp_ls/0/11144839145_640330/0","category":"ent","category1_chn":"娱乐","category1_id":"1","category2_chn":"欧美娱乐","category2_id":"110","category_chn":"娱乐","category_id":"1","comment_id":"4643310119","comment_num":0,"duration":0,"ext":{"tonality_score":2,"news_score":3,"img_type":3,"video_count":0},"flag":0,"fm_url":"","from":"cms_pool","id":"20200106A0MGE7","img":"https://inews.gtimg.com/newsapp_ls/0/11144839145_640330/0","img_count":9,"img_type":3,"imgs":{"294X195":"https://inews.gtimg.com/newsapp_ls/0/11144839145_294195/0","295X195":"https://inews.gtimg.com/newsapp_ls/0/11144839145_295195/0","400X320":"https://inews.gtimg.com/newsapp_ls/0/11144839145_400320/0","485X350":"https://inews.gtimg.com/newsapp_ls/0/11144834692_485350/0","580X328":"https://inews.gtimg.com/newsapp_ls/0/11144839145_580328/0","640X330":"https://inews.gtimg.com/newsapp_ls/0/11144839145_640330/0","640X470":"https://inews.gtimg.com/newsapp_ls/0/11144839145_640470/0","870X492":"https://inews.gtimg.com/newsapp_ls/0/11144839145_870492/0"},"intro":"当地时间2020年1月5日，第77届金球奖颁奖典礼在美国洛杉矶举办，男明星们纷纷穿西装出席格外干练端庄，而红毯作为女明星的第一“战场”，必然少不了一番争奇斗艳。相差17岁的好莱坞两大女神妮可·基德曼……","irs_imgs":{"294X195":["https://inews.gtimg.com/newsapp_ls/0/11144839145_294195/0"],"295X195":["https://inews.gtimg.com/newsapp_ls/0/11144839145_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144839146_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144839147_295195/0"],"400X320":["https://inews.gtimg.com/newsapp_ls/0/11144839145_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144839146_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144839147_400320/0"],"485X350":["https://inews.gtimg.com/newsapp_ls/0/11144834692_485350/0","https://inews.gtimg.com/newsapp_ls/0/11144834693_485350/0","https://inews.gtimg.com/newsapp_ls/0/11144834694_485350/0","https://inews.gtimg.com/newsapp_ls/0/11144834695_485350/0"],"580X328":["https://inews.gtimg.com/newsapp_ls/0/11144839145_580328/0"],"640X330":["https://inews.gtimg.com/newsapp_ls/0/11144839145_640330/0"],"640X470":["https://inews.gtimg.com/newsapp_ls/0/11144839145_640470/0"],"870X492":["https://inews.gtimg.com/newsapp_ls/0/11144839145_870492/0"]},"keywords":"斯嘉丽·约翰逊;妮可·基德曼","media_icon":"https://inews.gtimg.com/newsapp_ls/0/2819348763_200200/0","mini_img":"","multi_imgs":["https://inews.gtimg.com/newsapp_bt/0/11144387937/641","https://inews.gtimg.com/newsapp_bt/0/11144387948/641","https://inews.gtimg.com/newsapp_bt/0/11144387938/641"],"news_level":3,"play_url_high":"","play_url_medium":"","play_url_small":"","pool_type":"pc:article","publish_time":"2020-01-06 19:15:00","report":"{\"ext\":\"0+0\"+\"all\":\";;;0+0\"+\"3h\":\"5412073+383246;210815+11604;ent:1204926+93453#movie:13109+450;0+0;130+9\"+\"app\":\";;;1+0;;0;0;0;0;0.0;0.0;0.0\"+\"day\":\"32834617+2591064;2654316+137853;ent:6489037+524317#movie:69552+2394;0+0;488+40\"}","s_group":2,"showtype":"picture","source":"站酷海洛娱乐","source_fans":0,"source_id":"6667303","source_logo":"","strategy":2,"surl":"http://kuaibao.qq.com/s/20200106A0MGE700","tag_label":[["斯嘉丽·约翰逊","323309"],["妮可·基德曼","305841"]],"tags":"斯嘉丽·约翰逊;妮可·基德曼","title":"53岁妮可·基德曼和斯嘉丽穿红裙同框 身高相差20cm输赢一目了然","title_len":29.444445,"ts":1578309300,"update_time":"2020-01-06 19:15:00","url":"https://xw.qq.com/cmsid/20200106A0MGE700","view_count":0,"vurl":"https://new.qq.com/omn/20200106/20200106A0MGE700.html"},{"app_id":"20200106A0OBSQ00","article_type":0,"bimg":"https://inews.gtimg.com/newsapp_ls/0/11144850738_640330/0","category":"ent","category1_chn":"娱乐","category1_id":"1","category2_chn":"中国娱乐","category2_id":"112","category_chn":"娱乐","category_id":"1","comment_id":"4643589964","comment_num":5,"duration":0,"ext":{"tonality_score":2,"news_score":3,"img_type":1,"video_count":0},"flag":0,"fm_url":"","from":"cms_pool","id":"20200106A0OBSQ","img":"https://inews.gtimg.com/newsapp_ls/0/11144850738_640330/0","img_count":8,"img_type":1,"imgs":{"294X195":"https://inews.gtimg.com/newsapp_ls/0/11144850738_294195/0","295X195":"https://inews.gtimg.com/newsapp_ls/0/11144845355_294195/0","354X264":"https://inews.gtimg.com/newsapp_ls/0/11144845355_354264/0","580X328":"https://inews.gtimg.com/newsapp_ls/0/11144850738_580328/0","640X330":"https://inews.gtimg.com/newsapp_ls/0/11144850738_640330/0","640X470":"https://inews.gtimg.com/newsapp_ls/0/11144850738_640470/0","870X492":"https://inews.gtimg.com/newsapp_ls/0/11144850738_870492/0"},"intro":"序言：2020年1月6日，有八卦媒体曝光疑似马蓉的动态。马蓉疑似与一个中年男子一同现身，该男子身穿蓝色西装，看到镜头还特地护住女方。随后，两人挽手同回酒店，看起来关系亲密。有媒体就此向马蓉方求证，马……","irs_imgs":{"294X195":["https://inews.gtimg.com/newsapp_ls/0/11144850738_294195/0"],"295X195":["https://inews.gtimg.com/newsapp_ls/0/11144845355_294195/0","https://inews.gtimg.com/newsapp_ls/0/11144845373_294195/0","https://inews.gtimg.com/newsapp_ls/0/11144845366_294195/0"],"354X264":["https://inews.gtimg.com/newsapp_ls/0/11144845355_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144845373_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144845366_354264/0"],"580X328":["https://inews.gtimg.com/newsapp_ls/0/11144850738_580328/0"],"640X330":["https://inews.gtimg.com/newsapp_ls/0/11144850738_640330/0"],"640X470":["https://inews.gtimg.com/newsapp_ls/0/11144850738_640470/0"],"870X492":["https://inews.gtimg.com/newsapp_ls/0/11144850738_870492/0"]},"keywords":"马蓉;王宝强","media_icon":"https://inews.gtimg.com/newsapp_ls/0/8279574616_200200/0","mini_img":"","multi_imgs":["https://inews.gtimg.com/newsapp_bt/0/11144843000/641","https://inews.gtimg.com/newsapp_bt/0/11144843001/641","https://inews.gtimg.com/newsapp_bt/0/11144843002/641"],"news_level":3,"play_url_high":"","play_url_medium":"","play_url_small":"","pool_type":"pc:article","publish_time":"2020-01-06 19:18:32","report":"{\"ext\":\"2+5\"+\"all\":\";;;675+12\"+\"3h\":\"5412073+383246;2683251+201973;ent:1204926+93453;675+12;27039+1219\"+\"app\":\";;;0+0;;0;0;0;0;0;0;0\"+\"day\":\"32834617+2591064;16083534+1361251;ent:6489037+524317;675+12;247746+19247\"}","s_group":2,"showtype":"article","source":"今日影视头条","source_fans":0,"source_id":"5174463","source_logo":"","strategy":2,"surl":"http://kuaibao.qq.com/s/20200106A0OBSQ00","tag_label":[["马蓉","84720"],["王宝强","87921"]],"tags":"马蓉;王宝强","title":"马蓉方否认恋情，因为和王宝强的财产分割还没完成，不想谈恋爱","title_len":28.11111,"ts":1578309512,"update_time":"2020-01-06 19:18:32","url":"https://xw.qq.com/cmsid/20200106A0OBSQ00","view_count":2,"vurl":"https://new.qq.com/omn/20200106/20200106A0OBSQ00.html"},{"app_id":"20200106A0OC0A00","article_type":0,"bimg":"https://inews.gtimg.com/newsapp_ls/0/11144854265_640330/0","category":"ent","category1_chn":"娱乐","category1_id":"1","category2_chn":"中国娱乐","category2_id":"112","category_chn":"娱乐","category_id":"1","comment_id":"4643591511","comment_num":0,"duration":0,"ext":{"tonality_score":2,"news_score":3,"img_type":3,"video_count":0},"flag":0,"fm_url":"","from":"cms_pool","id":"20200106A0OC0A","img":"https://inews.gtimg.com/newsapp_ls/0/11144854265_640330/0","img_count":4,"img_type":3,"imgs":{"294X195":"https://inews.gtimg.com/newsapp_ls/0/11144854265_294195/0","295X195":"https://inews.gtimg.com/newsapp_ls/0/11144854265_295195/0","354X264":"https://inews.gtimg.com/newsapp_ls/0/11144849137_354264/0","400X320":"https://inews.gtimg.com/newsapp_ls/0/11144854265_400320/0","580X328":"https://inews.gtimg.com/newsapp_ls/0/11144854265_580328/0","640X330":"https://inews.gtimg.com/newsapp_ls/0/11144854265_640330/0","640X470":"https://inews.gtimg.com/newsapp_ls/0/11144854265_640470/0","870X492":"https://inews.gtimg.com/newsapp_ls/0/11144854265_870492/0"},"intro":"1月4日，陈翔30岁生日会在成都如期举办，近600名香橙从全国各地赶来，和自己喜欢了9年的男孩一起相聚庆祝。陈翔为粉丝们演唱了多首歌曲，还现场弹唱了自己未发布的新歌《有时》，作为礼物送给粉丝。1月的……","irs_imgs":{"294X195":["https://inews.gtimg.com/newsapp_ls/0/11144854265_294195/0"],"295X195":["https://inews.gtimg.com/newsapp_ls/0/11144854265_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144854266_295195/0","https://inews.gtimg.com/newsapp_ls/0/11144854267_295195/0"],"354X264":["https://inews.gtimg.com/newsapp_ls/0/11144849137_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144849120_354264/0","https://inews.gtimg.com/newsapp_ls/0/11144849142_354264/0"],"400X320":["https://inews.gtimg.com/newsapp_ls/0/11144854265_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144854266_400320/0","https://inews.gtimg.com/newsapp_ls/0/11144854267_400320/0"],"580X328