大数据新闻网的分析 先到先得吧 查重率6.2% 主页文件重率更低

摘  要

由于网络的便捷，越来越多的软件被我们创造了出来而开发者为顺应数据信息时代的需求也不断的开发出其他与之相符合的配套措施，从而满足用户们的需求，方便有不同需求用户的生活。因此，在这种环境下让我们对数据有了其他背后的想法。为了满足商家以及其他行业的用户需求等，各大公司都推行了基于大数据上更精准的推送给客户以及为商家用户分析目前的流行趋势等并在商户未来规划和发展上给予一定的指导。反观，对于客户来说在所使用的软件上能够浏览到和自己爱好更加贴合的信息，这对于用户的体验感有更好的增进，所以我拟设计并实现大数据可视化系统设计。开发本系统之后，商家可以根据分析出的数据调整自己的运营信息的占比以及公司主要投入的发展方向等方面。

本系统是基于Spark语言的大数据应用，使用了目前许多公司所采用的大数据框架。本系统按照功能需求将其分成了五个模块，其中包括数据收集模块、数据存储模块、实时处理模块、离线查询模块以及页面显示模块。数据收集模块包括集群对本地的日志信息的收集；实时处理模块运用Spark来进行数据分析，再通过MySQL来收集数据；离线查询模块运用Hive来查询离线的数据；页面显示层则在实时是运用Echarts来实现数据的展示，而在离线的状态下则运用Hue来展现数据信息。通过对程序的各项功能测试，程序的功能能够满足分析一个新闻网站中的数据流动和客户的兴趣程度方向。

关键词：Spark；大数据；数据挖掘

Abstract

Due to the convenience of the network, more and more people through the website to obtain and manage information. To meet people's needs and convenient people's life, there are still a lot of values behind these data for us to explore. In order to meet the needs of merchants, they can accurately locate customers' favorite information and analyze current fashion trends, and provide certain guidance for future planning and development of merchants. For customers, they can browse the information that is more suitable for their hobbies on the web site, which will better improve the user's experience. Since, I plan to design and implement the design of big data visualization system. After the development of this system, businesses can adjust the proportion of their own operating information and the development direction of the company's main investment according to the data analyzed.

The system is a big data application based on Spark language, using Web Socket and Echarts framework to develop visual pages, and data processing is the use of typical big data framework services. The system is mainly divided into five modules, including data collection module, data storage module, real-time processing module, offline processing module and page display layer. The data collection module includes the collection of data information by cluster. Real-time processing module uses Streaming to analyze data, and then collects data through MySQL. Offline processing module uses Map Reduce to process offline data. The page display layer uses Echarts to display data in real time, and Hue to display data in offline state.

Keywords：big data; Spark; data mining

目  录

第1章 绪  论................................................................................................................. 1

1.1 课题背景............................................................................................................ 1

1.2 目的意义............................................................................................................ 2

1.3 解决问题............................................................................................................ 2

1.4 研究现状............................................................................................................ 3

1.5 本章小结............................................................................................................ 3

第2章 大数据实时分析可视化系统分析.................................................................... 4

2.1 可行性分析........................................................................................................ 4

2.2 功能需求分析.................................................................................................... 5

2.3 本章小结............................................................................................................ 6

第3章 大数据实时分析可视化系统设计.................................................................... 7

3.1 总体设计............................................................................................................ 7

3.2 数据库表设计.................................................................................................... 8

3.3 技术框架选择.................................................................................................... 9

3.4 本章小结.......................................................................................................... 12

第4章 大数据实时分析系统可视化实现.................................................................. 13

4.1 实时分析系统实现.......................................................................................... 13

4.2 离线查询系统实现.......................................................................................... 22

4.3 本章小结.......................................................................................................... 25

第5章 大数据实时分析可视化系统测试.................................................................. 26

5.1 概念和意义...................................................................................................... 26

5.2 测试步骤.......................................................................................................... 26

5.3 测试内容.......................................................................................................... 27

5.4 测试结果.......................................................................................................... 29

5.5 本章小结.......................................................................................................... 30

结  论............................................................................................................................. 31

参考文献......................................................................................................................... 32

附  录............................................................................................................................. 34

致  谢............................................................................................................................. 35

第1章 绪  论

1.1 课题背景

目前，随着时代的不断发展，数据新闻可以说是大数据时代背景下的必然产物。处在大数据时代，好像人们生活的各种领域都能够和大数据相互联系在一起，这也是事物的普遍联系性所导致的。在这种情况下，大数据利用其自身的优势和特点在新闻生产过程中斩获并得到了了新闻媒体人的芳心。数据新闻[1]成为国际媒体工作者的另外一个代名词，数据新闻会在杂乱的新闻案件以及信息中用更加直观、清晰明了