springboot基于Hadoop的高校电子图书馆的大数据平台规划与设计(源码+vue+scrapy+hive+部署文档+可视化大屏展示等)

收藏关注不迷路！！

🌟文末获取源码+数据库🌟

感兴趣的可以先收藏起来，还有大家在毕设选题（免费咨询指导选题），项目以及论文编写等相关问题都可以给我留言咨询，希望帮助更多的人

---

前言

💗博主介绍：✨全网拥有20W+粉丝、CSDN作者、博客专家、全栈领域优质创作者、平台优质Java创作者、专注于Java、小程序、python、安卓技术领域和毕业项目实战✌💟！✨💗

👇🏻 精彩专栏 推荐订阅👇🏻

计算机毕业设计设计精品实战案例✅

感兴趣的可以先收藏起来，还有大家在毕设选题（免费咨询指导选题），项目以及论文编写等相关问题都可以给我留言咨询，博主免费解答、希望可以帮助更多人

  

详细视频演示

文章底部名片，获取项目的完整演示视频，免费解答技术疑问

项目介绍

  伴随着我国社会的发展，人民生活质量日益提高。于是对高校电子图书馆的大数据管理进行规范而严格是十分有必要的，所以许许多多的信息管理系统应运而生。此时单靠人力应对这些事务就显得有些力不从心了。所以本论文将设计一套高校电子图书馆的大数据平台，帮助图书馆进行文热门图书、用户投稿、教育等繁琐又重复的工作，提高工作效率的同时，也减轻了管理者的压力。
本论文的主要内容包括：
第一，研究分析当下主流的web技术，结合图书馆日常管理方式，进行高校电子图书馆的大数据平台的数据库设计，设计高校电子图书馆的大数据平台功能，并对每个模块进行说明。
第二，陈列说明该系统实现所采用的架构、系统搭建采用的服务器、系统开发环境和使用的工具，以及系统后台采用的数据库。
最后，对系统进行全面测试，主要包括功能测试、查询性能测试、安全性能测试。
分析系统存在的不足以及将来改进的方向。

技术介绍

开发语言：Java
框架：springboot
JDK版本：JDK1.8
服务器：tomcat7
数据库：mysql
数据库工具：Navicat11
开发软件：eclipse/myeclipse/idea
Maven包：Maven

功能介绍

满足高校电子图书馆的大数据平台的需求。此系统功能较为全面如下图系统功能结构如图4-1所示。

图4-1功能结构图

核心代码

# # -*- coding: utf-8 -*-

# 数据爬取文件

import scrapy
import pymysql
import pymssql
from ..items import JiaoyuItem
import time
from datetime import datetime,timedelta
import datetime as formattime
import re
import random
import platform
import json
import os
import urllib
from urllib.parse import urlparse
import requests
import emoji
from selenium.webdriver import ChromeOptions, ActionChains
from scrapy.http import TextResponse
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
class JiaoyuSpider(scrapy.Spider):
    name = 'jiaoyuSpider'
    spiderUrl = 'https://findujs.libsp.com/find/unify/search'
    start_urls = spiderUrl.split(";")
    protocol = ''
    hostname = ''
    realtime = False
    headers = {
        'Accept': 'application/json, text/plain, */*',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7',
        'Connection': 'keep-alive',
        'Content-Type': 'application/json;charset=UTF-8',
        'Cookie': 'SameSite=None',
        'Cache-Control': 'max-age=0',
        'Origin': 'https://findujs.libsp.com',
        'Referer': 'https://findujs.libsp.com/',
        'Sec-Fetch-Dest': 'empty',
        'Sec-Fetch-Mode': 'cors',
        'Sec-Fetch-Site': 'same-origin',
        'content-language': 'zh_CN',
        'groupCode': '200094',
        'sec-ch-ua': '"Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'x-lang': 'CHI'
    }

    def __init__(self,realtime=False,*args, **kwargs):
        super().__init__(*args, **kwargs)
        self.realtime = realtime=='true'

    def start_requests(self):

        plat = platform.system().lower()
        if not self.realtime and (plat == 'linux' or plat == 'windows'):
            connect = self.db_connect()
            cursor = connect.cursor()
            if self.table_exists(cursor, '191yf7f0_jiaoyu') == 1:
                cursor.close()
                connect.close()
                self.temp_data()
                return

        pageNum=1+1
        for page in range(1, pageNum):
            params = {
                'matchMode': '2',
                'sortField': 'relevance',
                'sortClause': 'asc',
                'page': page,
                'rows': 50,
                'searchField': 'keyWord',
                'searchFieldContent': '教育',
                'indexSearch': 1
            }
            response = requests.post("https://findujs.libsp.com/find/unify/search", json=params, headers=self.headers)
            if response.status_code == 200:
                yield scrapy.Request(url=self.spiderUrl, meta={"json":response.json()}, callback=self.parse,dont_filter=True)
            else:
                continue

    # 列表解析
    def parse(self, response):
        _url = urlparse(self.spiderUrl)
        self.protocol = _url.scheme
        self.hostname = _url.netloc
        plat = platform.system().lower()
        if not self.realtime and (plat == 'linux' or plat == 'windows'):
            connect = self.db_connect()
            cursor = connect.cursor()
            if self.table_exists(cursor, '191yf7f0_jiaoyu') == 1:
                cursor.close()
                connect.close()
                self.temp_data()
                return
        data = response.meta["json"]

        try:
            list = data["data"]["searchResult"]
        except:
            pass
        for item in list:
            fields = JiaoyuItem()

            try:
                fields["title"] = str( item["title"])

            except:
                pass
            try:
                fields["author"] = str( item["author"])

            except:
                pass
            try:
                fields["publisher"] = str( item["publisher"])

            except:
                pass
            try:
                fields["publishyear"] = str( item["publishYear"])

            except:
                pass
            try:
                fields["price"] = float( item["price"])
            except:
                pass
            try:
                fields["bookpage"] = int(item["bookPage"].replace("页",""))
            except:
                pass
            try:
                fields["callnoone"] = str( item["callNoOne"])

            except:
                pass
            try:
                fields["subjectword"] = str( item["subjectWord"])

            except:
                pass
            try:
                fields["adstract"] = str( item["adstract"])

            except:
                pass
            try:
                fields["laiyuan"] = str("https://findujs.libsp.com/#/searchList/bookDetails/"+ str(item["recordId"]))

            except:
                pass
            try:
                fields["docname"] = str( item["docName"])

            except:
                pass
            yield fields

    # 去除多余html标签
    def remove_html(self, html):
        if html == None:
            return ''
        pattern = re.compile(r'<[^>]+>', re.S)
        return pattern.sub('', html).strip()

    # 数据库连接
    def db_connect(self):
        type = self.settings.get('TYPE', 'mysql')
        host = self.settings.get('HOST', 'localhost')
        port = int(self.settings.get('PORT', 3306))
        user = self.settings.get('USER', 'root')
        password = self.settings.get('PASSWORD', '123456')

        try:
            database = self.databaseName
        except:
            database = self.settings.get('DATABASE', '')

        if type == 'mysql':
            connect = pymysql.connect(host=host, port=port, db=database, user=user, passwd=password, charset='utf8')
        else:
            connect = pymssql.connect(host=host, user=user, password=password, database=database)
        return connect

    # 断表是否存在
    def table_exists(self, cursor, table_name):
        cursor.execute("show tables;")
        tables = [cursor.fetchall()]
        table_list = re.findall('(\'.*?\')',str(tables))
        table_list = [re.sub("'",'',each) for each in table_list]

        if table_name in table_list:
            return 1
        else:
            return 0

    # 数据缓存源
    def temp_data(self):

        connect = self.db_connect()
        cursor = connect.cursor()
        sql = '''
            insert into `jiaoyu`(
                id
                ,title
                ,author
                ,publisher
                ,publishyear
                ,price
                ,bookpage
                ,callnoone
                ,subjectword
                ,adstract
                ,laiyuan
                ,docname
            )
            select
                id
                ,title
                ,author
                ,publisher
                ,publishyear
                ,price
                ,bookpage
                ,callnoone
                ,subjectword
                ,adstract
                ,laiyuan
                ,docname
            from `191yf7f0_jiaoyu`
            where(not exists (select
                id
                ,title
                ,author
                ,publisher
                ,publishyear
                ,price
                ,bookpage
                ,callnoone
                ,subjectword
                ,adstract
                ,laiyuan
                ,docname
            from `jiaoyu` where
                `jiaoyu`.id=`191yf7f0_jiaoyu`.id
            ))
        '''

        cursor.execute(sql)
        connect.commit()
        connect.close()

数据库参考

--
-- Current Database: `springbootb2p57440`
--

/*!40000 DROP DATABASE IF EXISTS `springbootb2p57440`*/;

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `springbootb2p57440` /*!40100 DEFAULT CHARACTER SET utf8mb4 */;

USE `springbootb2p57440`;

--
-- Table structure for table `191yf7f0_jiaoyu`
--

DROP TABLE IF EXISTS `191yf7f0_jiaoyu`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `191yf7f0_jiaoyu` (
  `id` bigint(20) NOT NULL DEFAULT '0' COMMENT '主键',
  `addtime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `title` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '标题',
  `author` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '责任者',
  `publisher` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '出版社',
  `publishyear` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '出版时间',
  `price` double DEFAULT NULL COMMENT '价格',
  `bookpage` int(11) DEFAULT NULL COMMENT '页数',
  `callnoone` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '索书号',
  `subjectword` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '学科主题',
  `adstract` longtext CHARACTER SET utf8 COMMENT '内容与摘要附注',
  `laiyuan` longtext CHARACTER SET utf8 COMMENT '来源',
  `docname` varchar(200) CHARACTER SET utf8 DEFAULT NULL COMMENT '类型'
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `config`
--

DROP TABLE IF EXISTS `config`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `config` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键',
  `name` varchar(100) NOT NULL COMMENT '配置参数名称',
  `value` varchar(100) DEFAULT NULL COMMENT '配置参数值',
  `url` varchar(500) DEFAULT NULL COMMENT 'url',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 COMMENT='配置文件';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `config`
--

LOCK TABLES `config` WRITE;
/*!40000 ALTER TABLE `config` DISABLE KEYS */;
INSERT INTO `config` VALUES (1,'picture1','upload/picture1.jpg',NULL),(2,'picture2','upload/picture2.jpg',NULL),(3,'picture3','upload/picture3.jpg',NULL);
/*!40000 ALTER TABLE `config` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `discussrementushu`
--

DROP TABLE IF EXISTS `discussrementushu`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `discussrementushu` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键',
  `addtime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `refid` bigint(20) NOT NULL COMMENT '关联表id',
  `userid` bigint(20) NOT NULL COMMENT '用户id',
  `avatarurl` longtext COMMENT '头像',
  `nickname` varchar(200) DEFAULT NULL COMMENT '用户名',
  `content` longtext NOT NULL COMMENT '评论内容',
  `score` double DEFAULT NULL COMMENT '评分',
  `reply` longtext COMMENT '回复内容',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='热门图书评论表';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `discussrementushu`
--

LOCK TABLES `discussrementushu` WRITE;
/*!40000 ALTER TABLE `discussrementushu` DISABLE KEYS */;
/*!40000 ALTER TABLE `discussrementushu` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `jiaoyu`
--

DROP TABLE IF EXISTS `jiaoyu`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `jiaoyu` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键',
  `addtime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `title` varchar(200) DEFAULT NULL COMMENT '标题',
  `author` varchar(200) DEFAULT NULL COMMENT '责任者',
  `publisher` varchar(200) DEFAULT NULL COMMENT '出版社',
  `publishyear` varchar(200) DEFAULT NULL COMMENT '出版时间',
  `price` double DEFAULT NULL COMMENT '价格',
  `bookpage` int(11) DEFAULT NULL COMMENT '页数',
  `callnoone` varchar(200) DEFAULT NULL COMMENT '索书号',
  `subjectword` varchar(200) DEFAULT NULL COMMENT '学科主题',
  `adstract` longtext COMMENT '内容与摘要附注',
  `laiyuan` longtext COMMENT '来源',
  `docname` varchar(200) DEFAULT NULL COMMENT '类型',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=59 DEFAULT CHARSET=utf8 COMMENT='教育';
/*!40101 SET character_set_client = @saved_cs_client */;

系统效果图

源码获取

下方名片联系我即可！！


大家点赞、收藏、关注、评论啦 、查看👇🏻获取联系方式👇🏻