知客旅游服务系统

最新推荐文章于 2024-01-21 14:02:00 发布

甜味橘阳

最新推荐文章于 2024-01-21 14:02:00 发布

阅读量738

点赞数

分类专栏：笔记文章标签： database

本文链接：https://blog.csdn.net/qq_49472679/article/details/124525379

版权

笔记专栏收录该内容

6 篇文章 0 订阅

订阅专栏

知客旅游服务系统

前言
logstash简单使用
一、项目需求概述
二、技术可行性
三、es+hbase构建搜索引擎
四、将es中的数据转移到MongoDB中
五、springboot整合hadoop
springboot+scala开发
vue+echarts构建中国地图
- - vuex基本使用

前言

    通过简单的学习了elasticsearch基础，想做一个简单的、类似于百度京东等搜索项目巩固学习知识。

logstash简单使用

下载使用logstash:
Logstash是一款轻量级的日志搜集处理框架，可以方便的把分散的、多样化的日志搜集起来，并进行自定义的处理，然后传输到指定的位置，比如某个服务器或者文件logstash 可以直接和数据库关联，并且自动根据数据库中的数据更新索引。

https://www.elastic.co/cn/downloads/past-releases/logstash-7-7-0

1.控制台采集数据，控制台输出数据:
logstash -e " intput {stdin {}} output {stdout {}}"
在这里插入图片描述
2.控制台采集数据，控制台输出数据（json）
bin/logstash -e "input { stdin { } } output { stdout {codec => json} }"

3.把命令放到文件中使用:
创建demo.conf,添加： intput {stdin {}} output {stdout {}}
启动 bin/logstash -f demo.conf
在这里插入图片描述

4.将mysql数据同步到es:

a.下载mysql驱动

b.在安装目录下，创建新的配置文件demo2.conf, 添加如下

input {
  # 多张表的同步只需要设置多个jdbc的模块就行了
  jdbc {
      # mysql 数据库链接
      jdbc_connection_string => "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC"
      # 用户名和密码
      jdbc_user => "root"
      jdbc_password => "986836"

      # 驱动
      jdbc_driver_library => "G:/Download/logstash-7.7.0/data/mysql-connector-java-8.0.27.jar"

      # 驱动类名
      jdbc_driver_class => "com.mysql.cj.jdbc.Driver"

      #是否分页,及分页大小
      jdbc_paging_enabled => "true"
      jdbc_page_size => "100"

      #直接执行sql语句
      statement =>"select * from book"
      # 执行的sql 文件路径+名称
      # statement_filepath => "..../test.sql"

      #设置监听间隔  各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
      schedule => "10 * * * *"

      # 索引类型
      #type => "jdbc"
    }

}

output {
  elasticsearch {
        #es的ip和端口
        hosts => ["http://hadoop:9200"]
        #ES索引名称（自己定义的）
        index => "books"
        #文档类型
        document_type => "_doc"
        #设置数据的id为数据库中的字段
        document_id => "%{isbn}"
    }
    stdout {
        codec => json_lines
    }
}

执行：logstash -f …/data/demo2.conf 在这里插入图片描述

一、项目需求概述

此系统满足用户的需求，功能齐全，操作方便，容易上手，主要功能旨在实现一个能够通过用户位置/评分推荐位置附近/性价比高的美食，酒店，休闲娱乐，足疗按摩，KTV，景点和丽人会所等吃喝玩乐的地点推荐到用户。且推荐的方式不仅由就近原则计算，还计算在系统上的商家评分情况。根据不同用户的需求推荐不同的品类项目。该系统不仅优化代码，而且提高了数据处理的速度，从而根据目标用户体验提高了开发过程中的用户舒适度。

考虑到开发效率和构建水平的问题，可以根据用户的视觉体验和使用情况以及系统的响应性来表达系统性能的优缺点。该系统不仅优化代码，而且提高了数据处理的速度，从而根据目标用户体验提高了开发过程中的用户舒适度。
用户可以在系统中通过搜索功能搜寻想要的品类项目，且搜索功能严谨的按照LBS计算用户的实际距离和门店的距离作为评分项，用户可以通过距离，评分以及低价对当前搜索到的商家进行重新排序，对搜索功能做到精确化。
当用户进行查看商家后可以在商家的店铺下进行评论和打分，提高店铺推荐的准确度。
也会通过用户的评论和、评分信息和用户点击行为等进行离线和实时的分析。

二、技术可行性

前后端分离已成为互联网项目开发的业界标准使用方式，本系统也将采用前后端分离vue+springboot架构；要实现本次系统，这里我选择java语言作为后端技术支持，vue+elementUI+echarts等前端架构支持，python爬虫、kettle、flume+kafka作为数据采集支持，mysql、mongdb作为数据存储支持，用elasticsearch实现全文搜索等。在Java语言的基础上，在采用一些中间件作为大数据开发的支持。
在互联网的世界里，任何具备市场因素的软件都具备爆发力，当软件的访问量或者用户量呈爆破式增长时，软件代码是否健壮以及是否有采用一些中间件作为削峰是十分重要的。在本次课题设计中，将充分采用redis作为缓存层框架以达到减少访问数据库次数，以及数据去重效果，地理位置计算等功能。在拥有大量DEMO数据或者真实用户数据时将采用hadoop(mapreduce)+Spark+scala作为日志分析的手段，包含但不限于记录页面访问量，HTTP请求次数，独立IP的访问次数，不同页面访问、刷新量等。同时还会采用Kafka进行流量削峰和消息队列的异步以及实时传输。
在推荐算法中利用als算法计算商家对用户的评分，公式定义为两个连续变量(X,Y)的pearson相关性系数P(x,y)等于它们之间的协方差cov(X,Y)除以它们各自标准差的乘积(σX,σY)。系数的取值总是在-1.0到1.0之间，接近0的变量被成为无相关性，接近1或者-1被称为具有强相关性。通常情况下通过以下取值范围判断变量的相关强度：相关系数 0.8-1.0 极强相关 0.6-0.8 强相关 0.4-0.6 中等程度相关 0.2-0.4 弱相关 0.0-0.2 极弱相关或无相关, 用echarts对数据结果进行可视化，最后通过springboot整合各项框架展示给用户。

三、es+hbase构建搜索引擎

在这里插入图片描述
指定索引库参数
接下来我们来手工指定一下索引库的settings和mapping参数。

{
        "settings":{
                "number_of_shards":5,   #分片数 
                "number_of_replicas":1  #副本数
        },
        "mappings":{
				"dynamic":"strict",
				"_source":{"excludes":["content"]},
				"properties":{
						"title":{"type":"text","analyzer":"ik_max_word"},
						"point":{"type":"keyword"},
						"price":{"type":"double"},
						"level":{"type":"keyword"},
						"address":{"type":"text","analyzer":"ik_max_word"}
						"describe":{"type":"text","analyzer":"ik_max_word"},
						"content":{"type":"text","analyzer":"ik_max_word"}
				}
        }
}

解释：
dynamic参数有4个选项值：
1、true是默认的，表示开启动态映射
2、false表示忽略没有定义的字段，
3、strict表示遇到未知字段时抛出异常
4、runtime表示遇到未知字段时将它作为运行时字段，运行时字段是在ES7.11版本中增加的，运行时字段不会被索引，但是可以从_source中获取运行时字段内容，所以runtime可以适合公共字段已知，并且想兼容未知扩展字段的场景。

在这里插入图片描述
爬取数据并保存到hbase中并建立es索引

package com.niit.data_manager.utils;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.stream.Stream;
/**
 * @author brett
 * @date 2022-05-08 20:25:43
 *
 * Jsoup爬取旅游数据
 */
public class mockData {

    private static final String[] arr = {
            //todo 用alt+光标  选中同时编辑多行
            "https://gs.ctrip.com/html5/you/place/1.html",
            "https://gs.ctrip.com/html5/you/place/2.html",
            "https://gs.ctrip.com/html5/you/place/39.html",
            "https://gs.ctrip.com/html5/you/place/38.html",
            "https://gs.ctrip.com/html5/you/place/21.html",
            "https://gs.ctrip.com/html5/you/place/32.html",
            "https://gs.ctrip.com/html5/you/place/61.html",
            "https://gs.ctrip.com/html5/you/place/100001.html",
            "https://gs.ctrip.com/html5/you/place/100007.html",
            "https://gs.ctrip.com/html5/you/place/100008.html",
            "https://gs.ctrip.com/html5/you/place/100003.html",
            "https://gs.ctrip.com/html5/you/place/100009.html",
            "https://gs.ctrip.com/html5/you/place/100076.html",
            "https://gs.ctrip.com/html5/you/place/100039.html",
            "https://gs.ctrip.com/html5/you/place/100058.html",
            "https://gs.ctrip.com/html5/you/place/100053.html",
            "https://gs.ctrip.com/html5/you/place/100064.html",
            "https://gs.ctrip.com/html5/you/place/100062.html",
            "https://gs.ctrip.com/html5/you/place/100065.html",
            "https://gs.ctrip.com/html5/you/place/100066.html",
            "https://gs.ctrip.com/html5/you/place/100068.html",
            "https://gs.ctrip.com/html5/you/place/100056.html",
            "https://gs.ctrip.com/html5/you/place/100055.html",
            "https://gs.ctrip.com/html5/you/place/100054.html",
            "https://gs.ctrip.com/html5/you/place/100051.html",
            "https://gs.ctrip.com/html5/you/place/100059.html",
            "https://gs.ctrip.com/html5/you/place/100038.html",
            "https://gs.ctrip.com/html5/you/place/100052.html",
            "https://gs.ctrip.com/html5/you/place/100060.html",
            "https://gs.ctrip.com/html5/you/place/100067.html",
            "https://gs.ctrip.com/html5/you/place/100063.html",
            "https://gs.ctrip.com/html5/you/place/100031.html",
            "https://gs.ctrip.com/html5/you/place/100032.html",
            "https://gs.ctrip.com/html5/you/place/19.html",
            "https://gs.ctrip.com/html5/you/place/14.html",
            "https://gs.ctrip.com/html5/you/place/5.html",
            "https://gs.ctrip.com/html5/you/place/508.html",
            "https://gs.ctrip.com/html5/you/place/23.html"
    };

    //todo 百度百科爬取数据
    public static Scenery searchScenery(Scenery scenery, String key) throws IOException {
        ArrayList<String> contentList = new ArrayList<>();
        ArrayList<String> imgList = new ArrayList<>();
        String url = "https://baike.baidu.com/item/";
        Document document = Jsoup.connect(url + key).get();  // 获取页面文档
        String describe = document.getElementsByClass("para").eq(0).text();
        scenery.setDescribe(describe);
        Stream<Element> para = document
                .getElementsByClass("para")
                .stream().limit(6);   // 获取前6条数据
        para.forEach(element -> contentList.add(element.text()));
        scenery.setContent(contentList);
        Stream<Element> img = document.getElementsByClass("lazy-img").stream().limit(6);
        img.forEach(element -> imgList.add(element.attr("data-src")));
        scenery.setImgs(imgList);
        return scenery;
    }

    //todo 设置es索引并将数据内容保存到hbase中
   public static void main(String[] args) throws Exception {
        HBaseUtil.createTable("scenery", "info"); //先创建数据表
        String url = "https://m.ctrip.com/webapp/you/gspoi/sight/";
        for (String s : arr) {
            String s2 = s.split("/")[6];
            List<String> linkList = new ArrayList<>();
            if (s2.length() <= 10) {
                linkList.add(s2);
            } else {
                Document document = Jsoup.connect(s).get();
                Elements sightList = document.getElementsByClass("js_list_view");
                Document parse = Jsoup.parse(sightList.toString());
                Elements js_toJump = parse.getElementsByClass("js_toJump");
                js_toJump.forEach(e -> {
                    String link = e.getElementsByIndexEquals(0).attr("href");
                    if (link.contains("html")) {
                        linkList.add(link.split("/")[6]);
                    }
                });
            }
            for (String s3 : linkList) {
                List<Put> sceneryList = new ArrayList<>();
                Document document = Jsoup.connect(url + s3).get();
                Object[] poiNames = document.getElementsByClass("poiName").stream().limit(5).toArray();
                Object[] distanceFields = document.getElementsByClass("distanceField").stream().limit(5).toArray();
                Object[] poiLevel = document.getElementsByClass("poiLevel").stream().limit(5).toArray();
                Object[] commentScoreNum = document.getElementsByClass("commentScoreNum").stream().limit(5).toArray();
                Object[] priceFont = document.getElementsByClass("priceFont").stream().limit(5).toArray();
                System.out.println("name:" + poiNames.length + " 地址:" + distanceFields.length + " 等级:" + poiLevel.length + " 评分:" + commentScoreNum.length + " 价格:" + priceFont.length);
                if (poiLevel.length == 5 && priceFont.length == 5 && distanceFields.length==5 && commentScoreNum.length==5) {
                    for (int i = 0; i < 5; i++) {
                        HashMap<String, String> indexMap = new HashMap<>();
                        String title = ((Element) poiNames[i]).getElementsByTag("a").attr("title");
                        String addr = ((Element) distanceFields[i]).text();
                        String level = ((Element) poiLevel[i]).text();
                        Double score = new Double(((Element) commentScoreNum[i]).text());
                        Double price = new Double(((Element) priceFont[i]).text());
                        Scenery scenery = new Scenery();
                        scenery.setName(title);
                        scenery.setAddress(addr);
                        scenery.setLevel(level);
                        scenery.setScore(score);
                        scenery.setTicketPrice(price);
                        scenery.setPoint(BaiDuApiUtil.getLatAndLngByAddress(title));
                        Scenery s02 = searchScenery(scenery, title);

                        // 设置数据在elasticsearch中的索引
                        indexMap.put("title", s02.getName());
                        indexMap.put("point", s02.getPoint());
                        indexMap.put("price", s02.getTicketPrice().toString());
                        indexMap.put("level", s02.getLevel());
                        indexMap.put("score", s02.getScore().toString());
                        indexMap.put("address", s02.getAddress());
                        indexMap.put("describe", s02.getDescribe());

                        //向索引库中添加
                        ESUtil.addIndex("scenery", s02.getName(), indexMap);

                        Put put = new Put(Bytes.toBytes(s02.getName()));
                        // 将数据入库到hbase
                        put.addColumn(Bytes.toBytes("info"), Bytes.toBytes("img"), Bytes.toBytes(s02.getImgs().toString()));
                        put.addColumn(Bytes.toBytes("info"), Bytes.toBytes("content"), Bytes.toBytes(s02.getContent().toString()));
                        sceneryList.add(put);
                    }
                    HBaseUtil.put2HBaseList("scenery", sceneryList);
                    try {
                        Thread.sleep(600);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        }
    }
}

如图hbase保存数据成功！
在这里插入图片描述
索引建立成功！

Scenery风景实体类

package com.example.ssm.domain;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
/**
 * @author brett
 * @date 2022-05-09 15:21:23
 */
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Scenery {
    private StringBuilder title;  // 景点名
    private String point; //经纬度
    private Double ticketPrice; //票价
    private String level; //景区等级
    private Double score; //评分
    private String address; //地址
    private StringBuilder describe; // 描述
    private String firstImg; //封面图
    private List<String> content; //景点简介
    private List<String> imgs; // 景点图片
}

通过地名名称查找对应的经纬度

package com.niit.data_manager.utils;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.StandardCharsets;
/**
 * @author brett
 * @date 2022-05-15 20:29:39
 */
public class BaiDuApiUtil {

    private static final String AK = "q4WRaidEQFHCqNC7Izof4GxWe8zAEbDj";

    public static String getLatAndLngByAddress(String addr) {
        String address = "";
        String result = "";
        try {
            address = java.net.URLEncoder.encode(addr, "UTF-8");
        } catch (UnsupportedEncodingException e1) {
            e1.printStackTrace();
        }
        String url = "https://api.map.baidu.com/geocoding/v3/?address=" + address +
                "&output=json&ak="+AK+"&callback=showLocation ";
        URL myURL = null;
        URLConnection httpsConn;
        //进行转码
        try {
            myURL = new URL(url);
        } catch (MalformedURLException ignored) {
            ignored.printStackTrace();
        }
        try {
            assert myURL != null;
            httpsConn = myURL.openConnection();
            if (httpsConn != null) {
                InputStreamReader insr = new InputStreamReader(
                        httpsConn.getInputStream(), StandardCharsets.UTF_8);
                BufferedReader br = new BufferedReader(insr);
                String data = br.readLine();
                String[] split = data.replace("{", ":").replace("}", "::").split("::");
                if (split.length >=3){
                    result = split[2];
                }
                insr.close();
            }
        } catch (IOException ignored) {
            ignored.printStackTrace();
        }
        return result;
    }
}

HBaseUtil工具类

package com.example.ssm.utils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class HBaseUtil {
    private static Connection connection;

    //初始化创建连接
    static {
        try{
            connection = getConnection();
        }catch(IOException e){
            e.printStackTrace();
        }
    }
    //todo 创建hbase连接
    public static Connection getConnection() throws IOException {
        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum","192.168.111.137");
        conf.set("hbase.zookeeper.property.clientPort","2181");
        return ConnectionFactory.createConnection(conf);
    }
    //TODO 创建hbase表
    public static void addTable(String tableName,String... cfs) throws IOException {
        //获取admin
        Admin admin = connection.getAdmin();
        //创建列族builder
        List<ColumnFamilyDescriptor> list = new ArrayList<>();
        for (String cf:cfs){
            ColumnFamilyDescriptor columnFamilyDescriptor = ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes(cf)).build();
            list.add(columnFamilyDescriptor);
        }
        //创建TableBuilder
        TableDescriptor tabBuilder = TableDescriptorBuilder.newBuilder(TableName.valueOf(Bytes.toBytes(tableName)))
                .setColumnFamilies(list).build();
        admin.createTable(tabBuilder);
        admin.close();
    }
    //todo 向hbase中添加一条数据
    public static void addData(String tabName,String rowKey,String cfName,String column,String value) throws IOException {
        Table table = connection.getTable(TableName.valueOf(Bytes.toBytes(tabName)));
        Put put = new Put(Bytes.toBytes(rowKey));
        put.addColumn(Bytes.toBytes(cfName),Bytes.toBytes(column),Bytes.toBytes(value));
        table.put(put);
        table.close();
    }
    //todo 向hbase中添加一组数据
    public static void addListData(String tabName,List<Put> list) throws IOException {
        Table table = connection.getTable(TableName.valueOf(tabName));
        table.put(list);
        table.close();
    }
    //todo 根据rowKey获取hbase中的数据
    public static Map<String,String> getDataByRowkey(String tabName,String rowkey) throws IOException {

        Table table = connection.getTable(TableName.valueOf(Bytes.toBytes(tabName)));
        Get get = new Get(Bytes.toBytes(rowkey));
        Result result = table.get(get);
        HashMap<String, String> map = new HashMap<>();
        map.put("title",rowkey);
        System.out.println("title:"+rowkey);
        for (Cell c:result.listCells()){
            //获取列信息
            byte[] column = CellUtil.cloneQualifier(c);
            //获取值
            byte[] value = CellUtil.cloneValue(c);
            map.put(new String(column),new String(value));
        }
        return map;
    }
}

ESUtil工具类

package com.example.ssm.utils;
import com.example.ssm.domain.Scenery;
import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchType;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.MatchQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
 * @author brett
 * @date 2022-05-09 18:49:34
 *
 * es工具类
 */
public class ESUtil {

    private ESUtil(){};

    private static RestHighLevelClient client;
    private static String host = "192.168.111.137";
    private static int port = 9200;

    static {
        //获取es连接
        RestClientBuilder restBuilder = RestClient.builder(new HttpHost(host, port, "http"));
        client = new RestHighLevelClient(restBuilder);
    }

    //获取es连接
    public RestHighLevelClient getClient(){
        return client;
    }
    // 建立索引
    public static void addIndex(String index, String id, Map<String,String> map) throws IOException {
        IndexRequest indexRequest = new IndexRequest(index);
        indexRequest.id(id);
        indexRequest.source(map);
        // 执行
        client.index(indexRequest, RequestOptions.DEFAULT);
    }
    //todo 全文检索功能
    public static Map<String,Object> search(String key,String index,int start,int row) throws IOException {

        if (start<=1){
            start=1;
        }
        SearchRequest searchRequest = new SearchRequest();
        // 指定索引库
        searchRequest.indices(index);
        //指定searchType
        searchRequest.searchType(SearchType.QUERY_THEN_FETCH);
        //组装查询条件
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //如果传递了搜索参数，则拼接查询条件
//        if (StringUtils.isNotBlank(key)){ //StringUtils.isNotBlank()判断参数是否为空
//            searchSourceBuilder.query(QueryBuilders.multiMatchQuery(key,"title"));
//        }

        //多条件查询
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
        MatchQueryBuilder q1 = QueryBuilders.matchQuery("title", key);
        MatchQueryBuilder q2 = QueryBuilders.matchQuery("describe", key);
        boolQuery.should(q1);
        boolQuery.should(q2);
        searchSourceBuilder.query(boolQuery);
        //分页
        searchSourceBuilder.from((start-1)*row);
        searchSourceBuilder.size(row);
        //高亮
        //设置高亮字段
        HighlightBuilder highlightBuilder = new HighlightBuilder()
                .field("title")
                .field("describe");  //支持多个高亮字段
        //设置高亮字段的前缀后缀
        highlightBuilder.preTags("<font color='red'>");
        highlightBuilder.postTags("</font>");
        searchSourceBuilder.highlighter(highlightBuilder);

        //指定查询条件
        searchRequest.source(searchSourceBuilder);

        //执行查询操作
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //存储返回给页面的数据
        HashMap<String, Object> map = new HashMap<>();
        //获取查询到的返回结果
        SearchHits hits = searchResponse.getHits();
        //获取数据总量
        long numHits = hits.getTotalHits().value;
        map.put("count",numHits);
        List<Scenery> sceneries = new ArrayList<>();
        //获取具体内容
        SearchHit[] searchHits = hits.getHits();
        //迭代解析具体内容
        for (SearchHit hit:searchHits){
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            StringBuilder title = new StringBuilder(sourceAsMap.get("title").toString());
            StringBuilder describe = new StringBuilder(sourceAsMap.get("describe").toString());
            String firstImg = sourceAsMap.get("firstImg").toString();
            String score;
            try {
                score = sourceAsMap.get("score").toString();
            } catch (NullPointerException e) {
                score = "3.0";
                e.printStackTrace();
            }
            String address = "";
            try {
                address = sourceAsMap.get("address").toString();
            } catch (NullPointerException e) {
                e.printStackTrace();
            }
            String level="";
            try {
                level = sourceAsMap.get("level").toString();
            } catch (NullPointerException e) {
                level = "3A";
                e.printStackTrace();
            }
            String point="";
            try {
                point = sourceAsMap.get("point").toString();
            } catch (NullPointerException e) {
                e.printStackTrace();
            }
            String price="";
            try {
                price = sourceAsMap.get("price").toString();
            } catch (NullPointerException e) {
                price = "30.0";
                e.printStackTrace();
            }

            //获取高亮字段
            Map<String, HighlightField> highlightFields = hit.getHighlightFields();
            //获取title字段的高亮内容
            HighlightField titleHighLight = highlightFields.get("title");
            if (titleHighLight != null){
                title = new StringBuilder();
                Text[] fragments = titleHighLight.getFragments();
                for (Text t:fragments){
                    title.append(t);
                }
            }
            //获取describe的高亮字段内容
            HighlightField describeHighLight = highlightFields.get("describe");
            if (describeHighLight!=null){
                describe = new StringBuilder();
                Text[] fragments = describeHighLight.getFragments();
                for (Text t:fragments){
                    describe.append(t);
                }
            }
            //把信息封装到Scenery对象中
            Scenery scenery = new Scenery();
            scenery.setDescribe(describe);
            scenery.setTitle(title);
            scenery.setDescribe(describe);
            scenery.setPoint(point);
            scenery.setFirstImg(firstImg);
            scenery.setLevel(level);
            scenery.setTicketPrice(Double.parseDouble(price));
            scenery.setAddress(address);
            scenery.setScore(new Double(score));
            sceneries.add(scenery);
        }
        map.put("dataList",sceneries);
        return map;
    }
}

对应的controller

package com.example.ssm.controller;
import com.example.ssm.utils.ESUtil;
import com.example.ssm.utils.HBaseUtil;
import org.springframework.web.bind.annotation.*;
import java.io.IOException;
import java.util.Map;
import java.util.regex.Pattern;
/**
 * @author brett
 * @date 2022-05-9 14:54:03
 */
@RestController
@RequestMapping("/ES")
public class ESController {

    @GetMapping("/searchData")
    public Map<String,Object> getData(@RequestParam String key,@RequestParam int pageNo)
            throws IOException {
      return ESUtil.search(key,"scenery",pageNo,10);
    }

    @GetMapping("/selectDataByHbase")
    public Map<String,String> getHbaseData(@RequestParam String rowkey)
            throws IOException {
        String regex = "^[\\u4e00-\\u9fa5]*$"; //匹配中文
        StringBuilder key = new StringBuilder();
        String[] arr = rowkey.split("");
        for (String s:arr){
            if (Pattern.matches(regex,s)){
                key.append(s);
            }
        }
        return HBaseUtil.getDataByRowkey("scenery", key.toString());
    }
}

注：运行前应该在虚拟机环境中启动elasticsearch，hadoop,hbase环境。

四、将es中的数据转移到MongoDB中

下载logstash-output-mongdb插件：
https://github.com/logstash-plugins/logstash-output-mongodb
打开后一定要切换版本至3.1.5，然后下载zip包

在这里插入图片描述
解压到logstash目录下

修改logstash里的 Gemfile文件

最后一行加上gem “logstash-output-mongodb”, :path => “logstash-output-mongodb-master”

用管理员打开CMD
切换到logstash 的bin目录运行：logstash-plugin install --no-verify
在这里插入图片描述
下载完成后查看安装版本：logstash-plugin list --verbose

创建配置文件mongdb-out.conf：

input {
  stdin {
  }
  elasticsearch {
 #ESIP地址与端口
  hosts => "hadoop:9200"
  #ES索引名称（自己定义的）
  index => "scenery"
  #自增ID编号
  # document_id => "%{id}"
  #定时字段 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
  #schedule => "* * * * *"
  #设定ES索引类型
  type => "message"
  }
}

filter {
json {
  source => "message"
  remove_field => ["message"]
  }
}

#目标mongodb地址信息
  output {
  stdout { codec => rubydebug }
  mongodb {
#目标mongodb集合
  collection => "scenery"
#目标库名称
  database => "mydb"
  uri => "mongodb://hadoop:27017"
  }
}

执行命令：logstash -f …/data/mongodb-out.conf
在这里插入图片描述
导入数据成功！！！
在MongoDB中查看数据：

数据导入成功

五、springboot整合hadoop

1.在pom.xml文件中导入hadoop依赖

		<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.3</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.3</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.7.3</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

hadoop和springboot冲突的主要有两个，一个是slf4j的日志包，一个是Tomcat 的servlet-api包，去掉这两个依赖就可以成功运行springboot了
在application.yml配置文件中添加基本星信息在这里插入图片描述

@Value("${hadoop.name-node}")
    private String nameNode;

    @Test
    public void createDir() throws URISyntaxException, IOException, InterruptedException {
        //1获取文件系统
        Configuration configuration = new Configuration();
        FileSystem fileSystem = FileSystem.get(new URI(nameNode), configuration,"root");
        //2执行操作 
        fileSystem.mkdirs(new Path("/jianxuhui"));
        fileSystem.close();
        System.out.println("结束！！！");
    }

2.从MongoDB中读取数据作为mapreduce输入源

添加mongodb-hadoop依赖：

<!-- mongodb整合hadoop,可以用mongodb作为mapreduce数据输入源 -->
        <!-- https://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core -->
        <dependency>
            <groupId>org.mongodb.mongo-hadoop</groupId>
            <artifactId>mongo-hadoop-core</artifactId>
            <version>2.0.2</version>
        </dependency>

eg:从mongodb中将一个集合中的数据通过mapreduce处理，然后将结果返回到mongodb存储

package com.example.ssm.MR;
import com.mongodb.BasicDBObject;
import com.mongodb.hadoop.MongoInputFormat;
import com.mongodb.hadoop.MongoOutputFormat;
import com.mongodb.hadoop.io.BSONWritable;
import com.mongodb.hadoop.util.MongoConfigUtil;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.bson.BSONObject;
import java.io.IOException;

/**
 * @author brett
 * @date 2022-05-24 11:46:25
 *
 * 以mongodb数据作为mapreduce数据输入源
 */
public class mongodbSource {
    static class mongodbMapper extends Mapper<Object, BSONObject, Text, BSONWritable>{
        @Override
        //实现mapper,mongodb读取出来就是一个BSONObject
        protected void map(Object key, BSONObject value, Context context) throws IOException, InterruptedException {
            System.out.println(key);
            context.write(new Text(key.toString()),new BSONWritable(value));
        }
    }
    static class mongodbReducer extends Reducer<Text,BSONWritable,Text,BSONWritable>{
        @Override
        //不做任何处理，直接将数据读取出来在存入到mongodb中
        protected void reduce(Text key, Iterable<BSONWritable> values, Context context) throws IOException, InterruptedException {
            for (BSONWritable value:values){
                context.write(key,value);
            }
        }
    }

    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
        args = new String[]{"mongodb://192.168.111.137:27017/mydb.info",
        "mongodb://192.168.111.137:27017/mydb.result"};
        Configuration configuration = new Configuration();
        MongoConfigUtil.setInputURI(configuration,args[0]);
        MongoConfigUtil.setOutputURI(configuration,args[1]);
        //可以构建条件BasicQuery查询
        BasicDBObject query = new BasicDBObject();
        MongoConfigUtil.setQuery(configuration,query);
        MongoConfigUtil.setCreateInputSplits(configuration,false);
        Job job = Job.getInstance(configuration,"Hadoop Mongodb");
        job.setJarByClass(mongodbSource.class);
        job.setMapperClass(mongodbMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(BSONWritable.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(BSONWritable.class);
        job.setReducerClass(mongodbReducer.class);
        job.setInputFormatClass(MongoInputFormat.class);
        job.setOutputFormatClass(MongoOutputFormat.class);
        job.setNumReduceTasks(1);
        System.exit(job.waitForCompletion(true)?0:1);
    }
}

结果：如图传输数据成功！
在这里插入图片描述

3、java计算两位置经纬度之间的距离

package com.example.ssm.utils;
/**
 * @author brett
 * @date 2022-05-26 22:28:51
 *
 * 通过经纬度计算距离
 */
public class CalculateDistance {
    /*
       S = acos( sin(第1个点纬度 * ∏ / 180) * sin(第2个点纬度 * ∏ / 180) + cos(第1个点纬度 * ∏ / 180) *
        cos(第2个点纬度 * ∏ / 180) * cos(第2个点经度 * ∏ / 180 - 第1个点经度 * ∏ / 180)) * 地球半径
    */
    //地球半径，单位km
    private static final double EARTH_RADIUS = 6378.137;

    public static double getDistance(Double lng1,Double lat1,Double lng2,Double lat2){
        //纬度
        double lat01 = Math.toRadians(lat1); //Math.toRadians()将数据转化为弧度制
        double lat02 = Math.toRadians(lat2);
        //经度
        double lng01 = Math.toRadians(lng1);
        double lng02 = Math.toRadians(lng2);
        // 计算两点距离的公式
        double s = Math.acos(Math.sin(lat01)*Math.sin(lat02)+Math.cos(lng01)*Math.cos(lng02)*Math.cos(lng02-lng01));
        // 弧长乘地球半径, 返回单位: 千米
        s =  s * EARTH_RADIUS;
        return s;
    }
}

根据经纬度查找省份城市

//传入经纬度, 返回查询的地区, lng: 经度, lat: 纬度
    public static String findByLatAndLng(String lat, String lng) {
        try {
            //移除坐标前后的 空格
            /*lng = lng.trim();
            lat = lat.trim();*/
            CloseableHttpClient httpClient = HttpClients.createDefault();
            // url中的ak值要替换成自己的:
            String url = "http://api.map.baidu.com/reverse_geocoding/v3/?ak=xeiyKzum1WHj3SE39P216gDI9cnIYbgi&output=json&coordtype=wgs84ll&location=" + lat + "," + lng;
            //System.out.println(url);
            HttpGet httpGet = new HttpGet(url);
            CloseableHttpResponse response = httpClient.execute(httpGet);
            HttpEntity httpEntity = response.getEntity();
            String json = EntityUtils.toString(httpEntity);
            Map<String, Object> result = JSONObject.parseObject(json, Map.class);
            if (result.get("status").equals(0)) {
                Map<String, Object> resultMap = (Map<String, Object>) result.get("result");
                resultMap = (Map<String, Object>) resultMap.get("addressComponent");
                String province = (String) resultMap.get("province");
                String city = (String) resultMap.get("city");
                return  province + city;
            }

        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

4、SpringBoot封装MapReduce的WordCount做字符统计

package com.example.ssm.MR;
import org.apache.hadoop.fs.FileSystem;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.net.URI;
/**
 * @author brett
 * @date 2022-05-28 10:13:21
 *
 * hadoop配置类
 */
@Configuration
public class HadoopConfig {

    @Bean("hdfsConfig")
    public org.apache.hadoop.conf.Configuration hdfsChannerl(){
        org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
        conf.set("dfs.replication","1");
        conf.set("dfs.client.use.datanode.hostname","true");
        conf.set("mapred.job.tracker", "hdfs://192.168.111.137:8020/");
        conf.set("fs.defaultFS", "hdfs://192.168.111.137:8020/");
        System.setProperty("HADOOP_USER_NAME","root");
        return conf;
    }

    @Bean("fileSystem")
    public FileSystem createFs(@Qualifier("hdfsConfig") org.apache.hadoop.conf.Configuration conf){
        FileSystem fs = null;
        try {
            URI uri = new URI("hdfs://192.168.111.137:8020/");
            fs = FileSystem.get(uri,conf);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return  fs;
    }
}

hadoopController类

@RestController
@RequestMapping(value = "/hadoop")
public class HadoopController {

    @Qualifier("hdfsConfig")
    @Autowired
    private org.apache.hadoop.conf.Configuration conf;

	@GetMapping(value = "/reduce")
    public Boolean reduce() throws IOException, ClassNotFoundException, InterruptedException {
        Job job = Job.getInstance(conf);
        job.setMapperClass(WordCountMapper.class);
        job.setReducerClass(WordCountReducer.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.setInputPaths(job, new Path(template.getNameSpace())); //输入路径
        FileOutputFormat.setOutputPath(job, new Path("/statics")); //输出路径
        return job.waitForCompletion(true);
    }
}

在这里插入图片描述

5、vue+百度地图api定位当前位置

//js文件导出
export function MP(ak) {
    return new Promise(function (resolve, reject) {
        window.onload = function () {
            resolve(window.BMap)//插入script标签后 会在window上挂一BMap属性,此为跨域获取的数据
        };
        var script = document.createElement("script");
        script.type = "text/javascript";
        script.src = "http://api.map.baidu.com/api?v=2.0&ak=" + ak + "&callback=init";
        script.onerror = reject;
        document.head.appendChild(script);//插入此标签后 会在window上挂一BMap属性,此为跨域获取的数据
    })
}

//在vue 生命周期函数created()中引用getCity函数
getCity() {
            this.$nextTick(()=>{
                MP(this.ak).then(BMap=> {
                    //在此调用api
                    var geolocation = new BMap.Geolocation();
                    geolocation.getCurrentPosition(a=>{
                        console.log(a)
                        this.city = a.address.province+a.address.city;
                        this.latitude = a.latitude;
                        this.longitude = a.longitude;
                    })
                })
            })
        }

springboot+scala开发

添加依赖：

<!--添加依赖-->
<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
</dependency>
...
<!-- 添加插件 -->
<plugin>
    <groupId>net.alchim31.maven</groupId>
    <artifactId>scala-maven-plugin</artifactId>
    <version>3.2.1</version>
    <executions>
        <execution>
            <id>compile-scala</id>
            <phase>compile</phase>
            <goals>
                <goal>add-source</goal>
                <goal>compile</goal>
            </goals>
        </execution>
        <execution>
            <id>test-compile-scala</id>
            <phase>test-compile</phase>
            <goals>
                <goal>add-source</goal>
                <goal>testCompile</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <recompileMode>incremental</recompileMode>
        <scalaVersion>${scala.version}</scalaVersion>
        <args>
            <arg>-deprecation</arg>
        </args>
        <jvmArgs>
            <jvmArg>-Xms64m</jvmArg>
            <jvmArg>-Xmx1024m</jvmArg>
        </jvmArgs>
    </configuration>
</plugin>

vue+echarts构建中国地图

下载地图离线包:
在这里插入图片描述

<template>
    <div class="content">
        <div ref="charts" style="width: 70%; height: 800px"></div>
    </div>
</template>
<script>
    import zhongguo from "@/assets/china.json"
    export default {
        created () {
            this.$nextTick(() => {
                this.initCharts();
            })
        },
        methods: {
            initCharts() {
                const charts = this.$echarts.init(this.$refs["charts"]);
                const option = {
                    title: {  //标题样式
                        text: '各省份景点访问量',
                        x: "center",
                        textStyle: {
                            fontSize: 18,
                            color: "black"
                        },
                    },
                    // 背景颜色
                    backgroundColor: "#eff3e9",
                    // 提示浮窗样式
                    tooltip: {
                        //数据项图形触发
                        trigger: 'item',
                        //提示框浮层的背景颜色。 (鼠标悬浮后的提示框背景颜色)
                        backgroundColor: "white",
                        //字符串模板(地图): {a}（系列名称），{b}（区域名称），{c}（合并数值）,{d}（无）
                        formatter: '地区：{b}<br/>客流量：{c}'
                    },
                    //视觉映射组件
                    visualMap: {
                        top: 'center',
                        left: 'left',
                        // 数据的范围
                        min: 10,
                        max: 500000,
                        text: ['High', 'Low'],
                        realtime: true,  //拖拽时，是否实时更新
                        calculable: true,  //是否显示拖拽用的手柄
                        inRange: {
                            // 颜色分布
                            color: ['lightskyblue', 'yellow', 'orangered']
                        }
                    },
                    series: [
                        {
                            name: '模拟数据',
                            type: 'map',
                            mapType: 'china',
                            roam: false,//是否开启鼠标缩放和平移漫游
                            itemStyle: {//地图区域的多边形 图形样式
                                normal: {//是图形在默认状态下的样式
                                    label: {
                                        show: true,//是否显示标签
                                        textStyle: {
                                            color: "black"
                                        }
                                    }
                                },
                                zoom: 1.5,  //地图缩放比例,默认为1
                                emphasis: {//是图形在高亮状态下的样式,比如在鼠标悬浮或者图例联动高亮时
                                    label: { show: true }
                                }
                            },
                            top: "3%",//组件距离容器的距离
                            bottom:"0%",
                            left:"3%",
                            right:"3%",
                            data: [
                                { name: '北京', value: 350000 },
                                { name: '天津', value: 120000 },
                                { name: '上海', value: 300000 },
                                { name: '重庆', value: 92000 },
                                { name: '河北', value: 25000 },
                                { name: '河南', value: 20000 },
                                { name: '云南', value: 500 },
                                { name: '辽宁', value: 3050 },
                                { name: '黑龙江', value: 80000 },
                                { name: '湖南', value: 2000 },
                                { name: '安徽', value: 24580 },
                                { name: '山东', value: 40629 },
                                { name: '新疆', value: 36981 },
                                { name: '江苏', value: 13569 },
                                { name: '浙江', value: 24956 },
                                { name: '江西', value: 15194 },
                                { name: '湖北', value: 41398 },
                                { name: '广西', value: 41150 },
                                { name: '甘肃', value: 17630 },
                                { name: '山西', value: 27370 },
                                { name: '内蒙古', value: 27370 },
                                { name: '陕西', value: 97208 },
                                { name: '吉林', value: 88290 },
                                { name: '福建', value: 19978 },
                                { name: '贵州', value: 94485 },
                                { name: '广东', value: 89426 },
                                { name: '青海', value: 35484 },
                                { name: '西藏', value: 97413 },
                                { name: '四川', value: 54161 },
                                { name: '宁夏', value: 56515 },
                                { name: '海南', value: 54871 },
                                { name: '台湾', value: 48544 },
                                { name: '香港', value: 49474 },
                                { name: '澳门', value: 34594 }
                            ]
                        }
                    ]
                };
                // 地图注册，第一个参数的名字必须和option.geo.map一致
               this.$echarts.registerMap("china",zhongguo)

                charts.setOption(option);
            },
        },
    };
</script>

结果：
在这里插入图片描述

vuex基本使用

cnpm install vuex --save

导入vuex包

import Vuex from 'vuex'
Vue.use(Vuesx)

创建store对象

const store = new Vuex.Store({
	states:{
		count:0
	}
})
export default store

将对象挂载到全局vue实例中

const vue = new Vue({
	el:"#app",
	rander :h=>h(app),
	store
	//将创建的共享数据对象，挂载到vue实例中
	//所有组件，就可以直接从store中获取全局的数据了
})

Mutation用于变更Store中的数据。
①只能通过mutation变更Store数据，不可以直接操作Store中的数据。
②通过这种方式虽然操作起来稍微繁琐一些，但是可以集中监控所有数据的变化。

甜味橘阳

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
知客旅游服务系统

用es做一个简单搜索项目前言一、项目构建二、爬取数据1.引入库前言通过简单的学习了elasticsearch基础，想做一个简单的、类似于百度京东等搜索项目巩固学习知识。一、项目构建二、爬取数据1.引入库........................
复制链接

扫一扫