ELK实践（一）使用es搭建商品搜索项目

一、了解数据及建模

实验使用的数据是类Airbnb网站上的房屋信息，结构如下所示：

数据建模：
自定义分词器，以实现搜索时的自动补全功能，针对不同的字段定义不同的数据类型，以达到最优的效果：例如dynamic设置为false，对于不需要分词的字段直接设置类型为keyword，不需要检索的字段index设置为false等等。附上具体配置以供参考：

PUT testairbnb
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "analysis": {
        "analyzer": {
          "autosuggest_analyzer": {
            "filter": [
              "lowercase",
              "asciifolding",
              "autosuggest_filter"
            ],
            "tokenizer": "standard",
            "type": "custom"
          },
          "ngram_analyzer": {
            "filter": [
              "lowercase",
              "asciifolding",
              "ngram_filter"
            ],
            "tokenizer": "standard",
            "type": "custom"
          }
        },
        "filter": {
          "autosuggest_filter": {
            "max_gram": "20",
            "min_gram": "1",
            "token_chars": [
              "letter",
              "digit",
              "punctuation",
              "symbol"
            ],
            "type": "edge_ngram"
          },
          "ngram_filter": {
            "max_gram": "9",
            "min_gram": "2",
            "token_chars": [
              "letter",
              "digit",
              "punctuation",
              "symbol"
            ],
            "type": "ngram"
          }
        }
      }
    }
  },
  "mappings": {
    "doc": {
      "dynamic": false,
      "properties": {
        "accommodates": {
          "type": "integer"
        },
        "bathrooms": {
          "type": "integer"
        },
        "bed_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "bedrooms": {
          "type": "integer"
        },
        "beds": {
          "type": "integer"
        },
        "date_from": {
          "type": "date",
          "format": "yyyyMMdd"
        },
        "date_to": {
          "type": "date",
          "format": "yyyyMMdd"
        },
        "has_availability": {
          "type": "boolean"
        },
        "host_image": {
          "type": "keyword",
          "ignore_above": 256,
          "index": false
        },
        "host_name": {
          "type": "text",
          "analyzer": "autosuggest_analyzer",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "image": {
          "type": "keyword",
          "ignore_above": 256,
          "index":false
        },
        "listing_url": {
          "type": "keyword",
          "ignore_above": 256
        },
        "location": {
          "type": "geo_point"
        },
        "name": {
          "type": "text",
          "analyzer": "autosuggest_analyzer",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "price": {
          "type": "float"
        },
        "property_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "room_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

二、导入数据

因为数据为csv格式，所有可以直接用csv插件将数据导入到es中。配置如下：

input{
    file {
        path => "/home/wfs/project/airbnb/airbnb.csv"
        sincedb_path => "/dev/null"
        start_position => "beginning"
    }
}

filter{
    csv{
        columns => ["accommodates","bathrooms","bed_type","bedrooms","beds","date_from","date_o","date_rom","date_to","has_availability","host_image","host_name","image","listing_url","location","name","price","property_type","room_type"]
    }

    mutate{
        remove_field=>["message"]
        lowercase=>["has_availability"]
    }
}
output{
    elasticsearch{
        hosts => ["192.168.20.101:9200","192.168.20.102:9200"]
        index => "testairbnb"
        user => "elastic"
        password => "123456"
    }

    stdout{
        codec=>rubydebug

    }
}

使用lowercase将has_availability字段内容转换为小写

注意数据在导入es后，通过kibna查看会发现image和host_image字段还是以url的形式存在，我们需要做进一步的格式处理来实现以图片的形式存在的效果。

对image和host_image字段做如下处理：

对listing_url 字段做如下处理：

最后实现的效果如下所示：

然后就可以针对不同的条件做搜索查询了。

三、搭建搜索界面

现在就可以通过ReactiveSearch来搭建一个简易的搜索界面了
ReactiveSearch主页：https://opensource.appbase.io/reactive-manual/
github地址：https://github.com/appbaseio/reactivesearch
demo: https://github.com/appbaseio-apps/airbeds

安装：nodejs、yarn

安装依赖关系：
# yarn

启动：默认3000端口
# yarn start

效果图：

这样就可以通过指明时间、价格等条件开进行搜索了。

配置说明：
从github上下载demo包：

git clone https://github.com/appbaseio-apps/airbeds

修改配置连接到es:

# cd airbeds/src/
# ls
App.css  App.js  App.test.js  index.css  index.js  logo.svg  registerServiceWorker.js

修改App.js文件，主要配置如下：

修改es配置文件：

# tail -3 elasticsearch.yml  
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: X-Requested-With, X-Auth-Token, Content-Type, Content-Length, Authorization, Access-Control-Allow-Headers, Accept
#修改es所有节点，然后重启服务

自定义组件：
ReactiveSearch主页上有对每个组件的详细说明及样例，可以直接使用，例如：

打包：
可以将调试好的项目进行打包yarn build，打包完成后将build放到任意有http服务的地方就可以运行。
package.json文件”homepage“设置为空
打包：yarn build
打包到build文件夹下，这样直接将build文件夹的文件放到httph或者nginx家目录下就可以直接80端口访问了。