2.8 Logstash安装与导入数据

安装 Logstash

下载并解压缩 Logstash

笔者下载的是二进制安装包中的 MACOS 版本

Elastic 官方网站：https://www.elastic.co/cn/

Logstash介绍页面：https://www.elastic.co/cn/logstash

Logstash下载页面：https://www.elastic.co/cn/downloads/logstash

准备 logstash.conf 配置文件

运行 Logstash

bin/logstash -f logstash.conf

下载Movielens测试数据集

下载最MovieLens最小测试数据集：https://grouplens.org/datasets/movielens/

编写logstash.conf配置文件

创建 logstash.conf 配置文件并写入一下内容（注意文件路径）

input {
  file {
    path => "/Users/mankvis/Work/elasticsearchStudy/ml-latest-small/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}

尝试通过 Logstash 导入数据

在 logstash/bin 目录下运行一下命令

sudo ./logstash -f logstash.conf

以下为输出信息

查看已经导入的数据

打开 Kibana 的 Dev Tools 并输入 GET /movies/_search

可以看到下载的测试数据集中的数据已经全部倒入 Elasticsearch 中