ELK学习6_Kafka->Logstash->Elasticsearch数据流操作

最新推荐文章于 2024-07-13 11:10:40 发布

Wang_Zhenwei

最新推荐文章于 2024-07-13 11:10:40 发布

阅读量1w

点赞数

分类专栏： ELK 原创文章标签： ELK kafka LOGSTASH elasticsearch

本文链接：https://blog.csdn.net/wang_zhenwei/article/details/49493131

版权

本文介绍了如何配置Logstash从Kafka消费数据，并将其转发到Elasticsearch。主要内容包括Logstash的配置文件设置，Kafka的Zookeeper启动，防火墙关闭，Kafka topic创建，以及Logstash和Elasticsearch的启动与数据验证。通过这个过程，实现了数据从Kafka到Elasticsearch的完整流转。

摘要由CSDN通过智能技术生成

Logstash配置过程

在Logstash中建立input和output的条件：

[hadoop@Slave1 ~]$ cd /usr/local/
[hadoop@Slave1 local]$ cd logstash/
[hadoop@Slave1 logstash]$ ls
bin           CONTRIBUTORS  Gemfile.jruby-1.9.lock  LICENSE     vendor
CHANGELOG.md  Gemfile       lib                     NOTICE.TXT
[hadoop@Slave1 logstash]$ mkdir -p conf
[hadoop@Slave1 logstash]$ ls
bin           conf          Gemfile                 lib      NOTICE.TXT
CHANGELOG.md  CONTRIBUTORS  Gemfile.jruby-1.9.lock  LICENSE  vendor
[hadoop@Slave1 logstash]$ cd conf
[hadoop@Slave1 conf]$ ls
[hadoop@Slave1 conf]$ touch kafkaInput_esOutPut.conf
[hadoop@Slave1 conf]$ ls
kafkaInput_esOutPut.conf
[hadoop@Slave1 conf]$ 
[hadoop@Slave1 conf]$ vim kafkaInput_esOutPut.conf

对kafkaInput_esOutPut.conf进行编辑，本机的具体内容如下：

input {  
kafka {  
  zk_connect => "192.168.154.158:2181,192.168.154.159:2181,192.168.154.160:2181"  
  group_id => "test-consumer-group"  
  topic_id => "logStash"  
  reset_beginning => false # boolean (optional)， default: false  
  consumer_threads => 5  # number (optional)， default: 1  
  decorate_events => true # boolean (optional)， default: false  
  }  
}  
  
filter{  
    mutate{  
            #以:号分割message内容，分割后以数据方式显示。  
            #比如abc:efg => message[0] = abc message[1]=efg  
        split => ["message",","]  
    }  
    #第一个数据的内容中ORA-xxxxx这种格式，则这条内容是ora错误。添加二个字段  
    mutate{  
        add_field =>   {  
                "source_Ip" => "%{[message][0]}"  
                "source_Port" => "%{[message][1]}"  
                "dest_Ip" => "%{[message][2]}"  
                "dest_Port" => "%{[message][3]}"  
                }  
    }  
}  
  
output {  
     elasticsearch {  
       
        host => "localhost"        
    }  
}

分别ssh链接Slave2和Slave3，将kafkaInput_esOutPut.conf拷贝到这两台机器上：

创建conf目录过程：

[hadoop@Slave1 conf]$ ssh Slave2
Last login: Wed Oct 14 10:58:06 2015 from slave1
[hadoop@Slave2 ~]$ cd /usr/local/logstash/
[hadoop@Slave2 logstash]$ mkdir -p conf
[hadoop@Slave2 logstash]$ ls
bin           conf          Gemfile                 lib      NOTICE.TXT
CHANGELOG.md  CONTRIBUTORS  Gemfile.jruby-1.9.lock  LICENSE  vendor
[hadoop@Slave2 logstash]$ exit
logout
Connection to Slave2 closed.
[hadoop@Slave1 conf]$ ssh Slave3
Last login: Wed Oct 14 10:59:01 2015 from slave2
[hadoop@Slave3 ~]$ cd /usr/local/logstash/
[hadoop@Slave3 logstash]$ mkdir -p conf
[hadoop@Slave3 logstash]$ ls
bin           conf          Gemfile                 lib      NOTICE.TXT
CHANGELOG.md  CONT