linux arm64 安装 ELK

linux arm64 安装 ELK

前言:本篇文章主要介绍如何在 linux 环境下安装 ELK 栈,包括 es、kibanalogstash

1. 安装elasticsearch

1.1 下载es linux aarch64 rpm 包

运行命令
rpm -ivh elasticsearch-7.17.4-aarch64.rpm

1.2 创建 es 数据和日志目录

mkdir /home/elasticsearch  
mkdir /home/elasticsearch/data  
mkdir /home/elasticsearch/logs  

1.3 对数据和日志目录授权

chown elasticsearch:elasticsearch -R /home/elasticsearch  
chown elasticsearch:elasticsearch -R /home/elasticsearch/data/     
chown elasticsearch:elasticsearch -R /home/elasticsearch/logs/

1.4 修改 es 配置文件

vi /etc/elasticsearch/elasticsearch.yml

cluster.name: "my-application"  
node.name: node-1   
path.data: /home/elasticsearch/data          # 数据目录,需要赋予权限,否则报错   
path.logs: /home/elasticsearch/logs          # 存储目录,需要赋予权限,否则报错   
network.host: "localhost"                      # 访问地址   
cluster.initial_master_nodes: ["node-1"]

完整配置文件

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /home/es/elasticsearch/data
# Path to log files:
path.logs: /home/es/elasticsearch/logs

# 备份路径
# path.repo: 'D:/ElasticSearch/backup'
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 0.0.0.0
# 当前节点的IP地址,任意IP
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
http.port: 9200
transport.port: 9300 
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# 集群中的节点可以相互发现并选举一个主节点
discovery.seed_hosts: ["localhost"]
#
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# 允许跨域
http.cors.enabled: true
http.cors.allow-origin: "*"
# 安全设置
# xpack.security.enabled: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features. 
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html

1.5 设置开机启动

systemctl enable elasticsearch

1.6 启动、停止、查看状态命令

systemctl start elasticsearch    # 启动    
systemctl stop elasticsearch     # 停止    
systemctl status elasticsearch   # 查看状态

1.7 配置IK分词器

IK Analysis for Elasticsearch
=============================

The IK Analysis plugin integrates Lucene IK analyzer (http://code.google.com/p/ik-analyzer/) into elasticsearch,
support customized dictionary.

Analyzer: `ik_smart` , `ik_max_word` , Tokenizer: `ik_smart` , `ik_max_word`

Versions
--------
1.7.1 Ik 版本与 ES版本
IK versionES version
master7.x -> master
6.x6.x
5.x5.x
1.10.62.4.6
1.9.52.3.5
1.8.12.2.1
1.7.02.1.1
1.5.02.0.0
1.2.61.0.0
1.2.50.90.x
1.1.30.20.x
1.0.00.16.2 -> 0.19.0
1.7.2 下载和编译

elasticsearch-analysis-ik下载地址

在es plugins创建目录ik,将ik压缩包解压到ik目录下
plugings目录:/usr/share/elasticsearch/plugins

解压命令:unzip filename.zip

* optional 1 - download pre-build package from here: https://github.com/medcl/elasticsearch-analysis-ik/releases

    create plugin folder `cd your-es-root/plugins/ && mkdir ik`
    
    unzip plugin to folder `your-es-root/plugins/ik`

* optional 2 - use elasticsearch-plugin to install ( supported from version v5.5.1 ):

    
    ./bin/elasticsearch-plugin install 
	https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.3.0/elasticsearch-analysis-ik-6.3.0.zip
    

   NOTE: replace `6.3.0` to your own elasticsearch version
1.7.3 ik操作示例
1.create a index
curl -XPUT http://localhost:9200/index


2.create a mapping
curl -XPOST http://localhost:9200/index/_mapping -H 'Content-Type:application/json' -d'
{
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            }
        }

}'

3.index some docs

curl -XPOST http://localhost:9200/index/_create/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index/_create/2 -H 'Content-Type:application/json' -d'
{"content":"公安部:各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index/_create/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查:韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index/_create/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'
4.query with highlighting
curl -XPOST http://localhost:9200/index/_search  -H 'Content-Type:application/json' -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'

Result

```json
{
    "took": 14,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 2,
        "max_score": 2,
        "hits": [
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "4",
                "_score": 2,
                "_source": {
                    "content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
                },
                "highlight": {
                    "content": [
                        "<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首 "
                    ]
                }
            },
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "3",
                "_score": 2,
                "_source": {
                    "content": "中韩渔警冲突调查:韩警平均每天扣1艘中国渔船"
                },
                "highlight": {
                    "content": [
                        "均每天扣1艘<tag1>中国</tag1>渔船 "
                    ]
                }
            }
        ]
    }
}
1.7.4 词典配置(Dictionary Configuration)

IKAnalyzer.cfg.xml can be located at {conf}/analysis-ik/config/IKAnalyzer.cfg.xml
or {plugins}/elasticsearch-analysis-ik-*/config/IKAnalyzer.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
	<comment>IK Analyzer 扩展配置</comment>
	<!--用户可以在这里配置自己的扩展字典 -->
	<entry key="ext_dict">custom/mydict.dic;custom/single_word_low_freq.dic</entry>
	 <!--用户可以在这里配置自己的扩展停止词字典-->
	<entry key="ext_stopwords">custom/ext_stopword.dic</entry>
 	<!--用户可以在这里配置远程扩展字典 -->
	<entry key="remote_ext_dict">location</entry>
 	<!--用户可以在这里配置远程扩展停止词字典-->
	<entry key="remote_ext_stopwords">http://xxx.com/xxx.dic</entry>
</properties>
1.7.5 热更新 IK 分词使用方法

目前该插件支持热更新 IK 分词,通过上文在 IK 配置文件中提到的如下配置

 	<!--用户可以在这里配置远程扩展字典 -->
	<entry key="remote_ext_dict">location</entry>
 	<!--用户可以在这里配置远程扩展停止词字典-->
	<entry key="remote_ext_stopwords">location</entry>

其中 location 是指一个 url,比如 http://yoursite.com/getCustomDict,该请求只需满足以下两点即可完成分词热更新。

  1. 该 http 请求需要返回两个头部(header),一个是 Last-Modified,一个是 ETag,这两者都是字符串类型,只要有一个发生变化,该插件就会去抓取新的分词进而更新词库。

  2. 该 http 请求返回的内容格式是一行一个分词,换行符用 \n 即可。

满足上面两点要求就可以实现热更新分词了,不需要重启 ES 实例。

可以将需自动更新的热词放在一个 UTF-8 编码的 .txt 文件里,放在 nginx 或其他简易 http server 下,当 .txt 文件修改时,http server 会在客户端请求该文件时自动返回相应的 Last-Modified 和 ETag。可以另外做一个工具来从业务系统提取相关词汇,并更新这个 .txt 文件。

常见问题

1.自定义词典为什么没有生效?

请确保你的扩展词典的文本格式为 UTF8 编码

2.如何手动安装?

git clone https://github.com/medcl/elasticsearch-analysis-ik
cd elasticsearch-analysis-ik
git checkout tags/{version}
mvn clean
mvn compile
mvn package

拷贝和解压release下的文件: #{project_path}/elasticsearch-analysis-ik/target/releases/elasticsearch-analysis-ik-*.zip 到你的 elasticsearch 插件目录, 如: plugins/ik
重启elasticsearch

3.分词测试失败
请在某个索引下调用analyze接口测试,而不是直接调用analyze接口
如:

curl -XGET "http://localhost:9200/your_index/_analyze" -H 'Content-Type: application/json' -d'
{
   "text":"中华人民共和国MN","tokenizer": "my_ik"
}'
  1. ik_max_word 和 ik_smart 什么区别?

ik_max_word: 会将文本做最细粒度的拆分,比如会将“中华人民共和国国歌”拆分为“中华人民共和国,中华人民,中华,华人,人民共和国,人民,人,民,共和国,共和,和,国国,国歌”,会穷尽各种可能的组合,适合 Term Query;

ik_smart: 会做最粗粒度的拆分,比如会将“中华人民共和国国歌”拆分为“中华人民共和国,国歌”,适合 Phrase 查询。

Changes

自 v5.0.0 起

  • 移除名为 ik 的analyzer和tokenizer,请分别使用 ik_smartik_max_word

Thanks

YourKit supports IK Analysis for ElasticSearch project with its full-featured Java Profiler.
YourKit, LLC is the creator of innovative and intelligent tools for profiling
Java and .NET applications. Take a look at YourKit’s leading software products:
YourKit Java Profiler and
YourKit .NET Profiler.

2.安装Kibana

2.1 rpm包(不成功)

安装、运行成功,但是启动后停止,原因是系统缺少c++依赖包

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.14.1-x86_64.rpm
rpm -ivh kibana-7.14.1-x86_64.rpm

2.1.1 编辑配置文件

vim /etc/kibana/kibana.yml

server.port: 5601   
server.host: "localhost"   
elasticsearch.hosts: ["http://localhost:9200"]    
i18n.locale: "zh-CN"   
2.1.2 设置开机自启动

sudo systemctl daemon-reload
sudo systemctl enable kibana

2.1.3 启动、停止、查看状态命令

sudo systemctl start kibana # 启动
sudo systemctl stop kibana # 停止
sudo systemctl status kibana.service # 查看状态
sudo systemctl cat kibana

2.1.4 卸载kibana

rpm -qa|grep kibana #查看是否安装了kibaba
rpm -e package

2.2 docker 安装kibana

2.2.1 创建配置目录

mkdir /home/kibana
mkdir /home/kibana/config
mkdir /home/kibana/data
mkdir /home/kibana/plugins

2.2.2 Default Kibana configuration for docker target

修改配置文件 vim /home/kibana/config/kibana.yml

server.name: kibana 
# 服务器主机地址
server.host: "0"   
# elasticsearch 主机地址  
elasticsearch.hosts: [ "http://localhost:9200" ]    
xpack.monitoring.ui.container.elasticsearch.enabled: true   
i18n.locale: "zh-CN"    
2.2.3 数据目录授权没有用

sudo chown -R elasticsearch:elasticsearch /home/kibana

2.2.4 运行 kibana

docker run -it -d --privileged --name kibana --restart=always -p 5601:5601 -v /home/kibana/config:/usr/share/kibana/config kibana:7.17.4

docker run -it -d --privileged --name kibana -p 5601:5601 \ -e ELASTICSEARCH_URL=http://172.25.18.243:9200 \ -e ELASTICSEARCH_HOSTS=http://172.25.18.243:9200 \ -e XPACK_SECURITY_ENABLED=false \ -e XPACK_MONITORING_ENABLED=true \ -e XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED=true \ -v /home/kibana/config:/usr/share/kibana/config kibana:7.17.4

3. 安装 logstash

3.1 rpm包

rpm -ivh logstash-7.17.4.rpm

3.2 logstash目录结构

目录说明
/etc/logstash/logstash.yml服务的配置文件。日志,数据目录
/etc/logstash/logstash-sample.conf书写的logstash的配置文件
/etc/logstash/conf.d/书写的logstash的配置文件
/etc/logstash/jvm.optionslogstash jvm配置 这里可以设置为512m

3.3 运行logstash报错

错误1(缺少权限)

unable to load ./jdbc-driver/postgresql-42.2.5.jre7.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)

对jdbc-driver目录及文件进行授权
chown logstash:logstash -R /etc/logstash/conf.d

错误2(相对路径)

将jdbc驱动改为绝对路径
"jdbc_driver_library" => "/etc/logstash/conf.d/jdbc-driver/postgresql-42.5.2.jar"

3.3 logstash配置

/etc/logstash/conf.d/ 文件夹下新建配置文件(logstash.conf),从官网教程(es7.17.4中已经弃用type文档类型,可以配置多个执行文件–*.conf)得知可以建立多个
.conf文件,在运行logstash时可以运行多个配置文件

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

# logstash config
input {

	# 连接PostgreSQL
	jdbc {
      	"jdbc_driver_class" => "org.postgresql.Driver"
      	"jdbc_driver_library" => "/etc/logstash/conf.d/jdbc-driver/postgresql-42.5.2.jar"
	  
	  	# 设定自动重连
      	"jdbc_connection_string" => "jdbc:postgresql://localhost:5433/poi?autoReconnect=true&characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true"
	  
	 	# 连接数据库配置
      	"jdbc_user" => "postgres"
      	"jdbc_password" => "pg123456"
	  
	  	# 时区设置
	  	# jdbc_default_timezone => "Asia/Shanghai"
	  	
	  	# 几何对象查询语句
	  	# -----------------------------------无法转换几何类型,而且必须要写上gid字段--------------------------------------------------
	  	# "statement"  => "select *, st_asgeojson(poi_geom) as geometry from poi2"
	  	# "statement"  => "select gid,name,adress,telephone,category,st_asewkt(poi_geom) as geometry from poi2"
	  	# "statement"  => "select gid,name,adress,telephone,category,st_asgeojson(poi_geom) as geom from poi2"
	  	# -----------------------------------无法转换几何类型,而且必须要写上gid字段--------------------------------------------------
	  
	 	# -----------------------------------以下方式有效-----------------------------------------------------------------------------
	  
	  	# 查询参数 
       #"parameters" => { "is_deleted" => "1"}
	  
	  	# 有效:获取经纬度坐标(String类型),需要转换经纬度类型,字符串是纬度在前,经度在后。	
	  	# "statement"  => "select id,name,address,telephone,category,concat_ws(',',st_y(geom),st_x(geom))as geometry from synchronous"
	  
	  	# 有效:获取经纬度坐标(数组类型)
	  	"statement"  => "select gid,bsm,ysdm,zjnr,bz,st_asgeojson(poi_geom)::json->>'coordinates' as location from poi"
	  
	  	# 开启追踪,如果为true,则需要指定tracking_column,用于增量同步,需是数据库字段
	  	use_column_value => true
	  	# 指定追踪的字段
	  	tracking_column => "gid"
	  
	  	# 开启分页查询
	  	"jdbc_paging_enabled" => true
	  
	  	# 文档最大查询数
	  	"jdbc_fetch_size" => "10000"

	 	# 单次分页查询数
	  	"jdbc_page_size" => "500"
	 
	  	#同步频率:每分钟执行一次
        	"schedule" => "* * * * *"
	  
	  	#处理中文乱码问题
	  	codec => plain { charset => "UTF-8"}
	  
	  	# 连接类型
	  	type => "poi"
    }
	
}

# 字段过滤
filter {
	
    json {
		# 略过无效JSON字段
		# skip_on_invalid_json => true
		# 存储json数据的字段
		source => "location"
		
		# 放置json数据的目标字段
		target => "geometry"
		
		#remove_field => ["@timestamp","@version","geom"]
		#remove_tag => ["@timestamp","@version"]
    }
	mutate { 
		#转换经纬度坐标
		#convert => {"x" => "float"}
		#convert => {"y" => "float"}
		
		# 将geometry数组转换为float类型
		#convert => {
			#"[geometry][0]" => "float"
			#"[geometry][1]" => "float"
		#}
		
		# 移除字段
        #remove_field => "@timestamp"    
        remove_field => "@version"    
    }
}
	
output {
	elasticsearch {
			hosts => ["http://localhost:9200"]
			index => "poi"	
			# 重要:需要指定唯一id
			document_id => "%{gid}"
	
			# 索引映射关系引用文件。配置为模板文件,与同步数据相对应
			# template =>"D:\ElasticSearch\logstash-7.13.4-windows-x86_64\template.json"
			# 为了更好的控制动态模板,需要将其设置为false
			# manage_template => false
		
			# 重写模板
			template_overwrite => true
	   
			# template_name => "test"
	   
			#user => "elastic"
			#password => "changeme"
	}
	stdout {
		codec =>  json_lines
	}
}

3.4 jdbc配置

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html

配置链接

3.5 postgres-jdbc 下载

https://jdbc.postgresql.org/

下载地址

  • 2
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
安装ELKElasticsearch, Logstash, Kibana)之前,需要先安装Docker和Docker Compose,并配置Docker的加速器。 以下是安装步骤: 1. 安装Docker Engine-Community: 在Linux系统上执行以下命令: ``` yum install docker-ce docker-ce-cli containerd.io ``` 2. 配置Docker加速器: 执行以下命令编辑Docker配置文件: ``` vi /etc/docker/daemon.json ``` 在配置文件中添加以下内容,以使用阿里云镜像加速器: ``` { "registry-mirrors": ["https://kgretkdm.mirror.aliyuncs.com"] } ``` 保存并关闭文件。 3. 启动Docker并设置开机自启: 执行以下命令启动Docker服务,并设置开机自启: ``` systemctl start docker && systemctl enable docker ``` 4. 安装Docker Compose: 在Linux系统上执行以下命令: ``` curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose ``` 5. 安装ELK: 在安装完Docker和Docker Compose之后,可以使用Docker Compose来部署ELK。首先,创建一个名为docker-compose.yml的文件,并将以下内容复制到文件中: ``` version: '3' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0 environment: - discovery.type=single-node ports: - 9200:9200 kibana: image: docker.elastic.co/kibana/kibana:7.12.0 ports: - 5601:5601 depends_on: - elasticsearch logstash: image: docker.elastic.co/logstash/logstash:7.12.0 volumes: - ./logstash/config/:/usr/share/logstash/pipeline/ depends_on: - elasticsearch ``` 6. 部署ELK: 在包含docker-compose.yml文件的目录中执行以下命令以启动ELK: ``` docker-compose up -d ``` 这样,就完成了Linux上使用Docker安装ELK的过程。请确保按照步骤正确执行,以确保ELK成功安装和运行。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值