基于Search Guard的Elasticsearch安全认证和授权配置

Install the Search Guard Plugin for your Elasticsearch version

Disable shard allocation

curl -XPUT 'http://es1:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

检查设置是否生效，运行如下命令

curl -XGET 'http://es1:9200/_cluster/settings?pretty'

若输出如下结果，则说明配置成功

{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "none"
        }
      }
    }
  },
  "transient" : { }
}

Stop all nodes

采用如下命令查询出ES服务的进程号（占用9200端口和9300端口的进程）

netstat -tnlp
kill -9 进程号

Download

关于Search Guard Plugin版本和Elasticsearch版本之间的对应关系，请参考：https://docs.search-guard.com/latest/search-guard-versions

这里以elasticsearch-6.6.1为例进行说明，因为生产服务器不能联网，这里采用离线方式安装，下载Search Guard Plugin

下载地址：https://oss.sonatype.org/service/local/repositories/releases/content/com/floragunn/search-guard-6/6.6.1-24.3/search-guard-6-6.6.1-24.3.zip

Install

将search-guard-6-6.6.1-24.3.zip拷贝到es所在服务器节点，进入<ES_HOME>/bin目录，执行如下命令进行安装

./elasticsearch-plugin install -b file:///usr/local/hadoop/elasticsearch/search-guard-6-6.6.1-24.3.zip

具体的安装步骤可参考官网：https://docs.search-guard.com/6.x-24/search-guard-installation#adding-the-tls-configuration

Generate all required TLS certificates

参考：https://docs.search-guard.com/6.x-24/offline-tls-tool

下载offline TLS tool

下载地址：https://search.maven.org/#search|ga|1|a%3A"search-guard-tlstool"

这里下载的是search-guard-tlstool-1.7.tar.gz

tar -zxvf search-guard-tlstool-1.7.tar.gz -C search-guard-tlstool //解压

Config node- and certificate configuration settings

进入/config目录

cp example.yml tlsconfig.yml
vi  tlsconfig.yml

Generate Root and Intermediate CA

修改后的Self-generated certificate authority部分配置如下

ca:
   root:
      # The distinguished name of this CA. You must specify a distinguished name.   
      dn: CN=root.ca.tomas.com,OU=CA,O=tomas,DC=tomas,DC=com

      # The size of the generated key in bits
      keysize: 2048

      # The validity of the generated certificate in days from now
      validityDays: 3650
      
      # Password for private key
      #   Possible values: 
      #   - auto: automatically generated password, returned in config output; 
      #   - none: unencrypted private key; 
      #   - other values: other values are used directly as password   
      pkPassword: root-ca-password
      
      # The name of the generated files can be changed here
      file: root-ca.pem
      
   # If you want to use an intermediate certificate as signing certificate,
   # please specify its parameters here. This is optional. If you remove this section,
   # the root certificate will be used for signing.         
   intermediate:
      # The distinguished name of this CA. You must specify a distinguished name.
      dn: CN=signing.ca.tomas.com,OU=CA,O=tomas,DC=tomas,DC=com
   
      # The size of the generated key in bits   
      keysize: 2048
      
      # The validity of the generated certificate in days from now      
      validityDays: 3650
  
      pkPassword: intermediate-ca-password

      file: intermediate-ca.pem

      # If you have a certificate revocation list, you can specify its distribution points here      
      # crlDistributionPoints: URI:https://raw.githubusercontent.com/floragunncom/unittest-assets/master/revoked.crl

进入/tools目录，运行如下命令生成Root和Intermediate证书

./sgtlstool.sh -c ../config/tlsconfig.yml -ca

若提示如下信息，表示生成Root和intermediate证书成功

Root certificate and signing certificate have been sucessfully created.

生成的证书默认存放在/tools/out目录下

Generate Node and Client CA

修改后的Default values and global settings部分配置如下

defaults:

      # The validity of the generated certificate in days from now
      validityDays: 3650 
      
      # Password for private key
      #   Possible values: 
      #   - auto: automatically generated password, returned in config output; 
      #   - none: unencrypted private key; 
      #   - other values: other values are used directly as password   
      pkPassword: auto      
      
      # Specifies to recognize legitimate nodes by the distinguished names
      # of the certificates. This can be a list of DNs, which can contain wildcards.
      # Furthermore, it is possible to specify regular expressions by
      # enclosing the DN in //. 
      # Specification of this is optional. The tool will always include
      # the DNs of the nodes specified in the nodes section.            
      nodesDn:
       - "CN=*.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com"
      # - 'CN=node.other.com,OU=SSL,O=Test,L=Test,C=DE'
      # - 'CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE'
      # - 'CN=elk-devcluster*'
      # - '/CN=.*regex/' 

      # If you want to use OIDs to mark legitimate node certificates, 
      # the OID can be included in the certificates by specifying the following
      # attribute
      
      nodeOid: "1.2.3.4.5.5"

      # The length of auto generated passwords            
      generatedPasswordLength: 12
      
      # Set this to true in order to generate config and certificates for 
      # the HTTP interface of nodes
      httpsEnabled: true
      
      # Set this to true in order to re-use the node transport certificates
      # for the HTTP interfaces. Only recognized if httpsEnabled is true
      
      reuseTransportCertificatesForHttp: false
      
      # Set this to true to enable hostname verification
      #verifyHostnames: false
      
      # Set this to true to resolve hostnames
      #resolveHostnames: false

修改后的Nodes部分配置如下

nodes:
  - name: es1
    dn: CN=es1.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
    dns: es1
  - name: es2
    dn: CN=es2.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
    dns: es2
  - name: es3
    dn: CN=es3.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
    dns: es3
  - name: es4
    dn: CN=es4.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
    dns: es4

修改后的Clients部分配置如下

clients:
  - name: spock
    dn: CN=spock.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
  - name: kirk
    dn: CN=kirk.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com
    admin: true

进入/tools目录，运行如下命令生成node和client证书

./sgtlstool.sh -c ../config/tlsconfig.yml -crt

若看到如下提示信息，证明生成node和client证书成功

Using signing certificate: /usr/local/hadoop/elasticsearch/search-guard-tlstool/tools/out/intermediate-ca.pem
Created 4 node certificates.
Passwords for the private keys of the node certificates have been auto-generated. The passwords are stored in the config snippet files.
Created 2 client certificates.
Passwords for the private keys of the client certificates have been auto-generated. The passwords are stored in the file "client-certificates.readme"

Config SSL

由于ElasticSearch节点之间通讯默值非加密，造成数据不安全，Search Guard强制ElasticSearch节点之间通讯为加密方式。

distribute cetificates

将上述生成的相关证书拷贝到相应ES节点的<ES_HOME>/config目录
例如将如下文件拷贝到hostname为es1的节点

root-ca.pem
root-ca.key
intermediate-ca.pem
intermediate-ca.key
es1.pem
es1.key
es1_http.pem
es1_http.key
es1_elasticsearch_config_snippet.yml
kirk.pem
kirk.key
spock.pem
spock.key

并采用如下命令将上述文件的权限更改为0600

chmod 600 root* intermediate* kirk* spock* es1*

将如下文件拷贝到hostname为es2的节点

root-ca.pem
root-ca.key
intermediate-ca.pem
intermediate-ca.key
es2.pem
es2.key
es2_http.pem
es2_http.key
es2_elasticsearch_config_snippet.yml
kirk.pem
kirk.key
spock.pem
spock.key

并采用如下命令将上述文件的权限更改为0600，同上,其他节点操作同上。

chmod 600 root-ca.* intermediate-ca.* es1* spock.* kirk.*

Change the permission on directory of <ES_HOME>/config

采用如下命令将ES集群各节点的<ES_HOME>/config目录的权限改为0700

chmod 700 <ES_HOME>/config

Disable X-Pack security

从6.3.0版本开始，Elasticsearch 和 Kibana绑定了X-Pack和OSS，如果Elasticsearch使用的是6.3.0以上的版本，需要关闭X-Pack security，即在ES集群各节点的<ES_HOME>/config/elasticsearch.yml文件中新增如下配置

xpack.security.enabled: false

Adding the TLS configuration

将es1_elasticsearch_config_snippet.yml文件中内容新增配置到hostname为es1的节点<ES_HOME>/config/elasticsearch.yml文件中

# This is a configuration snippet for the node es1
# This snippet needs to be inserted into the file config/elasticsearch.yml of the respective node.
# If the config file already contains SearchGuard configuration, this needs to be replaced.
# Furthermore, you need to copy the files referenced below into the same directory.
# Please refer to http://docs.search-guard.com/latest/configuring-tls for further configuration of your installation.

searchguard.ssl.transport.pemcert_filepath: es1.pem
searchguard.ssl.transport.pemkey_filepath: es1.key
searchguard.ssl.transport.pemkey_password: lZoD27XLoBry
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: es1_http.pem
searchguard.ssl.http.pemkey_filepath: es1_http.key
searchguard.ssl.http.pemkey_password: sualxs42muY1
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.authcz.admin_dn:
- CN=kirk.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com

searchguard.cert.oid: 1.2.3.4.5.5

将es2_elasticsearch_config_snippet.yml文件中内容新增配置到hostname为es2的节点<ES_HOME>/config/elasticsearch.yml文件中

# This is a configuration snippet for the node es2
# This snippet needs to be inserted into the file config/elasticsearch.yml of the respective node.
# If the config file already contains SearchGuard configuration, this needs to be replaced.
# Furthermore, you need to copy the files referenced below into the same directory.
# Please refer to http://docs.search-guard.com/latest/configuring-tls for further configuration of your installation.

searchguard.ssl.transport.pemcert_filepath: es2.pem
searchguard.ssl.transport.pemkey_filepath: es2.key
searchguard.ssl.transport.pemkey_password: d0aVkFO9ZW0W
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: es2_http.pem
searchguard.ssl.http.pemkey_filepath: es2_http.key
searchguard.ssl.http.pemkey_password: 153IQlKc2bul
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.authcz.admin_dn:
- CN=kirk.tomas.com,OU=Ops,O=tomas,DC=tomas,DC=com

searchguard.cert.oid: 1.2.3.4.5.5

Enable the REST management API

在ES集群各节点的<ES_HOME>/config/elasticsearch.yml文件中新增如下配置

searchguard.restapi.roles_enabled: ["sg_all_access"]

ES集群所有节点都配置好后，重启ES集群

Restart Elasticsearch

采用如下命令启动ES集群所有节点

cd /usr/local/hadoop/elasticsearch/
./bin/elasticsearch -d

Re-enable shard allocation by using sgadmin

使用admin证书、root证书，并结合sgadmin工具或curl来使能shard allocation，这里使用sgadmin，具体命令如下

cd /usr/local/hadoop/elasticsearch/plugins/search-guard-6/tools
chmod u+x sgadmin.sh //使sgadmin.sh有可执行权限
./sgadmin.sh -icl -nhnv -h es1 --enable-shard-allocation -cacert ../../../config/root-ca.pem -cert ../../../config/kirk.pem -key ../../../config/kirk.key -keypass zNbkgrtwctow

其中，zNbkgrtwctow为admin证书的私钥的密码，可以在client-certificates.readme文件中找到

Initializing Search Guard

具体参考：https://docs.search-guard.com/latest/sgadmin

Search Guard的配置（包括users、roles和permissions）都是存储在ES集群里的一个index中，这样就可以实现修改配置后立即生效，不用重启集群。Search Guard的配置信息需要通过sgadmin工具来加载到ES的一个index中（默认这个index是不存在的），所以sgadmin主要用来初始化index和配置认证和授权方式。

Change the permissions on that script and give it execution rights

chmod +x plugins/search-guard-7/tools/sgadmin.sh

提示，可以通过运行如下命令查看sgadmin.sh工具所有的命令行选项

./sgadmin.sh

Using sgadmin with PEM certificates

在ES集群的任一节点（以es1节点为例）执行如下命令

./sgadmin.sh -h es1 -cd ../sgconfig/ -icl -nhnv -cacert ../../../config/root-ca.pem -cert ../../../config/kirk.pem -key ../../../config/kirk.key -keypass zNbkgrtwctow

若看到如下提示信息，证明初始化集群成功

Search Guard Admin v6
Will connect to hadoopSvr3:9300 ... done
Elasticsearch Version: 6.6.1
Search Guard Version: 6.6.1-24.3
Connected as CN=kirk.wxtomas.com,OU=Ops,O=wxtomas Com\, Inc.,DC=wxtomas,DC=com
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: ES-cluster
Clusterstate: GREEN
Number of nodes: 2
Number of data nodes: 2
searchguard index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/local/elasticsearch-6.6.1/plugins/search-guard-6/sgconfig
Will update 'sg/config' with ../sgconfig/sg_config.yml 
   SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with ../sgconfig/sg_roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with ../sgconfig/sg_roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with ../sgconfig/sg_internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with ../sgconfig/sg_action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

Test the installation

visiting https://es1:9200

如果提示需要输入用户名和密码，输入admin/admin

Display information about the currently logged in user

visiting https://es1:9200/_searchguard/authinfo?pretty

Search Guard Health Check

为了检查Search Guard是否已经正确安装，并且已经在正常运行，检查Search Guard的健康状态，在浏览器地址栏输入如下地址
https://:9200/_searchguard/health
若输出结果如下

{"message":null,"mode":"strict","status":"UP"}

则说明Search Guard运行正常。

Accessing data Check

curl -u admin:admin -k -XGET 'https://<hostname>:9200/_cat?pretty'

至此，Search Guard 认证基本已经完成了,接下来关于账号权限的创建与管理，可以在 Kibana 直接操作。

Config Kibana

具体参考：https://docs.search-guard.com/6.x-24/kibana-plugin-installation

Installing the Search Guard Plugin

1、下载与Kibana版本匹配的Search Guard Kibana Plugin
具体下载地址：https://oss.sonatype.org/service/local/repositories/releases/content/com/floragunn/search-guard-kibana-plugin/6.6.1-18.4/search-guard-kibana-plugin-6.6.1-18.4.zip

2、关闭Kibana

3、将下载的插件search-guard-kibana-plugin-6.6.1-18.4.zip上传到Kibana的安装目录，并执行如下命令安装插件

bin/kibana-plugin install file:usr/local/kibana-6.6.1-linux-x86_64/search-guard-kibana-plugin-6.6.1-18.4.zip

若提示如下信息，则表示安装成功

Attempting to transfer from file:usr/local/kibana-6.6.1-linux-x86_64/search-guard-kibana-plugin-6.6.1-18.4.zip
Transferring 1046761 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

Disable X-Pack Security

在kibana安装目录下的config/kibana.yml文件中修改配置如下

xpack.security.enabled: false

Configuring the Kibana server user

在kibana安装目录下的config/kibana.yml文件中修改配置如下

elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"

Setting up SSL/TLS

因为在Elasticsearch REST layer开启了TLS，所以kibana访问Elasticsearch的协议需要从http改为https，在config/kibana.yml文件中修改配置如下

elasticsearch.hosts: ["https://ip:9200"]

Configuring the Root CA

有两种配置：
一是关闭 证书验证，在config/kibana.yml文件中修改配置如下

elasticsearch.ssl.verificationMode: none

二是提供root CA，在config/kibana.yml文件中修改配置如下

elasticsearch.ssl.certificateAuthorities: "/path/to/your/root-ca.pem"
elasticsearch.ssl.verificationMode: certificate

然后启动kibana。

Config logstash

具体参考：https://docs.search-guard.com/6.x-24/elasticsearch-logstash-search-guard

Check permissions for the logstash user

在Search Guard secured cluster中默认配置一个logstash user，因为Search Guard中已经包含了一个logstash用户，此处不再单独配置，在<ES_HOME>/plugins/search-guard-6/sgconfig/sg_roles.yml文件中查看sg_logstash用户的权限如下

# For logstash and beats
sg_logstash:  
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - indices:admin/template/get
    - indices:admin/template/put
  indices:
    'logstash-*':
      '*':
        - CRUD
        - CREATE_INDEX
    '*beat*':
      '*':
        - CRUD
        - CREATE_INDEX

也可以在用admin账号登录kibana，进入Search Guard菜单，在Permissions and Roles部分的Search Guard Roles查看。

set up a logstash user

将logstash用户配置到logstash的配置文件logstash.conf中elasticsearch输出部分，具体如下

output {
    elasticsearch {
       user => logstash
       password => logstash
       ...
    }
}

Setting up TLS/SSL

因为前面在ES中配置启动了REST层的TLS，所以需要配置logstash使用https来与elasticsearch通信，将root证书拷贝到logstash-6.4.2/config目录下，并在logstash的配置文件logstash.conf中elasticsearch输出部分配置如下

    ssl => true
    ssl_certificate_verification => false
    cacert => "/usr/local/logstash-6.4.2/config/root-ca.pem"
    index => "logstash-rs1_26_mongodlog"

注意：如果是自定义索引名，索引名必须匹配’logstash-*‘开头或’beat’，否则logstash用户无权限。
然后重启logstash。