Elasticsearch6.4专题之6:Set-up X-Pack

X-Pack

Installing X-Pack

By default, when you install Elasticsearch, X-Pack is installed. See Installing Elasticsearch.

Set up X-Pack

X-Pack is an Elastic Stack extension that provides security, alerting, monitoring, reporting, machine learning, and many other capabilities. By default, when you install Elasticsearch, X-Pack is installed.

If you want to try all of the X-Pack features, you can start a 30-day trial. At the end of the trial period, you can purchase a subscription to keep using the full functionality of the X-Pack components. For more information, see https://www.elastic.co/subscriptions.

Configuring monitoring

Configuring monitoring in Elasticsearch

If you enable the Elastic monitoring features in your cluster, you can optionally collect metrics about Elasticsearch. By default, monitoring is enabled but data collection is disabled.

Advanced monitoring settings enable you to control how frequently data is collected, configure timeouts, and set the retention period for locally-stored monitoring indices. You can also adjust how monitoring data is displayed.

1. To collect monitoring data about your Elasticsearch cluster:

   1. Verify that the xpack.monitoring.enabled and xpack.monitoring.collection.enabled settings are true on each node in the cluster.
      
      By default, data collection is disabled. For more information, see Monitoring Settings.

   2. Optional: Specify which indices you want to monitor.
      
      By default, the monitoring agent collects data from all Elasticsearch indices. To collect data from particular indices, configure the xpack.monitoring.collection.indices setting. You can specify multiple indices as a comma-separated list or use an index pattern to match multiple indices. For example:

      xpack.monitoring.collection.indices: logstash-*, index1, test2
      

      You can prepend + or - to explicitly include or exclude index names or patterns. For example, to include all indices that start with test except test3, you could specify +test*,-test3.

   3. Optional: Specify how often to collect monitoring data.
      
      The default value for the xpack.monitoring.collection.interval setting 10 seconds. See Monitoring Settings.

2. Optional: Configure your cluster to route monitoring data from sources such as Kibana, Beats, and Logstash to a monitoring cluster:

   1. Verify that xpack.monitoring.collection.enabled settings are true on each node in the cluster.
   2. Configure monitoring across the Elastic Stack.

3. Identify where to store monitoring data.
   
   By default, Elasticsearch monitoring features use a local exporter that indexes monitoring data on the same cluster. See Default exporters and Local Exporters.

   Alternatively, you can use an http exporter to send data to a separate monitoring cluster. See HTTP Exporters.

   important:
   The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.

   For more information about typical monitoring architectures, see How Monitoring Works.

4. If Elasticsearch security features are enabled and you are using an http exporter to send data to a dedicated monitoring cluster:

   1. Create a user on the monitoring cluster that has the remote_monitoring_agent built-in role. For example, the following request creates a remote_monitor user that has the remote_monitoring_agent role:

   POST /_xpack/security/user/remote_monitor
   {
       "password" : "changeme",
       "roles" : [ "remote_monitoring_agent"],
       "full_name" : "Internal Agent For Remote Monitoring"
   }
   或者
   curl -X POST "localhost:9200/_xpack/security/user/remote_monitor?pretty" 
   -H 'Content-Type: application/json' -d'
   {
     "password" : "changeme",
     "roles" : [ "remote_monitoring_agent"],
     "full_name" : "Internal Agent For Remote Monitoring"
   }'
   

   2. On each node in the cluster that is being monitored, configure the http exporter to use the appropriate credentials when data is shipped to the monitoring cluster.

      If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol in the host setting. You must also include the CA certificate in each node’s trusted certificates in order to verify the identities of the nodes in the monitoring cluster.

      The following example specifies the location of the PEM encoded certificate with the certificate_authorities setting:

      xpack.monitoring.exporters:
        id1:
          type: http
          host: ["https://es-mon1:9200", "https://es-mon2:9200"]
          auth:
          #The username and password parameters provide the user credentials.
            username: remote_monitor 
            password: changeme
          ssl:
            certificate_authorities: [ "/path/to/ca.crt" ]
        id2:
          type: local   
      

      Alternatively, you can configure trusted certificates using a truststore (a Java Keystore file that contains the certificates):

      xpack.monitoring.exporters:
        id1:
          type: http
          host: ["https://es-mon1:9200", "https://es-mon2:9200"]
          auth:
            username: remote_monitor
            password: changeme
          ssl:
            truststore.path: /path/to/file
            truststore.password: password
        id2:
          type: local
      

5. If the Elasticsearch security features are enabled and you want to visualize monitoring data in Kibana, you must create users that have access to the Kibana indices and permission to read from the monitoring indices.
   
   You set up Monitoring UI users on the cluster where the monitoring data is stored, that is to say the monitoring cluster. To grant all of the necessary permissions, assign users the monitoring_user and kibana_user roles. For more information, see Mapping users and groups to roles.

6. Optional: Configure the indices that store the monitoring data.

Configuring Indices for Monitoring

Index templates are used to configure the indices that store the monitoring data collected from a cluster.

You can retrieve the templates through the _template API:

GET /_template/.monitoring-*
或
curl -X GET "localhost:9200/_template/.monitoring-*?pretty"

By default, the template configures one shard and one replica for the monitoring indices. To override the default settings, add your own template:

1. Set the template pattern to .monitoring-*.
2. Set the template order to 1. This ensures your template is applied after the default template, which has an order of 0.
3. Specify the number_of_shards and/or number_of_replicas in the settings section.

For example, the following template increases the number of shards to five and the number of replicas to two.

PUT /_template/custom_monitoring
{
    "index_patterns": ".monitoring-*",
    "order": 1,
    "settings": {
        "number_of_shards": 5,
        "number_of_replicas": 2
    }
}
或
curl -X PUT "localhost:9200/_template/custom_monitoring?pretty" 
-H 'Content-Type: application/json' -d'
{
    "index_patterns": ".monitoring-*",
    "order": 1,
    "settings": {
        "number_of_shards": 5,
        "number_of_replicas": 2
    }
}'

Important：
Only set the number_of_shards and number_of_replicas in the settings section. Overriding other monitoring template settings could cause your monitoring dashboards to stop working correctly.

Configuring a Tribe Node to Work with Monitoring

WARN：
Deprecated in 5.4.0.
The tribe node is deprecated in favour of Cross-cluster search and will be removed in Elasticsearch 7.0.

If you connect to a cluster through a tribe node, and you want to monitor the tribe node, then you will need to install X-Pack on that node as well.

With this configuration, the tribe node is included in the node count displayed in the Monitoring UI, but is not included in the node list because it does not export any data to the monitoring cluster.

To include the tribe node in the monitoring data, enable Monitoring data collection at the tribe level:

node.name: my-tribe-node1

tribe:
  on_conflict: prefer_cluster1
  c1:
    cluster.name: cluster1
    discovery.zen.ping.unicast.hosts: [ "cluster1-node1:9300", "cluster1-node2:9300", "cluster1-node2:9300" ]
    ## enable data collection from the tribe node using a Local Exporter.
    xpack.monitoring.enabled: true 
  c2:
    cluster.name: cluster2
    discovery.zen.ping.unicast.hosts: [ "cluster2-node3:9300", "cluster2-node3:9300", "cluster2-node3:9300" ]
    ## Enable data collection from the tribe node using an HTTP Exporter.
    xpack.monitoring: 
      enabled: true
      exporters:
        id1:
          type: http
          host: [ "monitoring-cluster:9200" ]

When you enable data collection from the tribe node, it is included in both the node count and node list.

Monitoring Settings in Elasticsearch

By default, monitoring is enabled but data collection is disabled. To enable data collection, use the xpack.monitoring.collection.enabled setting.

You can configure these monitoring settings in the elasticsearch.yml file. Some of them can also be set across the cluster by using the cluster update settings API.

Tip：Cluster settings take precedence over settings in the elasticsearch.yml file.

To adjust how monitoring data is displayed in the monitoring UI, configure xpack.monitoring settings in kibana.yml. To control how monitoring data is collected from Logstash, configure xpack.monitoring settings in logstash.yml.

For more information, see Monitoring the Elastic Stack.

General Monitoring Settings

xpack.monitoring.enabled

Set to true (default) to enable Elasticsearch X-Pack monitoring for Elasticsearch on the node.

To enable data collection, you must also set xpack.monitoring.collection.enabled to true. Its default value is false.

Monitoring Collection Settings

refer to https://www.elastic.co/guide/en/elasticsearch/reference/6.4/monitoring-settings.html

The xpack.monitoring.collection settings control how data is collected from your Elasticsearch nodes.

xpack.monitoring.collection.enabled

[6.3.0] Added in 6.3.0.Set to true to enable the collection of monitoring data. When this setting is false (default), Elasticsearch monitoring data is not collected and all monitoring data from other sources such as Kibana, Beats, and Logstash is ignored.

You can update this setting through the Cluster Update Settings API.

xpack.monitoring.collection.interval

Setting to -1 to disable data collection has been deprecated. [6.3.0]

Controls how often data samples are collected. Defaults to 10s. If you modify the collection interval, set the xpack.monitoring.min_interval_seconds option in kibana.yml to the same value.

• You can update this setting through the Cluster Update Settings API.

xpack.monitoring.collection.cluster.stats.timeout

Sets the timeout for collecting the cluster statistics. Defaults to 10s.

xpack.monitoring.collection.indices

Controls which indices Monitoring collects data from. Defaults to all indices. Specify the index names as a comma-separated list, for example test1,test2,test3. Names can include wildcards, for example test*.You can explicitly include or exclude indices by prepending + to include the index, or - to exclude the index. For example, to include all indices that start with test except test3, you could specify +test*,-test3.

You can update this setting through the Cluster Update Settings API.

xpack.monitoring.collection.index.stats.timeout

Sets the timeout for collecting index statistics. Defaults to 10s.

xpack.monitoring.collection.index.recovery.active_only

Controls whether or not all recoveries are collected. Set to true to collect only active recoveries. Defaults to false.

xpack.monitoring.collection.index.recovery.timeout

Sets the timeout for collecting the recovery information. Defaults to 10s.

xpack.monitoring.history.duration

Sets the retention duration beyond which the indices created by a Monitoring exporter are automatically deleted. Defaults to 7d (7 days).

This setting has a minimum value of 1d (1 day) to ensure that something is being monitored, and it cannot be disabled.

Important:This setting currently only impacts local-type exporters. Indices created using the http exporter will not be deleted automatically.

If both X-Pack monitoring and Watcher are enabled, you can use this setting to affect the Watcher cleaner service too. For more information, see the xpack.watcher.history.cleaner_service.enabled setting in the Watcher Settings.

xpack.monitoring.exporters

Configures where the agent stores monitoring data. By default, the agent uses a local exporter that indexes monitoring data on the cluster where it is installed. Use an HTTP exporter to send data to a separate monitoring cluster. For more information, see Local Exporter Settings, HTTP Exporter Settings, and How Monitoring Works.

Local Exporter Settings

The local exporter is the default exporter used by Monitoring. As the name is meant to imply, it exports data to the local cluster, which means that there is not much needed to be configured.

If you do not supply any exporters, then Monitoring will automatically create one for you. If any exporter is provided, then no default is added.

xpack.monitoring.exporters.my_local:
  type: local

type

The value for a Local exporter must always be local and it is required.

use_ingest

Whether to supply a placeholder pipeline to the cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.

cluster_alerts.management.enabled

Whether to create cluster alerts for this cluster. The default value is true. To use this feature, Watcher must be enabled. If you have a basic license, cluster alerts are not displayed.

HTTP Exporter Settings

The following lists settings that can be supplied with the http exporter. All settings are shown as what follows the name you select for your exporter:

xpack.monitoring.exporters.my_remote:
  type: http
  host: ["host:port", ...]

type

    The value for an HTTP exporter must always be http and it is required.

host

    Host supports multiple formats, both as an array or as a single value. Supported formats include hostname, hostname:port, http://hostname http://hostname:port, https://hostname, and https://hostname:port. Hosts cannot be assumed. The default scheme is always http and the default port is always 9200 if not supplied as part of the host string.

xpack.monitoring.exporters:
  example1:
    type: http
    host: "10.1.2.3"
  example2:
    type: http
    host: ["http://10.1.2.4"]
  example3:
    type: http
    host: ["10.1.2.5", "10.1.2.6"]
  example4:
    type: http
    host: ["https://10.1.2.3:9200"]

auth.username

    The username is required if a auth.password is supplied.

auth.password

    The password for the auth.username.

connection.timeout

    The amount of time that the HTTP connection is supposed to wait for a socket to open for the request. The default value is 6s.

connection.read_timeout

    The amount of time that the HTTP connection is supposed to wait for a socket to send back a response. The default value is 10 * connection.timeout (60s if neither are set).

ssl

    Each HTTP exporter can define its own TLS / SSL settings or inherit them. See the TLS / SSL section below.

proxy.base_path

    The base path to prefix any outgoing request, such as /base/path (e.g., bulk requests would then be sent as /base/path/_bulk). There is no default value.

headers

    Optional headers that are added to every request, which can assist with routing requests through proxies.

xpack.monitoring.exporters.my_remote:
  headers:
    X-My-Array: [abc, def, xyz]
    X-My-Header: abc123

    Array-based headers are sent n times where n is the size of the array. Content-Type and Content-Length cannot be set. Any headers created by the Monitoring agent will override anything defined here.

index.name.time_format

    A mechanism for changing the default date suffix for the, by default, daily Monitoring indices. The default value is YYYY.MM.DD, which is why the indices are created daily.

use_ingest

    Whether to supply a placeholder pipeline to the monitoring cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.

cluster_alerts.management.enabled

    Whether to create cluster alerts for this cluster. The default value is true. To use this feature, Watcher must be enabled. If you have a basic license, cluster alerts are not displayed.

cluster_alerts.management.blacklist

    Prevents the creation of specific cluster alerts. It also removes any applicable watches that already exist in the current cluster.

    You can add any of the following watch identifiers to the blacklist:

• elasticsearch_cluster_status
• elasticsearch_version_mismatch
• elasticsearch_nodes
• kibana_version_mismatch
• logstash_version_mismatch
• xpack_license_expiration

For example: [“elasticsearch_version_mismatch”,“xpack_license_expiration”].

X-Pack monitoring TLS/SSL Settings

You can configure the following TLS/SSL settings. If the settings are not configured, the Default TLS/SSL Settings are used.

xpack.monitoring.exporters.$NAME.ssl.supported_protocols

Supported protocols with versions. Valid protocols: SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2. Defaults to TLSv1.2, TLSv1.1, TLSv1. Defaults to the value of xpack.ssl.supported_protocols.

xpack.monitoring.exporters.$NAME.ssl.verification_mode

Controls the verification of certificates. Valid values are none, certificate, and full. See xpack.ssl.verification_mode for a description of these values. Defaults to the value of xpack.ssl.verification_mode.

xpack.monitoring.exporters.$NAME.ssl.cipher_suites

Supported cipher suites can be found in Oracle’s Java Cryptography Architecture documentation. Defaults to the value of xpack.ssl.cipher_suites.

X-Pack monitoring TLS/SSL Key and Trusted Certificate Settings

The following settings are used to specify a private key, certificate, and the trusted certificates that should be used when communicating over an SSL/TLS connection. A private key and certificate are optional and would be used if the server requires client authentication for PKI authentication. If none of the settings below are specified, the Default TLS/SSL Settings are used.

PEM Encoded Files

Java Keystore Files

When using Java keystore files (JKS), which contain the private key, certificate and certificates that should be trusted, use the following settings:

xpack.monitoring.exporters.$NAME.ssl.keystore.path

Path to the keystore that holds the private key and certificate.

xpack.monitoring.exporters.$NAME.ssl.keystore.password

Password to the keystore.

xpack.monitoring.exporters.$NAME.ssl.keystore.secure_password (Secure)

Password to the keystore.

xpack.monitoring.exporters.$NAME.ssl.keystore.key_password

Password for the private key in the keystore. Defaults to the same value as xpack.monitoring.exporters.$NAME.ssl.keystore.password.

xpack.monitoring.exporters.$NAME.ssl.keystore.secure_key_password (Secure)

Password for the private key in the keystore.

xpack.monitoring.exporters.$NAME.ssl.truststore.path

Path to the truststore file.

xpack.monitoring.exporters.$NAME.ssl.truststore.password

Password to the truststore.

xpack.monitoring.exporters.$NAME.ssl.truststore.secure_password (Secure)

Password to the truststore.

PKCS#12 Files

PKCS#11 Tokens

Configuring security in Elasticsearch

X-Pack security enables you to easily secure a cluster. With X-Pack security, you can password-protect your data as well as implement more advanced security measures such as encrypting communications, role-based access control, IP filtering, and auditing. For more information, see Securing the Elastic Stack.

To use X-Pack security in Elasticsearch:

1. Verify that you are using a license that includes the X-Pack security feature.

If you want to try all of the X-Pack features, you can start a 30-day trial. At the end of the trial period, you can purchase a subscription to keep using the full functionality of the X-Pack components. For more information, see https://www.elastic.co/subscriptions and License Management.

2. Verify that the xpack.security.enabled setting is true on each node in your cluster. If you are using a trial license, the default value is false. For more information, see Security Settings in Elasticsearch.
3. If you plan to run Elasticsearch in a Federal Information Processing Standard (FIPS) 140-2 enabled JVM, see FIPS 140-2.
4. Configure Transport Layer Security (TLS/SSL) for internode-communication.

Note

This requirement applies to clusters with more than one node and to clusters with a single node that listens on an external interface. Single-node clusters that use a loopback interface do not have this requirement. For more information, see Encrypting Communications.

1. Generate node certificates for each of your Elasticsearch nodes.
2. Enable TLS on each Elasticsearch node.

5. If it is not already running, start Elasticsearch.
6. Set the passwords for all built-in users.

X-Pack security provides built-in users to help you get up and running. The elasticsearch-setup-passwords command is the simplest method to set the built-in users’ passwords for the first time.

For example, you can run the command in an “interactive” mode, which prompts you to enter new passwords for the built-in users:

bin/elasticsearch-setup-passwords interactive

For more information about the command options, see elasticsearch-setup-passwords.

Important
The elasticsearch-setup-passwords command uses a transient bootstrap password that is no longer valid after the command runs successfully. You cannot run the elasticsearch-setup-passwords command a second time. Instead, you can update passwords from the Management > Users UI in Kibana or use the security user API.

7. Choose which types of realms you want to use to authenticate users.

• Configure an Active Directory realm.
• Configure a file realm.
• Configure an LDAP realm.
• Configure a native realm.
• Configure a PKI realm.
• Configure a SAML realm.
• Configure a Kerberos realm.

8. Set up roles and users to control access to Elasticsearch.

For example, to grant John Doe full access to all indices that match the pattern events* and enable him to create visualizations and dashboards for those indices in Kibana, you could create an events_admin role and assign the role to a new johndoe user.

curl -XPOST -u elastic 'localhost:9200/_xpack/security/role/events_admin' -H "Content-Type: application/json" -d '{
  "indices" : [
    {
      "names" : [ "events*" ],
      "privileges" : [ "all" ]
    },
    {
      "names" : [ ".kibana*" ],
      "privileges" : [ "manage", "read", "index" ]
    }
  ]
}'

curl -XPOST -u elastic 'localhost:9200/_xpack/security/user/johndoe' -H "Content-Type: application/json" -d '{
  "password" : "userpassword",
  "full_name" : "John Doe",
  "email" : "john.doe@anony.mous",
  "roles" : [ "events_admin" ]
}'

9. Enable auditing to keep track of attempted and successful interactions with your Elasticsearch cluster:

   a. Add the following setting to elasticsearch.yml on all nodes in your cluster:

   xpack.security.audit.enabled: true
   

   For more information, see Auditing Security Events and Auditing Settings.

   b. Restart Elasticsearch.

By default, events are logged to a dedicated elasticsearch-access.log file in ES_HOME/logs. You can also store the events in an Elasticsearch index for easier analysis and control what events are logged.

Encrypting communications in Elasticsearch

Encrypting Communications in an Elasticsearch Docker Container

Enabling Cipher Suites for Stronger Encryption

Separating node-to-node and client traffic

Configuring an Active Directory realm

Configuring a file realm

Configuring an LDAP realm

Configuring a native realm

Configuring a PKI realm

Configuring a SAML realm

Configuring a Kerberos realm

FIPS 140-2

Security settings

Auditing Settings

Configuring X-Pack Java Clients

X-Pack Settings in Elasticsearch

You configure settings for X-Pack features in the elasticsearch.yml, kibana.yml, and logstash.yml configuration files.

X-Pack Feature	Elasticsearch Settings	Kibana Settings	Logstash Settings
APM UI	No	Yes	No
Development Tools	No	Yes	No
Graph	No	Yes	No
Machine learning	Yes	Yes	No
Management	No	No	Yes
Monitoring	Yes	Yes	Yes
Reporting	No	Yes	No
Security	Yes	Yes	No
.Auditing	Yes	No	No
Watcher	Yes	No	No

There are also X-Pack license settings in the elasticsearch.yml file.

License Settings

Machine Learning Settings

Watcher Settings

SQL Access Settings

Bootstrap Checks for X-Pack

In addition to the Elasticsearch bootstrap checks, there are checks that are specific to X-Pack features.

Encrypt sensitive data check

If you use Watcher and have chosen to encrypt sensitive data (by setting xpack.watcher.encrypt_sensitive_data to true), you must also place a key in the secure settings store.

To pass this bootstrap check, you must set the xpack.watcher.encryption_key on each node in the cluster. For more information, see Encrypting Sensitive Data in Watcher.

PKI realm check

If you use X-Pack security and a Public Key Infrastructure (PKI) realm, you must configure Transport Layer Security (TLS) on your cluster and enable client authentication on the network layers (either transport or http). For more information, see PKI User Authentication and Setting Up TLS on a Cluster.

To pass this bootstrap check, if a PKI realm is enabled, you must configure TLS and enable client authentication on at least one network communication layer.

Role mappings check

If you authenticate users with realms other than native or file realms, you must create role mappings. These role mappings define which roles are assigned to each user.

If you use files to manage the role mappings, you must configure a YAML file and copy it to each node in the cluster. By default, role mappings are stored in ES_PATH_CONF/role_mapping.yml. Alternatively, you can specify a different role mapping file for each type of realm and specify its location in the elasticsearch.yml file. For more information, see Using Role Mapping Files.

To pass this bootstrap check, the role mapping files must exist and must be valid. The Distinguished Names (DNs) that are listed in the role mappings files must also be valid.

SSL/TLS check

In 6.0 and later releases, if you have a gold, platinum, or enterprise license and X-Pack security is enabled, you must configure SSL/TLS for internode-communication.

Note
Single-node clusters that use a loopback interface do not have this requirement. For more information, see Encrypting Communications.

To pass this bootstrap check, you must set up SSL/TLS in your cluster.

Token SSL check

If you use X-Pack security and the built-in token service is enabled, you must configure your cluster to use SSL/TLS for the HTTP interface. HTTPS is required in order to use the token service.

In particular, if xpack.security.authc.token.enabled and http.enabled are set to true in the elasticsearch.yml file, you must also set xpack.security.http.ssl.enabled to true. For more information about these settings, see Security settings and HTTP.

To pass this bootstrap check, you must enable HTTPS or disable the built-in token service by using the X-Pack security settings.