备注:
Metron安装方式的一种,基于Ambari集群安装;
The following provides instructions on how to install Metron on an existing Ambari-managed HDP 2.4 cluster. These are not recommendations for production deployment but rather for dev/testing environments.
Prerequisites
For the purposes of this discussion, there will be 7 types of server roles:
- Hadoop Master
- Hadoop Slave
- Ambari Server
- Metron Installer
- Metron Search
- Metron Probe
- Metron Web
Specifications for Existing Hadoop Cluster
The following are the specifications for the Hadoop related nodes:
- All cluster nodes must be running CentOS 6.x.
- The cluster must be running HDP 2.4 managed by Ambari 2.2.x.
-
The cluster must have a minimum of 2 Hadoop Master nodes and 4 Hadoop slave nodes.
- The minimum spec for each of the Hadoop Slave and Master nodes should be (for Dev/Test/POC envs):
- 8-16 GB RAM
- 4 cores
- 100 GB of storage
- The following services must be installed across the Hadoop Master and Slave nodes:
-
Each of the following components must be installed on at least 4 Hadoop Slave Nodes:
- No other workloads should be running on the cluster. In other words, this cluster should be dedicated for Metron jobs.
Specifications for Metron Nodes
- At least three nodes must be dedicated for Metron-specific components.
- You should have root access on all Metron nodes.
-
All Metron nodes (Search, Probes, and Web) must have the following:
- Centos 6.7
- Oracle JDK 1.8
- The minimum spec for each of the Metron nodes should be (for Dev/Test/POC envs):
- 8-16 GB RAM
- 4 cores
- 100 GB of storage
-
The Metron Installer node must have Docker installed. (Note: If you are using Centos, 7.x is preferred for the Metron Installer node.)
-
The Metron Installer node requires SSH access to all Hadoop Cluster and Metron nodes. Passwordless is preferred.
Sample Deployment Architecture
Based on the above specs, here is a sample deployment architecture.
Install Instructions
Step 1: Setup
For the following instructions, set the following environmental variables. Plug in the necessary values for your deployment.
export METRON_HOME=/root/metron/incubator-metron-Metron_0.2.0BETA_rc2
export METRON_0_2_RELEASE_BINARIES=https://dist.apache.org/repos/dist/dev/incubator/metron/0.2.0BETA-RC2-incubating/apache-metron-0.2.0BETA-RC2-incubating.tar.gz
export INSTALL_CONF_DIRECTORY=my_install
Step 2: Install or Validate Existing Hadoop Cluster
- Install a new HDP 2.4 using the following instructions or ensure you have an existing HDP cluster that follows the specs outlined in the pre-requisites section.
Step 3: Download and Build Metron
- Log into the Metron Installer node.
- Verify Docker is installed and running.
- Ensure that this node has ssh access to each of the Metron and Hadoop/HDP nodes.
- Download the Metron Binaries from $METRON_0_2_RELEASE_BINARIES
- Untar it: tar -zxvf apache-metron-0.2.0BETA-RC2-incubating.tar.gz
- cd $METRON_HOME
- Run Docker installer image:
docker run -it -v $METRON_HOME:/root/incubator-metron dlyle65535/metron-ansible:2.0.0.2 bash - Build Metron
cd /root/incubator-metron/
mvn clean package -DskipTests
Step 4: Configure Metron for Install
- cd /root/incubator-metron/metron-deployment/inventory
- create a directory called $INSTALL_CONF_DIRECTORY that will hold all of your metron deployment configuration
- Copy the sample configs into the new directory:
- cp - R /root/incubator-metron/metron-deployment/inventory/metron_example/* /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY
- Setup your inventory hosts by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/hosts
-
Configure with your cluster details by defining which hosts the Metron services will be installed on by updating these groups:
[ambari_master] - host running Ambari
[ambari_slaves] - all Ambari-managed hosts
[metron_hbase_tables] - host used to create the HBase tables required by Metron. Requires a HBase client.
[metron_kafka_topics] - host used to create the Kafka topics required by Metron. Requires a Kafka broker.
[enrichment] - submits the topology code to Storm and requires a Storm client
[search] - host(s) where Elasticsearch will be installed
[web] - host where the Metron UI and underlying services will be installed
[sensors] - host where network data will be collected and published to Kafka - Setup your environment configuration by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/group_vars/all
-
Commonly Used Settings:
Ansible
-
-
-
- ansible_ssh_private_key_file: /Path/to/private/key/file Point to the private key file for ssh user on the target hosts
- ansible_ssh_user: root The name of the ssh user on the target hosts (requires sudo)
-
-
Ambari
-
-
-
- ambari_port: 8080 Change if your Ambari instance uses a non-default port
- ambari_user: admin Change to user on your Ambari instance
- ambari_password: admin Change to password for your Ambari user above
-
-
Kafka
-
-
-
- num_partitions: 3 Change to your desired number of partitions
- retention_in_gb: 25 Change to your desired retention size
-
-
Metron
-
-
-
- java_home: /usr/jdk64/jdk1.8.0_40 Location of Java on all hosts
-
-
Sensors
-
-
-
- sensor_test_mode: True Change to false if not running traffic replay
- sniff_interface: eth0 Interface that the Metron sensors will sniff on the [sensors] host
-
-
Search
-
-
-
- elasticsearch_network_interface: eth0 Bind interface for the Elasticsearch host(s)
-
-
-
-
- Ensure that you do't have duplicate entries of sniff_interface
-
实际操作时的位置:
Step 5: Install/Deploy Metron to the Cluster
Note: Adding -u user -k to the below command lines allows deployment in environments where passwordless SSH has not been enabled.
- cd /root/incubator-metron/metron-deployment/playbooks
- Run the following ansible command to deploy Metron with the standard services (Metron Topologies, WebUI, and Monit)
ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr" - Run the following Ansible command to deploy Metron without installing Sensors
ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr,sensors"
实际操作时的命令:
1. cd /home/szx/incubator-metron-Metron_0.2.0BETA_rc2/metron-deployment/playbooks
2. ansible-playbook -i /home/szx/incubator-metron-Metron_0.2.0BETA_rc2/metron-deployment/inventory/installConf metron_install.yml --skip-tags="solr"
3. ansible-playbook -i /home/szx/incubator-metron-Metron_0.2.0BETA_rc2/metron-deployment/inventory/installConf metron_install.yml --skip-tags="solr,sensors"