Metron学习10_Deploying a fresh Metron cluster via Ambari Management pack

Short Description:

This article describes the steps to deploy a fresh HCS/Metron cluster using Ambari Server installed with management packs for Metron.

Article

Note:

• This article is an extension of the instructions at this link.
• This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).
• Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.
• The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.

Prerequisites

For Generating RPMs and Management Pack

• Local system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.
• Docker service has been installed and is running.
• A cloned copy of the Metron git repo.

For Metron Cluster deployment

• 12 VMs running CentOS 7.x.
• JDK > 1.8.x installed on all nodes.
• Ambari Server > 2.4.x required

Step 1 - Build Metron RPMs and Management Pack

In this step, you will create the RPMs and Ambari management pack tarball on your local system. The RPMs and Ambari mpack need to be SCP'ed into the cluster hosts

a) Build Mpack

On your local system where you have cloned the Git repo, run the following command:

  
  
➜  incubator-metron git:(master) ✗ pwd
/home/metron-test/incubator-metron
➜  incubator-metron git:(master) ✗ mvn clean install

Once the above command has run successfully, the metron management pack will be generated in the path at:

incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz

b) Build RPMs

Run the following commands to generate the RPMs:

Note: Docker service should be running in order for the below command to work properly.

  
  
cd incubator-metron/metron-deployment/packaging/docker/rpm-docker
➜  rpm-docker git:(master) mvn clean install

The above command will build the RPMs and create them under incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch

c) Copy RPMs and Mpack to cluster nodes

Use either SCP or your favorite file transfer application to copy the mpack and RPMs over the to the cluster

• Copy the metron_mpack-1.0.0.0-SNAPSHOT.tar.gz file to the node which is going to run Ambari Server (For E.g. node #1)
• On the remaining 11 nodes (E.g. node 2 through node 11), create a directory call /localrepo and scp all of the following metron rpms to the /localrepo folder

  
  
metron-common-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-data-management-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-elasticsearch-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-enrichment-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-indexing-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-parsers-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-pcap-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm
metron-solr-<METRONVERSION>-<TIMESTAMP>-BETA.noarch.rpm

Step 2 - Install Ambari Server and Metron management pack

SSH to the node where Ambari server needs to be installed (node #1) and follow these steps.

Ensure CentOS repo for Ambari server is updated:

  
  
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.1.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

Install Ambari server, install mpack and start service

  
  
yum install ambari-server -y
ambari-server setup -s
ambari-server install-mpack --mpack=/path/to/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server start

Step 3 - Cluster Installation

Point your browser to http://<node#1>:8080 to start the cluster installation wizard. Follow the screenshots below to complete the Metron installation.

Start the wizard by choosing the "Create a Cluster" option and specify a name for the cluster.

Select HDP 2.5 stack for Metron 0.3.x from the version selection page.

Specify the list of hosts in your cluster along with the connection information (e.g. private key)

In the "Choose Services" page, select the following services -- which are required as a minimum for a working Metron deployment. You may select other additional services as per your need.

• HDFS
• YARN + MapReduce2
• HBase
• Zookeeper
• Storm
• Flume
• Kafka
• Elasticsearch
• Kibana
• Metron

Here are sample screenshots of the selection

In the "Assign Masters" page, ensure the following criteria is met. Observe that the Ambari wizard will display warning / error popups when the below criteria is not met.

a) Ensure that the Kibana Server, Metron Enrichment, MySQL Server, Metron Indexing and Metron Parsers components are all assigned to the _same node_. It is important to note that these _should not_ be on the same node as Ambari server. All of these, for e.g. could reside on the node #12

b) It is preferred to run the Elasticsearch Master on the same node as Metron components (node #12, in this example).

c) Add up to 4 Kafka Broker components. Ensure that one of the Kafka Broker components is installed on the Metron node #12.

d) [Optional] You may retain the Zookeeper server on only one host and remove the rest

The other components may be left at their defaults

Note: For some of the components, there is a warning message to remind the user for client selection. Choose "Continue Anyway" if the Validation Issues warning dialog pops up.

Here is a sample screenshot for reference:

In the "Assign Slaves and Clients" window, you need to ensure that the Metron node (#12 in this example) is selected for the following:

• DataNode
• NodeManager
• RegionServer
• Supervisor
• Flume
• ElasticsearchDataNode
• Client

Optionally, you can choose to install "Client" on all the nodes.

In the "Customize Services" window, refer to the screenshots below for filling in the respective tabs

1. Change NameNode heap size from the default 1024 MB to about 4096 MB

2. Under Elasticsearch -> Advanced elastic-site -> zen_discovery_ping_unicast_hosts, specify the location where Elasticsearch master is installed (E.g. node #12 in this case.

3. Under Kibana -> Advanced kibana-env, specify the kibana_es_url to the Elasticsearch master node URL with port as 9200.

4. Under Metron, change the parameters in the respective tabs below.

Tab #1 - Default Settings for Metron services.

Tab #2 - Repository settings - Remote vs. Local repository

If you choose to install using a Local repository, ensure that you have copied the Metron RPMs into the /localrepo folder in the Metron node.

If you choose to install using a remote repository, specify the URL where the repo file is available.

Hit the "Next" button and then hit the "Deploy" button to proceed with Metron deployment. This will start the cluster deployment and in time, all the services should be up and running.

原文来自：https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html