Spark,Livy,ES,Griffin单机版安装

最新推荐文章于 2024-06-08 09:31:18 发布

谷应泰

最新推荐文章于 2024-06-08 09:31:18 发布

阅读量516

点赞数 1

分类专栏：大数据文章标签： spark elasticsearch big data Griffin

本文链接：https://blog.csdn.net/m0_37135879/article/details/122780208

版权

大数据专栏收录该内容

24 篇文章 3 订阅

订阅专栏

本文详细介绍了如何安装和配置大数据质量监测平台Griffin，包括Spark、Livy、Elasticsearch的安装，以及Griffin的环境变量设置、配置文件修改、服务启动等步骤。在安装过程中，涉及到Hadoop、Hive、YARN的配置，并通过编译解决Griffin的依赖问题，最后成功启动服务并展示了Griffin的Web界面。

摘要由CSDN通过智能技术生成

一.Spark安装

有二进制包,不用编译

tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz

设置环境变量

export SPARK_HOME=/opt/lagou/servers/spark-2.2.1
export PATH=$PATH:$SPARK_HOME/bin

修改配置文件
/opt/lagou/servers/spark-2.2.1/conf/spark-defaults.conf
增加如下配置

spark.master                 yarn
spark.eventLog.enabled       true
spark.eventLog.dir     hdfs://Linux121:9000/spark/logs
spark.serializer       org.apache.spark.serializer.KryoSerializer
spark.yarn.jars        hdfs://Linux121:9000/spark/spark_2.2.1_jars/*

/opt/lagou/servers/spark-2.2.1/conf/spark-env.sh

export JAVA_HOME=/opt/lagou/servers/jdk1.8.0_301
export HADOOP_HOME=/opt/lagou/servers/hadoop-2.9.2
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

拷贝mysql驱动

ln -s /opt/lagou/servers/hive-2.3.7/lib/mysql-connector-java-5.1.46.jar /opt/lagou/servers/spark-2.2.1/jars/mysql-connector-java-5.1.46.jar

在hdfs创建配置的路径并且将spark jar上传

[root@Linux122 guchenfei]# hdfs dfs -mkdir -p /spark/logs
[root@Linux122 guchenfei]# hdfs dfs -mkdir -p /spark/spark_2.2.1_jars/
[root@Linux122 jars]# hdfs dfs -put /opt/lagou/servers/spark-2.2.1/jars/*.jar /spark_2.2.1_jars/

修改yarn配置,由于虚拟机资源不足
/opt/lagou/servers/hadoop-2.9.2/etc/hadoop/yarn-site.xml
将禁止检查真实内存情况

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>

重启yarn服务
测试能否进入spark-shell

[root@Linux122 conf]# spark-shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/lagou/servers/spark-2.2.1/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/lagou/servers/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/lagou/servers/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/02/03 00:02:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/02/03 00:03:02 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Spark context Web UI available at http://172.16.131.130:4040
Spark context available as 'sc' (master = yarn, app id = application_1643816826080_0001).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_301)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

测试计算

scala> val lines = sc.textFile("/wcinput/tmp.txt")
lines: org.apache.spark.rdd.RDD[String] = /wcinput/tmp.txt MapPartitionsRDD[5] at textFile at <console>:24
scala> lines.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect()
res4: Array[(String, Int)] = Array((JAVA,5), (Linux,4), (hadoop,7))

二.Livy安装

1.解压安装包设置环境变量

[root@Linux122 servers]# unzip livy-0.5.0-incubating-bin.zip
配置环境变量

export LIVY_HOME=/opt/lagou/servers/livy-0.5.0
export PATH=$PATH:$LIVY_HOME/bin

2.修改配置文件
/opt/lagou/servers/livy-0.5.0/conf/livy.conf

livy.server.host=127.0.0.1
livy.spark.master=yarn
livy.spark.deployMode=cluster
livy.repl.enable-hive-context=true

/opt/lagou/servers/livy-0.5.0/conf/livy-env.sh

export SPARK_HOME=/opt/lagou/servers/spark-2.2.1
export HADOOP_HOME=/opt/lagou/servers/hadoop-2.9.2/
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

创建日志文件

[root@Linux122 livy-0.5.0]# mkdir logs

启动服务

[root@Linux122 livy-0.5.0]# nohup bin/livy-server &

三.ES安装

1.解压

[root@Linux122 servers]# tar -zxvf elasticsearch-5.6.0.tar.gz

2.修改文件用户权限

ES不能用root用户启动,创建用户和用户组

[root@Linux122 servers]# groupadd elasticsearch
[root@Linux122 servers]# useradd elasticsearch -g elasticsearch
[root@Linux122 servers]# chown -R elasticsearch:elasticsearch elasticsearch-5.6.0/(-R递归去执行)
[root@Linux122 servers]# ll
drwxr-xr-x.  7 elasticsearch elasticsearch  131 Sep  7  2017 elasticsearch-5.6.0

[root@Linux122 elasticsearch-5.6.0]# ll
total 224
drwxr-xr-x.  2 elasticsearch elasticsearch   4096 Feb  3 13:52 bin
drwxr-xr-x.  2 elasticsearch elasticsearch     75 Sep  7  2017 config
drwxr-xr-x.  2 elasticsearch elasticsearch   4096 Sep  7  2017 lib
-rw-r--r--.  1 elasticsearch elasticsearch  11358 Sep  7  2017 LICENSE.txt
drwxr-xr-x. 13 elasticsearch elasticsearch    236 Sep  7  2017 modules
-rw-r--r--.  1 elasticsearch elasticsearch 194187 Sep  7  2017 NOTICE.txt
drwxr-xr-x.  2 elasticsearch elasticsearch      6 Sep  7  2017 plugins
-rw-r--r--.  1 elasticsearch elasticsearch   9549 Sep  7  2017 README.textile
[root@Linux122 elasticsearch-5.6.0]#

3.修改Linux内核参数

[root@Linux122 guchenfei]# vim /etc/security/limits.conf 
elasticsearch hard nofile 1000000
elasticsearch soft nofile 1000000
* soft nproc 4096
* hard nproc 4096

[root@Linux122 guchenfei]# vim /etc/sysctl.conf 
vm.max_map_count=262144

执行如下命令修改才能生效
[root@Linux122 guchenfei]# sysctl -p
vm.max_map_count = 262144

4.修改ES配置文件

[root@Linux122 guchenfei]# vim /opt/lagou/servers/elasticsearch-5.6.0/config/elasticsearch.yml 
network.host: 0.0.0.0

修改JVM内存分配(根据自己资源情况设置)
/opt/lagou/servers/elasticsearch-5.6.0/config/jvm.options
-Xms1g
-Xmx1g

5.启动ES服务

[root@Linux122 config]# su elasticsearch
[elasticsearch@Linux122 elasticsearch-5.6.0]$ ls
bin  config  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.textile
[elasticsearch@Linux122 elasticsearch-5.6.0]$ bin/elasticsearch -d(-d表示后台启动)

web浏览
http://linux122:9200
在这里插入图片描述

6.在ES里创建griffin索引

curl -XPUT http://Linux122:9200/griffin -d '
{
    "aliases": {},
    "mappings": {
        "accuracy": {
            "properties": {
                "name": {
                    "fields": {
                        "keyword": {
                            "ignore_above": 256,
                            "type": "keyword"
                        }
                    },
                    "type": "text"
                },
                "tmst": {
                    "type": "date"
                }
            }
        }
    },
    "settings": {
        "index": {
            "number_of_replicas": "2",
            "number_of_shards": "5"
        }
    }
}
'

响应

{"acknowledged":true,"shards_acknowledged":true,"index":"griffin"}

四.Griffin编译安装

1.解压Griffin

[root@Linux122 servers]# unzip griffin-griffin-0.5.0.zip

2.mysql创建数据库

mysql> create database quartz;

3.初始化数据库

/opt/lagou/servers/griffin-0.5.0/service/src/main/resources/Init_quartz_mysql_innodb.sql

在sql脚本加use quartz;
执行sql脚本

[root@Linux123 ~]# mysql -uroot -p123456 < Init_quartz_mysql_innodb.sql

mysql> show tables;
+--------------------------+
| Tables_in_quartz         |
+--------------------------+
| QRTZ_BLOB_TRIGGERS       |
| QRTZ_CALENDARS           |
| QRTZ_CRON_TRIGGERS       |
| QRTZ_FIRED_TRIGGERS      |
| QRTZ_JOB_DETAILS         |
| QRTZ_LOCKS               |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE     |
| QRTZ_SIMPLE_TRIGGERS     |
| QRTZ_SIMPROP_TRIGGERS    |
| QRTZ_TRIGGERS            |
+--------------------------+
11 rows in set (0.00 sec)

4.hadoop和hive配置

[root@Linux122 resources]# hdfs dfs -mkdir -p /spark/spark_conf
/opt/lagou/servers/hive-2.3.7/conf/hive-site.xml(那台机器装将对应的文件放入hdfs,这个文件有客户端和服务端之分)
[root@Linux122 conf]# hdfs dfs -put hive-site.xml /spark/spark_conf

5.确保环境变量配置

export JAVA_HOME=/opt/lagou/servers/jdk1.8.0_301
export SPARK_HOME=/opt/lagou/servers/spark-2.2.1
export LIVY_HOME=/opt/lagou/servers/livy-0.5.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

6.Griffin编译

(1).修改配置文件去掉这块的注释

/opt/lagou/servers/griffin-0.5.0/service/pom.xml

113         <dependency>
114             <groupId>mysql</groupId>
115             <artifactId>mysql-connector-java</artifactId>
116             <version>${mysql.java.version}</version>
117         </dependency>

(2)./opt/lagou/servers/griffin-0.5.0/service/src/main/resources/application.properties

[root@Linux122 resources]# cat application.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#
server.port=9876

spring.application.name=griffin_service
spring.datasource.url=jdbc:mysql://Linux123:3306/quartz?autoReconnect=true&useSSL=false
spring.datasource.username=root
spring.datasource.password=123456
spring.jpa.generate-ddl=true
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.show-sql=true
# Hive metastore
hive.metastore.uris=thrift://Linux123:9083
hive.metastore.dbname=hivemetadata
hive.hmshandler.retry.attempts=15
hive.hmshandler.retry.interval=2000ms
# Hive cache time
cache.evict.hive.fixedRate.in.milliseconds=900000
# Kafka schema registry
kafka.schema.registry.url=http://localhost:8081
# Update job instance state at regular intervals
jobInstance.fixedDelay.in.milliseconds=60000
# Expired time of job instance which is 7 days that is 604800000 milliseconds.Time unit only supports milliseconds
jobInstance.expired.milliseconds=604800000
# schedule predicate job every 5 minutes and repeat 12 times at most
#interval time unit s:second m:minute h:hour d:day,only support these four units
predicate.job.interval=5m
predicate.job.repeat.count=12
# external properties directory location
external.config.location=
# external BATCH or STREAMING env
external.env.location=
# login strategy ("default" or "ldap")
login.strategy=default
# ldap
ldap.url=ldap://hostname:port
ldap.email=@example.com
ldap.searchBase=DC=org,DC=example
ldap.searchPattern=(sAMAccountName={0})
# hdfs default name
fs.defaultFS=
# elasticsearch
elasticsearch.host=Linux122
elasticsearch.port=9200
elasticsearch.scheme=http
# elasticsearch.user = user
# elasticsearch.password = password
# livy
livy.uri=http://localhost:8998/batches
livy.need.queue=false
livy.task.max.concurrent.count=20
livy.task.submit.interval.second=3
livy.task.appId.retry.count=3
# yarn url
yarn.uri=http://Linux123:8088
# griffin event listener
internal.event.listeners=GriffinJobEventHook
[root@Linux122 resources]#

(3)./opt/lagou/servers/griffin-0.5.0/service/src/main/resources/quartz.properties

修改如下

 26 org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate

(4)./opt/lagou/servers/griffin-0.5.0/service/src/main/resources/sparkProperties.json

修改hdfs上边的配置路径

“spark.yarn.dist.files”: “hdfs:///spark/spark_conf/hive-site.xml”

(5)./opt/lagou/servers/griffin-0.5.0/service/src/main/resources/env/env_batch.json

"api": "http://Linux122:9200/griffin/accuracy",

创建配置里边路径

[root@Linux122 env]# hdfs dfs -mkdir -p /griffin/persist

(6).编译

/opt/lagou/servers/griffin-0.5.0
[root@Linux122 griffin-0.5.0]# mvn -Dmaven.test.skip=true clean install

错误

[ERROR]                                                                                          
[ERROR] ERROR in /opt/lagou/servers/griffin-0.5.0/ui/angular/node_modules/@types/jquery/JQuery.d.ts (1784,30): Cannot find name 'SVGElementTagNameMap'.
[ERROR] ERROR in /opt/lagou/servers/griffin-0.5.0/ui/angular/node_modules/@types/jquery/JQuery.d.ts (1784,73): Cannot find name 'SVGElementTagNameMap'.
[ERROR] ERROR in /opt/lagou/servers/griffin-0.5.0/ui/angular/node_modules/@types/jquery/JQuery.d.ts (8706,29): Cannot find name 'SVGElementTagNameMap'.
[ERROR] ERROR in /opt/lagou/servers/griffin-0.5.0/ui/angular/node_modules/@types/jquery/JQuery.d.ts (8706,84): Cannot find name 'SVGElementTagNameMap'.
[ERROR] 
[ERROR] npm ERR! Linux 3.10.0-1160.el7.x86_64
[ERROR] npm ERR! argv "/opt/lagou/servers/griffin-0.5.0/ui/.tmp/node/node" "/opt/lagou/servers/griffin-0.5.0/ui/.tmp/node/node_modules/npm/bin/npm-cli.js" "run" "build"
[ERROR] npm ERR! node v6.11.3
[ERROR] npm ERR! npm  v3.10.10
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! griffin@0.0.0 build: `ng build`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR! 
[ERROR] npm ERR! Failed at the griffin@0.0.0 build script 'ng build'.
[ERROR] npm ERR! Make sure you have the latest version of node.js and npm installed.
[ERROR] npm ERR! If you do, this is most likely a problem with the griffin package,
[ERROR] npm ERR! not with npm itself.
[ERROR] npm ERR! Tell the author that this fails on your system:
[ERROR] npm ERR!     ng build
[ERROR] npm ERR! You can get information on how to open an issue for this project with:
[ERROR] npm ERR!     npm bugs griffin
[ERROR] npm ERR! Or if that isn't available, you can get their info via:
[ERROR] npm ERR!     npm owner ls griffin
[ERROR] npm ERR! There is likely additional logging output above.
[ERROR] 
[ERROR] npm ERR! Please include the following file with any support request:
[ERROR] npm ERR!     /opt/lagou/servers/griffin-0.5.0/ui/angular/npm-debug.log
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Griffin 0.5.0 0.5.0:
[INFO] 
[INFO] Apache Griffin 0.5.0 ............................... SUCCESS [  9.269 s]
[INFO] Apache Griffin :: UI :: Default UI ................. FAILURE [01:03 min]
[INFO] Apache Griffin :: Web Service ...................... SKIPPED
[INFO] Apache Griffin :: Measures ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:13 min
[INFO] Finished at: 2022-02-03T22:16:37+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.6:npm (npm build) on project ui: Failed to run task: 'npm run build' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :ui
[root@Linux122 griffin-0.5.0]# mvn -Dmaven.test.skip=true clean install
[INFO] Scanning for projects...
[WARNING] The project org.apache.griffin:griffin:pom:0.5.0 uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Griffin 0.5.0                                               [pom]
[INFO] Apache Griffin :: UI :: Default UI                                 [pom]
[INFO] Apache Griffin :: Web Service                                      [jar]
[INFO] Apache Griffin :: Measures                                         [jar]
[INFO] 
[INFO] ---------------------< org.apache.griffin:griffin >---------------------
[INFO] Building Apache Griffin 0.5.0 0.5.0                                [1/4]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ griffin ---
[INFO] Deleting /opt/lagou/servers/griffin-0.5.0/target
[INFO] 
[INFO] --- mav

/opt/lagou/servers/griffin-0.5.0/ui/angular/node_modules/@types/jquery/JQuery.d.ts
根据错误信息删除对应行
继续编译(我编译用的是国外的源所以避免了一些错误)

[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ measure ---
[INFO] Installing /opt/lagou/servers/griffin-0.5.0/measure/target/measure-0.5.0.jar to /root/.m2/repository/org/apache/griffin/measure/0.5.0/measure-0.5.0.jar
[INFO] Installing /opt/lagou/servers/griffin-0.5.0/measure/pom.xml to /root/.m2/repository/org/apache/griffin/measure/0.5.0/measure-0.5.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Griffin 0.5.0 0.5.0:
[INFO] 
[INFO] Apache Griffin 0.5.0 ............................... SUCCESS [  5.451 s]
[INFO] Apache Griffin :: UI :: Default UI ................. SUCCESS [01:37 min]
[INFO] Apache Griffin :: Web Service ...................... SUCCESS [21:30 min]
[INFO] Apache Griffin :: Measures ......................... SUCCESS [09:11 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  32:25 min
[INFO] Finished at: 2022-02-03T22:51:17+08:00
[INFO] ------------------------------------------------------------------------

(7)jar拷贝

/opt/lagou/servers/griffin-0.5.0/service/target
[root@Linux122 target]# cp service-0.5.0.jar /opt/lagou/servers/griffin-0.5.0/

/opt/lagou/servers/griffin-0.5.0/measure/target
[root@Linux122 target]# cp measure-0.5.0.jar /opt/lagou/servers/griffin-0.5.0/griffin-measure.jar

上传hdfs
[root@Linux122 griffin-0.5.0]# hdfs dfs -mkdir /griffin
[root@Linux122 griffin-0.5.0]# hdfs dfs -put griffin-measure.jar /griffin/

(8)启动服务

[root@Linux122 griffin-0.5.0]# nohup java -jar service-0.5.0.jar > service.out 2>&1 &

http://Linux122:9876
[root@Linux122 griffin-0.5.0]# tail -f service.out