Livy安装&部署

官网
http://livy.incubator.apache.org/get-started/

Download

[hadoop@hadoop001 software]$ wget http://mirrors.hust.edu.cn/apache/incubator/livy/0.5.0-incubating/livy-0.5.0-incubating-bin.zip
[hadoop@hadoop001 software]$ unzip livy-0.5.0-incubating-bin.zip
[hadoop@hadoop001 software]$ mv livy-0.5.0-incubating-bin/ ../app/
[hadoop@hadoop001 software]$ cd ../app/livy-0.5.0-incubating-bin/
[hadoop@hadoop001 livy-0.5.0-incubating-bin]$ cd conf/
[hadoop@hadoop001 conf]$ cp livy-env.sh.template livy-env.sh
[hadoop@hadoop001 conf]$ vi livy-env.sh
JAVA_HOME=/opt/app/jdk1.8.0_45
HADOOP_CONF_DIR=/opt/app/hadoop-2.6.0-cdh5.7.0/conf
SPARK_HOME=/opt/app/spark-2.2.0-bin-2.6.0-cdh5.7.0

修改日志，使其信息能打印在控制台上

[hadoop@hadoop001 conf]$ cp log4j.properties.template log4j.properties

启动Livy

[hadoop@hadoop001 livy-0.5.0-incubating-bin]$ ./bin/livy-server

报错：

Exception in thread "main" java.io.IOException: Cannot write log directory /opt/app/livy-0.5.0-incubating-bin/logs
		        at org.eclipse.jetty.util.RolloverFileOutputStream.setFile(RolloverFileOutputStream.java:219)
		        at org.eclipse.jetty.util.RolloverFileOutputStream.<init>(RolloverFileOutputStream.java:166)
		        at org.eclipse.jetty.server.NCSARequestLog.doStart(NCSARequestLog.java:228)
		        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
		        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
		        at org.eclipse.jetty.server.handler.RequestLogHandler.doStart(RequestLogHandler.java:140)
		        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
		        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
		        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
		        at org.eclipse.jetty.server.Server.start(Server.java:387)
		        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
		        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
		        at org.eclipse.jetty.server.Server.doStart(Server.java:354)
		        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
		        at org.apache.livy.server.WebServer.start(WebServer.scala:92)
		        at org.apache.livy.server.LivyServer.start(LivyServer.scala:259)
		        at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
		        at org.apache.livy.server.LivyServer.main(LivyServer.scala)

解决办法：
权限问题，需要手动创建logs目录

[hadoop@hadoop001 livy-0.5.0-incubating-bin]$ mkdir logs

启动成功：
web ui： http://192.168.26.131:8998/ui

配置文件

• livy.conf：配置了一些server的信息
• spark-blacklist.conf
  会列出来一些spark配置中的一些东西，这些东西用户是不允许被修改掉的
  给用户的一些东西，有些是不能改的，比如：内存大小的设置、executor的设置
  这些给用户改，是不放心的；因此有些东西必然是不能够暴露的
• log4j.properties：日志信息

做如下修改：

[hadoop@hadoop001 conf]$ cp livy.conf.template livy.conf
[hadoop@hadoop001 conf]$ vi livy.conf
livy.server.host = 0.0.0.0
livy.server.port = 8998
livy.spark.master = local[2]