#Livy配置Kerberos,#调用Hadoop组件,#Java 实现Livy大数据调用,#java拉取hive数据同步到本地

项目场景：

78做java后端的, 让联调大数据, 78做java的都没看过大数据让去做, 又不是没做大数据的人了,而且 nm让做不懂得涨点工资, 发个福利吗?

自己写一个Java程序, 去调用别人Livy, Livy调用Spark, Spark执行作业, 回写到自己的库里!

不要磨磨蹭蹭说原理讲道理, show code, 愿意多bb, 就teach code,
问原理的都是sb, 问感触,问经验的才是正常人!zzJava面试官!

原因分析：

78一个问题分析一下, 我去手写个jdk给你看好不好? ：

老话都说了, 黑猫白猫78猫, 解决问题就是好猫, 能78记住下次不犯就很nb, 还分析, 分析个der

---

记录：

1, 搭建livy, livy单机的搭建和使用很简单, 下载livy的安装包吗, tar -xvf 解压, 完事
麻烦的是搞livy的配置, 路径为: livy安装路径/conf
还有就是kerberos的认证, 我这边麻烦的不是配置kerberos, 而是dm账号密码密钥路径, 说不明白, s*

…
livy.spark.master = yarn
livy.spark.deployMode = cluster
#这两个参数, 得78注意下, 看看你要调用的spark是不是集群模式, 好像默认是local, 本地模式, 单机的

livy.server.launch.kerberos.principal=GULIMALL.COM
livy.server.launch.kerberos.keytab=/home/root/gulimall.keytab
#这个是因为对方开启了kerberos的认证, 其实就是一个账号密码, 类似于ssh登录的过程一样, 对方提供给你一个账号和密钥 这里指定他的账号和路径

#livy.server.auth.type=kerberos
#livy.server.auth.kerberos.principal=HTTP/jt_bsdm@BIGDATA.TAIKANG.COM
#livy.server.auth.kerberos.keytab=/home/jt_bsdm/mygit/keytab/jt_bsdm.keytab
#livy.impersonation.enabled=true
#livy.server.auth.kerberos.name-rules = DEFAULT
#这些配置可以不开, 如果开了的话, 你访问livy 8089的时候,他会让你输入账号密码

注:
你可能也需要看下环境变量, 给你提供我当时用的几个例子:
而且我这边没有root, sudo权限, 所以只能每次执行前自己手动环境变量

…
export JAVA_HOME=/home/root/jdk8
export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin

…
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/data/spark

2, 至此, 你可以去开始测试下livy调用spark了, 我这边也是找了一个命令行提价spark的例子,
正常的话, 你应该能从hadoop的管理页面看到你提交的任务了, 而且是success的, 那么恭喜你, livy到spark初步调试通过!

…
1,
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --executor-memory 1G --total-executor-cores 2 ./spark-examples_2.11-2.4.0-cdh6.3.4.jar 100
2,
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --executor-memory 1G --total-executor-cores 2 ./spark-examples_2.11-2.4.0-cdh6.3.4.jar 100

注:
可以看到我这边也是有两个例子, 区别在于, --deploy-mode cluster , 你可以自行测试

3, 写个代码测试一下, 代码分两块, 一块是代码调用spark, 第二快是spark执行你需要运行的jar

• java调用livy, 执行spark调度
  先看看pom

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>sparktets</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>2.4.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.12</artifactId>
            <version>2.4.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.19</version>
        </dependency>
        <dependency>
            <groupId>org.apache.livy</groupId>
            <artifactId>livy-api</artifactId>
            <version>0.7.1-incubating</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.78</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifest>
                                    <mainClass>
                                        cn.tongdun.test.AppMain
                                    </mainClass>
                                </manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

重点:
这个打包78当时也浪费了好长时间, 当时公司大哥写了一个测试demo, 自己引入的spark-core, spark-sql, spark-hive等依赖, 我自己直接引入了spark-sql, 让他自己去找相关依赖, 然后公司大哥的一直跑不通, 我的可以了, 反正78咱也不明白,能自动引, 为啥非78自己分开导入,他78打的包100M, 我自己的30M, 还一直bibilailai打包有问题, 有问题, 是有问题, 你说说怎么解决, 我78看不出来有问题, 光78喊口号, nm, 有会的大哥, 也可以指导指导!

需要注意的点是:
1, spark-sql_2.12 这个的scope, 需要改成provider, 打包的时候排出去, 不然会报一个错, 好像是这个, java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.orc.OrcFileFormat could not be instantiated
2, 就是maven打包的时候的, builder模块得加上, 把你依赖的包都给完整打入进去, 不然有时候会报某些ClassNotFound啦, no suitable driver啥啥的

测试livy调用spark的类
这部分参考了一个大哥写的, 忘记人家连接了, 78的,

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONException;
import com.alibaba.fastjson.JSONObject;

public class TdTest {
    static String host = "http://1.1.1.1:1";  //你的livy地址

    public static int submitJob() throws JSONException {
        JSONObject data = new JSONObject();

        JSONObject conf = new JSONObject();//spark相关配置
        conf.put("spark.master","yarn");
        conf.put("spark.submit.deployMode","cluster");
        conf.put("spark.driver.extraClassPath","/user/jt_bsdm/jdbc.jar");

        data.put("conf",conf);
//        data.put("proxyUser","etluser");

        data.put("file","/data/hong1.jar");// 指定执行的spark jar (hdfs路径)
       // data.put("jars",new String[]{"/data/jdbc.jar"});//指定spark jar依赖的外部jars
        data.put("className", "TdSparkTest");
        data.put("name","Spakr本次作业的名字");
        data.put("executorCores",2);
        data.put("executorMemory","1g");
        data.put("driverCores",2);
        data.put("driverMemory","1g");
        data.put("numExecutors",3);
        data.put("queue","default");
//        data.put("args",new String[]{"杭州","yj_hangzhou","2019041719"});//传递参数

        System.out.println("参数构建完成， 发送请求给livy,{}"+(host + "/batches"));
        String res = HttpUtils.postAccess(host + "/batches", data);

        System.out.println("livy调用成功， 结果是：{}"+res);
        JSONObject resjson = JSON.parseObject(res);
        System.out.println("spark作业id：{}"+resjson);
        return resjson.getIntValue("id");
    }

    public static String getJobInfo(int id){
        JSONObject state = HttpUtils.getAccess(host + "/batches/"+id+"/state");
        String result = state.getString("state");
        System.out.println("livy状态：,{}"+result);
        return result;
    }

    public static void killJob(int id){
        // 可以直接kill掉spark任务
        if(HttpUtils.deleteAccess(host+"/batches/"+id)) {
            System.out.println("kill spark job success");
        }

    }

    public static void main(String[] args) throws  Exception {
        int id  = submitJob();

        int time = 1;
        while(true) {
            String result = getJobInfo(id);
//            if(!StringUtils.isEmpty(result)){
//                if("success".equals(result)|| "FAILED".equals(result)){
//                    break;
//                }
//            }
            time++;
            Thread.sleep(10000);
        }
//        if(time > 3){
//            killJob(9);
//        }
    }


}

还有一个工具类

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class HttpUtils {

    public static HttpURLConnection init(HttpURLConnection conn){
        conn.setDoInput(true);
        conn.setDoOutput(true);
        conn.setRequestProperty("charset","utf-8");
        conn.setRequestProperty("Content-Type","application/json");
        return conn;
    }

    /**
     * HttpGET请求
     */
    public static JSONObject getAccess(String urlStr) {

        HttpURLConnection conn = null;
        BufferedReader in = null;
        StringBuilder builder = null;
        JSONObject response = null;
        try {
            URL url = new URL(urlStr);
            conn = init((HttpURLConnection) url.openConnection());
            conn.setRequestMethod("GET");
            conn.connect();

            in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "utf-8"));
            String line = "";
            builder = new StringBuilder();
            while((line = in.readLine()) != null){
                builder.append(line);
            }

            response = JSON.parseObject(builder.toString());

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            if (conn!=null)
                conn.disconnect();
            try {
                if (in != null)
                    in.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        return response;
    }

    /**
     * HttpDelete请求
     */
    public static Boolean deleteAccess(String urlStr) {
        HttpURLConnection conn = null;
        try {
            URL url = new URL(urlStr);
            conn = init((HttpURLConnection) url.openConnection());
            conn.setRequestMethod("DELETE");
            conn.connect();

            conn.getInputStream().close();

            conn.disconnect();
        }catch (Exception e){
            e.printStackTrace();
            return false;
        }

        return true;
    }



    /**
     * HttpPost请求
     */
    public static String postAccess(String urlStr, JSONObject data)  {
        HttpURLConnection conn = null;
        BufferedReader in = null;
        StringBuilder builder = null;
        DataOutputStream out = null;
        try {
            URL url = new URL(urlStr);
            conn = init((HttpURLConnection) url.openConnection());
            conn.setRequestMethod("POST");
            conn.connect();

            out = new DataOutputStream(conn.getOutputStream());
            out.write(data.toString().getBytes("utf8"));
            out.flush();

            in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "utf-8"));
            String line = "";
            builder = new StringBuilder();
            while((line = in.readLine()) != null){
                builder.append(line);
            }

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            if (conn!= null)
                conn.disconnect();
            try {
                if (in!=null)
                    in.close();
                if (out!=null)
                    out.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (builder != null)
            return builder.toString();
        return "";
    }

}

• 上边不是需要执行一个Spark的jar嘛, 那来看看这个jar的内容

import org.apache.log4j.spi.LoggerFactory;
import org.apache.spark.sql.*;


import java.text.ParseException;
import java.util.Properties;

public class TdSparkTest {


    public static void main(String[] args) throws ParseException {
        SparkSession spark = SparkSession
                .builder()
                .appName("TdSparkTest")
                .master("yarn")
                .config("hadoop.home.dir", "/etc/hadoop/conf") //hadoop的配置文件
                .enableHiveSupport()
                //以下的这些是为了当时测试spark运行时候加载第三方jar包的, 看你需要,可以直接忽略
               // .config("jars","/data/jdbc.jar")
//                .config("driver-class-path","/user/jt_bsdm/jdbc.jar")
              //  .config("driverClassPath","/user/jt_bsdm/jdbc.jar")
               // .config("spark.driver.extraClassPath","/user/jt_bsdm/jdbc.jar")
               // .config("spark.executor.extraClassPath","/user/jt_bsdm/jdbc.jar")
                .getOrCreate();

        spark.sql("use testdb");

        Dataset<Row> sql = spark.sql("SELECT * FROM test");

        sql.show();
//        sql.write().csv("/user/jt_bsdm/td_test.csv");  //可以写到hdfs里


	//spark读取下对方hive的数据,然后写入到自己的库里, 可以是mysql, elasticsearch, hive等
        Properties properties = new Properties();
        properties.setProperty("user", "root"); // 用户名
        properties.setProperty("password", "123"); // 密码
        properties.setProperty("driver", "com.mysql.cj.jdbc.Driver");
//        properties.setProperty("numPartitions","10");

        //DataFrameWriter<Row> write = sql.write();
        sql.write().jdbc("jdbc:mysql://1.1.1.1:1/test","tdtest",properties);
        //write.option("spark.executor.extraClassPath","/user/jt_bsdm/jdbc.jar");
        write.mode(SaveMode.Overwrite).jdbc("jdbc:mysql://10.153.81.27:50670/test","table",properties);


    }

}

经验:

1, livy调用其他组件的时候, 不知道是因为配置了kerberos还是因为直接livy读取hadoop的配置文件, 竟然真的可以实现, livy为入口, 直接调用大数据组件的功能, 确实78不错
2, 78大数据不也是调用java API, d面试的, 公司大哥们, 问一堆原理原理, 你来, 你知道, 你写一个, 活都干不明白,问原来, s*
3, 做78个java后端, 啥nm都得管,s*

如果需要, 再看看hadoop需要提供的配置文件哪些:
如图