使用java代码来实现动态提交spark任务到集群

最新推荐文章于 2024-08-11 23:51:02 发布

L凝竹

最新推荐文章于 2024-08-11 23:51:02 发布

阅读量1.1w

点赞数 6

分类专栏：大数据技术之spark 文章标签：大数据 spark

本文链接：https://blog.csdn.net/xiaoliu_qq/article/details/80204757

版权

大数据技术之spark 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

场景

执行java代码的一个方法，这个动作触发提交spark任务到运行在yarn上的spark 集群

开始

Spark任务介绍

执行spark读取Hive中的一个表，这个表是用Hive来管理的HBASE表。统计这个表的总记录数。

具体代码如下：

        objectTable_count {
  def main(args: Array[String]): Unit = {

    import org.apache.spark.sql.{SaveMode, SparkSession}
    import org.apache.spark.{SparkConf}

    val sparkConf = new SparkConf().setAppName("test_hive_count").setMaster("yarn-client")
    val sparkSession = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate()
    val sqlContext = sparkSession.sqlContext

    val sql = " select count(1) from test.hbasehive_test "

    val result = sqlContext.sql(sql)

    result.write.format("jdbc").option("url", "jdbc:mysql://****:3306/test").option("dbtable", "result").option("user", "****").option("password", "******").mode(SaveMode.Overwrite).save()

    sparkSession.stop()

  }
}

尖叫提示：

这种方式提交spark任务读取Hive表，创建spark入口使用sqlContext做为入口直接使用sparkSession不行，会报错找不到Hive对应表

javaAPI调用提交介绍

使用的是spark提供的SparkLauncher 的API来提交任务

具体代码如下：

        import org.apache.spark.launcher.SparkLauncher;

import java.io.IOException;
import java.util.HashMap;

/**
 *
 * Created by xiaoliu on 2018-5-3.
 */
public class SubmitMain {

    public static void main(String args[]) {

        System.setProperty("HADOOP_USER_NAME", "hdfs");

        HashMap<String, String> map = new HashMap<String, String>();

        map.put("HADOOP_CONF_DIR", "/etc/hadoop/conf");

        map.put("YARN_CONF_DIR", "/etc/hadoop/conf");

        map.put("SPARK_CONF_DIR", "/etc/spark2/conf");

        map.put("SPARK_HOME", "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2");

        map.put("JAVA_HOME","/usr/java/jdk1.8.0_144");

        try {
            SparkLauncher spark = new SparkLauncher(map)

                    .setDeployMode("client")

                    .setAppResource("hdfs:///user/jars/spark_module-1.0-SNAPSHOT.jar")

                    .setMainClass("com.sinosoft.Table_count")

                    .setMaster("yarn-client")

                    .setConf(SparkLauncher.DRIVER_MEMORY, "1g")

                    .setConf(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH,"/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/jars/")

                    .setConf(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH,"/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/")

                    .setVerbose(true);

            // 启动spark任务

            Process process =spark.launch();
            InputStreamReaderRunnable inputStreamReaderRunnable = new InputStreamReaderRunnable(process.getInputStream(), "input");
            Thread inputThread = new Thread(inputStreamReaderRunnable, "LogStreamReader input");
            inputThread.start();

            InputStreamReaderRunnable errorStreamReaderRunnable = new InputStreamReaderRunnable(process.getErrorStream(), "error");
            Thread errorThread = new Thread(errorStreamReaderRunnable, "LogStreamReader error");
            errorThread.start();

            System.out.println("Waiting for finish...");
            int exitCode = process.waitFor();
            System.out.println("Finished! Exit code:" + exitCode);

        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

}

尖叫提示：

在上文中配置的hadoop、yarn、spark的配置目录必须有，否则会抛出找不到目录异常

这其中用到了一个自定义的log日志输出类

package com.liu.sparksubmit;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

/**
 *
 * Created by xiaoliu on 2018-5-5.
 */
public class InputStreamReaderRunnable implements Runnable {

    private BufferedReader reader;

    private String name;

    public InputStreamReaderRunnable(InputStream is, String name) {
        this.reader = new BufferedReader(new InputStreamReader(is));
        this.name = name;
    }

    public void run() {
        System.out.println("InputStream " + name + ":");
        try {
            String line = reader.readLine();
            while (line != null) {
                System.out.println(line);
                line = reader.readLine();
            }
            reader.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}