Spark SQL 通过hive读取hbase数据

最新推荐文章于 2021-08-06 20:39:07 发布
最新推荐文章于 2021-08-06 20:39:07 发布
阅读量1.8k 收藏 2
点赞数
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wx2007xing/article/details/88925346
版权

一、 环境

spark 2.3.3+hive-3.1.1+hbase-2.1.3

二、 配置

1、 配置hive的Metastore

参考连接：https://blog.csdn.net/wx2007xing/article/details/88878042

2、 复制hive目录下的hive-hbase-handler-3.1.1.jar到各个节点下spark安装目录的下的jars里

我的节点有4个：h1 h2 h3 h4

# scp /usr/local/software/hive-3.1.1/lib/hive-hbase-handler-3.1.1.jar 
root@h4:/usr/local/software/spark-2.3.3/jars/

3、 复制hbase目录lib下的jar包到各个节点spark的jars目录下

经测试，仅配置SPARK_CLASSPATH无效；理论上可行，一定是哪有点问题，以后再解决4/

# scp /usr/local/software/hbase-2.1.3/lib/*
root@h4:/usr/local/software/spark-2.3.3/jars/

4、 由于spark需要通过mysql连接hive的元数据，所有需要复制mysql-connector-java-5.1.35.jar到spark的jars目录，并在spark-env.sh里面添加

export SPARK_CLASSPATH=$SPARK_HOME/jars/mysql-connector-java-5.1.35.jar:$SPARK_CLASSPATH

5、 复制hive-site.xml和hbase-site.xml到spark的conf目录下

配置完毕

三、 测试

前提是hadoop已经启动

1、 启动hive的Metastore

2、 启动hbase

3、 启动spark

4、 使用spark-sql客户端

[root@h1 ~]# spark-sql 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/software/spark-2.3.3/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019-03-31 10:54:51 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-31 10:54:54 INFO  metastore:376 - Trying to connect to metastore with URI thrift://h1:9083
2019-03-31 10:54:54 INFO  metastore:472 - Connected to metastore.
2019-03-31 10:54:56 INFO  SessionState:641 - Created local directory: /tmp/0d35ee77-fbc6-4288-bf2d-f2900c0ec8cf_resources
2019-03-31 10:54:56 INFO  SessionState:641 - Created HDFS directory: /tmp/hive/root/0d35ee77-fbc6-4288-bf2d-f2900c0ec8cf
2019-03-31 10:54:56 INFO  SessionState:641 - Created local directory: /tmp/root/0d35ee77-fbc6-4288-bf2d-f2900c0ec8cf
2019-03-31 10:54:56 INFO  SessionState:641 - Created HDFS directory: /tmp/hive/root/0d35ee77-fbc6-4288-bf2d-f2900c0ec8cf/_tmp_space.db
2019-03-31 10:54:56 INFO  SparkContext:54 - Running Spark version 2.3.3
2019-03-31 10:54:57 INFO  SparkContext:54 - Submitted application: SparkSQL::192.168.247.11
2019-03-31 10:54:57 INFO  SecurityManager:54 - Changing view acls to: root
2019-03-31 10:54:57 INFO  SecurityManager:54 - Changing modify acls to: root
2019-03-31 10:54:57 INFO  SecurityManager:54 - Changing view acls groups to: 
2019-03-31 10:54:57 INFO  SecurityManager:54 - Changing modify acls groups to: 
2019-03-31 10:54:57 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2019-03-31 10:54:57 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 46069.
2019-03-31 10:54:57 INFO  SparkEnv:54 - Registering MapOutputTracker
2019-03-31 10:54:58 INFO  SparkEnv:54 - Registering BlockManagerMaster
2019-03-31 10:54:58 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-03-31 10:54:58 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2019-03-31 10:54:58 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-6e471f04-a669-48f0-afa8-83499db500f9
2019-03-31 10:54:58 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2019-03-31 10:54:58 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2019-03-31 10:54:58 INFO  log:192 - Logging initialized @10145ms
2019-03-31 10:54:58 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: 2018-09-05T05:11:46+08:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732
2019-03-31 10:54:58 INFO  Server:419 - Started @10496ms
2019-03-31 10:54:58 WARN  Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2019-03-31 10:54:59 INFO  AbstractConnector:278 - Started ServerConnector@3234d125{HTTP/1.1,[http/1.1]}{192.168.247.11:4041}
2019-03-31 10:54:59 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4041.
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48a0c8aa{/jobs,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@52ff99cd{/jobs/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4c2af006{/jobs/job,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7b676112{/jobs/job/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5578be42{/stages,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e49ce2b{/stages/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@136965e3{/stages/stage,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4364712f{/stages/stage/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1b7a52dd{/stages/pool,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f93dd4e{/stages/pool/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ad5be4a{/storage,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3ad85136{/storage/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@737d100a{/storage/rdd,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@12e5da86{/storage/rdd/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6535117e{/environment,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d1cbd0f{/environment/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fa13e6{/executors,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3af7d855{/executors/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77049094{/executors/threadDump,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f88bfbe{/executors/threadDump/json,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@59bbe88a{/static,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@53abfc07{/,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c8c16c0{/api,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f5b9db{/jobs/job/kill,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@507d64aa{/stages/stage/kill,null,AVAILABLE,@Spark}
2019-03-31 10:54:59 INFO  SparkUI:54 - Bound SparkUI to 192.168.247.11, and started at http://h1:4041
2019-03-31 10:54:59 INFO  Executor:54 - Starting executor ID driver on host localhost
2019-03-31 10:54:59 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42945.
2019-03-31 10:54:59 INFO  NettyBlockTransferService:54 - Server created on h1:42945
2019-03-31 10:54:59 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-03-31 10:54:59 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, h1, 42945, None)
2019-03-31 10:54:59 INFO  BlockManagerMasterEndpoint:54 - Registering block manager h1:42945 with 413.9 MB RAM, BlockManagerId(driver, h1, 42945, None)
2019-03-31 10:54:59 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, h1, 42945, None)
2019-03-31 10:54:59 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, h1, 42945, None)
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e4e8fdf{/metrics/json,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  SharedState:54 - loading hive config file: file:/usr/local/software/spark-2.3.3/conf/hive-site.xml
2019-03-31 10:55:00 INFO  SharedState:54 - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/root/spark-warehouse').
2019-03-31 10:55:00 INFO  SharedState:54 - Warehouse path is 'file:/root/spark-warehouse'.
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4364863{/SQL,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3625a016{/SQL/json,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d634127{/SQL/execution,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@d2e6b0b{/SQL/execution/json,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72b43104{/static/sql,null,AVAILABLE,@Spark}
2019-03-31 10:55:00 INFO  HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
2019-03-31 10:55:01 INFO  HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is file:/root/spark-warehouse
2019-03-31 10:55:01 INFO  metastore:291 - Mestastore configuration hive.metastore.warehouse.dir changed from /user/hive/warehouse to file:/root/spark-warehouse
2019-03-31 10:55:01 INFO  metastore:376 - Trying to connect to metastore with URI thrift://h1:9083
2019-03-31 10:55:01 INFO  metastore:472 - Connected to metastore.
2019-03-31 10:55:02 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
spark-sql> 

spark-sql> select * from test;
2019-03-31 10:56:08 INFO  deprecation:1173 - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2019-03-31 10:56:14 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 297.2 KB, free 413.6 MB)
2019-03-31 10:56:14 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.3 KB, free 413.6 MB)
2019-03-31 10:56:14 INFO  BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on h1:42945 (size: 25.3 KB, free: 413.9 MB)
2019-03-31 10:56:14 INFO  SparkContext:54 - Created broadcast 0 from 
2019-03-31 10:56:14 INFO  HBaseStorageHandler:219 - Configuring input job properties
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:host.name=h1
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_171
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.home=/usr/local/software/jdk1.8.0_171/jre
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.class.path=/usr/local/software/spark-2.3.3/conf/:/usr/local/software/spark-2.3.3/jars/activation-1.1.1.jar:/usr/local/software/spark-2.3.3/jars/aircompressor-0.8.jar:/usr/local/software/spark-2.3.3/jars/antlr-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/antlr-runtime-3.4.jar:/usr/local/software/spark-2.3.3/jars/antlr4-runtime-4.7.jar:/usr/local/software/spark-2.3.3/jars/aopalliance-1.0.jar:/usr/local/software/spark-2.3.3/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/local/software/spark-2.3.3/jars/apache-log4j-extras-1.2.17.jar:/usr/local/software/spark-2.3.3/jars/apacheds-i18n-2.0.0-M15.jar:/usr/local/software/spark-2.3.3/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/software/spark-2.3.3/jars/api-asn1-api-1.0.0-M20.jar:/usr/local/software/spark-2.3.3/jars/api-util-1.0.0-M20.jar:/usr/local/software/spark-2.3.3/jars/arpack_combined_all-0.1.jar:/usr/local/software/spark-2.3.3/jars/arrow-format-0.8.0.jar:/usr/local/software/spark-2.3.3/jars/arrow-memory-0.8.0.jar:/usr/local/software/spark-2.3.3/jars/arrow-vector-0.8.0.jar:/usr/local/software/spark-2.3.3/jars/automaton-1.11-8.jar:/usr/local/software/spark-2.3.3/jars/avro-1.7.7.jar:/usr/local/software/spark-2.3.3/jars/avro-ipc-1.7.7.jar:/usr/local/software/spark-2.3.3/jars/avro-mapred-1.7.7-hadoop2.jar:/usr/local/software/spark-2.3.3/jars/base64-2.3.8.jar:/usr/local/software/spark-2.3.3/jars/bcprov-jdk15on-1.58.jar:/usr/local/software/spark-2.3.3/jars/bonecp-0.8.0.RELEASE.jar:/usr/local/software/spark-2.3.3/jars/breeze-macros_2.11-0.13.2.jar:/usr/local/software/spark-2.3.3/jars/breeze_2.11-0.13.2.jar:/usr/local/software/spark-2.3.3/jars/calcite-avatica-1.2.0-incubating.jar:/usr/local/software/spark-2.3.3/jars/calcite-core-1.2.0-incubating.jar:/usr/local/software/spark-2.3.3/jars/calcite-linq4j-1.2.0-incubating.jar:/usr/local/software/spark-2.3.3/jars/chill-java-0.8.4.jar:/usr/local/software/spark-2.3.3/jars/chill_2.11-0.8.4.jar:/usr/local/software/spark-2.3.3/jars/commons-beanutils-1.7.0.jar:/usr/local/software/spark-2.3.3/jars/commons-beanutils-core-1.8.0.jar:/usr/local/software/spark-2.3.3/jars/commons-cli-1.2.jar:/usr/local/software/spark-2.3.3/jars/commons-codec-1.10.jar:/usr/local/software/spark-2.3.3/jars/commons-collections-3.2.2.jar:/usr/local/software/spark-2.3.3/jars/commons-compiler-3.0.8.jar:/usr/local/software/spark-2.3.3/jars/commons-compress-1.4.1.jar:/usr/local/software/spark-2.3.3/jars/commons-configuration-1.6.jar:/usr/local/software/spark-2.3.3/jars/commons-crypto-1.0.0.jar:/usr/local/software/spark-2.3.3/jars/commons-dbcp-1.4.jar:/usr/local/software/spark-2.3.3/jars/commons-digester-1.8.jar:/usr/local/software/spark-2.3.3/jars/commons-httpclient-3.1.jar:/usr/local/software/spark-2.3.3/jars/commons-io-2.4.jar:/usr/local/software/spark-2.3.3/jars/commons-lang-2.6.jar:/usr/local/software/spark-2.3.3/jars/commons-lang3-3.5.jar:/usr/local/software/spark-2.3.3/jars/commons-logging-1.1.3.jar:/usr/local/software/spark-2.3.3/jars/commons-math3-3.4.1.jar:/usr/local/software/spark-2.3.3/jars/commons-net-2.2.jar:/usr/local/software/spark-2.3.3/jars/commons-pool-1.5.4.jar:/usr/local/software/spark-2.3.3/jars/compress-lzf-1.0.3.jar:/usr/local/software/spark-2.3.3/jars/core-1.1.2.jar:/usr/local/software/spark-2.3.3/jars/curator-client-2.7.1.jar:/usr/local/software/spark-2.3.3/jars/curator-framework-2.7.1.jar:/usr/local/software/spark-2.3.3/jars/curator-recipes-2.7.1.jar:/usr/local/software/spark-2.3.3/jars/datanucleus-api-jdo-3.2.6.jar:/usr/local/software/spark-2.3.3/jars/datanucleus-core-3.2.10.jar:/usr/local/software/spark-2.3.3/jars/datanucleus-rdbms-3.2.9.jar:/usr/local/software/spark-2.3.3/jars/derby-10.12.1.1.jar:/usr/local/software/spark-2.3.3/jars/eigenbase-properties-1.1.5.jar:/usr/local/software/spark-2.3.3/jars/flatbuffers-1.2.0-3f79e055.jar:/usr/local/software/spark-2.3.3/jars/generex-1.0.1.jar:/usr/local/software/spark-2.3.3/jars/gson-2.2.4.jar:/usr/local/software/spark-2.3.3/jars/guava-14.0.1.jar:/usr/local/software/spark-2.3.3/jars/guice-3.0.jar:/usr/local/software/spark-2.3.3/jars/guice-servlet-3.0.jar:/usr/local/software/spark-2.3.3/jars/hadoop-annotations-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-auth-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-client-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-common-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-hdfs-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-app-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-common-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-core-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-api-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-client-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-common-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-common-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/local/software/spark-2.3.3/jars/hive-beeline-1.2.1.spark2.jar:/usr/local/software/spark-2.3.3/jars/hive-cli-1.2.1.spark2.jar:/usr/local/software/spark-2.3.3/jars/hive-exec-1.2.1.spark2.jar:/usr/local/software/spark-2.3.3/jars/hive-jdbc-1.2.1.spark2.jar:/usr/local/software/spark-2.3.3/jars/hive-metastore-1.2.1.spark2.jar:/usr/local/software/spark-2.3.3/jars/hk2-api-2.4.0-b34.jar:/usr/local/software/spark-2.3.3/jars/hk2-locator-2.4.0-b34.jar:/usr/local/software/spark-2.3.3/jars/hk2-utils-2.4.0-b34.jar:/usr/local/software/spark-2.3.3/jars/hppc-0.7.2.jar:/usr/local/software/spark-2.3.3/jars/htrace-core-3.1.0-incubating.jar:/usr/local/software/spark-2.3.3/jars/httpclient-4.5.4.jar:/usr/local/software/spark-2.3.3/jars/httpcore-4.4.8.jar:/usr/local/software/spark-2.3.3/jars/ivy-2.4.0.jar:/usr/local/software/spark-2.3.3/jars/jackson-annotations-2.6.7.jar:/usr/local/software/spark-2.3.3/jars/jackson-core-2.6.7.jar:/usr/local/software/spark-2.3.3/jars/jackson-core-asl-1.9.13.jar:/usr/local/software/spark-2.3.3/jars/jackson-databind-2.6.7.1.jar:/usr/local/software/spark-2.3.3/jars/jackson-dataformat-yaml-2.6.7.jar:/usr/local/software/spark-2.3.3/jars/jackson-jaxrs-1.9.13.jar:/usr/local/software/spark-2.3.3/jars/jackson-mapper-asl-1.9.13.jar:/usr/local/software/spark-2.3.3/jars/jackson-module-jaxb-annotations-2.6.7.jar:/usr/local/software/spark-2.3.3/jars/jackson-module-paranamer-2.7.9.jar:/usr/local/software/spark-2.3.3/jars/jackson-module-scala_2.11-2.6.7.1.jar:/usr/local/software/spark-2.3.3/jars/jackson-xc-1.9.13.jar:/usr/local/software/spark-2.3.3/jars/janino-3.0.8.jar:/usr/local/software/spark-2.3.3/jars/java-xmlbuilder-1.1.jar:/usr/local/software/spark-2.3.3/jars/JavaEWAH-0.3.2.jar:/usr/local/software/spark-2.3.3/jars/javassist-3.18.1-GA.jar:/usr/local/software/spark-2.3.3/jars/javax.annotation-api-1.2.jar:/usr/local/software/spark-2.3.3/jars/javax.inject-1.jar:/usr/local/software/spark-2.3.3/jars/javax.inject-2.4.0-b34.jar:/usr/local/software/spark-2.3.3/jars/javax.servlet-api-3.1.0.jar:/usr/local/software/spark-2.3.3/jars/javax.ws.rs-api-2.0.1.jar:/usr/local/software/spark-2.3.3/jars/javolution-5.5.1.jar:/usr/local/software/spark-2.3.3/jars/jaxb-api-2.2.2.jar:/usr/local/software/spark-2.3.3/jars/jcl-over-slf4j-1.7.16.jar:/usr/local/software/spark-2.3.3/jars/jdo-api-3.0.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-client-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-common-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-container-servlet-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-container-servlet-core-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-guava-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-media-jaxb-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jersey-server-2.22.2.jar:/usr/local/software/spark-2.3.3/jars/jets3t-0.9.4.jar:/usr/local/software/spark-2.3.3/jars/jetty-6.1.26.jar:/usr/local/software/spark-2.3.3/jars/jetty-util-6.1.26.jar:/usr/local/software/spark-2.3.3/jars/jline-2.12.1.jar:/usr/local/software/spark-2.3.3/jars/joda-time-2.9.3.jar:/usr/local/software/spark-2.3.3/jars/jodd-core-3.5.2.jar:/usr/local/software/spark-2.3.3/jars/jpam-1.1.jar:/usr/local/software/spark-2.3.3/jars/json4s-ast_2.11-3.2.11.jar:/usr/local/software/spark-2.3.3/jars/json4s-core_2.11-3.2.11.jar:/usr/local/software/spark-2.3.3/jars/json4s-jackson_2.11-3.2.11.jar:/usr/local/software/spark-2.3.3/jars/jsp-api-2.1.jar:/usr/local/software/spark-2.3.3/jars/jsr305-1.3.9.jar:/usr/local/software/spark-2.3.3/jars/jta-1.1.jar:/usr/local/software/spark-2.3.3/jars/jtransforms-2.4.0.jar:/usr/local/software/spark-2.3.3/jars/jul-to-slf4j-1.7.16.jar:/usr/local/software/spark-2.3.3/jars/kryo-shaded-3.0.3.jar:/usr/local/software/spark-2.3.3/jars/kubernetes-client-3.0.0.jar:/usr/local/software/spark-2.3.3/jars/kubernetes-model-2.0.0.jar:/usr/local/software/spark-2.3.3/jars/leveldbjni-all-1.8.jar:/usr/local/software/spark-2.3.3/jars/libfb303-0.9.3.jar:/usr/local/software/spark-2.3.3/jars/libthrift-0.9.3.jar:/usr/local/software/spark-2.3.3/jars/log4j-1.2.17.jar:/usr/local/software/spark-2.3.3/jars/logging-interceptor-3.8.1.jar:/usr/local/software/spark-2.3.3/jars/lz4-java-1.4.0.jar:/usr/local/software/spark-2.3.3/jars/machinist_2.11-0.6.1.jar:/usr/local/software/spark-2.3.3/jars/macro-compat_2.11-1.1.1.jar:/usr/local/software/spark-2.3.3/jars/mesos-1.4.0-shaded-protobuf.jar:/usr/local/software/spark-2.3.3/jars/metrics-core-3.1.5.jar:/usr/local/software/spark-2.3.3/jars/metrics-graphite-3.1.5.jar:/usr/local/software/spark-2.3.3/jars/metrics-json-3.1.5.jar:/usr/local/software/spark-2.3.3/jars/metrics-jvm-3.1.5.jar:/usr/local/software/spark-2.3.3/jars/minlog-1.3.0.jar:/usr/local/software/spark-2.3.3/jars/netty-3.9.9.Final.jar:/usr/local/software/spark-2.3.3/jars/netty-all-4.1.17.Final.jar:/usr/local/software/spark-2.3.3/jars/objenesis-2.1.jar:/usr/local/software/spark-2.3.3/jars/okhttp-3.8.1.jar:/usr/local/software/spark-2.3.3/jars/okio-1.13.0.jar:/usr/local/software/spark-2.3.3/jars/opencsv-2.3.jar:/usr/local/software/spark-2.3.3/jars/orc-core-1.4.4-nohive.jar:/usr/local/software/spark-2.3.3/jars/orc-mapreduce-1.4.4-nohive.jar:/usr/local/software/spark-2.3.3/jars/oro-2.0.8.jar:/usr/local/software/spark-2.3.3/jars/osgi-resource-locator-1.0.1.jar:/usr/local/software/spark-2.3.3/jars/paranamer-2.8.jar:/usr/local/software/spark-2.3.3/jars/parquet-column-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/parquet-common-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/parquet-encoding-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/parquet-format-2.3.1.jar:/usr/local/software/spark-2.3.3/jars/parquet-hadoop-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/parquet-hadoop-bundle-1.6.0.jar:/usr/local/software/spark-2.3.3/jars/parquet-jackson-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/protobuf-java-2.5.0.jar:/usr/local/software/spark-2.3.3/jars/py4j-0.10.7.jar:/usr/local/software/spark-2.3.3/jars/pyrolite-4.13.jar:/usr/local/software/spark-2.3.3/jars/RoaringBitmap-0.5.11.jar:/usr/local/software/spark-2.3.3/jars/scala-compiler-2.11.8.jar:/usr/local/software/spark-2.3.3/jars/scala-library-2.11.8.jar:/usr/local/software/spark-2.3.3/jars/scala-parser-combinators_2.11-1.0.4.jar:/usr/local/software/spark-2.3.3/jars/scala-reflect-2.11.8.jar:/usr/local/software/spark-2.3.3/jars/scala-xml_2.11-1.0.5.jar:/usr/local/software/spark-2.3.3/jars/scalap-2.11.8.jar:/usr/local/software/spark-2.3.3/jars/shapeless_2.11-2.3.2.jar:/usr/local/software/spark-2.3.3/jars/slf4j-api-1.7.16.jar:/usr/local/software/spark-2.3.3/jars/slf4j-log4j12-1.7.16.jar:/usr/local/software/spark-2.3.3/jars/snakeyaml-1.15.jar:/usr/local/software/spark-2.3.3/jars/snappy-0.2.jar:/usr/local/software/spark-2.3.3/jars/snappy-java-1.1.2.6.jar:/usr/local/software/spark-2.3.3/jars/spark-catalyst_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-core_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-graphx_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-hive-thriftserver_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-hive_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-kubernetes_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-kvstore_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-launcher_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-mesos_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-mllib-local_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-mllib_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-network-common_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-network-shuffle_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-repl_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-sketch_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-sql_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-streaming_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-tags_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-unsafe_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spark-yarn_2.11-2.3.3.jar:/usr/local/software/spark-2.3.3/jars/spire-macros_2.11-0.13.0.jar:/usr/local/software/spark-2.3.3/jars/spire_2.11-0.13.0.jar:/usr/local/software/spark-2.3.3/jars/ST4-4.0.4.jar:/usr/local/software/spark-2.3.3/jars/stax-api-1.0-2.jar:/usr/local/software/spark-2.3.3/jars/stax-api-1.0.1.jar:/usr/local/software/spark-2.3.3/jars/stream-2.7.0.jar:/usr/local/software/spark-2.3.3/jars/stringtemplate-3.2.1.jar:/usr/local/software/spark-2.3.3/jars/super-csv-2.2.0.jar:/usr/local/software/spark-2.3.3/jars/univocity-parsers-2.5.9.jar:/usr/local/software/spark-2.3.3/jars/validation-api-1.1.0.Final.jar:/usr/local/software/spark-2.3.3/jars/xbean-asm5-shaded-4.4.jar:/usr/local/software/spark-2.3.3/jars/xercesImpl-2.9.1.jar:/usr/local/software/spark-2.3.3/jars/xmlenc-0.52.jar:/usr/local/software/spark-2.3.3/jars/xz-1.0.jar:/usr/local/software/spark-2.3.3/jars/zjsonpatch-0.3.0.jar:/usr/local/software/spark-2.3.3/jars/zookeeper-3.4.6.jar:/usr/local/software/spark-2.3.3/jars/zstd-jni-1.3.2-2.jar:/usr/local/software/spark-2.3.3/jars/mysql-connector-java-5.1.35.jar:/usr/local/software/spark-2.3.3/jars/hive-hbase-handler-3.1.1.jar:/usr/local/software/spark-2.3.3/jars/aopalliance-repackaged-2.5.0-b32.jar:/usr/local/software/spark-2.3.3/jars/asm-3.1.jar:/usr/local/software/spark-2.3.3/jars/commons-daemon-1.0.13.jar:/usr/local/software/spark-2.3.3/jars/commons-io-2.5.jar:/usr/local/software/spark-2.3.3/jars/commons-lang3-3.6.jar:/usr/local/software/spark-2.3.3/jars/commons-math3-3.6.1.jar:/usr/local/software/spark-2.3.3/jars/commons-net-3.1.jar:/usr/local/software/spark-2.3.3/jars/curator-client-4.0.0.jar:/usr/local/software/spark-2.3.3/jars/curator-framework-4.0.0.jar:/usr/local/software/spark-2.3.3/jars/curator-recipes-4.0.0.jar:/usr/local/software/spark-2.3.3/jars/disruptor-3.3.6.jar:/usr/local/software/spark-2.3.3/jars/guava-11.0.2.jar:/usr/local/software/spark-2.3.3/jars/hadoop-annotations-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-auth-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-client-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-common-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-common-2.7.7-tests.jar:/usr/local/software/spark-2.3.3/jars/hadoop-hdfs-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-hdfs-2.7.7-tests.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-app-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-common-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-core-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-hs-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-jobclient-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-mapreduce-client-shuffle-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-minicluster-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-api-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-client-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-common-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-applicationhistoryservice-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-common-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-nodemanager-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-resourcemanager-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-tests-2.7.7-tests.jar:/usr/local/software/spark-2.3.3/jars/hadoop-yarn-server-web-proxy-2.7.7.jar:/usr/local/software/spark-2.3.3/jars/hamcrest-core-1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-annotations-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-annotations-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-client-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-common-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-common-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-endpoint-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-examples-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-external-blockcache-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-hadoop2-compat-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-hadoop2-compat-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-hadoop-compat-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-hadoop-compat-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-http-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-it-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-it-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-mapreduce-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-mapreduce-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-metrics-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-metrics-api-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-procedure-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-protocol-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-protocol-shaded-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-replication-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-resource-bundle-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-rest-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-rsgroup-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-rsgroup-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-server-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-server-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hbase-shaded-miscellaneous-2.1.0.jar:/usr/local/software/spark-2.3.3/jars/hbase-shaded-netty-2.1.0.jar:/usr/local/software/spark-2.3.3/jars/hbase-shaded-protobuf-2.1.0.jar:/usr/local/software/spark-2.3.3/jars/hbase-shell-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-testing-util-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-thrift-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-zookeeper-2.1.3.jar:/usr/local/software/spark-2.3.3/jars/hbase-zookeeper-2.1.3-tests.jar:/usr/local/software/spark-2.3.3/jars/hk2-api-2.5.0-b32.jar:/usr/local/software/spark-2.3.3/jars/hk2-locator-2.5.0-b32.jar:/usr/local/software/spark-2.3.3/jars/hk2-utils-2.5.0-b32.jar:/usr/local/software/spark-2.3.3/jars/htrace-core4-4.2.0-incubating.jar:/usr/local/software/spark-2.3.3/jars/httpclient-4.5.3.jar:/usr/local/software/spark-2.3.3/jars/httpcore-4.4.6.jar:/usr/local/software/spark-2.3.3/jars/jackson-annotations-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-core-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-databind-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-jaxrs-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/jackson-jaxrs-base-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-jaxrs-json-provider-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-module-jaxb-annotations-2.9.2.jar:/usr/local/software/spark-2.3.3/jars/jackson-xc-1.8.3.jar:/usr/local/software/spark-2.3.3/jars/jamon-runtime-2.4.1.jar:/usr/local/software/spark-2.3.3/jars/javassist-3.20.0-GA.jar:/usr/local/software/spark-2.3.3/jars/javax.el-3.0.1-b08.jar:/usr/local/software/spark-2.3.3/jars/javax.inject-2.5.0-b32.jar:/usr/local/software/spark-2.3.3/jars/java-xmlbuilder-0.4.jar:/usr/local/software/spark-2.3.3/jars/javax.servlet.jsp-2.3.2.jar:/usr/local/software/spark-2.3.3/jars/javax.servlet.jsp-api-2.3.1.jar:/usr/local/software/spark-2.3.3/jars/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/usr/local/software/spark-2.3.3/jars/javax.servlet.jsp.jstl-1.2.2.jar:/usr/local/software/spark-2.3.3/jars/jaxb-api-2.2.12.jar:/usr/local/software/spark-2.3.3/jars/jaxb-impl-2.2.3-1.jar:/usr/local/software/spark-2.3.3/jars/jcodings-1.0.18.jar:/usr/local/software/spark-2.3.3/jars/jersey-client-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-common-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-container-servlet-core-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-guava-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-media-jaxb-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jersey-server-2.25.1.jar:/usr/local/software/spark-2.3.3/jars/jets3t-0.9.0.jar:/usr/local/software/spark-2.3.3/jars/jettison-1.3.8.jar:/usr/local/software/spark-2.3.3/jars/jetty-http-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-io-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-jmx-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-jsp-9.2.26.v20180806.jar:/usr/local/software/spark-2.3.3/jars/jetty-schemas-3.1.M0.jar:/usr/local/software/spark-2.3.3/jars/jetty-security-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-server-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-servlet-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-sslengine-6.1.26.jar:/usr/local/software/spark-2.3.3/jars/jetty-util-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-util-ajax-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-webapp-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/jetty-xml-9.3.25.v20180904.jar:/usr/local/software/spark-2.3.3/jars/joni-2.1.11.jar:/usr/local/software/spark-2.3.3/jars/jsch-0.1.54.jar:/usr/local/software/spark-2.3.3/jars/junit-4.12.jar:/usr/local/software/spark-2.3.3/jars/libthrift-0.12.0.jar:/usr/local/software/spark-2.3.3/jars/metrics-core-3.2.1.jar:/usr/local/software/spark-2.3.3/jars/netty-all-4.0.23.Final.jar:/usr/local/software/spark-2.3.3/jars/org.eclipse.jdt.core-3.8.2.v20130121.jar:/usr/local/software/spark-2.3.3/jars/paranamer-2.3.jar:/usr/local/software/spark-2.3.3/jars/snappy-java-1.0.5.jar:/usr/local/software/spark-2.3.3/jars/spymemcached-2.12.2.jar:/usr/local/software/spark-2.3.3/jars/zookeeper-3.4.10.jar:/usr/local/software/hadoop-3.1.2/etc/hadoop/:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-compress-1.18.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-io-2.5.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/curator-client-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/curator-framework-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/hadoop-annotations-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/hadoop-auth-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-databind-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/json-smart-2.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/netty-3.10.5.Final.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/zookeeper-3.4.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/hadoop-nfs-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/common/hadoop-kms-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-auth-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-annotations-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-api-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-client-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-registry-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-common-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-router-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-api-3.1.2.jar:/usr/local/software/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-core-3.1.2.jar
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:os.name=Linux
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:os.version=3.10.0-862.3.3.el7.x86_64
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:user.name=root
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:user.home=/root
2019-03-31 10:56:15 INFO  ZooKeeper:100 - Client environment:user.dir=/root
2019-03-31 10:56:15 INFO  ZooKeeper:438 - Initiating client connection, connectString=h1:2181,h2:2181,h3:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$25/989714742@743514be
2019-03-31 10:56:15 INFO  ClientCnxn:975 - Opening socket connection to server h3/192.168.247.13:2181. Will not attempt to authenticate using SASL (unknown error)
2019-03-31 10:56:15 INFO  ClientCnxn:852 - Socket connection established to h3/192.168.247.13:2181, initiating session
2019-03-31 10:56:15 INFO  ClientCnxn:1235 - Session establishment complete on server h3/192.168.247.13:2181, sessionid = 0x3000001a9360012, negotiated timeout = 40000
2019-03-31 10:56:15 INFO  RegionSizeCalculator:76 - Calculating region sizes for table "test".
2019-03-31 10:56:47 INFO  ZooKeeper:684 - Session: 0x3000001a9360012 closed
2019-03-31 10:56:47 INFO  ClientCnxn:512 - EventThread shut down
2019-03-31 10:56:47 INFO  SparkContext:54 - Starting job: processCmd at CliDriver.java:376
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Got job 0 (processCmd at CliDriver.java:376) with 1 output partitions
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Final stage: ResultStage 0 (processCmd at CliDriver.java:376)
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Parents of final stage: List()
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Missing parents: List()
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[4] at processCmd at CliDriver.java:376), which has no missing parents
2019-03-31 10:56:47 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 17.0 KB, free 413.6 MB)
2019-03-31 10:56:47 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.2 KB, free 413.6 MB)
2019-03-31 10:56:47 INFO  BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on h1:42945 (size: 9.2 KB, free: 413.9 MB)
2019-03-31 10:56:47 INFO  SparkContext:54 - Created broadcast 1 from broadcast at DAGScheduler.scala:1039
2019-03-31 10:56:47 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at processCmd at CliDriver.java:376) (first 15 tasks are for partitions Vector(0))
2019-03-31 10:56:47 INFO  TaskSchedulerImpl:54 - Adding task set 0.0 with 1 tasks
2019-03-31 10:56:47 INFO  TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 8002 bytes)
2019-03-31 10:56:47 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2019-03-31 10:56:48 INFO  HadoopRDD:54 - Input split: HBase table split(table name: test, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":2,"maxResultSize":-1,"families":{"user":["id","name"]},"caching":-1,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: , end row: , region location: h3, encoded region name: bff331b5b3b02e41966096fd585d2d67)
2019-03-31 10:56:48 INFO  ZooKeeper:438 - Initiating client connection, connectString=h1:2181,h2:2181,h3:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$25/989714742@743514be
2019-03-31 10:56:48 INFO  ClientCnxn:975 - Opening socket connection to server h1/0.0.0.0:2181. Will not attempt to authenticate using SASL (unknown error)
2019-03-31 10:56:48 INFO  ClientCnxn:852 - Socket connection established to h1/0.0.0.0:2181, initiating session
2019-03-31 10:56:48 INFO  ClientCnxn:1235 - Session establishment complete on server h1/0.0.0.0:2181, sessionid = 0x100003534240006, negotiated timeout = 40000
2019-03-31 10:56:48 INFO  TableInputFormatBase:181 - Input split length: 0 bytes.
2019-03-31 10:56:49 INFO  CodeGenerator:54 - Code generated in 1065.228664 ms
2019-03-31 10:56:50 INFO  ZooKeeper:684 - Session: 0x100003534240006 closed
2019-03-31 10:56:50 INFO  ClientCnxn:512 - EventThread shut down
2019-03-31 10:56:50 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 1382 bytes result sent to driver
2019-03-31 10:56:50 INFO  TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 2459 ms on localhost (executor driver) (1/1)
2019-03-31 10:56:50 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool 
2019-03-31 10:56:50 INFO  DAGScheduler:54 - ResultStage 0 (processCmd at CliDriver.java:376) finished in 2.766 s
2019-03-31 10:56:50 INFO  DAGScheduler:54 - Job 0 finished: processCmd at CliDriver.java:376, took 3.014328 s
1	1	wx
2	2	user
Time taken: 43.769 seconds, Fetched 2 row(s)
2019-03-31 10:56:50 INFO  SparkSQLCLIDriver:951 - Time taken: 43.769 seconds, Fetched 2 row(s)
spark-sql>

 

猿星人崛起
关注
  • 0
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
【Spark实战系列】Spark SQL 读取 hbase 映射到 hive 表的数据
JasonLee实时计算
02-29 2606
这篇文章主要来介绍一下用Spark SQL 如何读取hbase映射到hive的表,有的时候我们的数据存储在hbase中,但是想要用spark去分析,这时候就要用spark读取hbase的数据,spark读取hbase中的数据有很多种方式,今天来介绍一种比较方便的方法. 之前也用newAPIHadoopRDD的方式读取过hbase,但是这种方式读取后是一个RDD,虽然通过RDD也可以注册成表,但是...
spark sql 读取hbase中数据保存到hive
lin的专栏
04-25 2290
新建SaprkReadHbase类： package sparksql import java.util.Properties import org.apache.hadoop.hbase.HBaseConfiguration import org.apache.hadoop.hbase.client.Result import org.apache.hadoop.hbase.io.Immut...
SparkSql集成hive访问hbase
zh350229319的专栏
05-02 1478
SparkSql集成hive访问hbase标签（空格分隔）： spark SparkSql需要用到hive的元数据，有两种方式集成方式为使用buildin的hive直接连接hive的mysql元数据库 注：Spark自带的hive版本为1.2.1，实际测试发现，如果使用了高版本的hive集成，spark会自动将hive源数据库中的metastore version改为1.2.0，导致hiv
贝壳基于Spark的HiveToHBase实践
大数据星球-浪尖
06-15 475
导读：本文详细介绍了如何将Hive里的数据快速稳定的写进HBase中。由于数据量比较大，我们采取的是HBase官方提供的bulkload方式来避免HBase put api写入压力大的缺陷...
手撸代码-Spark+BulkLoad从Hive读取数据到HBase
epitomizelu的专栏
08-06 435
object HBaseBulkLoader { /** * 依据不同表的数据文件，提取对应数据，封装到KeyValue对象中 */ def getLineToData(line: String, family: String, fieldNames: TreeMap[String, Int] ): List[(ImmutableBytesWritable, KeyValue)] = { val length =
Spark使用Java读Hive写入HBase
一个数据小开发的博客
01-28 782
Spark使用Java读Hive写入HBase Spark读取Hive数据得方法在下面之前得一篇博客中有提及，在此就不重复提了，如果感兴趣得可以查阅如下连接中得博客 Spark使用Java读Hive得博客：https://blog.csdn.net/Aaron_ch/article/details/113151157 rowDataset.foreachPartition((ForeachPartitionFunction<Row>) t -> { ...
JDBC
RoronaYasuo的博客
02-23 174
1//import java.sql.Connection; //import java.sql.DriverManager; import java.sql.*; //import javax.sql.ConnectionEvent; //import com.mysql.jdbc.Driver; /* 步骤 1. 注册驱动. 告知jvm使用的是哪一个数据库的驱动 2. 获得连接. 使用...
Atlas Spark SQL血缘分析，HBASE
05-25
Spark SQL的强大之处在于其能够与多种数据源（如HDFS、Hive、HBase等）交互，提供了高性能的数据处理能力。 HBase是基于Hadoop的分布式数据库，设计用于处理大规模、非结构化数据。它是一个NoSQL数据库，提供实时...
建立Hive和Hbase的映射关系，通过Spark将Hive表中数据导入ClickHouse
07-31
这样，Hive查询可以通过HBase的表获取数据，反之亦然。映射的关键在于定义正确的SerDe（序列化/反序列化）类和配置参数，以确保Hive能理解HBase的行键和列族结构。 接下来，Spark作为一个分布式计算框架，提供了一...
spark-2.4.0-hive-hbase-Api.7z
10-31
Spark与HBase的结合，通过Spark的HBase连接器，可以实现高效的数据导入导出和实时分析。 在这个压缩包中，"spark-2.4.0-old"可能指的是Spark的旧版本库，可能是为了兼容性或者回滚目的而保留的。这个文件可能包含了...
scala-hive-HBASE-Api.7z
10-31
2. **Hive**：Apache Hive是由Facebook开源的一个数据仓库工具，它可以将结构化的数据文件映射为一张数据库表，并提供SQL（HQL，Hive Query Language）查询功能，方便大数据分析。Hive主要针对大规模数据集进行...
清华大学精品大数据实战课程（Hadoop、Hbase、Hive、Spark）PPT课件含习题（29页） 第6章 Spark SQL.pptx
05-23
4. **多数据源支持**：Spark SQL能够读取和写入多种数据存储系统，包括Hadoop的文件系统（HDFS）、列式存储Parquet、NoSQL数据库HBase以及Hive的数据仓库。 5. **分布式SQL引擎**：Spark SQL可以通过JDBC/ODBC接口...
spark 读取hive数据 写入hbase
qq_31866793的博客
08-27 3328
1，saveAsNewAPIHadoopDataset批量写入（千万级别以下使用） import org.apache.hadoop.hbase.client._ import org.apache.hadoop.hbase.io.ImmutableBytesWritable import org.apache.hadoop.hbase.mapreduce.TableOutputForma...
Spark读取Hive数据写入Hbase
Baron_ND的博客
04-17 1953
使用spark方式写入 在使用Spark时经常需要把数据落入HBase中，如果使用普通的Java API，写入会速度很慢。还好Spark提供了Bulk写入方式的接口。那么Bulk写入与普通写入相比有什么优势呢？ BulkLoad不会写WAL，也不会产生flush以及split。 如果我们大量调用PUT接口插入数据，可能会导致大量的GC操作。除了影响性能之外，严重时甚至可能会对HBase节点的...
Spark读取hive表数据并将处理的数据写入HBase
石榴姐yyds
04-25 1140
package com.nbdpt.work4_hive2hbase2019 import com.nbdpt.util.BaseUtil import org.apache.hadoop.hbase.client.{ConnectionFactory, Get, Put} import org.apache.hadoop.hbase.io.ImmutableBytesWritable import org.apache.hadoop.hbase.mapreduce.TableOutputFormat .
《Spark系列》建立Hive和Hbase的映射关系，通过Spark将Hive表中数据导入ClickHouse
https://blog.datasource.space
07-31 2158
创建列簇 按行导入数据
spark-toHive、tomysql、tohbase
weixin_50831726的博客
01-10 284
一、Spark to Hive 1、打开虚拟机，将Hive的配置文件（conf下的hive-site.xml)拷贝到Spark的配置文件目录下(conf)，软拷贝硬拷贝皆可以 ln -s /opt/software/hadoop/hive110/conf/hive-site.xml /opt/software/hadoop/spark//conf/hive-site.xml 2、拷贝jar包（mysql-connector-java-5.1.32.jar 拷贝到Spark的jars目录下 cp /op
SparkSQL/SparkStreaming读写Hive/Mysql/Hbase/Kafka
u013998466的博客
03-27 1870
Spark读写Hive/Mysql/Hbase/KafkaSpark读写Hive/Mysql/Hbase/Kafka一、工具类、配置类（一）SparkUtils (SparkSession启动类)（二）ConnectUtils (连接/配置参数工具类)二、Mysql to Hive三、Hbase to Hive四、Hive to Hive五、Kafka to Hive/Hbase/Kafka（一）...
【中科院1区】Matlab实现天鹰优化算法AO-RF锂电池健康状态估计算法研究.rar
最新发布
08-02
1.版本：matlab2014/2019a/2024a 2.附赠案例数据可直接运行matlab程序。 3.代码特点：参数化编程、参数可方便更改、代码编程思路清晰、注释明细。 4.适用对象：计算机，电子信息工程、数学等专业的大学生课程设计、期末大作业和毕业设计。 5.作者介绍：某大厂资深算法工程师，从事Matlab算法仿真工作10年；擅长智能优化算法、神经网络预测、信号处理、元胞自动机等多种领域的算法仿真实验，更多仿真源码、数据集定制私信+。 替换数据可以直接使用，注释清楚，适合新手
spark sql与hive hbase mysql集成
03-16
Spark SQL可以与Hive、HBase和MySQL集成。 与Hive集成：Spark SQL可以通过HiveContext连接到Hive，使用Hive的元数据和查询语言。这样可以在Spark中使用Hive表和查询，同时也可以在Hive中使用Spark SQL的功能。 与HBase集成：Spark SQL可以通过HBase数据源连接到HBase，读取和写入HBase表。这样可以在Spark中使用HBase表，同时也可以在HBase中使用Spark SQL的功能。 与MySQL集成：Spark SQL可以通过JDBC连接到MySQL，读取和写入MySQL表。这样可以在Spark中使用MySQL表，同时也可以在MySQL中使用Spark SQL的功能。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值