SparkSQL配置和使用初探

最新推荐文章于 2021-07-14 16:23:43 发布
最新推荐文章于 2021-07-14 16:23:43 发布
阅读量1k 收藏 1
点赞数
分类专栏： Spark应用 文章标签： Spark 大数据
原文  http://www.cnblogs.com/byrhuangqiang/p/4012087.html

1.环境

  • OS:Red Hat Enterprise Linux Server release 6.4 (Santiago)
  • Hadoop:Hadoop 2.4.1
  • Hive：0.11.0
  • JDK:1.7.0_60
  • Spark:1.1.0（内置SparkSQL）
  • Scala:2.11.2

2.Spark集群规划

  • 账户:ebupt
  • master:eb174
  • slaves:eb174、eb175、eb176

3.SparkSQL发展历史

2014年9月11日，发布Spark1.1.0。Spark从1.0开始引入SparkSQL。Spark1.1.0变化较大是SparkSQL和MLlib。具体参见 release note 。

SparkSQL的前身是Shark。由于Shark自身的不完善，2014年6月1日Reynold Xin宣布：停止对Shark的开发。SparkSQL抛弃原有Shark的代码，汲取了Shark的一些优点，如内存列存储（In-Memory Columnar Storage）、Hive兼容性等，重新开发SparkSQL。

4.配置

  1. 安装配置同Spark-0.9.1（参见博文: Spark、Shark集群安装部署及遇到的问题解决 ）
  2. 将$HIVE_HOME/conf/hive-site.xml配置文件拷贝到$SPARK_HOME/conf目录下。
  3. 将$HADOOP_HOME/etc/hadoop/hdfs-site.xml配置文件拷贝到$SPARK_HOME/conf目录下。

5.运行

  1. 启动Spark集群
  2. 启动SparkSQL Client： ./spark/bin/spark-sql --master spark://eb174:7077 --executor-memory 3g
  3. 运行SQL，访问hive的表：spark-sql> select count(*) from test.t1;
14/10/08 20:46:04 INFO ParseDriver: Parsing command: select count(*) from test.t1
14/10/08 20:46:05 INFO ParseDriver: Parse Completed
14/10/08 20:46:05 INFO metastore: Trying to connect to metastore with URI thrift://eb170:9083
14/10/08 20:46:05 INFO metastore: Waiting 1 seconds before next connection attempt.
14/10/08 20:46:06 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@eb174:55408/user/Executor#1282322316] with ID 2
14/10/08 20:46:06 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@eb176:56138/user/Executor#-264112470] with ID 0
14/10/08 20:46:06 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@eb175:43791/user/Executor#-996481867] with ID 1
14/10/08 20:46:06 INFO BlockManagerMasterActor: Registering block manager eb174:54967 with 265.4 MB RAM
14/10/08 20:46:06 INFO BlockManagerMasterActor: Registering block manager eb176:60783 with 265.4 MB RAM
14/10/08 20:46:06 INFO BlockManagerMasterActor: Registering block manager eb175:35197 with 265.4 MB RAM
14/10/08 20:46:06 INFO metastore: Connected to metastore.
14/10/08 20:46:07 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/10/08 20:46:07 INFO MemoryStore: ensureFreeSpace(406982) called with curMem=0, maxMem=278302556
14/10/08 20:46:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 397.4 KB, free 265.0 MB)
14/10/08 20:46:07 INFO MemoryStore: ensureFreeSpace(25198) called with curMem=406982, maxMem=278302556
14/10/08 20:46:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.6 KB, free 265.0 MB)
14/10/08 20:46:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb174:49971 (size: 24.6 KB, free: 265.4 MB)
14/10/08 20:46:07 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
14/10/08 20:46:07 INFO SparkContext: Starting job: collect at HiveContext.scala:415
14/10/08 20:46:08 INFO FileInputFormat: Total input paths to process : 1
14/10/08 20:46:08 INFO DAGScheduler: Registering RDD 5 (mapPartitions at Exchange.scala:86)
14/10/08 20:46:08 INFO DAGScheduler: Got job 0 (collect at HiveContext.scala:415) with 1 output partitions (allowLocal=false)
14/10/08 20:46:08 INFO DAGScheduler: Final stage: Stage 0(collect at HiveContext.scala:415)
14/10/08 20:46:08 INFO DAGScheduler: Parents of final stage: List(Stage 1)
14/10/08 20:46:08 INFO DAGScheduler: Missing parents: List(Stage 1)
14/10/08 20:46:08 INFO DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[5] at mapPartitions at Exchange.scala:86), which has no missing parents
14/10/08 20:46:08 INFO MemoryStore: ensureFreeSpace(11000) called with curMem=432180, maxMem=278302556
14/10/08 20:46:08 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 10.7 KB, free 265.0 MB)
14/10/08 20:46:08 INFO MemoryStore: ensureFreeSpace(5567) called with curMem=443180, maxMem=278302556
14/10/08 20:46:08 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.4 KB, free 265.0 MB)
14/10/08 20:46:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb174:49971 (size: 5.4 KB, free: 265.4 MB)
14/10/08 20:46:08 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
14/10/08 20:46:08 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MapPartitionsRDD[5] at mapPartitions at Exchange.scala:86)
14/10/08 20:46:08 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
14/10/08 20:46:08 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 0, eb174, NODE_LOCAL, 1199 bytes)
14/10/08 20:46:08 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 1, eb176, NODE_LOCAL, 1199 bytes)
14/10/08 20:46:08 INFO ConnectionManager: Accepted connection from [eb176/10.1.69.176:49289]
14/10/08 20:46:08 INFO ConnectionManager: Accepted connection from [eb174/10.1.69.174:33401]
14/10/08 20:46:08 INFO SendingConnection: Initiating connection to [eb176/10.1.69.176:60783]
14/10/08 20:46:08 INFO SendingConnection: Initiating connection to [eb174/10.1.69.174:54967]
14/10/08 20:46:08 INFO SendingConnection: Connected to [eb176/10.1.69.176:60783], 1 messages pending
14/10/08 20:46:08 INFO SendingConnection: Connected to [eb174/10.1.69.174:54967], 1 messages pending
14/10/08 20:46:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb176:60783 (size: 5.4 KB, free: 265.4 MB)
14/10/08 20:46:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb174:54967 (size: 5.4 KB, free: 265.4 MB)
14/10/08 20:46:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb174:54967 (size: 24.6 KB, free: 265.4 MB)
14/10/08 20:46:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb176:60783 (size: 24.6 KB, free: 265.4 MB)
14/10/08 20:46:10 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 1) in 2657 ms on eb176 (1/2)
14/10/08 20:46:10 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 0) in 2675 ms on eb174 (2/2)
14/10/08 20:46:10 INFO DAGScheduler: Stage 1 (mapPartitions at Exchange.scala:86) finished in 2.680 s
14/10/08 20:46:10 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
14/10/08 20:46:10 INFO DAGScheduler: looking for newly runnable stages
14/10/08 20:46:10 INFO DAGScheduler: running: Set()
14/10/08 20:46:10 INFO DAGScheduler: waiting: Set(Stage 0)
14/10/08 20:46:10 INFO DAGScheduler: failed: Set()
14/10/08 20:46:10 INFO DAGScheduler: Missing parents for Stage 0: List()
14/10/08 20:46:10 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[9] at map at HiveContext.scala:360), which is now runnable
14/10/08 20:46:10 INFO MemoryStore: ensureFreeSpace(9752) called with curMem=448747, maxMem=278302556
14/10/08 20:46:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 9.5 KB, free 265.0 MB)
14/10/08 20:46:10 INFO MemoryStore: ensureFreeSpace(4941) called with curMem=458499, maxMem=278302556
14/10/08 20:46:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.8 KB, free 265.0 MB)
14/10/08 20:46:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on eb174:49971 (size: 4.8 KB, free: 265.4 MB)
14/10/08 20:46:10 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
14/10/08 20:46:11 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[9] at map at HiveContext.scala:360)
14/10/08 20:46:11 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
14/10/08 20:46:11 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 2, eb175, PROCESS_LOCAL, 948 bytes)
14/10/08 20:46:11 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@513f39c
14/10/08 20:46:11 INFO StatsReportListener: task runtime:(count: 2, mean: 2666.000000, stdev: 9.000000, max: 2675.000000, min: 2657.000000)
14/10/08 20:46:11 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:11 INFO StatsReportListener:     2.7 s   2.7 s   2.7 s   2.7 s   2.7 s   2.7 s   2.7 s   2.7 s   2.7 s
14/10/08 20:46:11 INFO StatsReportListener: shuffle bytes written:(count: 2, mean: 50.000000, stdev: 0.000000, max: 50.000000, min: 50.000000)
14/10/08 20:46:11 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:11 INFO StatsReportListener:     50.0 B  50.0 B  50.0 B  50.0 B  50.0 B  50.0 B  50.0 B  50.0 B  50.0 B
14/10/08 20:46:11 INFO StatsReportListener: task result size:(count: 2, mean: 1848.000000, stdev: 0.000000, max: 1848.000000, min: 1848.000000)
14/10/08 20:46:11 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:11 INFO StatsReportListener:     1848.0 B        1848.0 B        1848.0 B        1848.0 B        1848.0 B        1848.0 B    1848.0 B        1848.0 B        1848.0 B
14/10/08 20:46:11 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 86.309428, stdev: 0.103820, max: 86.413248, min: 86.205607)
14/10/08 20:46:11 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:11 INFO StatsReportListener:     86 %    86 %    86 %    86 %    86 %    86 %    86 %    86 %    86 %
14/10/08 20:46:11 INFO StatsReportListener: other time pct: (count: 2, mean: 13.690572, stdev: 0.103820, max: 13.794393, min: 13.586752)
14/10/08 20:46:11 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:11 INFO StatsReportListener:     14 %    14 %    14 %    14 %    14 %    14 %    14 %    14 %    14 %
14/10/08 20:46:11 INFO ConnectionManager: Accepted connection from [eb175/10.1.69.175:36187]
14/10/08 20:46:11 INFO SendingConnection: Initiating connection to [eb175/10.1.69.175:35197]
14/10/08 20:46:11 INFO SendingConnection: Connected to [eb175/10.1.69.175:35197], 1 messages pending
14/10/08 20:46:11 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on eb175:35197 (size: 4.8 KB, free: 265.4 MB)
14/10/08 20:46:12 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@eb175:58085
14/10/08 20:46:12 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 140 bytes
14/10/08 20:46:12 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 2) in 1428 ms on eb175 (1/1)
14/10/08 20:46:12 INFO DAGScheduler: Stage 0 (collect at HiveContext.scala:415) finished in 1.432 s
14/10/08 20:46:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
14/10/08 20:46:12 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@6e8030b0
14/10/08 20:46:12 INFO StatsReportListener: task runtime:(count: 1, mean: 1428.000000, stdev: 0.000000, max: 1428.000000, min: 1428.000000)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     1.4 s   1.4 s   1.4 s   1.4 s   1.4 s   1.4 s   1.4 s   1.4 s   1.4 s
14/10/08 20:46:12 INFO StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms
14/10/08 20:46:12 INFO StatsReportListener: remote bytes read:(count: 1, mean: 100.000000, stdev: 0.000000, max: 100.000000, min: 100.000000)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     100.0 B 100.0 B 100.0 B 100.0 B 100.0 B 100.0 B 100.0 B 100.0 B 100.0 B
14/10/08 20:46:12 INFO SparkContext: Job finished: collect at HiveContext.scala:415, took 4.787407158 s
14/10/08 20:46:12 INFO StatsReportListener: task result size:(count: 1, mean: 1072.000000, stdev: 0.000000, max: 1072.000000, min: 1072.000000)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     1072.0 B        1072.0 B        1072.0 B        1072.0 B        1072.0 B        1072.0 B    1072.0 B        1072.0 B        1072.0 B
14/10/08 20:46:12 INFO StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 80.252101, stdev: 0.000000, max: 80.252101, min: 80.252101)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     80 %    80 %    80 %    80 %    80 %    80 %    80 %    80 %    80 %
14/10/08 20:46:12 INFO StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:      0 %     0 %     0 %     0 %     0 %     0 %     0 %     0 %     0 %
14/10/08 20:46:12 INFO StatsReportListener: other time pct: (count: 1, mean: 19.747899, stdev: 0.000000, max: 19.747899, min: 19.747899)
14/10/08 20:46:12 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
14/10/08 20:46:12 INFO StatsReportListener:     20 %    20 %    20 %    20 %    20 %    20 %    20 %    20 %    20 %
5078
Time taken: 7.581 seconds
View Code

注意：

  1. 在启动spark-sql时，如果不指定master，则以local的方式运行，master既可以指定standalone的地址，也可以指定yarn；
  2. 当设定master为yarn时(spark-sql --master yarn)时，可以通过http://$master:8088页面监控到整个job的执行过程；
  3. 如果在$SPARK_HOME/conf/spark-defaults.conf中配置了spark.master spark://eb174:7077，那么在启动spark-sql时不指定master也是运行在standalone集群之上。

6.遇到的问题及解决方案

① 在spark-sql客户端命令行界面运行SQL语句出现无法解析UnknownHostException:ebcloud（这是hadoop的dfs.nameservices）

14/10/08 20:42:44 ERROR CliDriver: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, eb174): java.lang.IllegalArgumentException: java.net.UnknownHostException: ebcloud
  org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
  org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:240)
  org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:144)
  org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:579)
  org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:524)
  org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
  org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
  org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
  org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

原因：Spark无法正确获取HDFS的地址。因此，将hadoop的HDFS配置文件hdfs-site.xml拷贝到$SPARK_HOME/conf目录下。

②

14/10/08 20:26:46 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 
14/10/08 20:26:46 INFO SparkContext: Starting job: collect at HiveContext.scala:415 
14/10/08 20:29:19 WARN RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over eb171/10.1.69.171:8020. Not retrying because failovers (15) exceeded maximum allowed (15) 
java.net.ConnectException: Call From eb174/10.1.69.174 to eb171:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) 
at org.apache.hadoop.ipc.Client.call(Client.java:1414) 
at org.apache.hadoop.ipc.Client.call(Client.java:1363)

原因：hdfs连接失败，原因是hdfs-site.xml未全部同步到slaves的节点上。

7.参考资料

春宇大数据
关注
专栏目录
AutoSAR系列讲解（深入篇）14.1-EB Tresos使用初探
雪云飞星的博客
04-24 2万+
主要讲解了EB Tresos的基础功能
spark中读取hdfs中文件出错
smallboy2011的专栏
07-19 3605
scala> val rdd1 = sc.textFile("hdfs://master:9001/spark/spark02/directory/") 14/07/19 17:09:36 INFO MemoryStore: ensureFreeSpace(138763) called with curMem=0, maxMem=309225062 14/07/19 17:09:36 INFO
hadoop 集群 多 nameservices 多节点 集群搭建 过程---第二版
u010006687的博客
09-02 8176
环境: centos7.4 jkd1.7.0_80 hadoop2.6.4 在 /etc/profile中配置java,hadoop,zookeeper的export和PATH,方便启动调用 结构: cluster1: mini0 mini1 cluster2: mini7 mini8 zkfc : mini0 mini
Spark项目连接Hadoop HA配置文件编写方案
ciqingloveless的博客
06-20 594
val sparkConf = new SparkConf().setAppName("sparkRealTimeServer") .setMaster("yarn").setSparkHome("/app/data1/spark").set("spark.submit.deployMode", "client"). set("spark.testing.memory", ...
SparkSQL读写备份还原后的表报错UnknownHostException: nameservice01
OMars的博客
07-14 668
问题背景 使用的是Ambari 2.7.3大数据平台，原本集群要开启namenode高可用，但是因为某些搞不定的原因重新安装了集群，针对Hive数据做了备份，然后在当前集群环境做了数据恢复。所有的数据通过Hive都可查询。但是有程序执行读写Hive数据的时候报错UnknownHostException: nameservice01，具体如下 Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: namese
spark初体验
热门推荐
张兆坤
03-31 1万+
现在这个美好的时代，作为一个技术人，如果连Spark的大名都不知道，那显然是说不过去的，话说大数据的时候，必提Hadoop、Spark。我跟进Spark也好些日子，这次用最新搭建的Hadoop实验集群用最新的Spark来体验了一把酸甜苦辣!
SparkConf、spark-submit以及spark-defaults.conf
htfenght的博客
01-14 8458
北风网spark学习笔记 SparkConf、spark-submit以及spark-defaults.conf spark-submit脚本会自动加载conf/spark-defaults.conf文件中的配置属性，并传递给我们的spark应用程序 加载默认的配置属性，一大好处就在于，我们不需要在spark-submit脚本中设置所有的属性 比如说，默认属性中有一个spark.master属性...
云存储和云计算的使用初探
03-04
工银瑞信是中国工商银行和瑞士信贷合资成立的一家基金管理的公司，其业务不仅仅是基金层面，也会涉及到社保、养老金等与老百姓生活非常相关的领域。据悉，目前工银瑞信投资于全世界多个目标市场，因此业务也会受到...
MongoDB初探(二)----使用spring-data配置mongodb
03-28
在本篇“MongoDB初探(二)----使用spring-data配置mongodb”中，我们将深入探讨如何利用Spring Data框架来集成和操作MongoDB数据库。Spring Data是Spring生态系统的一个重要组成部分，它提供了与各种数据存储系统...
HDFS Client UnknownHostException: NAMESERVICE
独行的卡卡的博客
01-16 3857
问题现象 HDFS Client抛出如下异常： java.net.UnknownHostException: NAMESERVICE at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeProxi
在配置hbase中所遇到的问题--期待解决
sir_TI的博客
09-12 651
本人按照网上的教程，将core-site.xml和hdfs.xml放到hbase/conf的路径下，仍然是报一下错误： java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.          at org.apache.hadoop.hbas...
Exception while invoking getStats of class ClientNamenodeProtocolTranslatorPB over 故障分析
lepton126的专栏
07-21 9667
在执行 hdfs dfsadmin -report 命令时，出现故障Exception while invoking getStats of class ClientNamenodeProtocolTranslatorPB over ，在查阅log记录时发现 java.net.SocketTimeoutException: Call From clusternode***to clusterno
超全spark性能优化总结
journey_fan的专栏
02-22 1756
Spark是大数据分析的利器，在工作中用到spark的地方也比较多，这篇总结是希望能将自己使用spark的一些调优经验分享出来。 一、常用参数说明 --driver-memory 4g : driver内存大小，一般没有广播变量(broadcast)时，设置4g足够，如果有广播变量，视情况而定，可设置6G，8G，12G等均可 --executor-memory 4g : 每个executor的内存...
hive启动报错---java.net.UnknownHostException: cluster
sunqingok的博客
12-16 5730
hive启动时报错如下： Exception in thread “main” java.lang.IllegalArgumentException: java.net.UnknownHostException: cluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377) a...
5-1、Spark环境搭建
鲍礼彬的CSDN博客 ~_~
03-12 1794
2、Spark环境搭建 2.1、官网下载 Spark官网地址：http://spark.apache.org/     下载后如下：   Linux上安装部署Spark Jdk Scala SSH Hadoop Spark 2.2、安装模式 Local模式(学习、测试之用)   Standalone模式（内置的资源管理和调度框架） Mesos（Apache） Ya
sparksql限制一（多表join问题）
cmoaai42662的博客
03-10 1192
1、sqlContext中3个表不能同时join，需要两两join后再join一次；---经验证，此版本没这个限制（cdh5.3） scala> sqlContext.sql("select a.name,a.age,b...
第57课：SparkSQL案例实战学习笔记
slq1023的博客
03-27 3928
第57课：SparkSQL案例实战学习笔记 本期内容： 1.SparkSQL基础案例实战 2.SparkSQL商业类型的案例 进入Spark官网的sql-programming-guide： http://spark.apache.org/docs/latest/sql-programming-guide.html#getting-started 可以看到 The entry
Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: master
微电子学与固体电子学-俞驰
04-26 5191
$ spark-submit --class WordCountLocal --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 /home/appleyuchi/IdeaProjects/scala-learn/target/scala-learn-1.0-...
Java环境配置与Eclipse使用初探
"Java开发环境的搭建主要包括JDK的下载安装和环境变量的配置，以及集成开发环境Eclipse的基础使用。本章节详细介绍了JDK的下载、安装步骤，包括从Sun官方网站获取不同操作系统版本的JDK，特别是Windows版本的JDK-6u2...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值