2018年10月_hxhh

原创 TaskSchedulerImpl: Initial job has not accepted any resources (Spark报错)

18/10/20 10:08:16 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources如图所示：执行脚本的时候报错报错原因：...

2018-10-20 10:54:38 1096

原创 org.apache.spark.SparkException: Exception thrown in awaitResult （Spark报错）

WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master node1:7077org.apache.spark.SparkException: Exception thrown in awaitResult at org.apache.spark.rpc.RpcTimeout$$anonfun$1.a...

2018-10-20 09:18:49 27322

原创 Name node is in safe mode. 报错解决

错误如下： org.apache.hadoop.dfs.SafeModeException: Cannot delete/user/local/hadoop-2.7.4. Name node is in safe mode报错原因：Name node在safe mode，安全模式下不可以实现增删改操作。解决方案：到Hadoop的安装目录下：bin/hadoop dfsadmin -...

2018-10-18 21:09:15 2490

原创 java.lang.IllegalStateException: File name has been re-used with different files. (flume报错）

原因：重复使用了同一个数据文件解决方案1：更改flume的配置文件conf的目录a1.sources.r1.spoolDir = /usr/local/data/flumedata#push modea1.sources = r1a1.sinks = k1a1.channels = c1#sourcea1.sources.r1.channels = c1a1.sources....

2018-10-18 11:23:06 2690

原创 Spark Streaming整合flume(Poll方式和Push方式)

flume作为日志实时采集的框架，可以与SparkStreaming实时处理框架进行对接，flume实时产生数据，sparkStreaming做实时处理。Spark Streaming对接FlumeNG有两种方式，一种是FlumeNG将消息Push推给Spark Streaming，还有一种是Spark Streaming从flume 中Poll拉取数据。6.1 Poll方式（1）安装flu...

2018-10-18 10:58:13 935

原创 No suitable driver found for jdbc:mysql://192.168.25.121:3306/spark

INFO TaskSetManager: Lost task 1.0 in stage 4.0 (TID 7) on executor localhost: java.sql.SQLException (No suitable driver found for jdbc:mysql://192.168.25.121:3306/spark) [duplicate 1]18/10/16 18:13:...

2018-10-16 18:18:21 2150 1

原创 org.apache.spark.examples.SparkPi

Warning: Local jar /usr/local/spark/ does not exist, skipping.java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPiat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at jav...

2018-10-15 12:38:48 9368 10

原创搭建Spark高可用集群

通过zookeeper搭建高可用spark集群1、需要搭建一个zk集群2、配置文件修改（spark-env.sh）注释掉export SPARK_MASTER_HOST=hdp-node-01添加 SPARK_DAEMON_JAVA_OPTSexport SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -D...

2018-10-15 11:10:30 232

原创 Spark集群搭建

1.1下载spark安装包下载地址spark官网：http://spark.apache.org/downloads.html这里我们使用spark-2.0.2-bin-hadoop2.7版本.3.2规划安装目录/opt/bigdata1.3解压安装包tar-zxvf spark-2.0.2-bin-hadoop2.7.tgz1.4重命名目录mv spark-2.0.2-bin...

2018-10-15 10:20:43 124

原创 Reported time is too far out of sync with master.（HBase报错）

错误如图所示：查看日志文件发现：Reported time is too far out of sync with master.解决方案：方案一：推荐使用（使用alibaba的时间同步）ntpdate ntp6.aliyun.com方案二：常用的手动进行时间的同步（一般不使用） date -s "2017-03-03 03:03:03"方案三：（网络同步）yum i...

2018-10-14 11:14:12 1343

原创 Scala在idea打印的问题

错误示范： val scores = Array(77,88) val names = Array("hxh","hh","xx") names.zip(scores) println(names)正确示范：（需要定义一个新的变量再打印，否则打印额外都是原来names）val scores = Array(77,88) val names = A

2018-10-11 09:30:59 1313

原创 Cannot start compilation: the output path is not specified for module "scalaDemo". 完美解决

Cannot start compilation: the output path is not specified for module "scalaDemo". Specify the output path in the Project Structure dialog解决方案：

2018-10-10 15:35:36 9055 1

原创 java.net.UnknownHostException: master (HBase报错)

错误如下：Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: master at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:373) at org.apache.hadoop.hd...

2018-10-09 19:56:27 7149 1

原创 HBase的Shell操作

1 连接集群hbase shell2 创建表create 'user','base_info'3 插入数据put 'user','rowkey_10','base_info:username','张三'put 'user','rowkey_10','base_info:birthday','2014-07-10'put 'user','rowkey_10','base_info:...

2018-10-09 15:52:05 152

原创 HBase集群部署

1.1操作步骤说明： 1. 下载安装包 2. 修改配置文件 3. regionservers 4. hbase-site.xml 5. hbase-env.sh 6. 拷贝hadoop配置文件 7. 分发配置文件 8. 启动集群2.1 下载安装包wget http://mirrors.hust.edu.cn/apache/hbase/1.3.1/hbase-1.3.1-bi...

2018-10-09 15:11:54 274

原创 org.apache.hadoop.mapred.FileAlreadyExistsException 解决方案

Exception in thread “main”org.apache.hadoop.mapred.FileAlreadyExistsException: Outputdirectory temp/preparePreferenceMatrix/itemIDIndex already exists然而这个目录通过Hadoop的web界面并查询不到！！解决方法：（直接remove掉在...

2018-10-08 22:12:34 5319

原创 JedisPoolConfig的报错 setMaxIdle() setMaxTotal() setMaxWaitMillis() setTestOnBorrow() setTestOnReturn()

JedisPoolConfig的报错JedisPoolConfig config = new JedisPoolConfig();config.setMaxIdle(5);config.setMaxTotal(1000 * 100);config.setMaxWaitMillis(30);config.setTestOnBorrow(true);config.setTestOnRetu...

2018-10-04 14:52:23 6638 1

hxh的博客