huangdsky-CSDN博客

原创 superset 0.28.1 配置druid,保存chart后，chart无法进去报错

报错如下：/superset/connectors/druid/models.py" ,line 532, in linkTypeError: format() got multiple values for argment 'self'结合superset 0.30 代码，改532行为：return Markup(f'<a href="{self.url}"&...

2019-06-18 18:34:53 617

原创修改linux打开文件最大数与最大线程数

修改 /etc/security/limits.d/90-nproc.conf 将1024变为65525

2016-08-01 14:25:29 519

原创发现个学spark的地方，不错

http://www.hubwiz.com/

2016-07-29 09:05:42 427

原创 flume练手

一、牛刀小试（一）1、/home/ut/hadoop/apache-flume-1.6.0-bin/conf下创建vim avro.conf a1.sources = r1a1.sinks = k1a1.channels = c1 # Describe/configure the sourcea1.sources.r1.type = avroa1.sou

2016-07-25 14:55:38 300

原创 hbase 12.1 regionserver 宕机报错记录——1

"Append_min" : 0, "Append_max" : 0, "Append_mean" : 0.0, "Append_median" : 0.0, "Append_75th_percentile" : 0.0, "Append_90th_percentile" : 0.0, "Append_95th_percentile" :

2016-07-10 10:09:19 6807

原创 hadoop 错误1 journalnode与namenode超时

2016-06-04 18:27:49,728 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 7002 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248

2016-06-06 11:20:20 3124 1

原创 hbase 预分区官网例子

public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits)throws IOException { try { admin.createTable( table, splits ); return true; } catch (TableExist

2016-05-23 18:01:03 645

原创 hbase1.2.1 编译hadoop-2.7.2

根目录下MAVEN_OPTS="-Xmx2g" mvn site install assembly:single -Dhadoop-two.version=2.7.2 -DskipTests

2016-04-29 13:21:30 523

原创 spark rdd 去括号

val str2 = sqlContext.sql("select a.userID,b.column from log a,vtable b where a.mediaCode=b.mediacode ").map( t=> (t(0),t(1)).toString().replaceAll("\$","").replaceAll("\$",""))

2016-04-28 14:46:05 2401

原创记录一个map-site.xml -Xmx 1000m 写成了 1000报错问题

http://stackoverflow.com/questions/30231508/yarn-mapreduce-job-dies-with-strange-message

2016-03-24 17:32:59 735

原创 hdp安装地址

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/index.html

2016-03-21 16:45:25 387

原创很顺利地安装了spark 做一下记录

环境变量配置（hadoop-2.6.0安装请参考网络上的一些安装）：#jdkJAVA_HOME=/usr/lib/jdk1.8.0_51PATH=$JAVA_HOME/bin:$PATHCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport JAVA_HOMEexport PATHexport C

2015-11-11 15:31:04 467

原创由于加了mahout*.job.jar进hadoop2.X/share/hadoop/common/lib重启datanode闪退报错

org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/hcg/hadoop/hadoop-2.6.0/mydata/data. Reported: -56. Expecting = -55去掉mahout*.job.j

2015-11-04 14:56:56 686

原创 hadoop2.6.0下打包mahout调用数据模型程序运行时报错

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/common/AbstractJobat java.lang.ClassLoader.defineClass1(Native Method)at java.lang.ClassLoader.defineClass(ClassLoader.

2015-11-03 16:52:01 991

原创运行mahout 将学习集的序列化转化为向量报错Error: Java heap space(mapreduce运行内存调优)

mahout seq2sparse -i email-bydate-train-seq -o email-bydate-train-vector -lnorm -nv -wt tfidf mahout seq2sparse -i email-bydate-train-seq -o email-bydate-train-vector -lnorm -nv -wt tfidfMAHOUT_LO

2015-10-23 16:03:06 1120

原创 mahout hadoop2.6.0 编译失败

[DEBUG] Detected Maven Version: 3.3.3 is allowed in the range [3.3.3,).[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message:Detected JDK Version: 1.8.0-51 is

2015-10-16 15:55:45 1478

原创两个resourcemanager都处于standby，zkfc无法切换resourcemanager

该问题是对yarn-site.xml文件里的内存分配调整造成的，当强制变换yarn-resourcemanager为active报错yarn rmadmin -transitionToActive --forcemanual rm1You have specified the forcemanual flag. This flag is dangerous, as it c

2015-10-07 10:13:18 5173

原创 centos 7配置静态IP 后 service network restart 成功 IP仍然保持原来值不变

systemctl -f start NetworkManager

2015-09-17 13:17:50 6173

原创 nutch2.3 hadoop2.6.0 hbase0.98.8 分布式爬虫NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfigurati

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfigurationat org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:114)at org.apache.gora.sto

2015-09-14 17:43:45 1384

原创 hdfs namenode -format 报错

21:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)15/08/28 08:52:02 INFO ipc.Client: Retrying connect to server: hcg05/

2015-09-02 12:35:25 5166

原创 hadoop-2.6.0+zookeper-3.4.6之HA模式两个namenode不同状态active和standby的切换

经过多次的试验控制active和standby的切换的主要线程是 DFSZKFailoverController1.原来状态namenode1：active ;namenode2:standby 2.当某种原因致使namenode1失效时，重启namenode1后状态变为：namenode2：active ;namenode1:standby3.如果要恢复原有状态namenode1：a

2015-09-02 11:18:23 2099

原创 hbase0.94.14+hadoop2.6.0

一Hbase的HMaster闪退报错日志：2015-08-22 10:19:45,537 FATAL [master-hcg01,60000,1440209981755] master.HMaster: Unhandled exception. Starting shutdown.java.lang.UnsupportedOperationException: Not implemente

2015-08-22 12:12:14 618

原创 zookeeper-3.4.6格式化问题 hdfs zkfc -formatZk报错

感谢朋友的提醒：错误在配置hdfs-site.xml文件

2015-08-20 16:57:23 18124 5

原创 centos 7 第一次启动hadoop生态之zookeeper-3.4.6

问题：启动./zkServer.sh status 报错Error contacting service. It is probably not running.我发现的情况：1、将各个设置好的节点都启动完2、systemctl disable firewalld.service #禁用3、配置错误

2015-08-20 13:32:37 577

qq_23280769的专栏