- 博客(71)
- 资源 (27)
- 收藏
- 关注
原创 hive1.2以前版本的where条件NullPointerException
1、异常背景:hive版本1.1.0,表是orc格式,使用条件where name in ('支付金额','订单量','客单价','毛利率','全链路达成率','猫超重点商品在架率','基准价毛利率','商品缺货率')2、日志如下:Diagnostic Messages for this Task:Error: java.lang.RuntimeException: org.apache.had...
2018-03-20 17:54:26 588
原创 flume sink hdfs异常
1、异常消息如下:016-08-26 14:19:17,704 (hdfs-sink1-call-runner-2) [ERROR - org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:267)] Error while trying to hflushOrSync!2016-
2016-08-26 15:47:16 9215 1
原创 sqoop导数据到hive失败
1、sqoop异常现象:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:There is no database named dw)2、hive.log: 2016-08-23 16:1
2016-08-23 16:59:48 9900
原创 datanode Bad connect ack with firstBadLink
1、每次启动job很慢并有异常信息:ERROR - java.io.IOException: Bad connect ack with firstBadLink as 10.21.232.114:5001023-08-2016 14:13:21 CST import_ucord01_order_discount ERROR - at org.apache.hadoop.hdfs.DF
2016-08-23 16:58:10 1579
原创 mapreduce job任务非常慢
1、application日志2016-08-11 14:48:15,174 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all scheduled reduces:02016-08-11 14:48:15,174 INFO [
2016-08-11 22:23:08 7974
原创 hive和hbase集成异常
一、偶尔出现两个异常Error: java.lang.IllegalArgumentException: Illegal character code:-1, at 0. User-space table qualifiers can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: � at org.apach
2016-07-19 16:51:25 7215 1
原创 reids定时异常
1、最近业务总是报redis异常,而且总是凌晨1点左右16/07/18 01:02:30 ERROR ShardedDaoImpl: sharded redis lrange:redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection timed ou
2016-07-18 09:41:52 451
原创 mapreduce job一直卡住
16/07/15 17:34:41 INFO input.FileInputFormat: Total input paths to process : 116/07/15 17:34:42 INFO mapreduce.JobSubmitter: number of splits:116/07/15 17:34:42 INFO mapreduce.JobSubmitter: Submitti
2016-07-15 17:41:34 4235 4
原创 StandbyException
16/07/15 16:16:48 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hdfs01.beta1.fn/10.202.249.230:9000 after 2 fail over attempt
2016-07-15 16:45:44 630
原创 hadoop-2.5.0-cdh5.3.0 HA在线升级
本文升级到hadoop2.61、hadoop升级前准备,namenode元数据和配置备份等信息参考上篇文章:http://blog.csdn.net/linux_ja/article/details/519084662、备份信息:hdfs dfsadmin -rollingUpgrade preparehdfs dfsadmin -rollingUpgrade quer
2016-07-15 15:16:38 2059
原创 hdfs dfsadmin -rollingUpgrade解读
hdfs dfsadmin -rollingUpgrade prepare 源码中是CheckpointSignature rollEditLog() throws IOException { getEditLog().rollEditLog(); // Record this log segment ID in all of the storage directori
2016-07-14 17:57:17 1180
原创 Hadoop 2.3.0-cdh5.0.2升级Hadoop 2.5.0-cdh5.3.1
一、准备升级包for line in `cat /home/hadoop/platform/hadoop.list|awk '{print $1}'`; do echo $line;ssh $line "/bin/mkdir /home/hadoop/platform";donefor line in `cat /home/hadoop/platform/hadoop.list|a
2016-07-14 13:41:08 1263
原创 flume kafka-sink high cpu
flume sink到kfka时候,导致cpu过高,以下是分析原因:一、flume的kafka的sink cpu太高分析:1、获取flume的进程id[root@datanode conf]$ toptop - 10:17:58 up 14 days, 18:09, 2 users, load average: 1.37, 1.11, 0.65Tasks: 436 total,
2015-08-04 14:54:15 2953 2
原创 org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out
storm诡异问题查找:一个经常容易报的异常:Exception in thread "main" org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out at org.apache.thrift7.transport.TSocket.open(TS
2015-07-16 17:02:05 6582
原创 hive on tez遇到问题
一、环境hive1.3.1,tez0.5.0二、运行异常如下:return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. org.apache.tez.mapreduce.hadoop.MRHelpers.getBaseMRConfiguration(Lorg/apache/hadoop/conf/Configurat
2015-07-14 14:59:39 3214
原创 hive on tez集成
一、1.下载apache-tez-0.5.4-src.tar.gz2.下载apache-maven-3.2.5-bin.tar3.下载protobuf-2.5.0.tar.gz二、安装1.安装mvn,省略2.安装protobuf: a.tar -zxf protobuf-2.5.0.tar.gz命令解压后得到是protobuf-2.5.0的源码, b.
2015-07-14 11:14:21 1347
原创 jmx监控spark executor配置
jmx监控spark比storm稍微有点繁琐:首先在spark-defaults.conf中添加 ,但是8711端口不能重复,也就是说不能在一个节点上启动两个executor,或者端口冲突,没有storm友好 spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dc
2015-06-16 16:42:00 5027
原创 storm work配置JMX监控
配置JMX监控,可以查看GC和线程等等信息,方便debug,以下是在sorm.yaml文件中配置参数配置,%ID%取的时每个work的进程号,因为一个节点上可以有多个work,所以防止端口号重复导致启动失败,所以用动态代替worker.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSi
2015-06-16 16:04:44 2308
原创 FairSync与NonFairSync比较
通过源码比较二者区别,: static final class FairSync extends Sync { private static final long serialVersionUID = -3000897897090466540L; final void lock() { acquire(1);//直接调用tryAcqui
2015-06-16 11:18:24 2500
原创 sparkSQL 集成hive异常问题解决
1、报:Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient是由于元数据默认到derby中找,所以提供mysql的解决:在spar
2015-06-15 17:21:19 3631
原创 分析storm各个线程堵塞原因
1、Thread-4-kafkaSpout-EventThread线程处于长时间等待状态,是正常行为,该线程的作用: 一开始就进入无限循环,从队列waitingEvents中获取事件,如果没有就阻塞等待2、Thread-4-kafkaSpout-SendThread,该线程的作用: 是个IO线程,用于发起和接受请求,并处理相应,同时产生心跳信息 3、Threa
2015-06-11 22:01:53 3300
原创 netty学习之二:ByteBuf解读
ByteBuf原理跟之前的ByteBuffer原理类似,只是扩展和弥补了ByteBuffer的不足,设计原理:1、初始化writerIndex,readerIndex为02、写入N个字节后:3、读入M4、discardReadBytes之后:5、discardReadBytes前后操作:源码: public ByteBuf discardRead
2015-06-07 23:51:11 2057
原创 netty学习之一:java.nio.ByteBuffer
public class App { public static void main(String[] args) { ByteBuffer buffer = ByteBuffer.allocate(88); String value = "权威指南"; buffer.put(value.getBytes()); System.out.println(buffer.capacit
2015-06-07 22:29:21 1236
原创 java多线程总结
(1)执行完同步代码块,就会释放锁(2)在执行同步代码块的过程中,遇到异常而导致线程终止,锁也会被释放(3)在执行同步代码块的过程中,执行了锁所属对象的wait()方法,这个线程会释放锁,进 入对象的等待池除了以上情况以外,只要持有锁的线程还没有执行完同步代码块,就不会释放锁。在下面情况下,线程是不会释放锁的:(1)执行同步代码块的过程中,执行了Threa
2015-06-05 08:13:46 519
原创 sbt.ResolveException: download failed: net.sf.json-lib#json-lib;2.4!json-lib.jar
用sbt构建json-lib失败,以下是配置方法:libraryDependencies += "net.sf.json-lib" % "json-lib" % "2.4" from "http://repo1.maven.org/maven2/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar"
2015-04-10 10:29:57 2007
转载 KafkaSpout的核心逻辑都是由PartitionManager来实现的。
KafkaSpout的核心逻辑都是由PartitionManager来实现的。但是这个类实现时候需要考虑的东西有些多,0.92至0.93,至当前(2015.3.14)的master一直在变化。在这里,先分析一下最近的发布版0.93里的逻辑。也提出一些问题,希望以后Apache Storm会把这个类实现地更完美一些。PartitionManager的主要功能Partition
2015-04-09 15:02:26 649
转载 解析operators
package test.util.ip;import java.io.File;import java.io.FileInputStream;import java.io.IOException;import java.nio.ByteBuffer;import java.nio.ByteOrder;import java.util.Arrays;import j
2015-03-20 19:58:02 522
转载 net
import os,socket,timeimport threadingdef check_port(host,port): s = None for res in socket.getaddrinfo(host, port, socket.AF_UNSPEC,socket.SOCK_STREAM): af, socktype, proto, canonnam
2015-03-20 19:51:49 428
原创 storm spout,bolt超时处理和fail处理
如果toplogy在debug模式下,如果发现__tick打印,则是ack超时导致的,如果打印__ack_fail打印则是消息fail导致的分别对应的变量backtype.storm.SYSTEM_TICK_STREAM_ID = "__tick";(def ACKER-FAIL-STREAM-ID "__ack_fail")
2015-02-08 10:50:17 2551
原创 hbase0.98分布式部署
两个master,两个regionserverhbaes-env.sh:# The directory where pid files are stored. /tmp by default.export HBASE_PID_DIR=/home/master/platform/hbase-0.98.1-cdh5.1.0/pids# Seconds to sleep
2015-02-04 19:33:23 720
原创 hive导数据到hbase
hbase0.98,hive1.2一,copy hbase/lib/hbase-client-0.98.1-cdh5.1.0.jarhbase-server-0.98.1-cdh5.1.0.jarhbase-common-0.98.1-cdh5.1.0.jarhbase-protocol-0.98.1-cdh5.1.0.jarhtrace-core-2.04.jar
2015-02-04 19:06:44 1147
原创 auto.offset.reset解释
auto.offset.reset largest What to do when there is no initial offset in ZooKeeper or if an offset is out of range:* smallest : automatically reset the offset to the smallest offset* largest : aut
2015-01-27 20:36:17 5190 1
原创 简单配置mysql主从复制
1,搭建slave节点,简单配置my.cnf文件如下:server_id = 1#要唯一,不能和其他mysql一样datadir=/usr/local/mysql/datacharacter-set-server=utf8log-bin=mysql-binbinlog-format=ROWbinlog_checksum = NONEreplicate_do_db = e
2015-01-15 11:49:34 531
原创 用mysql5.6.21源码安装mysql出现问题解决
安装地址:https://www.cnhzz.com/mysql-5-6-21_install/登陆mysql时候出现:Segmentation fault(dumped)是因为缺少mysql-libs-5.1.61-4.el6.x86_64然后在执行yum install mysql-devel这个命令就行了
2015-01-13 10:29:05 968
原创 解析mysql binlog 报错:ErrorPacket[packetMarker=255,errorCode=1236,slash=#,sqlState=HY000,errorMessage=bo
在用java的openpaser解析mysql binlog报错如下异常:ErrorPacket[packetMarker=255,errorCode=1236,slash=#,sqlState=HY000,errorMessage=bogus data in log event]解决办法:因为读取mysql master的binlog的时候,setBinlogPosition
2015-01-12 11:35:27 3349
原创 [WARN] Network error when fetching messages:storm-kafka-0.8导致kafka读取数据丢失
2015-01-09T16:17:10.090+0800 s.k.KafkaUtils [WARN] Network error when fetching messages:java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_51]
2015-01-09 17:44:18 2764
原创 storm-kafka spout获取数据的异常解决办法
20391 [Thread-8-kfkaSpout] WARN storm.kafka.KafkaUtils - Got fetch request with offset out of range: [140885176]; retrying with default start offset time from configuration. configured start offset t
2015-01-09 13:50:51 6784 1
原创 mysql replace语句使用
今天发现一个奇怪的现象,表中主键是递增的,突然发现好多主键id数据不连续,通过binlog日志查看,确实有该数据的id,究其原因原来是因为使用了replace语句,导致递增主键id也被修改了,所以出现断断续续的现象
2015-01-06 15:57:41 543
原创 storm NotSerializableException: java.io.BufferedWriter解决
org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Thread Thread[main,5,main] diedjava.lang.RuntimeException: java.io.NotSerializableException: java.io.BufferedWriter at backtype.storm.seriali
2014-12-20 16:10:37 1756
原创 java.io.IOException: No FileSystem for scheme: hdfs异常解决
Configuration conf = new Configuration();conf.set("fs.hdfs.impl",org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());fs = FileSystem.get(new URI(HDFS_PATH), conf);若出现java.io.IOExc
2014-12-15 17:31:51 13938 4
hadoop的mapreduce把oracle/mysq导入到hbase和hdfs中的程序
2014-02-28
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人