luyee2010的专栏_luyee2010_CSDN博客-hadoop之－站在巨人的肩膀上,Hbase入门,Hadoop入门领域博主

原创 idea 报错 java.lang.ExceptionInInitializerError: com.sun.tools.javac.code.TypeTags

主要是jar包依赖冲突，增加dependencyManagement，升级lombok为高版本<dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.16</version> <scope>provided</scope>&lt

2021-01-20 20:54:11 7

注意引入项目是examples下的scala为scala项目，编译与delta官网一致```build/sbt compile```注意在window下运行hadoop相关一堆事，我这里直接通过System.setProperty设置hadoop.home.dirSystem.setProperty("hadoop.home.dir", "E:\\code\\github\\winutils\\hadoop-3.0.0")配置好之后就可以运行调试Quicksta...

2020-11-06 19:59:42 75

原创数据湖DeltaLake导入idea报错DeltaSqlBaseBaseListener is already defined as class DeltaSqlBaseBaseListener

sbt编译完导入idea后报错DeltaSqlBaseBaseListener is already defined as class DeltaSqlBaseBaseListener这个应该是重复文件吧，删除了target\scala-2.12\src_managed\main 只保留了target\scala-2.12\src_managed\main\antlr4就正常了...

2020-09-18 16:47:20 67 1

原创记一次produces引发的404

发现有个服务快应用过来的请求突然都404(no mapping found)了，但是自己用postman调或者浏览器都正常开始怀疑是url中有乱码，加了个Filter，这个是正常的，参数什么的也正常，按理是一样的最后实在没招了，想到可能是Header的问题，如果打印出所有的Header，对比快应用与postman的请求请求头Accept跟请求消息体Content-type居然都是application/x-protobuf,而自己的@RequestMapping又设置了prod..

2020-09-02 11:34:05 65

原创数据湖DeltaLake编译scalastyle检查错误Failing because of negative scalastyle result

delta编译报错很明显是scalastyle检查的问题，但是sbt不像maven有-skip可以不执行检查，用scalastyle:on /scalastyle:off 太麻烦，只好把scalastyle-config.xml文件的内容全部删除$ build/sbt compileUsing E:\dev_tools\jdk1.8.0_202\jdk as default JAVA_HOME.Note, this will be overridden by -java-home if it i

2020-07-02 17:23:33 134

原创一次日志引发的事故

排查问题加了不少log,直接开启info模式以及console ,导致cpu

2020-06-22 17:12:37 89

原创多个泛型CompletableFuture获取结果

获取feed列表通常里边需要依据Id查询用户信息import java.util.HashMap;import java.util.Map;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutionException;import java.util.concurrent.atomic.AtomicLong;public class BatchCompletableFutureTes

2020-05-26 16:54:02 233

原创 Elasticsearch "reason": "date_time_parse_exception: Text '1588707487000' could not be parsed at inde

mapping{ "properties": { "createDate": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, "id": { "type": "long" }

2020-05-11 21:03:57 1405

原创 ArrayBlockingQueue 自己坑了CPU搞出100%

写监控给自己挖了个坑哈哈，入队列肯定是不阻塞的，不入成功出队列写了个while poll

2020-05-07 10:38:50 222

原创 elasticsearch long类型出现精度丢失问题

建立schema添加doc17位postman结果es 后台查询结果js, 精度会丢失实际19位也可以？20位好像不行

2020-04-29 10:53:24 2303

原创 fidder抓取firefox请求

1.选着tools--->options--->Connections，点击Copy Brower Proxy Configuration URL2.还是在Firefox浏览器:选项中->常规->网络代理->设置->自动代理配置的Url(PAC),ctrl+V粘贴刚刚复制的fiddler代理地址...

2020-03-25 15:00:03 80

原创 301 Moved Permanently > websocket: bad handshake

1.SSL 配置 FULL2.Authenticated Origin Pulls，开启ON

2019-09-20 15:39:48 1917

原创 mybatis 返回主键

插入返回主键 <insert id="insertSelective" useGeneratedKeys="true" keyProperty="id" keyColumn="Fid" parameterType="com.dao.domain.calculationv2.ComputeDebug" >1，useGeneratedKeys="true" 2，keyP...

2019-09-18 16:01:46 70

原创 mybatis-generator-maven-plugin不生成WithBLOB结尾命名的domain

pom <build> <finalName>MybatisGen</finalName> <plugins> <plugin> <groupId>org.mybatis.generator</groupId> ...

2019-09-12 09:51:21 509

原创 flink1.9 源码编译

之前编译过blink,感觉很顺利，这次1.9 release 编译好几次都没成功。。。git clone https://github.com/apache/flink.gitcd flinkgit checkout release-1.9mvn clean install -DskipTests -Drat.skip=true一开始习惯的使用,编译好几次都失败了，-D...

2019-08-31 21:00:20 539

原创 java.lang.UnsupportedOperationException List.clear

用Set实现List去重的代码 public static <S> List<S> removeDedupElement(List<S> list) { TreeSet<S> set = new TreeSet<S>(list); list.clear(); list.addAll(set); return list...

2019-08-21 19:57:46 543

原创 ProcessBuilder java.io.IOException: Cannot run program error=2, No such file or directory

通过javaProcessBuilder执行如下命令， String killCommand = String.format("ps -ef|grep java|grep 'kafka_sink_partitition_test_2019_0730/conf/'|awk '{print $2}'|xargs kill -9",taskName); try {...

2019-08-13 16:00:12 4537

原创 http json参数乱码

通过传递json参数，生成需要的spark任务模板，生成的Properties文件乱码。主要是http传参与服务端接收参数的问题解决方案客户端 httpPost.setEntity(new StringEntity(GsonUtils.toJsonString(map).toString(),"utf-8"));服务端 HttpServletRequest.setCh...

2019-06-05 18:23:10 292

原创 fatal: You are not currently on a branch. To push the history leading to the current (detached HEAD)

git push$ git pushfatal: You are not currently on a branch.To push the history leading to the current (detached HEAD)state now, usegit status$ git statusHEAD detached from da74766nothing...

2019-06-05 10:53:43 6118

原创 spark idea java.lang.NoClassDefFoundError

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:73) at org.apache.spark.Spar...

2019-05-29 15:57:25 1561

原创 Warning:scala: skipping Scala files without a Scala SDK in module(s)

scala 与java 混合的maven项目模块右键--》Add Framework Supporthttps://intellij-support.jetbrains.com/hc/en-us/community/posts/206411529-Add-Scala-SDK-to-module

2019-05-29 10:39:03 1090

原创 eclipse 不能识别scala代码红叉

eclipse Version: Oxygen.3a Release (4.7.3a)安装了scalaIDE插件4.7.0 Release 与Oxygen兼容安装完后导入包含java,scala的maven项目，scala代码不能识别，都是红叉解决方案：右键单击工程--》Configure--》add scala nature;参考链接：https://stackover...

2019-05-28 15:18:11 602

原创 spark sql functions

org.apache.spark.sql.functions是一个Object，提供了很多函数。https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/functions.html

2019-03-22 10:26:33 193

原创 printStackTrace&getMessage

package com.ly.test.fql;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public class TestPrintStackTrace { private static final Logger LOG = LoggerFactory.getLogger(TestPrintStackTrac...

2019-02-18 21:19:10 147

原创 Error:(52, 75) not found: value TCLIService public abstract class ThriftCLIService extends AbstractS

maven编译完spark2.2.0，然后导入maven项目File-->new->Import project from external model-->maven方便hive-thriftserver的代码，记得select profiles 页勾选hive-thriftserver导入maven项目到idea 出现错误Error:(52, 75) not...

2019-02-13 13:27:03 230

原创 Error occurred during initialization of VM Could not reserve enough space for object heap

maven 编译报错D:\code\github\spark> set MAVEN_OPTS= -Xms2000m -Xmx2000mD:\code\github\spark> mvn clean install package -DskipTests -Dcheckstyle.skip=trueError occurred during initializatio...

2019-02-13 10:53:03 660

原创 EINVRES Request to https://bower.herokuapp.com/packages/ failed with 502

解决方案：在.bowerrc文件添加：{ "registry": "https://registry.bower.io"}

2019-01-25 17:41:50 889

原创 “Received fatal alert: protocol_version” or “peer not authenticated”

maven 编译的时候碰到解决方案：export MAVEN_OPTS=-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2也可以通过在mvn前面添加 mvn -Dhttps.protocols=TLSv1.2 install

2019-01-24 14:37:03 1119

原创 python 日期时间

>>> print datetime.date.today().strftime('%Y%m%d')20180302>>> print (datetime.date.today()-datetime.timedelta(days=30)).strftime('%Y%m%d') 20180131

2018-03-02 11:27:56 527

原创 java.lang.AbstractMethodError: com.alibaba.otter.canal.common.zookeeper.ZooKeeperx.writeDat aReturnS

错误java.lang.AbstractMethodErrorjava.lang.AbstractMethodError: com.alibaba.otter.canal.common.zookeeper.ZooKeeperx.writeDataReturnStat(Ljava/lang/String;[BI)Lorg/apache/zookeeper/data/Stat;

2018-01-16 17:49:39 859

原创 mac 下du --max-depth

du -hd1

2018-01-15 11:35:02 7709

原创钉钉报警

自定义机器人更多-->机器人管理-->自定义机器人-->添加-->复制Webhook地址 https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxx简单点的，直接给json参数就好了curl https://oapi.dingtalk.com/robot/send?access_token=xxx

2017-10-23 15:38:49 3213

原创 KUDU-Got out-of-order key column: name: "ID" type: STRING is_key: true is_nullable: false

Got out-of-order key column: name: "ID" type: STRING is_key: true is_nullable: falseThe columns which make up the primary key must be listed first in the schema.https://community

2017-09-25 16:21:26 3678

原创避免mybatis SUM求和返回null

SELEC SUM(ORDER_MONEY ) FROM gm_order WHERE ADD_UID = #{userCode}如上写法如果没有结果的话就会返回null,其实我们希望返回的是0.00这种情况SELECT COALESCE(SUM(ORDER_MONEY),0) FROM gm_order WHERE ADD_UID = #{userCode}

2017-09-01 03:57:03 13358

原创 Spring+Mybatis 多数据源配置Invalid bound statement (not found)

单独使用某一个数据源配置都是ok的，同时两个以上就会报错。几个数据源的mapper.xml，dao放在同一个目录下了，分开之后分别放在不同的目录就ok了

2017-08-29 17:16:47 5934 1

原创 Confluent Platform部署

下载链接地址：http://packages.confluent.io/archive/3.3/confluent-oss-3.3.0-2.11.tar.gz点击打开链接Confluent Platform Quickstart

2017-08-29 03:05:47 4046

原创 mybatis Invalid bound statement (not found)

org.apache.ibatis.binding.BindingException: Invalid bound statement (not found)将select 语句加上 ']]>"，居然好了

2017-08-29 03:02:19 553

原创 kudu1.4.0 安装配置

包下载地址archive.cloudera.com/kudu

2017-08-25 01:45:26 1075

原创 Percona-XtraBackup安装配置

下载安装# wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.8/binary/redhat/6/x86_64/percona-xtrabackup-24-2.4.8-1.el6.x86_64.rpm# rpm -ivh percona-xtrabackup-24-2.4.8-1.el6.x86_

2017-08-18 11:16:23 911