merrily01
Talk is cheap 、show me the code .
展开
-
IDEA中Spark源码接口或方法少实现类
1. 现象:举例说明如:SparkApplication # start方法,被多个类重写,但有时候点左侧overrides看实现状况时,可能会缺少某几个类的实现调用;2. 原因:比如缺少yarn或kubernetes具体类的实现,原因一般是因为Spark项目中module的引用出了问题;我的问题是因为总切换2.4和3.0版本,modules中各个模块同时引入了2.11和2.12版本,module引用冲突导致的;3. 解决办法:File -> Project..原创 2020-12-16 10:48:32 · 195 阅读 · 2 评论 -
Error: Could not find or load main class XXX 或 Main class cannot resolve XXX
非maven项目,简简单单的Scala Object main函数调方法,想run或debug起来,结果idea报错Error: Could not find or load main class XXX不是啥sdk或 *..iml的问题,需要确认的是,是否将要调试的class类所在的根目录设置为sources root具体:参考:https://stackoverflow.com/questions/21884217/cant-choose-class-as-main-class-i.原创 2020-10-09 15:28:13 · 1090 阅读 · 0 评论 -
spark on k8s: master pod kube-proxy-mkbp7 Evicte -- The node was low on resource: ephemeral-storag
spark on k8s 集群发现kube-proxy-mkbp7异常,总是Evicted状态,其他node节点上pod正常。1. 通过kubectl describe pod XXXXX -n kube-system 查看pod信息,发现报错日志如下:The node was low on resource: ephemeral-storage. Container kube-pr...原创 2019-11-18 17:10:40 · 1294 阅读 · 2 评论 -
ERROR: unsatisfiable constraints:
背景:测试环境,基于spark-2.4.4代码,打镜像,供spark on k8s使用。问题:执行打包镜像命令,报错如下:docker build -t spark-2.4.4-merrily01-spark -f kubernetes/dockerfiles/spark/Dockerfile . ---> Running in 43e58e4f3a6f+ ...原创 2019-11-15 12:26:03 · 12720 阅读 · 0 评论 -
Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile
环境:spark:2.4.4maven:apache-maven-3.6.2java: 1.8背景:修改了spark源码,想基于此用spark的make-distribution.sh脚本重新打可部署的tar包,命令如下:./dev/make-distribution.sh --name maruilei-spark --tgz -Phadoop-2.7 -Phiv...原创 2019-11-15 12:17:35 · 3117 阅读 · 1 评论 -
Spark on Kubernetes提交测试任务失败报错:User "system:serviceaccount:default:default" cannot get resource "pods
Spark On Kubernetes 通过cluster方式提交spark-submit example.jar包测试任务,driver-pod创建成功,任务失败,driver pod报错日志如下:External scheduler cannot be instantiatedCaused by: io.fabric8.kubernetes.client.KubernetesClien...原创 2019-11-06 14:27:39 · 8275 阅读 · 0 评论 -
Spark on Kubernetes提交任务报错:Expected HTTP 101 response but was '403 Forbidden'
环境:Spark版本: 2.4.3Kubernetes版本:v1.16.2问题:提交spark-submit example.jar 以cluster方式到k8s集群,driver-pod报错如下:19/11/06 07:06:54 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes...原创 2019-11-06 15:28:43 · 7475 阅读 · 0 评论 -
Spark On Kubernetes报错Permission denied: user=root, access=WRITE, inode="/user/spark/log":hadoop:supe
Spark On Kubernetes 提交测试任务,driver pod 报错日志,如下:19/11/06 07:38:05 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!19/11/06 07:38:05 INFO SparkContext: S...原创 2019-11-06 15:54:15 · 773 阅读 · 0 评论 -
Failed to initialize compiler: object scala.runtime in compiler mirror-Spark Shell由于Scala编译器原因不能正常启动
有日子没用之前搭的测试集群,今天想用来测试下 Spark一个问题,发现spark-shell启动失败了,报错信息如下:hadoop@master:/software/servers/spark$ spark-shellListening for transport dt_socket at address: 900119/11/06 09:34:26 WARN NativeCodeLoa...原创 2019-11-06 10:50:32 · 1007 阅读 · 2 评论 -
scala [ERROR] Failed to construct terminal; falling back to unsupported
使用scala-2.11.12版本,启动scala时,报错如下:Welcome to Scala 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191).Type in expressions for evaluation. Or try :help.[ERROR] Failed to construct terminal;...原创 2019-10-30 17:27:45 · 4646 阅读 · 7 评论 -
Jenkins+Gitlab配置Private token不生效,报错:Client error: HTTP 401 Unauthorized
问题描述:Jenkins + Gitlab配置Private token拉取git代码,设置好正确的GitLab API token后,Test Connection 失败,报错:Client error: HTTP 401 Unauthorized , 效果如下图:解决办法:点击 Advanced 高级按钮,将API-Level设置由autodetect改为v3,Test Co...原创 2019-02-12 10:34:20 · 15690 阅读 · 15 评论 -
Enable VT-x in your BIOS security settings (refer to document for your computer)
联想T450AS中用虚拟机调试时,报错:Enable VT-x in your BIOS security settings (refer to document for your computer)即:此主机支持Inter VT-x,但Intel VT-x处于禁用状态,未开启。此报错,跟AS开发环境无关,需要在BIOS中打开。方法如下:重启电脑---出现Lenovo时按F2进入-原创 2016-07-11 22:32:50 · 28146 阅读 · 8 评论 -
Android Studio 真机不能调试,报错Error:java.lang.UnsupportedClassVersionError: com/android/dx/command/Main :
android studio出现这个问题是因为buildToolsVersion的原因 我原来 buildToolsVersion "24.0.0 rc3"出现了以上的错误 改为buildToolsVersion "23.0.2"后这个问题解决了 大家尝试找一个正常运行的项目 看看里面的buildToolsVersion用的什么,复制过来就行了。我刚刚也遇到这个问题,现在解决了:原创 2016-07-29 11:31:46 · 6809 阅读 · 0 评论 -
java.net.SocketTimeoutException: connect timed out
报错记录:java.net.SocketTimeoutException: connect timed outat java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:334)原创 2017-01-15 01:35:28 · 39254 阅读 · 5 评论