flink1.13.5编译，各种填坑

已于 2022-04-07 17:49:33 修改
阅读量3.3k 收藏 2
点赞数 2
分类专栏： flink 文章标签： flink
于 2022-04-07 17:47:33 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/spark_dev/article/details/124008534
版权

系列文章目录

文章目录

前言

问题：使用官网flink1.13.5,hadoop2.7.2，无法把jar提交到集群进行standalone和yarn模式的部署
解决方法：自编译源码

一、源码准备

到官网或者github下载源码

二、 修改pom

pom中把hadoop和hive的版本改为与自己的版本一致。

三、编译

1.编译失败

编译命令：
mvn clean install -DskipTests

[INFO] > fsevents@1.2.7 install /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/fsevents
[INFO] > node install
[INFO] 
[INFO] 
[INFO] > node-sass@4.11.0 install /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass
[INFO] > node scripts/install.js
[INFO] 
[ERROR] Unable to save binary /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/vendor/linux-x64-64 : { Error: EACCES: permission denied, mkdir '/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/vendor'
[ERROR]     at Object.mkdirSync (fs.js:729:3)
[ERROR]     at sync (/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/mkdirp/index.js:71:13)
[ERROR]     at Function.sync (/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/mkdirp/index.js:77:24)
[ERROR]     at checkAndDownloadBinary (/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/scripts/install.js:114:11)
[ERROR]     at Object.<anonymous> (/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/scripts/install.js:157:1)
[ERROR]     at Module._compile (internal/modules/cjs/loader.js:689:30)
[ERROR]     at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
[ERROR]     at Module.load (internal/modules/cjs/loader.js:599:32)
[ERROR]     at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
[ERROR]     at Function.Module._load (internal/modules/cjs/loader.js:530:3)
[ERROR]   errno: -13,
[ERROR]   syscall: 'mkdir',
[ERROR]   code: 'EACCES',
[ERROR]   path:
[ERROR]    '/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/vendor' }
[INFO] 
[INFO] > node-sass@4.11.0 postinstall /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass
[INFO] > node scripts/build.js
[INFO] 
[INFO] Building: /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node/node /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
[ERROR] gyp info it worked if it ends with ok
[ERROR] gyp verb cli [ '/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node/node',
[ERROR] gyp verb cli   '/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-gyp/bin/node-gyp.js',
[ERROR] gyp verb cli   'rebuild',
[ERROR] gyp verb cli   '--verbose',
[ERROR] gyp verb cli   '--libsass_ext=',
[ERROR] gyp verb cli   '--libsass_cflags=',
[ERROR] gyp verb cli   '--libsass_ldflags=',
[ERROR] gyp verb cli   '--libsass_library=' ]
[ERROR] gyp info using node-gyp@3.8.0
[ERROR] gyp info using node@10.9.0 | linux | x64
[ERROR] gyp verb command rebuild []
[ERROR] gyp verb command clean []
[ERROR] gyp verb clean removing "build" directory
[ERROR] gyp verb command configure []
[ERROR] gyp verb check python checking for Python executable "python2" in the PATH
[ERROR] gyp verb `which` succeeded python2 /usr/bin/python2
[ERROR] gyp verb check python version `/usr/bin/python2 -c "import sys; print "2.7.5
[ERROR] gyp verb check python version .%s.%s" % sys.version_info[:3];"` returned: %j
[ERROR] gyp verb get node dir no --target version specified, falling back to host node version: 10.9.0
[ERROR] gyp verb command install [ '10.9.0' ]
[ERROR] gyp verb install input version string "10.9.0"
[ERROR] gyp verb install installing version: 10.9.0
[ERROR] gyp verb install --ensure was passed, so won't reinstall if already installed
[ERROR] gyp verb install version not already installed, continuing with install 10.9.0
[ERROR] gyp verb ensuring nodedir is created /root/.node-gyp/10.9.0
[ERROR] gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/10.9.0"
[ERROR] gyp WARN EACCES attempting to reinstall using temporary dev dir "/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/.node-gyp"
[ERROR] gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
[ERROR] gyp verb command install [ '--node_gyp_internal_noretry', '10.9.0' ]
[ERROR] gyp verb install input version string "10.9.0"
[ERROR] gyp verb install installing version: 10.9.0
[ERROR] gyp verb install --ensure was passed, so won't reinstall if already installed
[ERROR] gyp verb install version not already installed, continuing with install 10.9.0
[ERROR] gyp verb ensuring nodedir is created /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/.node-gyp/10.9.0
[ERROR] gyp WARN install got an error, rolling back install
[ERROR] gyp verb command remove [ '10.9.0' ]
[ERROR] gyp verb remove using node-gyp dir: /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/.node-gyp
[ERROR] gyp verb remove removing target version: 10.9.0
[ERROR] gyp verb remove removing development files for version: 10.9.0
[ERROR] gyp WARN install got an error, rolling back install
[ERROR] gyp verb command remove [ '10.9.0' ]
[ERROR] gyp verb remove using node-gyp dir: /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/.node-gyp
[ERROR] gyp verb remove removing target version: 10.9.0
[ERROR] gyp verb remove removing development files for version: 10.9.0
[ERROR] gyp ERR! configure error 
[ERROR] gyp ERR! stack Error: EACCES: permission denied, mkdir '/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass/.node-gyp'
[ERROR] gyp ERR! System Linux 3.10.0-1160.49.1.el7.x86_64
[ERROR] gyp ERR! command "/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node/node" "/root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
[ERROR] gyp ERR! cwd /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/node-sass
[ERROR] gyp ERR! node -v v10.9.0
[ERROR] gyp ERR! node-gyp -v v3.8.0
[ERROR] gyp ERR! not ok 
[ERROR] Build failed with error code: 1
[INFO] 
[INFO] > husky@1.3.1 install /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard/node_modules/husky
[INFO] > node husky install
[INFO] 
[INFO] husky > setting up git hooks
[INFO] HUSKY_SKIP_INSTALL environment variable is set to 'true', skipping Git hooks installation.
[ERROR] added 1250 packages in 16.206s
[INFO] 
[INFO] --- frontend-maven-plugin:1.6:npm (npm run build) @ flink-runtime-web_2.11 ---
[INFO] Running 'npm run build' in /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard
[INFO] 
[INFO] > flink-dashboard@2.0.0 build /root/opensource/flink-1.13.5/flink-runtime-web/web-dashboard
[INFO] > ng build --prod --base-href ./
[INFO] 
[ERROR] Browserslist: caniuse-lite is outdated. Please run next command `npm update`
Killed
[root@hadoop103 flink-1.13.5]# npm update caniuse-lite browserslist
[root@hadoop103 flink-1.13.5]# npm update caniuse-lite browserslist
[root@hadoop103 flink-1.13.5]# npm i caniuse-lite browserslist -S
npm ERR! fetch failed https://registry.npmjs.org/browserslist/-/browserslist-4.20.2.tgz
npm WARN retry will retry, error on last attempt: Error: Parse Error
npm ERR! fetch failed https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001325.tgz
npm WARN retry will retry, error on last attempt: Error: Parse Error
npm ERR! fetch failed https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001325.tgz
npm WARN retry will retry, error on last attempt: Error: Parse Error
npm ERR! fetch failed https://registry.npmjs.org/browserslist/-/browserslist-4.20.2.tgz
npm WARN retry will retry, error on last attempt: Error: Parse Error
npm ERR! fetch failed https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001325.tgz
npm ERR! fetch failed https://registry.npmjs.org/browserslist/-/browserslist-4.20.2.tgz
npm ERR! Linux 3.10.0-1160.49.1.el7.x86_64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "i" "caniuse-lite" "browserslist" "-S"
npm ERR! node v6.14.2
npm ERR! npm  v3.10.10
npm ERR! code HPE_UNEXPECTED_CONTENT_LENGTH

npm ERR! Parse Error
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /root/opensource/flink-1.13.5/npm-debug.log

2. 填坑

2.1 填坑1

cd xxxx/flink
mvn clean install -DskipTests -Dfast -T 4 -Dmaven.compile.fork=true  -Dscala-2.11

分解：
mvn clean install \
  -DskipTests \ # 跳过测试部分
  -Dfast \ # 跳过doc检查等
  -T 4 \ # 支持多处理器或者处理器核数参数,加快构建速度,推荐Maven3.3及以上
  -Dmaven.compile.fork=true #允许多线程编译,推荐maven在3.3及以上

（1）flink-runtime-web这个包访问外网npm下载难，需要改一下这个包下的pom文件（执行编译命令前需要做的事）
搜索“ci --cache-max=0 --no-save”替换为“install -registry=https://registry.npm.taobao.org --cache-max=0 --no-save”
即：install -registry=https://registry.npm.taobao.org --cache-max=0 --no-save

作者：FishMAN__
链接：https://www.jianshu.com/p/66a3cf379042

重新编译：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Flink : Table : Planner 1.13.5:
[INFO] 
[INFO] Flink : Table : Planner ............................ SUCCESS [02:49 min]
[INFO] Flink : Formats : .................................. SUCCESS [  4.462 s]
[INFO] Flink : Format : Common ............................ SUCCESS [  0.279 s]
[INFO] Flink : Table : SQL Parser Hive .................... SUCCESS [ 16.250 s]
[INFO] Flink : Table : Runtime Blink ...................... SUCCESS [ 21.968 s]
[INFO] Flink : Table : Planner Blink ...................... FAILURE [ 58.828 s]
[INFO] Flink : Formats : Json ............................. SKIPPED
[INFO] Flink : Connectors : Elasticsearch base ............ SKIPPED
[INFO] Flink : Connectors : Elasticsearch 5 ............... SKIPPED
[INFO] Flink : Connectors : Elasticsearch 6 ............... SKIPPED
[INFO] Flink : Connectors : Elasticsearch 7 ............... SKIPPED
[INFO] Flink : Connectors : HBase base .................... SUCCESS [  7.488 s]
[INFO] Flink : Connectors : HBase 1.4 ..................... SKIPPED
[INFO] Flink : Connectors : HBase 2.2 ..................... SKIPPED
[INFO] Flink : Formats : Hadoop bulk ...................... SUCCESS [  1.868 s]
[INFO] Flink : Formats : Orc .............................. SKIPPED
[INFO] Flink : Formats : Orc nohive ....................... SKIPPED
[INFO] Flink : Formats : Avro ............................. SKIPPED
[INFO] Flink : Formats : Parquet .......................... SKIPPED
[INFO] Flink : Formats : Csv .............................. SKIPPED
[INFO] Flink : Connectors : Hive .......................... SKIPPED
[INFO] Flink : Connectors : JDBC .......................... SKIPPED
[INFO] Flink : Connectors : RabbitMQ ...................... SUCCESS [  2.197 s]
[INFO] Flink : Connectors : Twitter ....................... SUCCESS [  6.866 s]
[INFO] Flink : Connectors : Nifi .......................... SUCCESS [  2.562 s]
[INFO] Flink : Connectors : Cassandra ..................... SKIPPED
[INFO] Flink : Metrics : JMX .............................. SUCCESS [  1.353 s]
[INFO] Flink : Formats : Avro confluent registry .......... SKIPPED
[INFO] Flink : Connectors : Kafka ......................... SKIPPED
[INFO] Flink : Connectors : Google PubSub ................. SUCCESS [  4.299 s]
[INFO] Flink : Connectors : Kinesis ....................... SKIPPED
[INFO] Flink : Connectors : SQL : Elasticsearch 6 ......... SKIPPED
[INFO] Flink : Connectors : SQL : Elasticsearch 7 ......... SKIPPED
[INFO] Flink : Connectors : SQL : HBase 1.4 ............... SKIPPED
[INFO] Flink : Connectors : SQL : HBase 2.2 ............... SKIPPED
[INFO] Flink : Connectors : SQL : Hive 1.2.2 .............. SKIPPED
[INFO] Flink : Connectors : SQL : Hive 2.2.0 .............. SKIPPED
[INFO] Flink : Connectors : SQL : Hive 2.3.6 .............. SKIPPED
[INFO] Flink : Connectors : SQL : Hive 3.1.2 .............. SKIPPED
[INFO] Flink : Connectors : SQL : Kafka ................... SKIPPED
[INFO] Flink : Connectors : SQL : Kinesis ................. SKIPPED
[INFO] Flink : Formats : Sequence file .................... SUCCESS [  1.608 s]
[INFO] Flink : Formats : Compress ......................... SUCCESS [  1.434 s]
[INFO] Flink : Formats : Avro AWS Glue Schema Registry .... SKIPPED
[INFO] Flink : Formats : SQL Orc .......................... SKIPPED
[INFO] Flink : Formats : SQL Parquet ...................... SKIPPED
[INFO] Flink : Formats : SQL Avro ......................... SKIPPED
[INFO] Flink : Formats : SQL Avro Confluent Registry ...... SKIPPED
[INFO] Flink : Examples : Streaming ....................... SKIPPED
[INFO] Flink : Examples : Table ........................... SKIPPED
[INFO] Flink : Examples : Build Helper : .................. SUCCESS [  0.401 s]
[INFO] Flink : Examples : Build Helper : Streaming Twitter  SKIPPED
[INFO] Flink : Examples : Build Helper : Streaming State machine SKIPPED
[INFO] Flink : Examples : Build Helper : Streaming Google PubSub SKIPPED
[INFO] Flink : Container .................................. SUCCESS [  1.244 s]
[INFO] Flink : Queryable state : Runtime .................. SUCCESS [  1.906 s]
[INFO] Flink : Mesos ...................................... SUCCESS [ 57.883 s]
[INFO] Flink : Kubernetes ................................. SUCCESS [ 19.016 s]
[INFO] Flink : Yarn ....................................... SUCCESS [  6.343 s]
[INFO] Flink : Libraries : Gelly .......................... SUCCESS [  8.045 s]
[INFO] Flink : Libraries : Gelly scala .................... SUCCESS [ 42.796 s]
[INFO] Flink : Libraries : Gelly Examples ................. SKIPPED
[INFO] Flink : External resources : ....................... SUCCESS [  0.458 s]
[INFO] Flink : External resources : GPU ................... SUCCESS [  0.462 s]
[INFO] Flink : Metrics : Dropwizard ....................... SUCCESS [  1.061 s]
[INFO] Flink : Metrics : Graphite ......................... SUCCESS [  0.773 s]
[INFO] Flink : Metrics : InfluxDB ......................... SUCCESS [  3.366 s]
[INFO] Flink : Metrics : Prometheus ....................... SUCCESS [  1.748 s]
[INFO] Flink : Metrics : StatsD ........................... SUCCESS [  0.825 s]
[INFO] Flink : Metrics : Datadog .......................... SUCCESS [  1.147 s]
[INFO] Flink : Metrics : Slf4j ............................ SUCCESS [  0.882 s]
[INFO] Flink : Libraries : CEP Scala ...................... SUCCESS [ 34.466 s]
[INFO] Flink : Table : Uber ............................... SKIPPED
[INFO] Flink : Table : Uber Blink ......................... SKIPPED
[INFO] Flink : Python ..................................... SKIPPED
[INFO] Flink : Table : SQL Client ......................... SKIPPED
[INFO] Flink : Libraries : State processor API ............ SUCCESS [  3.106 s]
[INFO] Flink : Dist ....................................... SKIPPED
[INFO] Flink : Yarn Tests ................................. SKIPPED
[INFO] Flink : E2E Tests : ................................ SKIPPED
[INFO] Flink : E2E Tests : CLI ............................ SKIPPED
[INFO] Flink : E2E Tests : Parent Child classloading program SKIPPED
[INFO] Flink : E2E Tests : Parent Child classloading lib-package SKIPPED
[INFO] Flink : E2E Tests : Dataset allround ............... SKIPPED
[INFO] Flink : E2E Tests : Dataset Fine-grained recovery .. SKIPPED
[INFO] Flink : E2E Tests : Datastream allround ............ SKIPPED
[INFO] Flink : E2E Tests : Batch SQL ...................... SKIPPED
[INFO] Flink : E2E Tests : Stream SQL ..................... SKIPPED
[INFO] Flink : E2E Tests : Distributed cache via blob ..... SKIPPED
[INFO] Flink : E2E Tests : High parallelism iterations .... SKIPPED
[INFO] Flink : E2E Tests : Stream stateful job upgrade .... SKIPPED
[INFO] Flink : E2E Tests : Queryable state ................ SKIPPED
[INFO] Flink : E2E Tests : Local recovery and allocation .. SKIPPED
[INFO] Flink : E2E Tests : Elasticsearch 5 ................ SKIPPED
[INFO] Flink : E2E Tests : Elasticsearch 6 ................ SKIPPED
[INFO] Flink : Quickstart : ............................... SUCCESS [  1.541 s]
[INFO] Flink : Quickstart : Java .......................... SUCCESS [  2.873 s]
[INFO] Flink : Quickstart : Scala ......................... SUCCESS [  0.258 s]
[INFO] Flink : E2E Tests : Quickstart ..................... SKIPPED
[INFO] Flink : E2E Tests : Confluent schema registry ...... SKIPPED
[INFO] Flink : E2E Tests : Stream state TTL ............... SKIPPED
[INFO] Flink : E2E Tests : SQL client ..................... SKIPPED
[INFO] Flink : E2E Tests : File sink ...................... SKIPPED
[INFO] Flink : E2E Tests : State evolution ................ SKIPPED
[INFO] Flink : E2E Tests : RocksDB state memory control ... SKIPPED
[INFO] Flink : E2E Tests : Common ......................... SKIPPED
[INFO] Flink : E2E Tests : Metrics availability ........... SKIPPED
[INFO] Flink : E2E Tests : Metrics reporter prometheus .... SKIPPED
[INFO] Flink : E2E Tests : Heavy deployment ............... SKIPPED
[INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SKIPPED
[INFO] Flink : E2E Tests : Streaming Kafka base ........... SKIPPED
[INFO] Flink : E2E Tests : Streaming Kafka ................ SKIPPED
[INFO] Flink : E2E Tests : Plugins : ...................... SKIPPED
[INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SKIPPED
[INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SKIPPED
[INFO] Flink : E2E Tests : TPCH ........................... SKIPPED
[INFO] Flink : E2E Tests : Streaming Kinesis .............. SKIPPED
[INFO] Flink : E2E Tests : Elasticsearch 7 ................ SKIPPED
[INFO] Flink : E2E Tests : Common Kafka ................... SKIPPED
[INFO] Flink : E2E Tests : TPCDS .......................... SKIPPED
[INFO] Flink : E2E Tests : Netty shuffle memory control ... SKIPPED
[INFO] Flink : E2E Tests : Python ......................... SKIPPED
[INFO] Flink : E2E Tests : HBase .......................... SKIPPED
[INFO] Flink : E2E Tests : AWS Glue Schema Registry ....... SKIPPED
[INFO] Flink : State backends : Heap spillable ............ SUCCESS [  1.198 s]
[INFO] Flink : Contrib : .................................. SUCCESS [  0.504 s]
[INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [  1.566 s]
[INFO] Flink : FileSystems : Tests ........................ SKIPPED
[INFO] Flink : Docs ....................................... SKIPPED
[INFO] Flink : Walkthrough : .............................. SUCCESS [  0.476 s]
[INFO] Flink : Walkthrough : Common ....................... SUCCESS [  1.539 s]
[INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [  0.261 s]
[INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [  0.295 s]
[INFO] Flink : Tools : CI : Java .......................... SUCCESS [  1.627 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:52 min (Wall Clock)
[INFO] Finished at: 2022-04-07T11:19:11+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project flink-table-planner-blink_2.12: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project flink-table-planner-blink_2.12: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
    at scala_maven.ScalaMojoSupport.execute (ScalaMojoSupport.java:490)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 137 (Exit value: 137)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:377)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:160)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:147)
    at scala_maven_executions.JavaMainCallerByFork.run (JavaMainCallerByFork.java:100)
    at scala_maven.ScalaCompilerSupport.compile (ScalaCompilerSupport.java:161)
    at scala_maven.ScalaCompilerSupport.doExecute (ScalaCompilerSupport.java:99)
    at scala_maven.ScalaMojoSupport.execute (ScalaMojoSupport.java:482)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :flink-table-planner-blink_2.12

2.2 填坑２ 仓库没有confluent.version>5.3.0版本

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  28.374 s (Wall Clock)
[INFO] Finished at: 2022-04-07T11:34:06+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-avro-confluent-registry: Could not resolve dependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.13.5: io.confluent:kafka-schema-registry-client:jar:5.5.2 was not found in http://maven.aliyun.com/nexus/content/groups/public during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of nexus-aliyun has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:

原因： 到mvnrepo查，发现： 只有5.3.0版本
解决方法：把pom 版本改了

[root@hadoop103 flink-formats]# vim flink-avro-confluent-registry/pom.xml
<confluent.version>5.3.0</confluent.version>

2.3 填坑3 编译出错，从错误地方继续跑的方法

-rf : 指定失败的module ，从失败的地方继续跑

编译命令：

[root@hadoop103 flink-1.13.5]# mvn clean install -DskipTests -Dfast -T 4 -Dmaven.compile.fork=true -Dscala-2.12 -rf :flink-avro-confluent-registry

[INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SUCCESS [ 21.321 s]
[INFO] Flink : E2E Tests : Streaming Kafka base ........... SUCCESS [  0.210 s]
[INFO] Flink : E2E Tests : Streaming Kafka ................ SUCCESS [  7.944 s]
[INFO] Flink : E2E Tests : Plugins : ...................... SUCCESS [  0.113 s]
[INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SUCCESS [  0.095 s]
[INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SUCCESS [  0.148 s]
[INFO] Flink : E2E Tests : TPCH ........................... SUCCESS [  1.719 s]
[INFO] Flink : E2E Tests : Streaming Kinesis .............. SUCCESS [ 19.961 s]
[INFO] Flink : E2E Tests : Elasticsearch 7 ................ SUCCESS [  4.154 s]
[INFO] Flink : E2E Tests : Common Kafka ................... SUCCESS [ 56.202 s]
[INFO] Flink : E2E Tests : TPCDS .......................... SUCCESS [  1.079 s]
[INFO] Flink : E2E Tests : Netty shuffle memory control ... SUCCESS [  0.152 s]
[INFO] Flink : E2E Tests : Python ......................... SUCCESS [  8.451 s]
[INFO] Flink : E2E Tests : HBase .......................... SUCCESS [  2.208 s]
[INFO] Flink : E2E Tests : AWS Glue Schema Registry ....... SUCCESS [ 23.854 s]
[INFO] Flink : State backends : Heap spillable ............ SUCCESS [  0.517 s]
[INFO] Flink : Contrib : .................................. SUCCESS [  0.108 s]
[INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [  0.858 s]
[INFO] Flink : FileSystems : Tests ........................ SUCCESS [  2.662 s]
[INFO] Flink : Docs ....................................... SUCCESS [  3.249 s]
[INFO] Flink : Walkthrough : .............................. SUCCESS [  0.423 s]
[INFO] Flink : Walkthrough : Common ....................... SUCCESS [  0.433 s]
[INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [  0.133 s]
[INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [  0.222 s]
[INFO] Flink : Tools : CI : Java .......................... SUCCESS [  2.507 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:46 min (Wall Clock)
[INFO] Finished at: 2022-04-07T11:59:03+08:00
[INFO] ------------------------------------------

最后的安装包在：build-target目录下

/root/opensource/flink-1.13.5/build-target

总结

编译成功，经过测试，服务端，能客户端提交的任务

*星星之火*
关注
  • 2
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
1、Flink1.12.7或1.13.5详细介绍及本地安装部署、验证
alanchanchn的专栏
07-04 4万+
本示例以1.12版本进行介绍，当前版本更新至1.17。Flink 诞生于欧洲的一个大数据研究项目 StratoSphere。该项目是柏林工业大学的一个研究性项目。早期， Flink 是做 Batch 计算的，但是在 2014 年， StratoSphere 里面的核心成员孵化出 Flink，同年将 Flink 捐赠 Apache，并在后来成为 Apache 的顶级大数据项目，同时 Flink 计算的主流方向被定位为 Streaming， 即用流式计算来做所有大数据的计算。
flink-connector-hive_2.11-1.11.0.jar
07-30
修改过源码的jar包
Flink1.13.5 SQL与Hive集成开发
大数据研习社
05-10 1861
1 添加依赖包 Flink SQL与Hive集成需要添加相关依赖包如下所示： #Flink与hadoop兼容包 flink-shaded-hadoop-2-uber-2.8.3-10.0.jar #Flink与hive集成包 flink-sql-connector-hive-2.3.6_2.11-1.13.6.jar #其他依赖包（在hadoop安装目录的shere/hadoop/common目录下） commons-configuration-1.6.jar commons-loggi
flink13.5整合hudi10
wudonglianga的专栏
02-07 1873
1. 版本 组件 版本 hudi 10.0 flink 13.5 2.hudi 源码下载 https://github.com/apache/hudi/releases 2.1 需要改flink 版本为13.5 根目录下面的pom 文件 <flink.version>1.13.5</flink.version> <hive.version>3.1.0</hive.version> <hadoop.version>3.1.
编译flink1.11版本问题总结
L13763338360的博客
08-17 2405
1.解决npm问题 1.1 报错如下 [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.6:npm (npm install) on project flink-runtime-web_2.11: Failed to run task: 'npm ci --cache-max=0 --no-save --registry=https://registry.npm.taobao.org' failed..
flink 1.13.2的pom.xml文件模板
最新发布
m0_49303490的博客
11-20 787
flink 1.13.2的pom.xml文件模板
FLink1.13.1+FlinkCDC2.0.2+Hudi0.10构建流批一体数仓
jekongoo的博客
09-28 2473
构建hudi （1）通过国内镜像拉取源码 git clone https://github.com/apache/hudi.git （2）修改pom.xml vim pom.xml -- 直接添加 <repository> <id>nexus-aliyun</id> <name>nexus-aliyun</name> <url>http://maven.aliyun.com/nexus/
Flink编程报错集锦
泛音青年的博客
03-09 7493
1、Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath. Exception in thread "main" org.apache.flink.table.api.TableException: findAndCr...
hive CLI 命令选项
木下瞳的博客
03-22 751
$ hive --help 一个简单说明的选项列表，Service List 后面的内容，是提供的服务 $ hive--help --service cli 显示了 CLI 所提供的选项列表 命名空间 $hive -define foo=bar 定义一个变量 foo set 显示或修改变量值，直接 set 显示全部变量，只截取了部分，还可以...
Hive 启动时报错
Jron_的博客
03-08 572
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.
Hive搭建
luoyepiaoxue2014的博客
11-25 1819
Hive系列第二章
【flink】Flink 1.13 源码 编译
九师兄
01-13 457
今天编译flink 1.13 这里我是去掉了很多不想要的模块的，大家如果也像加快速度，或者不想出现那么多问题，也可以去掉不想要的模块。自己不想要的模块都可以移除，这样编译速度快一些，而且可能没有那么多问题。测试模块 不想要的模块我都给排除了。...
Flink+hudi遇到的坑（error原因及处理，随时补充）
qq_40788398的博客
04-15 6098
1. java.lang.LinkageError: org/apache/hudi/org/apache/parquet/column/ColumnDescriptor 原因：主键问题 处理方式：检查建表语句主键，参考官网主键设置。
Flink pom文件模板
weixin_43947279的博客
05-06 819
创建Flink项目时，如果是新手最好是根据官方的建议创建相应的Flink项目。 官方这边推荐两个方式创建 第一种方式，使用maven进行创建项目，前提你得安装maven： $ mvn archetype:generate \ -DarchetypeGroupId=org.apache.flink \ -DarchetypeArtifactId=flink-quickstart-java \ -DarchetypeVersion=1.13.0 Parameter
【Flink原理和应用】：记一次Flink源码编译的过程
hxcaifly的博客
01-12 5430
1. 克隆Flink源码 首先，把Flink源码克隆到本地： git clone https://github.com/apache/flink.git 2. 直接进行打包编译 执行： mvn clean package -DskipTests 报错了，错误日志： Failed to execute goal org.apache.maven.plugins:maven-checkstyle...
flink 1.12本地编译报错解决--flink-fs-hadoop-shaded等
iwonder123的博客
12-15 1868
flink 1.12本地编译报错解决问题描述：原因分析：解决方案：解决方案： 问题描述： 找不到jar包 Failure to find org.apache.flink:flink-fs-hadoop-shaded:jar:tests:1.11-byted-SNAPSHOT in https://maven.byted.org/repository/public was cached in the local repository, resolution will not be reattempted
解决 storm 本地启动失败问题
你看你细细看
10-26 707
报错内容 Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/topology/IRichSpout at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.priv
beeline连接hiveserver2报错：User: root is not allowed to impersonate root
热门推荐
Magician的博客
08-29 2万+
最近在生产中搭建HA机制的集群，碰到不少坑，会在接下来的时间里好好总结下，先说下遇到的一个大坑。我们的需求是：希望通过hive的thrift服务来实现跨语言访问Hive数据仓库。但是第一步，你得需要在节点中打通服务器端（启动hiveserver2的节点）和客户端（启动beeline的节点）的链接。整体的结构如下图所示： 但是整个的配置过程可谓是一波三折，现在数据1节点启动hiveserver...
flink cdc mysql一定要编译源码吗
08-30
一般来说，使用Flink CDC时不需要编译源码。用户可以直接在Flink CDC官网下载官方编译好的二进制包或在项目的pom.xml文件中添加相关依赖即可使用。只有在以下几种情况下才需要进行源码编译： 1. 用户对Flink CDC源码进行了修改。 2. Flink CDC某依赖项的版本与运行环境不一致。 3. 官方未提供最新版本的Flink CDC二进制安装包。 所以，一般情况下，使用Flink CDC不需要编译源码，可以直接使用官方提供的二进制包或相关依赖。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* *2* [Flink CDC 系列（2）—— Flink CDC 源码编译](https://blog.csdn.net/weixin_47298890/article/details/123029835)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"] - *3* [最新版Flink CDC MySQL同步MySQL（一）](https://blog.csdn.net/weixin_43114209/article/details/131553658)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"] [ .reference_list ]
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值