背景
最近一直在调查数据分析以及集群的软件,接下来,又开始了BI的调查了,今天就收到邮件,让调查apache falcon。各种环境折腾又开始了。
先理解一下Falcon的架构:
事前准备
- apache falcon源码 (本次使用的是最新的apache-falcon-0.9-sources.tar.gz)
- jdk(1.7/1.8)
- maven(3.X) apache-maven-3.0.3-bin.tar.gz 【后续提示出错,换成maven3.1.0】
安装
JDK
- JDK的安装,此处就不再说明了,很简单,装完后记得配置/etc/profile里JAVA_HOME,完后source一下就OK了,确认如下:
[root@sv004 project]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
maven
- maven的安装,本人第一次,也很简单,解压,配置MAVEN_BIN,然后使用mvn -v确认是否安装。效果如下所示:
[root@sv004 apache-maven-3.0.3]# ./bin/mvn -v
Apache Maven 3.0.3 (r1075438; 2011-03-01 02:31:09+0900)
Maven home: /home/project/apache-maven-3.0.3
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
本次的重点来了
使用maven编译falcon源码
1.解压falcon源码
tar -xzvf apache-falcon-${project.version}-source.tar.gz
cd falcon-sources-${project.version}
2.编译falcon
export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install
但此时报错了
[INFO] Scanning for projects...
Downloading: http://repo1.maven.org/maven2/org/apache/apache/17/apache-17.pom
Downloading: https://raw.github.com/InMobi/mvn-repo/master/releases/org/apache/apache/17/apache-17.pom
Downloading: http://repo.hortonworks.com/content/repositories/releases/org/apache/apache/17/apache-17.pom
Downloading: https://repository.apache.org/content/groups/snapshots/org/apache/apache/17/apache-17.pom
Downloading: https://repository.apache.org/content/groups/public/org/apache/apache/17/apache-17.pom
Downloading: https://maven.java.net/content/groups/public/org/apache/apache/17/apache-17.pom
Downloading: https://repository.jboss.org/nexus/content/groups/public/org/apache/apache/17/apache-17.pom
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/17/apache-17.pom
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.falcon:falcon-main:0.9 (/home/project/apache-maven-3.0.3/falcon-sources-0.9/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:17 from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org and 'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown host repo1.maven.org -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
原因
根据Unkown host信息调查了一下,应该是代理设置的有问题,我的maven使用的是默认的,没有设置,坑就被我踩到了。
参考URL:http://stackoverflow.com/questions/15334394/could-not-transfer-artifact-org-apache-maven-pluginsmaven-surefire-pluginpom2/15334627#15334627
解决策
修改我的maven/conf/setting.xml,如下所示:
<proxies>
<proxy>
<active>true</active>
<protocol>http</protocol>
<host>***.***.***.***</host> ⇒代理的IP地址
<port>8080</port>
<nonProxyHosts>maven</nonProxyHosts>
</proxy>
</proxies>
完后重新执行
export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install
第一次时间比较长,取决你的网速,因为要下载好多依赖包文件。
好事多磨呀.....
Exception while executing SCM command.
[INFO] Storing buildScmBranch: UNKNOWN_BRANCH
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ falcon-ui ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Falcon ..................................... SUCCESS [18:56.122s]
[INFO] Apache Falcon UI .................................. FAILURE [3.810s]
[INFO] Checkstyle ........................................ SKIPPED
[INFO] Build Tools ....................................... SKIPPED
[INFO] Apache Falcon Java client ......................... SKIPPED
[INFO] Apache Falcon CLI client .......................... SKIPPED
[INFO] Apache Falcon Metrics ............................. SKIPPED
[INFO] Apache Falcon Hadoop Dependencies ................. SKIPPED
[INFO] Apache Falcon Test Utility ........................ SKIPPED
[INFO] Apache Falcon Commons ............................. SKIPPED
[INFO] Apache Falcon Oozie EL Extension .................. SKIPPED
[INFO] Apache Falcon Embedded Hadoop - Test Cluster ...... SKIPPED
[INFO] Apache Falcon Sharelib Hive - Test Cluster ........ SKIPPED
[INFO] Apache Falcon Sharelib Pig - Test Cluster ......... SKIPPED
[INFO] Apache Falcon Sharelib Hcatalog - Test Cluster .... SKIPPED
[INFO] Apache Falcon Sharelib Oozie - Test Cluster ....... SKIPPED
[INFO] Apache Falcon Test Tools - Test Cluster ........... SKIPPED
[INFO] Apache Falcon Messaging ........................... SKIPPED
[INFO] Apache Falcon LIfecycle Module .................... SKIPPED
[INFO] Apache Falcon Oozie Adaptor ....................... SKIPPED
[INFO] Apache Falcon Scheduler ........................... SKIPPED
[INFO] Apache Falcon Acquisition ......................... SKIPPED
[INFO] Apache Falcon Distcp Replication .................. SKIPPED
[INFO] Apache Falcon Retention ........................... SKIPPED
[INFO] Apache Falcon Archival ............................ SKIPPED
[INFO] Apache Falcon Rerun ............................... SKIPPED
[INFO] Apache Falcon Prism ............................... SKIPPED
[INFO] falcon-unit ....................................... SKIPPED
[INFO] Apache Falcon Web Application ..................... SKIPPED
[INFO] Apache Falcon Documentation ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:04.642s
[INFO] Finished at: Thu Apr 14 02:38:36 JST 2016
[INFO] Final Memory: 18M/237M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.23:install-node-and-npm (install node and npm) on project falcon-ui: The plugin com.github.eirslett:frontend-maven-plugin:0.0.23 requires Maven version 3.1.0 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginIncompatibleException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :falcon-ui
根据报错信息来看,maven的版本与下载的插件的版本不一致导致的,根据提示信息,遂决定使用maven3.1.0。
构建好Maven3.1.0的环境后,重新再次编译falcon源码,结果悲剧了,install-node-and-npm无法下载.......
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Falcon ..................................... SUCCESS [2.380s]
[INFO] Apache Falcon UI .................................. FAILURE [1:02.489s]
[INFO] Checkstyle ........................................ SKIPPED
[INFO] Build Tools ....................................... SKIPPED
[INFO] Apache Falcon Java client ......................... SKIPPED
[INFO] Apache Falcon CLI client .......................... SKIPPED
[INFO] Apache Falcon Metrics ............................. SKIPPED
[INFO] Apache Falcon Hadoop Dependencies ................. SKIPPED
[INFO] Apache Falcon Test Utility ........................ SKIPPED
[INFO] Apache Falcon Commons ............................. SKIPPED
[INFO] Apache Falcon Oozie EL Extension .................. SKIPPED
[INFO] Apache Falcon Embedded Hadoop - Test Cluster ...... SKIPPED
[INFO] Apache Falcon Sharelib Hive - Test Cluster ........ SKIPPED
[INFO] Apache Falcon Sharelib Pig - Test Cluster ......... SKIPPED
[INFO] Apache Falcon Sharelib Hcatalog - Test Cluster .... SKIPPED
[INFO] Apache Falcon Sharelib Oozie - Test Cluster ....... SKIPPED
[INFO] Apache Falcon Test Tools - Test Cluster ........... SKIPPED
[INFO] Apache Falcon Messaging ........................... SKIPPED
[INFO] Apache Falcon LIfecycle Module .................... SKIPPED
[INFO] Apache Falcon Oozie Adaptor ....................... SKIPPED
[INFO] Apache Falcon Scheduler ........................... SKIPPED
[INFO] Apache Falcon Acquisition ......................... SKIPPED
[INFO] Apache Falcon Distcp Replication .................. SKIPPED
[INFO] Apache Falcon Retention ........................... SKIPPED
[INFO] Apache Falcon Archival ............................ SKIPPED
[INFO] Apache Falcon Rerun ............................... SKIPPED
[INFO] Apache Falcon Prism ............................... SKIPPED
[INFO] falcon-unit ....................................... SKIPPED
[INFO] Apache Falcon Web Application ..................... SKIPPED
[INFO] Apache Falcon Documentation ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:05.883s
[INFO] Finished at: Thu Apr 14 20:57:47 JST 2016
[INFO] Final Memory: 30M/303M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.23:install-node-and-npm (install node and npm) on project falcon-ui: Could not extract the Node archive: Could not extract archive: '/home/project/apache-maven-3.1.0/falcon-sources-0.9/falcon-ui/node_tmp/node.tar.gz': EOFException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apach