java hbase创建_本地Java大数据环境基础配置

创建项目

准备pom.xml文件配置(附在文档最后)

在下载jar过程中极其缓慢，简直不能忍受，查阅了下资料，得到下面的办法：

1、 maven仓库中没有支持cdh的相关依赖。cloudera自己建立了一个相关的仓库。要想利用maven添加相关依赖，则必须单独添加cloudera仓库

2、 在国内使用maven中央仓库一般会有网络问题，大部分人会使用aliyun仓库或者其他开源的仓库。所以需要修改setting.xml (以下配置中 *,!cloudera 表示除了aliyun仓库还使用cloudera仓库)

配置：

         nexus-aliyun         *,!cloudera         Nexus aliyun                                        http://maven.aliyun.com/nexus/content/groups/public         

配置环境

下面两个配置表示是在本地运行还是到Linux环境中运行

dev

编写内容

local

编写内容

准备log4j的日志配置，这个网上很多，可以直接复制一份，改下关键地方就可以

HDFS连接地址的配置

Hbase的连接配置

配置文件transformer-env.xml

文件内容

pom.xml文件配置

  4.0.0  com.xlgl.wzy  localenv  0.0.1-SNAPSHOT       2.7.3    1.2.5    2.1.1    5.1.6    2.6.2    0.6.1                                                                          aliyun                          http://maven.aliyun.com/nexus/content/groups/public/                                                              cloudera                          https://repository.cloudera.com/artifactory/cloudera-repos/                                                              jboss                          http://repository.jboss.org/nexus/content/groups/public                                        org.apache.hadoop      hadoop-client      ${hadoop.version}               org.apache.hbase      hbase-client      ${hbase.version}               org.apache.hbase      hbase-server      ${hbase.version}               org.apache.hive      hive-exec      ${hive.version}                   mysql      mysql-connector-java      ${mysql.version}                   cz.mallat.uasparser      uasparser      ${uasparser.version}               com.google.code.gson      gson      ${gson.version}                             local              true                                                             configuration/local                                                         src/main/resources                                                                   org.codehaus.mojo            build-helper-maven-plugin            1.4                                          add-source                 generate-sources                                    add-source                                                                          ${basedir}/src/main/java                    ${basedir}/src/main/extr                                                                                                            dev                                                       configuration/dev                                                         src/main/resources                                                       org.codehaus.mojo            build-helper-maven-plugin            1.4                                          add-source                 generate-sources                                    add-source                                                                          ${basedir}/src/main/java                                                                                                            org.apache.maven.plugins            maven-shade-plugin            2.1                                          package                                  shade                                                                                            cz.mallat.uasparser:uasparser                      net.sourceforge.jregex:jregex                      mysql:mysql-connector-java                                                                                                                             src/test/java                       maven-assembly-plugin                                                         jar-with-dependencies                                                                                           make-assembly                       package                                                  assembly                                                                                maven-compiler-plugin        3.3                  1.8          1.8          utf8                                                                                                           org.eclipse.m2e                              lifecycle-mapping                              1.0.0                                                                                                                                                                                                                                                                                                                         org.codehaus.mojo                                                                                                                                                             build-helper-maven-plugin                                                                                                                                                    [1.4,)                                                                                                                                                             add-source