1、安装idea
2、新建Maven项目
其中GroupId一般是公司域名倒写,例:www.baidu.com,一般包名为com.baidu,这是一种规范,ArtifactId一般是指项目的具体作用,比如testHdfs。
3、项目完成后打开项目目录下的pom.xml文件,并添加如下依赖
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>1.8</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
</dependencies>
4、将linux环境下core-site.xml和hdfs-site.xml文件拷贝到idea项目目录下resources文件夹下,具体如下图
5、测试是否成功,先将集群启动
1)在xshell中启动集群
bash>start-dfs.sh
bash>start-yarn.sh
2)在java文件夹下新建类,并输入以下代码
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;
public class HadoopTest2 {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://192.168.59.100:9000");
FileSystem fs = FileSystem.get(conf);
fs.mkdirs(new Path("/1104/clover"));
fs.close();
System.out.println("over");
}
}
以上是调试成功的
调试中出现以下问题:
1)出现如下图异常,则修改hadoop100为其对应的ip地址
2)出现如下异常,则点击右上角Edit Configrations
第一种方法:
在VMoption中输入:-DHADOOP_USER_NAME=root,root是你登陆linux的用户名称即可
第二种方法:
在代码中进行添加
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
public class HadoopTest2 {
public static void main(String[] args) throws IOException, URISyntaxException, InterruptedException {
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://192.168.59.100:9000");
//FileSystem fs = FileSystem.get(conf);
FileSystem fs = FileSystem.get(new URI("hdfs://192.168.59.100:9000"),conf,"clover");
fs.mkdirs(new Path("/1107/clover"));
fs.close();
System.out.println("over");
}
}