1.虚拟机版本VMware® Workstation 12 Pro 12.1.1 build-3770994
安装略
2.os为centos7.2 ,jdk1.8
centos安装略
jdk官网下载,tar -zxvf 解压,配置/etc/profile jdk环境变量
3.官网下载hadoop,我这里用我原来下的hadoop2.7,安装的单机伪分布模式,笔记本配置不高,没办法。
我的安装路径:
[root@localhost hadoop-2.7.0]# pwd
/usr/soft/hadoop-2.7.0
进入etc/hadoop里修改
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.67.100:9000</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
hadoop-env.sh
export JAVA_HOME=/usr/include/jdk1.8
4.配置ssh免登陆
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
5.格式化namenode
bin/hdfs namenode -format 启动hdfs$ sbin/start-dfs.sh 启动yarn$ sbin/start-yarn.sh
6.启动后,登陆验证是否起来了
http://192.168.67.100:8088/cluster
http://192.168.67.100:50070/dfshealth.html#tab-overview
7eclipse测试。
创建maven项目,pox文件增加依赖
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.0</version>
</dependency>
项目结构
static final String PATH = "hdfs://192.168.67.100:9000/";
public static void main(String[] args) throws Exception {
FileSystem ls = FileSystem.get(URI.create(PATH), new Configuration());
// 列出所有目录及文件
FileStatus[] fss = ls.listStatus(new Path("/"));
Arrays.asList(fss).parallelStream().forEach(a -> {
System.out.println("path=" + a.getPath());
});
// 创建目录
ls.mkdirs(new Path("/test2"));
// 创建空文件
ls.create(new Path("/test2/file2.txt"));
System.out.println("新增后重新列出。。。。。。。");
//重新列出
fss = ls.listStatus(new Path("/"));
Arrays.asList(fss).parallelStream().forEach(a -> {
System.out.println("path=" + a.getPath());
});
//递归删除
ls.delete(new Path("/test1"), true);
System.out.println("删除后重新列出。。。。。。。");
//重新列出
fss = ls.listStatus(new Path("/"));
Arrays.asList(fss).parallelStream().forEach(a -> {
System.out.println("path=" + a.getPath());
});
}
运行结果:
path=hdfs://192.168.67.100:9000/test1
path=hdfs://192.168.67.100:9000/cy
新增后重新列出。。。。。。。
path=hdfs://192.168.67.100:9000/test1
path=hdfs://192.168.67.100:9000/cy
path=hdfs://192.168.67.100:9000/test2
删除后重新列出。。。。。。。
path=hdfs://192.168.67.100:9000/test2
path=hdfs://192.168.67.100:9000/cy