Spring boot 与 hadoop整合

2019独角兽企业重金招聘Python工程师标准>>>

Spring boot 与 hadoop 整合

获取目录

HadoopTest.java

import org.apache.hadoop.fs.FileStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.hadoop.fs.FsShell;

@SpringBootApplication
public class HadoopTest implements CommandLineRunner {

	@Autowired
	private FsShell shell;

	[@Override](https://my.oschina.net/u/1162528)
	public void run(String... args) {
		for (FileStatus s : shell.lsr("/")) {
			System.out.println("> " + s.getPath());
		}
	}

	public static void main(String[] args) {
		SpringApplication.run(HadoopTest.class, args);
	}
}

application.yml

spring:
  main:
	show_banner: false
  hadoop:
	fsUri: hdfs://172.16.100.91:8020

pom.xml

<dependency>
	<groupId>org.springframework.data</groupId>
	<artifactId>spring-data-hadoop</artifactId>
	<version>2.2.0.RELEASE</version>
	<exclusions>
		<exclusion>
			<groupId>org.springframework</groupId>
			<artifactId>spring-context-support</artifactId>
		</exclusion>
		<exclusion>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
		</exclusion>
		<exclusion>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-to-slf4j</artifactId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-client</artifactId>
	<version>2.7.0</version>
	<exclusions>
		<exclusion>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-to-slf4j</artifactId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-common</artifactId>
	<version>${hadoop.version}</version>
	<scope>compile</scope>
	<exclusions>
		<exclusion>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-to-slf4j</artifactId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>org.springframework.data</groupId>
	<artifactId>spring-data-hadoop-boot</artifactId>
	<version>2.2.0.RELEASE</version>
</dependency>
<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-client</artifactId>
	<version>2.6.0-cdh5.7.0</version>
</dependency>

Windows下需要hadoop运行环境

添加环境变量

变量名：HADOOP_USER_NAME
变量值：root

变量名：HADOOP_HOME
变量值：D:\lwenhao\apache-hadoop-3.1.0-winutils-master

需要把HADOOP_HOME配置到PATH中

下载地址apache-hadoop-3.1.0-winutils-master

提取码：suik

转载于:https://my.oschina.net/lwenhao/blog/3025710