Spring Hadoop概述
官方链接
http://spring.io/projects/spring-hadoop
官方文档
https://docs.spring.io/spring-hadoop/docs/2.5.0.RELEASE/reference/html/
Hadoop Configuration官方文档:
https://docs.spring.io/spring-hadoop/docs/2.5.0.RELEASE/reference/html/springandhadoop-config.html
Spring Hadoop开发环境搭建及访问HDFS
添加pom文件
<!--添加spring Hadoop依赖-->
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>2.5.0.RELEASE</version>
</dependency>
在resources下添加beans.xml配置文件
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hdp="http://www.springframework.org/schema/hadoop"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
<bean/>
<hdp:configuration id="hadoopConfiguration">
fs.defaultFS=${spring.hadoop.fsUri}
</hdp:configuration>
<context:property-placeholder location="application.properties"/>
<hdp:file-system id="fileSystem" configuration-ref="hadoopConfiguration" user="hadoop"/>
</beans>
xml中file-system为bean下文要用到,里面的hadoopConfiguration对应hdp:configuration的id为hadoopConfiguration的配置信息,user对应具有读写权限的hadoop的用户。
这些东西可在官方网站找到配置格式开头已给出官网链接
创建application.properties
这个文件是hadoop的uri
spring.hadoop.fsUri=hdfs://hadoop:9000
package com.kun.hadoop.spring;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
public class SpringHadoopHDFSApp {
private ApplicationContext ctx;
private FileSystem fileSystem;
/**
* 创建hdfs文件夹
*/
@Test
public void testMkdir() throws Exception{
fileSystem.mkdirs(new Path("/springhdfs"));
}
/**
* 读取HDFS文件内容
*/
@Test
public void testText() throws Exception{
FSDataInputStream in = fileSystem.open(new Path("/springhdfs/test.txt"));
IOUtils.copyBytes(in, System.out, 1024);
in.close();
}
@Before
public void setUp(){
ctx= new ClassPathXmlApplicationContext("bean.xml");
fileSystem = (FileSystem)ctx.getBean("fileSystem");//得到bean对象
}
@After
public void tearDown(){
ctx=null;
}
}
Spring Boot访问HDFS
添加pom依赖
<!--添加spring boot依赖-->
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop-boot</artifactId>
<version>2.5.0.RELEASE</version>
</dependency>
使用
package com.kun.hadoop.spring;
import org.apache.hadoop.fs.FileStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.hadoop.fs.FsShell;
/**
* 使用Spring Boot的方式访问HDFS
*/
@SpringBootApplication
public class SpringBootHDFSApp implements CommandLineRunner {
@Autowired
FsShell fsShell;
public void run(String... strings) throws Exception{
for(FileStatus fileStatus:fsShell.lsr("/springboothdfs")){
System.out.println(">"+ fileStatus.getPath());
}
}
public static void main(String[] args) {
SpringApplication.run(SpringBootHDFSApp.class,args);
}
}
同时整合mapreduce和hive等都可以通过下面链接进行配置
https://docs.spring.io/spring-hadoop/docs/2.5.0.RELEASE/reference/html/springandhadoop.html