记录一下Java API 连接hadoop操作hdfs的实现流程(使用连接池管理)。
以前做过这方面的开发,本来以为不会有什么问题,但是做的还是坑坑巴巴,内心有些懊恼,记录下这烦人的过程,警示自己切莫眼高手低!
一:引入相关jar包如下
org.apache.hadoop
hadoop-common
2.8.2
org.apache.hadoop
hadoop-hdfs
2.8.2
org.apache.commons
commons-pool2
2.6.0
二:连接池开发的基本流程
2.1项目基本环境是SpringBoot大集成···
2.2hadoop相关包结构如下(自己感觉这结构划分的也是凸显了low逼水平【手动笑哭】)
2.2 画个图表达下开发思路
三、上代码
importcom.cmcc.datacenter.hdfs.client.HdfsClient;importcom.cmcc.datacenter.hdfs.client.HdfsFactory;importorg.springframework.beans.factory.annotation.Value;importorg.springframework.context.annotation.Bean;importorg.springframework.context.annotation.Configuration;
@Configurationpublic classHdfsConfig {
@Value("${hadoop.hdfs.ip}")privateString hdfsServerIp;
@Value("${hadoop.hdfs.port}")privateString hdfsServerPort;
@Value("${hadoop.hdfs.pool.maxTotal}")private intmaxTotal;
@Value("${hadoop.hdfs.pool.maxIdle}")private intmaxIdle;
@Value("${hadoop.hdfs.pool.minIdle}")private intminIdle;
@Value("${hadoop.hdfs.pool.maxWaitMillis}")private intmaxWaitMillis;
@Value("${hadoop.hdfs.pool.testWhileIdle}")private booleantestWhileIdle;
@Value("${hadoop.hdfs.pool.minEvictableIdleTimeMillis}")private long minEvictableIdleTimeMillis = 60000;
@Value("${hadoop.hdfs.pool.timeBetweenEvictionRunsMillis}")private long timeBetweenEvictionRunsMillis = 30000;
@Value("${hadoop.hdfs.pool.numTestsPerEvictionRun}")private int numTestsPerEvictionRun = -1;
@Bean(initMethod= "init", destroyMethod = "stop")publicHdfsClient HdfsClient(){
HdfsClient client= newHdfsClient();returnclient;
}/*** Test