从ftp下载日志到hbase

最新推荐文章于 2021-08-12 14:09:21 发布

pigHead_chen

最新推荐文章于 2021-08-12 14:09:21 发布

阅读量1.2k

点赞数

分类专栏： Hadoop

本文链接：https://blog.csdn.net/pigHead_chen/article/details/8677730

版权

Hadoop 专栏收录该内容

11 篇文章 0 订阅

订阅专栏

1，写Java代码

import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;
import java.net.URISyntaxException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.ftp.FTPFileSystem;
import org.apache.hadoop.io.IOUtils;

public class ImportData
{
	public static void main(String[] args) throws IOException, URISyntaxException
	{	
		if (args.length == 0) {
			System.err.println("log name needed.");
			System.exit(-1);
		}	
		String src="logs/" + args[0];	//args[0]是文件名
		Configuration conf = new Configuration();
		FTPFileSystem ftpfs = new FTPFileSystem();
		ftpfs.setConf(conf);
		ftpfs.initialize(new URI("ftp://userip:password@xxx.xx.xx.xx"), conf);
		FSDataInputStream fsdin = ftpfs.open(new Path(src), 1000);
		FileSystem fileSystem=FileSystem.get(conf);
		OutputStream outputStream=fileSystem.create(new Path(args[1]));
		IOUtils.copyBytes(fsdin, outputStream, conf, true);
	}

}

说明：args[0]是日志的文件名，args[1]是hdfs里的文件名

编译运行即可

2，在eclipse下运行会报错

异常：

Exception in thread"main" java.lang.IllegalArgumentException: Wrong FS:hdfs://localhost:9000/user/hduser/2012-09-02, expected: file:///

atorg.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)

atorg.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)

atorg.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:335)

atorg.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:492)

atorg.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:377)

atorg.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)

atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)

atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)

atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)

atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)

atorg.Importer.DownloadFromFtp.main(DownloadFromFtp.java:31)

解决办法1，从本地加载配置文件

Configuration conf = new Configuration();
conf.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml"));