Hadoop Sequence File 文件的读取和写入

写入代码

下面是写入了100条(key,value)的信息,其中以LongWriable为key,以Text作为value.

        Configuration config = new Configuration();
        FileSystem fs  = FileSystem.get(conf);
        int i = 0;
        Path path = new Path("/home/lake/hello.xml");
        SequenceFile.Writer writer = null;
        SequenceFile.Writer.Option optPath = SequenceFile.Writer.file(path);
        //定义key
        SequenceFile.Writer.Option optKey = SequenceFile.Writer.keyClass(LongWritable.class);
        //定义value
        SequenceFile.Writer.Option optVal = SequenceFile.Writer.valueClass(Text.class);

        writer = SequenceFile.createWriter(conf, optPath, optKey, optVal);
        //写入的数据可以根据你的情况来定,我这只是测试
        String value = "hello world";

        while(i < 100){
            writer.append(new LongWritable(i),new Text(value));
            i ++;
        }
        writer.close();

上面程序运行完成之后,就可以在指定的路径上看到产生的文件。

读取的代码

                Configuration config = new Configuration();
                FileSystem fs  = FileSystem.get(conf);
                Path path = new Path("/home/lake/hello.xml");
                SequenceFile.Reader reader = new SequenceFile.Reader(fs.getConf(), SequenceFile.Reader.file(path));
                List<Object> sampleValues = new ArrayList<Object>();
                Writable key = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(), fs.getConf());
                Writable value = (Writable) ReflectionUtils.newInstance(reader.getValueClass(), fs.getConf());
                int count = 0;
                String keyName = "Key";
                String valueName = "Value";
                //change data to json format
                while (reader.next(key, value) && count < 12) {
sampleValues.add("{\"" + keyName + "\": \"" + key + "\", \"" + valueName + "\": \"" + value + "\"}");
                    count++;
                }
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页