hadoop解决中文输出乱码（二）

[size=medium] [b]hadoop涉及输出文本的默认输出编码统一用[color=red]没有BOM的UTF-8[/color]的形式[/b]，有很多软件只能靠文件的BOM才能成够识别编码为UTF-8的文件，因此当用此类软件打开文件时，显示的是乱码。为了能够正常显示文件内容，在输出内容到文件内之前，先写入BOM（0xEFBBBF）即可。[/size]
[size=medium]Map Reduce运行的结果是一些列文件，通过使用如下函数将输出文件归并到一个文件中：[/size]

org.apache.hadoop.fs.FileUtil.copyMerge(FileSystem srcFS, Path srcDir, FileSystem dstFS, Path dstFile, boolean deleteSource, Configuration conf, String addString)

[size=medium]其内的代码如下：[/size]

/** Copy all files in a directory to one output file (merge). */
  public static boolean copyMerge(FileSystem srcFS, Path srcDir, 
                                  FileSystem dstFS, Path dstFile, 
                                  boolean deleteSource,
                                  Configuration conf, String addString) throws IOException {
    dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);

    if (!srcFS.getFileStatus(srcDir).isDirectory())
      return false;

    OutputStream out = dstFS.create(dstFile);

    try {
      FileStatus contents[] = srcFS.listStatus(srcDir);
      Arrays.sort(contents);
      for (int i = 0; i < contents.length; i++) {
        if (contents[i].isFile()) {
          InputStream in = srcFS.open(contents[i].getPath());
          try {
            IOUtils.copyBytes(in, out, conf, false);
            if (addString!=null)
              out.write(addString.getBytes("UTF-8"));

          } finally {
            in.close();
          } 
        }
      }
    } finally {
      out.close();
    }


    if (deleteSource) {
      return srcFS.delete(srcDir, true);
    } else {
      return true;
    }
  } 

[size=medium]在第12行加入如下代码即可完成对UTF-8 BOM的写入：[/size]

out.write(new byte[]{(byte)0xEF,(byte)0xBB,(byte)0xBF});

[size=medium]最终的结果为：[/size]

/** Copy all files in a directory to one output file (merge). */
  public static boolean copyMerge(FileSystem srcFS, Path srcDir, 
                                  FileSystem dstFS, Path dstFile, 
                                  boolean deleteSource,
                                  Configuration conf, String addString) throws IOException {
    dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);

    if (!srcFS.getFileStatus(srcDir).isDirectory())
      return false;

    OutputStream out = dstFS.create(dstFile);
    out.write(new byte[]{(byte)0xEF,(byte)0xBB,(byte)0xBF}); // 写入BOM
    try {
      FileStatus contents[] = srcFS.listStatus(srcDir);
      Arrays.sort(contents);
      for (int i = 0; i < contents.length; i++) {
        if (contents[i].isFile()) {
          InputStream in = srcFS.open(contents[i].getPath());
          try {
            IOUtils.copyBytes(in, out, conf, false);
            if (addString!=null)
              out.write(addString.getBytes("UTF-8"));

          } finally {
            in.close();
          } 
        }
      }
    } finally {
      out.close();
    }


    if (deleteSource) {
      return srcFS.delete(srcDir, true);
    } else {
      return true;
    }
  }