GDAL访问HDFS集群中的数据

1.集群搭建

参考文章：hadoop2.10.0完全分布式集群搭建+ HA(QJM)高可用集群搭建_hadoop 2.10 ha-CSDN博客

创建文件夹  hdfs dfs -mkdir -p hdfs://192.168.80.132:9000/test  
开放权限  hdfs dfs -chmod -R 777 /test
上传文件  hadoop fs -put /home/wh/data/result_0.tiff hdfs://192.168.80.132:9000/test

2.GDAL+HDFS编译

参考文章：GDAL编译安装指南 - 简书

可以不编译hapoop，直接使用官网的库

需要新增的环境变量

export JAVA_HOME=/home/wuhao/project/jdk1.8.0_411
export HADOOP_HOME=/home/wuhao/project/hadoop-2.10.2

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath --glob)

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native:$JAVA_HOME/jre/lib/amd64/server

        同时需要把第一步的core-site.xml配置同步到客户机上，可以直接拷贝hadoop/etc目录，不然会一直报下面的错误。

        使用gdalinfo验证

 3. 使用c++测试GDAL接口读取tif文件

#include <iostream>
#include <gdal.h>
#include "gdal_priv.h"

using namespace std;

int main()
{
        string hdfsPath = "hdfs://192.168.80.132:9000/wuhao/test/result_0.tiff";
        cout << "hdfsPath:" << hdfsPath << endl;

        GDALAllRegister();
        string gdalPath = "/vsihdfs/" + hdfsPath;

        GDALDataset* ds = (GDALDataset*)GDALOpen(gdalPath.c_str(), GA_ReadOnly);
        int x = ds->GetRasterXSize();
        int y = ds->GetRasterYSize();

        cout << "x：" << x << endl;
        cout << "y：" << y << endl;

        return 0;
}

4.使用Java测试GDAL接口读取tif文件

import org.gdal.gdal.Dataset;
import org.gdal.gdal.gdal;
import org.gdal.gdalconst.gdalconst;
import java.io.IOException;

class MyMain {

    public static void main(String[] args) throws IOException {
        String hdfsPath = "hdfs://192.168.80.132:9000/wuhao/test/result_0.tiff";
        System.out.println("hdfsPath:" + hdfsPath);

        // 使用 GDAL 访问文件
        gdal.AllRegister();
        String gdalPath = "/vsihdfs/" + hdfsPath;
        System.out.println("gdalPath:" + gdalPath);

        Dataset ds = gdal.Open(gdalPath, gdalconst.GA_ReadOnly);
        int x = ds.GetRasterXSize();
        int y = ds.getRasterYSize();

        System.out.println("x：" + x);
        System.out.println("y：" + y);
    }
}

编译并运行，需要配置hadoop/etc/core-site.xml文件或者把该文件放到java统一目录运行

使用i打包时，在pom文件中配置依赖同时要确保core-site.xml打包进jar中

<dependencies>
        <dependency>
            <groupId>org.gdal</groupId>
            <artifactId>gdal</artifactId>
            <version>2.4.4</version>
<!--            <scope>system</scope>-->
<!--            <systemPath>${pom.basedir}/lib/gdal-2.4.4.jar</systemPath>-->
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

    </dependencies>