一 LIB HDFS接口简介
hadoop filesystem apis是JAVA CLIENT API,HDFS并没有提供原生的c语言访问接口。但HDFS提供了基于C调用接口LIBHDFS,为C语言访问HDFS提供了很大的便利
头文件:hdfs.h --- ${HADOOP_HOME}/include
库文件:libhdfs.so --{HADOOP_HOME}/lib/native
二 程序执行
以apache hadoop官网示例代码为例:
#include "hdfs.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char **argv)
{
hdfsFS fs = hdfsConnect("hostname", 9000);
const char* writePath = "/maxf/testfile1.txt";
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
if(!writeFile)
{
fprintf(stderr, "Failed to open %s for writing!\n", writePath);
exit(-1);
}
char* buffer = "Hello, World!\n";
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
if (hdfsFlush(fs, writeFile))
{
fprintf(stderr, "Failed to 'flush' %s\n", writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
return 0;
}
编译程序:gcc hdfs.c -o c_hdfs -I /bata/lzp/hadoop-2.7.1/include -L /bata/lzp/hadoop-2.7.1/lib/native -lhdfs
出现错误:
/usr/bin/ld: warning: libjvm.so, needed by /home/hadoop/hadoop-2.7.1/lib/native/libhdfs.so, not found (try using -rpath or -rpath-link)
/home/hadoop/hadoop-2.7.1/lib/native/libhdfs.so: undefined reference to `JNI_CreateJavaVM@SUNWprivate_1.1'
/home/hadoop/hadoop-2.7.1/lib/native/libhdfs.so: undefined reference to `JNI_GetCreatedJavaVMs@SUNWprivate_1.1'
collect2: ld returned 1 exit status
1.在/etc/ld.so.conf中添加libhdfs.so库文件路径,libjvm.so库文件路径
2.可通过ldconfig -v | grep hdfs查看设置的库文件路径是否生效
设置完成之后编译程序可以正常通过
三 代码执行
1.缺失commons-lang-2.6.jar
2.缺失commons-configuration-1.6.jar,hadoop-common-2.7.1.jar,commons-logging-1.2.jar三个包
3.缺失guava-12.0.1.jar包
4.缺失commons-collections-3.2.2.jar,slf4j-api-1.6.1.jar,slf4j-log4j12-1.6.1.jar,log4j-1.2.17.jar,hadoop-hdfs-2.7.1.jar,hadoop-auth-2.5.1.jar包
5.缺失htrace-core-3.1.0-incubating.jar
还有好多包需要进行配置,丢了会有各种各样的NoClassDefFoundError错误,在这里就不一一列出了;
可参考以下配置文件进行配置:
export CLASSPATH=.:$HADOOP_HOME/lib/commons-lang-2.6.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/commons-configuration-1.6.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/commons-logging-1.2.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/guava-12.0.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/commons-collections-3.2.2.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/slf4j-api-1.6.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/slf4j-log4j12-1.6.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/common/lib/log4j-1.2.17.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/hadoop-auth-2.5.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/htrace-core-3.1.0-incubating.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/commons-cli-1.2.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/protobuf-java-2.5.0.jar:$CLASSPATH
export CLASSPATH=.:$HADOOP_HOME/lib/commons-io-2.4.jar:$CLASSPATH
export CLASSPATH=.:/bata/lzp/maxf/apache-drill-1.12.0/jars/classb/javax.servlet-api-3.1.0.jar::$CLASSPATH
配置完成后即可运行c程序的bin文件
查看hdfs中生成的文件