hdfs的C++接口编译测试

 项目中要做一个数据包管理服务,我们主要项目开发都是C++的,所以这个数据包管理也是c++开发的,但是数据包的存储是个问题,最后选择了本地存储和hdfs存储结合。
    昨天调试了以下hdfs的C++接口,简单这里记录一下:
代码程序是网上随处可以搜到的测试代码:

#include <stdlib.h> 
#include <stdio.h> 
#include <string.h> 
#include “hdfs.h” 

int main(int argc, char **argv) { 

hdfsFS fs = hdfsConnect(“127.0.0.1”, 9000); 
const char* writePath = “/home/testfile2.txt”; 
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0); 
if(!writeFile) { 
fprintf(stderr, “Failed to open %s for writing!\n”, writePath); 
exit(-1); 

char* buffer = “Hello, World!”; 
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1); 
if (hdfsFlush(fs, writeFile)) { 
fprintf(stderr, “Failed to ‘flush’ %s\n”, writePath); 
exit(-1); 

hdfsCloseFile(fs, writeFile);

return 0;
}

编译我这边是使用scons来编译的,脚本如下:

env = Environment() # Initialize the environment
env.Append(CCFLAGS = [‘-g’,’-O3′])
env.Append(LIBS = [‘hdfs’])
env.Append(LIBPATH = [‘/data/hadoop/hadoop/lib/native/’])
env.Append(CPPPATH = [‘/data/hadoop/hadoop/include/’])

env.Program(
target = ‘test’,
source = [‘test.cc’],
)

直接编译即可。

但是运行时需要设置一些其他的环境变量才可以用,应为使用到了jvm的库,但是jvm的库只有so库,目前没有找到如何编译链接静态库的方法,理论上应该可以编译链接静态库的。如果谁知道了,请告诉我一下。

下面是环境变量设置方式,直接添加在 ~/.bashrc文件种即可

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/
export JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/data/hadoop/hadoop/

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$STORM_HOME/bin
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native/

jar_list=`find $HADOOP_HOME/ -name *.jar`
for jar_item in $jar_list
do
CLASSPATH=$CLASSPATH:$jar_item
done
export CLASSPATH

这是运行之后创建的文件:

helight:hadoop$ hdfs dfs -ls /home
Found 7 items
-rw-r–r– 1 helight supergroup 6995 2014-03-24 19:12 /home/dbconn.cc
-rw-r–r– 1 helight supergroup 133716 2014-03-24 18:59 /home/mozilla.pdf
drwxr-xr-x – helight supergroup 0 2014-03-28 09:53 /home/test
-rw-r–r– 3 helight supergroup 14 2014-05-09 09:55 /home/testfile.txt
-rw-r–r– 3 helight supergroup 14 2014-05-09 10:26 /home/testfile1.txt
-rw-r–r– 3 helight supergroup 14 2014-05-09 09:56 /home/testfile2.txt
-rw-r–r– 3 helight supergroup 14 2014-05-09 10:28 /home/testfile3.txt
helight:hadoop$

Leave a Reply

Your email address will not be published. Required fields are marked *