Thursday, March 14, 2013

Using C++ or C to interact with hadoop

Are you a C++ or c programmer and you are not willing to write java code to interact with Hadoop/HDFS ha? Ok you have an option that is : llbhdfs native library that enables you to write programs in c or cpp to interact with Hadoop.

Current Hadoop distributions contain the pre-compiled libhdfs libraries for 32-bit and 64-bit Linux operating systems. You may have to download the Hadoop standard distribution and compile the libhdfs library from the source code, if your operating system is not compatible with the pre-compiled libraries.

For more information read following:

http://wiki.apache.org/hadoop/MountableHDFS

https://ccp.cloudera.com/display/CDHDOC/Mountable+HDFS

http://xmodulo.com/2012/06/how-to-mount-hdfs-using-fuse.html

Writing code in cpp or c Follows:

#include "hdfs.h"
int main(intargc, char **argv) {
hdfsFS fs =hdfsConnect( "NAMENODE_HOSTNAME,PORT);

if (!fs) {
fprintf(stderr, "
Cannot connect to HDFS.\n");
exit(-1);
}
char* readFile = "
cppcexample.file";
char* message = "
HDFS C API!!!";
int size = strlen(message);
int exists = hdfsExists(fs, readFile);
if (exists > -1) {
fprintf(stdout, "
File %s exists!\n", readFile);
}else{
// Create and open file for writing
hdfsFile outFile = hdfsOpenFile(fs, readFile, O_WRONLY|O_CREAT,
0, 0, 0);
if (!outFile) {
fprintf(stderr, "
Failed to open %s for writing!\n", readFile);
exit(-2);
}
// write to file
hdfsWrite(fs, outFile, (void*)message, size);
hdfsCloseFile(fs, outFile);
}
// Open file for reading
hdfsFile inFile = hdfsOpenFile(fs, readFile, O_RDONLY, 0, 0, 0);
if (!inFile) {
fprintf(stderr, "
Failed to open %s for reading!\n", readFile);
exit(-2);
}
char* data = malloc(sizeof(char) * size);
// Read from file.
tSize readSize = hdfsRead(fs, inFile, (void*)data, size);
fprintf(stdout, "
%s\n", data);
free(data);
hdfsCloseFile(fs, inFile);
hdfsDisconnect(fs);
return 0;
}

No comments:

Post a Comment

Live

Your Ad Here