Using C++ or C to interact with hadoop

Are you a C++ or c programmer and you are not willing to write java code to interact with Hadoop/HDFS ha? Ok you have an option that is : llbhdfs native library that enables you to write programs in c or cpp to interact with Hadoop.

Current Hadoop distributions contain the pre-compiled libhdfs libraries for 32-bit and 64-bit Linux operating systems. You may have to download the Hadoop standard distribution and compile the libhdfs library from the source code, if your operating system is not compatible with the pre-compiled libraries.

For more information read following:

http://wiki.apache.org/hadoop/MountableHDFS

https://ccp.cloudera.com/display/CDHDOC/Mountable+HDFS

http://xmodulo.com/2012/06/how-to-mount-hdfs-using-fuse.html

Writing code in cpp or c Follows:

Finding out block location and block size of file on HDFS

 

Have you ever needed to find out the Block location and Block size for a file which is lying on hdfs hadoop? if so here is the command you can use to find out that.

For that we need “fsck” command which hadoop provide.

Here goes the command:

bin/hadoop fsck /filepath/filenameonhdfs –block –files  -location

This command will provide information about Block location on what data node its lying, what are the blocks for that file

Just go and play with the command you will understand more.

Phoenix: A SQL layer over HBase 'We put the SQL back in the NoSQL'

 

Phoenix is a SQL layer over HBase, delivered as a client-embedded JDBC driver, powering the HBase use cases at Salesforce.com. Phoenix targets low-latency queries (milliseconds), as opposed to batch operation via map/reduce. To see what's supported, go to our language reference guide, and read more on our wiki.

FAILED FSError: java.io.IOException: No Space left on device

 

FAILED FSError: java.io.IOException: No Space left on device

org.apache.hadoop.util.DiskChecker$DiskErrorException : Could not find any valid local directory for taskTracker/jobcache

When you run hive queries and you queries fails, just go the the job tracker of Hadoop : mostly on 50030 port on your Hadoop nodes, and check the failed or killed jobs, you may find these errors.

The reason behind will be following:

Hadoop Interview Questions

What are the default configuration files that are used in Hadoop

As of 0.20 release, Hadoop supported the following read-only default configurations - src/core/core-default.xml - src/hdfs/hdfs-default.xml - src/mapred/mapred-default.xml

How will you make changes to the default configuration files

Hadoop does not recommends changing the default configuration files, instead it recommends making all site specific changes in the following files - conf/core-site.xml - conf/hdfs-site.xml - conf/mapred-site.xml Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath: - core-default.xml : Read-only defaults for hadoop. - core-site.xml: Site-specific configuration for a given hadoop installation. Hence if same configuration is defined in file core-default.xml and src/core/core-default.xml then the values in file core-default.xml (same is true for other 2 file pairs) is used.

Hadoop Interview Question

 Here are Some Hadoop Administration question you May expect.  answers you need to find.... :) i can give but i wont : if you find good answer share with me also :) hope you will right ? If you are not able to find let me know through comments i will post the answers too.

• What is Hadoop? Brief about the components of  Hadoop.

• What are the Hadoop daemon processes tell the components of Hadoop and functionality?

• Tell steps for configuring Hadoop?
• What is architecture of HDFS and flow?

• Can we have more than one configuration setting for Hadoop cluster how can you switch between these configurations?

• What will be your troubleshooting approach in Hadoop?
• What are the exceptions you have come through while working on Hadoop, what was your approach for getting rid of those exceptions or errors?

100 C Questions and Answers

(Q :  1)

Code:

int z,x=5,y=-10,a=4,b=2; 

z = x++ - --y * b / a;

What number will z in the sample code above contain? 

Option 1

5 

Option 2

6 

Option 3

10 [Ans] Corrected by buddy by running the program

Option 4

11 

Option 5

12 

____________________________________________________________

(Q :  2)

With every use of a memory allocation function, what function should be used to release allocated memory which is no longer needed? 

Demystifying Hadoop concepts Series: Safe mode

 

What is is safe mode of hadoop, may time we come across this exception “ org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException”  or some other exceptions Which contains safe mode in it .

 

First let me tell what Safe mode is in context to Hadoop : as we all know Name node contains fsimage (metadata) of the data present on the cluster, which can be large or small based on the size of the cluster and the size of date present on the cluster, so when the name node starts it loads this fsimage and the edit logs from the disk in the Primary memory RAM for fast processing, and after loading it waits for data nodes to report about the present on those data nodes, so during this process that is loading the fsimage and edit logs and waiting for data nodes to report about the data block in safe mode, which is a read only mode for name node this is done to maintain the consistency of the data present, this is just like saying “ i will not receive any thing till i know what i already have”. And during this period no modification to the file blocks are allowed as to maintain the correctness of the data.

 

How long safemode exist :

Generally name node automatically comes out of safe mode in 30 seconds if all data present are consistent according the fsimage and the editlogs.

 

Related commands :

Put Namenode in Safemode: bin/hadoop dfsadmin –safemode

Leave Safemode : bin/hadoop dfsadmin -safemode leave

 

What to do if you encounter this exception :

 

First, wait a minute or two and then retry your command. If you just started your cluster, it's possible that it isn't fully initialized yet. If waiting a few minutes didn't help and you still get a "safe mode" error, check your logs to see if any of your data nodes didn't start correctly (either they have Java exceptions in their logs or they have messages stating that they are unable to contact some other node in your cluster). If this is the case you need to resolve the configuration issue (or possibly pick some new nodes) before you can continue.

 

What do you mean by Object Slicing?

When a derived class object is assigned to a base class, only the base class's part of content in the derived object are copied to the base class, leaving behind the derived class specific contents. This is referred as Object Slicing.

Class BaseClass
{
public int i;
};

class DerivedClass : public BaseClass
{
public int j;
};

int main()
{
BaseClass ObjectOfB;
DerivedClass ObjectOfD;
ObjectOfB = ObjectOfD;
//Here ObjectOfD contains both i and j.
//But only i is copied to ObjectOfB.
}

What is difference between overloading and overriding?

Having same name methods with different parameters is called overloading, while having same name and parameter functions in base and drive class called overriding.