List of steps of datanode failure.

• Namenode marks Datanodes without recent heartbeat as dead
• Does not forward any new I/O requests
• Constantly tracks which blocks must be replicated with BlockMap
• Initiates replication when necessary

Please write steps of checkpointing in hadoop.

• Performed by Namenode
• Two versions of FsImage
• One stored on disk
• One in memory
• Applies all transactions in EditLog to in-memory FsImage
• Flushes FsImage to disk
• Truncates EditLog

Note : Currently only occurs on start-up

What is the namenode startup process steps?

• Namenode enters Safemode
• Replication does not occur in Safemode
• Each Datanode sends Heartbeat 
• Each Datanode sends Blockreport
• Lists all HDFS data blocks
• Namenode creates Blockmap from Blockreports
• Namenode exits Safemode
• Replicate any under-replicated blocks 

What makes HDFS unavailable ?

Ans: Failure of datanodes

What is one of the underacted problem which may occur with map reduce submission to Hadoop.

Due to some condition of bad code it goes into infinity loop.

What the functions of a scheduling algorithm?

• Reduce the total amount of computation necessary to complete a job
• Allow multiple users to share clusters in a predictable, policy-guided manner.
• Run jobs at periodic times of the day.
• Reduce job latencies in an environment with multiple jobs of different sizes.

Write file to HDFS/Hadoop Read File From HDFS/Hadoop Using Java

 

import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DistributedFileSystem;

/**
 *
 * @author    Shashwat Shriparv
 * @email     dwivedishashwat@gmail.com
 * @Web       helpmetocode.blogspot.com
 */
public class WritetoHDFSReadFromHDFSWritToLocal {
    private static byte[] buffer;
    private static int bytesRead;

    public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
        
    FileSystem fs =new DistributedFileSystem();

fs.initialize(new URI("hdfs://master1:9000/"), new Configuration());

final File folder = new File("C:\\Shared\\files");
  for (final File fileEntry : folder.listFiles()) {
        if (fileEntry.isDirectory()) {
            readallfilefromfolder(fileEntry);
        } else {
            fs.copyFromLocalFile(new Path("C:\\shashwat\\files"+fileEntry.getName()),new Path("/Test/"));
            System.out.println(fileEntry.getName());
            fs.copyToLocalFile(new Path("/Test/"+fileEntry.getName()),new Path("d:\\shashwat\\"));
        }
    }

//fs.copyFromLocalFile(new Path("C:\\Shared\\HadoopLibs"),new Path("/Test/1.jpg"));

System.out.println("Done");
}
    public static void readallfilefromfolder(final File folder) {
    for (final File fileEntry : folder.listFiles()) {
        if (fileEntry.isDirectory()) {
            readallfilefromfolder(fileEntry);
        } else {
            System.out.println(fileEntry.getName());
        }
    }
}
}

Note : While writing to hdfs create a directory and change its permission to 777 to avoid security related exception.

WebHDFS REST API

The HTTP REST API supports the complete FileSystem interface for HDFS.

Operations

• HTTP GET
  • OPEN (see FileSystem.open)
  • GETFILESTATUS (see FileSystem.getFileStatus)
  • LISTSTATUS (see FileSystem.listStatus)
  • GETCONTENTSUMMARY (see FileSystem.getContentSummary)
  • GETFILECHECKSUM (see FileSystem.getFileChecksum)
  • GETHOMEDIRECTORY (see FileSystem.getHomeDirectory)
  • GETDELEGATIONTOKEN (see FileSystem.getDelegationToken)
• HTTP PUT
  • CREATE (see FileSystem.create)
  • MKDIRS (see FileSystem.mkdirs)
  • RENAME (see FileSystem.rename)
  • SETREPLICATION (see FileSystem.setReplication)
  • SETOWNER (see FileSystem.setOwner)
  • SETPERMISSION (see FileSystem.setPermission)
  • SETTIMES (see FileSystem.setTimes)
  • RENEWDELEGATIONTOKEN (see DistributedFileSystem.renewDelegationToken)
  • CANCELDELEGATIONTOKEN (see DistributedFileSystem.cancelDelegationToken)
• HTTP POST
  • APPEND (see FileSystem.append)
• HTTP DELETE
  • DELETE (see FileSystem.delete)

For More Please Visit WebHDFS

Jobtracker API error - Call to localhost/127.0.0.1:50030 failed on local exception: java.io.EOFException

Try the port number listed in your $HADOOP_HOME/conf/mapred-site.xml under the mapred.job.tracker property. Here's my pseudo mapred-site.xml conf

<property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
</property>

If you look at the JobTracker.getAddress(Configuration) method, you can see it uses this property if you don't explicitly specify the jobtracker host / port:

public static InetSocketAddress getAddress(Configuration conf) {
  String jobTrackerStr =
    conf.get("mapred.job.tracker", "localhost:8012");
  return NetUtils.createSocketAddr(jobTrackerStr);
}

Adding Scheduler to Hadoop Cluster

 

As we know we we execute task or jobs on hadoop it follows FIFO Scheduling, but if you are in multi user hadoop environment the you will need better scheduler for the consistency and correctness of the task scheduling.

Hadoop comes with other schedulers too those are:

Fair Schedulers : This defines pools and over time; each pool gets around the same amount of resources.

Capacity Schedulers : This defines queues, and each queue has a guaranteed capacity. The capacity scheduler shares computer resources allocated to a queue with other queues if those resources are not in use.

For changing the scheduler you need to take your cluster offline and make some configuration changes, first make sure that the correct scheduler jar files are there. In older version of hadoop you need to put the jar file if not ther in lib directory but from hadoop 1 these jars available in the lib folder and if you are using the newer hadoop good news for you

Steps will be: