The Data Nodes are responsible for serving read and write requests from the file system’s clients. Name node basically has the name and path of the data blocks along with the IP address of commodity machine where data blocks reside.įile system operations such as opening, closing, renaming files and directories are executed by name node.

Only the name node in the whole Hadoop eco system has the information about which data blocks resides on what commodity machine. Name node keeps the metadata information about the data blocks residing on commodity machines or slave machines in Hadoop cluster. The pertinent question here is what name node and secondary name node does. It is important to understand that secondary name node is not a backup machine. 2 GB/4GB RAM machines which we call it as commodity machines or slave machines.ĭata blocks which by default is 64 MB chunk resides on commodity machines or slave machines. Name Node is a high-end server.Īlong with master we have one more high-end server which is called secondary name node.Īll other servers are normal machines i.e. HDFS file system has master-slave architecture. To achieve availability of each node where 64 MB chunk is residing, HDFS replicates these 64 MB chunks onto two additional servers by default. To reconstruct the cluster in order to achieve the output as count file, we have to write a program which will go to each node in the cluster and get the aggregated count information which will finally be summed up to get the final output. As soon as the file will arrive at HDFS file system it will break into default 64 MB chunk scattered across the commodity servers. Our goal is to get the number of people whose first name starts with “A”. An example of HDFS file system would be phone number of everyone across the globe stored in a file. The file system also copies each 64 MB chunk to multiple machines depending on the replication factor mentioned in HDFS configuration file. When the data enters into HDFS file system, it breaks down into smaller chunks default 64 MB chunk and then gets distributed across the different nodes in the cluster allowing it for parallel processing. Hadoop Distributed File System (HDFS) is a file system that provides reliable data storage for large data-set in distributed computing environment using cluster of commodity servers.

Hadoop hdfs tutorial