Datanodes come and go.
Proper removal of a Datanode is important, otherwise you might end up with missing blocks or unconsolidated Ambari meta database.
In this post, I am removing 2 Datanodes from my cluster. Before doing this, I have to know the replication factor (3 in my case) and the number of Datanodes left after the decommission. This is important because of the following rule:
If replication factor is higher than the number of existing datanodes after the removal, the removal process is not going to succeed!
In my case, I have 5 datanodes in the cluster, so I am going to be left with 3 Datanodes after removing 2.
Following procedure is needed to remove Datanode(s) properly:
- From the Hosts list, check the Datanodes you wish to remove.
- Decommission Nodemanagers
- Decommision Datanodes
- Stop Ambari Metrics on each Datanode
- While on the same page, stop all components
Repeat this for every Datanode.Click OK on the confirmation window.
Background Operations window informs you when the components are stopped.
- Stop the Ambari Agent.
Log in to each to-be-removed Datanode and stop the Ambari Agent.
sudo ambari-agent stop
- Delete Host
While on the same page, from the Host Actions menu, choose Delete Host
Repeat this for every Datanode.
- Restart HDFS and YARN services from Ambari.
The soon to be removed Datanode(s) are in decommissioned status and will remain in it until HDFS service is restarted. Ambari reminds you to restart HDFS and YARN.
The unwanted Datanodes have now been removed. I have a cluster with 3 Datanodes after removing 2.