Removing Datanode(s) from a cluster with Ambari

Datanodes come and go.
Proper removal of a Datanode is important, otherwise you might end up with missing blocks or unconsolidated Ambari meta database.
In this post, I am removing 2 Datanodes from my cluster. Before doing this, I have to know the replication factor (3 in my case) and the number of Datanodes left after the decommission. This is important because of the following rule:

If replication factor is higher than the number of existing datanodes after the removal, the removal process is not going to succeed!

In my case, I have 5 datanodes in the cluster, so I am going to be left with 3 Datanodes after removing 2.

Following procedure is needed to remove Datanode(s) properly:

  1. From the Hosts list, check the Datanodes you wish to remove.
  2. Decommission Nodemanagers
    decomission nodemanager
  3. Decommision Datanodesdecommission datanode
  4. Stop Ambari Metrics on each Datanode
    ambari metrics stop
  5. While on the same page, stop all components
    stop all componentsRepeat this for every Datanode.Click OK on the confirmation window.
    stop all components confirmation
    Background Operations window informs you when the components are stopped.
    status report
  6. Stop the Ambari Agent.
    Log in to each to-be-removed Datanode and stop the Ambari Agent.

    sudo ambari-agent stop
  7. Delete Host
    While on the same page, from the Host Actions menu, choose Delete Host
    delete host
    Repeat this for every Datanode.
  8. Restart HDFS and YARN services from Ambari.
    The soon to be removed Datanode(s) are in decommissioned status and will remain in it until HDFS service is restarted. Ambari reminds you to restart HDFS and YARN.HDFS status

The unwanted Datanodes have now been removed. I have a cluster with 3 Datanodes after removing 2.

Advertisements

2 thoughts on “Removing Datanode(s) from a cluster with Ambari

  1. In case datanode is on the same machine as primary node, should I avoid the step 7 Delete host ?
    Many thanks in advance

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s