Installing Flume on Hortonworks cluster using Ambari

Add Flume in Ambari

  1. Click on Aded Service from the Ambari interface.
    flume-add service
  2. Flume service available in HDP is 1.5.2. Choose this service to be installed.
    flume-available version
  3. Pick where to install the Flume service. In this case, Flume is added to the namenode. The services can be moved to another node by using Ambari.
    flume-choose node
  4. In step Customize Services, Flume agent can be configured. This can be done after the service is installed. For now, let it be empty.
    flume-agent config
  5. In step Review, click on Deploy
    flume-deploy
  6. After the install, the service is started and tested. If everything goes well, the green progress bar shows up
    flume-install start and test
  7. The summary warns you that some services would have to be restarted so that Flume can function properly. This is a generic message. In case of installing only Flume, no restart of existing services is needed.
    flume-summary

Work in Linux

  1. User flume is added automatically by Ambari and it belongs to group hadoop.
    flume-linux group

Work in HDFS

  1. In order for user flume to work properly on HDFS, flume folder has to be created under /user in HDFS. For example, in case of deleting files in HDFS as user flume, deleted files are moved to /user/flume.Create /user/flume in HDFS.
    sudo -u hdfs hadoop fs -mkdir /user/flume

    Give ownership to user flume.

    sudo -u hdfs hadoop fs -chown flume /user/flume

    Give read, write and execute to flume and flume’s HDFS group – hdfs.

    sudo -u flume hadoop fs -chmod 770 /user/flume

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s