Installing Ambari Infra for enabling Ranger Audit Access

About the key services mentioned in this post:
Apache Solr – an open-source enterprise search platform. Ranger is using it to store audit logs.
Ambari Infra – core shared service used by Ambari managed components. The database is Solr.

Using a database for Audit Access in Ranger is not an option anymore with HDP 2.5. What is being offered now is Solr and HDFS. It is recommended that Ranger audits are written to Solr and HDFS.
Solr takes care of the search queries from th Ranger Web interface, while HDFS is for more persistent  storing of audits.

This was done on an HDP 2.5 cluster on AWS.

Installing Ambari Infra

Even though the HDP’s documentation says Solr should be installed before Ranger, I installed Ranger service first because of my previous Ranger experience when I used MySql for audit logs.

So installing Ambari Infra is really a clicking job. The only thing to check is where the service is going to be installed. I installed it on NameNode. Remember, it is easy to move services from on node to another.

Configuring Ranger with Solr

Click on Ranger and click on Configs -> Ranger Audit. From there Turn on Audit to Solr and SolrCloud.

You should now have enabled both Solr and HDFS for collecting audit logs.

If you now log in to Ranger, you should see audit logs.

If you plan to build an application in Solr, do not use the solr that is intended for Ambari Infra but install Solr.

Very useful documentation on this topic is available here.

Configuring Ranger Plugins in Ambari

In previous post, I described how to install Ranger in Ambari on HDP.

HDFS

Ranger allows (through configuration) both Ranger policies and HDFS permissions to be checked for a user request. When a user request is received in NameNode, Ranger plugin will check for policies set through Ranger admin. If there are no policies, Ranger plugin will check for permission set in HDFS.

It is recommended to have restrictive permission at HDFS level and create permission in Ranger security admin.

Configuring HDFS Plugin happens in two places – HDFS service and Ranger service.

HDFS service

Select HDFS service from the Services menu.

Open Advanced ranger-hdfs-plugin-properties ad check the Enable Ranger for HDFS checkbox.

Change the following property by replacing NAMENODE_HOSTNAME with the RANGER_HOST.

NAMENODE_HOSTNAME

If you are using an older HDP version, check Audit to DB.

audit to db

Change HDFS umask from 022 to 077.

umask 077

Save the properties and restart the service.

The following message appears, click OK to restart HDFS.

dependent configurations

Ranger service

In Ranger, under tab Config

Switch on HDFS Ranger Plugin

hdfs-switch

 

Change the audit source type from default solr to db.

audit source type

Save and restart Ranger service.

Hive

comming soon…

Adding and configuring service Ranger in Ambari

Ranger is a framework to enable, monitor and manage data security in Hadoop cluster. The service comes from Hortonworks and is a part of Apache family now.

This post describes how Ranger 0.5.0 is installed and configured  with audit data stored in a database. Default setting is Solr, my cluster does not have Solr, but it has a MySql database.

My Hadoop distribution is Hortonworks and versions mentioned in this post are 2.3.4 and 2.5.

 

Database preparation

Install MySql

(If not installed yet)

sudo apt-get install mysql-server -y

Set up Ranger database

Note for HDP 2.3.4!
Ranger database has to be created manually otherwise the installation will not go through. If you are using HDP 2.5, this is done through Ambari Add Service Wizard. Move on to “Adding Service in Ambari”.

create database ranger;
CREATE USER 'ranger'@'localhost' IDENTIFIED BY 'ranger';
GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'localhost';
CREATE USER 'ranger'@'%' IDENTIFIED BY 'ranger';
GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'localhost' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%' WITH GRANT OPTION;

If the MySql database is on another server than Ranger, check from RANGER_SERVER if you can log in to the database

mysql -u ranger -pranger -h MYSQL_SERVER

Adding Service in Ambari

Start Add Service Wizard and choose service Ranger

Add service

Some requirements have to be fulfilled.

Ranger Requirements

Check if MySql Java Connector is present on Ambari Server

ls /usr/share/java/mysql-connector-java.jar

Run the following on Ambari Server if the file is present

sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Output:

Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.

Assign masters for both Ranger services. In this case, the services are installed on the NameNode.

Assign masters

Choose DB flavor, tye in ranger DB host and ranger password (same as in the script from the previous chapter)

Wizard - Ranger Admin

Type password for root user and test the connection.

Wizard root password

If the MySql database is on another server, user has to be created and grants for root from Ranger server have to be granted.

CREATE USER 'root'@'RANGER_SERVER' IDENTIFIED BY 'root';
GRANT RELOAD ON *.* TO 'root'@'RANGER_SERVER';
GRANT ALL PRIVILEGES ON mysql.* TO 'root'@'RANGER_SERVER';
GRANT ALL PRIVILEGES ON ranger.* TO 'root'@'RANGER_SERVER' WITH GRANT OPTION;

In the Audit tab:
– switch off Audit to Solr
– switch on Audit to HDFS
– switch on Audit to DB and type in password for Ranger Audit user. (HDP 2.3.4)

HDP 2.5: Audit to DB is not an option anymore.

Wizard - audit storage

Ranger is now installed and can be accessed on the RANGER_SERVER:6080.

Note: the Ranger WEB UI not showing up?
Make sure port 6080 is open.

If the URL is an internal IP address read on:
External URL has to be corrected to ranger host. Authentication in this example is UNIX.

Wizard - ranger url only 2-3-4

Continue to the next step.

Review of the installation follows, if everything is ok, start with the Install, Start and Test.