Upgrading Ambari 2.4 to 2.5

This post describes how an upgrade from Ambari 2.4.1.0 to 2.5 has been done. The reason for that is to be able to further upgrade HDP to 2.6. Upgrade of HDP from 2.5 to 2.6 is described here.

Ambari Server is installed on Ubuntu 14.04. The same OS is used across the whole HDP cluster.

The following services are upgraded using this blog post:

  • Ambari Server
  • Ambari Agent
  • Ambari Infra
  • Ambari Metrics
  • Ambari Collector
  • Grafana

Backup

It is important to do a database backup of the Ambari database. Metadata for my Ambari is stored in MySql database.

Create a directory for backup

mkdir /home/ubuntu/ambari24-backup

Backup the database (enter password when prompted)

mysqldump -u ambari -p ambari_db > /home/ubuntu/ambari24-backup/ambari.mysql

Make a safe copy of the Ambari Server configuration file

sudo cp /etc/ambari-server/conf/ambari.properties ambari24-backup/

Prepare for installation of Ambari Agent and Server

Stop Ambari Metrics from the Ambari Web UI

Stop Ambari Server on Ambari Server instance

sudo ambari-server stop

Stop all Ambari Agents on all instances in the cluster where it is running

sudo ambari-agent stop

On all instances running Ambari Server or Ambari Agent do the following

sudo mv /etc/apt/sources.list.d/ambari.list db-backups/
sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.0.3/ambari.list -O /etc/apt/sources.list.d/ambari.list

Upgrade Ambari Server

sudo apt-get clean all
sudo apt-get update -y
sudo apt-cache show ambari-server | grep Version

The last command should output something like this

Version: 2.5.0.3-7
Version: 2.4.1.0-22

This means version 2.5 is available, Ambari Server can be installed

Install Ambari Server

sudo apt-get install ambari-server

Some lines from the output

The following packages will be upgraded:
  ambari-server

Unpacking ambari-server (2.5.0.3-7) over (2.4.1.0-22) ...

Setting up ambari-server (2.5.0.3-7) ...

Confirm that there is only one ambari server jar file

ll /usr/lib/ambari-server/ambari-server*jar

Output

-rw-r--r-- 1 root root 5806966 Apr  2 23:33 /usr/lib/ambari-server/ambari-server-2.5.0.3.7.jar

Install Ambari Agent

On each host running Ambari agent

sudo apt-get update
sudo apt-get install ambari-agent

Check if the Ambari agent install was a success

dpkg -l ambari-agent

Output from one node

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                         Version        Architecture    Description
+++-============================-==============-===============-=================
ii  ambari-agent                 2.5.0.3-7      amd64           Ambari Agent

Upgrade Ambari DB schema

On Ambari Server instance, run the following command

sudo ambari-server upgrade

The following question shows up. The backup has been done at the beginning. Type y and press Enter.

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (y)?

Output

INFO: Upgrading database schema
INFO: Return code from schema upgrade command, retcode = 0
INFO: Schema upgrade completed
Adjusting ambari-server permissions and ownership...
Ambari Server 'upgrade' completed successfully.

Start the services

Start Ambari Server

sudo ambari-server start

Start Ambari Agent on all instances where it is installed

sudo ambari-agent start

Post-installation tasks

Hive and Oozie (which I have installed in HDP) are using MySql, so I have to put the jar file in place

sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Ambari Upgrade 1: Upgrading Hortonworks Ambari from 2.1 to 2.2

Introduction

This post explains how to upgrade from Ambari 2.1 to either version 2.2.1.1 or 2.2.2.0.

Im using an external database MySql as Ambari database. My operating system is Ubuntu 14.04 Trusty. Hive service is using external database – MySql (important information for later).

The cluster does NOT have the following services installed:

  • Ranger
  • Storm
  • Ganglia
  • Nagios

The upgraded cluster does not use LDAP, nor Active Directory.

If you have any of the above mentioned services, check this link to learn how to handle them in the upgrade process.

Backup

The following steps are done on the Ambari server, unless explicity mentioned otherwise.

  1. Create a folder for backup files on all nodes in the cluster.
    mkdir /home/ubuntu/ambari-backup
  2. Backup the Ambari MySql database.
    DAT=`date +%Y%m%d_%H%M%S`
    mysqldump -u root -proot ambari_db > /home/ubuntu/ambari-backup/ambari_db_$DAT.sql
  3. Backup the ambari.properties file.
    sudo cp /etc/ambari-server/conf/ambari.properties /home/ubuntu/ambari-backup

Upgrade

  1. Make sure you have Java 1.7+ on the Ambari server.
  2. Stop Ambari Metrics from the Ambari web UI.
  3. Stop Ambari server
    sudo ambari-server stop
  4. Stop all Ambari agents on all nodes.
    sudo ambari-agent stop
  5. Remove old repository file ambari.list from all nodes. Different Linux flavours might have different file name check here, page 6.
    sudo mv /etc/apt/sources.list.d/ambari.list /home/ubuntu/ambari-backup
  6. Download new repository file on all nodes.
    Ambari 2.2.1 for Ubuntu 14:

    sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.1/ambari.list -O /etc/apt/sources.list.d/ambari.list

    Ambari 2.2.2 for Ubuntu 14:

    sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.2.0/ambari.list -O /etc/apt/sources.list.d/ambari.list
  7. Update Ubuntu packages and check version.
    sudo apt-get clean all
    sudo apt-get update
    sudo apt-cache show ambari-server | grep Version

    If you are installing to 2.2.1, you should see the following output:
    Version: 2.2.1.1-70
    If you are installing to 2.2.2, you should see the following output:
    Version: 2.2.2.0-460

  8. Install Ambari server on the node dedicated for Ambari server.
    sudo apt-get install ambari-server

    Confirm that there is only one ambari-server*.jar file in /usr/lib/ambari-server.
    Jar files related to upgrade 2.2.1:

    ambari-metrics-common-2.2.1.1.70.jar
    ambari-server-2.2.1.1.70.jar
    ambari-views-2.2.1.1.70.jar

    Jar files related to upgrade 2.2.2:

    ambari-metrics-common-2.2.2.0.460.jar
    ambari-server-2.2.2.0.460.jar
    ambari-views-2.2.2.0.460.jar

  9. Install Ambari agents on all nodes in the cluster.
    sudo apt-get update -y && sudo apt-get install ambari-agent
  10. Upgrade Ambari database.
    sudo ambari-server upgrade
  11. The following question show up: “Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (y)?”
    Press “y”, since that was done in the backup process.When the installation is completed, the following output concludes the installation process:

    Ambari Server ‘upgrade’ completed successfully.

  12. Start Ambari server.
    sudo ambari-server start
  13. On all nodes where Ambari agent is installed, start the agent.
    sudo ambari-agent start
  14. Hive in the cluster is using external database – MySql, so this step is mandatory. Reinstall mysql connector file
    sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
  15. Log in to the upgraded Ambari (same URL, same port, same username and password)
  16. Restart all services in Ambari

Ambari Metrics upgrade

  1. Stop all Ambari Metrics services in Ambari.
  2. On every node in the cluster, where Metrics Monitor is installed, execute the following commands.
    sudo apt-get clean all
    sudo apt-get update
    sudo apt-get install ambari-metrics-assembly
  3. On every node in the cluster, where Metrics Collector is installed, execute the following commands (yes, the command is the same as in previous step).
    sudo apt-get install ambari-metrics-assembly
  4. Start Ambari Metrics services in Ambari.

Warning!

After the upgrade, it is possible to run into the following message when accessing Ambari Web UI.

Ambari post upgrade message in browser

Ctrl+Shift+R solves the problem. The text in the message is quite descriptive and explains why this message is showing.

 

Next step is installing Grafana. This is covered in post Ambari Upgrade 2.

Additional links

The link takes you to the Hortonworks Ambari 2.2.1.1 upgrade document.
The link takes you to the Hortonworks Ambari 2.2.2.0 upgrade document.