In earlier posts, I describe how to build and configure Zeppelin 0.6.0 and Zeppelin-With-R. Here is how the interpreters are configured.

The following interpreters are mentioned in this post:

• Spark
• Hive

Spark interpreter configuration in this post has been tested and works on the following Apache Spark versions:

• 1.5.0
• 1.5.2
• 1.6.0

Basic Spark

Once the page loads, click on Interpreter tab.

Edit spark interpreter

Change the master parameter

master yarn-client

Add 2 properties

spark.driver.extraJavaOptions -Dhdp.version=2.3.4.0-3485
spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.4.0-3485

The bare minimum is now reached. How to get control over the Spark ports is described here.

Hive

Edit default.url parameter under Hive interpreter

default.url jdbc:hive2://hiveserver2:10000

Add 3 properties to hive interpreter

hive.hiveserver2.password hive
hive.hiveserver2.url jdbc:hive2://hiveserver2:10000
hive.hiveserver2.user hive

Testing Zeppelin

Create new Notebook and test it.

Scala

SparkSQL

Hive

PySpark

Application Programming Interface

REpresentational State Transfer

REST is a “software architectural style” of the WWW. REST API communicates between a client and a server through HTTP (most common, but not always!) by using the same verbs (GET, POST, PUT, DELETE) that web browsers use to retrieve web pages and send data (Wikipedia).

The results can be returned in many different file formats, where JSON is the most common one. The following examples return the results in JSON format.

Facebook example

Let us see how Facebook REST API (Facebook Graph API) works:

In a browser, the Facebook page of The University of St. Gallen loads as usual:

https://www.facebook.com/HSGUniStGallen

If “www” is replaced by “graph” and the following URL is loaded:

https://graph.facebook.com/HSGUniStGallen

The first result in a JSON format is returned:

{
  "error": {
    "message": "An access token is required to request this resource.",
    "type": "OAuthException",
    "code": 104,
    "fbtrace_id": "GhoWNOCPeFx"
  }
}

The reason for that is sometime at the beginning of 2015, Facebook has made its policies more strict whne it comes to who can access what. Now, an access token is needed. Running the following from the browser should return a JSON file with some data.

https://graph.facebook.com/HSGUniStGallen?access_token=ose0cBAMc4OZCs4w508tpKbu0ikeTNrZAyBY4Ipkj2phzXkq5cf2VZAngySd9lq5KWIq5sl1ZBo0iaIIvpTiyXcnM1YavhzqEh8cYnVPxn3ZC9y1ZB6xAgJYZCNiFfLFobOZAi0yAmyPEXoSIRXAF2yQluoZBbDB0C6e4yhgHg1CC9RwkoLQbaIJXypVdFylTbANfV1bgZDZD

Access token is invalid. Try with your own.

The result is the following:

{
“id”: “118336184931317”,
“about”: “Herzlich willkommen auf der offiziellen Facebook-Seite der Universit\u00e4t St.Gallen (HSG). // Welcome to the official University of St.Gallen (HSG) Facebook page.”,
“can_post”: false,
“category”: “University”,
“category_list”: [
{
“id”: “108051929285833”,
“name”: “College & University”
},
{
“id”: “216161991738736”,
“name”: “Campus Building”
}
],
“checkins”: 8596,
“cover”: {
“cover_id”: “233550823409852”,
“offset_x”: 0,
“offset_y”: 0,
“source”: “https://scontent.xx.fbcdn.net/v/t1.0-9/s720x720/404165_233550823409852_1060305907_n.jpg?oh=a6150b98691c08457fe200c9f0ad8765&oe=57AC6C2C”,
“id”: “233550823409852”
},
“description”: “Hier informieren wir Sie \u00fcber Neuigkeiten aus der Universit\u00e4t sowie aus den HSG-Themenhubs START, CAMPUS, PROFESSIONAL und RESEARCH. Maria Schmeiser \u003Cms> schreibt hier f\u00fcr Sie.\n\nHSG START bietet Informationen f\u00fcr Studieninteressierte u. a. \u00fcber Themen wie Informationstage, Messen, Zulassung und Anmeldung und Weiterbildung. https://www.facebook.com/HSGStart\n\nHSG CAMPUS b\u00fcndelt Informationen zu Studierenden-Themen aus der Studentenschaft, Bibliothek, Sport und Karrierethemen. Dar\u00fcberhinaus informiert HSG CAMPUS \u00fcber das Campus-Leben in St.Gallen. https://www.facebook.com/HSGCampus\n\nHSG RESEARCH b\u00fcndelt Informationen aus der Universit\u00e4t f\u00fcr Forschende und Forschungsinteressierte. https://www.facebook.com/HSGResearch\n\nHSG PROFESSIONAL informiert \u00fcber \u00d6ffentlichkeits-, Medien- und Alumni-Themen sowie Stellenangebote der Universit\u00e4t St.Gallen (HSG). https://www.facebook.com/HSGProfessional\n\n____________________________________\n\nHere you\u2019ll find information about the university as well our specialist hubs: START, CAMPUS, PROFESSIONAL and RESEARCH. Maria Schmeiser \u003Cms> writes for you here.\n\nHSG START gives you a head start on subjects such as open days, fairs, entry requirement and our courses if you’re interested in studying at the University of St.Gallen (HSG). https://www.facebook.com/HSGStart\n\nHSG CAMPUS curates information from around the University, such as the student union, the library, sports and careers, as well as documenting life at the University of St.Gallen (HSG). https://www.facebook.com/HSGCampus\n\nHSG RESEARCH curates information from around the University concerning research projects. https://www.facebook.com/HSGResearch\n\nHSG PROFESSIONAL curates information from around the University such as media, press and Alumni as well as informing you about current career opportunities at the University. https://www.facebook.com/HSGProfessional”,
“founded”: “1898”,
“general_info”: “Die Universit\u00e4t St.Gallen (HSG) wurde 1898 \u2013 in der Hochbl\u00fcte der St.Galler Stickereiindustrie \u2013 als Handelsakademie gegr\u00fcndet und ist heute eine Hochschule f\u00fcr Wirtschafts-, Rechts- und Sozialwissenschaften sowie Internationale Beziehungen. 1899 fanden die ersten Vorlesungen statt. Praxisn\u00e4he und eine integrative Sicht zeichnen unsere Ausbildung seit jenen Gr\u00fcndungstagen aus. Wir geh\u00f6ren wir zu den f\u00fchrenden Wirtschaftsuniversit\u00e4ten in Europa und sind EQUIS- und AACSB-akkreditiert.\n\nIm Jahr 2001 haben wir Bachelor- und Master-Studieng\u00e4nge integral eingef\u00fchrt und gleichzeitig unsere Ausbildung tiefgreifend reformiert. Seither steht nicht mehr nur die Fachausbildung im Vordergrund, sondern auch die Pers\u00f6nlichkeitsbildung. Bei uns k\u00f6nnen Abschl\u00fcsse auf Bachelor-, Master- und Doktorats/ Ph.D.-Stufe erreicht werden. Die enge Vernetzung von Studium, Weiterbildung und Forschung ist uns wichtig.\n\n____________________________________\n\nThe University of St.Gallen (HSG) was founded as a business academy in 1898 \u2013 in the heyday of the St.Gallen embroidery industry and is nowadays a School of Management, Economics, Law, Social Sciences and International Affairs. The first lectures were held in 1899. The practice-oriented approach and integrative view have characterised the education we offer since those early days. Today, we are one of Europe\u2019s leading business schools and are EQUIS and AACSB accredited.\n\nIn 2001, we introduced Bachelor\u2019s and Master\u2019s degree courses across the board and fundamentally reformed the education we offer. Since then, we no longer focus just on professional training but also on character building. At our university, you can complete studies at Bachelor\u2019s, Master\u2019s and Ph.D. level. The close integration of studies, further education and research is important to us.”,
“has_added_app”: false,
“is_community_page”: false,
“is_published”: true,
“likes”: 20791,
“link”: “https://www.facebook.com/HSGUniStGallen/”,
“location”: {
“city”: “Saint Gallen”,
“country”: “Switzerland”,
“latitude”: 47.431544844559,
“longitude”: 9.3751522771065,
“street”: “Dufourstrasse 50”,
“zip”: “9000”
},
“name”: “Universit\u00e4t St.Gallen (HSG)”,
“parking”: {
“lot”: 0,
“street”: 0,
“valet”: 0
},
“phone”: “+41 (0)71 224 21 11”,
“talking_about_count”: 455,
“username”: “HSGUniStGallen”,
“website”: “www.unisg.ch”,
“were_here_count”: 8596
}

Only selected values (name, likes, street and zip) can be fetched:

https://graph.facebook.com/HSGUniStGallen?fields=name,likes,location.street,location.zip&access_token=ose0cBAMc4OZCs4w508tpKbu0ikeTNrZAyBY4Ipkj2phzXkq5cf2VZAngySd9lq5KWIq5sl1ZBo0iaIIvpTiyXcnM1YavhzqEh8cYnVPxn3ZC9y1ZB6xAgJYZCNiFfLFobOZAi0yAmyPEXoSIRXAF2yQluoZBbDB0C6e4yhgHg1CC9RwkoLQbaIJXypVdFylTbANfV1bgZDZD

This is what is returned:

{
  "name": "Universit\u00e4t St.Gallen (HSG)",
  "likes": 20791,
  "location": {
  "street": "Dufourstrasse 50",
  "zip": "9000"
},
  "id": "118336184931317"
}

These calls can be triggered from a piece of code as well. Here is how it is done in Python. JSON object is parsed and name, number of likes, street address and postal code are returned.

#!/usr/bin/env python

import urllib2
import json

facebook_token = 'ose0cBAMc4OZCs4w508tpKbu0ikeTNrZAyBY4Ipkj2phzXkq5cf2VZAngySd9lq5KWIq5sl1ZBo0iaIIvpTiyXcnM1YavhzqEh8cYnVPxn3ZC9y1ZB6xAgJYZCNiFfLFobOZAi0yAmyPEXoSIRXAF2yQluoZBbDB0C6e4yhgHg1CC9RwkoLQbaIJXypVdFylTbANfV1bgZDZD'

#fetch name, likes, street and postal code
url = 'https://graph.facebook.com/HSGUniStGallen?fields=name,likes,location.street,location.zip&access_token=' + facebook_token

json_str = urllib2.urlopen(url)
fdata = json.load(json_str)

print 'Name: ' + str(fdata['name'].encode('utf-8'))
print 'Address: ' + str(fdata['location']['street']) + ', ' + str(fdata['location']['zip'])
print 'Likes: ' + str(fdata['likes'])

The output of the code is the following:

Name: Universität St.Gallen (HSG)
Address: Dufourstrasse 50, 9000
Likes: 20792

A lot more user friendly to read than JSON.

REST API and Hadoop administration

Administrating Hadoop from Ambari sometimes requires running REST API commands. Here is an example of how to remove a service from the cluster by using ambari REST API:

curl -u admin:admin -H “X-Requested-By: ambari” -X DELETE http://ambari-server:8080/api/v1/clusters/testiwimd/services/MAPREDUCE2/components/HISTORYSERVER