Adding service to HDP using REST API

Create configuration

Services have one or more mandatory configuration files that hold properties the service needs to start and function. In case of Pig, there are 3 such configuration files. Without them, the service will not start and it will output a descriptive error saying the configuration is missing.
The next three files are added in the order Pig searches for them. The files are copied from an existing HDP cluster and have not been changed.

pig-env

Create file pig-env-INITIAL.json
{"type": "pig-env", "tag": "INITIAL", "properties" : { "content" : "\nJAVA_HOME={{java64_home}}\nHADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\n\nif [ -d \"/usr/lib/tez\" ]; then\n  PIG_OPTS=\"$PIG_OPTS -Dmapreduce.framework.name=yarn\"\nfi\n    "}}

The command is going to read from the JSON file.

curl -u admin:admin -H 'X-Requested-By:admin' -i -X POST -d '@pig-env-INITIAL.json' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME'/configurations'
Output

“201 Created” status is returned:

create config pig-env output.png

Apply configuration
curl -v -u admin:admin -H 'X-Requested-By:admin' -i -X PUT -d '{ "Clusters" : {"desired_configs": {"type": "pig-env", "tag" : "INITIAL" }}}' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME''<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>

Configuration type “pig-env” has been added.

apply config pig-env output.png

The change in Ambari

apply config pig-env ambari.png

pig-properties

Create file pig-properties-INITIAL.json
{"type" : "pig-properties",
          "tag" : "INITIAL",
          "version" : 1,
          "properties" : {
            "content" : "\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Pig default configuration file. All values can be overwritten by pig.properties and command line arguments.\n# see bin/pig -help\n\n# brief logging (no timestamps)\nbrief=false\n\n# debug level, INFO is default\ndebug=INFO\n\n# verbose print all log messages to screen (default to print only INFO and above to screen)\nverbose=false\n\n# exectype local|mapreduce|tez, mapreduce is default\nexectype=tez\n\n# Enable insertion of information about script into hadoop job conf \npig.script.info.enabled=true\n\n# Do not spill temp files smaller than this size (bytes)\npig.spill.size.threshold=5000000\n\n# EXPERIMENT: Activate garbage collection when spilling a file bigger than this size (bytes)\n# This should help reduce the number of files being spilled.\npig.spill.gc.activation.size=40000000\n\n# the following two parameters are to help estimate the reducer number\npig.exec.reducers.bytes.per.reducer=1000000000\npig.exec.reducers.max=999\n\n# Temporary location to store the intermediate data.\npig.temp.dir=/tmp/\n\n# Threshold for merging FRJoin fragment files\npig.files.concatenation.threshold=100\npig.optimistic.files.concatenation=false;\n\npig.disable.counter=false\n\nhcat.bin=/usr/bin/hcat\n\n    "
          }
}

Post the configuration change

curl -u admin:admin -H 'X-Requested-By:admin' -i -X POST -d '@pig-properties-INITIAL.json' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME'/configurations'
Apply configuration
curl -v -u admin:admin -H 'X-Requested-By:admin' -i -X PUT -d '{ "Clusters" : {"desired_configs": {"type": "pig-properties", "tag" : "INITIAL" }}}' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME''

pig-log4j

Create file pig-log4j-INITIAL.json
{"type" : "pig-log4j",
"tag" : "INITIAL",
"version" : 1,
"properties" : {
"content" : "\n#\n#\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n#\n#\n#\n\n# ***** Set root logger level to DEBUG and its only appender to A.\nlog4j.logger.org.apache.pig=info, A\n\n# ***** A is set to be a ConsoleAppender.\nlog4j.appender.A=org.apache.log4j.ConsoleAppender\n# ***** A uses PatternLayout.\nlog4j.appender.A.layout=org.apache.log4j.PatternLayout\nlog4j.appender.A.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n\n    "
}
}

Post the configuration change

curl -u admin:admin -H 'X-Requested-By:admin' -i -X POST -d '@pig-log4j-INITIAL.json' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME'/configurations'
Apply configuration
curl -v -u admin:admin -H 'X-Requested-By:admin' -i -X PUT -d '{ "Clusters" : {"desired_configs": {"type": "pig-log4j", "tag" : "INITIAL" }}}' 'http://'$AMBARI_SERVER':8080/api/v1/clusters/'$CLUSTER_NAME''

Creating hosts components is on next page

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s