RHive - Integration of R and Hive with simple demo

-
RHive is package that can be used for writing hive queries in R.User can import RHive library in R and then one can start writing hive queries in it. Today we are going to do installation of R and some simple example to demonstrate power of R.

Notes: All commands shown below are executed and tested on Cent OS 6.x

Before we proceed , we need certain things in place. Since I have used Cents OS, I installed couple of pre-requisite that is required before we begin with RHive installation.

Install Ant : This is required for building and packaging project.Below is a command to install it.

sudo yum install ant                                                                                               

Install JDK

sudo yum install java-1.6.0-openjdk                                                                 

Set JAVA_HOME in .bashrc file

export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.36.x86_64                    

Install git

 sudo yum install git                                                                                                         
Set up your HIVE_HOME and HADOOP_HOME 

export HIVE_HOME=/usr/lib/hive                                                                                
export HADOOP_HOME=/usr/lib/hadoop                                                                  

Once this is done, you will have to install R and RStudio for accessing hive and demonstrating example.

For R installation , you can simply do 

sudo yum install R and for RStudio you can simply run below command.


wget https://download2.rstudio.org/rstudio-server-rhel-0.99.484-x86_64.rpm
sudo yum install --nogpgcheck rstudio-server-rhel-0.99.484-x86_64.rpm                   

Note: Above RStudio  command is with respect to my machine configuration , you can check on rstudio site for command compatible with your machine.

To check RStudio server is accessible, open your browser and type below address

http://ip:8787

ip :IP address of your machine where you have installed RSutdio server.

Above URL should give RStudio web page.

Now you have installed and configured all pre-requisite required for RHive installation. Now get ready to clone RHive repository and install RHive package in R.

git clone https://github.com/nexr/RHive.git
cd Rhive
ant build
R CMD build RHive
wget https://cran.r-project.org/src/contrib/rJava_0.9-7.tar.gz
wget https://cran.r-project.org/src/contrib/Rserve_1.7-3.tar.gz
R CMD INSTALL rJava_0.9-7.tar.gz
R CMD INSTALL Rserve_1.7-3.tar.gz
R CMD INSTALL RHive_2.0-0.10.tar.gz             

Once above command are completed successfully, you are ready to use RHive.
Now just to test your installation start R shell and type below command to check if you get output.

library(RHive)
rhive.init()
rhive.connect(ip,port,hiveServer2)
rhive.query("show databases")

Above command should list all databases present in hive , if doesn't then please re-check your configurations.

Lets draw 3D pie chart using R now. Before this we need to have sample data in hive which can be accessed through R.

For this I have created a simple Student table. Here is a command to create student hive table.

create table Students(
Sname String,
score int,
subject String
)
row format delimited 
fields terminated by '|'
stored as TextFile;

Sample Input file for student table.

Sname|Score|Subject
Griffith|20|Computer
Rigel|19|Computer
Amos|44|Computer
Mason|35|Computer
Otto|67|Computer
Chester|16|Computer
Henry|15|Computer
Jerry|96|Computer
Grant|86|Computer

Load data into table and start rstudio server web page (http://ip:8787)

You should see scree something like this.



Here I am going to create  student performance pie chart.  Based marks scored in computer subject, they will be given Poor,Good etc grade.

Here is my R script.I am not a good R programmer but I have tried my best perfect result.:P

library(RHive)
rhive.init()
rhive.connect(ip,10000)
poor <- rhive.query("select count(score) from students where score <=25")
avgs <- rhive.query("select count(score) from students where score >25 and score <=40")
goodst <- rhive.query("select count(score) from students where score >40 and score <=75")
vgoods <- rhive.query("select count(score) from students where score >75")
a = as.integer(poor)
b = as.integer(avgs)
c = as.integer(goodst)
d = as.integer(vgoods)
marks <- c(a,b,c,d)
lbls <- c("Very Poor", "Average", "Good" ,"Very Good")
pct <- round(marks/sum(marks)*100)
lbls <- paste(lbls, pct) # add percents to labels 
lbls <- paste(lbls,"%",sep="") # ad % to labels 
pie3D(marks,labels = lbls, col=rainbow(length(lbls)),
   main="Pie Chart of Student Performance")
Output from above 

Reference:
https://github.com/nexr/RHive

Hope this will help....:)

Comments

Post a Comment

Popular posts from this blog

JDBC Hive Connection fails : Unable to read HiveServer2 uri from ZooKeeper

-
Image
I have been struggling from couple of days to get my jdbc hive connection working using Zookeeper URI and after putting lot of effort finally I realised problem. In this post I will sharing my experience  and steps I took to rectify problem. Notes : I am having a HDP 2.6 sandbox along with my JDBC program . Use case : Many companies does not provide HiveServer2 url for configuring your JDBC string and its important because if your hiveserver goes down  , your whole job will abort (You will be missing Hive HA capabilities). So How do I dynamically determine second Hive Server URI ? Solution is to use Zookeeper Service Discovery feature. Zookeepers keeps all HiveServer 2 URI into its namespace. When client tries to make a connection to Hive , Zookeeper will look for available hive server and return URL to client program. You can quickly verify HiveServer namespace in zk Path with below command on HDP.  /usr/hdp/current/zookeeper-client/bin/zookeeper-client...
Read more

Access Kubernetes ConfigMap in Spring Boot Application

-
Image
 Hello There, Welcome back to our new post on Kubernetes tutorials. In this example we will see how can we configure and access ConfigMap in Spring boot application. ConfigMaps are helpful where you have multiple environment (Dev,Test Prod etc) and you want to promote your container with different environment configuration. There are three ways Pods can access ConfigMap in containers. For todays Demo , we will use Environment Variable approach. Pre-requisite : MiniKube Cluster   For Demo Purpose I have created Spring Boot application which will expose below end point.  curl localhost:8080/configMap Docker Image : shashivish123/springboot  Below is how Controller class looks like and value is being autowired from application.properties. Remember CONFIG_KEY value will be coming from Kubernetes ConfigMap Environment Configuration. Now Let's Begin , We will start by Creating firs ConfigMap in Kubernetes. Step 1: Create a namespace where we will ...
Read more

Developing Custom Processor in Apache Nifi

-
Image
Apache Nifi was developed to automate the flow of data between different systems. Apache NiFi is based on technology previously called “Niagara Files” that was in development and used at scale within the NSA for the last 8 years and was made available to the Apache Software Foundation through the NSA Technology Transfer Program. Nifi is based on FlowFiles which are heart of it. A FlowFile is a data record, which consists of a pointer to its content (payload) and attributes to support the content, that is associated with one or more provenance events. The attributes are key/value pairs that act as the metadata for the FlowFile, such as the FlowFile filename. The content is the actual data or the payload of the file. Provenance is a record of what’s happened to the FlowFile. Each one of these parts has its own repository (repo) for storage. Each flowfile is processed by FlowFile processor . Processors have access to attributes of a given FlowFile and its content stream. Processo...
Read more