Cleared Databricks Spark Certification

-
Last Week I cleared Databricks spark developer certification. Well exam was not that much difficult but it had some tricky question and bunch of scala/python spark program challenges.

1. You must go through ‘Learning Spark’ book. Try practicing all transformation and action given in book.

2. There were 40 questions in exam and all were MCQ. I am not sure about passing percentage; I scored 70% to pass.

3. 90% question were from Spark Core , Spark SQL and Spark Streaming . There were 2 questions from Machine learning and 1 question from GraphX. For GraphX I referred MapR blog , it has got nice explanation to begin with.

https://www.mapr.com/blog/how-get-started-using-apache-spark-graphx-scala

4. Make sure you do enough hands on before appearing for the exam. Most of the questions were related to programming so must have through understanding how API works in detail.

https://github.com/databricks/learning-spark

5. Try examples in all three language, you don’t need to be expert but you should be able to understand code. In my case most of the questions were in Scala and python.

Topics  one must cover:
  1. All transformation and Actions on basic and pairRDD.
  2. Accumulator and Broadcast variable.
  3. Lazy evaluation model.
  4. Lineage graph in spark.
  5. Word count in all three languages.
  6. Spark SQL – DataFrame example – Infer Schema , Programmed Schema
  7. Window operation in Spark Streaming , Check Pointing , DStream recovery on failure and Statefull transformation.
  8. Machine learning : K-mean , Regression , Clustering
  9. GrapghX basic : Vertex , Edge RDD and Triplets in graphs.


Reference Link :

  • Apache Spark Documentation : http://spark.apache.org/
  • DataBricks Knowledge Base : https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/
  • Spark Training : https://spark-summit.org/2014/training





Comments

  1. Unknown7 September 2016 at 08:11

    Thanks for sharing the roadmap

    ReplyDelete
  2. Anupam Sharma5 October 2016 at 06:16

    Hi Shashi,

    i have few questions -

    1. How was your preparation strategy ? Which books and material you used ?
    2. Is there distributed network required for handson or standalone laptop (localmode) is enough from test perspective ?
    3. The test was "multiple option answer test" or it was real time whiteboard programming test?

    Thanks for the sharing the detail with us.

    ReplyDelete
  3. Anupam Sharma5 October 2016 at 06:18

    Hi Shashi,

    i have few questions -

    1. How was your preparation strategy ? Which books and material you used ?
    2. Is there distributed network required for handson or standalone laptop (localmode) is enough from test perspective ?
    3. The test was "multiple option answer test" or it was real time whiteboard programming test?

    Thanks for the sharing the detail with us.

    ReplyDelete
  4. Shashi9 October 2016 at 06:24

    Hi Anupam,

    I have referred learning Spark book and tried all examples given in book and apache spark documentation is also good resource.

    I think Sandbox is good enough to practice.

    Test was MCQ based.

    ReplyDelete
    Replies
    1. Anupam Sharma14 October 2016 at 20:00

      thanks shashi
      i will appear the exam next month, will let you know about my experience

      Delete
  5. Shashi16 October 2016 at 07:01

    All the Best...

    ReplyDelete
  6. dhruba25 April 2017 at 05:53

    Very Well written . Good Work bro

    ReplyDelete

Post a Comment

Popular posts from this blog

JDBC Hive Connection fails : Unable to read HiveServer2 uri from ZooKeeper

-
Image
I have been struggling from couple of days to get my jdbc hive connection working using Zookeeper URI and after putting lot of effort finally I realised problem. In this post I will sharing my experience  and steps I took to rectify problem. Notes : I am having a HDP 2.6 sandbox along with my JDBC program . Use case : Many companies does not provide HiveServer2 url for configuring your JDBC string and its important because if your hiveserver goes down  , your whole job will abort (You will be missing Hive HA capabilities). So How do I dynamically determine second Hive Server URI ? Solution is to use Zookeeper Service Discovery feature. Zookeepers keeps all HiveServer 2 URI into its namespace. When client tries to make a connection to Hive , Zookeeper will look for available hive server and return URL to client program. You can quickly verify HiveServer namespace in zk Path with below command on HDP.  /usr/hdp/current/zookeeper-client/bin/zookeeper-client   Once
Read more

Access Kubernetes ConfigMap in Spring Boot Application

-
Image
 Hello There, Welcome back to our new post on Kubernetes tutorials. In this example we will see how can we configure and access ConfigMap in Spring boot application. ConfigMaps are helpful where you have multiple environment (Dev,Test Prod etc) and you want to promote your container with different environment configuration. There are three ways Pods can access ConfigMap in containers. For todays Demo , we will use Environment Variable approach. Pre-requisite : MiniKube Cluster   For Demo Purpose I have created Spring Boot application which will expose below end point.  curl localhost:8080/configMap Docker Image : shashivish123/springboot  Below is how Controller class looks like and value is being autowired from application.properties. Remember CONFIG_KEY value will be coming from Kubernetes ConfigMap Environment Configuration. Now Let's Begin , We will start by Creating firs ConfigMap in Kubernetes. Step 1: Create a namespace where we will deploy container and
Read more

Developing Custom Processor in Apache Nifi

-
Image
Apache Nifi was developed to automate the flow of data between different systems. Apache NiFi is based on technology previously called “Niagara Files” that was in development and used at scale within the NSA for the last 8 years and was made available to the Apache Software Foundation through the NSA Technology Transfer Program. Nifi is based on FlowFiles which are heart of it. A FlowFile is a data record, which consists of a pointer to its content (payload) and attributes to support the content, that is associated with one or more provenance events. The attributes are key/value pairs that act as the metadata for the FlowFile, such as the FlowFile filename. The content is the actual data or the payload of the file. Provenance is a record of what’s happened to the FlowFile. Each one of these parts has its own repository (repo) for storage. Each flowfile is processed by FlowFile processor . Processors have access to attributes of a given FlowFile and its content stream. Processo
Read more