#Install spark locally on a mac mac osx#

eval '$(jenv init -)' export JAVA_HOME='$HOME/.jenv/versions/`jenv version-name`' alias jenv_set_java_home='export JAVA_HOME='$HOME/.jenv/versions/`jenv version-name`'' Tip 1: If you are using jenv to manage your versions, then you might need the following additional lines in your.

You may also want to setup a bunch of variables early on, to be of use later CDH='5.4.7' export HADOOP_HOME='$HOME/bin/cdh/$ While you are at it, also grab what version of Spark (pre-built for Hadoop 2.6x) from here, and untar to a directory like below, which we will call $SPARK_INSTALL $ ls $HOME/bin/spark-1.5.0-bin-hadoop2.6/ CHANGES.txt LICENSE NOTICE R README.md RELEASE bin conf data ec2 examples lib python sbin You have to download the CDH tarballs for Hadoop, Zookeeper, Hive from the tarball page (CDH 5.4.x page) and untar them under a folder (refered to as CDH_HOME going forward) as hadoop, zookeeper $ ls $HOME/bin/cdh/5.4.7 hadoop hadoop-2.6.0-cdh5.4.7.tar.gz hive-1.1.0-cdh5.4.7 hive-1.1.0-cdh5.4.7.tar.gz zookeeper zookeeper-3.4.5-cdh5.4.7.tar.gz Download Stuffįirst off all, make sure you have Java 7/8 installed, with JAVA_HOME variable setup to point to the correct location. Most of the methodology is borrowed from here, we just link the four pieces together nicely in this blog. We will use the latest stable Cloudera distribution, and work off the jars. Hive (So we can create some tables and play with it).Spark (We will illustrate with Spark Shell, but should work on YARN mode as well).

Here, we attempt to get some pretty heavy-weight stuff working locally on your mac, namely If you are like me, who loves to have everything you are developing against working locally in a mini-integration environment, read on

#Install spark locally on a mac mac osx#

Setting up Hadoop/YARN/Spark/Hive on Mac OSX