How to Install Apache Spark on Ubuntu Linux

Install Java JDK

Installing Java JDK is the first step to get Apache Spark running on Ubuntu. This program is required for Spark to work properly. You can install the default Java Development Kit by running two simple commands in your terminal to update your software list and then install Java.

sudo apt update
sudo apt install default-jdk

After installing, run the commands below to verify the version of Java installed.

java --version

This should show you lines like the ones below:

openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)

Install Scala

To install Scala on Ubuntu and get Apache Spark working, you’ll use your system’s software manager. Scala is another important piece that Apache Spark needs to function correctly. A single command lets you easily install Scala, and then you can check its version.

sudo apt install scala

To verify the version of Scala installed, run the commands below:

scala -version

This should display a line like this:

Apache Spark installation on Ubuntu begins after Java and Scala are installed. This guide shows you how to download the latest Apache Spark package, version 3.4.1, and place it correctly on your computer. Follow these simple steps to download and extract the necessary files to finish the Apache Spark installation.

Download the latest version of Spark.

cd /tmp
wget https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6-bin-hadoop2.7.tgz

Then, extract the downloaded file and move it to the /opt directory.

tar -xvzf spark-2.4.6-bin-hadoop2.7.tgz
sudo mv spark-2.4.6-bin-hadoop2.7 /opt/spark

nano ~/.bashrc

Add the following lines to the bottom of the file and save.

export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Finally, apply your environment changes by running these commands.

After Apache Spark installation on Ubuntu, starting the Spark master process requires running one command, and starting the Spark worker process requires running a second command. Once both the Spark master and worker processes are running, you can check Spark's status in your web browser.

start-master.sh

Next, start the Spark work process by running the commands below.

start-slave.sh spark://localhost:7077

You can swap `localhost` with your server’s hostname or IP address. When the process starts, open your browser and navigate to the server hostname or IP address.

To connect to Apache Spark using its command shell, you can execute the following commands. This connection allows you to interact directly with the Spark cluster, enabling data processing and analysis tasks.

spark-shell

The commands above will launch Spark Shell.

Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/_,_/_/ /_/_   version 2.4.6
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 11.0.10)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

That should do it!

Conclusion:

This post showed you how to install Apache Spark on Ubuntu 20.04 | 18.04. If you find any error above, please use the form below to report.