How to Install Apache Spark on Ubuntu Linux

Install Java JDK

Installing Java JDK is the first step to get Apache Spark running on Ubuntu. Spark requires this program to function correctly. The default Java Development Kit can be installed by running two commands in your terminal: one to update your software list, and another to install Java.

sudo apt update
sudo apt install default-jdk

After installing, run the commands below to verify the version of Java installed.

java --version

This should show lines like the ones below:

openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)

Install Scala

To install Scala on Ubuntu for Apache Spark, use your system’s software manager. Scala is another important piece that Apache Spark needs to function correctly. A single command installs Scala, after which you can check its version.

sudo apt install scala

To verify the version of Scala installed, run the commands below:

scala -version

This should display a line like this:

With Java and Scala installed, Apache Spark installation on Ubuntu can begin. This guide details downloading the latest Apache Spark package, version 3.4.1, and placing it correctly on your computer. Follow these steps to download and extract the necessary files for the Apache Spark installation.

Download the latest version of Spark.

cd /tmp
wget https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6-bin-hadoop2.7.tgz

Then, extract the downloaded file and move it to the /opt directory.

tar -xvzf spark-2.4.6-bin-hadoop2.7.tgz
sudo mv spark-2.4.6-bin-hadoop2.7 /opt/spark

nano ~/.bashrc

Add the following lines to the bottom of the file and save.

export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Finally, apply your environment changes by running these commands.

After installing Apache Spark on Ubuntu, starting the master process requires one command, and the worker process requires a second. Once both the Spark master and worker processes are running, you can check Spark’s status in your web browser.

start-master.sh

Next, start the Spark work process by running the commands below.

start-slave.sh spark://localhost:7077

`localhost` can be swapped with your server’s hostname or IP address. When the process starts, open your browser and navigate to the server hostname or IP address.

To connect to Apache Spark using its command shell, execute the following commands. This connection allows you to interact directly with the Spark cluster, enabling data processing and analysis tasks.

spark-shell

The commands above will launch Spark Shell.

Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/_,_/_/ /_/_   version 2.4.6
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 11.0.10)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

That should do it!

Conclusion:

This post covered installing Apache Spark on Ubuntu 20.04 | 18.04. If you find any error above, please use the form below to report.