Local PySpark dev environment

This repo provides everything needed for a self-contained, local PySpark 1-node "cluster" running on your laptop, including a Jupyter notebook environment.

It uses Visual Studio Code and the devcontainer feature to run the Spark/Jupyter server in Docker, connected to a VS Code dev environment frontend.

Requirements

Install Docker Desktop (you don't have to be a Docker super-expert :-))
Install Visual Studio Code
Install the VS Code Remote Development pack

Setup

Install required tools
Git clone this repo to your laptop
Open the local repo folder in VS Code
Open the VS Code command palette and select/type 'Reopen in Container'
Wait while the devcontainer is built and initialized, this may take several minutes
Open test.ipynb in VS Code
If you get an HTTP warning, click 'Yes'
Wait a few moments for the Jupyter kernel to initialize... if after about 30 seconds or so the button on the upper-right still says 'Select Kernel', click that and select the option with 'ipykernel'
Run the first cell... it will take a few seconds to initialize the kernel and complete. You should see a message to browse to the Spark UI... click that for details of how your Spark session executes the work defined in your notebook on your 1-node Spark "cluster"
Run the remaining cells in the notebook, in order... see the output of cell 3
Have fun exploring PySpark!