pyspark-project-example

A simple example for PySpark based project.

APACHE-2.0 License

Stars

11

Committers

View Code on GitHub

Ecosystems: Python

pyspark-project-example

A simple example for PySpark based project.

Prerequisites

JDK (>=7) (as the local test downloads Spark 1.6.0, which dropped JDK 6 support)
Python PEP 8 (for checking code-style in local test)
```
pip install pep8
```
Python Py4J == 0.9.0 (for initiating Spark in local test)
```
pip install -r requirements.txt
```

Test

For test in your local, just run below:
```
./dev/run_tests
```
This contains some codes to download Spark 1.6.0 to run tests. So, it is okay to run the script without some specific modifications.

Warnings

As currently PySpark cannot be installed via pip install due to this issue,
Spark has to be installed and SPARK_HOME="your Spark path" and PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH should be set in order to use this library outside after building.

Badges

Extracted from project README

Build Status

codecov.io

Related Projects

spark-examples

Spark examples

pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

28 Dec 2017 1,639

spark

Apache Spark - A unified analytics engine for large-scale data processing

25 Feb 2014 38,255

sarplus

pronounced sUrplus as it's simply better if not best!

spark-tdd-example

A simple Spark TDD example

pySparkUtils

A collection of utilities for handling pySpark's SparkContext

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

09 May 2015 261

data-analysis-with-python-and-pyspark

spark-prometheus

Prometheus Connector for Apache Spark