pyspark_pandas

Pyspark + pandas. This may get merged into the SparklingPandas project.

Downloads
392.5K
Stars
6
Committers
1
View Code on GitHub
Ecosystems: Python

There is already an existing project, SparklingPandas, that integrates pandas and pyspark. You should look at that one as this project may get merged into that one. This project aims to provide useful tools and algorithms for distributing Pandas objects on Spark.

  • Apache Spark is a fast and general engine for large-scale data processing. It is written in Scala and also supports Python via PySpark.

  • Pandas is a library providing high-performance and easy-to-use data structures and data analysis tools for the Python programming language.

Did you check out SparklingPandas?

Package Rankings
Top 14.47% on Pypi.org
Related Projects
spark-py-notebooks

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython /...

06 May 2015 1,614
pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

09 May 2015 261
data_science_blogs

A repository to keep track of all the code that I end up writing for my blog posts.

25 Dec 2019 252
datacompy

Pandas and Spark DataFrame comparison for humans and more!

23 Mar 2018 399
data-analysis-with-python-and-pyspark

12 Dec 2020 10
PyNotes

My notebook on using Python with Jupyter Notebook, PySpark etc

23 Aug 2019 10
sarplus

pronounced sUrplus as it's simply better if not best!

24 Oct 2018 12
pyspark-maestro

This repo contains implementations of PySpark for real-world use cases for batch data processing,...

23 Jul 2024 1
TensorFlowOnSpark

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

20 Jan 2017 3,872
APACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB

Apache Spark Machine Learning project using MLlib and Linear Regression on Databricks!

07 Aug 2024 0
pyspark-examples

Pyspark RDD, DataFrame and Dataset Examples in Python language

26 Nov 2019 1,158
pyspark-project-example

A simple example for PySpark based project.

03 Feb 2016 11
spark

Apache Spark - A unified analytics engine for large-scale data processing

25 Feb 2014 38,255
PandasGUI

A GUI for Pandas DataFrames

12 Jun 2019 3,134
Spark-with-Python

Fundamentals of Spark with Python (using PySpark), code examples

20 Aug 2018 328