Apache Spark Ecosystem

spark

Apache Spark - A unified analytics engine for large-scale data processing

spark-nlp

State of the Art Natural Language Processing

SynapseML

Simple and Distributed Machine Learning

zeppelin

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more

redash

Make Your Company Data Driven

h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc

delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

deeplearning4j

Suite of tools for deploying and training deep learning models using the JVM

TensorFlowOnSpark

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters

spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs

fugue

A unified interface for distributed computing

spark-cassandra-connector

DataStax Connector for Apache Spark to Apache Cassandra

horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet

kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses

alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud

spark

apache-spark-internals

The Internals of Apache Spark

ytsaurus

YTsaurus is a scalable and fault-tolerant open-source big data platform

adam

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet

mlflow

Open source platform for the machine learning lifecycle

bigdata---train-analysis

batch processing and realtime tains(railway) data analysis to help Station Masters refreshing each 20 seconds

maven-hocon-extension

Apache maven

mcdd-big-data-study

Study project for big data (Hadoop, Zookeeper, Kafka, Flink, Spark)

nifi-api

Apache NiFi API

Olympics_data_Project

A personal project that builds an end-to-end data pipeline using the 2024 Olympics data

daph

Daph是一个通用的数据同步与数据处理平台级工具，既具有丰富的数据同步能力，又具有强大的数据处理能力，一站式满足数据开发所有需求，可用于构建可视化配置化的数据同步与数据处理平台。

Tokyo-Olympics-2021-Analytics

An end-to-end ETL pipeline for analyzing and visualizing Tokyo Olympics 2021 data using Azure tools and Power BI

airflow-spark

If you want to use airflow with spark, ready to use ;-)

StreamlineDE-

Welcome to StreamlineDE, an end-to-end data engineering project designed to demonstrate real-time data ingestion, processing, and storage using a modern data engineering stack

Reddit-Analysis

Sentiment Analysis on streaming data and batch data from Reddit

Daily-Youtube-Extraction

Projeto que completa a criação de um ambiente para extração, armazenamento e processamento de dados do Youtube

ozhera-site

Website sources for the Apache OzHera(Incubating) website

spark-cluster-multi-node-setup

Quickly setup and simulate a multi node spark cluster using docker and docker-compose

arrow-go

Official Go implementation of Apache Arrow

Stark

基于Spark+Debezium打造的简单易用、超高性能大数据治理引擎，适用于批流一体的数据集成和数据分析场景，支持CDC实时数据采集，支持海量数据同步、数据建模和OLAP数据分析

BigBanyanTree

Gathering insights from Common Crawl using Apache Spark and LLMs

BigDataETLAndSentimentAnalysis

A Java based project aims to extract news articles from large

pulsar-java-contrib

Contributor repository for code samples, plugins and libraries in Java for Apache Pulsar

polaris-site

Apache polaris

case-study-accidents

Spark analysis on the accidents-data