An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage...
Streaming data processing using Hadoop HDFS, Spark, Kafka, Minio, Elasticsearch
⏱ Real-Time Sentiment Analysis using PySpark and simulation of Twitter/X API using FastAPI
Spark Structured Streaming / Kafka / Cassandra / Elastic
Welcome to StreamlineDE, an end-to-end data engineering project designed to demonstrate real-time...
Spark structured streaming with Kafka data source and writing to Cassandra
Big-Data with Apache Spark and Python.
Java Application, uses Apache Spark, handles batch as well as streaming processing
Sentiment Analysis on streaming data and batch data from Reddit
Utilizing my background and love for Apache Airflow and Data to build a real-time data streaming ...
End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to sche...
In this project, we developed an ETL pipeline using Apache Airflow to process delivery data and t...