🚀 Dive into the world of Node.js, MongoDB, Redis, Postgres, RabbitMQ, and the ELK stack with this comprehensive sample backend project! Explore how these powerful technologies can be integrated to build robust, scalable, and efficient backend solutions.
MIT License
This project is a sample Node.js Backend project that demonstrates how to connect to MongoDB, Redis, PostgreSQL, Kafka, and RabbitMQ. It demonstrates how these services can be used and interact with each other in a Node.js application.
Additionally, the project demonstrates how to use the ELK stack (Elasticsearch, Logstash, Kibana) for logging, as well as how to implement a round-robin load balancing algorithm using Redis.
And we call this the NMRPKER-Stack! 🚀
/api/test/route1
and /api/test/route2
.Here is a table of the features demonstrated in this project:
Feature | Description |
---|---|
MongoDB | Connect to MongoDB and perform CRUD operations. |
Redis-Mongo-Flow | Use Redis as a cache layer for MongoDB. |
RabbitMQ | Connect to RabbitMQ and publish and consume messages. |
Apache Kafka | Connect to Apache Kafka and produce and consume messages. |
Round-Robin Load Balancing Algorithm | Implement round-robin load balancing using Redis. |
ELK Stack | Use the ELK stack (Elasticsearch, Logstash, Kibana) for logging. |
PostgreSQL | Connect to PostgreSQL and perform CRUD operations. |
Express.js | Use Express.js as the web server framework. |
Node.js | Use Node.js as the runtime environment. |
JavaScript | Use JavaScript as the programming language. |
NPM | Use NPM as the package manager. |
REST API | Use REST API for communication. |
The project has the following structure:
node-mongo-redis-project
├── index.js # Main entry point for the project for testing the connections
├── config.js # Configuration file for the project
├── package.json # NPM package file
├── publish.js # Script to publish a message to RabbitMQ
├── README.md
├── apache-kafka
│ └── kafkaService.js # Core logic for Apache Kafka
├── round-robin
│ ├── index.js # Core logic for round-robin load balancing & for testing the algorithm
│ └── config.js # Configuration file for Redis
├── routes
│ └── test.js # Sample routes for the project
├── redis-mongo-flow
│ ├── app.js # Core logic of the Redis-Mongo flow
│ ├── config.js # Configuration file for Redis and MongoDB
│ ├── seed.js # Script to populate MongoDB with sample data
│ └── test.js # Script to test the Redis-Mongo flow
├── elk-stack
│ └── index.js # Core logic for logging using the ELK stack & testing the stack
└── postgresql
└── app.js # Core logic for PostgreSQL
└── config.js # Configuration file for PostgreSQL
To get started, run the following commands:
Start the MongoDB service:
brew services start mongodb-community
Start the Redis service:
redis-server
Start the RabbitMQ service:
brew services start rabbitmq
Start the Apache Kafka service:
cd kafka_2.13-3.8.0
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
cd
back into the project directory (fix the path if necessary):
cd node-mongo-redis-project
Start the Node.js server:
node index.js
(Optional) Test the RabbitMQ service by publishing a message (go to another terminal window):
node publish.js
Verify that the message is received by the consumer by going to the terminal that is running the Node.js server (the one you started the index.js script). You should receive the following messages in the terminal:
Server running on port 5000
Visit http://localhost:5000/ to test the connection.
Redis Connected
MongoDB Connected
Redis Test: Hello from Redis
Kafka Producer and Consumer Connected
Aggregation result: [
{ _id: 'Frank', totalOrderValue: 60 },
{ _id: 'David', totalOrderValue: 100 },
{ _id: 'Charlie', totalOrderValue: 35 },
{ _id: 'Grace', totalOrderValue: 75 },
{ _id: 'Alice', totalOrderValue: 50 },
{ _id: 'Bob', totalOrderValue: 80 },
{ _id: 'Ivy', totalOrderValue: 55 },
{ _id: 'Eve', totalOrderValue: 45 },
{ _id: 'Helen', totalOrderValue: 90 }
]
Sent message to Kafka: Hello Kafka from Express!
{
topic: 'test-topic',
partition: 0,
value: 'Hello Kafka from Express!'
}
RabbitMQ Connected
[*] Waiting for messages in task_queue. To exit press CTRL+C
[x] Received 'This is a test message!'
Visit http://localhost:5000/ to test the connections.
(Optional) Test the Round-Robin Load Balancing Algorithm:
cd round-robin
node index.js
(Optional) Test the ELK Stack:
cd elk-stack
node index.js
Note: Before you get started, be sure to have the following installed on your machine by running the following commands (the following instructions are for MacOS):
Install Homebrew:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Install MongoDB:
brew tap mongodb/brew
brew install mongodb-community
Install Redis:
brew install redis
Install RabbitMQ:
brew install rabbitmq
Install Apache Kafka:
wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
Ensure Kafka is running:
cd kafka_2.13-3.8.0
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
Install Postgres:
brew install postgresql
Start the Postgres service:
brew services start postgresql
Create a new database named davidnguyen
(or any name you prefer - as long as you change the database credentials in the postgresql/config.js
file):
createdb davidnguyen
Connect to the database:
psql davidnguyen
Install Node.js:
brew install node
Install NPM:
brew install npm
Then cd
into the project directory and run:
bash npm init -y
Install all the required packages:
npm install
For Windows, refer to the official installation guides for MongoDB, Redis, RabbitMQ, Apache Kafka, and PostgreSQL.
This project is licensed under the MIT License - see the LICENSE file for details.
Feel free to use this project for your own learning purposes or as a reference for your own projects!
Thank you for checking out this demo backend project! 🚀 Feel free to reach out if you have any questions or feedback. 😊