RESTful API with strongly-typed JSON schema for distributed systems. Uses Elasticsearch-Kibana for indexed searching and RabbitMQ as a message broker.
MIT License
A course on distributed systems and big data indexing techniques.
To install all application dependencies [Elasticsearch, Kibana and RabbitMQ], use the docker-compose.yml
file to deploy all application dependencies using docker compose
.
To create the cluster, run:
docker compose up
To delete the cluster and the network:
docker compose down
The default username and password for RabbitMQ is defined in the docker-compose.yml
file.
guest
guest
The following is a list of the endpoints for the dependencies being created in docker:
Install redis
# only for MacOs
brew install redis
Install the dependencies
yarn
# or if you are using npm: npm i
Start the server locally
yarn start:dev
# npm run start:dev
We'll use a docker container to connect to the elasticsearch service. So make sure you have docker installed.
To run the elasticsearch image in a docker container (single-node cluster):
Create a new docker network
# docker network create elasticnet
docker network create <network-name>
Pull the elastic docker image
# Running an elastic docker container only works with a specified tag, not with
# latest versions.
# docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker pull docker.elastic.co/elasticsearch/elasticsearch:[tag]
Use the -m flag to set a memory limit for the container. This removes the need to manually set the JVM size.
Start an elasticsearch container
# docker run --name es01 --net elasticnet -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker run --name <container-name> --net <network-name> -p 9200:9200 -it -m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:[tag]
The command prints the
elastic
user password and an enrollment token for Kibana.
Copy the generated elastic
password and enrollment token. These credentials are only shown when you start Elasticsearch for the first time. You can regenerate the credentials using the following commands.
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
We recommend storing the elastic password as an environment variable in your shell
export ELASTIC_PASSWORD="your_password"
Copy the http_ca.crt
SSL certificate from the container to your local machine.
# docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt /Users/sid/Developer/api/src/certs
docker cp <container-name>:/usr/share/elasticsearch/config/certs/http_ca.crt /path/to/your/folder
Make a REST API call to Elasticsearch to ensure the Elasticsearch container is running.
curl --cacert /Users/sid/Developer/api/src/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200
This should printout a success message that looks similar to this:
{
"name" : "686b14641876",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "C4TdqRy3QKSQM9Cri2HBKA",
"version" : {
"number" : "8.11.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "6f9ff581fbcde658e6f69d6ce03050f060d1fd0c",
"build_date" : "2023-11-11T10:05:59.421038163Z",
"build_snapshot" : false,
"lucene_version" : "9.8.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
To install the Kibana docker image:
Pull the Kibana Docker image.
# docker pull docker.elastic.co/kibana/kibana:8.11.1
docker pull docker.elastic.co/kibana/kibana:[tag]
Start a Kibana container.
# docker run --name kib01 --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.11.1
docker run --name <container-name> --net <network-name> -p 5601:5601 docker.elastic.co/kibana/kibana:[tag]
NOTE: When Kibana starts, it outputs a unique generated link to the terminal. To access Kibana, open this link in a web browser. In your browser, enter the enrollment token that was generated when you started Elasticsearch.
To regenerate the token
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
Log in to Kibana as the elastic
user with the password that was generated when you started Elasticsearch
# to regenerate the password, run:
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
To install the RabbitMQ docker image:
Pull the RabbitMQ docker image
# docker pull rabbitmq:3.12-management-alpine
docker pull rabbitmq:[tag]
Create the RabbitMQ docker container
# docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management-alpine
docker run -it --rm --name <container-name> -p 5672:5672 -p 15672:15672 rabbitmq:[tag]
To remove the containers and their network, run:
# Remove the Elastic network
# docker network rm elastic
docker network rm <network-name>
# Remove Elasticsearch, Kibana and RabbitMQ containers
# docker rm es01
docker rm <elastic-container-name>
# docker rm kib01
docker rm <kibana-container-name>
# docker rm rabbitmq
docker rm <rabbitmq-container-name>
This project is licensed under the MIT License. (see license file for details)