Microservices with GRPC and Kafka on Node.js

This is an example microservices architecture which uses GRPC communication between microservices for requests that need instant reply, and uses fault-tolerant messaging queue for HA.

Each microservice supports both GRPC communication and consumption through Kafka topics. If you need to get a response from a certain microservice, GRPC is recommended. If the task can be done asyncronously, please do it within Kafka's boundaries.

Architecture Overview

Requirements

Node (preferably >= 15, since ESM is enabled by default)
PostgreSQL (preferably >= 12)
Kafka

Development

Install dependencies using npm install on the root directory
Run docker-compose up on the root directory
Prepare database using the directions in packages/subscription

Deployment

Public:

has /metrics endpoint where it exposes certain memory and cpu consumption metrics for Prometheus support.
has /health endpoint for readiness and liveliness probe checks.

GRPC microservices:

supports grpc.health.v1.Health grpc health checks

For further information regarding Kubernets deployment, please look into the k8s folder.

Security

inter microservice communication
- In general all communications between microservices should be encrypted, preferably using SSL. Right now, all of the requests are signed but have insecure SSL configuration.
All microservices hide stacktraces and error messages from client by default.
Rate limiting is enabled on public microservice. the default configuration is max 100 requests per minute.
- Right now we're using memory based rate limiting which is bad for production since it will reset on deployment, and work be suitable if you have scale to more than 1 horizontal pods.
- Exposes x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset and retry-after headers to client.
CORS configuration: Before using this code on production make sure that the necessary CORS headers should only allow requests from your owned domains to disallow unwanted requests from browsers and websites.

OpenAPI 3 Specification

You can access OpenAPI 3 based documentation generated by public microservice using docs next.js project which consumes the OpenAPI specification and creates a developer friendly website.

Privacy Issues

Subscription microservice is not GDPR friendly. We shouldn't store any personal information unless we particularly need it. Since some of the issues regarding personal information was inside the task itself, I left it there, but in order comply with the regulations, I didn't soft delete a subscription but removed it completely upon cancellation.

Development Issues

Lerna npmClientArgs does not work for Node 15 & NPM 7. Please refer to https://github.com/n8n-io/n8n/issues/1149. In order to solve installation issues you need to run npm config set legacy-peer-deps true. By default using npm install triggers prepare-build.js command which handles this error.