NLP Stock SA

This is a stock sentiment analysis project.

Resources:

• DB potential schema- lucidchart
• Example DB tables
• General pipeline - graph
• General pipeline building

Requirements

• Docker
• Python 3.10
• Poetry 1.7.1
• Node 18.19.0
• Yarn 4.1.0
• PostgreSQL 16.2.0

Extensions and Settings for VS Code

Extensions

• Black
• Flake8
• ESLint
• Prettier
• Prettier ESLint

Settings

cp .vscode.example .vscode

Details

For Docker, make sure you bump the resources it can be allocated:

• CPU limit: 16
• Memory limit: 16 GB
• Swap: 1.5 GB
• Virtual disk limit: 160 GB

Installation

Add following to /etc/hosts

TODO: this might not be necessary still?

127.0.0.1 nlp-ssa.dev *.nlp-ssa.dev database

NOTE: For M1 Macs only

export DOCKER_DEFAULT_PLATFORM=linux/amd64

brew install nvm [email protected] postgresql@16
curl -sSL https://install.python-poetry.org | python 3 - --version 1.8.2
nvm install $(cat .nvmrc) && nvm use

Create .env:

cp .env.example .env

Make scripts for reverse proxy executable and run install.sh:

chmod +x nginx-reverse-proxy/generate-certs.sh nginx-reverse-proxy/install.sh nginx-reverse-proxy/uninstall.sh
./nginx-reverse-proxy/install.sh

Initial install for frontend:

cd app
nvm use #=> this should set your Node version to 18.20.2
corepack yarn install

Initial install for backend:

cd server
poetry install

Required Software

Docker

Download Docker Desktop and start it

PostgreSQL

brew install postgresql@16
brew services stop postgresql@16 # <= need to stop server started by Homebrew as it'll interfere with our container

Python & Poetry

brew install [email protected]
curl -sSL https://install.python-poetry.org | python 3 - --version 1.8.2

Node & Yarn

brew install nvm
nvm install $(cat .nvmrc) && nvm use
cd web
corepack enable && yarn install

Operations??

Quick Start

docker compose build
./admin.sh db init && ./admin.sh db seed
docker compose up all -d

Then navigate to https://nlp-ssa.dev/app 🙂

To run the scraper:

docker compose up scraper --build -d

DB Migrations

Run current database migrations:

docker compose run --rm server alembic upgrade head

Reset to base version

docker compose run --rm server alembic downgrade

Run revisions:

docker compose run --rm server alembic revision -m "some migration message"

Autogenerate revisions:

docker compose run --rm server alembic revision --autogenerate -m "some migration message"

Frontend

TODO 🫠

Tests

Server

Run current database migrations for test database:

docker compose run -e DATABASE_NAME=test_the_money_maker server alembic upgrade head

Run tests with:

docker compose run -e DATABASE_NAME=test_the_money_maker -e ENV=test --rm server python -m pytest -s --import-mode=append

Or run in watch mode:

docker compose run -e DATABASE_NAME=test_the_money_maker -e ENV=test --rm server pytest-watch