CaRMa

About documentation

This is important. Project lives relatively for many years (since 2012), and developers switches from one to another and this makes it more clear we need a good documentation and comments for the code and everything must be written in English (some parts still written in Russian, since for a long period of time it have been used and developed only in single country) to make it available to work on the project to anyone (any serious developer supposed to be able to deal with English, if you don't then go learn it or please leave the profession, no offence).

Any documentation related to the project supposed to be placed in docs directory using Markdown (but for some exceptional cases it may be something different but please use open standard formats such as XML, HTML, ODF, CSV, SVG, LaTeX, etc. but not DOC, DOCX, XLS, PSD and other formats from closed ecosystems).

Some documents for the project have been written as separated Google Docs files, they're supposed to be migrated to this repo (to the docs docs), they also was written in Russian, supposed to be rewritten in English. Only very secret documents may live separately.

Please keep in mind that you, as a developer, may die unexpectedly (or you just may leave the poject) as any other creature, and even if you don't (at least not unexpectedly) after you other people probably will work with this project. Please do your job like you die just after your last commit that means not to be paranoid, depressive or crazy but means that it would be a good idea to leave a good comments for your code, for your changes, it would be a good idea to add or expand some documents which explains things for developers, and try to avoid ad-hoc fixes/features which is almost always hard to deal with. Just try to not leave stuff with thoughs like "I'll refactor this later", "I'll add comments for this later", etc., please make sure you did right thing before you commit, at least make sure another developer can deal with it after you, it may be adding TODO or FIXME comments which explains what you or another developer supposed to do. For a function or a monad it would be a good idea to write few simple usage examples and explain what issues it helps/supposed to solve.

When your changes or new features affects deploy process, please check that docs are still exhaustive, and project may be deployed just using these instructions without contacting you personally.

Building

Refer to .circleci/config.yml for full building instructions.

tools/builder.sh

There's tool that could build everything from scratch in parallel for you, just run:

tools/builder.sh -p all

Or get detailed usage info of this tool to be able to run particular tasks with different options:

tools/builder.sh --help

Production release

If your next release doesn't require to apply any migrations, to create or update any configs then your production release could be built by this command:

tools/builder.sh -p --production --clean all

If your server setup implies CaRMa's binaries to be presented in ~/.local/bin/ then you should run after that:

stack install

And restart proper CaRMa services.

P.S. You could find instructions about how to apply database migrations below.

Backend (Haskell)

Generic

Prepare database

Install PostgreSQL 9.3 (this is what production servers use) and PostGIS (extension for PostgreSQL).

We have docker container with exact PostgreSQL version for development, you could use it.

Create a database named carma:
```
createdb carma
```

Add roles:

createuser -s carma
createuser carma_db_sync
createuser carma_geo
createuser carma_sms
createuser mail_svc
createuser fmuser
createuser pavel.golovnin
createuser reportgen

carma is the superuser which owns the database and is used to run DB migration scripts.

Set passwords:

psql carma -c "alter user carma with password 'pass'"
psql carma -c "alter user carma_db_sync with password 'pass'"

Unpack and recover a database snapshot (note that this needs to run as a PostgreSQL superuser - for example, prepend with sudo -u postgres):
```
psql carma -f 2017-05-29_03-15_carma.sql
```
Or if you use docker/dev-pg-9.3 mentioned above, you could do it by this command:
```
psql -h 127.0.0.1 -U carma_db_sync -d carma -f 2017-05-29_03-15_carma.sql
```
In case you have a gzipped database dump (.sql.gz extension):
```
zcat 2017-05-29_03-15_carma.sql.gz | psql carma
```
For docker/dev-pg-9.3:
```
zcat 2017-05-29_03-15_carma.sql.gz | psql -h 127.0.0.1 -U carma_db_sync -d carma
```
You might need to apply some migrations if your database snapshot is older than current state of git-branch. To do so run:
```
(cd database && ./db.sh update)
```
Or if you use docker/dev-pg-9.3:
```
(cd database && env PGHOST=127.0.0.1 PGUSER=carma_db_sync PGPASSWORD=pass ./db.sh update-devel)
```

Build backend executables

Install Haskell Stack. Check that it works:
```
stack --numeric-version
# 1.6.5
```

Clone CaRMa Git repository:

git clone [email protected]:ruamk/carma
cd carma

Add log files:

mkdir -p srv/log && > srv/log/access.log && > srv/log/error.log

Build the backend:
```
stack build --install-ghc
```
P.S. When you rebuild backend on production servers you need to run
```
stack install
```
after to install executables to $HOME/.local/bin dir which is used to run backend from by init-scripts.
Copy default configuration files to your local directory (which is ignored from git-index):
```
cp -r srv/snaplets-default srv/snaplets
cp -r carma-mobile-server/snaplets-default carma-mobile-server/snaplets
```
Keep in mind that if you change database configuration or credentials to access to it you need to update these files (links points to default ones):
WARNING! Before going to next step you supposed to build frontend first, because templates for backend must be built by frontend toolchain before start.

Run the server from srv/ directory:

(cd srv && stack exec carma -- -p 8000)

Check that the server works:
```
curl localhost:8000/meta
```
WARNING! To work with Geo API (e.g. search on maps, addresses, etc.) you need to run Nominatim Mediator microservice, to do so locally:
1. Go to the directory of that package:
```
cd carma-nominatim-mediator
```
2. Copy default config:
```
cp app.cfg.default app.cfg
```
3. Run the microservice (being in carma-nominatim-mediator directory):
```
stack exec carma-nominatim-mediator
```
4. Now you can ashure that it's working by this command:
```
curl -v 'http://127.0.0.1:8165/reverse-search/ru-RU,ru/37.378847,55.558741'
```

macOS

On macOS with openssl and icu4c installed via Homebrew, build with

stack build --extra-include-dirs=/usr/local/opt/openssl/include/ --extra-include-dirs=/usr/local/opt/icu4c/include/ --extra-lib-dirs=/usr/local/opt/openssl/lib/ --extra-lib-dirs=/usr/local/opt/icu4c/lib/

Docker

Alternatively, to build the server inside Docker and package all executables in a container:

stack docker pull
stack --docker image container

If you're on Linux, you can just build with stack --docker build and run the app outside Docker. You will not be able to run the app built this way on macOS or Windows.

If the container has no access to your SSH keys (which may be the case when using a token), use stack dot prior to building the image to quickly fetch dependencies (.stack-work/downloaded directory is shared between the container and the host anyways).

Note that this container lacks frontend resources necessary to launch CaRMa - see below for how to build the whole bundle.

Frontend

Check that you have proper versions of node.js and npm:

printf 'node: %s, npm: %s\n' "`node --version`" "`npm --version`"
# node: v8.11.1, npm: 5.6.0

To build front-end from scratch (development build):

cd srv
npm install
npm run build
npm run build-backend-templates

If you need to just rebuild it again (after some changes for instance) run:

npm run build

To build bundle for production instead of for development run:

cd srv
npm install
npm run prod-build
npm run build-backend-templates

If you're about to deploy next release to the production server, just run (this includes npm i, clean-build-backend-templates and prod-clean-build):

npm run prod-release

You also could run a watcher in background to rebuild client code automatically when sources change by using this command (development mode):

npm run watch

To clean everything run:

npm run clean
npm run clean-backend-templates

There's also single command to do the same:

npm run full-clean

"Pure" frontend

WARNING! Do not forget to also build "pure" frontend, the commands look almost the same as you saw above. See "pure" frontend README for details.

This is partly rewritten frontend (some rewritten parts of it), some parts of it included to the main frontend using <iframe>s.

Ideally the whole frontend must be rewritten in this "pure" version.

CI build

CircleCI is configured to build CaRMa on every push. If you have Docker installed, you may use CircleCI CLI tool to perform builds in the same environment as on CI server:

circleci build --job build_client
circleci build --job build_server

Jobs run in an auto-removed container, so at the moment you may only use them to test that the build finishes successfully.

Docker bundle

This is an EXPERIMENTAL workflow

If you build the frontend and the backend Docker image, you can also build a bundle image containing both:

cd srv/
docker build -t carma-bundle .

CaRMa server executable inside the container is located at /usr/local/bin/carma.

This is what build_bundle CI step does.

Alternatively, you may skip building CaRMa altogether and just pick an image from ruamc/carma-bundle repo on Docker Hub.

To run the bundle, use carma-bundle.yml Docker Compose file, which combines a CaRMa bundle and PostgreSQL.

Required steps:

Create carma-db-data volume using docker/dev-pg-9.3 container.
Clone carma-configs and edit CaRMa configs path in volumes: section of carma-bundle.yml.

The command to run the bundle is

docker-compose -f docker/carma-bundle.yml up

Running

Get a recent database snapshot. Backup folder with daily production snapshots is mounted on carma-test server at /var/backups/allbackups/postgresql_carma:
```
scp <YOUR_LOGIN>@192.168.10.13:/var/backups/allbackups/postgresql_carma/2017-05-29_03-15_carma.sql.gz .
```

Badges

Extracted from project README

carma-ruamc-fork