MIT License
The backend API for OpenDOSM that serves data available via dosm-malaysia/aksara-data
to the frontend at dosm-malaysia/aksara-front
.
aksara
postgresql
<your python exe> -m venv env
# activate and install dependencies
source env/bin/activate
pip install -r requirements.txt
.env
file.python manage.py migrate
dosm-malaysia/aksara-data
repo and populate or update the DB: python manage.py loader UPDATE
python manage.py loader REBUILD
python manage.py runserver
API will be running at http://127.0.0.1:8000/
Despite OpenDOSM having multiple dashboards and data variables, there is only 1 endpoint, dedicated to each type, to serve all variables and dashboards.
dashboard/
'dashboard'
.dashboard
parameter indicates which dashboard API should be returned.handle_request
method to execute them accordingly.data-variable/
'id'
.'id'
represents the unique identifier for each variable, within the data catalog.chart/
'dashboard'
and 'chart'
.update/
data-catalog/
As of now, OpenDOSM uses GitHub Actions to trigger a rebuild (when there is new code) or update of data (when there is new data). Alternatively, with your preferred deployment setup, there are 2 ways to update the data.
https://<your url here>/update/
, which would by default trigger an update, and populate the DB with the new data.Alternatively, if your desired cron / task scheduler runs locally, you could use the command line to trigger an update and populate the DB with the new data.
python manage.py loader category operation files
category
- Possible values are either DASHBOARDS
or DATA_CATALOG
, depending on whichever you choose to update.
operation
- Possible values are either UPDATE
or REBUILD
. UPDATE
will update the datasets, by inserting new rows, or updating existing rows by their ID. Whereas REBUILD
would delete all rows from the database, and populate from an empty database.
files
- When using the UPDATE
operation, will update specific files of your choice. The values must be a string, concatenated with a ,
dividing each dataset. E.g : 'meta_1,meta_2,meta_3'
. There is no need to add the additional .json
suffix. This parameter is optional, and if left empty, will update every existing dataset of the chosen category.
aksara-data
repo.trigger
module for the loader to work without Telegram updates.Before explaining the logic of OpenDOSM's backend operations, here is a brief description of the thinking behind the architecture choices:
Therefore, to minimise complexity, it was vital that we used an architecture that could keep the number of endpoints minimal, while simultaneously ensuring that chart data could be served in as versatile a manner as possible. Here are the key ingredients we used to do this:
Meta Json version 1.0
A META Json contains all information required to build a dashboard. These files can be found within the dashboard
folder of the aksara-data
repo. Each one is responsible for either a dashboard, or a dedicated chart.
chart_source
keys contain data which is fed to the chart builders, which serves the final data.Catalog Json version 1.0
A Catalog Json contains all information required to build a data variable, or a set of data-variables. These files can be found within the catalog
forlder of the aksara-data
repo. Each one is responsible for 1 or more data variables.
file
key contains the description of the dataset, as well as the source of the data set.file
key, there is a key named variables
. This key, is an array of objects, containing the id, name, title, and description of each data-variable within the dataset.-1
: Indicates a variable should be used in the explanation, but has no dedicated chart / page for itself.0
: Indicates the variables excluded for a data-variable of the table chart.>= 1
: Indicates the variables included within the dataset, AND has a dedicated chart / page for itself.catalog_data
key, contains an array of objects, which represents the meta-data, chart variables and chart filters, required to build the respective data-variable page. These id in these variables will also correspond to the id within the variables
key, nested in the file
key.Chart Builders version 1.0
Critical design principle: All charts of the same type should be built using only 1 method. For instance, all time series charts on the site should (as far as possible) be handled by the same code unit.
utils
folder in the project, the chart_builder
file contains several methods for various charts, such as heatmaps, bar charts, choropleths, etc.variable_structures.py
file, within the utils
folder.Pull requests are welcome. For any pull request, please make sure to either create a new branch, or submit a pull request to the dev
branch. Please make sure to briefly describe the changes introduced in the pull request.