APACHE-2.0 License
This repository contains some experiments on what a modern data platform could look like. A strong emphasis lies on how observability can be achieved throughout all actions occurring on a data platform.
Currently the documentation is not published, but can be viewed in this repository.
mkdocs serve
To run unit tests for the rust crates execute
cargo test
To build crates and services
cargo build
TODO explain mock framework
To set up the python environment run
poetry install
The python bindings for rust dependent crates have to be build and installed separately. Make sure you have the environment created by poetry activated and run.
make python-develop
To see the Dagster integrations in play, create two folders in the project root: .dagster
and .fusion
,
then start an instance of the fusion service in a separate shell.
fusion server start --host 127.0.0.1 --port 50051 --log-level info
An example Dagster repository using most of the features from the dagster-fusion
package is
provided within examples/model-training
. To inspect it and play with the configurations,
run a local instance of dagster.
docker build -f docker/Dockerfile
docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest
To generate proto definitions for use in python client.
make python-proto