Streamdal is an open-source 'Code-Native Data Privacy' solution for running data tasks directly in your application code.
Think of it as a "workflow engine" or a "pre/post data processor" that is executed client-side via WebAssembly in your application code.
It is at least 10x faster, 10x cheaper and 10x easier to operate than traditional data pipelines.
This is what it looks like:
Manifesto Benefits Demo Getting Started How Does It Work? Community Resources
There are major benefits to running pipelines directly within your app:
You don't have to install the server, the console or instrument any of your apps to see Streamdal in action. We've got a live demo :)
While the demo is read-only, it will give you a good idea about the capabilities of Streamdal.
The demo showcases real-time:
You can read more about how this is achieved in the "how does it work?" docs.
Getting started consists of two steps:
The easiest way to get Streamdal running is via curl | bash
:
curl -sSL https://sh.streamdal.com | bash
git
, docker
and docker-compose
installed~/streamdal
docker-compose
Once done:
Openhttp://localhost:8080
in your browser!
You should be presented with a beautiful (but empty) UI! To populate it, we will need to instrument some code. Onto the next section!
For alternative installation methods, check the docs dir.
Once you have installed the server and console, you can begin instrumenting your code.
Instrumentation involves calling our SDKs .Process()
method either after
your application reads data from somewhere OR before your app is about to
write data somewhere.
To see an example of a complete instrumentation, take a look at one of our shim examples or the Go demo client that is bundled with the ./apps/server.
Streamdal consists of three main components:
The basic flow is that you install the server and console and wrap any reads or writes in your app with one of our SDKs. Once that's done, you will be able to see the app and the data your app is reading or writing in the console (or use the CLI).
You will also be able to enforce rules on your data (such as "this should be
valid JSON", "message should contain a field called foo
", "strip all email
addresses" and so on).
[!IMPORTANT] For a more in-depth explanation of the flow and the various components, visit our docs.
This repo is a monorepo that has the following layout and usage:
# assets <--- Static assets
# img
# ...
# apps
# cli <--- CLI UI
# console <--- Web-based UI
# docs <--- https://docs.streamdal.com
# server <--- Server component
# ...
# docs
# install
# bare-metal
# docker
# ...
# | instrument
# | ...
# libs
# protos <--- Common protobuf schemas
# wasm <--- Wasm funcs used in pipeline steps
# wasm-detective <--- Wasm lib used for data parsing and validation
# wasm-transformer <--- Wasm lib used for data transformation
# ...
# scripts
# install
# install.sh <--- Install script for installing Streamdal
# ...
# LICENSE
# Makefile <--- Makefile with common tasks; run `make help` for more info
# README.md
The Streamdal platform is made up of several components that must all be at specific versions to function properly together.
When running in production, verify versions of the components against this list:
Date | Server | Console | Protos | Wasm | Helm |
---|---|---|---|---|---|
05.08.2024 | 0.0.38 | 0.0.69 | 0.1.56 | 0.1.38 | 0.1.61 |
We're building Streamdal in the open and we'd love for you to join us!
Join our Discord!
Stuck? Something not working right? Have questions?
You have control over what we're building - our roadmap is 100% public!
Feel free to stop by to discuss features, suggest new ones or just keep an eye on what's in the pipeline.
We ❤️ contributions! But... before you craft a beautiful PR, please read through our contributing docs.
This project is licensed under the Apache-2.0
license.
See the LICENSE file for more info.