The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like a software artifact.
APACHE-2.0 License
Does one of these scenarios sounds familiar to you?
If so, you are one of the people we've built refinery for. refinery helps you to build better NLP models in a data-centric approach. Semi-automate your labeling, find low-quality subsets in your training data, and monitor your data in one place.
refinery doesn't get rid of manual labeling, but it makes sure that your valuable time is spent well. Also, the makers of refinery currently work on integrations to other labeling tools, such that you can easily switch between different choices.
DEMO: You can interact with the application in a (mostly read-only) online playground. Check it out here
refinery is a multi-repository project, you can find all integrated services in the architecture below. The app builds on top of 🤗 Hugging Face and spaCy to leverage pre-built language models for your NLP tasks, as well as qdrant for neural search.
There are already many other tools available to build training data. Why did we decide to build yet another one?
We believe that developers can have crazy ideas, and we want to lower the barrier for them to go for that idea. refinery is designed to build labeled training data much faster, so that it takes you very little time to prototype an idea. We've received much love for exactly that, so make sure to give it a try for your next project.
refinery is more than a labeling tool. It has a built-in labeling editor, but its main advantages come with automation and data management. You can integrate any kind of heuristic to label what is possible automatically, and then focus on headache-causing subsets afterwards. Whether you do the labeling in refinery or any other tool (even crowd labeled) doesn't matter!
refinery is the tool that brings new perspectives into your data. You're working on multilingual, human-written texts? Via our integration to bricks, you can easily enrich your texts with metadata such as the detected language, sentence complexity and many more. You can use this both to analyze your data, but also to orchestrate your labeling workflow.
While doing so, we aim to improve the collaboration between engineers and subject matter experts (SMEs). In the past, we've seen how our application was being used in meetings to discuss label patterns in form of labeling functions and distant supervisors. We believe that data-centric AI is the best way to leverage collaboration.
We hate the idea that there are still use cases in which the training data is just a plain CSV-file. That is okay if you really just quickly want to prototype something at hand with a few records, but any serious software should be maintainable. We believe an open-source solution for training data management is what's needed here. refinery is the tool helping you to document your data. That's how you treat training data as a software artifact.
Lastly, refinery supports SDK actions like pulling and pushing data. Data-centric AI redefines labeling to be more than a one-time job by giving it an iterative workflow, so we aim to give you more power every day by providing end-to-end capabilities, growing the large-scale availability of high-quality training data. Use our SDK to program integrations with your existing landscapes.
You can automate tons of repetitive tasks, gain better insights into the data labeling workflow, receive an implicit documentation for your training data, and can ultimately build better models in shorter time.
Our goal is to make training data building feel more like a programmatic and enjoyable task, instead of something tedious and repetitive. refinery is our contribution to this goal. And we're constantly aiming to improve this contribution.
If you like what we're working on, please leave a ⭐!
You won't believe how often we get that question - and it is a fair one 🙂 Put short, the open-source version of refinery is currently a single-user version, and you can get access to a multi-user environment with our commercial options. Additionally, we have commercial products on top of refinery, e.g. to use the refinery automations as an actual realtime prediction API.
Generally, we are passionate about open-source and want to contribute as much as possible.
For a detailed overview of features, please look into our docs.
pip install kern-refinery
Once the library is installed, go to the directory where you want to store the data and run refinery start
. This will automatically git clone
this repository first if you haven't done so yet. To stop the server, run refinery stop
.
TL;DR:
$ git clone https://github.com/code-kern-ai/refinery.git
$ cd refinery
If you're on Mac/Linux:
$ ./start
If you're on Windows:
$ start.bat
To stop, type ./stop
(Mac/Linux) or stop.bat
.
refinery consists of multiple services that need to be run together. To do so, we've set up a setup file, which will automatically pull and connect the respective services for you. The file is part of this repository, so you can just clone it and run ./start
(Mac/Linux) or start.bat
(Windows) in the repository. After some minutes (now is a good time to grab a coffee ☕), the setup is done and you can access http://localhost:4455
in your browser. To stop the server, run ./stop
(Mac/Linux) or ./stop.bat
(Windows).
You're ready to start! 🙌 🎉
If you run into any issues during installation, please don't hesitate to reach out to us (see community section below).
By default, we store the data to the directory refinery/postgres-data
. If you want to change that path, you need to modify the variable LOCAL_VOLUME
of the start
script of your operating system. To remove data, simply delete the volume folder. Make sure to delete only if you don't need the data any longer - this is irreversible!
The best way to start with refinery is our quick start.
You can find extensive guides in our docs and tutorials on our YouTube channel. We've also prepared a repository with sample projects which you can clone.
If you need help writing your first labeling functions, look into our open-source content library bricks.
You can find our changelog here.
No worries, we've got you. If you have questions, reach out to us on Discord, or open a ticket in the "q&a" category of our forum.
Feel free to join our Discord, where we'll happily help you building your training data:
We send out a (mostly) weekly newsletter about recent findings in data-centric AI, product highlights in development and more. You can subscribe to the newsletter here.
Also, you can follow us on Twitter and LinkedIn.
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated. You can do so by providing feedback about desired features and bugs you might detect.
If you actively want to participate in extending the code base, reach out to us. We'll explain you how the architecture is set up, so you can customize the application as you desire.
[
{
"running_id": "0",
"headline": "T. Rowe Price (TROW) Dips More Than Broader Markets",
"date": "Jun-30-22 06:00PM\u00a0\u00a0",
"headline__sentiment__MANUAL": null,
"headline__sentiment__WEAK_SUPERVISION": "NEGATIVE",
"headline__sentiment__WEAK_SUPERVISION__confidence": 0.62,
"headline__entities__MANUAL": null,
"headline__entities__WEAK_SUPERVISION": [
"STOCK", "STOCK", "STOCK", "STOCK", "STOCK", "STOCK", "O", "O", "O", "O", "O"
],
"headline__entities__WEAK_SUPERVISION__confidence": [
0.98, 0.98, 0.98, 0.98, 0.98, 0.98, 0.00, 0.00, 0.00, 0.00, 0.00
]
}
]
You can extend your projects by using our Python SDK. With it, you can easily export labeled data of your current project and import new files both programmatically and via CLI (rsdk pull
and rsdk push <file_name>
). It also comes with adapters, e.g. to Rasa.
Our architecture follows some main patterns:
Service overview (maintained by Kern AI)
Service | Description |
---|---|
ml-exec-env | Execution environment for the active learning module. Containerized function as a service to build active learning models using scikit-learn and sequence-learn. |
embedder | Embedder for refinery. Manages the creation of document- and token-level embeddings using the embedders library. |
weak-supervisor | Weak supervision for refinery. Manages the integration of heuristics such as labeling functions, active learners or zero-shot classifiers. Uses the weak-nlp library for the actual integration logic and algorithms. |
record-ide-env | Execution environment for the record IDE. Containerized function as a service to build record-specific "quick-and-dirty" code snippets for exploration and debugging. |
config | Configuration of refinery. Amongst others, this manages endpoints and available language models for spaCy. |
tokenizer | Tokenizer for refinery. Manages the creation and storage of spaCy tokens for text-based record attributes and supports multiple language models. |
gateway | Gateway for refinery. Manages incoming requests and holds the workflow logic. To interact with the gateway, the UI or Python SDK can be used. |
authorizer | Evaluates whether a user has access to certain resources. |
websocket | Websocket module for refinery. Enables asynchronous notifications inside the application. |
lf-exec-env | Execution environment for labeling functions. Containerized function as a service to execute user-defined Python scripts. |
ac-exec-env | Execution environment for attribute calulaction. Containerized function as a service to generate new attributes using Python scripts. |
updater | Updater for refinery. Manages migration logic to new versions if required. |
neural-search | Neural search for refinery. Manages similarity search powered by Qdrant and outlier detection, both based on vector representations of the project records. |
zero-shot | Zero-shot module for refinery. Enables the integration of 🤗 Hugging Face zero-shot classifiers as an off-the-shelf no-code heuristic. |
entry | Login and registration screen for refinery. Implemented via Ory Kratos. |
ui | UI for refinery. Used to interact with the whole system; to find out how to best work with the system, check out our docs. |
doc-ock | Usage statistics collection for refinery. If users allow it, this collects product insight data used to optimize the user experience. |
gateway-proxy | Gateway proxy for refinery. Manages incoming requests and forwards them to the gateway. Used by the Python SDK. |
parent-images | Shared images used by refinery. Used to reduce the required space for refinery. Not yet listed in architecture diagram |
ac-exec-env | Execution environment for attribute calculation in refinery. Containerized function as a service to build custom attributes derived from the original data. Not yet listed in architecture diagram |
alfred | Controls the start process of the refinery app. Named after Batman's butler Alfred. Not yet listed in architecture diagram |
Service overview (open-source 3rd party)
Service | Description |
---|---|
qdrant/qdrant | Qdrant - Vector Search Engine for the next generation of AI applications |
postgres/postgres | PostgreSQL: The World's Most Advanced Open Source Relational Database |
minio/minio | Multi-Cloud ☁️ Object Storage |
mailhog/MailHog | Web and API based SMTP testing |
ory/kratos | Next-gen identity server (think Auth0, Okta, Firebase) with Ory-hardened authentication, MFA, FIDO2, TOTP, WebAuthn, profile management, identity schemas, social sign in, registration, account recovery, passwordless. Golang, headless, API-only - without templating or theming headaches. Available as a cloud service. |
ory/oathkeeper | A cloud native Identity & Access Proxy / API (IAP) and Access Control Decision API that authenticates, authorizes, and mutates incoming HTTP(s) requests. Inspired by the BeyondCorp / Zero Trust white paper. Written in Go. |
Integrations overview (maintained by Kern AI)
Integration | Description |
---|---|
refinery-python | Official Python SDK for Kern AI refinery. |
sequence-learn | With sequence-learn, you can build models for named entity recognition as quickly as if you were building a sklearn classifier. |
embedders | With embedders, you can easily convert your texts into sentence- or token-level embeddings within a few lines of code. Use cases for this include similarity search between texts, information extraction such as named entity recognition, or basic text classification. Integrates 🤗 Hugging Face transformer models |
weak-nlp | With weak-nlp, you can integrate heuristics like labeling functions and active learners based on weak supervision. Automate data labeling and improve label quality. |
Integrations overview (open-source 3rd party)
Integration | Description |
---|---|
huggingface/transformers | 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. |
scikit-learn/scikit-learn | scikit-learn: machine learning in Python |
explosion/spaCy | 💫 Industrial-strength Natural Language Processing (NLP) in Python |
Submodules overview
Not listed in the architecture, but for internal code management, we apply git submodules.
Submodule | Description |
---|---|
submodule-model | Data model for refinery. Manages entities and their access for multiple services, e.g. the gateway. |
submodule-s3 | S3 related AWS and Minio logic. |
Term | Meaning |
---|---|
Weak supervision | Technique/methodology to integrate different kinds of noisy and imperfect heuristics like labeling functions. It can be used not only to automate data labeling, but generally as an approach to improve your existing label quality. |
Neural search | Embedding-based approach to retrieve information; instead of telling a machine a set of constraints, neural search analyzes the vector space of data (encoded via e.g. pre-trained neural networks). Can be used e.g. to find nearest neighbors. |
Active learning | As data is labeled manually, a model is trained continuously to support the annotator. Can be used e.g. stand-alone, or as a heuristic for weak supervision. |
Vector encoding (embedding) | Using pre-trained models such as transformers from 🤗 Hugging Face, texts can be transformed into vector space. This is both helpful for neural search and active learning (in the latter case, simple classifiers can be applied on top of the embedding, which enables fast re-training on the vector representations). |
Missing anything in the glossary? Add the term in an issue with the tag "enhancement".
refinery is licensed under the Apache License, Version 2.0. View a copy of the License file.