argilla

0.13.2 (2022-04-12)

Bug Fixes

• #1265: persist pagination size after query (#1358) (49ca243), closes #1265
• #1367: remove record text from metadata modal (#1385) (1782724), closes #1367
• #1368: long list of entities in Token Classifier (#1388) (829269f), closes #1368 #1393
• #1387: improve metadata distinct values computation (be9f68f), closes #1387
• install: remove loguru dependency (#1372) (9e52414), closes #1331 #1305
• search: compute dataset schema properly for advanced query dsl (#1380) (f71ab91)
• visualization: force break word in selectors (#1406) (5ac1950)

0.13.1 (2022-04-01)

Bug Fixes

• #1244: compute capitalness based on python methods (#1359 #1371) (218f099), closes #1244
• #1362: using active api method instead instance (#1363) (bcf446d), closes #1362
• #1365: create rules with regex queries (#1369) (c2afc9c), closes #1365

0.13.0 (2022-03-30)

🗂 Multilabel weak supervision

You can now build multilabel text classification datasets using query-based rules

If you want to get started, check out this tutorial.

https://user-images.githubusercontent.com/1107111/160930404-7b909f1e-b871-4e4c-b1c8-ea9eabfcad21.mp4

🤗 Reading Hugging Face datasets from the Hub

You can now read ANY text classification, NER, or text2text dataset directly from the Hub and load it into Rubrix.

To understand how Rubrix datasets work check out this guide.

👥 Redesigned team workspaces

Organizing teams and datasets is a key Rubrix feature. After several rounds of feedback with early users, we've completely redesigned the user experience. Let us know what you think.

You can get started and configure users and workspaces following this guide

🔎 Guide for the query language and model

We have included a new in-depth guide about the Lucene-based query language and data model used for search, weak labeling, loading subsets of data, and metrics.

Features

• #1119: users without personal datasets (#1282) (555d41d), closes #1119 #1318 #1317 #1323 #1324
• #1130: cleanup rb namespace by refactoring client API (#1160) (a0fdd8e), closes #1130
• #1144: weak supervision for multilabel datasets (#1166) (fd95bae), closes #1144 #1190 #1237 #1233 #1326
• datasets: simplify load flow from hf datasets with no rb format (#1234) (a6da1cd), closes #1327
• #1180: show Rubrix version in the webapp (#1243) (8c71ad9), closes #1180 #1350 #1349
• #1225: prepare tokenclass dataset for hf training (#1231) (ae5e7cd), closes #1225
• #950: using record search_keywords for highlighting (#1235) (47616bf), closes #950 #1278 #1316 #1315
• #981: add majority voter with multi label support (#1228) (8052aa8), closes #981
• Introduce a 'text' argument for the TextClassificationRecord (#1246) (bb7d93e)

Bug Fixes

• #1347: allow tooltip record overlapping in Token Classifier (#1352) (87174d3), closes #1347
• #1103: remove "Error Distribution" from metrics (#1255) (b9bb5b4), closes #1103
• #1149: fix vulnerable dependencies (node-sass) (#1263) (7f8c1d1), closes #1149
• #1211: fix score scale (#1261) (8a72281), closes #1211
• #1238: show prediction labels when annotating rule (#1239) (0321b88), closes #1238
• #1241, #1245: show new line char in metrics plot & increase mentions in entity consistency (#1257) (38930cb), closes #1241 #1245
• #1311: small defects about hover style (#1313) (442703c), closes #1311
• #1320: render car return in Token Classifier (#1328) (b7f1b7b), closes #1320
• #1335: force line break in rules summary (#1336) (2d77a76), closes #1335
• #1337: number of records in the overall annotated coverage (#1338) (d384713), closes #1337
• #1339: metrics and status not updated when the query is refreshed (#1340) (6fc0a58), closes #1339
• #984: manage super user workspaces (#1268) (9b24921), closes #984 #1288 #1290
• datasets: prevent error when no annotated records found in dataset (#1284) (c20028f)
• install: make starlette an optional dependency (#1295) (32afb3d)
• NER: create record annotation from tags (also in from_datasets) (#1283) (adcf1b1)
• rules: store single-label rules with a comp. format for old versions (#1334) (eb310d3)

0.12.1 (2022-03-11)

Bug Fixes

• #1238: show prediction labels when annotating rule (#1239) (6c1b975), closes #1238

0.11.1 (2022-03-11)

Bug Fixes

• #1238: show prediction labels when annotating rule (#1239) (28e97c6), closes #1238

0.12.0 (2022-03-08)

Features

• #1029: improve server api logging (#1148) (d4a121a), closes #1029 #1224
• #1183: token classification fine-tuning (#1199) (2cdd30b), closes #1183
• #1192: disable ssl verify for elasticsearch http client (#1193) (631a729), closes #1192
• #950: include search keywords as part of record results (#1201) (2dd5853), closes #950
• #970: header redesign (#1185) (fa9c639), closes #970 #1218 #1214 #1223
• Implement 'prepare_for_training' for text classification datasets (#1209) (f7fd59c)

Bug Fixes

• #1207: using api sdk wrapper for init (#1208) (2495c75), closes #1207

0.10.0 (2022-02-12)

Now you can use filters in the Define Rules mode (weak labeling). These filters are useful for seeing the impact of rules on specific dataset subpopulations/subsets (e.g., with certain metadata fields, annotated records, etc.):

Features

• #1061: unify records results title (#1111) (54ebb15), closes #1061
• #982: show filters in labelling rules view (#1038) (7ff677b), closes #982

Bug Fixes

• #1054: reduce collapsable area. Optimize for annotation (#1106) (48024ba), closes #1054
• #1054: remove old scroll padlock button (a1d6444), closes #1054
• #1094: remove computed record fields returned in API results (#1095) (cd61d1e), closes #1094
• #831: Remove sort field when only one is applied (#1116) (36b276b), closes #831
• convert pd.NaT to None for event_timestamp (#1105) (21e78e4)

0.11.0 (2022-02-19)

Highlights

Introducing rb.Dataset* and 🤗 Hub integration

The Dataset classes are lightweight containers for Rubrix records. These classes facilitate importing from and exporting to different formats (e.g., pandas.DataFrame, datasets.Dataset) as well as sharing and versioning Rubrix datasets using the Hugging Face Hub.

With this release, Rubrix users and teams can use the Hugging Face Hub to share and read both public and private Rubrix datasets for TextClassification, TokenClassification, and Text2Text datasets. This opens up a whole new world of possibilities for data reproducibility and sharing. Let's see an example:

import rubrix as rb
from datasets import load_datasets

# 👧🏻 🏷️ Leire has labeled a text classification dataset using a local Rubrix instance
dataset_rb = rb.load("text_classification_ds", as_pandas=False)

# 👧🏻 exports a Rubrix Dataset to a hf Dataset
dataset_ds = dataset_rb.to_datasets()

# 👧🏻 🚀 Leire shares the labelled dataset with the world 
dataset_ds.push_to_hub("text_classification_ds")

# 👨 John downloads the dataset from the Hugging Face Hub
dataset_ds = load_dataset("leire/text_classification_ds", split="train")

# 👨 reads in dataset
dataset_rb = rb.read_datasets(dataset_ds, task="TextClassification")

# 👨 🏷️ logs the dataset and continues labeling with his own Rubrix instance
rb.log(dataset_rb, "john_text_classification_ds")

You can read more at https://rubrix.readthedocs.io/en/stable/guides/datasets.html

For each record type, there’s a corresponding Dataset class called DatasetFor<RecordType>. You can look up their API in the reference section.

Improving NER UI and UX

The UI for Token Classification has been completely redesigned to provide a better user experience for exploration and annotation. This is the first of a set of changes focusing on annotation productivity for token classification.

Features

• #1051: keep predictions labels when annotating (#1077) (f1824ba), closes #1051
• #1063: Token Classifier fine tuning content selection (#1084) (9e14d05), closes #1063
• #1127: raise startup app error from es connection error (#1145) (7e7e9d8), closes #1127
• #422: introducing the rb.Dataset* classes (#1109) (b5bbca6), closes #422
• #821: token classifier show predictions in explore view (#1009) (6ba6764), closes #821
• #951: new "not covered records by rules" filter (#991) (0649f2a), closes #951 #1156

Bug Fixes

• #1140: fix/make client models more consistent (#1147) (926bb16), closes #1140
• client: parse unauthorized api error properly (#1164) (1a5a08d)
• search: prevent metrics computation breaks searches (#1175) (9f2adc9)

🎉 0.9.0 (2022-02-02)

• Improve logging
• Small improvements to the labelling module and weak labeling mode
• Better setup documentation (python -m rubrix)

Features

• #932: label models now modify the prediction_agent when calling LabelModel.predict (#1049) (4a024ee), closes #932
• #953: add additional metrics to LabelModel.score method (#979) (2887907), closes #953
• #955: add default for rules in WeakLabels (#976) (34389d3), closes #955 #1011

Bug Fixes

• #1045: calculate overall precision from overall correct/incorrect in rules (#1086) (1c76d81), closes #1045 #1087
• #1053: metadata modal position (#1068) (09b88cc), closes #1053 #1053
• #1054: optimize Long records (#1080) (fdd797a), closes #1054
• #1067: fix rule definition link when no labels are defined (#1069) (eb958bf), closes #1067
• #1081: prevent add records of different task (#1085) (5296e52), closes #1081 #1081
• #924: parse new error format in UI (#1082) (f26c79c), closes #924

0.8.2 (2022-01-31)

Features

• #1036: remove prediction ok/ko in labelling rules (#1037) (672b852), closes #1036
• #735: add warning when agent but no prediction/annotation is provided (#987) (ba88c34), closes #735

Bug Fixes

• #1008: set the event_timestamp when annotating (#1024) (c24fdad), closes #1008
• #1015: manage emojis in Token Classification records (#1016) (8b570fb), closes #1015
• #1023: handle elasticsearch connection problems on server startup (#1030) (e8c8d86), closes #1023
• #1027: Improve client models by reordering fields + forbidding extra args (#1032) (6c1ae7f), closes #1027
• #1028: Add videos to Monitoring tutorial (#1033) (6ff3326), closes #1028
• #1050: generalizes entity span validation (#1055) (37207bc), closes #1050
• #1058: sort by % data in rules list (#1062) (9735f22), closes #1058
• #1065: 'B' tag for beginning tokens (#1066) (a5ed329), closes #1065
• cleanlab: set cleanlab n_jobs=1 as default (#1059) (189cbcb)

0.8.1 (2022-01-20)

Bug Fixes

• #1002: Show 0 records overall metrics when no rules defined (#1013) (a8a5c79), closes #1002 #1002
• Breadcrumbs: copy workspace from the breadcrumbs when dataset loading has errors #1003 (33e372d), closes #844
• statics: handle 404 errors for static files (#1006) (f4b656a)
• #800: compute common aggregations one by one (#990) (8cf420a), closes #800
• #800: limit number of metadata fields (#993) (bb6b76b), closes #800
• #905: copy dataset with rules (#948) (8597b83), closes #905
• #974: display the dropdown in the last record of the scroll (#986) (e5f8d53), closes #974
• #977: Remove redirection when accessing login (#996) (b3fe2cb), closes #977

0.8.1-alpha.2 (2022-01-20)

Bug Fixes

• #1002: Show 0 records overall metrics when no rules defined (#1007) (a890e17), closes #1002 #1002
• Breadcrumbs: copy workspace from the breadcrumbs when dataset loading has errors #1003 (33e372d), closes #844
• statics: handle 404 errors for static files (#1006) (f4b656a)

0.8.1-alpha.0 (2022-01-19)

Bug Fixes

• #800: compute common aggregations one by one (#990) (8cf420a), closes #800
• #800: limit number of metadata fields (#993) (bb6b76b), closes #800
• #905: copy dataset with rules (#948) (8597b83), closes #905
• #974: display the dropdown in the last record of the scroll (#986) (e5f8d53), closes #974
• #977: Remove redirection when accessing login (#996) (b3fe2cb), closes #977

Introducing interactive Weak labeling (Define rules mode) 🚀

We are glad to introduce the most important feature to date: now it's possible to iterate on labeling queries directly in the UI with initial support for multi-class text classification. Multilabel and token classification support is coming soon.

See the video for the recommended workflow:

https://user-images.githubusercontent.com/1107111/149346471-93cbd7ee-96a2-451a-8f5e-f9e26b246407.mp4

Check the updated tutorial: https://rubrix.readthedocs.io/en/master/tutorials/weak-supervision-with-rubrix.html

What's changed

• [WeakSupervision] Change load_rules import path in guide and tutorial (#939)
• fix links to new web app reference (#936)
• Bugfixes/avoid infinite loop when dataset loading (#934)
• show nan instead of 0 for precision in summary (#930)
• fix(api): include_metrics param only for search endponts (#929)
• [Documentation] Update title page video for docs (#928)
• update skweak tutorial (#922)
• [Documentation] Updating the web app docu (#827)
• publish python package to test.pypi for master and releases branches (#927)
• [WeakLabels] Align WeakLabels.summary() with web app (#925)
• UI: show rules without precision properly (#919)
• chore(build): build docker images for release branches (#921)
• Docs: Updates readme front video (#923)
• Docs: Updates weak supervision resources (#920)
• feat(rules): compute total & ann. coverage before label selection (#916)
• fix(rules): compute annotated coverage when no label properly (#915)
• Tutorial: Human-in-the-loop weak supervision with skweak (#869)
• UI: include affected #records to overall coverage/ann. coverage metrics (#914)
• fix lint build (#913)
• UI: manage precision and rules without annotation coverage (#909)
• fix(#876): process 400 response detail properly (#889)
• feat(rules): allow compute partial query rule metrics (#907)
• fix(security): providing default workspace should pass check (#911)
• UI: reset filters from define rules view (#908)
• UI: Show number of created rules in rules management view (#910)
• UI: drop access to rule name field (#904)
• fix(rules): prevent lost rules with dataset updates (#892)
• fix(datasets): process owner as part of dataset id (#870)
• (UI) Rules summary metrics format (#888)
• UI: Improve code snippet for empty workspace (#886)
• fix(UI): Remove case sensitive when filtering labels (#882)
• Docs: Updates Flair zeroshot tutorial (#887)
• removing wrong video (#885)
• Update readme (#883)
• fix(UI) Metrics value by default if no metric (#875)
• feat(metrics): add token level metrics for token classification from client (#849)
• UI: New rule metrics layout (#861)
• chore: expose load_rules from base module (#866)
• Docs: Regenerates graphs metrics guide (#865)
• updating loss video (#864)
• Docs: Update weak supervision guide (#863)
• Update README.md (#862)
• Fix: Link loss tutorial (#859)
• Docs: Improve loss tutorial (#858)
• Docs: Improve AL and ws tutorials (#857)
• chore(ci): Include component testing configuration (#839)
• fix/loss video updated (#853)
• Docs: Weak supervision guide update (#855)
• chore(app): upgrade lint dependencies (#841)
• feat: weak supervision mode (#814)
• Docs: Review hf tutorial (#852)
• fix: error link to workspace home (#845)
• fix(metrics): compute token length for each token (#850)
• add streaming (#851)
• fix(rules): prevent division by 0 for overall metrics (#848)
• small change
• [Tutorials] Update media structure, remove TLDR heading (#847)
• Updating videos and images for sentiment classification tutorial (#846)
• fix(rules): prevent division by zero (#843)
• new folder and videos for model loss tutorial (#805)
• feat(token class): add metrics at token level (#838)
• new folder and images for active learning tutorial (#796)
• [Tutorials] Typo fix in find label errors tutorial (#842)
• [Tutorials] Add the new find_label_errors tutorial (#833)
• [Rule] Modify the client API to the server's weak supervision feature (#840)
• [LabelModel] Improve Snorkel to not modify the passed in WeakLabels object (#836)
• feat (search): allow to filtering record metrics fields in search (#837)
• fix(ui): remove workspace home from code snippet api url (#834)
• ui: Hide validate button for binary cases in Text classifier (#830)
• fix print message (#829)
• feat: Include workspace in url path (#820)
• fix(ui): align records and global action layouts #825
• fix(ui): Show labels as selected after validate (#826)
• feat(labeling rule): implements api endpoint to fetch a single rule (#817)
• [LabelErrors] Add find_label_errors method (#775)
• fix(ui): Fix styles in Safari (#815)
• docs: Add contributors to readme (#822)
• add missing rubrix import (#819)
• new folder and images for spacy tutorial (#794)
• feat(labeling rules): allow edition for rule label and description (#813)
• refactor(labeling rules): optional label for rule metrics (#811)
• Fix token alignment on CreationTokenClassificationRecord (#812)
• feat(server): add overall dataset labeling rules metrics (#807)
• feat(labeling rules): add coverage for annotated records (#806)
• fix(ui): Unique ID for scroll state to avoid same state for different dataset records (#809)
• new folder and images for zeroshot ner tutorial (#804)
• new folder and images for zeroshot data annotation tutorial (#803)
• fix(log): check multi-label integrity without search aggregations (#802)
• updated images, added folder for fastapi tutorial (#801)
• added folder for weak supervision tutorial (#795)
• feat(weak supervision): client labeling rules from server (#799)
• feat(server): labeling rule metrics (#790)
• fix/edit zero-shot tutorial (#774)
• fix/edited fastapi tutorial (#773)
• Fix/edit ner flair tutorial (#766)
• Fix/edit weaksupervision tutorial (#759)
• fix(ui): Little changes in fonts (#793)
• fix(ui): Allow open dataset in new tab from datasets list (#792)
• feat(server): rubrix namespaces for elasticsearch indices (#789)
• fix(ui): Show annotation after global validation (#786)
• remove reload arg launching server using python (#787)
• updated readme with conda install instruction (#788)
• fix(ui): Hide scroller component when loading or paginate (#784)
• fix(ui): allow remove metadata filter from record metadata modal (#772)
• fix(ui): Token Classifier: validate record without annotation or prediction (#782)
• Fix/edit active learning tutorial (#760)
• Docs:minor changes to loss tutorial (#778)
• Fix/edit model loss tutorial (#767)
• fix(server): missing deprecated dep (#777)
• fix(ui): Global validate for records without annotation or prediction (#746)
• Fix/edit spacy tutorial (#758)
• Fix/edit labeling tutorial (#750)
• fix(server) - misaligned entity mentions on CreationTokenClassificationRecord (#771)
• [Requirements] Require python>=3.7 (#770)
• [Labeling] Add FlyingSquid label model (#755)
• Update README.md (#769)
• Adds Flair example to guide (#762)
• docs: Updates huggingface examples and adds monitor for Flair (#761)
• feat(search): show boolean values in metadata (#753)
• feat(server): allow handle labeling rules for datasets from API (#744)
• fix(imports): import monitoring with spacy<3.0 fails (#754)
• [UI] new fonts families (#751)
• fix(scroll): using new scroll component (#710)
• fix(ui): filter "validatable" records for global action validate button (#741)
• feat(monitor): flair ner auto-monitor (#738)

New Contributors

• @sugatoray made their first contribution
• @ruanchaves made their first contribution

• Bugfixes/avoid infinite loop when dataset loading (#934)
• show nan instead of 0 for precision in summary (#930)
• fix(api): include_metrics param only for search endponts (#929)
• [Documentation] Update title page video for docs (#928)
• update skweak tutorial (#922)
• [Documentation] Updating the web app docu (#827)
• revert test.pypi publish
• publish python package to test.pypi for master and releases branches (#927)
• [WeakLabels] Align WeakLabels.summary() with web app (#925)
• UI: show rules without precision properly (#919)
• chore(build): build docker images for release branches (#921)
• Docs: Updates readme front video (#923)
• Docs: Updates weak supervision resources (#920)
• feat(rules): compute total & ann. coverage before label selection (#916)
• fix(rules): compute annotated coverage when no label properly (#915)
• Tutorial: Human-in-the-loop weak supervision with skweak (#869)
• UI: include affected #records to overall coverage/ann. coverage metrics (#914)
• fix lint build (#913)
• UI: manage precision and rules without annotation coverage (#909)
• fix(#876): process 400 response detail properly (#889)
• feat(rules): allow compute partial query rule metrics (#907)
• fix(security): providing default workspace should pass check (#911)
• UI: reset filters from define rules view (#908)
• UI: Show number of created rules in rules management view (#910)
• UI: drop access to rule name field (#904)
• fix(rules): prevent lost rules with dataset updates (#892)
• fix(datasets): process owner as part of dataset id (#870)
• (UI) Rules summary metrics format (#888)
• UI: Improve code snippet for empty workspace (#886)
• fix(UI): Remove case sensitive when filtering labels (#882)
• Docs: Updates Flair zeroshot tutorial (#887)
• removing wrong video (#885)
• Update readme (#883)
• fix(UI) Metrics value by default if no metric (#875)
• feat(metrics): add token level metrics for token classification from client (#849)
• UI: New rule metrics layout (#861)
• chore: expose load_rules from base module (#866)
• Docs: Regenerates graphs metrics guide (#865)
• updating loss video (#864)
• Docs: Update weak supervision guide (#863)
• Update README.md (#862)
• Fix: Link loss tutorial (#859)
• Docs: Improve loss tutorial (#858)
• Docs: Improve AL and ws tutorials (#857)
• chore(ci): Include component testing configuration (#839)
• fix/loss video updated (#853)
• Docs: Weak supervision guide update (#855)
• chore(app): upgrade lint dependencies (#841)
• feat: weak supervision mode (#814)
• Docs: Review hf tutorial (#852)
• fix: error link to workspace home (#845)
• fix(metrics): compute token length for each token (#850)
• chore: improve dockerignore files
• add streaming (#851)
• fix(rules): prevent division by 0 for overall metrics (#848)
• small change
• [Tutorials] Update media structure, remove TLDR heading (#847)
• Updating videos and images for sentiment classification tutorial (#846)
• fix(rules): prevent division by zero (#843)
• new folder and videos for model loss tutorial (#805)
• feat(token class): add metrics at token level (#838)
• new folder and images for active learning tutorial (#796)
• [Tutorials] Typo fix in find label errors tutorial (#842)
• [Tutorials] Add the new find_label_errors tutorial (#833)
• [Rule] Modify the client API to the server's weak supervision feature (#840)
• [LabelModel] Improve Snorkel to not modify the passed in WeakLabels object (#836)
• feat (search): allow to filtering record metrics fields in search (#837)
• fix(ui): remove workspace home from code snippet api url (#834)
• ui: Hide validate button for binary cases in Text classifier (#830)
• fix print message (#829)
• feat: Include workspace in url path (#820)
• fix(ui): align records and global action layouts #825
• fix(ui): Show labels as selected after validate (#826)
• feat(labeling rule): implements api endpoint to fetch a single rule (#817)
• [LabelErrors] Add find_label_errors method (#775)
• fix(ui): Fix styles in Safari (#815)
• docs: Add contributors to readme (#822)
• add missing rubrix import (#819)
• new folder and images for spacy tutorial (#794)
• feat(labeling rules): allow edition for rule label and description (#813)
• refactor(labeling rules): optional label for rule metrics (#811)
• Fix token alignment on CreationTokenClassificationRecord (#812)
• feat(server): add overall dataset labeling rules metrics (#807)
• feat(labeling rules): add coverage for annotated records (#806)
• fix(ui): Unique ID for scroll state to avoid same state for different dataset records (#809)
• new folder and images for zeroshot ner tutorial (#804)
• new folder and images for zeroshot data annotation tutorial (#803)
• fix(log): check multi-label integrity without search aggregations (#802)
• updated images, added folder for fastapi tutorial (#801)
• added folder for weak supervision tutorial (#795)
• feat(weak supervision): client labeling rules from server (#799)
• feat(server): labeling rule metrics (#790)
• fix/edit zero-shot tutorial (#774)
• fix/edited fastapi tutorial (#773)
• Fix/edit ner flair tutorial (#766)
• Fix/edit weaksupervision tutorial (#759)
• fix(ui): Little changes in fonts (#793)
• fix(ui): Allow open dataset in new tab from datasets list (#792)
• feat(server): rubrix namespaces for elasticsearch indices (#789)
• fix(ui): Show annotation after global validation (#786)
• remove reload arg launching server using python (#787)
• updated readme with conda install instruction (#788)
• fix(ui): Hide scroller component when loading or paginate (#784)
• fix(ui): allow remove metadata filter from record metadata modal (#772)
• fix(ui): Token Classifier: validate record without annotation or prediction (#782)
• Fix/edit active learning tutorial (#760)
• Docs:minor changes to loss tutorial (#778)
• Fix/edit model loss tutorial (#767)
• fix(server): missing deprecated dep (#777)
• fix(ui): Global validate for records without annotation or prediction (#746)
• Fix/edit spacy tutorial (#758)
• Fix/edit labeling tutorial (#750)
• fix(server) - misaligned entity mentions on CreationTokenClassificationRecord (#771)
• [Requirements] Require python>=3.7 (#770)
• [Labeling] Add FlyingSquid label model (#755)
• Update README.md (#769)
• Adds Flair example to guide (#762)
• docs: Updates huggingface examples and adds monitor for Flair (#761)
• feat(search): show boolean values in metadata (#753)
• feat(server): allow handle labeling rules for datasets from API (#744)
• fix(imports): import monitoring with spacy<3.0 fails (#754)
• [UI] new fonts families (#751)
• fix(scroll): using new scroll component (#710)
• fix(ui): filter "validatable" records for global action validate button (#741)
• feat(monitor): flair ner auto-monitor (#738)

Full Changelog: https://github.com/recognai/rubrix/compare/v0.7.0...v0.8.0-alpha.0

🔆 Highlights

Rubrix Workspaces

Rubrix Workspaces enable you to organize your data collection and monitoring workflows much more flexibly than before. Workspaces can be project-based (for separating the work across different projects), team-based (for organizing the work across teams), model-based (for organizing data collection and monitoring on a per-model or model group basis), or anything you can think about. A workspace is a Rubrix “space” where users can collaborate, both using the Webapp and the Python client. There are two types of workspace:

Team workspace: Where one or several users have read/write access.

User workspace: Every user gets its own user workspace. This workspace is the default workspace when users log in and log and load data with the Python client. The name of this workspace corresponds to the username.

Additionally, you can still use tags and metadata to structure datasets inside a workspace.

The setup should be pretty straight forward, you can find all details here: https://rubrix.readthedocs.io/en/stable/getting_started/user-management.html.

From the Python library side, to know how to log and load data from different workspaces, check the Python client API docs: https://rubrix.readthedocs.io/en/stable/reference/python/python_client.html

Weak Supervision

1. Implementation of the first built-in Label Model (Snorkel): https://rubrix.readthedocs.io/en/stable/guides/weak-supervision.html#Built-in-label-models
2. New tutorial using weak supervision for news classification: https://rubrix.readthedocs.io/en/stable/tutorials/weak-supervision-with-rubrix.html
3. Example using Weasel for training a downstream classifier directly with weak labels using PyTorch and Hugging Face transformers: https://rubrix.readthedocs.io/en/stable/guides/weak-supervision.html#Joint-Model-with-Weasel

The API docs for the weak supervision model can be found here: https://rubrix.readthedocs.io/en/stable/reference/python/python_labeling.html#python-labeling

Improved UX for text classification annotation

Refined the annotation module for text classification, especially for text classification with a high number of labels

Rubrix Metrics

Increased the support for Rubrix Metrics, check this guide for more information: https://rubrix.readthedocs.io/en/stable/guides/metrics.html

• Support for queries to compute metrics for dataset slices
• Support for F1 in Token Classification
• Support for common metrics across tasks (string length)
• Support for Token classification predictions (model outputs) and annotations (training data)

💻 Upgrading

To use this new release, do not forget to run:

Update the client library:

pip install -U rubrix

If you are using Docker:

docker-compose pull
docker-compose up

If you are using the python server:

pip install -U rubrix[server]

What's Changed

• Refactor: Move RubrixClient out of init (#563) by David Fidalgo
• Remove dynamic metadata, move it to setup.cfg (#562) by David Fidalgo
• fix tab titles of our docs (#561) by David Fidalgo
• [UI] Token classifier: Arrow styles are broken in Firefox (#576) by leiyre
• Fix: rb.load for ids with mixed types (#577) by David Fidalgo
• fix the build process (#583) by David Fidalgo
• fix: limit agent length (#585) by Francisco Aranda
• refactor(client): moves asgi module to rubrix.monitoring (#584) by Francisco Aranda
• fix(client): clear client cache after delete dataset (#580) by Francisco Aranda
• fixes(server): avoid mix single and multi label records for text-class (#582) by Francisco Aranda
• fix: assert tokens and text have content (#598) by Francisco Aranda
• tests: include basic tests for server.security module (#593) by Alex Jakubko
• [Docs] Make building the docs faster (#599) by David Fidalgo
• feat(client): compute metrics with query filter (#600) by Francisco Aranda
• refactor(server): normalizes token classification metrics (#602) by Francisco Aranda
• bugfixes(metrics): prevent index out of range for tokenclass metrics (#608) by Francisco Aranda
• feat(metrics): use stacked bar for entity consistency (#607) by Francisco Aranda
• fix(UI): Mention values in Stats sidebar sort when updating (#613) by leiyre
• Add tqdm to rb.log (#609) by David Fidalgo
• feat(metrics): include mention length metrics at char level (#615) by Francisco Aranda
• fix(monitoring): support old zeroshot versions (#614) by Francisco Aranda
• fix: enable nested fields in search dsl (#587) by Francisco Aranda
• hotfix: fix test for build ci by Francisco Aranda
• Typo fix in 05-active_learning.ipynb (#619) by Sebastian Raschka
• feat(metrics): annotated mentions metrics (#618) by Francisco Aranda
• [UI] Text classifier: annotation task interaction enhancement (#611) by leiyre
• docs: Introduce monitoring guide (#625) by Daniel Vila Suero
• docs: review monitoring guide (#626) by Daniel Vila Suero
• refactor: rename teams to workspaces (#622) by Francisco Aranda
• docs: update monitoring guide (#631) by Daniel Vila Suero
• fix(client): Adds verbose kwarg to rb.log (#632) by David Fidalgo
• [stats] Keywords in stats re-sort when query is updated (#639) by leiyre
• hotfix(server): wrong email user validation regex by Francisco Aranda
• Introduce LabelModel and Snorkel implementation (#624) by David Fidalgo
• refactor(UI): normalize multi-label dataset access (#635) by Francisco Aranda
• [QA] text classification labels (#636) by leiyre
• fix(metrix): empty metrics visualization (#642) by Francisco Aranda
• Add F1 metrics to token classification task (#640) by David Fidalgo
• fix(doc): prevent 'Mixed Content:...' error (#645) by Francisco Aranda
• NoRecordsFoundError when rb.load results in empty list in WeakLabels (#641) by David Fidalgo
• [UI styles] QA annotation buttons styles (#654) by leiyre
• refactor(metrics): module shortcut for compute_for and enum def (#651) by Francisco Aranda
• hotfix(user): empty workspaces list checks to default workspace by Francisco Aranda
• format doc strings according to the google style + small improvements (#656) by David Fidalgo
• fix(search): prevent ignore 0s for aggregation result keys (#655) by Francisco Aranda
• feat(server): accepts workspace as http header (#659) by Francisco Aranda
• refactor(user): bypass ws for super users (#660) by Francisco Aranda
• feat(server): common task metrics (#657) by Francisco Aranda
• feat(client): user workspace management from client (#661) by Francisco Aranda
• feat(UI): select user workspace (#662) by Francisco Aranda
• UI: Add hover effect on selected label in Text Classification (#663) by leiyre
• UI: Button-icon active state improvement (#664) by leiyre
• [BUG] Annotation agent is user.username (#666) by leiyre
• by default do not pass on Y_dev when fitting (#670) by David Fidalgo
• Docs: Adds weak supervision tutorial (#672) by Daniel Vila Suero
• [Client] Add metrics parameter to all client models (#671) by David Fidalgo
• [UI] QA: button active state color duration (#675) by leiyre
• [bug] Sticky top-bar glitch when scrolling (#674) by leiyre
• fix(docs): .rubrix_* -> .rubrix* (#680) by Francisco Aranda
• fix(server): metadata keys with empty meta will be omitted (#678) by Francisco Aranda
• docs: fix small typo in ws tuto (#684) by Daniel Vila Suero
• feat(client): dataset copy with workspace param (#683) by Francisco Aranda
• [UI] Limit pagination in UI (#668) by leiyre
• fix(server): single label annotation validator (#687) by Francisco Aranda
• fix(app): read all dataset labels for annotation (#688) by Francisco Aranda
• [UI] Message for empty home (datasets list) (#691) by leiyre
• [UI] Fix: Text classifier explore record width (#696) by leiyre
• [Labeling] Throw error when encountering duplicated rule names (#693) by David Fidalgo
• [UI] Fix: Text Classification annotation record width (#699) by leiyre
• [Metrics] Normalize F1 metrics for Text-/TokenClassification (#694) by David Fidalgo
• fix link for models (#703) by Leire Rosado
• [Docs] First attempt to devise a testing workflow for the tutorials (#649) by David Fidalgo
• docs: Updates metrics guide (#647) by Daniel Vila Suero
• [UI] "Validate" button align left in Text classification and Token classification (#707) by leiyre
• feat(metrics): improve common dataset metrics #709 by Francisco Aranda
• [Docs] Add WeaSEL example to weak supervision guide (#578) by David Fidalgo
• [UI] Workspaces QA (#697) by leiyre
• small typo/grammar fixes for the weak supervision guide by dcfidalgo
• Fix/loss tutorial (#714) by Leire Rosado
• Fix/spacy_transformers (#711) by Leire Rosado
• fix(ui): refresh dataset before initalize it (#721) by Francisco Aranda
• [UI] Fix: Refresh button mantains pagination configuration (#715) by leiyre
• remove kglab tutorial (#720) by David Fidalgo
• fix(ui): refresh aggregations to paginated dataset (#722) by Francisco Aranda
• fix(ui): preserving the annotate/explore state on browser refresh (#724) by Francisco Aranda
• docs: Adds User and Workspaces management guide (#726) by Daniel Vila Suero

New Contributors

• @rasbt made their first contribution in https://github.com/recognai/rubrix/pull/619
• @leireropl made their first contribution in https://github.com/recognai/rubrix/pull/703

Full Changelog: https://github.com/recognai/rubrix/compare/v0.6.2...v0.7.0

What's Changed

• Refactor: Move RubrixClient out of init (#563) by David Fidalgo
• Remove dynamic metadata, move it to setup.cfg (#562) by David Fidalgo
• fix tab titles of our docs (#561) by David Fidalgo
• [UI] Token classifier: Arrow styles are broken in Firefox (#576) by leiyre
• Fix: rb.load for ids with mixed types (#577) by David Fidalgo
• fix the build process (#583) by David Fidalgo
• fix: limit agent length (#585) by Francisco Aranda
• refactor(client): moves asgi module to rubrix.monitoring (#584) by Francisco Aranda
• fix(client): clear client cache after delete dataset (#580) by Francisco Aranda
• fixes(server): avoid mix single and multi label records for text-class (#582) by Francisco Aranda
• fix: assert tokens and text have content (#598) by Francisco Aranda
• tests: include basic tests for server.security module (#593) by Alex Jakubko
• [Docs] Make building the docs faster (#599) by David Fidalgo
• feat(client): compute metrics with query filter (#600) by Francisco Aranda
• refactor(server): normalizes token classification metrics (#602) by Francisco Aranda
• bugfixes(metrics): prevent index out of range for tokenclass metrics (#608) by Francisco Aranda
• feat(metrics): use stacked bar for entity consistency (#607) by Francisco Aranda
• fix(UI): Mention values in Stats sidebar sort when updating (#613) by leiyre
• Add tqdm to rb.log (#609) by David Fidalgo
• feat(metrics): include mention length metrics at char level (#615) by Francisco Aranda
• fix(monitoring): support old zeroshot versions (#614) by Francisco Aranda
• fix: enable nested fields in search dsl (#587) by Francisco Aranda
• hotfix: fix test for build ci by Francisco Aranda
• Typo fix in 05-active_learning.ipynb (#619) by Sebastian Raschka
• feat(metrics): annotated mentions metrics (#618) by Francisco Aranda
• [UI] Text classifier: annotation task interaction enhancement (#611) by leiyre
• docs: Introduce monitoring guide (#625) by Daniel Vila Suero
• docs: review monitoring guide (#626) by Daniel Vila Suero
• refactor: rename teams to workspaces (#622) by Francisco Aranda
• docs: update monitoring guide (#631) by Daniel Vila Suero
• fix(client): Adds verbose kwarg to rb.log (#632) by David Fidalgo
• [stats] Keywords in stats re-sort when query is updated (#639) by leiyre
• hotfix(server): wrong email user validation regex by Francisco Aranda
• Introduce LabelModel and Snorkel implementation (#624) by David Fidalgo
• refactor(UI): normalize multi-label dataset access (#635) by Francisco Aranda
• [QA] text classification labels (#636) by leiyre
• fix(metrix): empty metrics visualization (#642) by Francisco Aranda
• Add F1 metrics to token classification task (#640) by David Fidalgo
• fix(doc): prevent 'Mixed Content:...' error (#645) by Francisco Aranda
• NoRecordsFoundError when rb.load results in empty list in WeakLabels (#641) by David Fidalgo
• [UI styles] QA annotation buttons styles (#654) by leiyre
• refactor(metrics): module shortcut for compute_for and enum def (#651) by Francisco Aranda
• hotfix(user): empty workspaces list checks to default workspace by Francisco Aranda
• format doc strings according to the google style + small improvements (#656) by David Fidalgo
• fix(search): prevent ignore 0s for aggregation result keys (#655) by Francisco Aranda
• feat(server): accepts workspace as http header (#659) by Francisco Aranda
• refactor(user): bypass ws for super users (#660) by Francisco Aranda
• feat(server): common task metrics (#657) by Francisco Aranda
• feat(client): user workspace management from client (#661) by Francisco Aranda
• feat(UI): select user workspace (#662) by Francisco Aranda
• UI: Add hover effect on selected label in Text Classification (#663) by leiyre
• UI: Button-icon active state improvement (#664) by leiyre
• [BUG] Annotation agent is user.username (#666) by leiyre
• by default do not pass on Y_dev when fitting (#670) by David Fidalgo
• Docs: Adds weak supervision tutorial (#672) by Daniel Vila Suero
• [Client] Add metrics parameter to all client models (#671) by David Fidalgo
• [UI] QA: button active state color duration (#675) by leiyre
• [bug] Sticky top-bar glitch when scrolling (#674) by leiyre
• fix(docs): .rubrix_* -> .rubrix* (#680) by Francisco Aranda
• fix(server): metadata keys with empty meta will be omitted (#678) by Francisco Aranda
• docs: fix small typo in ws tuto (#684) by Daniel Vila Suero
• feat(client): dataset copy with workspace param (#683) by Francisco Aranda
• [UI] Limit pagination in UI (#668) by leiyre
• fix(server): single label annotation validator (#687) by Francisco Aranda
• fix(app): read all dataset labels for annotation (#688) by Francisco Aranda
• [UI] Message for empty home (datasets list) (#691) by leiyre
• [UI] Fix: Text classifier explore record width (#696) by leiyre
• [Labeling] Throw error when encountering duplicated rule names (#693) by David Fidalgo
• [UI] Fix: Text Classification annotation record width (#699) by leiyre
• [Metrics] Normalize F1 metrics for Text-/TokenClassification (#694) by David Fidalgo
• fix link for models (#703) by Leire Rosado
• [Docs] First attempt to devise a testing workflow for the tutorials (#649) by David Fidalgo
• docs: Updates metrics guide (#647) by Daniel Vila Suero
• [UI] "Validate" button align left in Text classification and Token classification (#707) by leiyre
• feat(metrics): improve common dataset metrics #709 by Francisco Aranda
• [Docs] Add WeaSEL example to weak supervision guide (#578) by David Fidalgo
• [UI] Workspaces QA (#697) by leiyre
• small typo/grammar fixes for the weak supervision guide by dcfidalgo
• Fix/loss tutorial (#714) by Leire Rosado
• Fix/spacy_transformers (#711) by Leire Rosado
• fix(ui): refresh dataset before initalize it (#721) by Francisco Aranda
• [UI] Fix: Refresh button mantains pagination configuration (#715) by leiyre
• remove kglab tutorial (#720) by David Fidalgo
• fix(ui): refresh aggregations to paginated dataset (#722) by Francisco Aranda
• fix(ui): preserving the annotate/explore state on browser refresh (#724) by Francisco Aranda
• docs: Adds User and Workspaces management guide (#726) by Daniel Vila Suero

New Contributors

• @rasbt made their first contribution in https://github.com/recognai/rubrix/pull/619
• @leireropl made their first contribution in https://github.com/recognai/rubrix/pull/703

Full Changelog: https://github.com/recognai/rubrix/compare/v0.6.2...v0.7.0-alpha.1

What's Changed

• Refactor: Move RubrixClient out of init (#563) by David Fidalgo
• Remove dynamic metadata, move it to setup.cfg (#562) by David Fidalgo
• fix tab titles of our docs (#561) by David Fidalgo
• [UI] Token classifier: Arrow styles are broken in Firefox (#576) by leiyre
• Fix: rb.load for ids with mixed types (#577) by David Fidalgo
• fix the build process (#583) by David Fidalgo
• fix: limit agent length (#585) by Francisco Aranda
• refactor(client): moves asgi module to rubrix.monitoring (#584) by Francisco Aranda
• fix(client): clear client cache after delete dataset (#580) by Francisco Aranda
• fixes(server): avoid mix single and multi label records for text-class (#582) by Francisco Aranda
• fix: assert tokens and text have content (#598) by Francisco Aranda
• tests: include basic tests for server.security module (#593) by Alex Jakubko
• [Docs] Make building the docs faster (#599) by David Fidalgo
• feat(client): compute metrics with query filter (#600) by Francisco Aranda
• refactor(server): normalizes token classification metrics (#602) by Francisco Aranda
• bugfixes(metrics): prevent index out of range for tokenclass metrics (#608) by Francisco Aranda
• feat(metrics): use stacked bar for entity consistency (#607) by Francisco Aranda
• fix(UI): Mention values in Stats sidebar sort when updating (#613) by leiyre
• Add tqdm to rb.log (#609) by David Fidalgo
• feat(metrics): include mention length metrics at char level (#615) by Francisco Aranda
• fix(monitoring): support old zeroshot versions (#614) by Francisco Aranda
• fix: enable nested fields in search dsl (#587) by Francisco Aranda
• hotfix: fix test for build ci by Francisco Aranda
• Typo fix in 05-active_learning.ipynb (#619) by Sebastian Raschka
• feat(metrics): annotated mentions metrics (#618) by Francisco Aranda
• [UI] Text classifier: annotation task interaction enhancement (#611) by leiyre
• docs: Introduce monitoring guide (#625) by Daniel Vila Suero
• docs: review monitoring guide (#626) by Daniel Vila Suero
• refactor: rename teams to workspaces (#622) by Francisco Aranda
• docs: update monitoring guide (#631) by Daniel Vila Suero
• fix(client): Adds verbose kwarg to rb.log (#632) by David Fidalgo
• [stats] Keywords in stats re-sort when query is updated (#639) by leiyre
• hotfix(server): wrong email user validation regex by Francisco Aranda
• Introduce LabelModel and Snorkel implementation (#624) by David Fidalgo
• refactor(UI): normalize multi-label dataset access (#635) by Francisco Aranda
• [QA] text classification labels (#636) by leiyre
• fix(metrix): empty metrics visualization (#642) by Francisco Aranda
• Add F1 metrics to token classification task (#640) by David Fidalgo
• fix(doc): prevent 'Mixed Content:...' error (#645) by Francisco Aranda
• NoRecordsFoundError when rb.load results in empty list in WeakLabels (#641) by David Fidalgo
• [UI styles] QA annotation buttons styles (#654) by leiyre
• refactor(metrics): module shortcut for compute_for and enum def (#651) by Francisco Aranda
• hotfix(user): empty workspaces list checks to default workspace by Francisco Aranda
• format doc strings according to the google style + small improvements (#656) by David Fidalgo
• fix(search): prevent ignore 0s for aggregation result keys (#655) by Francisco Aranda
• feat(server): accepts workspace as http header (#659) by Francisco Aranda
• refactor(user): bypass ws for super users (#660) by Francisco Aranda
• feat(server): common task metrics (#657) by Francisco Aranda
• feat(client): user workspace management from client (#661) by Francisco Aranda
• feat(UI): select user workspace (#662) by Francisco Aranda
• UI: Add hover effect on selected label in Text Classification (#663) by leiyre
• UI: Button-icon active state improvement (#664) by leiyre
• [BUG] Annotation agent is user.username (#666) by leiyre
• by default do not pass on Y_dev when fitting (#670) by David Fidalgo
• Docs: Adds weak supervision tutorial (#672) by Daniel Vila Suero
• [Client] Add metrics parameter to all client models (#671) by David Fidalgo
• [UI] QA: button active state color duration (#675) by leiyre
• [bug] Sticky top-bar glitch when scrolling (#674) by leiyre
• fix(docs): .rubrix_* -> .rubrix* (#680) by Francisco Aranda
• fix(server): metadata keys with empty meta will be omitted (#678) by Francisco Aranda
• docs: fix small typo in ws tuto (#684) by Daniel Vila Suero
• feat(client): dataset copy with workspace param (#683) by Francisco Aranda
• [UI] Limit pagination in UI (#668) by leiyre
• fix(server): single label annotation validator (#687) by Francisco Aranda
• fix(app): read all dataset labels for annotation (#688) by Francisco Aranda
• [UI] Message for empty home (datasets list) (#691) by leiyre
• [UI] Fix: Text classifier explore record width (#696) by leiyre
• [Labeling] Throw error when encountering duplicated rule names (#693) by David Fidalgo
• [UI] Fix: Text Classification annotation record width (#699) by leiyre
• [Metrics] Normalize F1 metrics for Text-/TokenClassification (#694) by David Fidalgo
• fix link for models (#703) by Leire Rosado
• [Docs] First attempt to devise a testing workflow for the tutorials (#649) by David Fidalgo
• docs: Updates metrics guide (#647) by Daniel Vila Suero
• [UI] "Validate" button align left in Text classification and Token classification (#707) by leiyre
• feat(metrics): improve common dataset metrics #709 by Francisco Aranda
• [Docs] Add WeaSEL example to weak supervision guide (#578) by David Fidalgo
• [UI] Workspaces QA (#697) by leiyre
• small typo/grammar fixes for the weak supervision guide by dcfidalgo
• Fix/loss tutorial (#714) by Leire Rosado
• Fix/spacy_transformers (#711) by Leire Rosado
• fix(ui): refresh dataset before initalize it (#721) by Francisco Aranda
• [UI] Fix: Refresh button mantains pagination configuration (#715) by leiyre
• remove kglab tutorial (#720) by David Fidalgo
• fix(ui): refresh aggregations to paginated dataset (#722) by Francisco Aranda
• fix(ui): preserving the annotate/explore state on browser refresh (#724) by Francisco Aranda
• docs: Adds User and Workspaces management guide (#726) by Daniel Vila Suero

New Contributors

• @rasbt made their first contribution in https://github.com/recognai/rubrix/pull/619
• @leireropl made their first contribution in https://github.com/recognai/rubrix/pull/703

Full Changelog: https://github.com/recognai/rubrix/compare/v0.6.2...v0.7.0-alpha.0