1.16.0

Added

• Added ArgillaTrainer integration with sentence-transformers, allowing fine tuning for sentence similarity (#3739)
• Added ArgillaTrainer integration with TrainingTask.for_question_answering (#3740)
• Added Auto save record to save automatically the current record that you are working on (#3541)
• Added ArgillaTrainer integration with OpenAI, allowing fine tuning for chat completion (#3615)
• Added workspaces list command to list Argilla workspaces (#3594).
• Added datasets list command to list Argilla datasets (#3658).
• Added users create command to create users (#3667).
• Added whoami command to get current user (#3673).
• Added users delete command to delete users (#3671).
• Added users list command to list users (#3688).
• Added workspaces delete-user command to remove a user from a workspace (#3699).
• Added datasets list command to list Argilla datasets (#3658).
• Added users create command to create users (#3667).
• Added users delete command to delete users (#3671).
• Added workspaces create command to create an Argilla workspace (#3676).
• Added datasets push-to-hub command to push a FeedbackDataset from Argilla into the HuggingFace Hub (#3685).
• Added info command to get info about the used Argilla client and server (#3707).
• Added datasets delete command to delete a FeedbackDataset from Argilla (#3703).
• Added created_at and updated_at properties to RemoteFeedbackDataset and FilteredRemoteFeedbackDataset (#3709).
• Added handling PermissionError when executing a command with a logged in user with not enough permissions (#3717).
• Added workspaces add-user command to add a user to workspace (#3712).
• Added workspace_id param to GET /api/v1/me/datasets endpoint (#3727).
• Added workspace_id arg to list_datasets in the Python SDK (#3727).
• Added argilla script that allows to execute Argilla CLI using the argilla command (#3730).
• Added server_info function to check the Argilla server information (also accessible via rg.server_info) (#3772).

Changed

• Move database commands under server group of commands (#3710)
• server commands only included in the CLI app when server extra requirements are installed (#3710).
• Updated PUT /api/v1/responses/{response_id} to replace values stored with received values in request (#3711).
• Display a UserWarning when the user_id in Workspace.add_user and Workspace.delete_user is the ID of an user with the owner role as they don't require explicit permissions (#3716).
• Rename tasks sub-package to cli (#3723).
• Changed argilla database command in the CLI to now be accessed via argilla server database, to be deprecated in the upcoming release (#3754).
• Changed visible_options (of label and multi label selection questions) validation in the backend to check that the provided value is greater or equal than/to 3 and less or equal than/to the number of provided options (#3773).

Fixed

• Fixed remove user modification in text component on clear answers (#3775)
• Fixed Highlight raw text field in dataset feedback task (#3731)
• Fixed Field title too long (#3734)
• Fixed error messages when deleting a DatasetForTextClassification (#3652)
• Fixed Pending queue pagination problems when during data annotation (#3677)
• Fixed visible_labels default value to be 20 just when visible_labels not provided and len(labels) > 20, otherwise it will either be the provided visible_labels value or None, for LabelQuestion and MultiLabelQuestion (#3702).
• Fixed DatasetCard generation when RemoteFeedbackDataset contains suggestions (#3718).
• Add missing draft status in ResponseSchema as now there can be responses with draft status when annotating via the UI (#3749).
• Searches when queried words are distributed along the record fields (#3759).
• Fixed Python 3.11 compatibility issue with /api/datasets endpoints due to the TaskType enum replacement in the endpoint URL (#3769).

New Contributors

• @sdiazlor made their first contribution in https://github.com/argilla-io/argilla/pull/3384

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.15.1...v1.16.0

Changelog 1.15.1

Fixed

• Fixed Text component text content sanitization behavior just for markdown to prevent disappear the text (#3738)
• Fixed Text component now you need to press Escape to exit the text area (#3733)
• Fixed SearchEngine was creating the same number of primary shards and replica shards for each FeedbackDataset (#3736).

🔆 Highlights

Argilla 1.15.0 comes with an enhanced FeedbackDataset settings page enabling the update of the dataset settings, an integration of the TRL package with the ArgillaTrainer, and continues adding improvements to the Python client for managing FeedbackDatasets.

⚙️ Update FeedbackDataset settings from the UI

FeedbackDataset settings page has been updated and now it allows to update the guidelines and some attributes of the fields and questions of the dataset. Did you misspell the title or description of a field or question? Well, you don't have to remove your dataset and create it again anymore! Just go to the settings page and fix it.

🤖 TRL integration with the ArgillaTrainer

The famous TRL package for training Transformers with Reinforcement Learning techniques has been integrated with the ArgillaTrainer, that comes with four new TrainingTask: SFT, Reward Modeling, PPO and DPO. Each training task expects a formatting function that will return the data in the expected format for training the model.

Check this 🆕 tutorial for training a Reward Model using the Argilla Trainer.

🐍 Filter FeedbackDataset and remove suggestions

In the 1.14.0 release we added many improvements for working with remote FeedbackDatasets. In this release, a new filter_by method has been added that allows to filter the records of a dataset from the Python client. For now, the records can be only filtered using the response_status, but we're planning adding more complex filters for the upcoming releases. In addition, new methods have been added allowing to remove the suggestions created for a record.

1.15.0

Added

• Added Enable to update guidelines and dataset settings for Feedback Datasets directly in the UI (#3489)
• Added ArgillaTrainer integration with TRL, allowing for easy supervised finetuning, reward modeling, direct preference optimization and proximal policy optimization (#3467)
• Added formatting_func to ArgillaTrainer for FeedbackDataset datasets add a custom formatting for the data (#3599).
• Added login function in argilla.client.login to login into an Argilla server and store the credentials locally (#3582).
• Added login command to login into an Argilla server (#3600).
• Added logout command to logout from an Argilla server (#3605).
• Added DELETE /api/v1/suggestions/{suggestion_id} endpoint to delete a suggestion given its ID (#3617).
• Added DELETE /api/v1/records/{record_id}/suggestions endpoint to delete several suggestions linked to the same record given their IDs (#3617).
• Added response_status param to GET /api/v1/datasets/{dataset_id}/records to be able to filter by response_status as previously included for GET /api/v1/me/datasets/{dataset_id}/records (#3613).
• Added list classmethod to ArgillaMixin to be used as FeedbackDataset.list(), also including the workspace to list from as arg (#3619).
• Added filter_by method in RemoteFeedbackDataset to filter based on response_status (#3610).
• Added list_workspaces function (to be used as rg.list_workspaces, but Workspace.list is preferred) to list all the workspaces from an user in Argilla (#3641).
• Added list_datasets function (to be used as rg.list_datasets) to list the TextClassification, TokenClassification, and Text2Text datasets in Argilla (#3638).
• Added RemoteSuggestionSchema to manage suggestions in Argilla, including the delete method to delete suggestios from Argilla via DELETE /api/v1/suggestions/{suggestion_id} (#3651).
• Added delete_suggestions to RemoteFeedbackRecord to remove suggestions from Argilla via DELETE /api/v1/records/{record_id}/suggestions (#3651).

Changed

• Changed Optional label for * mark for required question (#3608)
• Updated RemoteFeedbackDataset.delete_records to use batch delete records endpoint (#3580).
• Included allowed_for_roles for some RemoteFeedbackDataset, RemoteFeedbackRecords, and RemoteFeedbackRecord methods that are only allowed for users with roles owner and admin (#3601).
• Renamed ArgillaToFromMixin to ArgillaMixin (#3619).
• Move users CLI app under database CLI app (#3593).
• Move server Enum classes to argilla.server.enums module (#3620).

Fixed

• Fixed Filter by workspace in breadcrumbs (#3577)
• Fixed Filter by workspace in datasets table (#3604)
• Fixed Query search highlight for Text2Text and TextClassification (#3621)
• Fixed RatingQuestion.values validation to raise a ValidationError when values are out of range i.e. [1, 10] (#3626).

Removed

• Removed multi_task_text_token_classification from TaskType as not used (#3640).
• Removed argilla_id in favor of id from RemoteFeedbackDataset (#3663).
• Removed fetch_records from RemoteFeedbackDataset as now the records are lazily fetched from Argilla (#3663).
• Removed push_to_argilla from RemoteFeedbackDataset, as it just works when calling it through a FeedbackDataset locally, as now the updates of the remote datasets are automatically pushed to Argilla (#3663).
• Removed set_suggestions in favor of update(suggestions=...) for both FeedbackRecord and RemoteFeedbackRecord, as all the updates of any "updateable" attribute of a record will go through update instead (#3663).
• Remove unused owner attribute for client Dataset data model (#3665)

As always, thanks to our amazing contributors

• @peppinob-ol made their first contribution in https://github.com/argilla-io/argilla/pull/3472
• @eshwarhs made their first contribution in https://github.com/argilla-io/argilla/pull/3605

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.14.1...v1.15.0

Changelog 1.14.1

Fixed

• Fixed PostgreSQL database not being updated after begin_nested because of missing commit (#3567).

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.14.0...v1.14.1

🔆 Highlights

Argilla 1.14.0 comes packed with improvements to manage Feedback Datasets from the Python client. Here are the most important changes in this version:

Pushing and pulling a dataset

Pushing a dataset to Argilla will now create a RemoteFeedbackDataset in Argilla. To make changes to your dataset in Argilla you will need to make those updates to the remote dataset. You can do so by either using the dataset returned when using the push_to_argilla() method (as shown in the image above) or by loading the dataset like so:

import argilla as rg
# connect to Argilla
rg.init(api_url="...", api_key="...")
# get the existing dataset in Argilla
remote_dataset = rg.FeedbackDataset.from_argilla(name="my-dataset", workspace="my-workspace")
# add a list of FeedbackRecords to the dataset in Argilla
remote_dataset.add_records(...)

Alternatively, you can make a local copy of the dataset using the pull() method.

local_dataset = remote_dataset.pull()

Note that any changes that you make to this local dataset will not affect the remote dataset in Argilla.

Adding and deleting records

How to add records to an existing dataset in Argilla was demonstrated in the first code snippet in the "Pushing and pulling a dataset" section. This is how you can delete a list of records using that same dataset:

records_to_delete = remote_dataset.records[0:5]
remote_dataset.delete_records(records_to_delete)

Or delete a single record:

record = remote_dataset.records[-1]
record.delete()

Add / update suggestions in existing records

To add and update suggestions in existing records, you can simply use the update() method. For example:

for record in remote_dataset.records:
    record.update(suggestions=...)

Note that adding a suggestion to a question that already has one will overwrite the previous suggestion. To learn more about the format that the suggestions must follow, check our docs.

Delete a dataset

You can now easily delete datasets from the Python client. To do that, get the existing dataset like demonstrated in the first section and just use:

remote_dataset.delete()

Create users with workspace assignments

Now you can create a user and directly assign existing workspaces to grant them access.

user = rg.User.create(username="...", first_name="...", password="...", workspaces=["ws1", "ws2"])

Changelog 1.14.0

Added

• Added PATCH /api/v1/fields/{field_id} endpoint to update the field title and markdown settings (#3421).
• Added PATCH /api/v1/datasets/{dataset_id} endpoint to update dataset name and guidelines (#3402).
• Added PATCH /api/v1/questions/{question_id} endpoint to update question title, description and some settings (depending on the type of question) (#3477).
• Added DELETE /api/v1/records/{record_id} endpoint to remove a record given its ID (#3337).
• Added pull method in RemoteFeedbackDataset (a FeedbackDataset pushed to Argilla) to pull all the records from it and return it as a local copy as a FeedbackDataset (#3465).
• Added delete method in RemoteFeedbackDataset (a FeedbackDataset pushed to Argilla) (#3512).
• Added delete_records method in RemoteFeedbackDataset, and delete method in RemoteFeedbackRecord to delete records from Argilla (#3526).

Changed

• Improved efficiency of weak labeling when dataset contains vectors (#3444).
• Added ArgillaDatasetMixin to detach the Argilla-related functionality from the FeedbackDataset (#3427)
• Moved FeedbackDataset-related pydantic.BaseModel schemas to argilla.client.feedback.schemas instead, to be better structured and more scalable and maintainable (#3427)
• Update CLI to use database async connection (#3450).
• Limit rating questions values to the positive range [1, 10] (#3451).
• Updated POST /api/users endpoint to be able to provide a list of workspace names to which the user should be linked to (#3462).
• Updated Python client User.create method to be able to provide a list of workspace names to which the user should be linked to (#3462).
• Updated GET /api/v1/me/datasets/{dataset_id}/records endpoint to allow getting records matching one of the response statuses provided via query param (#3359).
• Updated POST /api/v1/me/datasets/{dataset_id}/records endpoint to allow searching records matching one of the response statuses provided via query param (#3359).
• Updated SearchEngine.search method to allow searching records matching one of the response statuses provided (#3359).
• After calling FeedbackDataset.push_to_argilla, the methods FeedbackDataset.add_records and FeedbackRecord.set_suggestions will automatically call Argilla with no need of calling push_to_argilla explicitly (#3465).
• Now calling FeedbackDataset.push_to_huggingface dumps the responses as a List[Dict[str, Any]] instead of Sequence to make it more readable via 🤗datasets (#3539).

Fixed

• Fixed issue with bool values and default from Jinja2 while generating the HuggingFace DatasetCard from argilla_template.md (#3499).
• Fixed DatasetConfig.from_yaml which was failing when calling FeedbackDataset.from_huggingface as the UUIDs cannot be deserialized automatically by PyYAML, so UUIDs are neither dumped nor loaded anymore (#3502).
• Fixed an issue that didn't allow the Argilla server to work behind a proxy (#3543).
• TextClassificationSettings and TokenClassificationSettings labels are properly parsed to strings both in the Python client and in the backend endpoint (#3495).
• Fixed PUT /api/v1/datasets/{dataset_id}/publish to check whether at least one field and question has required=True (#3511).
• Fixed FeedbackDataset.from_huggingface as suggestions were being lost when there were no responses (#3539).
• Fixed QuestionSchema and FieldSchema not validating name attribute (#3550).

Deprecated

• After calling FeedbackDataset.push_to_argilla, calling push_to_argilla again won't do anything since the dataset is already pushed to Argilla (#3465).
• After calling FeedbackDataset.push_to_argilla, calling fetch_records won't do anything since the records are lazily fetched from Argilla (#3465).
• After calling FeedbackDataset.push_to_argilla, the Argilla ID is no longer stored in the attribute/property argilla_id but in id instead (#3465).

As always, thanks to our amazing contributors

• @plaguss made their first contribution in https://github.com/argilla-io/argilla/pull/3454

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.13.3...v1.14.0

1.13.3

Fixed

• Fixed ModuleNotFoundError caused because the argilla.utils.telemetry module used in the ArgillaTrainer was importing an optional dependency not installed by default (#3471).
• Fixed ImportError caused because the argilla.client.feedback.config module was importing pyyaml optional dependency not installed by default (#3471).

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.13.2...v1.13.3

1.13.2

Fixed

• The suggestion_type_enum ENUM data type created in PostgreSQL didn't have any value (#3445).

1.13.1

Fixed

• Fix database migration for PostgreSQL (See #3438)

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.13.0...v1.13.1

🔆 Highlights

✨ Suggestions

All question types in the Feedback task support suggestions, but you can only add one suggestion per question.

Learn more about this feature in our docs.

🗄️ List workspaces

We've added functionalities to list all the workspaces that a user has access to. From the Python client you will be able to list all workspaces of the current user using rg.Workspace.list() and in the UI you will be able to see the list of workspaces in the user settings page.

Read more in the docs.

🏋️‍♂️ Extended training support

We are extending the support we give to help preparing data from Feedback datasets to use during training. As part of this release we include strategies to unify responses to RankingQuestions and also provide a task mapping for text classification TrainingTaskMapping.for_text_classification.

Read more about how to use these methods to train models with Feedback collected in Argilla here.

Changelog 1.13.0

Added

• Added GET /api/v1/users/{user_id}/workspaces endpoint to list the workspaces to which a user belongs (#3308 and #3343).
• Added HuggingFaceDatasetMixin for internal usage, to detach the FeedbackDataset integrations from the class itself, and use Mixins instead (#3326).
• Added GET /api/v1/records/{record_id}/suggestions API endpoint to get the list of suggestions for the responses associated to a record (#3304).
• Added POST /api/v1/records/{record_id}/suggestions API endpoint to create a suggestion for a response associated to a record (#3304).
• Added support for RankingQuestionStrategy, RankingQuestionUnification and the .for_text_classification method for the TrainingTaskMapping (#3364)
• Added PUT /api/v1/records/{record_id}/suggestions API endpoint to create or update a suggestion for a response associated to a record (#3304 & 3391).
• Added suggestions attribute to FeedbackRecord, and allow adding and retrieving suggestions from the Python client (#3370)
• Added allowed_for_roles Python decorator to check whether the current user has the required role to access the decorated function/method for User and Workspace (#3383)
• Added API and Python Client support for workspace deletion (Closes #3260)
• Added GET /api/v1/me/workspaces endpoint to list the workspaces of the current active user (#3390)

Changed

• Updated output payload for GET /api/v1/datasets/{dataset_id}/records, GET /api/v1/me/datasets/{dataset_id}/records, POST /api/v1/me/datasets/{dataset_id}/records/search endpoints to include the suggestions of the records based on the value of the include query parameter (#3304).
• Updated POST /api/v1/datasets/{dataset_id}/records input payload to add suggestions (#3304).
• The POST /api/datasets/:dataset-id/:task/bulk endpoints don't create the dataset if does not exists (Closes #3244)
• Added Telemetry support for ArgillaTrainer (closes #3325)
• User.workspaces is no longer an attribute but a property, and is calling list_user_workspaces to list all the workspace names for a given user ID (#3334)
• Renamed FeedbackDatasetConfig to DatasetConfig and export/import from YAML as default instead of JSON (just used internally on push_to_huggingface and from_huggingface methods of FeedbackDataset) (#3326).
• The protected metadata fields support other than textual info - existing datasets must be reindex. See docs for more detail (Closes #3332).
• Updated Dockerfile parent image from python:3.9.16-slim to python:3.10.12-slim (#3425).
• Updated quickstart.Dockerfile parent image from elasticsearch:8.5.3 to argilla/argilla-server:${ARGILLA_VERSION} (#3425).

Removed

• Removed support to non-prefixed environment variables. All valid env vars start with ARGILLA_ (See #3392).

Fixed

• Fixed GET /api/v1/me/datasets/{dataset_id}/records endpoint returning always the responses for the records even if responses was not provided via the include query parameter (#3304).
• Values for protected metadata fields are not truncated (Closes #3331).
• Big number ids are properly rendered in UI (Closes #3265)
• Fixed ArgillaDatasetCard to include the values/labels for all the existing questions (#3366)

Deprecated

• Integer support for record id in text classification, token classification and text2text datasets.

As always, thanks to our amazing contributors

• @manijhariya made their first contribution in https://github.com/argilla-io/argilla/pull/3295

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.12.1...1.13.0

1.12.1

Fixed

• Using rg.init with default argilla user skips setting the default workspace if not available. (Closes #3340)
• Resolved wrong import structure ArgillaTrainer TrainingTaskMapping (Closes #3345)
• Pin pydantic dependency to version < 2 (Closes 3348)

🔆 Highlights

New RankingQuestion in Feedback Task datasets

Now you will be able to include RankingQuestions in your Feedback datasets. These are specially designed to gather feedback on labeler's preferences, by providing a set of options that labelers can order.
Here's how you can add a RankingQuestion to a FeedbackDataset:

dataset = FeedbackDataset(
    fields=[
        rg.TextField(name="prompt"),
        rg.TextField(name="reply-1", title="Reply 1"),
        rg.TextField(name="reply-2", title="Reply 2"),
        rg.TextField(name="reply-3", title="Reply 3"),
    ],
    questions=[
        rg.RankingQuestion(
            name="ranking",
            title="Order replies based on your preference",
            description="1 = best, 3 = worst. Ties are allowed.",
            required=True,
            values={"reply-1": "Reply 1", "reply-2": "Reply 2", "reply-3": "Reply 3"} # or ["reply-1", "reply-2", "reply-3"]
    ]
)

More info in our docs.

Extended training support

You can now format responses from RatingQuestion, LabelQuestion and MultiLabelQuestion for your preferred training framework using the prepare_for_training method.

Also, we've added support for spacy-transformers in our Argilla Trainer.

Here's an example code snippet:

import argilla.feedback as rg

dataset = rg.FeedbackDataset.from_huggingface(
    repo_id="argilla/stackoverflow_feedback_demo"
)
task_mapping = rg.TrainingTaskMapping.for_text_classification(
    text=dataset.field_by_name("question"),
    label=dataset.question_by_name("tags")
)
trainer = rg.ArgillaTrainer(
    dataset=dataset,
    task_mapping=task_mapping,
    framework="spacy-transformers",
    fetch_records=False
)
trainer.update_config(num_train_epochs=2)
trainer.train(output_dir="my_awesone_model")

To learn more about how to use Argilla Trainer check our docs.

Changelog 1.12.0

Added

• Added RankingQuestionSettings class allowing to create ranking questions in the API using POST /api/v1/datasets/{dataset_id}/questions endpoint (#3232)
• Added RankingQuestion in the Python client to create ranking questions (#3275).
• Added Ranking component in feedback task question form (#3177 & #3246).
• Added FeedbackDataset.prepare_for_training method for generaring a framework-specific dataset with the responses provided for RatingQuestion, LabelQuestion and MultiLabelQuestion (#3151).
• Added ArgillaSpaCyTransformersTrainer class for supporting the training with spacy-transformers (#3256).

Changed

• All docker related files have been moved into the docker folder (#3053).
• release.Dockerfile have been renamed to Dockerfile (#3133).
• Updated rg.load function to raise a ValueError with a explanatory message for the cases in which the user tries to use the function to load a FeedbackDataset (#3289).
• Updated ArgillaSpaCyTrainer to allow re-using tok2vec (#3256).

Fixed

• Check available workspaces on Argilla on rg.set_workspace (Closes #3262)

New Contributors

• @garimau made their first contribution in https://github.com/argilla-io/argilla/pull/3255
• @adurante92 made their first contribution in https://github.com/argilla-io/argilla/pull/3242

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.11.0...v1.12.0

🔆 Highlights

New owner role and user update command

We've added a new user role, owner, that has permissions over all users, workspaces and datasets in Argilla (like the admin role in earlier versions). From this version, the admin role will only have permissions over datasets and users in workspaces assigned to them.
You can change a user from admin to owner using a simple CLI command: python -m argilla users update argilla --role owner.

Improved user and workspace management

You can now get lists of users and workspaces, create new ones and give users access to workspaces directly from the Python SDK. Note that only owners will have permissions for all these actions. Admins will be able to give users access to workspaces where they have access.

Metadata fields for Feedback records

You can now add metadata information to your records. This is useful to store information that's not needed for the labeling UI but important for downstream usage (e.g., prompt id, model IDs, etc.)

Changelog 1.11.0

Fixed

• Replaced np.float alias by float to avoid AttributeError when using find_label_errors function with numpy>=1.24.0 (#3214).
• Fixed format_as("datasets") when no responses or optional respones in FeedbackRecord, to set their value to what 🤗 Datasets expects instead of just None (#3224).
• Fixed push_to_huggingface() when generate_card=True (default behaviour), as we were passing a sample record to the ArgillaDatasetCard class, and UUIDs introduced in 1.10.0 (#3192), are not JSON-serializable (#3231).
• Fixed from_argilla and push_to_argilla to ensure consistency on both field and question re-construction, and to ensure UUIDs are properly serialized as str, respectively (#3234).

Added

• Added metadata attribute to the Record of the FeedbackDataset (#3194)
• New users update command to update the role for an existing user (#3188)
• New Workspace class to allow users manage their Argilla workspaces and the users assigned to those workspaces via the Python client (#3180)
• Added User class to let users manage their Argilla users via the Python client (#3169).
• Added an option to display tqdm progress bar to FeedbackDataset.push_to_argilla when looping over the records to upload (#3233).

Changed

• The role system now support three different roles owner, admin and annotator (#3104)
• admin role is scoped to workspace-level operations (#3115)
• The owner user is created among the default pool of users in the quickstart, and the default user in the server has now owner role (#3248), reverting (#3188).

Deprecated

• As of Python 3.7 end-of-life (EOL) on 2023-06-27, Argilla will no longer support Python 3.7 (#3188). More information at https://peps.python.org/pep-0537/

As always, thanks to our amazing contributors!

• @damianpumar made their first contribution in https://github.com/argilla-io/argilla/pull/2950
• @MedAmine-SUDO made their first contribution in https://github.com/argilla-io/argilla/pull/3204
• @manulpatel made their first contribution in https://github.com/argilla-io/argilla/pull/3233

🔆 Highlights

Search records in Feedback Task

We've added a search bar in the Feedback Task UI so you can filter records based on specific words or phrases.

Extended markdown support

Annotation guidelines are now rendered as markdown text to make them easier to read and have a more flexible format.

Train button in Feedback Task

Admin users have access to a Train </> button in the Feedback Task UI with quick links to all the information needed to train a model with the feedback gathered in Argilla.

Changelog 1.10.0

Added

• Added search component for feedback datasets (#3138)
• Added markdown support for feedback dataset guidelines (#3153)
• Added Train button for feedback datasets (#3170)

Changed

• Updated SearchEngine and POST /api/v1/me/datasets/{dataset_id}/records/search to return the total number of records matching the search query (#3166)

Fixed

• Replaced Enum for string value in URLs for client API calls (Closes #3149)
• Resolve breaking issue with ArgillaSpanMarkerTrainer for Named Entity Recognition with span_marker v1.1.x onwards.
• Move ArgillaDatasetCard import under @requires_version decorator, so that the ImportError on huggingface_hub is handled properly (#3174)
• Allow flow FeedbackDataset.from_argilla -> FeedbackDataset.push_to_argilla under different dataset names and/or workspaces (#3192)

As always, thanks to our amazing contributors!

• @hjain5164 made their first contribution in https://github.com/argilla-io/argilla/pull/3146
• @Fancman made their first contribution in https://github.com/argilla-io/argilla/pull/3150
• @preetgami made their first contribution in https://github.com/argilla-io/argilla/pull/3196

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.9.0...v1.10.0

🔆 Highlights

New question types in Feedback Datasets

We've included two new question types in Feedback Datasets: LabelQuestion and MultiLabelQuestion. These are specially useful for applying one or multiple labels to a record, for example, for text classification tasks. In this new view, you can add multiple classification questions and even combine them with the other question types available in Feedback Datasets: RatingQuestion and TextQuestion.

Markdown support in Feedback Fields and Text Questions

You can now add the use_markdown=True tag to a TextField or a TextQuestion to have the UI render the text as markdown. You can use this to read and write code, tables or even add images.

Further improvements in Feedback Datasets

We continue to add improvements to our new Feedback Datasets:

• We've added checks to avoid having fields and questions with repeated names.
• Dataset cards generated using FeedbackDataset.push_to_huggingface(generate_card=True) now follow the official Hugging Face template.

Changelog 1.9.0

Added

• Added boolean use_markdown property to TextFieldSettings model (#3000)
• Added boolean use_markdown property to TextQuestionSettings model (#3000).
• Added new status draft for the Response model (#3033)
• Added LabelSelectionQuestionSettings class allowing to create label selection (single-choice) questions in the API (#3005)
• Added MultiLabelSelectionQuestionSettings class allowing to create multi-label selection (multi-choice) questions in the API (#3010).
• Added POST /api/v1/me/datasets/{dataset_id}/records/search endpoint (#3068).
• Added new components in feedback task Question form: MultiLabel (#3064) and SingleLabel (#3016).
• Added docstrings to the pydantic.BaseModels defined at argilla/client/feedback/schemas.py (#3137)

Changed

• Updated GET /api/v1/me/datasets/:dataset_id/metrics output payload to include the count of responses with draft status (#3033)
• Database setup for unit tests. Now the unit tests use a different database than the one used by the local Argilla server (Closes #2987).
• Updated alembic setup to be able to autogenerate revision/migration scripts using SQLAlchemy metadata from Argilla server models (#3044)
• Improved DatasetCard generation on FeedbackDataset.push_to_huggingface when generate_card=True, following the official HuggingFace Hub template, but suited to FeedbackDatasets from Argilla (#3110)

Fixed

• Disallow fields and questions in FeedbackDataset with the same name (#3126).

As always, thanks to our amazing contributors!

• @gitrock made their first contribution in https://github.com/argilla-io/argilla/pull/3091
• @ChadDa3mon made their first contribution in https://github.com/argilla-io/argilla/pull/3092

🔆 Highlights

New Feedback Task 🎉

In addition, these datasets support multiple annotations: all users with access to the dataset can give their responses.

The FeedbackDataset has an enhanced integration with the Hugging Face Hub, so that saving a dataset to the Hub or pushing a FeedbackDataset from the Hub directly to Argilla is seamless.

Check all the things you can do with Feedback Tasks in our docs

New LLM section in our docs

We've added a new section in our docs that covers:

• Useful concepts around work with LLMs
• How-to guides that cover all the functionalities of the new Feedback Task
• End-to-end examples

More training integrations

We've added new frameworks for the ArgillaTrainer: ArgillaPeftTrainer for Text and Token Classification and ArgillaAutoTrainTrainer for Text Classification.

Changelog 1.8.0

Added

• /api/v1/datasets new endpoint to list and create datasets ([#2615]).
• /api/v1/datasets/{dataset_id} new endpoint to get and delete datasets ([#2615]).
• /api/v1/datasets/{dataset_id}/publish new endpoint to publish a dataset ([#2615]).
• /api/v1/datasets/{dataset_id}/questions new endpoint to list and create dataset questions ([#2615])
• /api/v1/datasets/{dataset_id}/fields new endpoint to list and create dataset fields ([#2615])
• /api/v1/datasets/{dataset_id}/questions/{question_id} new endpoint to delete a dataset questions ([#2615])
• /api/v1/datasets/{dataset_id}/fields/{field_id} new endpoint to delete a dataset field ([#2615])
• /api/v1/workspaces/{workspace_id} new endpoint to get workspaces by id ([#2615])
• /api/v1/responses/{response_id} new endpoint to update and delete a response ([#2615])
• /api/v1/datasets/{dataset_id}/records new endpoint to create and list dataset records ([#2615])
• /api/v1/me/datasets new endpoint to list user visible datasets ([#2615])
• /api/v1/me/dataset/{dataset_id}/records new endpoint to list dataset records with user responses ([#2615])
• /api/v1/me/datasets/{dataset_id}/metrics new endpoint to get the dataset user metrics ([#2615])
• /api/v1/me/records/{record_id}/responses new endpoint to create record user responses ([#2615])
• showing new feedback task datasets in datasets list ([#2719])
• new page for feedback task ([#2680])
• show feedback task metrics ([#2822])
• user can delete dataset in dataset settings page ([#2792])
• Support for FeedbackDataset in Python client (parent PR [#2615], and nested PRs: [#2949], [#2827], [#2943], [#2945], [#2962], and [#3003])
• Integration with the HuggingFace Hub ([#2949])
• Added ArgillaPeftTrainer for text and token classification #2854
• Added predict_proba() method to ArgillaSetFitTrainer
• Added ArgillaAutoTrainTrainer for Text Classification #2664
• New database revisions command showing database revisions info [#2615]: https://github.com/argilla-io/argilla/issues/2615

Fixes

• Avoid rendering html for invalid html strings in Text2text ([#2911]https://github.com/argilla-io/argilla/issues/2911)

Changed

• The database migrate command accepts a --revision param to provide specific revision id
• tokens_length metrics function returns empty data (#3045)
• token_length metrics function returns empty data (#3045)
• mention_length metrics function returns empty data (#3045)
• entity_density metrics function returns empty data (#3045)

Deprecated

• Using argilla with python 3.7 runtime is deprecated and support will be removed from version 1.9.0 (#2902)
• tokens_length metrics function has been deprecated and will be removed in 1.10.0 (#3045)
• token_length metrics function has been deprecated and will be removed in 1.10.0 (#3045)
• mention_length metrics function has been deprecated and will be removed in 1.10.0 (#3045)
• entity_density metrics function has been deprecated and will be removed in 1.10.0 (#3045)

Removed

• Removed mention density, tokens_length and chars_length metrics from token classification metrics storage (#3045)
• Removed token char_start, char_end, tag, and score metrics from token classification metrics storage (#3045)
• Removed tags-related metrics from token classification metrics storage (#3045)

As always, thanks to our amazing contributors!

• Fix image alignment on token classification by @cceyda in https://github.com/argilla-io/argilla/pull/2779
• Update cloud_providers.md by @chainyo in https://github.com/argilla-io/argilla/pull/2866

🔆 Highlights

OpenAI fine-tuning support

Use your data in Argilla to fine-tune OpenAI models. You can do this by getting your data in the specific format through the prepare_for_training method or train directly using ArgillaTrainer.

Argilla Trainer improvements

We’ve added CLI support for Argilla Trainer and two new frameworks for training: OpenAI & SpanMarker.

Logging and loading enhancements

We’ve improved the speed and robustness of rg.log and rg.load methods.

typer CLI

A more user-friendly command line interface with typer that includes argument suggestions and colorful messages.

Changelog 1.7.0

Added

• add max_retries and num_threads parameters to rg.log to run data logging request concurrently with backoff retry policy. See #2458 and #2533
• rg.load accepts include_vectors and include_metrics when loading data. Closes #2398
• Added settings param to prepare_for_training (#2689)
• Added prepare_for_training for openai (#2658)
• Added ArgillaOpenAITrainer (#2659)
• Added ArgillaSpanMarkerTrainer for Named Entity Recognition (#2693)
• Added ArgillaTrainer CLI support. Closes (#2809)

Changed

• Argilla quickstart image dependencies are externalized into quickstart.requirements.txt. See #2666
• bulk endpoints will upsert data when record id is present. Closes #2535
• moved from click to typer CLI support. Closes (#2815)
• Argilla server docker image is built with PostgreSQL support. Closes #2686
• The rg.log computes all batches and raise an error for all failed batches.
• The default batch size for rg.log is now 100.

Fixed

• argilla.training bugfixes and unification (#2665)
• Resolved several small bugs in the ArgillaTrainer.

Deprecated

• The rg.log_async function is deprecated and will be removed in next minor release.

As always, thanks to out amazing contributors!

• docs: Fix broken links in README.md (#2759) by @stephantul
• Update how_to.ipynb by @chainyo
• Update log_load_and_prepare_data.ipynb by @ignacioct

🔆 Highlights

User roles & settings page

We've introduced two user roles to help you manage your annotation team: admin and annotator. admin users can create, list and delete other users, workspaces and datasets. The annotator role is specifically designed for users who focus solely on annotating datasets.

We've also added a page to see your user's settings in the Argilla UI. To access it click on your user avatar at the top right corner and then select My settings.

Argilla Trainer

The new Argilla.training module deals with all data transformations and basic default configurations to train a model with annotations from Argilla using popular NLP frameworks. It currently supports spacy, setfit and transformers.

Additionally, admin users can access ready-made code snippets to copy-paste directly from the Argilla UI. Just go to the dataset you want to use, click the </> Train button in the top banner and select your preferred framework.

Learn more about Argilla.training in our docs.

Database support

Argilla will now create a default SQLite database to store users and workspaces. PostgreSQL is also officially supported. Simply set a custom value for the ARGILLA_DATABASE_URL environment variable pointing to your PostgreSQL instance.

Changelog 1.6.0

Added

• ARGILLA_HOME_PATH new environment variable (#2564).
• ARGILLA_DATABASE_URL new environment variable (#2564).
• Basic support for user roles with admin and annotator (#2564).
• id, first_name, last_name, role, inserted_at and updated_at new user fields (#2564).
• /api/users new endpoint to list and create users (#2564).
• /api/users/{user_id} new endpoint to delete users (#2564).
• /api/workspaces new endpoint to list and create workspaces (#2564).
• /api/workspaces/{workspace_id}/users new endpoint to list workspace users (#2564).
• /api/workspaces/{workspace_id}/users/{user_id} new endpoint to create and delete workspace users (#2564).
• argilla.tasks.users.migrate new task to migrate users from old YAML file to database (#2564).
• argilla.tasks.users.create new task to create a user (#2564).
• argilla.tasks.users.create_default new task to create a user with default credentials (#2564).
• argilla.tasks.database.migrate new task to execute database migrations (#2564).
• release.Dockerfile and quickstart.Dockerfile now creates a default argilladata volume to persist data (#2564).
• Add user settings page. Closes #2496
• Added Argilla.training module with support for spacy, setfit, and transformers. Closes #2504

Fixes

• Now the prepare_for_training method is working when multi_label=True. Closes #2606

Changed

• ARGILLA_USERS_DB_FILE environment variable now it's only used to migrate users from YAML file to database (#2564).
• full_name user field is now deprecated and first_name and last_name should be used instead (#2564).
• password user field now requires a minimum of 8 and a maximum of 100 characters in size (#2564).
• quickstart.Dockerfile image default users from team and argilla to admin and annotator including new passwords and API keys (#2564).
• Datasets to be managed only by users with admin role (#2564).
• The list of rules is now accessible while metrics are computed. Closes#2117
• Style updates for weak labelling and adding feedback toast when delete rules. See #2626 and #2648

Removed

• email user field (#2564).
• disabled user field (#2564).
• Support for private workspaces (#2564).
• ARGILLA_LOCAL_AUTH_DEFAULT_APIKEY and ARGILLA_LOCAL_AUTH_DEFAULT_PASSWORD environment variables. Use python -m argilla.tasks.users.create_default instead (#2564).
• The old headers for API Key and workspace from python client
• The default value for old API Key constant. Closes #2251

As always, thanks to our amazing contributors!

• feat: add ArgillaSpaCyTrainer for both TokenClassification and TextClassification (#2604) by @alvarobartt
• Move dataset dump to train, ignored unnecessary imports, & remove _required_fields attribute (#2642) by @alvarobartt
• fix: update field name in metadata for image url (#2609) by @burtenshaw
• fix Install doc spell error by @PhilipMay
• fix: broken README.md link (#2616) by @alvarobartt

1.5.1

Fixes

• Copying datasets between workspaces with proper owner/workspace info. Closes #2562
• Copy dataset with empty workspace to the default user workspace. See #2618
• Using elasticsearch config to request backend version. Closes #2311
• Remove sorting by score in labels. Closes #2622

Changed

• Update field name in metadata for image url. See #2609

1.4.1

Bug Fixes

• Copying datasets between workspaces with proper owner/workspace info. Closes #2562
• Copy dataset with empty workspace to the default user workspace 905d4de
• Using elasticsearch config to request backend version. Closes #2311

1.3.2

Bug Fixes

• Copying datasets between workspaces with proper owner/workspace info. Closes #2562
• Copy dataset with empty workspace to the default user workspace 905d4de
• Using elasticsearch config to request backend version. Closes #2311