SDV

Internal

• [dtypes] FixedIncrements Fails with New Numerical Data Types - Issue #2157 by @R-Palazzo

This release enables the HMASynthesizer and other utility functions to work with null foreign key values! It also adds an anonymization method to the metadata classes. Additionally, it patches a bug that lets SDV work with more Pandas data types.

New Features

• Add metadata anonymization to public SDV - Issue #2137 by @R-Palazzo
• Switch drop_missing_values in in drop_unknown_references to support null foreign keys by default - Issue #2076 by @R-Palazzo
• Support nullable foreign keys in HMA - Issue #2063 by @rwedge
• Remove input error from base synthesizer class once nullable foreign keys are supported - Issue #2057 by @rwedge
• Support null foreign keys in get_random_subset - Issue #2056 by @R-Palazzo
• Warn the user if they are trying to save an unfit synthesizer - Issue #1961 by @fealho

Bugs Fixed

• Using FixedCombinations constraint with an integer constraint column causes sampling to fail - Issue #2183 by @R-Palazzo
• Metadata Detection Fails with new Data Type - Issue #2182 by @R-Palazzo
• Unable visualize just the real data (or just the synthetic data) in a multi-table setting - Issue #2160 by @R-Palazzo
• [dtypes] Numerical Formatter Fails to Learn Format of New Data Types - Issue #2156 by @R-Palazzo
• Primary keys may not be unique for variable length regexes - Issue #2116 by @amontanez24
• Confusing warning when using GANs that suggests that CUDA isn't being used - Issue #2052 by @fealho
• PAR DiagnosticReport not 1.0 with float categorical columns - Issue #1910 by @lajohn4747
• In PARSynthesizer I cannot pass in datetime context (InvalidDataError during fitting) - Issue #1485 by @lajohn4747

Internal

• Enabling sdv logging causes tests to fail locally - Issue #2162 by @amontanez24
• Separate primary key detection functionality - Issue #2101 by @amontanez24

Maintenance

• [dtypes] Update the NumericalFormatter to use the learn_rounding_digits from RDT - Issue #2164 by @R-Palazzo
• Mock every usage of is_faker_function to speed up the unit tests - Issue #2163 by @R-Palazzo
• Review docs-related dev dependencies - Issue #2148 by @rwedge
• Cap boto and botocore - Issue #2123 by @lajohn4747

This release adds a new utils function called get_random_sequence_subset, that allows users to get a subset of sequential data.

New Features

• Add utils to the Top Level Package. - Issue #2119 by @pvk-developer
• Add a utility function get_random_sequence_subset - Issue #2085 by @amontanez24

Bugs Fixed

• Context column cannot be a sequence key: Need better error message for this case - Issue #2097 by @gsheni
• Primary key and sequential key cannot be the same - Issue #2096 by @lajohn4747
• Error when applying FixedCombinations constraint on a child table with multiple parents in HMASynthesizer - Issue #2087 by @pvk-developer
• PARSynthesizer errors during fit if sequence_index is numerical sdtype - Issue #2079 by @lajohn4747
• Cap numpy to less than 2.0.0 until SDV supports - Issue #2075 by @gsheni
• Rename the file_name parameter to filepath parameter in ExcelHandler - Issue #2065 by @lajohn4747
• HMA sampling crashes when unknown sdtype detected for numerical column - Issue #2064 by @lajohn4747
• HMA Synthesizer's scale parameter doesn't work for small values - Issue #2045 by @lajohn4747
• PAR DiagnosticReport not 1.0 with float categorical columns - Issue #1910 by @lajohn4747
• If a parent has 0/1 children, HMASynthesizer may create constant data - Issue #1895 by @gsheni

Internal

• Add timeouts to requests in release notes script - Issue #2067 by @gsheni
• Investigate HMA case where parent is missing num_rows column - Issue #1703 by @gsheni

Maintenance

• Release notes should not include PRs - Issue #2074 by @amontanez24
• Switch to using ruff for Python linting and code formatting - Issue #1803 by @gsheni

This release provides a number of new features. A big one is that it adds the ability to fit the HMASynthesizer on disconnected schemas! It also enables the PARSynthesizer to work with constraints in certain conditions. More specifically, the PARSynthesizer can now handle constraints as long as the columns involved in the constraints are either exclusively all context columns or exclusively all non-context columns.

Additionally, a verbose parameter was added to the TVAESynthesizer to get a more detailed progress bar. Also, a bug was corrected that renamed the file_path parameter in the ExcelHandler.read() method to filepath as specified in the official SDV docs.

Internal

• Add workflow to generate release notes - Issue #2050 by @amontanez24

Bugs Fixed

• PARSynthesizer: Duplicate sequence index values when sequence_length is higher than real data - Issue #2031 by @lajohn4747
• PARSynthesizer model won't fit if sequence_index is missing - Issue #1972 by @lajohn4747
• DataProcessor never gets assigned a table_name. - Issue #1964 by @fealho

New Features

• Rename file_path to filepath parameter in ExcelHandler - Issue #2055 by @amontanez24
• Enable the ability to run multi table synthesizers on disjointed table schemas - Issue #2047 by @lajohn4747
• Add header to log.csv file - Issue #2046 by @lajohn4747
• If no filepath is provided, do not create a file during sample - Issue #2042 by @lajohn4747
• Add verbosity to TVAESynthesizer - Issue #1990 by @fealho
• Allow constraints in PARSynthesizer (for all context cols, or all non-context columns) - Issue #1936 by @lajohn4747
• Improve error message when sampling on a non-CPU device - Issue #1819 by @fealho
• Better data validation message for auto_assign_transformers - Issue #1509 by @lajohn4747

Miscellaneous

• Do not enforce min/max on sequence index column - Issue #2043
• Include validation check for single table auto_assign_transformers - Issue #2021
• Add the dummy context column to metadata and not to extra_context_column - Issue #2019

This release fixes the ModuleNotFoundError error that was causing the 1.13.0 release to fail.

This release adds a utility function called get_random_subset that helps users get a subset of their multi-table data so that modeling can be done quicker. Given a dictionary of table names mapped to DataFrames, metadata, a main table and a desired number of rows to use for the main table, it will subsample the data in a way that maintains referential integrity.

This release also adds two new local file handlers: the CSVHandler and the ExcelHandler. This enables users to easily load from and save synthetic data to these files types. These handlers return data and metadata in the multi-table format, so we also added the function get_table_metadata to get a SingleTableMetadata object from a MultiTableMetadata object.

Finally, this release fixes some bugs that prevented synthesizers from working with data that had numerical column names.

New Features

• Add get_random_subset poc utility function - Issue #1877 by @R-Palazzo
• Add usage logging - Issue #1903 by @pvk-developer
• Move function drop_unknown_references from poc to be directly under utils - Issue #1947 by @R-Palazzo
• Add CSVHandler - Issue #1949 by @pvk-developer
• Add ExcelHandler - Issue #1950 by @pvk-developer
• Add get_table_metadata function - Issue #1951 by @R-Palazzo
• Save usage log file as a csv - Issue #1974 by @frances-h
• Split out metadata creation from data import in the local files handlers - Issue #1975 by @pvk-developer
• Improve error message when trying to sample before fitting (single table) - Issue #1978 by @R-Palazzo

Bugs Fixed

• Metadata detection crashes when the column names are integers (AttributeError: 'int' object has no attribute 'lower') - Issue #1933 by @lajohn4747
• Synthesizers crash when column names are integers (TypeError: unsupported operand) - Issue #1935 by @lajohn4747
• Switch parameter order in drop_unknown_references - Issue #1944 by @R-Palazzo
• Unexpected NaN values in sequence_index when dataframe isn't reset - Issue #1973 by @fealho
• Fix pandas DtypeWarning in download_demo - Issue #1980 by @fealho

Maintenance

• Only run unit and integration tests on oldest and latest python versions for macos - Issue #1948 by @frances-h

Internal

• Update code to remove FutureWarning related to 'enforce_uniqueness' parameter - Issue #1995 by @pvk-developer

This release makes a number of changes to how id columns are generated. By default, id columns with a regex will now have their values scrambled in the output. Id columns without a regex that are numeric will be created randomly. If they're not numeric, they will have a random suffix.

Additionally, improvements were made to the visibility of the get_loss_values_plot.

New Features

• Create unique id for each synthesizer - Issue #1902 by @pvk-developer
• Generator Discriminator Loss Chart Color Change - Issue #1916 by @lajohn4747
• If using regex to generate values, scramble them - Issue #1921 by @amontanez24
• When generating ids without a regex, create them randomly - Issue #1922 by @frances-h

Maintenance

• Cleanup automated PR workflows - Issue #1926 by @R-Palazzo

Internal

• Add add-on modules to sys.modules - Issue #1924 by @amontanez24

This release adds support for Python 3.12! It also adds a number of feature improvements. It adds a simplify_schema utility function to the sdv.utils.poc module which simplifies multi-table schemas so they can be run using HMASynthesizer. Multi-table data dictionaries can now be saved directly to CSVs using the sdv.datasets.local.save_csvs utility function. Additionally, generator-discriminator loss values can now be plotted directly from CTGAN using the get_loss_values_plot method. This release also adds error messages when trying to load an SDV synthesizer on an older version of the SDV, or when trying to re-fit a synthesizer from an older version of the SDV.

This release also fixes a number of bugs. Metadata auto-detection now validates that all primary keys are unique, and the metadata correctly validates sdtypes in a column relationship. Bugs in the HMASynthesizer that would cause the diagnostic score to not be equal to 1.0 for cardinality and data validity were fixed. Finally, errors in constraints now correctly raise a ConstraintsNotMetError instead of an InvalidData error.

New Features

• sdv helper function for generating generator-discriminator loss charts - Issue #1828 by @lajohn4747
• Add utility function to simplify multi-table schemas - Issue #1832 by @R-Palazzo
• Show an error if I accidentally load an SDV synthesizer on an older version of SDV - Issue #1837 by @pvk-developer
• Show an error when attempting to re-train a synthesizer that was created on a previous SDV version - Issue #1838 by @pvk-developer
• Add warning when user tries to train a model using datetime values without a datetime_format set - Issue #1847 by @pvk-developer
• Add a function to save my multi-table data as CSVs - Issue #1849 by @R-Palazzo
• Deprecate SingleTablePreset (including FastML Preset) - Issue #1855 by @lajohn4747
• Missing error message if the user forgets to add a sequence_key when using PARSynthesizer - Issue #1883 by @frances-h

Internal

• Add dependency checker - Issue #1818 by @frances-h

Bugs Fixed

• Metadata isn't validating sdtypes in a column relationship (public SDV only) - Issue #1781 by @R-Palazzo
• Contextual Anonymization transformers shouldn't be used for primary keys - Issue #1807 by @fealho
• HMASynthesizer diagnostic score is not 1.0 when using 'truncnorm' distribution - Issue #1831 by @frances-h
• InvalidDataError: The provided data does not match the metadata (although it matches) - Issue #1833 by @pvk-developer
• HMA likelihood match should respect cardinality - Issue #1834 by @fealho
• When inappropriately applying ScalarRange constraint, InvalidDataError is being returned instead of ConstraintsNotMetError - Issue #1842 by @pvk-developer
• When inappropriately applying a CustomConstraint, an InvalidDataError is being returned instead of ConstraintsNotMetError - Issue #1856 by @pvk-developer
• Error in Setting IDGenerator for Primary Key columns - Issue #1862 by @lajohn4747
• Metadata auto-detection should ensure primary keys are unique (special sdtypes are not exempt from this rule!) - Issue #1871 by @R-Palazzo

Maintenance

• Support Python 3.12 - Issue #1704 by @fealho
• Add dependency checker - Issue #1818 by @frances-h
• Add bandit workflow - Issue #1881 by @amontanez24

This release adds the poc utility submodule to help users more easily create a proof-of-concept with multi-table datasets. The poc submodule includes the drop_unknown_references utility function to automatically drop unknown references in a multi-table dataset. Additionally, multiple columns in the metadata can now be updated at once using the update_columns and update_columns_metadata methods. The SDV now also warns users when a synthesizer is loaded that was fitted on a different version of the SDV.

New Features

• Make the get_parameters function consistent between synthesizers - Issue #1756 by @fealho
• Reinstate get_table_parameters for the multi-table synthesizers - Issue #1757 by @fealho
• Validate condition and provide user-friendly messages for NaN/missing values (currently unsupported) - Issue #1758 by @fealho
• Improved error message if a column is already present in a relationship - Issue #1770 by @R-Palazzo
• Better error messaging for nullable foreign keys - Issue #1780 by @fealho
• Add a utility to drop unknown references (and enforce referential integrity) - Issue #1792 by @R-Palazzo
• Add update_columns and update_columns_metadata methods to metadata - Issue #1804 by @R-Palazzo
• Add get_column_names method to metadata - Issue #1805 by @frances-h
• Show original error message when plugin fails to load - Issue #1816 by @rwedge
• Show warning when loading a synthesizer on a previously-saved SDV version - Issue #1836 by @pvk-developer
• Add verbosity to drop_unknown_references - Issue #1845 by @R-Palazzo
• Create a poc module for utilities that help with proof-of-concept - Issue #1846 by @pvk-developer

Internal

• Cleanup utils module: Make internal functions private - Issue #1793 by @R-Palazzo
• Transition from using setup.py to pyroject.toml to specify project metadata - Issue #1801 by @R-Palazzo
• Remove bumpversion and use bump-my-version - Issue #1802 by @R-Palazzo

Maintenance

• Transition from using setup.py to pyroject.toml to specify project metadata - Issue #1801 by @R-Palazzo
• Remove bumpversion and use bump-my-version - Issue #1802 by @R-Palazzo
• Add support for RDT 1.10.0 - Issue #1850 by @amontanez24

Bugs Fixed

• INFO logs mention table name as None for single table data - Issue #1814 by @lajohn4747
• Fix drop_unknown_references for null foreign keys - Issue #1820 by @R-Palazzo

This release adds multiple improvements to handling premium transformers and column relationships, including using premium transformers even if the PII flag is set to true. Additionally, the SDV now warns users to save the metadata after auto-detection has been used. Semantic sdtype detection has also been improved to tokenize column names to prevent unexpected substring matches.

This release also fixes a few warning bugs and fixes an issue that would cause metadata.to_dict to fail for metadata loaded from older versions of the SDV. A few synthesizer bugs were also resolved. The quality of the sequence_index for the PARSynthesizer has been improved, and an issue that would cause CTGANSynthesizer, TVAESynthesizer, and CopulaGANSynthesizer to crash if all columns were to be generated from scratch has been fixed.

Bugs Fixed

• HMASynthesizer sometimes creates null values (out-of-bounds parameters synthesized) - Issue #1691 by @fealho
• Unable to conditionally sample some rows when using a ScalarRange constraint - Issue #1737 by @fealho
• Metadata.to_dict fails on metadata instances pre 1.9.0 - Issue #1739 by @amontanez24
• Metadata auto-detection should not assign a primary key if there are NaN values in it - Issue #1740 by @R-Palazzo
• '' object has no attribute '_model' - Issue #1741 by @fealho
• Column relationship warning should be raised during synthesizer initialization only - Issue #1750 by @R-Palazzo
• Improve quality of sequence_index: Move the start dates into the context model - Issue #1760 by @frances-h
• Add-ons warning is raised twice for multi table synthesizers. - Issue #1768 by @R-Palazzo

New Features

• Metadata auto-detection should tokenize words before determining PII - Issue #1725 by @fealho
• Provide a friendlier error if data is stored as dtype 'category' (CTGAN, TVAE) - Issue #1735 by @frances-h
• Allow the ability to easily remove primary keys - Issue #1742 by @frances-h
• Constraint should not be set on columns inside a gps relationship - Issue #1748 by @R-Palazzo
• Set the default transformer for GPS column relationship - Issue #1749 by @R-Palazzo
• Add a version module to align with SDV Enterprise - Issue #1761 by @R-Palazzo
• Warn users to save their metadata file after auto-detecting/updating it - Issue #1762 by @R-Palazzo
• Set the GPSNoiser as default transformer for GPS column relationship - Issue #1767 by @R-Palazzo
• Update transformer assignment logic for handling pii - Issue #1775 by @R-Palazzo

This release makes a number of improvements. It introduces a new concept to the metadata known as column relationships! Column relationships can be used to define when certain groups of columns in a table should be treated as a special concept (eg. address). You can add a column relationship by using the new add_column_relationship method. The metadata detection was also improved by allowing semantic sdtypes (eg. 'email', 'phone_number') to be detected as primary keys.

This release also patches some bugs. An issue messing up the likelihood matching in the HMASynthesizer was resolved. The CTGANSynthesizer no longer fails when using the FixedCombinations constraint. The Inequality constraint was also patched to handle datetimes better.

Deprecations

• The set_address_columns method is deprecated in favor of add_column_relationship.

New Features

• Improve error messages for composite keys - Issue #1684 by @frances-h
• Add column relationship validation to single table metadata - Issue #1698 by @frances-h
• Add add_column_relationship method to single table metadata - Issue #1699 by @frances-h
• Make synthesizers work with column_relationships - Issue #1700 by @R-Palazzo
• Metadata auto-detection should find primary keys of semantic sdtypes - Issue #1724 by @fealho

Bugs Fixed

• InvalidDataError for Inequality constraint (even though data is valid) - Issue #1692 by @fealho
• BaseIndependentSampler crashes because it tries to cast id columns - Issue #1712 by @pvk-developer
• KeyError in CTGANSynthesizer when applying FixedCombinations constraint - Issue #1717 by @pvk-developer
• Fix _get_likelihoods not generating likelihood values - Issue #1720 by @frances-h

This release adds support for the new Diagnostic Report from SDMetrics. This report calculates scores for three basic but important properties of your data: data validity, data structure and in the multi table case, relationship validity. Data validity checks that the columns of your data are valid (eg. correct range or values). Data structure makes sure the synthetic data has the correct columns. Relationship validity checks to make sure key references are correct and the cardinality is within ranges seen in the real data.

Additionally, a few bugs were fixed and functionality was improved around synthesizers. It is now possible to access the loss values for the TVAESynthesizer and CTGANSynthesizer by using the get_loss_values method. The get_parameters method is now more detailed and returns all the parameters used to make a synthesizer. The metadata is now capable of detecting some common pii sdtypes. Finally, a bug that made every parent row generated by the HMASynthesizer have at least one child row was patched. This should improve cardinality.

Maintenance

• Address SettingWithCopyWarning (HMASynthesizer) - Issue #1557 by @pvk-developer
• Bump SDMetrics version - Issue #1702 by @amontanez24

New Features

• Allow me to access loss values for GAN-based synthesizers - Issue #1671 by @frances-h
• Create a unified get_parameters method for all multi-table synthesizers - Issue #1674 by @frances-h
• Set credentials key as variables - Issue #1680 by @R-Palazzo
• Identifying PII Sdtypes in Metadata - Issue #1683 by @R-Palazzo
• Make SDV compatible with the latest SDMetrics - Issue #1687 by @fealho
• SingleTablePreset uses FrequencyEncoder - Issue #1695 by @fealho

Bugs Fixed

• HMASynthesizer creates too much synthetic data (always creates a child for every parent row) - Issue #1673 by @frances-h

This release adds an alert to the CTGANSynthesizer during preprocessing. The alert informs the user if the fitting of the synthesizer is likely to be slow on their schema. Additionally, it is now possible to enforce that sampled datetime values stay within the range of the fitted data!

This release also makes internal changes to support address data in SDV Enterprise.

New Features

• Add set_address_columns method - Issue #1593 by @R-Palazzo
• Update_transformers should raise error on address columns - Issue #1594 by @R-Palazzo
• add_constraints should raise error on address columns - Issue #1595 by @R-Palazzo
• Print alert if CTGANSynthesizer is likely to be slow - Issue #1658 by @fealho
• Set enforce_min_max_values to True for datetime transformers - Issue #1676 by @R-Palazzo

Bugs Fixed

• Unable to visualize metadata (Error: bad label format and CalledProcessError) - Issue #1625 by @fealho
• Can't set address columns after fitting - Issue #1661 by @R-Palazzo

This release improves user messaging in multiple ways. The most notable is that users will now see an alert if the HMASynthesizer is likely to be slow for their data's schema. Additionally, the logger messaging for constraints and the error messaging when setting distributions on non-parametric models was made more detailed.

The visualization plots in the sdv.evaluation sub-package all got a new parameter called plot_type, allowing the users to specify the plot type to use if the one being inferred is not useful. The sdv.datasets.local.load_csvs method now has a parameter called read_csv_parameters, that allow users to specify how the csvs should be read during loading. The same change was also made to the sdv.metadata.multi_table.detect_table_from_csv, sdv.metadata.multi_table.detect_from_csvs and sdv.metadata.single_table.detect_from_csv methods.

Multiple bugs were resolved including one that caused new categories to be created during the sample step of CTGANSynthesizer.

New Features

• Improve debug messages when a constraint falls back to reject sampling approach - Issue #1478 by @amontanez24
• Constraints should work with timezone-aware datetime columns - Issue #1576 by @fealho
• Better error message when trying to get distributions from non-parametric models - PR #1633 by @frances-h
• Add options to read CSV files - Issue #1644 by @lajohn4747
• Print alert if HMASynthesizer is likely to be slow - Issue #1646 by @lajohn4747
• Make SDV compatible with SDMetrics 0.12.1 - Issue #1650 by @pvk-developer
• Make function to estimate number of columns CTGAN produces - Issue #1657 by @fealho

Bugs Fixed

• In get_available_demos, the num_tables column should be an int - Issue #1420 by @lajohn4747
• AttributeError when using specific locale strings (es_AR, fr_BE) - Issue #1439 by @lajohn4747
• Confusing error when passing in an empty dataframe (with constraints) - Issue #1455 by @lajohn4747
• HMASynthesizer: Better error message for learned distributions (misleading fit error) - Issue #1579 by @fealho
• Fix tests in SDV after update in RDT 1.7.1 - Issue #1638 by @lajohn4747
• CTGAN sometimes creates new categories (int data) - Issue #1647 by @pvk-developer
• CTGAN sometimes creates new categories (object data) - Issue #1648 by @pvk-developer
• Better error message if I provide an incompatible sdtype/locale combo - Issue #1653 by @pvk-developer

Several improvements and bug fixes were made in this release. Most notably, the metadata detection was substantially improved. Support for the 'unknown' sdtype was added, providing more flexibility in data representation. The software now attempts to intelligently detect primary keys and identify parent-child relationships in the metadata, streamlining the metadata creation process.

Additionally, issues related to conditional sampling with negative float values, the inability to update transformers for columns created by constraints, and compatibility with numpy version 1.25 and higher were addressed. The default branch was also switched from 'master' to 'main' for better development practices. Various bugs and errors, including those involving HMA and datetime format detection, were also resolved.

New Features

• Improve metadata detection - Issue #1515 by @R-Palazzo
• Support 'unknown' sdtype - Issue #1516 by @R-Palazzo
• Detect primary keys in metadata - Issue #1521 by @frances-h
• Detect relationships in MultiTableMetadata - Issue #1522 by @frances-h
• Make function to estimate number of columns HMA produces. - Issue #1572 by @fealho
• Add wrapper for get_cardinalty_plot - Issue #1573 by @frances-h
• [Metadata detection] Add a cardinality cap when choosing between categorical vs. numerical - Issue #1584 by @pvk-developer
• [Metadata Detection] Only make primary/foreign keys sdtype id (leave others as unknown) - Issue #1598 by @amontanez24
• Check and supply a more descriptive error when trying to use 'gaussian_kde' with HMA - Issue #1604 by @frances-h

Bugs Fixed

• Conditional sampling with negative float values doesn't work - Issue #1161 by @fealho
• Cannot update transformers for columns that get created by constraints (KeyError) - Issue #1454 by @frances-h
• HMA produces KeyError for a schema with 3+ levels of depth - Issue #1558 by @fealho
• Columns consisting of only Nones are being detected as datetime - Issue #1589 by @pvk-developer
• HMASynthesizer throws an error when sampling multi table models with three levels of depths - Issue #1600 by @amontanez24
• ValueError: Invalid distribution specification when setting numerical_distributions on child table (HMA) - Issue #1605 by @fealho
• Bug: updating transformers in DataProcessor resets warning filters - Issue #1618 by @rwedge

Maintenance

• Investigate how to get numpy >1.25 to pass - Issue #1501 by @rwedge
• Switch default branch from master to main - Issue #1550 by @amontanez24

This release makes multiple improvements to the metadata. Both the single and multi table metadata classes now have a validate_data method. This method runs checks to validate the data against the current specifications in the metadata. The SingleTableMetadata.visualize is also improved. The sequence index is now shown in the same section as the sequence key. It also now shows all key and index information (eg. sequence key, primary key, sequence index) in one section.

The CTGANSynthesizer has been made more efficient in the following ways:

1. Boolean columns are now being skipped during preprocess like categorial columns are.
2. It is possible to apply other transformations to categorical columns and have CTGAN skip the one-hot encoding step.

Additional changes include that the columns labeled with the sdtype id will now go through the IDGenerator transformer by default and constraint transformations that were being overwritten during sampling will now be respected.

New Features

• Add validate_data method to Metadata - Issue #1518 by @fealho
• Use IDGenerator for ID columns - Issue #1519 by @frances-h
• Metadata visualization for sequential data: Only create 2 sections - Issue #1543 by @frances-h

Bugs Fixed

• Inefficient CTGAN modeling when adding categorical transformers - Issue #1450 by @fealho
• CTGANSynthesizer is assigning LabelEncoder to boolean columns (instead of None) - Issue #1530 by @fealho
• Metadata visualization for sequential data: Missing sequence index - Issue #1542 by @frances-h
• Constraint outputs are being overwritten in DataProcessor.reverse_transform - Issue #1551 by @amontanez24

This release adds two new methods to the MultiTableMetadata: detect_from_csvs and detect_From_dataframes. These methods allow you to detect metadata for a whole dataset at once by either loading them from a folder or a dictionary mapping table names to the pandas.DataFrames. The SingleTableMetadata can now be visualized! Additionally, there is now a summarized option in the show_table_details parameter of the visualize methods. This will print each sdtype in the table and the number of columns that have that sdtype.

Additionally, this release patches a bug that prevented custom constraints from working on columns that were primary or alternate keys. It also adds support for Python 3.11!

New Features

• Align default transformers between SDV and RDT - Issue #1484 by @R-Palazzo
• Add visualize method to SingleTableMetadata - Issue #1517 by @pvk-developer
• Add detect_from_csvs and detect_from_dataframes methods to MultiTableMetadata - Issue #1520 by @R-Palazzo
• Allow empty tables to be fitted using fit_processed_data - Issue #1524 by @fealho
• Summarized metadata visualization - Issue #1525 by @pvk-developer

Bugs Fixed

• Cannot use custom constraint transforms for certain columns (inconsistent ordering in forward vs. reverse) - Issue #1476 by @fealho
• Cannot create custom constraint with primary key - Issue #1514 by @amontanez24

Maintenance

• Add support for Python 3.11 - Issue #1459 by @fealho

This release fixes a bug that caused the Inequality constraint and others to fail if there were None values in a datetime column.

Bugs Fixed

• Inequality fails with None and datetime - Issue #1471 by @pvk-developer

Maintenance

• Drop support for Python 3.7 - Issue #1487 by @pvk-developer

Internal

• Make HMA use hierarchical sampling mixin - Issue #1428 by @frances-h
• Move progress bar out of base multi table synthesizer - Issue #1486 by @R-Palazzo

This release adds a parameter called verbose to the HMASynthesizer's initialization. Setting it to True will show progress bars during the fitting steps. Additionally, performance optimizations were made to the modeling and initialization of the HMASynthesizer.

Multiple changes were made to enhance constraints. The Range constraint was improved to be able to generate more accurate data when null values are provided. Constraints are also now validated against the data when running validate() on any synthesizer.

Finally, some warnings were resolved.

New Features

• Report fitting progress for the HMASynthesizer - Issue #1440 by @pvk-developer

Bugs Fixed

• Range constraint does not produce cases of missing values & may create violative data - Issue #1393 by @R-Palazzo
• Synthesizers don't validate constraints during validate() - Issue #1402 by @pvk-developer
• Confusing error during metadata validation - Issue #1417 by @frances-h
• SettingWithCopyWarning when conditional sampling - #1436 by @pvk-developer
• HMASynthesizer is modeling child tables - Issue #1442 by @pvk-developer
• ValueError when sampling PII columns - Issue #1445 by @pvk-developer

Internal

• Add BaseHierarchicalSampler Mixin - Issue #1394 by @frances-h
• Add BaseIndependentSampler Mixin - Issue #1395 by @frances-h
• Synthesizers created twice during HMA init - Issue #1418 by @frances-h
• Get rid of unnecessary methods for single table sampling - Issue #1430 by @amontanez24
• Detect all addons from top level init - PR #1453 by @frances-h

Maintenance

• Upgrade to torch 2.0 - Issue #1365 by @fealho
• During fit, there is a FutureWarning (due to RDT 1.5.0) - Issue #1456 by @amontanez24

This release adds a new initialization parameter to synthesizers called locales that allows users to set the locales to use for all columns that have a locale based sdtype (eg. address or phone_number). Additionally, it adds support for Pandas 2.0!

Multiple enhancements were made to improve the performance of data and metadata validation in synthesizers. The Inequality constraint was improved to be able to generate more scenarios of data concerning the presence of NaNs. Finally, many warnings have been resolved.

New Features

• Add add-on detection for new constraints - Issue #1397 by @frances-h
• Add add-on detection for multi and single table synthesizers - Issue #1385 by @frances-h
• Setting a locale for all my anonymized (PII) columns - Issue #1371 by @frances-h

Bugs Fixed

• Skip checking for Faker function if its a default sdtype - PR #1410 by @pvk-developer
• Inequality constraint does not produce all possibilities of missing values - Issue #1392 by @R-Palazzo
• Deprecated locale warning - Issue #1400 by @frances-h

Maintenance

• Use cached Faker instance to discover if an sdtype is a Faker function. - Issue #1405 by @pvk-developer
• Upgrade to pandas 2.0 - Issue #1366 by @pvk-developer

Internal

• Refactor Multi Table Modeling - Issue #1387 by @pvk-developer
• PytestConfigWarning: Unknown config option: collect_ignore - Issue #1376 by @amontanez24
• Pandas FutureWarning: Could not cast to int64 - Issue #1357 by @R-Palazzo
• RuntimeWarning: invalid value encountered in cast. - Issue #1369 by @amontanez24