soda-core

What's Changed

• [dremio] fixes for profiling and discovery by @vijaykiran in https://github.com/sodadata/soda-core/pull/1764
• Attributes: add timezone to dates by @m1n0 in https://github.com/sodadata/soda-core/pull/1765
• deprecation/fix: actually remove execution of row count query in table discovery by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1763
• add dataset filter test case starting with % by @baturayo in https://github.com/sodadata/soda-core/pull/1756

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.19...v3.0.20

What's Changed

• Sqlserver: fix regex like method by @m1n0 in https://github.com/sodadata/soda-core/pull/1669
• Update GH Actions version by @vijaykiran in https://github.com/sodadata/soda-core/pull/1759

Full Changelog: https://github.com/sodadata/soda-core/compare/3.0.18...v3.0.19

What's Changed

• [core] apply in-check filters to duplicate check by @m1n0 in https://github.com/sodadata/soda-core/pull/1748
• [core] Deprecate global data source 'disable_samples', use one from sampler by @m1n0 in https://github.com/sodadata/soda-core/pull/1749
• [profiling] feat: add row count to profiling by @baturayo in https://github.com/sodadata/soda-core/pull/1747
• [core] Fix for user defined query check failure on zero result by @vijaykiran in https://github.com/sodadata/soda-core/pull/1750
• [cloud] Get available check attributes schema using correct cloud api by @m1n0 in https://github.com/sodadata/soda-core/pull/1751
• [core] Remove unused soda_cloud property by @m1n0 in https://github.com/sodadata/soda-core/pull/1753
• [core] Better warning message for invalid check without valid spec by @m1n0 in https://github.com/sodadata/soda-core/pull/1752
• [dremio] fix profiling for schemas with s by @vijaykiran in https://github.com/sodadata/soda-core/pull/1757
• [cloud] Skip all checks if invalid attributes found by @m1n0 in https://github.com/sodadata/soda-core/pull/1758

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.17...v3.0.18

What's Changed

• Custom query null result handling by @m1n0 in https://github.com/sodadata/soda-core/pull/1739
• Duplicate count workaround for complex types by @m1n0 in https://github.com/sodadata/soda-core/pull/1738
• Warn about unsupported multi-argument numeric checks by @m1n0 in https://github.com/sodadata/soda-core/pull/1741 and https://github.com/sodadata/soda-core/pull/1742
• Send check attributes to cloud by @m1n0 in https://github.com/sodadata/soda-core/pull/1743
• [API] Experimental connecton API by @vijaykiran in https://github.com/sodadata/soda-core/pull/1745
• [DREMIO] fix profiling issues by @vijaykiran in https://github.com/sodadata/soda-core/pull/1746

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.16...v3.0.17

What's Changed

• Bump dependency versions by @vijaykiran in https://github.com/sodadata/soda-core/pull/1728
• Remove cloud traces from telemetry by @m1n0 in https://github.com/sodadata/soda-core/pull/1729
• [dremio] fix profiling query by @vijaykiran in https://github.com/sodadata/soda-core/pull/1730
• log metric identity when getting historical metrics for anomaly check for easier debugging by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1731
• Check attributes validation by @m1n0 in https://github.com/sodadata/soda-core/pull/1727
• [databricks/spark] fix listing of tables by @vijaykiran in https://github.com/sodadata/soda-core/pull/1736
• Fix schema info with partition info by @vijaykiran in https://github.com/sodadata/soda-core/pull/1737

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.15...v3.0.16

What's Changed

• Core 267 variables by @m1n0 in https://github.com/sodadata/soda-core/pull/1716
• Add snowflake arg to allow temp credential file in Linux by @wintersrd in https://github.com/sodadata/soda-core/pull/1714
• Do not upload more than 100 rows to Cloud when no limit is specified by @m1n0 in https://github.com/sodadata/soda-core/pull/1719
• Bump requests version by @m1n0 in https://github.com/sodadata/soda-core/pull/1721
• Fix history-loss when custom identity is provided by @vijaykiran in https://github.com/sodadata/soda-core/pull/1720
• Check attributes by @m1n0 in https://github.com/sodadata/soda-core/pull/1718
• Fix json serialisation for HTTPSampler by @vijaykiran in https://github.com/sodadata/soda-core/pull/1723

New Contributors

• @wintersrd made their first contribution in https://github.com/sodadata/soda-core/pull/1714

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.14...v3.0.15

What's Changed

• Updates to list of compatible data sources. by @janet-can in https://github.com/sodadata/soda-core/pull/1694
• Date format fixes by @vijaykiran in https://github.com/sodadata/soda-core/pull/1691
• Sqlserver: fix email regex, do not allow empty string by @m1n0 in https://github.com/sodadata/soda-core/pull/1688
• Spark: respect verbose setting when running a test query by @m1n0 in https://github.com/sodadata/soda-core/pull/1697
• fix: ensure that profiling does not lowercase columns by @tituskx in https://github.com/sodadata/soda-core/pull/1687
• Variables everywhere by @m1n0 in https://github.com/sodadata/soda-core/pull/1700
• Update docker by @vijaykiran in https://github.com/sodadata/soda-core/pull/1699
• Refactor duplicate check into two queries by @m1n0 in https://github.com/sodadata/soda-core/pull/1698
• Updates to HTTP Sampler: by @vijaykiran in https://github.com/sodadata/soda-core/pull/1702
• deprecation: remove row-count derivation from dataset discovery by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1706
• Support variables in configuration by @m1n0 in https://github.com/sodadata/soda-core/pull/1705
• Fix schema checks with table filter by @m1n0 in https://github.com/sodadata/soda-core/pull/1704
• duckdb support (experimental) by @vijaykiran in https://github.com/sodadata/soda-core/pull/1709
• Denodo support (experimental) by @vijaykiran in https://github.com/sodadata/soda-core/pull/1710

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.13...v3.0.14

What's Changed

• Support Date type in freshness checks by @m1n0 in https://github.com/sodadata/soda-core/pull/1667
• feat: support sampling for distribution checks by @baturayo in https://github.com/sodadata/soda-core/pull/1666
• Log current time to logs by @m1n0 in https://github.com/sodadata/soda-core/pull/1676
• Add file/check location to scan summary by @m1n0 in https://github.com/sodadata/soda-core/pull/1675
• Regex: support '+' in email format by @m1n0 in https://github.com/sodadata/soda-core/pull/1677
• Generate passing query for built-in checks by @m1n0 in https://github.com/sodadata/soda-core/pull/1668
• Test statistical functions on all data sources by @m1n0 in https://github.com/sodadata/soda-core/pull/1678
• Core 349 samples limit by @m1n0 in https://github.com/sodadata/soda-core/pull/1685
• Add message and link configuration options by @vijaykiran in https://github.com/sodadata/soda-core/pull/1686
• Oracle DB Support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1682
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1684
• Fixes to sampler, add logging by @vijaykiran in https://github.com/sodadata/soda-core/pull/1690

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.12...v3.0.13

What's Changed

• [core] Duplicate percent check by @m1n0 in https://github.com/sodadata/soda-core/pull/1649
• [core] Change over time - remove 'same day last month' by @m1n0 in https://github.com/sodadata/soda-core/pull/1648
• [cloud] don't send null file ref, when failed rows are disabled by @vijaykiran in https://github.com/sodadata/soda-core/pull/1650
• [core] Modify Test Column Names by @tdstark in https://github.com/sodadata/soda-core/pull/1652
• [scientific] feat: allow use of in-check filters for distribution checks by @tituskx in https://github.com/sodadata/soda-core/pull/1655
• [trino] Update trino_data_source.py by @ScottAtDisney in https://github.com/sodadata/soda-core/pull/1658
• [sqlserver] Change count to big_count by @vijaykiran in https://github.com/sodadata/soda-core/pull/1660
• [core] Failed rows exclude columns by @m1n0 in https://github.com/sodadata/soda-core/pull/1657
• [core] Introduce http sampler by @vijaykiran in https://github.com/sodadata/soda-core/pull/1665

New Contributors

• @tdstark made their first contribution in https://github.com/sodadata/soda-core/pull/1652

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.11...v3.0.12

What's Changed

• fix: limit the bin size and handle zero division for continious DRO by @baturayo in https://github.com/sodadata/soda-core/pull/1624
• fix: handle DRO generation for columns with 0 rows by @baturayo in https://github.com/sodadata/soda-core/pull/1627
• chore: pin prophet to >=1.1 by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1629
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1630
• Parse cli variables correctly, fix cli tests to actually assert result. by @m1n0 in https://github.com/sodadata/soda-core/pull/1634
• refactor: add bins and weights doc link to DRO exception handling logs by @baturayo in https://github.com/sodadata/soda-core/pull/1633
• Verify data source connection command by @m1n0 in https://github.com/sodadata/soda-core/pull/1636
• Redshift: use SVV_COLUMNS to get table metadata by @m1n0 in https://github.com/sodadata/soda-core/pull/1635
• Snowflake: use upper case in table metadata query by @m1n0 in https://github.com/sodadata/soda-core/pull/1639
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1643
• Trino: fix py310 type hints by @m1n0 in https://github.com/sodadata/soda-core/pull/1641
• fix: variable substitution in schema check query by @ceyhunkerti in https://github.com/sodadata/soda-core/pull/1628
• fix(anomaly_check): only send outcomeReasons with severity "warn" or "error" by @tituskx in https://github.com/sodadata/soda-core/pull/1640
• fixing bq separate compute storage project by @thiagodeschamps in https://github.com/sodadata/soda-core/pull/1638
• Bigquery: fix distribution check by @m1n0 in https://github.com/sodadata/soda-core/pull/1647
• Change over time - add same day/month support by @m1n0 in https://github.com/sodadata/soda-core/pull/1645

New Contributors

• @ceyhunkerti made their first contribution in https://github.com/sodadata/soda-core/pull/1628
• @thiagodeschamps made their first contribution in https://github.com/sodadata/soda-core/pull/1638

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.10...v3.0.11

What's Changed

• Failed rows configurable sample size by @m1n0 in https://github.com/sodadata/soda-core/pull/1608
• Skip change over time checks when historical measurements not available by @m1n0 in https://github.com/sodadata/soda-core/pull/1615
• Add Check source field for cloud by @m1n0 in https://github.com/sodadata/soda-core/pull/1614
• Use correct dicts when building scan result by @m1n0 in https://github.com/sodadata/soda-core/pull/1612
• Failed rows configurable sample size for duplicate_count check by @m1n0 in https://github.com/sodadata/soda-core/pull/1613
• feat: check historical metrics are not None or log helpful message by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1600
• Dremio support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1618
• fix: handle very large bin sizes by filtering out outliers for dro generation by @baturayo in https://github.com/sodadata/soda-core/pull/1616
• Include psycopg2 requirement for redshift by @m1n0 in https://github.com/sodadata/soda-core/pull/1620
• fix: ensure PSI and SWD can deal with decimal.Decimal type by @tituskx in https://github.com/sodadata/soda-core/pull/1611

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.9...v3.0.10

What's Changed

• Limit failed rows sample limit to 1000 by @m1n0 in https://github.com/sodadata/soda-core/pull/1599
• Add scan result getter by @m1n0 in https://github.com/sodadata/soda-core/pull/1602
• BQ separate project for compute and storage. by @m1n0 in https://github.com/sodadata/soda-core/pull/1598
• Scan results file argument by @vijaykiran in https://github.com/sodadata/soda-core/pull/1603
• Chore/move snowflake account by @jmarien in https://github.com/sodadata/soda-core/pull/1607
• Use filename in check identity by @m1n0 in https://github.com/sodadata/soda-core/pull/1606

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.8...v3.0.9

Please note that installing soda-core may fail with latest version of protobuf (https://github.com/protocolbuffers/protobuf/issues/10571), if the bug isn't fixed yet, try installing 3.19.4 or 3.20.1

What's Changed

• feat: raise more user-friendly log message when improting sci library fails by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1584
• Fix dbt table name by @vijaykiran in https://github.com/sodadata/soda-core/pull/1587
• SODA-914: add context authentication and impersonation for BigQuery by @tooobsias in https://github.com/sodadata/soda-core/pull/1588
• Basic Sqlserver regex support by @m1n0 in https://github.com/sodadata/soda-core/pull/1586
• Fix change over time with percentage with no extra config by @m1n0 in https://github.com/sodadata/soda-core/pull/1592
• Fix mysql/mariadb compatibility for regex by @vijaykiran in https://github.com/sodadata/soda-core/pull/1591
• Minor edits to cli help messages. by @janet-can in https://github.com/sodadata/soda-core/pull/1590
• fix: prevent empty message in outcomeReasons by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1596
• Scan results dict by @m1n0 in https://github.com/sodadata/soda-core/pull/1595
• Add variable resolution to query level by @vijaykiran in https://github.com/sodadata/soda-core/pull/1597

New Contributors

• @tooobsias made their first contribution in https://github.com/sodadata/soda-core/pull/1588

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.7...v3.0.8

What's Changed

• Add dbt: prefix to dbt checks by @vijaykiran in https://github.com/sodadata/soda-core/pull/1574
• Update freshness value to be milliseconds and add measure by @vijaykiran in https://github.com/sodadata/soda-core/pull/1575
• Add configurable API URL for dbt cloud by @vijaykiran in https://github.com/sodadata/soda-core/pull/1576
• Resolve variables in user defined queries by @vijaykiran in https://github.com/sodadata/soda-core/pull/1577
• Fix dbt cloud ingest, improve logging. by @m1n0 in https://github.com/sodadata/soda-core/pull/1578
• Fixed port option and regexp by @ScottAtDisney in https://github.com/sodadata/soda-core/pull/1579
• Fix dbt checks not being sent properly by @vijaykiran in https://github.com/sodadata/soda-core/pull/1580
• Fix regex tests for mysql by @vijaykiran in https://github.com/sodadata/soda-core/pull/1583

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.6...v3.0.7

What's Changed

• fix: add identityB to add datasource name in identity by @vijaykiran in https://github.com/sodadata/soda-core/pull/1556
• Databricks SQL Warehouse support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1559
• SODA-239 Added application flag to snowflake connect by @tombaeyens in https://github.com/sodadata/soda-core/pull/1561
• Add identites by @vijaykiran in https://github.com/sodadata/soda-core/pull/1569
• Add support for custom sampler by @vijaykiran in https://github.com/sodadata/soda-core/pull/1570
• SAS-322 Handle numerical column/table names by @m1n0 in https://github.com/sodadata/soda-core/pull/1572
• dbt ingestion support by @m1n0 in https://github.com/sodadata/soda-core/pull/1552

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.5...v3.0.6

What's Changed

• Update README to include package for mysql by @janet-can in https://github.com/sodadata/soda-core/pull/1532
• Fix duplicate query by @m1n0 in https://github.com/sodadata/soda-core/pull/1543
• Add comments explaining cloud payload by @m1n0 in https://github.com/sodadata/soda-core/pull/1545
• Data source contributing docs by @m1n0 in https://github.com/sodadata/soda-core/pull/1546
• refactor: turn no matching table error into a warning to avoid scan failing when all tables are excluded by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1533
• feat(profiling): add support for extra numeric and text datatypes by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1534
• Fix 'missing format' in numeric metrics by @m1n0 in https://github.com/sodadata/soda-core/pull/1549
• Trino data source by @ScottAtDisney in https://github.com/sodadata/soda-core/pull/1553
• Change spark installation to decouple deps by @vijaykiran in https://github.com/sodadata/soda-core/pull/1554

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.4...v3.0.5

What's Changed

• testing switch to 22.04 for GA by @jmarien in https://github.com/sodadata/soda-core/pull/1521
• Log and trace Soda Cloud trace IDs by @m1n0 in https://github.com/sodadata/soda-core/pull/1520
• Update docker image for sqlserver support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1522
• Add option to set scan datatime by @vijaykiran in https://github.com/sodadata/soda-core/pull/1531
• MySQL Support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1526

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.3...v3.0.4

New Features

• SQLServer support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1515

Fixes

• fix: better logging messages for profiling and discover datasets by @baturayo in https://github.com/sodadata/soda-core/pull/1498
• Fix config file creation when first path is not writable by @m1n0 in https://github.com/sodadata/soda-core/pull/1504
• fix: Failed rows don't consider filter by @vijaykiran in https://github.com/sodadata/soda-core/pull/1505
• Fix log message by @m1n0 in https://github.com/sodadata/soda-core/pull/1507
• Fix reference check for null values in source column by @m1n0 in https://github.com/sodadata/soda-core/pull/1509
• Attach sample rows to reference check by @m1n0 in https://github.com/sodadata/soda-core/pull/1508
• Make sure results to sodacloud are sent when there is an exception by @vijaykiran in https://github.com/sodadata/soda-core/pull/1510
• Fix for regex on collated columns in Snowflake by @ScottAtDisney in https://github.com/sodadata/soda-core/pull/1516

Enhancements

• Check name refactor by @m1n0 in https://github.com/sodadata/soda-core/pull/1502
• Set basic telemetry scan data even in case of exceptions by @m1n0 in https://github.com/sodadata/soda-core/pull/1512
• Improve athena text fixture auth setup by @m1n0 in https://github.com/sodadata/soda-core/pull/1501
• Publish data source packages for python 3.7 by @m1n0 in https://github.com/sodadata/soda-core/pull/1514
• Inform about wrong check indentation in logs by @m1n0 in https://github.com/sodadata/soda-core/pull/1517
• feat: skip row count query during column profiling by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1518
• feat: support 'text' data type in column profiling by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1519

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.2...v3.0.3

What's Changed

• fix: use exception rather than exc_info to render traceback in soda-core logger's call of prophet model by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1462
• feat: Add samples limit to failed rows checks by @vijaykiran in https://github.com/sodadata/soda-core/pull/1464
• fix: make name optional for failed rows by @vijaykiran in https://github.com/sodadata/soda-core/pull/1466
• Reduce the Docker image size to acceptable levels again by @jmarien in https://github.com/sodadata/soda-core/pull/1465
• fix: stored row count in cloud is wrong by @vijaykiran in https://github.com/sodadata/soda-core/pull/1467
• fix: handle exceptions from scientific library and log them instead or letting them raise by @tituskx in https://github.com/sodadata/soda-core/pull/1463
• BQ expose remaining client params and auth methods by @m1n0 in https://github.com/sodadata/soda-core/pull/1456
• Support cli --version to output core version by @m1n0 in https://github.com/sodadata/soda-core/pull/1476
• Spark DF: update example api usage by @m1n0 in https://github.com/sodadata/soda-core/pull/1474
• Change default scan definition name by @m1n0 in https://github.com/sodadata/soda-core/pull/1475
• BQ: remove schema, use dataset only by @m1n0 in https://github.com/sodadata/soda-core/pull/1470
• feat: warn users when quotes are present in include excludes identifiers by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1472
• fix: use default distribution comparison method when user has not provided one by @tituskx in https://github.com/sodadata/soda-core/pull/1469
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1478
• Async Docker image building through Actions and dispatch by @jmarien in https://github.com/sodadata/soda-core/pull/1479
• SODA-944 Enable Snowflake Tokens by @vijaykiran in https://github.com/sodadata/soda-core/pull/1480
• Fix utc timezone handling by @m1n0 in https://github.com/sodadata/soda-core/pull/1481
• test: Improve profiling test for all tables and all columns by @baturayo in https://github.com/sodadata/soda-core/pull/1445
• Fix utc timezone handling by @m1n0 in https://github.com/sodadata/soda-core/pull/1488
• Set redshift host before trying to fetch credentials by @bbakersmith in https://github.com/sodadata/soda-core/pull/1486
• fix: change unassigned min and max variables for profiling logs by @baturayo in https://github.com/sodadata/soda-core/pull/1492
• Use check name in Metric checks by @m1n0 in https://github.com/sodadata/soda-core/pull/1489
• fix: if anomaly detection fails other check results are not sent to cloud by @baturayo in https://github.com/sodadata/soda-core/pull/1490
• Treat zero missing or invalid rows as zero percent by @bbakersmith in https://github.com/sodadata/soda-core/pull/1487
• fix: prevent empty table list from running all tables by @baturayo in https://github.com/sodadata/soda-core/pull/1494
• fix: profile column parsing fails when user provides illegal column spec by @baturayo in https://github.com/sodadata/soda-core/pull/1491
• Join check text with newlines instead of /n by @bbakersmith in https://github.com/sodadata/soda-core/pull/1484
• IBM db2 support by @vijaykiran in https://github.com/sodadata/soda-core/pull/1473
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1495

New Contributors

• @bbakersmith made their first contribution in https://github.com/sodadata/soda-core/pull/1486

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.1...v3.0.2

What's Changed

• re-introduce Spark for the Docker image by @jmarien in https://github.com/sodadata/soda-core/pull/1458
• build: require strict prophet v1.0.0 in scientific library by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1459
• Comment for pinned prophet version by @m1n0 in https://github.com/sodadata/soda-core/pull/1460
• fix the e parameter by @jmarien in https://github.com/sodadata/soda-core/pull/1461

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0...v3.0.1