soda-core

What's Changed

• Updates to README for GA. by @janet-can in https://github.com/sodadata/soda-core/pull/1454
• Changes to readme at Maarten's request. Banner for announcement. by @janet-can in https://github.com/sodadata/soda-core/pull/1455
• Re-add telemetry data source attributes by @m1n0 in https://github.com/sodadata/soda-core/pull/1457

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0rc3...v3.0.0

What's Changed

• doc: add comment about ordinal_position ordering by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1428
• refactor: use filesystem abstractions in distribution check by @baturayo in https://github.com/sodadata/soda-core/pull/1423
• fix: distribution check athena compatibility by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1429
• feat: profile and discover view tables by @baturayo in https://github.com/sodadata/soda-core/pull/1416
• Code style section in contrib docs by @m1n0 in https://github.com/sodadata/soda-core/pull/1432
• Unify data source api, remove redundant code. by @m1n0 in https://github.com/sodadata/soda-core/pull/1433
• fix: support athena in column profiling by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1430
• column profiling metadata fix by @tombaeyens in https://github.com/sodadata/soda-core/pull/1431
• feat: Support profile columns inclusion/exclusion behaviour for Spark by @baturayo in https://github.com/sodadata/soda-core/pull/1437
• CORE-63 Added relative percentage change over time by @tombaeyens in https://github.com/sodadata/soda-core/pull/1435
• feat: Raise a MissingBinsAndWeights exception if soda scan runs without distribution_reference present by @tituskx in https://github.com/sodadata/soda-core/pull/1421
• Flatten data source configuration schema by @m1n0 in https://github.com/sodadata/soda-core/pull/1441
• fix: Suppress prophet's pandas: frame.append deprecation warning by @tituskx in https://github.com/sodadata/soda-core/pull/1440
• feat: send outcome reason to cloud for anomaly detection and schema checks by @baturayo in https://github.com/sodadata/soda-core/pull/1390
• Add private key and other extra params to snowflake by @m1n0 in https://github.com/sodadata/soda-core/pull/1446
• feat: refer to DROs by name by @tituskx in https://github.com/sodadata/soda-core/pull/1422
• Change: rename the update command to update-dro as it better describes what the command is used for by @tituskx in https://github.com/sodadata/soda-core/pull/1444
• feat/fix: ensure empty bins for integer columns are not created and fix bin width derivation by @baturayo in https://github.com/sodadata/soda-core/pull/1447
• Do not quote table names in for-each block by @m1n0 in https://github.com/sodadata/soda-core/pull/1449
• feat: add env based option to run tests on views by @vijaykiran in https://github.com/sodadata/soda-core/pull/1442

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0rc2...rc3

What's Changed

• feat: add wasserstein distance and PSI methods to distribution checks by @tituskx in https://github.com/sodadata/soda-core/pull/1395
• CORE-24 New freshness syntax by @tombaeyens in https://github.com/sodadata/soda-core/pull/1400
• Verify that in spark-df arrays & structs don't break anything by @tombaeyens in https://github.com/sodadata/soda-core/pull/1397
• feat: add column exclusion to profile columns by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1396
• feat: log no threshold error during parsing and provide more informative error during check summary by @tituskx in https://github.com/sodadata/soda-core/pull/1401
• CORE-44 Fixed some extra timestamps to utc by @tombaeyens in https://github.com/sodadata/soda-core/pull/1405
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1407
• SODA-23 table dataset rename by @tombaeyens in https://github.com/sodadata/soda-core/pull/1404
• feat: send distribution check results to cloud so that they can be plotted by @tituskx in https://github.com/sodadata/soda-core/pull/1402
• Update README to include support for Amazon Athena by @stuart-robinson in https://github.com/sodadata/soda-core/pull/1409
• refactor: Refactor scan.py to remove code duplicates by @baturayo in https://github.com/sodadata/soda-core/pull/1391
• Update CONTRIBUTING to stipulate that users fork the repo. by @janet-can in https://github.com/sodadata/soda-core/pull/1413
• Core 70 clean test schemas by @tombaeyens in https://github.com/sodadata/soda-core/pull/1415
• fix: hotfix for historic measurements having none values by @baturayo in https://github.com/sodadata/soda-core/pull/1418
• CORE-26 Fix change over time results value parsing by @vijaykiran in https://github.com/sodadata/soda-core/pull/1419
• CORE-57 improved exception handling when creating data source by @tombaeyens in https://github.com/sodadata/soda-core/pull/1411
• Another approach for the Docker image for Soda Core by @jmarien in https://github.com/sodadata/soda-core/pull/1398
• Added 5 random chars to CI schema names by @tombaeyens in https://github.com/sodadata/soda-core/pull/1424
• Fix drop table statement in test suite by @m1n0 in https://github.com/sodadata/soda-core/pull/1425
• SODA-44 Added Z to timestamps in soda cloud json by @tombaeyens in https://github.com/sodadata/soda-core/pull/1408
• Added docs on running tests by @tombaeyens in https://github.com/sodadata/soda-core/pull/1426
• Fix schema check title by @vijaykiran in https://github.com/sodadata/soda-core/pull/1427
• fix: more useful profiling warnings by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1420
• CORE-37 Fixed schema type comparison for BigQuery by @tombaeyens in https://github.com/sodadata/soda-core/pull/1410

New Contributors

• @stuart-robinson made their first contribution in https://github.com/sodadata/soda-core/pull/1409

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0rc1...v3.0.0rc2

What's Changed

• 1175 spark by @m1n0 in https://github.com/sodadata/soda-core/pull/1382

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b19...v3.0.0rc1

What's Changed

• fix: handle %.% in profile columns properly and other bugs by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1377
• Fix: cope with cloud disabled samples. by @m1n0 in https://github.com/sodadata/soda-core/pull/1393
• BQ: regex switch to 'r' instead of backslash escaping by @m1n0 in https://github.com/sodadata/soda-core/pull/1394

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b18...v3.0.0b19

What's Changed

• Scientific package tests on Athena. by @m1n0 in https://github.com/sodadata/soda-core/pull/1374
• Update OT with scan/check counts by @vijaykiran in https://github.com/sodadata/soda-core/pull/1386
• feat: add ability to send dataset samples to soda cloud (SODA-284) by @baturayo in https://github.com/sodadata/soda-core/pull/1372
• fix: typo in data source package import by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1387
• 627 Added default sampler returning a sample that is is not persistent by @tombaeyens in https://github.com/sodadata/soda-core/pull/1385
• feat: cap distribution check to 1M rows by default by @tituskx in https://github.com/sodadata/soda-core/pull/1379
• refactor: clean up logging for anomaly detection by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1389
• fix: avoid parsing DRO name in distribution check until fully implemented by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1388
• Downgrade markupsafe dependency by @m1n0 in https://github.com/sodadata/soda-core/pull/1392

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b17...v3.0.0b18

What's Changed

• Pin versions in core by @vijaykiran in https://github.com/sodadata/soda-core/pull/1383

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b16...v3.0.0b17

What's Changed

• Fixing suffix scanning of configuration and check files by @tombaeyens in https://github.com/sodadata/soda-core/pull/1365
• Refactored to actual table and actual column names by @tombaeyens in https://github.com/sodadata/soda-core/pull/1370
• Send Soda Cloud logs by @tombaeyens in https://github.com/sodadata/soda-core/pull/1380
• Prevent upload when no sample rows are present by @tombaeyens in https://github.com/sodadata/soda-core/pull/1378
• refactor: inform when columns are skipped in profiling via logs by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1375

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b15...v3.0.0b16

What's Changed

• refactor: remove darts dependency by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1362
• refactor: remove code duplication in sodacl_parser by @baturayo in https://github.com/sodadata/soda-core/pull/1361
• SODA-248 fixed change over time checks by @tombaeyens in https://github.com/sodadata/soda-core/pull/1366
• Athena support by @m1n0 in https://github.com/sodadata/soda-core/pull/1367
• Added for each schema check by @tombaeyens in https://github.com/sodadata/soda-core/pull/1368

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b14...v3.0.0b15

What's Changed

• Add defaultDataSource to cloud payload by @vijaykiran in https://github.com/sodadata/soda-core/pull/1359
• fix: provide docker image with soda-scientific packaged by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1355
• Fixing data source validity error message by @tombaeyens in https://github.com/sodadata/soda-core/pull/1357
• Added cython to the setup.py file by @tituskx in https://github.com/sodadata/soda-core/pull/1360
• #1353 SODA-494 Fixing recursive loading of files by @tombaeyens in https://github.com/sodadata/soda-core/pull/1356

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b13...v3.0.0b14

What's Changed

• 1237 samples2 by @tombaeyens in https://github.com/sodadata/soda-core/pull/1340
• Getting disable samples from cloud config by @tombaeyens in https://github.com/sodadata/soda-core/pull/1348
• Test anomaly detection for numeric metrics by @baturayo in https://github.com/sodadata/soda-core/pull/1349
• Updated contributing, fixed logs and added hint in comment how to add… by @tombaeyens in https://github.com/sodadata/soda-core/pull/1350
• Fix: Automated monitoring revert issues SODA-489 by @baturayo in https://github.com/sodadata/soda-core/pull/1351
• Soda 159 - Test anomaly detection for nested metrics by @baturayo in https://github.com/sodadata/soda-core/pull/1354

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b12...v3.0.0b13

What's Changed

• Fix date eu/us formats. by @m1n0 in https://github.com/sodadata/soda-core/pull/1334
• 1237 samples by @tombaeyens in https://github.com/sodadata/soda-core/pull/1328
• feat: column profiling by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1322
• Switch to latest prophet by @vijaykiran in https://github.com/sodadata/soda-core/pull/1335
• throw log error and return empty string if histogram assumption broken by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1337
• Freshness send microseconds to cloud. by @m1n0 in https://github.com/sodadata/soda-core/pull/1338
• Cloud: timestamps use seconds resolution by @m1n0 in https://github.com/sodadata/soda-core/pull/1339
• Deleted docs folder by @tombaeyens in https://github.com/sodadata/soda-core/pull/1343
• feat: implement automated monitoring executor/runner by @baturayo in https://github.com/sodadata/soda-core/pull/1323
• feat: add table discovery by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1341
• fix(profiling): allow null results in text column aggregates by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1344
• Fix: update check identity in case of automated monitoring by @baturayo in https://github.com/sodadata/soda-core/pull/1346

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b11...v3.0.0b12

What's Changed

• Switched check identity to hash of definition by @tombaeyens in https://github.com/sodadata/soda-core/pull/1302
• Dockerfile by @m1n0 in https://github.com/sodadata/soda-core/pull/1310
• Fail freshness if variable is missing. by @m1n0 in https://github.com/sodadata/soda-core/pull/1304
• Corrected typo in README. by @janet-can in https://github.com/sodadata/soda-core/pull/1312
• Inform user if no checks found. by @m1n0 in https://github.com/sodadata/soda-core/pull/1303
• Disable scientific package for now by @m1n0 in https://github.com/sodadata/soda-core/pull/1313
• Disable scientific tests on standard test run by @m1n0 in https://github.com/sodadata/soda-core/pull/1315
• Separate requirements with sci package by @m1n0 in https://github.com/sodadata/soda-core/pull/1316
• Test for 'for each' check identity. by @m1n0 in https://github.com/sodadata/soda-core/pull/1317
• Automated monitoring feature skip in soda-core scan SODA-202 by @baturayo in https://github.com/sodadata/soda-core/pull/1320
• Minimal python version bump to 3.8 by @m1n0 in https://github.com/sodadata/soda-core/pull/1324
• Fix automated monitoring not implemented warning by @baturayo in https://github.com/sodadata/soda-core/pull/1325
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1327
• CLI return codes. by @m1n0 in https://github.com/sodadata/soda-core/pull/1326
• Fix historic data fetching for change overtime by @vijaykiran in https://github.com/sodadata/soda-core/pull/1331
• 1332 skip sending skipped checks by @vijaykiran in https://github.com/sodadata/soda-core/pull/1333

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b10...v3.0.0b11

What's Changed

• Hotfix: continue with publish pipeline on error by @m1n0 in https://github.com/sodadata/soda-core/pull/1306

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b9...v3.0.0b10

What's Changed

• Update CONTRIBUTING.md on tests. by @m1n0 in https://github.com/sodadata/soda-core/pull/1279
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/sodadata/soda-core/pull/1278
• 1175 refactor data source methods by @m1n0 in https://github.com/sodadata/soda-core/pull/1268
• Handle missing data source. by @m1n0 in https://github.com/sodadata/soda-core/pull/1291
• Handle invalid data source type by @m1n0 in https://github.com/sodadata/soda-core/pull/1292
• feat: anomaly detection check by @bastienboutonnet in https://github.com/sodadata/soda-core/pull/1227
• Use UTC time for scan start/end. by @m1n0 in https://github.com/sodadata/soda-core/pull/1301

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b8...v3.0.0b9

What's Changed

• Remove jinja dependency version restriction by @vijaykiran in https://github.com/sodadata/soda-core/pull/1269
• Support overriding project id in BQ by @m1n0 in https://github.com/sodadata/soda-core/pull/1266

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b7...v3.0.0b8

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b6...v3.0.0b7

What's Changed

• Use thresholds in check name bugfix by @m1n0 in https://github.com/sodadata/soda-core/pull/1247
• Expand user path in file system everytime by @m1n0 in https://github.com/sodadata/soda-core/pull/1249
• Unify datasource metadata query. by @m1n0 in https://github.com/sodadata/soda-core/pull/1264
• Remove antlr dev dependency by @m1n0 in https://github.com/sodadata/soda-core/pull/1271

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b5...v3.0.0b6

What's Changed

• Add missing cloud diagnostics to user defined failed rows check. Fixes #1156 by @m1n0 in https://github.com/sodadata/soda-core/pull/1157
• [soda-cloud] Add user agent request header by @vijaykiran in https://github.com/sodadata/soda-core/pull/1165
• Fix version import by @m1n0 in https://github.com/sodadata/soda-core/pull/1166
• fix: typo in README.md by @duyet in https://github.com/sodadata/soda-core/pull/1172
• Include threshold in check name by @m1n0 in https://github.com/sodadata/soda-core/pull/1170
• Use thresholds in check name only if available by @m1n0 in https://github.com/sodadata/soda-core/pull/1182
• Fix bq json by @m1n0 in https://github.com/sodadata/soda-core/pull/1220

New Contributors

• @duyet made their first contribution in https://github.com/sodadata/soda-core/pull/1172

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b4...v3.0.0b5

Full Changelog: https://github.com/sodadata/soda-core/compare/v3.0.0b3...v3.0.0b4