pramen

What's Changed

• #410 Improve Spark sink to allow output to table and connection data targets in https://github.com/AbsaOSS/pramen/pull/412
• #398 Do not correct decimals in schema if it is correct already in https://github.com/AbsaOSS/pramen/pull/413
• #415 Extend notification target interface to include table definition of the table in https://github.com/AbsaOSS/pramen/pull/416

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.8...v1.8.9

What's Changed

• #407 Signal handling always stopping the pipeline, forward the signal to the original handler instead in https://github.com/AbsaOSS/pramen/pull/408

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.7...v1.8.8

What's Changed

• #403 Add the ability to get currently completed tasks from anywhere from user's code in https://github.com/AbsaOSS/pramen/pull/404
• #405 Update the version of the channels library in https://github.com/AbsaOSS/pramen/pull/405

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.6...v1.8.7

What's Changed

• #372 Add global notification targets in https://github.com/AbsaOSS/pramen/pull/400
• #399 Add ability to use date expressions and formatting in JDBC Native sql expressions in https://github.com/AbsaOSS/pramen/pull/401

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.5...v1.8.6

What's Changed

• #394 Fix PostgreSQL with numeric fields with no permission and scale in https://github.com/AbsaOSS/pramen/pull/395
• #349 Allow add partition for updating hive tables in https://github.com/AbsaOSS/pramen/pull/396

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.4...v1.8.5

What's Changed

• #388 Handle OS signals and include thread stack traces in email notifications in https://github.com/AbsaOSS/pramen/pull/390
• #389 Add Spark application id to jobs that fail before running a task in https://github.com/AbsaOSS/pramen/pull/391
• #387 Implement handling DATE overflows that can happen with PotgreSQL infinity date values in https://github.com/AbsaOSS/pramen/pull/392

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.3...v1.8.4

What's Changed

• Hotfix: handle null values when sanitizing timestamps in https://github.com/AbsaOSS/pramen/pull/385

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.2...v1.8.3

What's Changed

• #375 Add workaround for timestamp overflows in https://github.com/AbsaOSS/pramen/pull/377
• #376 Add support for 'input.table' when JDBC Native is used with supported SQL dialects in https://github.com/AbsaOSS/pramen/pull/380
• #378 Add support for MySQL databases in the JDBC source in https://github.com/AbsaOSS/pramen/pull/381
• #382 Update versions of Scala, Spark and sbt for builds and CI/CD in https://github.com/AbsaOSS/pramen/pull/383

⚠️ A bug is introduced in this version: Ingestion jobs that use JDBCNative source will fail if a column having timestamp type contains nulls. Please, use Pramen 1.8.3 instead.

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.1...v1.8.2

What's Changed

• #362 Get column descriptions from JDBC connection when possible in https://github.com/AbsaOSS/pramen/pull/364
• #361 Add the ability of EnceladusSink to send a REST request to delete temporary version files on S3 in https://github.com/AbsaOSS/pramen/pull/363
• #369 Fix lazy sinks incorrectly made lazy if they depend on transient tables in https://github.com/AbsaOSS/pramen/pull/370
• #367 Do not create directories on object stores unless strictly needed in https://github.com/AbsaOSS/pramen/pull/368
• #366 Track raw files at sources and in the metastore by size instead by count in https://github.com/AbsaOSS/pramen/pull/371

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.8.0...v1.8.1

What's Changed

• #245 Make transient transformers are now executed on-demand in https://github.com/AbsaOSS/pramen/pull/341
• #301 Make SQL generators extensible so that custom generators could be plugged in without Pramen recompilation in https://github.com/AbsaOSS/pramen/pull/352
• #353 Add the option to disable count query to optimize performance of very large JDBC datasets by @yruslan in https://github.com/AbsaOSS/pramen/pull/355
• #351 Add a method to SQL generators interface to return an alias expression. by @yruslan in https://github.com/AbsaOSS/pramen/pull/357
• #354 Add Denodo dialect to support 'timestamp with timezone' data type by @yruslan in https://github.com/AbsaOSS/pramen/pull/356
• #338 Do not include records per seconds in email notification for tables that do not have enough records in https://github.com/AbsaOSS/pramen/pull/338
• #343 Add a GitHub action for automatic releases in https://github.com/AbsaOSS/pramen/pull/343

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.7.4...v1.8.0

What's Changed

• #311 Add the ability to run multiple workflows in a single Spark Session in https://github.com/AbsaOSS/pramen/pull/334
• Fix table header gradient CSS style for some of email clients in https://github.com/AbsaOSS/pramen/pull/334

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.7.3...v1.7.4

What's Changed

• #327 Add status indicators to the email subject, extend statuses with new 'Warning' status in https://github.com/AbsaOSS/pramen/pull/328
• #325 Add quoting policy that help validate and quote identifiers that are passed to SQL queries in https://github.com/AbsaOSS/pramen/pull/329
• #324 Make string to date conversion for MS SQL Server queries graceful in https://github.com/AbsaOSS/pramen/pull/331

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.7.2...v1.7.3

What's Changed

• #314 Make information.date.sql.formatobsolete in https://github.com/AbsaOSS/pramen/pull/318
• #317 Escape column names in SQL queries when it contains non-alphanumeric characters or a possible keyword in https://github.com/AbsaOSS/pramen/pull/320
• #319 Add a trait (ExternalChannelFactoryV2) for creating custom sources or sinks with access to teh workflow configuration in https://github.com/AbsaOSS/pramen/pull/321

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.7.1...v1.7.2

What's Changed

• #286 Add options to set informtation start date relative to the current date in https://github.com/AbsaOSS/pramen/pull/313
• #306 Report a job as New when it never ran, but a rerun is requested in https://github.com/AbsaOSS/pramen/pull/308
• #312 Fix the requirement of 'information.date.format.sql' to be present for the JDBC source. by @yruslan in https://github.com/AbsaOSS/pramen/pull/315

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.7.0...v1.7.1

What's Changed

• #279 When 'trackDays=0', Pramen never checks source data record count if the table is already loaded for the day in https://github.com/AbsaOSS/pramen/pull/280
• #283 Add support for using Spark JDBC connectors for SELECT SQL queries in https://github.com/AbsaOSS/pramen/pull/300
• #291 Combine 'core' and 'runner' modules into 'core' in https://github.com/AbsaOSS/pramen/pull/297
• #292 Fix the last expected info date calculation in https://github.com/AbsaOSS/pramen/pull/293
• #294 Allow multiple transformers output to the same table in https://github.com/AbsaOSS/pramen/pull/296
• #295 Make 'format' parameter of SparkSource optional in https://github.com/AbsaOSS/pramen/pull/295
• #298 Do not include Delta Lake classes in the runner bundle by default in https://github.com/AbsaOSS/pramen/pull/299
• #302 Improve handling of fatal errors in parallel pipeline executions in https://github.com/AbsaOSS/pramen/pull/303
• #307 Open access to more methods of SQL Generators in https://github.com/AbsaOSS/pramen/pull/309

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.5...v1.7.0

What's Changed

• #284 Fix lock/semaphore/channel resource release on fatal errors in https://github.com/AbsaOSS/pramen/pull/285
• #287 Fix sending of email notifications if instantiating of hooks failed in https://github.com/AbsaOSS/pramen/pull/289

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.4...v1.6.5

What's Changed

• #263 Add ability to specify custom startup and shutdown hooks in https://github.com/AbsaOSS/pramen/pull/274
• #275 Implement custom notification signatures for email notifications in https://github.com/AbsaOSS/pramen/pull/277
• #276 Add maximumLength field metadata for JDBC 'varchar' fields in https://github.com/AbsaOSS/pramen/pull/278

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.3...v1.6.4

What's Changed

• #263 Extend custom notification entries with lists, attached files and custom HTML blocks in https://github.com/AbsaOSS/pramen/pull/265
• #266 Allow copying of files between file systems in https://github.com/AbsaOSS/pramen/pull/267
• #268 Add support for Varchar type when exposing tables in Hive in https://github.com/AbsaOSS/pramen/pull/269
• #270 Add Spark 3.5.0 support, update Scala and Spark dependencies to the latest patches in https://github.com/AbsaOSS/pramen/pull/271

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.1...v1.6.3

What's Changed

• #252 Add support for keeping arbitrary metadata for metastore table pertitions in https://github.com/AbsaOSS/pramen/pull/259
• #258 Make Spark Application name configurable in https://github.com/AbsaOSS/pramen/pull/260
• #261 Add postprocessing methods to transformers and sources in https://github.com/AbsaOSS/pramen/pull/262

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.1...v1.6.2

What's Changed

• #244 Implement 'always.allow' flag that allows jobs to proceed even if dependent jobs failed in https://github.com/AbsaOSS/pramen/pull/247
• #248 fFle pattern support in RawFileSource ib https://github.com/AbsaOSS/pramen/pull/251
• #249 Fix 'empty.max' issue popping up when running tasks in parallel in https://github.com/AbsaOSS/pramen/pull/250
• #248 Add support for case-insensitive file patterns in RawFileSource in https://github.com/AbsaOSS/pramen/pull/254
• #255 Change 'Hive' notification column to 'Catalog' in https://github.com/AbsaOSS/pramen/pull/256

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.6.0...v1.6.1