pramen

What's Changed

• #221 Add support for transient metastore tables in https://github.com/AbsaOSS/pramen/pull/243

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.5.2...v1.6.0

What's Changed

• #234 Allow custom email notification info in https://github.com/AbsaOSS/pramen/pull/236
• #237 Fix redundant schema tracking of metastore tables that have 'raw' format in https://github.com/AbsaOSS/pramen/pull/238
• #239 Add support for column comments in inilne transformations in https://github.com/AbsaOSS/pramen/pull/241

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.5.1...v1.5.2

What's Changed

• #225 Log executor nodes in https://github.com/AbsaOSS/pramen/pull/226
• #227 Add support for verbose log suppression for Spark 3.3+. in https://github.com/AbsaOSS/pramen/pull/228
• #229 Allow passing transformer configuration to the constructor in https://github.com/AbsaOSS/pramen/pull/231
• #232 Add the ability to specify alternative path to Hive tables if it differs from the table path in https://github.com/AbsaOSS/pramen/pull/233

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.5.0...v1.5.1

What's Changed

• #216 Split metastore unit tests into multiple files by @jirifilip in https://github.com/AbsaOSS/pramen/pull/216
• #166 Fix get_latest_available_date for delta format by @jirifilip in https://github.com/AbsaOSS/pramen/pull/217
• #204 Run Python transformations on Databricks by @jirifilip in https://github.com/AbsaOSS/pramen/pull/204
• #165 Read delta tables directly from catalog by @jirifilip in https://github.com/AbsaOSS/pramen/pull/218
• #222 Add file based source and metastore table format by @yruslan in https://github.com/AbsaOSS/pramen/pull/223

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.4.3...v1.5.0

What's Changed

• Hotfix: make sure fields that have empty names are not part of Hive DDL. by @yruslan in https://github.com/AbsaOSS/pramen/pull/213

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.4.2...v1.4.3

What's Changed

• #203 Add spark catalog support as one of Hive integration options by @yruslan in https://github.com/AbsaOSS/pramen/pull/206
• #208 Make AppRunner recreate AppContext on every run by @jirifilip in https://github.com/AbsaOSS/pramen/pull/208
• #209 Improve EnceladusSink to support Hive external table creation by @yruslan in https://github.com/AbsaOSS/pramen/pull/209
• #210 Improve StandardizationSink to support Delta as publish format by @yruslan in https://github.com/AbsaOSS/pramen/pull/210
• #205 Allows spreading object array elements among multiple config files. by @yruslan in https://github.com/AbsaOSS/pramen/pull/211

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.4.1...v1.4.2

What's Changed

• #196 Make Hive enablement an explicit parameter. by @yruslan in https://github.com/AbsaOSS/pramen/pull/199
• #197 Add Spark application Id to email notifications email notifications by @yruslan in https://github.com/AbsaOSS/pramen/pull/200
• #198 Improve exceptions rendering in notifications by @yruslan in https://github.com/AbsaOSS/pramen/pull/200

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.4.0...v1.4.1

What's Changed

• #192 Add ability for a data source to be validated and return validation errors and warnings by @yruslan in https://github.com/AbsaOSS/pramen/pull/193
• #189 Allow expoisng hive tables from metastore by @yruslan in https://github.com/AbsaOSS/pramen/pull/189
• #190 Fix failed tasks log level. by @yruslan in https://github.com/AbsaOSS/pramen/pull/191

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.3.0...v1.4.0

What's Changed

• #178 Add the workaround for the file appending issue. by @yruslan in https://github.com/AbsaOSS/pramen/pull/179
• #175 New Source and Sink interfaces by @yruslan in https://github.com/AbsaOSS/pramen/pull/180
• #182 Add custom query type by @yruslan in https://github.com/AbsaOSS/pramen/pull/183
• #184 Add the method to the Source trait by @yruslan in https://github.com/AbsaOSS/pramen/pull/185

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.2.2...v1.3.0

What's Changed

• #132 Make command line sink more generic by @yruslan in https://github.com/AbsaOSS/pramen/pull/142
• #146 Fix failing tests from extras module on Windows. by @jirifilip in https://github.com/AbsaOSS/pramen/pull/147
• #145 Add an option for failing the pipeline if no data is available at the source by @yruslan in https://github.com/AbsaOSS/pramen/pull/148
• #150 Updated Spark Application Name (Syncwatcher -> Pramen) by @dwfchu in https://github.com/AbsaOSS/pramen/pull/150
• #153 Fix pipeline does not publish task completion events to the journal table. by @yruslan in https://github.com/AbsaOSS/pramen/pull/154
• #151 Specify number of threads to consume by a task. by @jirifilip in https://github.com/AbsaOSS/pramen/pull/156
• #149 Autodetect Enceladus info version by @yruslan in https://github.com/AbsaOSS/pramen/pull/155
• #162 Standardization sink by @yruslan in https://github.com/AbsaOSS/pramen/pull/163
• #167 Make checking of the workflow configuration file existence smarter. by @yruslan in https://github.com/AbsaOSS/pramen/pull/168
• #170 Delete empty folder on write error so readers won't be confused that the partition exists. by @yruslan in https://github.com/AbsaOSS/pramen/pull/172
• #164 Allow creation of Hive tables from StandardizationSink by @yruslan in https://github.com/AbsaOSS/pramen/pull/171
• #173 Unify JDBC retries config by @yruslan in https://github.com/AbsaOSS/pramen/pull/174

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.2.1...v1.2.2

What's Changed

• #129 Add passive dependencies. by @yruslan in https://github.com/AbsaOSS/pramen/pull/131
• #135 Correct rendering of recursive dependencies. by @jirifilip in https://github.com/AbsaOSS/pramen/pull/136
• Readme corrections by @dwfchu in https://github.com/AbsaOSS/pramen/pull/134
• #137 Add support for sequential-only tasks by @yruslan in https://github.com/AbsaOSS/pramen/pull/138
• #139 fail the job if hive refresh failed. by @yruslan in https://github.com/AbsaOSS/pramen/pull/140
• #130 Remove Maven from the project. by @jirifilip in https://github.com/AbsaOSS/pramen/pull/141

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.2.0...v1.2.1

What's Changed

• #78 Added notification component by @yruslan in https://github.com/AbsaOSS/pramen/pull/114
• #84 Fixed metastore get latest available date for delta by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/116
• #85 Implemented passing of format-specific options to pramen-py MetaSto… by @jirifilip in https://github.com/AbsaOSS/pramen/pull/118
• #110 Security update versions of several dependencies. by @yruslan in https://github.com/AbsaOSS/pramen/pull/110
• #111 Fixed issues related to differences between Windows and Unix filesystem paths by @jirifilip in https://github.com/AbsaOSS/pramen/pull/111
• #113 Improved rendering of config arrays and objects in logs. by @jirifilip in https://github.com/AbsaOSS/pramen/pull/117
• #115 Added Spark sink by @jirifilip in https://github.com/AbsaOSS/pramen/pull/125
• #122 Added the ability to override root log level by @yruslan in https://github.com/AbsaOSS/pramen/pull/123
• #123 Changed IdentityTransformer so it won't create a new column. by @yruslan in https://github.com/AbsaOSS/pramen/pull/126

New Contributors

• @jirifilip made their first contribution in https://github.com/AbsaOSS/pramen/pull/111

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.1.2...v1.2.0

What's Changed

• Release Pramen and Pramen-Py 1.1.0 by @yruslan in https://github.com/AbsaOSS/pramen/pull/92
• Update README.md by @dk1844 in https://github.com/AbsaOSS/pramen/pull/94
• #96 Add support for Spark 3.3.x and Delta 2.2.0. by @yruslan in https://github.com/AbsaOSS/pramen/pull/97
• Feature/86 access metastore as a library by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/95
• #88 Added format logging if assert_df_equality return error by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/99
• Bugfix/100 access to metasore reader on asgard by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/103
• #91 Add support of Hive table refresh for the Enceladus sink by @yruslan in https://github.com/AbsaOSS/pramen/pull/104

New Contributors

• @dk1844 made their first contribution in https://github.com/AbsaOSS/pramen/pull/94

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.1.0...v1.1.2

What's Changed

New features

• #65 Add support for Delta metastore format in Pramen-Py by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/81
• #54 Add support for transfer jobs which send data directly from a source to a sink by @yruslan in https://github.com/AbsaOSS/pramen/pull/59
• #36 Implement Spark config per transformers by @zhukovgreen in https://github.com/AbsaOSS/pramen/pull/30
• #36 Allow specifying Pramen-Py transformer Spark config / resources by @yruslan in https://github.com/AbsaOSS/pramen/pull/37
• https://github.com/AbsaOSS/pramen/pull/51 Add support for DB2 by @yruslan in https://github.com/AbsaOSS/pramen/pull/51
• https://github.com/AbsaOSS/pramen/pull/42 Add the ability to specify last day of month as a scheduled day. by @yruslan in https://github.com/AbsaOSS/pramen/pull/42
• https://github.com/AbsaOSS/pramen/pull/44 Add support for configurable column special characters. by @yruslan in https://github.com/AbsaOSS/pramen/pull/44
• https://github.com/AbsaOSS/pramen/pull/69 Expose source and sink configs for options defined for all kinds of sources/sinks. by @yruslan in https://github.com/AbsaOSS/pramen/pull/69
• #40 Add customization the namespace for keeping transformations by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/64
• #68 Add minimum records for ingestion by @yruslan in https://github.com/AbsaOSS/pramen/pull/70
• #72 Allow creation of empty CSV files for the local CSV sink. by @yruslan in https://github.com/AbsaOSS/pramen/pull/74
• #73 Implement local spark source by @yruslan in https://github.com/AbsaOSS/pramen/pull/79
• #75 implement DCE launcher from Pramen by @yruslan in https://github.com/AbsaOSS/pramen/pull/80
• #82 Add defaults to commonly used configuration options. by @yruslan in https://github.com/AbsaOSS/pramen/pull/83

Bugfixes

• #45 java gateway error by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/48
• #55 Fix linter errors in CI (original fix by @zhukovgreen) by @yruslan in https://github.com/AbsaOSS/pramen/pull/56
• #47 Fix bug: Shell completion unit tests fail on Windows by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/57

Code and documentation improvements

• Use idiomatic way to skip tests on a condition by @zhukovgreen in https://github.com/AbsaOSS/pramen/pull/60
• Correct spelling and grammar by @dwfchu in https://github.com/AbsaOSS/pramen/pull/77
• #66 Made that list-configuration-options command was independent 2 by @ValeriiKhalimendik in https://github.com/AbsaOSS/pramen/pull/71

New Contributors

• @ValeriiKhalimendik made their first contribution in https://github.com/AbsaOSS/pramen/pull/48

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.0.1...v1.1.0

What's Changed

• Fix spelling and very minor grammar by @dwfchu in https://github.com/AbsaOSS/pramen/pull/26
• Add an explanation regarding run dates and info dates to README. by @yruslan in https://github.com/AbsaOSS/pramen/pull/27
• Add '--verbose' command line option to generate more detailed logs. by @yruslan in https://github.com/AbsaOSS/pramen/pull/31
• Remove unnecessary flags and update docs by @yruslan in https://github.com/AbsaOSS/pramen/pull/32
• Fix the warning log message when a sourcing table needs update. by @yruslan in https://github.com/AbsaOSS/pramen/pull/33

New Contributors

• @dwfchu made their first contribution in https://github.com/AbsaOSS/pramen/pull/26

Full Changelog: https://github.com/AbsaOSS/pramen/compare/v1.0.0...v1.0.1

What's Changed

• feature/uppercase col on read by @zhukovgreen in https://github.com/AbsaOSS/pramen/pull/11
• Add assembly for fat jars by @yruslan in https://github.com/AbsaOSS/pramen/pull/12
• Add license scan report and status by @fossabot in https://github.com/AbsaOSS/pramen/pull/13
• Add FOSSA badge. by @yruslan in https://github.com/AbsaOSS/pramen/pull/14
• Improve logging and notification message for missing dependencies. by @yruslan in https://github.com/AbsaOSS/pramen/pull/15
• Add IdentityTranformer into included transformations by @zhukovgreen in https://github.com/AbsaOSS/pramen/pull/16
• Add spark source for ingestion jobs by @yruslan in https://github.com/AbsaOSS/pramen/pull/17
• Fix a specific date in a generic transformation identity transformer. by @yruslan in https://github.com/AbsaOSS/pramen/pull/18
• Apply special characters replacement for all sources, not just JDBC. by @yruslan in https://github.com/AbsaOSS/pramen/pull/19
• Add version info in JAR generated by sbt assembly. by @yruslan in https://github.com/AbsaOSS/pramen/pull/21
• Fix Python publish in CI by @yruslan in https://github.com/AbsaOSS/pramen/pull/20
• Fix handling counts for ranged source tables. by @yruslan in https://github.com/AbsaOSS/pramen/pull/22
• Update the documentation about building the project and CSV source. by @yruslan in https://github.com/AbsaOSS/pramen/pull/23
• Cross compile across Scala and Spark versions by @yruslan in https://github.com/AbsaOSS/pramen/pull/24

Full Changelog: https://github.com/AbsaOSS/pramen/compare/pramen-py...v1.0.0

What's Changed

• feature/upper case column names on read by @zhukovgreen in https://github.com/AbsaOSS/pramen/pull/11