Smart Automation Tool for building modern Data Lakes and Data Pipelines
GPL-3.0 License
Bot releases are hidden (Show)
transform(session: SparkSession, dfDataObject1: DataFrame, dfDataObject2: DataFrame, isExec: Boolean, nbOfDts: Option[Int]): DataFrame
Published by zzeekk 12 months ago
Hotfix Release with the following changes:
Published by pgruetter about 1 year ago
dbutils-api: Update from 0.0.5 to 0.0.6
Published by pgruetter over 1 year ago
Spark: Update from Spark 3.2 to 3.3
Delta Lake: Update from 2.0 to 2.2
Published by ddeuber over 1 year ago
saveMode=Overwrite
for JdbcTableDataObject
, allow writing to the database table even if the column order in the dataframe is different (#633)JdbcTableConnection
in order to configure the commit behaviour in JDBC connections (#633)Note: this release is created as Hotfix Release on top of version 2.4.1, as develop-spark3 branch is already on 2.5.0-SNAPSHOT.
Published by zzeekk over 1 year ago
Note: this release is created as Hotfix Release on top of version 2.4.0, as develop-spark3 branch is already on 2.5.0-SNAPSHOT.
Published by pgruetter almost 2 years ago
#518 Schema Viewer shows wrong information
#580 Can't use same ExcelFileDataObject for write and read
#600 Schema viewer does not indicate whether a field is required
#601 Loading Schema from file should be done lazy
Leading underscores are preserved when normalizing column names
ExecutionMode and executionCondition are only applied in exec phase
#591 Column encryption
#610 Support DataObjectStateIncrementalMode for KafkaTopicDataObject
Bump commons-net from 3.1 to 3.9.0
Published by pgruetter almost 2 years ago
Published by pgruetter almost 2 years ago
This is mainly a bugfix release, see:
#583
#584
#578
#579
One new Feature:
#575
Published by zzeekk about 2 years ago
Full Changelog: https://github.com/smart-data-lake/smart-data-lake/compare/2.2.1...2.3.0
Published by zzeekk over 2 years ago
Published by pgruetter over 2 years ago
Update to Spark 3.2 (#406)
Update delta lake to version 1.1 (#406)
Implement CustomSnowparkAction (rudimentary Snowpark support, #376)
Implement script support and CustomScriptAction (#422)
Implement AirbyteDataObject (#365)
Implement basic ScalaNotebookDfTransformer (#401)
Implement SDL json schema creator (#440)
Add Atlas metadata exporter implementation
Extend StateListener.notifyState with parameter indicating change Action
Adapted StateChangeLogger to log only for the action for the notification was emitted
Refactor Actions SubFeed handling
Refactor integrating SparkSession into ActionPipelineContext and usage of implicit parameters
Add SASL Authentication for Kafka
Avoid loosing full error response text from webservice calls
Improve build stability by using linesIterator, otherwise on some environments the java:String.lines has precedence over scala:StringLike:lines, which causes compile problems.
Use json4s instead of hocon/configs to write json-state-files
Allow using custom class loader in order to find classes defined or loaded from notebooks (polynote) when parsing configuration
Extend ScalaJWebserviceClient so it can be re-used in getting-started
Force SaveMode.Overwrite for DeduplicateAction and HistorizeAction if mergeModeEnable=false
Make runtime info public (#454)
Published by zzeekk almost 3 years ago
Improved Delta Lake support
Data Objects extensions
Streaming improvements
New SDLSaveMode.merge to do upsert statemetns
New sdl-azure module
Small bugfixes & improvements
Cleanup
Published by zzeekk almost 3 years ago
Improved Delta Lake support
Data Objects extensions
Streaming improvements
New SDLSaveMode.merge to do upsert statemetns
New sdl-azure module
Small bugfixes & improvements
Cleanup
Published by pgruetter about 3 years ago
New features
Bugfixes
Published by pgruetter about 3 years ago
New features
Bugfixes
Published by pgruetter over 3 years ago
New features
Minor improvements
Bugfixes
Published by pgruetter over 3 years ago
New features
Minor improvements
Bugfixes
Published by pgruetter over 3 years ago
New Features
Minor improvements
Bugfixes
Published by pgruetter over 3 years ago
New Features
Minor improvements
Bugfixes