spark-redis

What's Changed

• Fix python docs (2.4) by @fe2s in https://github.com/RedisLabs/spark-redis/pull/184
• issue #250: fixed streaming doc page (added ssc.start()) by @fe2s in https://github.com/RedisLabs/spark-redis/pull/251
• issue-257 : mostly typo / manually tested in python by @tgrall in https://github.com/RedisLabs/spark-redis/pull/259
• Issue 265: Remove filterKeysByType step when getting data from redis by @mayankasthana in https://github.com/RedisLabs/spark-redis/pull/266
• Issue 49: added toRedisHASHes function (#49) by @phyok in https://github.com/RedisLabs/spark-redis/pull/269
• fix arg exception msg in redis source relation by @bsipin20 in https://github.com/RedisLabs/spark-redis/pull/273
• issue #328: add user/password authentication (Redis 6 ACL feature) by @fe2s in https://github.com/RedisLabs/spark-redis/pull/333

New Contributors

• @tgrall made their first contribution in https://github.com/RedisLabs/spark-redis/pull/259
• @mayankasthana made their first contribution in https://github.com/RedisLabs/spark-redis/pull/266
• @phyok made their first contribution in https://github.com/RedisLabs/spark-redis/pull/269
• @bsipin20 made their first contribution in https://github.com/RedisLabs/spark-redis/pull/273

Full Changelog: https://github.com/RedisLabs/spark-redis/compare/v2.6.0...v2.7.0

• Added new API: toRedisHASHes function
• Performance optimization (removed filterKeysByType step when getting data from Redis)
• Minor fixes of documentation and exception messages

• Add ssl params to all invocations of RedisEndpoint ctor. (#235)

• #195: support scala 2.12 (#223)
• #217: added toRedisLISTs() function (#221)

• Dataframe created from a specific key (not a pattern) and specified key doesn't exist in Redis (#194)
• A prototype of toRedisByteLIST() function (#205)

Version upgrades:

• Support for Spark 2.4.0
• Upgrade jedis to 3.1.0-m1

Backward incompatible changes:

• Issue #165 - RDD and Streaming implicit were split to two different packages to avoid extra spark-streaming dependency when only RDD functions are used. Streaming functions are now enabled through "import com.redislabs.provider.redis.streaming._"

Fixes:

• Issue #134 - Fixes an issue when SBT cannot resolve jedis dependency ()

Enhancements:

• Enables Structured Streaming output to Redis using foreachBatch, docs updated
• Added dev environment docs

Upgrading notice from 2.3.1 to 2.4.0:

• If Spark Streaming functionality (createRedisXStream, createRedisStream or createRedisStreamWithoutListname) is used, add additional import statement "import com.redislabs.provider.redis.streaming._" to prevent compilation errors

• Add support for DataFrames
• Add support for Structured Streaming

• Add support for Structured stream

• dataframes implementation
• dataframes docs (scala and python)
• issue #99 - pipeline batches (affects RDDs and DataFrames)
• issue #100 - SCAN count option (affects RDDs and DataFrames)

Support Apache Spark 1.4.1

Support Apache Spark 2.3.0

This release adds support for SparkSQL

Provides Redis access functionality to basic data types from Spark.