tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV

APACHE-2.0 License

Downloads
233
Stars
883
Committers
47
View Code on GitHub View on X
Ecosystems: Apache Spark

Bot releases are hidden (Show)

tispark - v3.1.6

Published by shiyuhang0 3 months ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.1.5...v3.1.6

tispark - v3.2.3 Latest Release

Published by shiyuhang0 7 months ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.2.2...v3.2.3

tispark - v3.1.5

Published by shiyuhang0 about 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.1.4...v3.1.5

tispark - v3.1.4

Published by shiyuhang0 about 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.1.3...v3.1.4

tispark - v3.2.2

Published by shiyuhang0 over 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.2.1...v3.2.2

tispark - v3.2.1

Published by shiyuhang0 over 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.2.0...v3.2.1

tispark - v3.1.3

Published by shiyuhang0 over 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.1.2...v3.1.3

tispark - v3.0.3

Published by shiyuhang0 over 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.0.2...v3.0.3

tispark - 3.2.1-rc1

Published by shiyuhang0 over 1 year ago

What's Changed

Full Changelog: https://github.com/pingcap/tispark/compare/v3.2.0...v3.2.1-RC1

tispark - TiSpark v3.2.0

Published by shiyuhang0 almost 2 years ago

New Features

  • Normalize the Java client in TiSpark and use the official client-java #2491
  • Compatible with TiDB v6.5.0 #2598
  • Support writing into the table with auto random primary key #2545
  • Support follower read #2546
  • Support writing into partition table with new collations #2565
  • Support partition pruning with to_days function when read from TiKV/TiFlash #2593
  • Support reading TiFlash load balancing with Round-Robin strategy client-java #662

Fixes

  • Bump Spark version from 3.0.2 to 3.0.3, from 3.1.1 to 3.1.3, from 3.2.1 to 3.2.3, from 3.3.0 to 3.3.1 #2544 #2607
  • Fix break change in partition read. For example, to_days is not supported #2552
  • Fix exception will be thrown when we insert into a table partitioned by year and the first col of the table is not date type #2554
  • Fix the bug that cluster index can not be used if the clustered index is not the Integer type #2560
  • Fix the bug that CBO may not choose min cost between TiKV table scan, TiKV index scan, and TiFlash scan #2563
  • Fix the bug that statistics are not collected which may affect the choice of the plan #2578
  • Fix the bug that partition pruning fails with the uppercase column #2593

Documents

  • Optimize user guide and dev guide #2533
tispark - TiSpark 3.1.2

Published by shiyuhang0 almost 2 years ago

New Features

  • Support write into partition table with new collations #2570
  • Support read TiFlash load balancing with Round-Robin strategy #2576
  • Support partition pruning with to_days function when read from TiKV/TiFlash #2594 #2600

Fixes

  • Fix CBO to let TiSpark choose the min cost between TiKV table scan, TiKV index scan and TiFlash scan correctly #2568
  • Fix the region not find error when reading from TiKV #2575
  • Fix the bug that statistics is not collected which may affect the choice of plans #2589
  • Compatible with TiDB v6.5.0 #2602
tispark - TiSpark 2.4.4

Published by shiyuhang0 about 2 years ago

Compatibility Changes

  • We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460.

Fixes

  • Fix the bug that single column condition is in the incorrect if branch #2393.
  • Fix region may be missed with too many tables #2442.
  • Fix the bug that count can not be pushed down #2468 #2483
tispark - TiSpark 3.1.1

Published by shiyuhang0 about 2 years ago

Fixes

  • Fix fail to read from some partition table #2553
  • Bump spark version to avoid vulnerabilities #2555
  • Fix error will be thrown when date col is not the first col in hash partition table #2556
tispark - TiSpark 3.1.0

Published by xuanyu66 about 2 years ago

Compatibility Changes

  • We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2457

New Features

  • Add authorization check for datasource api #2366
  • Make TiSpark's Explain clearer and easier to read #2439
  • Support host mapping in TiSpark #2436
  • Support bypass-TiDB write into partition table #2451
  • Support insert sql #2471
  • Support Spark 3.3 #2492
  • Only do auth check for tables in TiDB #2489
  • Support new Collation #2524

Fixes

  • Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw table not found exceptions #2433
  • Fix the bug that count/avg can not push down #2445
  • Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict. And the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2455
  • Fix the bug that exception would through when the size of pdAddresse is > 1 #2473
  • Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2476
  • Upgraded Spark 3.1 support version from 3.0.2 to 3.0.3, upgraded Spark 3.1 support version from 3.1.1 to 3.1.3, upgraded Spark 3.2 support version from 3.2.1 to 3.2.2 #2486
  • Fix the bug that exception will be throw when date col is not the first col ref #2538
tispark - TiSpark 3.0.2

Published by shiyuhang0 about 2 years ago

Compatibility Changes

  • We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460

Fixes

  • Fix the bug that single column condition is in the incorrect if branch #2395
  • Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2440
  • Fix the bug that count/avg can not push down #2470
  • Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2515
  • Fix exception would through when the size of pdAddresse is > 1 #2478
  • Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2485
  • Upgraded Spark3.1 support version from 3.0.2 to 3.0.3, Upgraded Spark3.1 support version from 3.1.1 to 3.1.3, Upgraded Spark3.2 support version from 3.2.1 to 3.2.2 #2488
  • Only do auth check for tables in TiDB #2500
  • Changed profile #2518
tispark - TiSpark 2.5.2

Published by shiyuhang0 about 2 years ago

Compatibility Changes

  • We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2461

Fixes

  • Fix the bug that single column condition is incorrect if branch #2394
  • Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2441
  • Fix the bug that count/avg can not push down #2469
  • Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2516
  • Fix exception would through when the size of pdAddresse is > 1 #2477
  • Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2484
  • Upgraded Spark3.1 support version from 3.0.2 to 3.0.3，Upgraded Spark3.1 support version from 3.1.1 to 3.1.3 #2487
  • Only do auth check for tables in TiDB #2502
  • Change spark profile #2517
tispark - TiSpark 3.0.1

Published by shiyuhang0 over 2 years ago

Fixes

  • Fix the bug that the single column condition is in the incorrect if branch #2395
  • Fix the bug that the spark-shell is stuck when exiting #2402
  • Fix the bug that the maven dependency of TiSpark 3.0.0 can not be imported #2401
  • Shutdown recycler when closing ChannelFactory to avoid resource leak #2405
tispark - TiSpark 3.0.0

Published by shiyuhang0 over 2 years ago

Compatibility Changes

  • TiSpark without catalog plugin is no more supported. You must configure catalog configs and use tidb_catalog now #2252
  • TiSpark's jar has a new naming rule like tispark-assembly-{$spark_version}_{$scala_version}-{$tispark_verison} #2370

New Feature

  • Support DELETE statement #2276
  • Support Spark 3.2 #2287
  • Support telemetry to collect information #2316
  • Support stale read to read historical versions of data #2322
  • Support TLS with reload capability #2306 #2349 #2365 #2377

Fixes

  • Fix the wrong result of _tidb_rowid when set spark.tispark.show_rowid=true #2270
  • Fix sum not push down bug #2314
  • Fix limit not push down bug #2329
  • Avoid NoSuchElementException when setting catalog #2220
  • Avoid ClassCastException when cluster index with type Timestamp and Date #2319
  • Improve retry logic in write so that it will not retry in some scenarios which needn't retry #2279
  • Delete unused configuration spark.tispark.statistics.auto_load #2300
  • Upgrade jackson-databind from 2.9.10.8 to 2.12.6.1 #2285
  • Upgrade guava from 26.0-android to 29.0-android #2340
  • Upgrade mysql-connector-java from 5.1.44 to 5.1.49 #2367

Documents

  • Update communication channels #2228
  • Add limitation: new collations are not supported #2238
tispark - TiSpark 2.5.1

Published by shiyuhang0 over 2 years ago

Fixes

  • Fix limit not push down bug #2335
  • Fix ClassCastException when cluster index with type Timestamp and Date #2323
  • Upgrade jackson-databind from 2.9.10.8 to 2.12.6.1 #2288
  • Fix the wrong result of _tidb_rowid #2278
  • Fix set catalog throw NoSuchElementException #2254

Documents

  • Add limitation: TLS is not supported #2281
  • Add limitation: new collations are not supported #2251
  • Update communication channels #2244
tispark - TiSpark 2.5.0

Published by zhangyangyu over 2 years ago

New feature

  • Support Spark 3.1.x and 3.0.x version.
  • Support Spark SQL's authentication and authorization through TiDB #2185.

Fixes

  • Fix duplicate range in RowIDAllocator #2156.
  • upgrade log4j to 2.17.1 #2197.
  • upgrade jackson-databind to 2.9.10.8 #2161.

Documents

  • Fix in user docs forget to use options while using BatchWrite #2169

Known issue

  • Spark SQL's authentication and authorization through TiDB will affect other datasource such as hive. See details in this issue #2224.
  • Batch write needs more tests and may be unstable. See details in this issue #2222.
Package Rankings
Top 9.62% on Repo1.maven.org
Top 8.17% on Proxy.golang.org
Top 13.57% on Pypi.org
Badges
Extracted from project README
Maven Central License
Related Projects
spark-cassandra-connector

DataStax Connector for Apache Spark to Apache Cassandra

27 Jun 2014 1,931
kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to ha...

01 Jun 2020 443
sparksql-for-hbase

Learn how to use Spark SQL and HSpark connector package to create / query data tables that reside...

31 Aug 2017 69
sparkle

Haskell on Apache Spark.

09 Nov 2015 444
spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

22 Apr 2019 2,020
iotdb

Apache IoTDB

24 Nov 2018 5,552
twut

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

29 Nov 2019 9
spark

Apache Spark - A unified analytics engine for large-scale data processing

25 Feb 2014 38,255
setl

A simple Spark-powered ETL framework that just works 🍺

20 Dec 2019 177
spark

Performance Observability for Apache Spark

28 Sep 2023 171
spark-authorizer

A Spark SQL extension which provides SQL Standard Authorization for Apache Spark | This repo is c...

08 Dec 2017 171
SynapseML

Simple and Distributed Machine Learning

05 Jun 2017 4,986
spark-state-tools

Spark Structured Streaming State Tools

30 Mar 2019 34
azure-kusto-spark

Apache Spark Connector for Azure Kusto

09 Jan 2019 77
spark-utils

Basic framework utilities to quickly start writing production ready Apache Spark applications

14 Apr 2018 36