tispark

New feature

• scala 2.11 and 2.12 are both supported

Fixes

• Fix duplicate range in RowIDAllocator #2208.
• upgrade log4j to 2.17.1 #2209.
• upgrade jackson-databind to 2.9.10.8 #2209.

Documents

• Fix in user docs forget to use options while using BatchWrite #2171.

Known issue

• Batch write needs more tests and may be unstable. See details in this issue #2222.

Since TiSpark v2.4.2, Only Scala v2.12 is supported.

What's Changed

• BatchWrite: support clustered index (#2060) by @marsishandsome in https://github.com/pingcap/tispark/pull/2079
• Fix region manager race (#1987) by @ti-srebot in https://github.com/pingcap/tispark/pull/2092
• Fix ResolveLock not called in Scan API (#2089) by @ti-srebot in https://github.com/pingcap/tispark/pull/2091
• fix region split: region split key should be a valid key by @marsishandsome in https://github.com/pingcap/tispark/pull/2101
• Optimize region cache miss (#2106) by @ti-srebot in https://github.com/pingcap/tispark/pull/2107
• fix catalog test (#2110) by @ti-srebot in https://github.com/pingcap/tispark/pull/2113
• Fix channel shut-downed due to ping failure (#2120) by @ti-srebot in https://github.com/pingcap/tispark/pull/2122
• Close TTLManager before creating a new one (#2114) by @ti-srebot in https://github.com/pingcap/tispark/pull/2118
• Close PDClient without closing channel factory (#2123) by @ti-srebot in https://github.com/pingcap/tispark/pull/2125
• Fix DateTime shift before the year 1582 (#2104) by @birdstorm in https://github.com/pingcap/tispark/pull/2116
• Fix enum with trailing spaces (#2127) by @ti-srebot in https://github.com/pingcap/tispark/pull/2134
• Fix not leader error due to region not invalidated (#2135) by @ti-srebot in https://github.com/pingcap/tispark/pull/2138
• Fix BatchGet StackOverFlow Error (#2126) by @ti-srebot in https://github.com/pingcap/tispark/pull/2140
• Update to support scala-2.12 by @birdstorm in https://github.com/pingcap/tispark/pull/2149
• Update spark version info for v2.4 by @birdstorm in https://github.com/pingcap/tispark/pull/2155

Full Changelog: https://github.com/pingcap/tispark/compare/v2.4.1...v2.4.2

Fixes

• Change PARTITION_PER_SPLIT default value from 1 to 10 #2029
• Fix array index out of bound when encoding #2041
• fix batch write missing unpersist #2046
• Fix bug: write decimal(16, 4) using batch write and read using tiflash #2055
• Fix mistake setting current_ts to 0 #2067
• Filter views in catalog #2071
• BatchWrite: add argument deduplicate #2073

Fixes

• Fix bug: write decimal(16, 4) using batch write and read using tiflash #2055
• Fix mistake setting current_ts to 0 #2067
• Filter views in catalog #2071
• BatchWrite: add argument deduplicate #2073

New Features

• Support TiDB-5.0

Fixes

• Fix GregorianCalendar #1952
• Fix tiflash timestamp pushdown bug #1966
• Do not select tidb_row_format_version if tidb is 3.x #1993
• Update table statistics after batch write #2009
• Fix oom cause missing data bug #2010

Fixes

• Disable index scan on partition table #1923
• Fix jackson-databind #1910
• Fix BatchGet retry #1907

Fixes

• Add retry in RangeSplitter #1886
• Fix TiDAGRequest concurrent visit #1893
• Fix partition table with partition column name date #1892

• Fix the resolved txn status cache for pessimistic txn #1868
• Fix overflow when convert big MyDecimal to BigDecimal #1869
• Add retry in RowIDAllocator #1856
• Fix select count where values contains null #1841
• Do not retry for some key error #1799
• Disable year type index scan #1790

New Features

• Support auto_random column #1838
• Support log desensitization #1805
• Support insert null on auto increment column #1786

Fixes

• Fix maven-shade-plugin relocation cannot handle folder name with dot #1737
• BatchWrite: add parameter maxRegionSplitNum #1729
• Fix string insert overflow check #1727

Fixes

• Revert "fix TiSession Close bug" #1713
• Fix typo in DateTime calculation #1710

Fixes

• Fix negative column id when there are more than 128 columns #1673
• Fix incorrect date type value when timezone is set #1704
• BatchWrite: fix auto increment column not primary key #1694
• Fix TiSession Close bug #1677
• Optimize region split number in small data size #1681

Fixes

• BatchWrite: enable user provided auto increment value in update mode #1658
• BatchWrite: fix initial ttl expired #1660
• BatchWrite: fix commit ts expired #1661

Fixes

• Fix covering index error when alter-primary-key is enabled #1640
• Fix alter primary key for write #1644

Fixes

• Fix scala match error when predicate includes boolean type conversion #1620
• Fix encoder v2 with type conversion and set default batch write version to v2 #1623
• Filter sequence table from catalog #1627
• Fix batch write failed because of MetaCodec.hashGet failed #1636

Fixes

• Filter Tombstone store #1582
• Increase gRPC response default limit size to 2G #1597
• Update txnPrewriteBatchSize & txnCommitBatchSize default value to 16K #1599
• Update writeThreadPerTask default value to 2 #1600
• Fix multiple database scan test bug #1603
• Update grpc version to 1.29.0 #1609
• BatchWrite: add parameter maxWriteTaskNumber #1616

Fixes

• Optimize scatter region logic #1582
• Set regionSplitUsingSize default value to true #1584
• User provided auto increment value is not supported #1586
• Fix partition table on tiflash #1588
• Use ShardRowIDBits to generate row id #1585

Fixes

• Fix batch write bug and optimize batch write execution #1562
• Fix columnar batch execution #1566
• Fix covering index generates incorrect plan when first column is not included in index #1573
• Fix pytispark #1557

Fixes

• Ignore get version sql exception #1516