Merged PRs

dolt

• 4675: deeper comparison for foregin key indexes
  If we have committed tables before this change, where the tables have a foreign key that references an index that could've been replace with an existing primary key, apply those changes, and we drop and recreate the parent/child tables in exactly the same way, we stage the parent table, but not the child table, which causes our foreign keys to enter a bad state.
  repro for clarity:

  // SWITCH TO DOLT v0.50.8
  rm -rf .dolt
  dolt init
  dolt sql -q "create table payment(payment_id int primary key);"
  dolt sql -q "create table subscription(id int primary key, payment_id int);"
  dolt sql -q "alter table subscription add constraint subscription_pm_fk foreign key (payment_id) references payment(payment_id);"
  dolt add .
  dolt commit -m "commit from the past"
  // SWITCH TO MAIN
  dolt sql -q "drop table subscription";
  dolt sql -q "drop table payment";
  dolt sql -q "create table payment(payment_id int primary key);"
  dolt sql -q "create table subscription(id int primary key, payment_id int);" #A
  dolt sql -q "alter table subscription add constraint subscription_pm_fk foreign key (payment_id) references payment(payment_id);" #B
  dolt add .
  dolt commit -m "commit from the future"
  

  You can see this by running dolt status after line (A) and then line (B).
  The fix was to have our DeepEquals comparison for foreign key definitions also check if the referencing table index names have changed, and hope that that doesn't break anything else.
• 4672: liuliu/handle-nil-interface-value
  fixes https://github.com/dolthub/dolthub-issues/issues/352
• 4667: go/libraries/doltcore/sqle/cluster: interceptors.go: Add some initial support for server interceptor allowing non-cluster clients to access the remotesapi.
• 4629: Added database collation support
  This allows setting the database collation, such that all newly created tables within a database (that do not explicitly set their collation) will inherit the database collation.
  Builds on
  • https://github.com/dolthub/vitess/pull/199
  • https://github.com/dolthub/go-mysql-server/pull/1351
    Fixes
  • https://github.com/dolthub/dolt/issues/4170

go-mysql-server

• 1361: Fixed collation check on foreign key columns
• 1354: Bug fix for pushdownSort handling missing cols qualified with a table name
  The Dolt bump for my GMS change to fix an alias issue in sort node pushdown triggered an error with matching missing column names now that we can include qualified column names.
  This PR adds a repro for that case to GMS and fixes the issue by ensuring we create a UnresolvedQualifiedColumn when the missing column is qualified with a table name. I've run Dolt tests locally and confirmed there shouldn't be any other test failures in the next bump.
• 1351: Add support for database collations
  This allows setting the database collation, such that all newly created tables within a database (that do not explicitly set their collation) will inherit the database collation.
  Builds on https://github.com/dolthub/vitess/pull/199

vitess

• 199: Added ALTER DATABASE parsing
  Adds support for parsing ALTER DATABASE queries.
• 196: allow dual only for select from statements if used without back-ticks

Closed Issues

• 4674: Add support for DROP TABLE CASCADE
• 4376: DATE_ADD truncates time
• 4170: Create database with charset and collation not supported
• 4662: Very large storage footprint
• 1897: Addition of Docker image
• 4430: Feature request: add a progress bar to dolt sql < file.sql that shows the progress of a SQL load

Latency

Merged PRs

dolt

• 4661: Bats tests for sql-client fixes
• 4659: go/libraries/doltcore/sqle/cluster: Add the ability to configure SAN assertions on URIs and DNS names in the server certificate when configuring TLS.
• 4658: Remove server_query in favor of dolt sql-client in bats
  This PR depends on #4640.
• 4656: Improve dump-docs error and add bats, fix diff docs
• 4647: Handle removed tables in //.../doltcore/migrate
  Fixes #4602
  Validated changes against repos that previously failed with duplicate tag errors:
  • https://dolthub.awsdev.ld-corp.com/repositories/dolthub/pdap-datasets/data/master
  • https://www.dolthub.com/repositories/dolthub/us-president-precinct-results
• 4646: Support two and three dot syntax in dolt_diff_summary table function
  Two dot: dolt_diff_summary('main..feature', 'table') (equivalent to dolt_diff_summary('main', 'feature', 'table'))
  Three dot: dolt_diff_summary('main...feature', 'table')
• 4644: Support two and three dot syntax in dolt_diff table function
  Two dot: dolt_diff('main..feature', 'table') (equivalent to dolt_diff('main', 'feature', 'table'))
  Three dot: dolt_diff('main...feature', 'table')
• 4641: integration-tests/bats/sql-server.bats: Move the temporary table session specific test to go-sql-server-driver.
• 4640: SQL client --query fixes
  Fixes the following issues regarding dolt sql-client --query
  • https://github.com/dolthub/dolt/issues/4623
  • https://github.com/dolthub/dolt/issues/4628
  • https://github.com/dolthub/dolt/issues/4635
• 4639: Add sleeps to a couple bats tests that were starting a SQL Server
  Give the server a chance to start up. Was failing weirdly on my Mac without this.
• 4638: go/libraries/doltcore/sqle/cluster: Add support for configured tls_{cert,key,ca} on the cluster.remotesapi.
• 4637: Add two and three dot diff syntax to CLI
  Two dot:
  dolt diff A..B
dolt diff A..
  Three dot:
  dolt diff A...B
dolt diff A...
dolt diff --merge-base A B
• 4634: go/store/nbs: Removed chunkReader.extract(), converted public methods to private methods for…
  … chunkReader, chunkSource and tableSet
• 4626: more permissive constraints and types when importing
  There was a change added that samples the rows when doing large imports, but this can incorrectly give columns a NOT NULL constraint. We can either not sample tables or just not apply the NOT NULL constraintl. I chose to not apply the NOT NULL constraint. Additionally, the sampling change makes it possible to incorrectly make certain columns UNISIGNED INT, when they contain negative numbers. I changed it so infer will always treat columns as signed, so integers that are too large/small for signed int will become strings.
  fix for: https://github.com/dolthub/dolt/issues/4620
• 4622: Partially remove server_query in favor of dolt sql-client in bats tests
  server_query requires python and has a non-descriptive interface. dolt sql-client allows for a native dolt interface to a running dolt sql-server that is easier to read and maps better to the rest of the bats tests.
• 4583: avoid hidden secondary indexes when creating foreign keys that prefix of pk
  TODO: maybe use empty string instead of PRIMARY for name of index used by foreign key.
  When creating foreign keys, we typically create a new secondary index so violations and cascades are handled quickly.
  However, there is no need to create a secondary index when the foreign key references a prefix to an existing primary key.
  This PR makes it so that the Primary Key shows up as an Index with the name "PRIMARY".

go-mysql-server

• 1354: Bug fix for pushdownSort handling missing cols qualified with a table name
  The Dolt bump for my GMS change to fix an alias issue in sort node pushdown triggered an error with matching missing column names now that we can include qualified column names.
  This PR adds a repro for that case to GMS and fixes the issue by ensuring we create a UnresolvedQualifiedColumn when the missing column is qualified with a table name. I've run Dolt tests locally and confirmed there shouldn't be any other test failures in the next bump.
• 1353: Fix panic for show keys from information_schema.columns
  Does this by removing the special handling of information_schema.columns as a separate node type, treats it just like any other ResolvedTable.
  In the process, effectively rewrote how we handle column default values by 1) moving most logic to happen in the OnceBefore batch, rather than default rules, and 2) splitting it up into multiple passes that each have a single purpose. I found in the process of 1) that the previous rules had a lot of side effects and unintended ordering constraints, so introduced new rules and tweaked others to eliminate those.
• 1352: Update sort node to use alias reference when it will resolve missing columns
  Fixes: https://github.com/dolthub/dolt/issues/3016
  Other changes:
  • Refactors the existing OrderBy/GroupBy tests into a ScriptTest.
  • Introduces a new interface, sql.Projector, that unites GroupBy, Window, and Project if a caller just needs to get the projected expressions.
• 1350: Subquery row iter fix field indexes
  Recent changes to subquery scope visibility use the scope to communicate column definition availability; i.e., we do not pass the scope into subqueries we have determined to not depend on the outer scope, marking the same scope as cacheable. This analysis change needs a corresponding runtime change to indicate whether the scope row is expected at execution time.
• 1349: Skip process tracking when prepreparing queries
  When we prepare a statement, the QueryProcess node we create is bound to the current context's PID. This is not the same PID as the context that will execute a statement created from that template, which results in ProcessList metadata not being properly cleaned up after a query has finished processing.
  Fixes: https://github.com/dolthub/dolt/issues/4601
  I couldn't find a great way to test this in the GMS package, but I'm working on a test in dolt that I'll link to shortly.
• 1290: New join planning
  Join planning has many high level components:
  • table order
  • tree shape (left deep, right deep, bushy, etc)
  • indexes for lookup joins
  • filter arrangement
  • choosing between logically equivalent plans (costing)
    This PR does not fix all of these components, but it does provides a structure for gracefully increasing the complexity of each.
    The new memo data structure is a memory efficient IR for SQL queries that 1) groups equivalent plans by their output schemas, and 2) generalizes child relationships. For example, here is an example memo for a join query:

  select * from ab
  inner join uv on a = u
  full join pq on a = p
  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (leftJoin 4 3)
  

  "Relational expression" each have their own expression group, within which several physical/concrete implementation can reside. An expression group is defined in terms of expression group relationships, not physical implementations, because the output from any relational expression within a group will be the same. Groups are usually keyed by a hash of the operator type and the child group keys, or in the case of scalar expression by the literal parameter values (this last point will be useful for prepared statements).
  All valid reorderings of the join tree are added to the memo by joinOrderBuilder.
  Applying indexes is a separate step, where we consider an indexed version for every join implementation where the outer (right) table is an indexable data source. Adding indexed plans to the query above yields:

  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (indexedJoin 1 2) (indexedJoin 2 1) (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (indexedJoin 4 3) (leftJoin 4 3)
  

  Hashed joins are added in the same manner:

  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (hashJoin 1 2) (hashJoin 2 1) (indexedJoin 1 2) (indexedJoin 2 1) (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (hashJoin 4 3) (indexedJoin 4 3) (leftJoin 4 3)
  

  This memo is missing several components that would make it more useful, including other node and expression types. For example, filter pushdown would be much easier in the memo. You could imagine extending the memo above with sql.Expression tree memo groups (scalar relations). Canonical (simplified) expression groups only requiring one allocation, and the scalar expression group caches properties like table and column dependencies similar to relProp column and table dependencies. We would need to build a memo prior to join planning to use filters for join planning, and run other transformation rules on the memo. I'd expect the memo to absorb rules over time, which will simplify transform logic and improve analyzer memory consumption. I added some starters to make this easier in the future, like codegen'd memo expressions, and a move towards visitor interfaces for IR transformations.
  A fully exhausted memo tree is converted back into a physical plan first through a costing process. Costing builds the fastest implementation for child group expressions bottom-up, using only the best child implementations for costing parent expressions and eventually the root node.
  The coster is designed to use histogram statistics to pick the fastest join plan. We only provide a subset of index metadata and table sizes at this moment. Additionally, costing should apply to generic SQL nodes and expressions, not just join trees and join leaves.
  Other notes:
  • we miss join plans that result from transitive predicates (ex: ab.a = uv.u + uv.u = xy.x => ab.a = xy.x).
  • we miss join plan optimizations resulting from table functional dependencies (ex: eliminate redundant filters if we already join on a a primary key), and null-allowing joins (ex: IS NULL).
  • we could more aggressively apply filters to join trees.
  • converting from the memo to an execution plan is fraught for the same reasons it was before this PR, but tree building and fixup is condensed in exec_builder, which i'd expect to expand to include more exprGroup types in the future.
  • there is a hack for fixing up field indexes in join trees not considered for reordering.
  • right joins are converted to left joins after expandStars in transposeRightJoins.
  • non-left deep trees necessitate passing parent rows down the right side of join trees, which were previously only table sources (left deep).
  • joinOrder hinting is rewritten to more easily fit within exprGroup costing.
  • I started to use bitmaps to track expression group column dependencies, which should also be used for filter applicability.
  • indexedJoinIter is duplicate and should be deprecated, but i had a hard time quickly removing it.
  • every join node is condensed into *plan.JoinNode and differentiated based on its join type, which lets me bucket different join types more easily (refer to JoinType type methods).
  • applyHashLookups should be deprecated, given that we will preemptively build hash joins. If hash joins do not select the lookup, we should either not cache the subquery or improve the coster to make better decisions.
  • there are probably bugs where we apply hash joins to non-cacheable subquery scopes for derived tables. Need more testing.
    Additional refs:
  • https://www.researchgate.net/publication/262216932_On_the_correct_and_complete_enumeration_of_the_core_search_space
  • https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/xform/join_order_builder.go.
  • https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/xu-columbia-thesis1998.pdf

Closed Issues

• 4652: upgrade dolt from 0.26.6 to 0.50.2， sql error
• 1345: SET @@mydb_head = HASHOF() panics when session variables is spelled wrong
• 4602: dolt migrate: Fails to migrate when a table has been renamed
• 2082: Don't create duplicate primary key index for referenced tables in foreign keys
• 4625: [dolt migrate]: Incorrect datetime value
• 3016: Strange error with Large Join Query
• 1897: Addition of Docker image
• 4620: import table returns error

Latency

Merged PRs

dolt

• 4632: update readmes for docker images
• 4631: go/doltcore/schema: Set primary key ordinals on schema migration
• 4630: [dolt migrate]: Validate ws migration, skip table validation if table did not change
• 4624: go/doltcore/migrate: Account for types.BoolKind in very old databases
• 4619: [dolt migrate]: don't prefetch leaf nodes when validating migrated data
  Bounds memory usage to size of table file indices and number of cpu cores
  Migrating https://www.dolthub.com/repositories/dolthub/museum-collections pre-pr with 8 cores and 64gb of ram takes 5,049 seconds.
  Post-pr it takes 6,017 seconds on the same machine.
  Only a (5049 - 6017) / 6017 = -16.09% performance hit.
  Previously memory usage was spiking to 42gb, now it remains under 2gb.
• 4618: go/libraries/doltcore/remotesrv: Add a symmetric key encryption layer to seal URL returned from the gRPC service an unseal them in the HTTP service.
  This will allow us to add authn/z to the gRPC layer.
• 4617: go/store/nbs: Faster prefix lookup for table file indices

  goos: darwin
  goarch: amd64
  pkg: github.com/dolthub/dolt/go/store/nbs
  cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  BenchmarkFindPrefix
  BenchmarkFindPrefix/benchmark_prefixIdx()
  BenchmarkFindPrefix/benchmark_prefixIdx()-12         	16069794	        71.68 ns/op
  BenchmarkFindPrefix/benchmark_findPrefix
  BenchmarkFindPrefix/benchmark_findPrefix-12          	22866360	        52.23 ns/op
  PASS
  

• 4613: Fixing variable name in getPORT helper function
  I noticed some errors with grep syntax in a failed BATS test. Looks like we just had an empty env var so there wasn't anything passed to grep, which caused errors and prevented us from fully checking if a port is in use.

Closed Issues

• 4601: sql-server: kill query spam with parameterized statements
• 4281: slow execution of complex query

Merged PRs

dolt

• 4609: use github actions for docker
  Use GitHub Actions to push to dolthub/dolt and dolthub/dolt-sql-server images.
• 4608: go/doltcore/migrate: Patch schemas to use utf8mb4_0900_bin to match e…
  …xisting index order
  fix for https://github.com/dolthub/dolt/issues/4603
• 4605: Add dolt sql server docker image
  Adds Docker Image for dolt sql-server. The server starts at 0.0.0.0:3306, which allow connections from outside the container through port-mapping. User can define host and port through yaml configuration. The data directory is default to /var/lib/dolt/ directory in the container. It can be mounted with directory on host system. It's not recommended to define data_dir in configuration, which will cause mounting host system directory fail.
• 4594: Make dolt log work with ^ ancestor spec
• 4585: Branch Control Pt. 9
  This PR:
  • Moves the branch controller into the context
  • Adds the remaining stored procedures
  • Handles declaring the super user via CLI arguments or YAML
  • Fixes bugs found through additional testing
  • Enables branch control globally
    Things still missing:
  • Bats tests (will mainly copy the engine tests, but makes use of saving/loading from disk)
  • Ways to interact with the binlog
  • Support for roles
    Role support is actually a bit more involved than originally anticipated. It involves coordination with the privilege tables, which were not built to be interacted with outside of their specific GMS context. I'm thinking it should get pushed back to a v2 implementation, unless it's high priority.
• 4516: join ordering GMS bump
  companion PR: https://github.com/dolthub/go-mysql-server/pull/1290

go-mysql-server

• 1343: allow adding new primary key to table with > 1 row iff it has auto_increment
  fix for: https://github.com/dolthub/dolt/issues/4581
  tests in dolt because memory.Table doesn't implement RewriteableTable
  https://github.com/dolthub/dolt/pull/4593
• 1310: Derived table outer scope visibility
  Add support for outer scope visibility for derived tables, as introduced in MySQL 8.0.14.
  References:
  • MySQL Blog Post: Outer references for derived tables
  • MySQL Worklog task and notes for derived table outer scope visibility
    Resolves: https://github.com/dolthub/dolt/issues/4534
    Dolt CI tests: https://github.com/dolthub/dolt/pull/4472
• 1290: New join planning
  Join planning has many high level components:
  • table order
  • tree shape (left deep, right deep, bushy, etc)
  • indexes for lookup joins
  • filter arrangement
  • choosing between logically equivalent plans (costing)
    This PR does not fix all of these components, but it does provides a structure for gracefully increasing the complexity of each.
    The new memo data structure is a memory efficient IR for SQL queries that 1) groups equivalent plans by their output schemas, and 2) generalizes child relationships. For example, here is an example memo for a join query:

  select * from ab
  inner join uv on a = u
  full join pq on a = p
  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (leftJoin 4 3)
  

  "Relational expression" each have their own expression group, within which several physical/concrete implementation can reside. An expression group is defined in terms of expression group relationships, not physical implementations, because the output from any relational expression within a group will be the same. Groups are usually keyed by a hash of the operator type and the child group keys, or in the case of scalar expression by the literal parameter values (this last point will be useful for prepared statements).
  All valid reorderings of the join tree are added to the memo by joinOrderBuilder.
  Applying indexes is a separate step, where we consider an indexed version for every join implementation where the outer (right) table is an indexable data source. Adding indexed plans to the query above yields:

  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (indexedJoin 1 2) (indexedJoin 2 1) (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (indexedJoin 4 3) (leftJoin 4 3)
  

  Hashed joins are added in the same manner:

  memo:
  ├── G1: (tablescan: ab)
  ├── G2: (tablescan: uv)
  ├── G3: (tablescan: pq)
  ├── G4: (hashJoin 1 2) (hashJoin 2 1) (indexedJoin 1 2) (indexedJoin 2 1) (innerJoin 2 1) (innerJoin 1 2)
  └── G5: (hashJoin 4 3) (indexedJoin 4 3) (leftJoin 4 3)
  

  This memo is missing several components that would make it more useful, including other node and expression types. For example, filter pushdown would be much easier in the memo. You could imagine extending the memo above with sql.Expression tree memo groups (scalar relations). Canonical (simplified) expression groups only requiring one allocation, and the scalar expression group caches properties like table and column dependencies similar to relProp column and table dependencies. We would need to build a memo prior to join planning to use filters for join planning, and run other transformation rules on the memo. I'd expect the memo to absorb rules over time, which will simplify transform logic and improve analyzer memory consumption. I added some starters to make this easier in the future, like codegen'd memo expressions, and a move towards visitor interfaces for IR transformations.
  A fully exhausted memo tree is converted back into a physical plan first through a costing process. Costing builds the fastest implementation for child group expressions bottom-up, using only the best child implementations for costing parent expressions and eventually the root node.
  The coster is designed to use histogram statistics to pick the fastest join plan. We only provide a subset of index metadata and table sizes at this moment. Additionally, costing should apply to generic SQL nodes and expressions, not just join trees and join leaves.
  Other notes:
  • we miss join plans that result from transitive predicates (ex: ab.a = uv.u + uv.u = xy.x => ab.a = xy.x).
  • we miss join plan optimizations resulting from table functional dependencies (ex: eliminate redundant filters if we already join on a a primary key), and null-allowing joins (ex: IS NULL).
  • we could more aggressively apply filters to join trees.
  • converting from the memo to an execution plan is fraught for the same reasons it was before this PR, but tree building and fixup is condensed in exec_builder, which i'd expect to expand to include more exprGroup types in the future.
  • there is a hack for fixing up field indexes in join trees not considered for reordering.
  • right joins are converted to left joins after expandStars in transposeRightJoins.
  • non-left deep trees necessitate passing parent rows down the right side of join trees, which were previously only table sources (left deep).
  • joinOrder hinting is rewritten to more easily fit within exprGroup costing.
  • I started to use bitmaps to track expression group column dependencies, which should also be used for filter applicability.
  • indexedJoinIter is duplicate and should be deprecated, but i had a hard time quickly removing it.
  • every join node is condensed into *plan.JoinNode and differentiated based on its join type, which lets me bucket different join types more easily (refer to JoinType type methods).
  • applyHashLookups should be deprecated, given that we will preemptively build hash joins. If hash joins do not select the lookup, we should either not cache the subquery or improve the coster to make better decisions.
  • there are probably bugs where we apply hash joins to non-cacheable subquery scopes for derived tables. Need more testing.
    Additional refs:
  • https://www.researchgate.net/publication/262216932_On_the_correct_and_complete_enumeration_of_the_core_search_space
  • https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/xform/join_order_builder.go.
  • https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/xu-columbia-thesis1998.pdf

Closed Issues

• 4526: Duplicate tag error
• 4603: dolt migrate: differing rows
• 4534: Derived Table access to Outer Query Scope

Latency

Merged PRs

dolt

• 4598: go/doltcore/doltdb: Account for drop/add column tag reuse when generating new columns
• 4597: adding add autoinc col tests
• 4596: Made server_query error on bad queries where we expect no results.
  Then fixed all the broken bats tests.
• 4577: Add --not, --min-parents, and --merges to dolt_log table function

go-mysql-server

• 1341: fix comparison for geometry types
  fix for: https://github.com/dolthub/dolt/issues/3451
  Maybe all geometry comparisons should just default to their EWKB formats; pretty confident this is what MySQL does.

Closed Issues

• 4581: Can't add an auto_increment primary key to a keyless table

Latency

Merged PRs

dolt

• 4592: Fix for dropping and re-adding an existing table with a column tag of 0 incorrectly regenerates its tag.
• 4590: go/doltcore/schema: Added doc comments to schema.Schema
• 4586: go/doltcore/sqle: Removing unused DDL code
• 4574: Add --parents and --decorate to dolt_log table function

Closed Issues

• 4531: calling set @@session.global_var = value should fail

Latency

Merged PRs

dolt

• 4582: Fix panic when merging tables that have a check constraint with some string literal
  This PR fixes a panic that occurs when merging tables that define a check constraint using some string literal. The panic's stacktrace is:

  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x2 addr=0x10 pc=0x10193c7f8]
  goroutine 1 [running]:
  github.com/dolthub/go-mysql-server/sql/parse.convertVal(0x0?, 0x14000e80520)
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/parse/parse.go:3392 +0x658
  github.com/dolthub/go-mysql-server/sql/parse.ExprToExpression(0x0, {0x10282ae00?, 0x14000e80520?})
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/parse/parse.go:3100 +0x1ee4
  github.com/dolthub/go-mysql-server/sql/parse.comparisonExprToExpression(0x10281e901?, 0x14000e86c80)
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/parse/parse.go:3491 +0x54
  github.com/dolthub/go-mysql-server/sql/parse.ExprToExpression(0x0, {0x10282ab00?, 0x14000e86c80?})
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/parse/parse.go:3089 +0x1cb8
  github.com/dolthub/go-mysql-server/sql/parse.ExprToExpression(0x0, {0x10282ada0?, 0x14000200e70?})
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/parse/parse.go:3164 +0x1594
  github.com/dolthub/go-mysql-server/sql/analyzer.ConvertCheckDefToConstraint(0x14000cbdf88?, 0x14000cbe288)
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/sql/analyzer/check_constraints.go:317 +0xc4
  github.com/dolthub/go-mysql-server.ColumnsFromCheckDefinition(0x10250f760?, 0x14000cbe3d0?)
  /Users/dhruv/go/pkg/mod/github.com/dolthub/[email protected]/engine.go:583 +0x1c
  github.com/dolthub/dolt/go/libraries/doltcore/merge.mergeChecks({0x102833588, 0x14000666e70}, {0x102833588, 0x14000667020}, {0x102833588, 0x14000667080})
  /Users/dhruv/dolt/go/libraries/doltcore/merge/merge_schema.go:807 +0x31c
  github.com/dolthub/dolt/go/libraries/doltcore/merge.SchemaMerge(0x1036fac40?, {0x10284f700, 0x1400065d920}, {0x10284f700, 0x1400065dc20}, {0x10284f700, 0x1400065dd40}, {0x1036fac40, 0x1})
  /Users/dhruv/dolt/go/libraries/doltcore/merge/merge_schema.go:169 +0x5e0
  github.com/dolthub/dolt/go/libraries/doltcore/merge.(*RootMerger).MergeTable(0x14000cbf0e0, {0x1028344d8, 0x14000e863c0}, {0x1036fac40, 0x1}, {0x0, {0x102824888, 0x140006611a0}, {0x1400066c6c0, 0x2c}}, ...)
  /Users/dhruv/dolt/go/libraries/doltcore/merge/merge_rows.go:123 +0x1ec
  github.com/dolthub/dolt/go/libraries/doltcore/merge.MergeRoots({0x1028344d8, 0x14000e863c0}, 0x1400067f1d0, 0x1400067f220, 0x1400067f3b0, {0x102824130, 0x14000e86500}, {0x102824130, 0x14000e86980}, {0x0, ...}, ...)
  /Users/dhruv/dolt/go/libraries/doltcore/merge/merge.go:131 +0x62c
  github.com/dolthub/dolt/go/libraries/doltcore/merge.MergeCommits({0x1028344d8, 0x14000e863c0}, 0x14000cbf3a8?, 0xb9ce7a80f36fd6ca?, {0x0, {0x102824888, 0x140006611a0}, {0x1400066c6c0, 0x2c}})
  /Users/dhruv/dolt/go/libraries/doltcore/merge/merge.go:66 +0xf0
  github.com/dolthub/dolt/go/libraries/doltcore/merge.ExecuteMerge({0x1028344d8, 0x14000e863c0}, 0x14000e863c0?, 0x14000eb6000)
  /Users/dhruv/dolt/go/libraries/doltcore/merge/action.go:223 +0x9c
  github.com/dolthub/dolt/go/cmd/dolt/commands.performMerge({0x1028344d8, 0x14000e863c0}, 0x14000eb6000?, 0x14000eb6000, {0x14000656280, 0x1e})
  /Users/dhruv/dolt/go/cmd/dolt/commands/merge.go:497 +0xfc
  github.com/dolthub/dolt/go/cmd/dolt/commands.MergeCmd.Exec({}, {0x1028344d8, 0x14000e863c0}, {0x14000668210, 0xa}, {0x1400004e1d0, 0x1, 0x1}, 0x1400066a090)
  /Users/dhruv/dolt/go/cmd/dolt/commands/merge.go:184 +0x998
  github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.handleCommand({{0x101e529aa, 0x4}, {0x101e6f2be, 0x11}, {0x0, 0x0}, {0x14000db6000, 0x2f, 0x2f}, 0x0}, ...)
  /Users/dhruv/dolt/go/cmd/dolt/cli/command.go:237 +0x450
  github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.Exec({{0x101e529aa, 0x4}, {0x101e6f2be, 0x11}, {0x0, 0x0}, {0x14000db6000, 0x2f, 0x2f}, 0x0}, ...)
  /Users/dhruv/dolt/go/cmd/dolt/cli/command.go:186 +0x390
  main.runMain()
  /Users/dhruv/dolt/go/cmd/dolt/dolt.go:395 +0x6ec
  main.main()
  /Users/dhruv/dolt/go/cmd/dolt/dolt.go:140 +0x1c
  

• 4576: Add Docker Image support for linux/arm64
  Added Docker Image for linux/arm64 addition to linux/amd64 architecture. Each will be pushed to DockerHub as separate tags, but both image will be available for latest or <release_version> tags.
• 4575: go/store/val: Add env var for disabling fast fixed-width access
  partial fix for #4527
• 4572: Support two dot log in dolt_log table function
  Supports these two dot log syntaxes:

  select * from dolt_log(<revision>..<revision>)
  select * from dolt_log(^<revision>, <revision>)
  

  Adding --not and other arguments/options next
• 4571: Branch Control Pt. 8
  Per the suggestion, this is a rewrite of branch control persistence using flatbuffers. Using the rest of the codebase as an example, it looks like we write vectors and such first before writing them to the object that they belong to, so I'm following that style.
• 4564: Add basic dolt_log table function
  Supports select * from dolt_log(<revision>) (<revision> is optional). Adding more options in a follow up PR
• 4561: Support two dot log for CLI
• 4560: go/doltcore/{table, rowconv}: Cleanup dead code
• 4559: Branch Control Pt. 6
  This fixes a few issues found through testing, and also adds persistence for the tables. Also re-added the binlog_table.go file. I originally removed it, but misread the feedback that said to not expose it rather than to remove it.
• 4558: dolt support for GeometryCollection
  fix for: https://github.com/dolthub/dolt/issues/3638
• 4553: Branch Control Pt. 7
  Tests for branch control tables. Things left to do:
  • Bats tests
  • Logic and tests for the rest of the dolt procedures
  • Moving the tables into the context
  • Support for roles
• 4546: fix filters on dolt_diff system table

go-mysql-server

• 1341: fix comparison for geometry types
  fix for: https://github.com/dolthub/dolt/issues/3451
  Maybe all geometry comparisons should just default to their EWKB formats; pretty confident this is what MySQL does.
• 1337: add support for GeometryCollection pt. 3
  Part 10 of fix for: https://github.com/dolthub/dolt/issues/3638
  Changes:
  • these functions now work with GeometryCollection
  • st_srid
  • st_swap
• 1336: can't alter blob, json, and geometry columns when other defaults are defined
  fix for: https://github.com/dolthub/dolt/issues/4543
  Note: the syntax here is actually invalid in MySQL?
• 1335: add support for GeometryCollection pt. 2
  Part 9 of fix for: https://github.com/dolthub/dolt/issues/3638
  Changes:
  • refactor a bit of wkt code
  • added missing implementation and tests for wkt for multipolygon
  • these functions now work with GeometryCollection
  • st_geometrycollectionfromtext
  • st_geomcollfromtext
  • st_geomcollfromtxt
  • st_geomfromtext
  • st_astext
  • st_aswkt
  • st_dimension
• 1332: add support for GeometryCollection pt. 1
  Part 8 of fix for: https://github.com/dolthub/dolt/issues/3638
  Changes:
  • have the deserialize data and write data methods return their own byte counts, to more easily track where we are in main deserailizer
  • adding GeometryCollection type
  • these functions now work with GeometryCollection
  • GeometryCollection
  • GeomCollection
  • st_aswkb
  • st_geometrycollectionfromwkb
  • st_geomcollfromwkb

Closed Issues

• 3451: Wrong result from select statement with filter on spatial types
• 3638: Support MultiLineString, MultiPolygon, MultiPoint and GeometryCollection geometry types
• 4557: Inserting bulk records is too slow
• 1319: Filter pushdown doesn't show in explain plan even though pushdown was successful

Latency

Merged PRs

dolt

• 4555: update Dockerfile on release
• 4554: go/doltcore/schema: Remove schema.CollCollection.GetAtIndex() in favor of GetByIndex
• 4550: get next port number if random port is in use
  fixed in bats testing:
  • if PORT in use, get the next unused PORT number when starting a sql-server
  • kill a process only if it's still running when stopping a sql-server
• 4542: dolt support for MULTIPOLYGON
  Fix for: https://github.com/dolthub/dolt/issues/3638
• 4538: add list of dolt config variables to command description
  Supported config variables are added to long description of dolt config, which can be shown with --help flag.

go-mysql-server

• 1336: can't alter blob, json, and geometry columns when other defaults are defined
  fix for: https://github.com/dolthub/dolt/issues/4543
  Note: the syntax here is actually invalid in MySQL?
• 1334: multipolygon support
• 1333: prevent creating views with same name as existing table
  also fix for: https://github.com/dolthub/dolt/issues/4549
• 1330: add support for MultiPolygon pt. 3
  Part 7 of fix for: https://github.com/dolthub/dolt/issues/3638
  TODO: rebase to main, left as a merge to james/mpoly2 for better readability
  Changes:
  • these functions now work with MultiPolygon
  • ST_ASGEOJSON
  • ST_GEOMFROMGEOJSON
  • lots of tests
• 1327: add support for MultiPolygon pt. 2
  Part 6 of fix for: https://github.com/dolthub/dolt/issues/3638
  TODO: rebase to main, left as a merge to james/mpoly for better readability
  Changes:
  • these functions now work with MultiPolygon
  • ST_ASWKT
  • ST_GEOMFROMTEXT
  • added engine tests
• 1324: add support for MultiPolygon pt. 1
  Part 5 of fix for: https://github.com/dolthub/dolt/issues/3638
  TODO: rebase to main, left as a merge to james/mline3 for better readability
  Changes:
  • contains small fixes to multilinestring comments and allocation length
  • these functions now work with MultiPolygon
  • MULTIPOLYGON
  • ST_ASWKB
  • ST_GEOMFROMWKB
  • ST_MPOLYFROMWKB
  • ST_MULTIPOLYGEOMFROMWKB
  • added engine tests
• 1321: add support for MultiLineString pt. 1
  Part 2 of fix for: https://github.com/dolthub/dolt/issues/3638
  Changes:
  • small copy-pasta comment fixes
  • adding multilinestring type and structs
  • MultiLineString support for these sql functions
  • ST_MULTILINESTRINGFROMWKB
  • ST_MULTILINESTRINGFROMTEXT
  • ST_MLINESTRINGFROMWKB
  • ST_MLINESTRINGFROMTEXT
  • ST_GEOMFROMWKB
  • ST_ASWKB
  • ST_ASTEXT

Closed Issues

• 4543: alter table t alter column set default c1 ('[]') fails with TEXT, BLOB, GEOMETRY, and JSON types may only have expression default values
• 4549: views with same name as table, causes inf recursion
• 4430: Feature request: add a progress bar to dolt sql < file.sql that shows the progress of a SQL load

Latency

Merged PRs

dolt

• 4545: Enable new clones to begin replicating on the primary
• 4539: Replace incorrect character.
  Replaces an incorrect ' character with the correct ` character.
  The current code results in broken docs html:
  
• 4528: Branch Control Pt. 5
  This is an aggregate of
  • https://github.com/dolthub/dolt/pull/4507
  • https://github.com/dolthub/dolt/pull/4508
  • https://github.com/dolthub/dolt/pull/4509
  • https://github.com/dolthub/dolt/pull/4510
    along with each PR's respective feedback.
• 4524: add file read progress info for dolt sql --file=<file.sql> command
  There will be percentage progress output printed for --file flag of dolt sql command. It can be used with or without --batch flag.
• 4512: Tests for unique index creation on keyless tables
  tests for: https://github.com/dolthub/dolt/issues/4420
  companion pr: https://github.com/dolthub/go-mysql-server/pull/1320
• 4505: dolt support for MULTILINESTRING
  Fix for: https://github.com/dolthub/dolt/issues/3638
  Companion PR: https://github.com/dolthub/go-mysql-server/pull/1323

go-mysql-server

• 1329: explain output includes FilteredTable
  We already print filtered table in the debug string for IndexedTableAccess, this adds it to regular explain and ResolvedTable.
• 1323: add support for MultiLineString pt. 3
  Part 4 of fix for: https://github.com/dolthub/dolt/issues/3638
  TODO: rebase to main, left as a merge to james/mline2 for better readability
  Changes:
  • these functions now work with MultiLineString
  • ST_GEOMFROMGEOJSON
  • ST_ASGEOJSON
  • added engine tests
• 1322: add support for MultiLineString pt. 2
  Part 3 of fix for: https://github.com/dolthub/dolt/issues/3638
  TODO: rebase to main, left as a merge to james/mline for better readability
  Changes:
  • small fix to polygon test
  • these functions now work with MultiLineString
  • ST_SWAP
  • ST_SRID
  • ST_DIMENSION
  • MULTILINESTRING
• 1321: add support for MultiLineString pt. 1
  Part 2 of fix for: https://github.com/dolthub/dolt/issues/3638
  Changes:
  • small copy-pasta comment fixes
  • adding multilinestring type and structs
  • MultiLineString support for these sql functions
  • ST_MULTILINESTRINGFROMWKB
  • ST_MULTILINESTRINGFROMTEXT
  • ST_MLINESTRINGFROMWKB
  • ST_MLINESTRINGFROMTEXT
  • ST_GEOMFROMWKB
  • ST_ASWKB
  • ST_ASTEXT
• 1320: rewrite table on unique key creation
  Fix for: https://github.com/dolthub/dolt/issues/4420
  In-memory tables don't support unique keys, so tests are in dolt:
  https://github.com/dolthub/dolt/pull/4512
• 1311: internal/sockstate: Fix sockets ending up in blocking mode when we monitor them for disconnects.
  As documented in os, calling *os.File.Fd() will put the socket into blocking mode, makes SetDeadline methods stop working, etc. This syscall to SetNonblocking restores desired functionality.
  This was discovered when testing clustering control plane operations in Dolt, which rely on the timely ability to terminate all client connections by calling Close() on them.
  We were able to reproduce the issue on macOS by doing the same File() behavior there, despite not having an implementation to actually use the fd for external monitoring of the connection state. To keep as much behavioral parity going forward, I left the fd translation in, since we've observed that it can radically change behavior.

Closed Issues

• 4430: Feature request: add a progress bar to dolt sql < file.sql that shows the progress of a SQL load
• 4518: dolt_remotes system table not accessible when connecting with a branch string
• 4270: Diff shows rows that have no diffs when a column has been added in the past.
• 4211: Inserting large JSON object in a prepared statement causes panic
• 4420: Dolt allows me to create an unique index that violates uniqueness
• 1331: CreateTable: DEFAULT CHARACTER SET missing
• 945: Client hangs if go-mysql-server takes too long to return query results

Latency

Merged PRs

dolt

• 4523: use auto-generated commit message if editor cannot be opened or got empty result
  Dolt tries getting system default editor if it's set, else it tries getting dolt core.editor if it's set, else it uses vim editor to get commit message. If vim is not installed on user's system, then it uses auto-generated commit message.
• 4522: fix insert ... on duplicate key update ... for unique keys
  fix for: https://github.com/dolthub/dolt/issues/4475
  only happens in new format, solution was just to validate any key violations before inserting anything.
• 4519: go/doltcore/sqle: Initialize remotes, backups, and branch config when constructing revision databases
  fix for #4518
• 4513: /go/performance/import_benchmarker: better err handling
• 4488: adding multipoint support to dolt
  Companion PR: https://github.com/dolthub/go-mysql-server/pull/1309
  Changes:
  • added support for MultiPoint type
  • read_geometry.go and write_geometry.go are part of types package
  • read_geometry.go and write_geometry.go passthrough to sql implementation of Serialize and Deserialize
  • prolly_indexes uses references sql implementations directly
  • moved helper Conversion functions to/from sql/types to read/write_geometry
  • included some checks to ensure noms kinds don't suddely change

go-mysql-server

• 1311: internal/sockstate: Fix sockets ending up in blocking mode when we monitor them for disconnects.
  As documented in os, calling *os.File.Fd() will put the socket into blocking mode, makes SetDeadline methods stop working, etc. This syscall to SetNonblocking restores desired functionality.
  This was discovered when testing clustering control plane operations in Dolt, which rely on the timely ability to terminate all client connections by calling Close() on them.
  We were able to reproduce the issue on macOS by doing the same File() behavior there, despite not having an implementation to actually use the fd for external monitoring of the connection state. To keep as much behavioral parity going forward, I left the fd translation in, since we've observed that it can radically change behavior.
• 1309: Adding Support for MULTIPOINT and some refactoring
  Part 1 of fix for: https://github.com/dolthub/dolt/issues/3638
  It looks like PRs for this are gonna get pretty big, so I'm going to split them up for easier reviewing.
  Changes:
  • greatly improved code reuse for spatial types
  • refactored code to be more readable
  • some functions were misnamed FROMWKT, when they should've been FROMTEXT
  • ST_POINTFROMWKT -> ST_POINTFROMTEXT
  • ST_LINEFROMWKT -> ST_LINEFROMTEXT
  • ST_POLYFROMWKT -> ST_POLYFROMTEXT
  • missing function aliases
  • ST_LINESTRINGFROMTEXT
  • ST_LINESTRINGFROMWKB
  • ST_POLYGONFROMTEXT
  • ST_POLYGONFROMWKB
  • ST_GEOMETRYFROMTEXT
  • ST_GEOMETRYFROMWKB
  • these functions now work with MULTIPOINT
  • ST_SWAP
  • ST_SRID
  • ST_ASWKB
  • ST_MULTIPOINTFROMWKB
  • ST_MULTIPOINTFROMTEXT
  • ST_MPOINTFROMWKB
  • ST_MPOINTFROMTEXT
  • ST_GEOMFROMWKB
  • ST_ASGEOJSON
  • ST_FROMGEOJSON
  • ST_DIMENSION
  • fixed bug where x and y geojson values were being swapped
• 1307: allow having node access to all tables in itself
  HAVING clause can reference column that is not in its select result or group by result (there can be no group by clause). These column references are from tables in HAVING node that it is not in its immediate children nodes.

Closed Issues

• 4270: Diff shows rows that have no diffs when a column has been added in the past.
• 4211: Inserting large JSON object in a prepared statement causes panic
• 4475: Dolt not respecting composite unique index.
• 4521: Pull fails due to missing commit message
• 4436: Support BEGIN WORK
• 3791: Dolt backup sync from SQL context returns success 0 instead of success 1.
• 4463: Official Docker hub image?
• 945: Client hangs if go-mysql-server takes too long to return query results
• 1164: Support the DO statement
• 1162: Support the EXTRACT function
• 1128: superusers should be hidden
• 1289: Some queries using HAVING can't be executed without GROUPBY

Latency

Merged PRs

dolt

• 4492: go/doltcore/env: Fix format logic for multi-database environments
• 4471: go/doltcore/actions: Sample CSV rows when inferring schema on import
• 4470: push docker image to dockerhub
  Build Docker Image for dolt and push on every release
  Dockerfile is WIP
• 4467: store/datas/pull: If the BPS is less than 1, report 0
  
  ^ prevents the push / pull progress reporter from reporting very small BP/s units.
• 4466: go: sqle: remotesrv: Successfully replicate CREATE DATABASE commands by instantiating new databases on the standby on demand.
• 4461: wrap spatial type string in single quotes when exporting
• 4458: Persist a commit mapping to the chunkstore after migration
  The commit mapping is persisted as dangling chunks. We don't expect it to remain after a GC and it will not be picked up during a push or pull.
• 4449: integration-tests/go-sql-server-driver: Proof of concept a declarative approach to integration tests for sql-server interactions.
  Converts bats/sql-server-config.bats to use a new approach.

go-mysql-server

• 1316: Slight port correction on README example
• 1315: Updated README example to fit current usage
  We haven't used []string{""} for JSON values in a long time, the README definitely needed an update.
• 1308: parallelize static lookups
• 1307: allow having node access to all tables in itself
  HAVING clause can reference column that is not in its select result or group by result (there can be no group by clause). These column references are from tables in HAVING node that it is not in its immediate children nodes.
• 1304: integration plan regressions
  This PR addresses two problems:
  1. Optimization rules that depend on pattern matching can fail to trigger with exchange nodes, which we failed to account for in testing. This fixes a small indexed join bug and runs integration query plans with parallelism = 2.
  2. IndexedInSubqueryFilter should be prevented when the static side of the join takes a dependency on the index lookup. The check is now looser, and opaque nodes cannot disallow the transform. When we introduce subqueries that do reference outer scopes, we will need to be more careful and do deeper validation.
• 1303: adding support for st_area, st_perimeter, st_length
  Only the Cartesian portion of these function is working, geodetic calculations are quite a bit harder.
  I just throw an error for any unsupported functionality.
  st_area is calculated using shoelace formula.
  Basic idea is to slice polygon into a bunch of triangles and sum up their areas. The function is not defined for polygons that intersect themselves, but from my testing it seems like MySQL returns the same values for these edge cases.
  st_perimeter is a NOT supported in MySQL, but it is in PostGIS (a postgres plugin for spatial types): https://postgis.net/docs/ST_Perimeter.html
  Fix for: https://github.com/dolthub/dolt/issues/4451
• 1301: Ensuring read-only check is executed for prepared and non-prepared statements
  Quick fix for: https://github.com/dolthub/dolt/issues/4434
  Tested locally with a python script to verify that prepared statements to read-only dbs are now correctly blocked. After getting this fix out, we will follow up with more formal, automated test to cover this.
• 1293: no column reference can be made on dual table
  Depends on https://github.com/dolthub/vitess/pull/196
  No GetField reference can be made on dual table. To differentiate between dual table and `dual` table(which can be created).
  Dual table is constructed as ResolvedTable with empty table name and a single column with empty name in the parser.
  Any column reference that becomes deferredColumn is replaced into alias that is present in projectedAliases in reorderProjection rule.
  Any GetField column reference is not allowed on dual table.

vitess

• 197: adding better support for transaction statements
  MySQL reference: https://dev.mysql.com/doc/refman/8.0/en/commit.html
  adds support for

  BEGIN [WORK]
  COMMIT [WORK] [AND [NO] CHAIN] [[NO] RELEASE]
  ROLLBACK [WORK] [AND [NO] CHAIN] [[NO] RELEASE]
  

  fix for: https://github.com/dolthub/dolt/issues/4436
• 196: allow dual only for select from statements if used without back-ticks

Closed Issues

• 3039: Implement the CHECK TABLE syntax.
• 4498: Incorrect Group by syntax passes validation and executes
• 3008: Support CASCADE syntax for ALTER TABLE
• 4497: Support all MySQL logical operators
• 4483: Support JOIN in UPDATE for unkeyed tables
• 1225: VALUES not accessible by variable name in ON DUPLICATE KEY UPDATE
• 3337: Difficult to identify differences in dolt_docs when in conflict
• 3205: QGIS plugin
• 809: dolt sql-server should ignore subdirectories that are not Dolt repos with a warning
• 3959: Transactional commit does not work when autocommit is off and a database is not selected
• 3149: Upload units do not properly update
• 4416: Can't round trip dolt dump
• 3729: Data from export table failed to load via LOAD DATA
• 4463: Official Docker hub image?
• 1841: Make dolt_branches table read-only.
• 4459: New table partitions code doesn't appear to be concurrent safe.
• 3763: concurrency errors with dolt_merge
• 3470: Subscribe to changes to a document/query (live query)
• 3445: Go index servers didn't pick up the dolt project
• 3219: TestLoadData fails with Dolt integrator
• 3160: Dolt panic on attempt to push to branch without permission
• 3130: Prevent altering table tables to have FULLTEXT key property
• 4451: Implement the ST_AREA() and ST_PERIMETER() functions
• 3018: Audit Error Codes
• 3020: UNION does not respect ORDER BY or LIMIT
• 575: Bad error message for omitting branch in push command
• 515: git-dolt update should be able to update dolt repo and handle branch references
• 4256: dual table has different behavior from MySQL, some cause panics
• 4435: Question on REPLACE with trigger
• 4434: read_only mode in sql-server not working properly
• 1289: Some queries using HAVING can't be executed without GROUPBY
• 1314: Cannot connect to database to query MemoryTable
• 750: Copy the mysql behaviour regarding reserved keywords
• 738: mysql SERIAL alias unsupported
• 644: SHOW TABLE CREATE PRIMARY KEY ordering
• 611: Bad error message for type conversion error on insert
• 610: EXISTS expects a single column in subquery result, should support any number
• 592: Creating a table without a current DB give confusing error message
• 562: Intermittent DATA RACE error when creating MySQL Servers
• 561: Support for generated columns
• 526: Cleanup PrimaryKey error vs UniqueKey error vs DuplicateEntry
• 513: Need tests of PREPARE
• 498: Inserting into UINT32 column can cause error with leading 0
• 479: Cannot use expression index in indexed join
• 476: Can a Table implement sql.IndexAddressableTable without implementing sql.DriverIndexableTable?
• 383: Unbound variables in subqueries
• 463: anyone writing the rocksdb backend?

Latency

Merged PRs

dolt

• 4448: go/doltcore/sqle: Update system tables tests in pkg sqle
• 4447: bats: cluster: Add a test for a couple failovers and the new primary having all writes.
• 4446: go: sqle: cluster: commithook: Log warnings when the commithook sees commits when we are not primary.
• 4445: go: sqle: cluster: Have call dolt_assume_cluster_role() transition gracefully when going primary -> standby. Fail if it cannot true-up the peer.
• 4444: go: sqle: cluster: Have remotesapi client and server interceptors transition server cluster role if they notice a new primary or broken cluster configuration.
• 4443: go: sqle: cluster: When the server has role "standby", make all dolt databases read-only.
• 4441: go: sqle: cluster: Add a mechanism to kill inflight queries and connections when transition server role.
• 4440: Fix broken schema table test
• 4439: .github/workflows: Run full CI against both __DOLT__ and __LD_1__
  The default NomsBinFormat is now __DOLT__. This change updates CI to continue running golang tests and BATS against the old NBF __LD_1__
• 4438: Walk artifacts ref
  • Reordered table serial message address walk to match the definitions in table.fbs
  • Now walks artifact ref hash. Previously we were not walking it :(
• 4437: Add integration tests
• 4433: go: sqle: cluster: Add a ChunkStore wrapper for use in our remotesrvServer to record successful commits on the standby.

go-mysql-server

• 1301: Ensuring read-only check is executed for prepared and non-prepared statements
  Quick fix for: https://github.com/dolthub/dolt/issues/4434
  Tested locally with a python script to verify that prepared statements to read-only dbs are now correctly blocked. After getting this fix out, we will follow up with more formal, automated test to cover this.
• 1297: Add query plan integration tests
  1. New integration test queries
  2. Fix some join planning bugs related to i) column casing and ii) TableWrappers
• 1296: server: context.go: Expose *SessionManager.KillConnection.

vitess

• 197: adding better support for transaction statements
  MySQL reference: https://dev.mysql.com/doc/refman/8.0/en/commit.html
  adds support for

  BEGIN [WORK]
  COMMIT [WORK] [AND [NO] CHAIN] [[NO] RELEASE]
  ROLLBACK [WORK] [AND [NO] CHAIN] [[NO] RELEASE]
  

  fix for: https://github.com/dolthub/dolt/issues/4436
• 195: Allow other options to precede COLLATE in column type defintions
  Fixes https://github.com/dolthub/dolt/issues/4403
  Previously, COLLATE had to immediately follow the column type in a column definition. As per the above issue, this is not a required rule in MySQL. The fix involved moving all collation-related options from the column_type rule to the column_type_options. This caused a conflict in column_type_options, as column_default also has a COLLATE rule. The conflicting rule resolved to the following:

  column_type_options DEFAULT value_expression COLLATE ID
  

  For reference, this is the new COLLATE rule in column_type_options:

  column_type_options COLLATE ID
  

  Given the MySQL expression DEFAULT "xyz" COLLATE utf8mb4_bin, it could match either rule, which caused the conflict. value_expression is too permissive in this context, and although we filter out invalid expressions in GMS, we need to be more restrictive here in the parser to prevent conflicts. In addition, the above example should put the collation on the column, as it is not possible to add a collation to a default string literal (must use the expression form: DEFAULT ("xyz" COLLATE utf8mb4_bin)).
  To fix this, the column_default rule was updated to be vastly more restrictive. This also highlighted some tests in GMS that enforce incorrect behavior, but those have been fixed ahead-of-time, and will be incorporated into the bump PR.
  NOTE: For some reason, we had tests allowing the use of utc_timestamp, utc_date, etc. as default values without the need for parentheses. This is not allowed in MySQL, so I am unsure as to why we allowed them in the first place. Perhaps because they're similar to CURRENT_TIMESTAMP, so we just allowed all of them? Regardless, the tests have been removed as they no longer pass (as they should not pass).

Closed Issues

• 4436: Support BEGIN WORK
• 1300: Inserts for some values get dropped when accessed via a trigger for inserts.

Latency

Merged PRs

dolt

• 4431: Make NomsBinFormat __DOLT__ the default format
• 4429: go: sqle: cluster: Add a dolt_cluster database, exposing a dolt_cluster_status table which queries replication status.
• 4426: Reference new socket interface functionality in README
• 4422: go/libraries/doltcore/sqle: Implement standby replication hook.
• 4318: implement call dolt_conflicts_resolve()
  Closes https://github.com/dolthub/dolt/issues/2475

go-mysql-server

• 1296: server: context.go: Expose *SessionManager.KillConnection.

Closed Issues

• 2587: Can't add columns to a table with an AUTO_INCREMENT column that isn't primary key
• 2475: dolt_conflicts_resolve() SQL procedure
• 4176: Collation not respected properly by LIKE and REGEXP_LIKE
• 1144: Is it possible to run raw sql queries to mock the database?

Latency

Current Format (__DOLT__)

Old Format (__LD_1__)

Merged PRs

dolt

• 4413: turn off opt-in feature for unix socket
  Initial implementation of unix socket has opt-in feature for unix socket support. The default of mysql unix socket when socket flag is not defined is when host is defined as 'localhost' or undefined.
• 4395: improve error message when current branch on session is force deleted
  An active branch in session can be force deleted. The session on deleted branch cannot run any queries that requires dolt transaction. It gives error message with possible actions that user can take, either to run USE <database>/<branch> query or reconnect to the server.
  Depends on https://github.com/dolthub/go-mysql-server/pull/1288

go-mysql-server

• 1291: allow adding column to table with autoincrement column
  fix for: https://github.com/dolthub/dolt/issues/2587
• 1288: add ValidateSession() to Session interface
  https://github.com/dolthub/dolt/pull/4395 depends on this PR
• 1280: More join types
  Add FullOuterJoin, SemiJoin, and AntiJoin.
  None of these new join nodes are safe for join ordering transformations yet. They are explicitly excluded from join planning.
  The getField indexes for these three nodes' join conditions deserve more consideration. I excluded them from auto-fixup after manually correcting join condition get fields for the appropriate schemas.
  FullOuterJoin uses a union distinct execution operator, which is correct but a lot slower than a merge join-esque operator.
  SemiJoin and AntiJoin rearrange subquery expression scopes. I separate resolve and finalizeSubqueryExpressions to perform decorrelation before predicate pushdown (where we were panicking on FixUpExpressions) and join ordering (we want to decorrelate scopes before join planning).
  Other:
  • query plan tests added for exist hoisting edge cases i did not catch on first pass
  • fixed bug with CTE stars

vitess

• 195: Allow other options to precede COLLATE in column type defintions
  Fixes https://github.com/dolthub/dolt/issues/4403
  Previously, COLLATE had to immediately follow the column type in a column definition. As per the above issue, this is not a required rule in MySQL. The fix involved moving all collation-related options from the column_type rule to the column_type_options. This caused a conflict in column_type_options, as column_default also has a COLLATE rule. The conflicting rule resolved to the following:

  column_type_options DEFAULT value_expression COLLATE ID
  

  For reference, this is the new COLLATE rule in column_type_options:

  column_type_options COLLATE ID
  

  Given the MySQL expression DEFAULT "xyz" COLLATE utf8mb4_bin, it could match either rule, which caused the conflict. value_expression is too permissive in this context, and although we filter out invalid expressions in GMS, we need to be more restrictive here in the parser to prevent conflicts. In addition, the above example should put the collation on the column, as it is not possible to add a collation to a default string literal (must use the expression form: DEFAULT ("xyz" COLLATE utf8mb4_bin)).
  To fix this, the column_default rule was updated to be vastly more restrictive. This also highlighted some tests in GMS that enforce incorrect behavior, but those have been fixed ahead-of-time, and will be incorporated into the bump PR.
  NOTE: For some reason, we had tests allowing the use of utc_timestamp, utc_date, etc. as default values without the need for parentheses. This is not allowed in MySQL, so I am unsure as to why we allowed them in the first place. Perhaps because they're similar to CURRENT_TIMESTAMP, so we just allowed all of them? Regardless, the tests have been removed as they no longer pass (as they should not pass).
• 194: adding missing keywords to token.go
  I forgot to add some keywords used in ROW_FMT table option, which caused a bats test to fail

Closed Issues

• 4403: Modifying a column is order-sensitive for NOT NULL and COLLATE
• 387: Pushing to DoltHub with incorrect credentials does not fast fail
• 629: Peer-to-peer dolt server?
• 4419: Unicode error running a utf8mb4 MySQL database export
• 4360: Deleting branch causes error in another session if that branch is active
• 4358: support PARTITION syntax in CREATE TABLE
• 4353: Incorrect result from aggregating on table joined in both subquery and CTE
• 4344: Subquery alias naming conflict

Merged PRs

dolt

• 4408: reorder arguments of dolt_diff() table function
  This PR updates argument ordering of dolt_diff() table function to match dolt_diff_summary() table function, which matches CLI command argument order.
• 4401: go/commands: Add sql-server option for validating query results against a MySQL instance
  depends on https://github.com/dolthub/go-mysql-server/pull/1279
• 4400: allow drop current database directory and fix drop db with '-'
  This PR fixes two issues:
  • if the server is running in the database itself, the database can be dropped regardless of being selected database.
  • drop database with '-' in dir name and '_' in database name.
• 4399: adding prepared tests for dolt stored procedures
  Fix for: https://github.com/dolthub/dolt/issues/4392
• 4397: print spatial types as hex in shells only
  Related PR: https://github.com/dolthub/go-mysql-server/pull/1278
  Since the SQL methods for Point, LineString, Polygon, and Geometry no longer return the hex formatted string, need to do a little extra work for prettier shell output.
• 4394: sql-server: Implement dolt_cluster_role, dolt_cluster_role_epoch variables, dolt_assume_cluster_role procedure.
  This is just the variable persistence and some validation and rules around how they can transition.
• 4389: Fix race for concurrent dolt_commit calls
  This fixes https://github.com/dolthub/dolt/issues/4361
• 4387: remotestorage,remotesrv: Be less restrictive in repository paths for remotesapi repos that are not served through dolthub.
  This uses the repo_path parameter to allow single and 3+ path component paths for repository names.
• 4386: go/cmd/dolt/commands/sqlserver: yaml_config.go: Parse and validate cluster: config.
• 4385: Updated bats README and a test to make it easier to get started
  Also improved parquet tests
• 4384: Implement merge stats for the new format
• 4383: go/libraries/doltcore/sqle: ReadReplicaDatabase: Add variable for force pulling remote branches.
  This adds @@global.dolt_read_replica_force_pull, which will turn read replica fetches into force pulls, allowing a read replica to true-up with a remote which has experienced a force push.
• 4363: support dolt_diff_summary_table_function
  dolt_diff_summary table function is the same as CLI dolt diff --summary command. It takes fromCommit, toCommit and tableName, which is optional.
  If tableName is not defined, the table function will output all tables with data diff.
  If the table does not have data diff, it will output empty result.
  If the table is keyless, like CLI command, only row added and row deleted will have appropriate information, and the rest is nil.
  Current issues:
  • drop-column causes row values to be modified into NULL, it should be different from user-defined NULL or NULL added from add-column.
  • for keyless table, update table causes delete old row and insert new row, which affects the row added and deleted result to be incorrect.
• 4268: Feat/support aliyun oss store
  support aliyun oss as backend store

go-mysql-server

• 1281: Alias resolution updates
  This change moves our alias identification and column name qualification code closer to MySQL's behavior. The major changes are:
  • identify available alias, table, and column names on a per-scope level (instead of the previous "nestingLevel" that was based off of the node depth of the execution plan tree)
  • supply additional context to the qualifyExpression method so that the containing node can be used to switch on different alias visibility rules
    There are still more nuanced edge cases in MySQL's alias behavior that we should continue working on matching (e.g. https://github.com/dolthub/go-mysql-server/issues/1285, https://github.com/dolthub/go-mysql-server/issues/1286), but this change moves us a nice step forward.
    Dolt CI Tests:
  • https://github.com/dolthub/dolt/pull/4410
    Fixes:
  • https://github.com/dolthub/go-mysql-server/issues/525
  • https://github.com/dolthub/dolt/issues/4344
• 1280: More join types
  Add FullOuterJoin, SemiJoin, and AntiJoin.
  None of these new join nodes are safe for join ordering transformations yet. They are explicitly excluded from join planning.
  The getField indexes for these three nodes' join conditions deserve more consideration. I excluded them from auto-fixup after manually correcting join condition get fields for the appropriate schemas.
  FullOuterJoin uses a union distinct execution operator, which is correct but a lot slower than a merge join-esque operator.
  SemiJoin and AntiJoin rearrange subquery expression scopes. I separate resolve and finalizeSubqueryExpressions to perform decorrelation before predicate pushdown (where we were panicking on FixUpExpressions) and join ordering (we want to decorrelate scopes before join planning).
  Other:
  • query plan tests added for exist hoisting edge cases i did not catch on first pass
  • fixed bug with CTE stars
• 1279: Andy/mysql validator
• 1278: should not be returning hex on the wire for geometry types
  Fix for: https://github.com/dolthub/dolt/issues/4390
  For improved display purposes, I changed the SQL method to convert the raw binary of spatial types to their hex equivalent (https://github.com/dolthub/go-mysql-server/pull/1068).
  DBeaver expects the binary in WKB format to generate their maps, so our SQL method must return them in binary
• 1276: Apply sql_select_limit to first scope only
  re: https://github.com/dolthub/dolt/issues/4391 and https://github.com/dolthub/dolt/issues/4353
• 1275: Throw error in alter table set default if default fails rule(s)
  This PR makes alter column set default error when its default expression fails the default expression rules. Previously the DDL succeeded and when an insert or any other query was performed the default would cause a continuous error.

  alter table t alter column col1 set default '{\"bye\":1}'
  -- errors with: TEXT, BLOB, GEOMETRY, and JSON types may only have expression default values
  

• 1272: Removed exponential time complexity for foreign key analysis
  This fixes https://github.com/dolthub/go-mysql-server/issues/1268
  Foreign key analysis created acyclical trees that were traversed during query execution to emulate cascade operations. This meant that cyclical foreign keys were converted to an acyclical tree. Normally this isn't possible as cyclical trees are infinitely traversable, but MySQL has a depth limit of 15, which allowed us to materialize an acyclic tree with a maximum height of 15 nodes. This, however, lead to trees with an exponential number of nodes: roughly (number_of_fks)¹⁵ × 1.5 nodes in the tree. With just 3 foreign keys, we'd get a tree with roughly 22 million nodes, which would take forever to process.
  This PR completely changes the analysis step to now generate cyclical trees. In addition, depth checks are now properly implemented (during query execution rather than during analysis), being represented by a returned error once the depth limit has been reached. Interestingly, MySQL is supposed to process up to 15 operations (returning an error on the 16th), but cyclical foreign keys will error on the 15th operation. I think this is a bug in MySQL, but nonetheless the behavior has been duplicated here.
  I also updated the timestamp_test.go file to grab an unused port. This prevents test failures due to requesting an already-in-use port. Not related to this PR in particular, but it was annoying to deal with so I fixed it.

vitess

• 194: adding missing keywords to token.go
  I forgot to add some keywords used in ROW_FMT table option, which caused a bats test to fail
• 193: adding support for PARTITION syntax for table creation
  Parsing table_options for statements like CREATE TABLE (column_definitions) table_options better matches MySQL
  Can parse partition_options based off of https://dev.mysql.com/doc/refman/8.0/en/create-table.html
  Fix for: https://github.com/dolthub/dolt/issues/4358
• 192: add full join parsing
• 191: go/mysql: query.go: Put strings into BindVars as copies, not slices of the recyclable wire packet buffers.
  Cherry picks https://github.com/vitessio/vitess/pull/5562.
• 190: Adding back support for using reserved keywords unquoted in InsertInto
  I previously excluded reserved_sql_id and only included sql_id because I didn't think reserved keywords should be allowed as identifiers in insert into columns without being quoted, but it appears we have tests in Dolt that rely on that (e.g. sql-diff.bats calls: INSERT INTO test (pk, int, string, boolean, float, uint, uuid) values ...) and I figured if we can support those without the quoting, it seems like a nice feature for customers, so I switched back to reserved_sql_id.
  It is confusing that we refer to identifiers as "reserved" keywords when they don't require backtick quoting though and would be nice to tidy that up in our grammar in the future.

Closed Issues

• 4390: POLYGON column not working/decoding with DBeaver
• 4321: Bad error message on DROP DATABASE when called on a database that initialized the sql-server
• 4319: Can't drop a database with a - in it
• 4404: Dolt doesn't respect the original order of CHECK, UNIQUE constraints in schemas
• 2673: Reorder of columns in UNIQUE INDEX when re-create table
• 4043: Add DOLT_DIFF_SUMMARY table function
• 2636: dolt diff --summary with cell-wise statistics
• 4270: Diff shows rows that have no diffs when a column has been added in the past.
• 4392: Bindvars not filled in for procedure calls
• 4391: sql_select_limit should not apply to subqueries
• 4361: concurrency problems with dolt_commit()
• 4373: dolt fetch upstream main fails on VPS with 1GB RAM
• 4220: Merge not showing correct statistics for rows merged
• 4329: Doltpy v2: Lib to can use Dolt from python
• 4328: Publish libs that can use Dolt
• 4340: Feature Request: Generate JSON_TABLE from table column reference
• 525: Column alias in where clause should be an error
• 1268: Analyzing self-referential foreign keys triggering an infinite loop

Latency

Current Default Format (__LD_1__)

New Format (__DOLT__)

Merged PRs

dolt

• 4372: go/dotlcore/sqle/writer: Handle UNIQUE KEYs overlapping with primary keys
  fixed #4369
• 4371: .github/workflows: ci-bats-*.yaml: Try to paper over flakey sql-server port selection in tests with BATS_TEST_RETRIES.
• 4370: proto,go: Add RepoPath field to remotesapi Request messages. Pass it along in remotestorage ChunkStore.
• 4366: Fix bug in colDiffs
  Fixes #4326
  • Fix a bug in splitDiffResultRow where a key existence is not being checked in a map. Specifically, ds.fromTo[i]] defaulted to 0 (the zero value for int) in the case where i did not exist as a key in ds.fromTo.
    
• 4364: JSON read perf
  Re: https://github.com/dolthub/go-mysql-server/pull/1270
  types_table_scan from 425 ms/q -> 134 ms/q, ~70% faster. The nightly benchmark should be around 2.5 (from 7.9).
• 4355: json_table tests
  Test PR for https://github.com/dolthub/go-mysql-server/pull/1261
• 4347: Add -A (--ALL) to dolt commit and CALL DOLT_COMMIT
-A adds all tables (including new tables) to the staged set before committing.
  Closes #4309
• 4301: have TempTableFilesDir() return err if .dolt is not found
  depends on https://github.com/dolthub/go-mysql-server/pull/1256
  If .dolt dir is removed during runinng sql-server. Any connections will fail and it will cause the server to stop.

go-mysql-server

• 1273: fix test ordering
  some tests on dolt were failing for JSON_TABLES, due to abiguous ORDER BY
  Companion PR: https://github.com/dolthub/dolt/pull/4355
• 1272: Removed exponential time complexity for foreign key analysis
  This fixes https://github.com/dolthub/go-mysql-server/issues/1268
  Foreign key analysis created acyclical trees that were traversed during query execution to emulate cascade operations. This meant that cyclical foreign keys were converted to an acyclical tree. Normally this isn't possible as cyclical trees are infinitely traversable, but MySQL has a depth limit of 15, which allowed us to materialize an acyclic tree with a maximum height of 15 nodes. This, however, lead to trees with an exponential number of nodes: roughly (number_of_fks)¹⁵ × 1.5 nodes in the tree. With just 3 foreign keys, we'd get a tree with roughly 22 million nodes, which would take forever to process.
  This PR completely changes the analysis step to now generate cyclical trees. In addition, depth checks are now properly implemented (during query execution rather than during analysis), being represented by a returned error once the depth limit has been reached. Interestingly, MySQL is supposed to process up to 15 operations (returning an error on the 16th), but cyclical foreign keys will error on the 15th operation. I think this is a bug in MySQL, but nonetheless the behavior has been duplicated here.
  I also updated the timestamp_test.go file to grab an unused port. This prevents test failures due to requesting an already-in-use port. Not related to this PR in particular, but it was annoying to deal with so I fixed it.
• 1271: enginetest: Add repro enginetest for unique key error
  repro for Dolt bug #4372
• 1270: No new context for every json SQL convert
  perf here: https://github.com/dolthub/dolt/pull/4364
• 1269: cleanup top-level directories
• 1261: use json_table in crossjoin that references table column on left
  Fix for: https://github.com/dolthub/dolt/issues/4340
  To use a column as the expr argument to json_table, the column must be part of a CROSS JOIN and has to be on the left side of a cross join (reference). Oddly, you cannot reference the column directly nor through a subquery (stackoverflow reference).
  As a result, this is partially an aliasing issue, which I do address in this PR.
  Added special column resolving rules for JSONTable for when its under a Join and subquery expression
• 1256: add check for chan is closed in Listener.Close()
  https://github.com/dolthub/dolt/pull/4301 depends on this PR.
  A test for it is added in https://github.com/dolthub/dolt/pull/4301
• 1253: Support Varchar(MAX)
  related to https://github.com/dolthub/dolt/issues/2261

Closed Issues

• 2261: Support VARCHAR(MAX)
• 4369: Dolt panic on duplicate insertion when a column is part of both a primary key and unique key
• 4326: UI issue for dolt diff and dropped columns
• 4235: Deleting database directory when a running server is using it panics
• 1441: dolt_diff_* tables can't handle dropped columns from the tables they describe.
• 4309: Add -A option to dolt commit to add all new and modified tables
• 4176: Collation not respected properly by LIKE and REGEXP_LIKE
• 4349: Is there a way to release momory？
• 1268: Analyzing self-referential foreign keys triggering an infinite loop

Latency

Current Default Format (__LD_1__)

New Format (__DOLT__)

Merged PRs

dolt

• 4348: go/libraries/doltcore/remotesrv: When uploading a file, pass table file details to the HTTP handler as query params in the URL, instead of stashing them in a shared map.
• 4336: sql-server: Let sql-server expose a remotesapi service to allow cloning, fetching and read replication from its databases.
  dolt sql-server gets a new configuration option, --remotesapi-port, which if supplied will cause the sql-server process to run a remotesapi service. The service will expose the databases in the sql-server for access as a dolt remote from a dolt CLI or sql-server process.
  Includes the following changes in the behavior of go/utils/remotesrv:
  • go/utils/remotesrv now logs with logrus
  • go/utils/remotesrv can serve gRPC and HTTP traffic over the same port.
  • go/utils/remotesrv gRPC service, when generating file server URLs, will echo the request's :authority header as the hostname component of the host URL component if -http-host is not supplied. This changes the old default, which was localhost.
• 4334: Fixes progress reporting bugginess
  Fixes #3149 and similar issues with dolt clone, dolt push, dolt pull.
  This PR largely simplifies the EphemeralPrinter code and adds a test to ensure that the printer leaves the cursor at the start of a line. Both the POSIX and Windows clear line implementations rely on that behavior which I believe was previously broken.

go-mysql-server

• 1259: Add sponsor button to GitHub
• 1258: Bug Fix: change unix_timestamp(<expr>) to return 0 when it can't convert expr to a date
  MySQL's implementation of unix_timestamp(<expr>) returns 0 when the expression can't be converted to a date and logs a warning. This PR changes Dolt to have the same behavior. It also includes a bug fix for ctx.ClearWarnings to correctly clear warnings.

  mysql> select unix_timestamp(1577995200);
  +----------------------------+
  | unix_timestamp(1577995200) |
  +----------------------------+
  |                          0 |
  +----------------------------+
  1 row in set, 1 warning (0.00 sec)
  mysql> select unix_timestamp("jason");
  +-------------------------+
  | unix_timestamp("jason") |
  +-------------------------+
  |                0.000000 |
  +-------------------------+
  1 row in set, 1 warning (0.00 sec)
  mysql> show warnings;
  +---------+------+-----------------------------------+
  | Level   | Code | Message                           |
  +---------+------+-----------------------------------+
  | Warning | 1292 | Incorrect datetime value: 'jason' |
  +---------+------+-----------------------------------+
  1 row in set (0.00 sec)
  

• 1257: allow compare decimals with different precision and scale
• 1252: Reimplemented LIKE expressions to add collation support
  Previously, we were converting the patterns from LIKE expressions into patterns that the standard regex parser would understand. Said parser does not support collations (as implemented in MySQL), therefore this is a custom-developed pattern matcher for LIKE expressions that fully supports collations.
  This is ONLY for patterns on the LIKE expression. We still need to implement full regex parsing to support REGEXP_LIKE.. Once we have REGEXP_LIKE completed, we may revert this back to the regex conversion process, but that will be no time soon.
  This also fixes https://github.com/dolthub/dolt/issues/4303
• 1244: use decimal type for arithmetic functions
  fixes https://github.com/dolthub/dolt/issues/4269
  sum will be either float64 or decimal.Decimal type, but return type is decided on the type of the child expression of sum expression. This causes some test results to be modified into int from float.
  the test results in decimal.Decimal type is hard to get the exact matching using its convert method, so result is checked in string format by using StringFixed() method of the decimal.Decimal result value.

vitess

• 190: Adding back support for using reserved keywords unquoted in InsertInto
  I previously excluded reserved_sql_id and only included sql_id because I didn't think reserved keywords should be allowed as identifiers in insert into columns without being quoted, but it appears we have tests in Dolt that rely on that (e.g. sql-diff.bats calls: INSERT INTO test (pk, int, string, boolean, float, uint, uuid) values ...) and I figured if we can support those without the quoting, it seems like a nice feature for customers, so I switched back to reserved_sql_id.
  It is confusing that we refer to identifiers as "reserved" keywords when they don't require backtick quoting though and would be nice to tidy that up in our grammar in the future.
• 189: Grammar fixes for customer issue using non-reserved keywords in insert into statements
  A customer reported an issue on Discord with using insert into t(pk, comment) values (1, 1). Comment is a keyword, but not a reserved keyword, so this is valid syntax, but our parser was failing on it.
  A few changes to tidy up:
  • Updated tests to trigger the customer reported parsing bug when an (unquoted) keyword is used in a list of columns in an insert into statement.
  • renamed column_name_safe_reserved_keyword to column_name_safe_keyword, since the contents are keywords, but are not "reserved" keywords that need to be backtick quoted
  • fixed a bug with recursion for insert into column names (ins_column_list rule)
  • removed avg,count,sum,min,max from lists of reserved keywords, since they are not reserved in MySQL and can parse correctly in tests without being backtick quoted.
  • added some more comments to try and make the intent of different grammar rules clearer.
• 188: Support Varchar(MAX)
  related to https://github.com/dolthub/dolt/issues/2261
• 187: CTEs on unions parse

Closed Issues

• 4055: Can't migrate dolthub/restaurants on dev
• 4349: Is there a way to release momory？
• 3149: Upload units do not properly update
• 4342: Better build instructions
• 4343: Problem with flatbuffers submodule
• 1232: Case-sensitive comparison of string literal

Latency

New Format (__DOLT__)

Merged PRs

dolt

• 4335: cmd user/password arguments always override yaml config
  Closes: https://github.com/dolthub/dolt/issues/4333
• 4324: go/store/prolly/{message, tree}: Reduce memory footprint of message.ItemAccess, tree.Node
  • Refactored message.ItemAccess (previously ItemArray) to decrease memory footprint from 48 bytes to 10.
  • Refactored tree.Node to decrease memory footprint from 152 bytes to 56
  • tree.nodeCache now caches tree.Node instead of chunks.Chunk. This eliminates decode time for cache hits.
  • Reworked lock striping to tree.nodeCache to improved access latency.
• 4323: go/utils/remotesrv: Make remotesrv implementation into a library which can be used from dolt proper.
  Adds some functionality to remotesrv:
  • -http-port and -grpc-port can be the same, in which case there is only one listener and both gRPC and file server traffic are served over the same port.
  • the grpc server will echo the incoming request's :authority header, with the supplied -http-port, if no -http-host is supplied. Changes the default behavior of remotesrv to be that, rather than generating URLs with an authority component containing localhost as the hostname.
• 4316: Trim trailing nil fields from tuples
  This PR enforces a canonical form for tuples. Any tuple that has trailing nil field(s) will be truncated to the first non-nil field. This fixes #4270.
• 4314: go/store/prolly: Bound maximum pending writes to prolly.MutableMap

go-mysql-server

• 1255: fix numberTypeImpl.SQL() to convert from any type
  The SQL method for numberTypeImpl expects v to already be the right type for the baseType, this is not always the case.
  CASE statements have an error guard for this behavior, which shouldn't be there.
  There are likely other types that have SQL() methods that don't work for all types.
  Changes:
  • added type switch for types that are unconvertable to number
  • remove analyzer rule 'validateCaseResultTypesId`
    Closes https://github.com/dolthub/dolt/issues/4306
• 1252: Reimplemented LIKE expressions to add collation support
  Previously, we were converting the patterns from LIKE expressions into patterns that the standard regex parser would understand. Said parser does not support collations (as implemented in MySQL), therefore this is a custom-developed pattern matcher for LIKE expressions that fully supports collations.
  This is ONLY for patterns on the LIKE expression. We still need to implement full regex parsing to support REGEXP_LIKE.. Once we have REGEXP_LIKE completed, we may revert this back to the regex conversion process, but that will be no time soon.
  This also fixes https://github.com/dolthub/dolt/issues/4303
• 1249: UpdateJoin bug fixes
  Bug fixes for https://github.com/dolthub/dolt/issues/4288
  • qualifying check constraint expressions when multiple tables are being updated
  • support finding UpdatableTables through ProcessIndexableTable and ProcessTable nodes
  • erroring out when a table updater isn't available for a SetField expression, instead of silently dropping it
    This does not address another UpdateJoin bug I just found while testing: https://github.com/dolthub/dolt/issues/4304. Mentioning that one here so we have this context connected when we go to fix it.
    I verified that Dolt's enginetest suite runs correctly with these changes.

Closed Issues

• 4269: sum() of decimal types with negative numbers is incorrect
• 4333: Support --user and --password sql-server args even if their is a config.yaml
• 4270: Diff shows rows that have no diffs when a column has been added in the past.
• 4306: Panic on "conflicting" result types in IF() function
• 4288: Query to revert a single column fails
• 4303: Double backslash results in unexpected LIKE comparison

Latency

Current Default Format (__LD_1__)

New Format (__DOLT__)

Merged PRs

dolt

• 4315: create db when no current db is set should use the dbs available in session
  If there is no current database selected, use databases that are available within the session and use the common format for those databases.
• 4302: Changes to SQL output, including row summary, timing and empty set output
  This changes the SQL shell to more closely match the MySQL shell by adding an "empty set" output, row count, and timing info.
  It also gets rid of the use of pipelines for printing, replacing with SqlRowWriter implementations.
• 4296: add --author flag to merge
• 4267: Add dolt diff --skinny flag
  Closes #790
  Adds a --skinny flag to dolt diff, which only shows primary keys and columns with changes.
  Note that writeDiffResults has been split into 2 functions - writeFilteredRows and getDiffRows. The former is responsible for printing the rows, while the latter function collects the relevant columns, depending on if the --skinny flag is active.
  Also note that in the event of row addition/deletions, all columns will be included when the --skinny flag is active.

go-mysql-server

• 1250: stripNodes in subquery block apply_hash_lookup optimization
  A HashLookup requires two rules, 1) converting a JOIN -> SUBQUERY_ALIAS => JOIN => CACHED_RESULT(SQ), 2) converting the CACHED_RESULT into a HASHED_LOOKUP.
  Prevent StripRowNode from breaking the pattern matcher for the first step. Other nodes can still interrupt the rule, but we have a test now at least.
• 1249: UpdateJoin bug fixes
  Bug fixes for https://github.com/dolthub/dolt/issues/4288
  • qualifying check constraint expressions when multiple tables are being updated
  • support finding UpdatableTables through ProcessIndexableTable and ProcessTable nodes
  • erroring out when a table updater isn't available for a SetField expression, instead of silently dropping it
    This does not address another UpdateJoin bug I just found while testing: https://github.com/dolthub/dolt/issues/4304. Mentioning that one here so we have this context connected when we go to fix it.
    I verified that Dolt's enginetest suite runs correctly with these changes.
• 1248: feat:(sql function) support last_insert_id with expr
  mysql support parameter for LAST_INSERT_ID(expr), it is useful in upsert query when we want to get the updated recored id
  ref: https://dev.mysql.com/doc/refman/8.0/en/information-functions.html#function_last-insert-id
• 1246: adding LEAD() window function
  Partial fix for: https://github.com/dolthub/dolt/issues/4260

Closed Issues

• 4306: Panic on "conflicting" result types in IF() function
• 790: dolt diff --skinny: Only show primary keys and columns with changes. Hide other columns.
• 4311: Diff shows rows that have no diffs when a column has been added in the past.
• 4290: CALL DOLT_CHECKOUT() on an existing branch give "branch already exists" error
• 4289: dolt_diff_<table> only works if to_commit is in your current branch
• 4307: Feature Request: Way to create db in new format via sql
• 4295: Specify explicit author in dolt_merge procedure

Merged PRs

dolt

• 4300: SQL: DOLT_CHECKOUT: Support checking out a branch which does not have an associated working set.
• 4275: Point lookup optimizations

go-mysql-server

• 1246: adding LEAD() window function
  Partial fix for: https://github.com/dolthub/dolt/issues/4260
• 1245: adding LAST_VALUE() window function
  Partial fix for: https://github.com/dolthub/dolt/issues/4260
• 1239: More point range optimizations
  SQL range now exposes whether a range is a point lookup, so the integrator does not have to check for itself.
• 1226: Better support for recursive CTEs with limit, filter and sort.

Closed Issues

• 4260: Support LAST_VALUE() and LEAD() window functions
• 4209: Fix a panic on import