Merged PRs

dolt

• 6600: Separate out log arg parsers
  Creates a separate arg parser for dolt log and dolt_log to avoid supporting unneeded flags.
• 6599: Bump gopkg.in/yaml.v3 from 3.0.0-20200313102051-9f266ea9e77c to 3.0.0 in /integration-tests/transactions
  Bumps gopkg.in/yaml.v3 from 3.0.0-20200313102051-9f266ea9e77c to 3.0.0.
• 6598: Move storage format version to only be displayed with dolt version --verbose
dolt version displayed the storage format version by default, but this information is less relevant now that almost all Dolt databases have been migrated to the new storage format. This change adds a new --verbose / -v flag that customers can specify if they do want to see the storage format version for a repository.
  Fixes https://github.com/dolthub/dolt/issues/6528
• 6568: updates dolt log to use sql backend
  Updates dolt log to use sql backend to generate results.
  Related: https://github.com/dolthub/dolt/issues/3922

go-mysql-server

• 1981: Make the IntSequence test function consistently use int64 for its generated values
  The previous implementation had an issue where it assumed the type used in the received IndexLookup, but this type can actually depend on exactly how the lookup was generated (and whether the bounds value was parsed from the query or generated internally.) This caused a panic if it was used in Lookup joins.
  This makes no such assumptions and adds extra tests.
• 1980: FreeBSD sockstate workaround
  Close #2138
• 1979: Refactor event resolving
  Inline loadEvents and move event resolving out of execution time.
• 1976: Refactor prepareds to cache AST
  Prepared statements were rewritten recently to cache a query string
  with marked bind variable locations. Executing prepared statements
  would generate a full SQL query string for re-parsing, binding, and
  analysis. This update caches the AST. Executing a prepared statement
  starts at binding an AST. Bind variables will be substituted in the
  process of converting AST expression to plan expressions.
  GMS bump: https://github.com/dolthub/dolt/pull/6593

Closed Issues

• 6446: dolt_patch() table function should consider filters for schema or data only changes
• 2138: Can't build from source on FreeBSD
• 6528: dolt version should not care about format anymore
• 6442: Automatically merge changes that enlarge varchar fields to text types
• 6486: Allow pattern-matching in replicate_heads variable

Latency

Merged PRs

dolt

• 6595: Feature: Allow * wildcard in @@dolt_replicate_heads
  This change allows customers to use * as a wildcard in the @@dolt_replicate_heads system variable to match zero or more characters in a branch name. For example, to replicate the main branch and all branches that start with feature, the following configuration can now be applied:

  dolt config --local --add sqlserver.global.dolt_replicate_heads main,feature*
  

  Documentation: https://github.com/dolthub/docs/pull/1698
  Fixes: https://github.com/dolthub/dolt/issues/6486
• 6562: Bug fix: Building secondary prefix indexes from existing data
  When building a secondary index from existing table data and the secondary index contains a prefix of a TEXT/BLOB column, we don't currently dereference the TEXT/BLOB pointer to get the content. Instead, we take the TEXT/BLOB pointer address and incorrectly put that into the secondary index. This manifested as errors in collation sorting, since one string was not a valid, encoded string and was instead a pointer address. Additionally, we weren't applying the prefix to the content for any existing data when building a prefix index – i.e. any VARCHAR data was inserted in full to the prefix index without trimming.
  It's difficult to test indexes directly, so I updated the index validation code in our test suite to support validating prefix indexes. I added some more tests for prefix indexes in GMS, too: https://github.com/dolthub/go-mysql-server/pull/1971
  Fixes https://github.com/dolthub/dolt/issues/6520
• 6517: Automatically merge schema changes that widen VARCHAR fields to TEXT
  This enhancement allows more schema changes to merge cleanly (instead of being reported as a schema conflict) when we can determine that a type is being widened and all existing data is guaranteed to be safe to convert. This includes:
  • VARCHAR → TINYTEXT, TEXT, MEDIUMTEXT, LONGTEXT
  • CHAR → TINYTEXT, TEXT, MEDIUMTEXT, LONGTEXT
  • BINARY → TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB
  • VARBINARY → TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB
    Trying to merge a change to a VARCHAR column larger than 255 characters to make it a TINYTEXT column will still generate a schema conflict, since TINYTEXT has a maximum size of 255 characters.
    One of the more interesting conversions is converting to VARBINARY(N) or TINYTEXT to BINARY(N). MySQL right-pads all BINARY(N) values with null bytes, up to N, which requires us to call Type.Convert as part of remapping values in a schema merge, and also to rebuild secondary indexes in some cases.
    Fixes: https://github.com/dolthub/dolt/issues/6442

go-mysql-server

• 1978: Fix table lookup bug for nil asOf
  Edit: Show keys for info schema table should work the same way as MySQL now.
  Prev: This error message is not correct, MySQL finds the table and does not error. But this at least reverts the behavior to what Dolt did before the name resolution refactors.
• 1976: Refactor prepareds to cache AST
  Prepared statements were rewritten recently to cache a query string
  with marked bind variable locations. Executing prepared statements
  would generate a full SQL query string for re-parsing, binding, and
  analysis. This update caches the AST. Executing a prepared statement
  starts at binding an AST. Bind variables will be substituted in the
  process of converting AST expression to plan expressions.
  GMS bump: https://github.com/dolthub/dolt/pull/6593
• 1975: Union NULL types cast to same type as neighbor
  3+-way unions are subject to a bug where a NULL projection will be cast to a char type, and then have a type clash with the subsequent table relation. Rearranging the query to have the NULL projection last appears to fix the bug because the convert cast is moved to a position where it cannot have a type clash with a table relation to the right.
• 1974: Re-implement generated stored columns with new analyzer
  Testing revealed a bug in explicit DEFAULT values in insert statements, added a skipped test for same.
  Also rewrote column default tests as scripts.
• 1945: server: Add a warning message at startup if the server is run without secure_file_priv being set to a non-empty value.

Closed Issues

• 6442: Automatically merge changes that enlarge varchar fields to text types
• 6486: Allow pattern-matching in replicate_heads variable
• 6497: LAST_INSERT_ID() always returns 0
• 6520: Panic when inserting into prefix index on TEXT column
• 6462: go.mod causes go install to fail (must not contain directives ... interpreted differently than if it were the main module)
• 6224: When executing a query in SQL, the error message is unrelated to the SQL statement.

Latency

Merged PRs

dolt

go-mysql-server

• 1973: Fixed behavior for CONVERT(x USING y)
  The CONVERT(x USING y) expression allows us to convert between character sets. When a string does not have a representing rune in the target character set, it should be replaced by a question mark. This is consistent with MySQL. Our previous behavior logged an error, which is valid in many scenarios, but not for this particular expression.
• 1970: sql: ExternalStoredProcedureDetails: Add the ability to mark stored procedures read-only.
• 1969: Added a QueryEngine abstraction to engine tests to allow us to run engine tests on a running server
  Implementation is partially working. Lots of errors when used in tests, some of which are real and some of which are issues with test definitions. Will keep working on this over time, but want to get the necessary refactoring in before continuing.

Closed Issues

Latency

Merged PRs

dolt

• 6584: Migrate the dolt_procedures table whenever stored procedures are used
  Fixes: https://github.com/dolthub/dolt/issues/6579
  Before, we were only migrating the table when a new stored procedure was added. This means that any branches that predate our recent change to dolt_procedures would be broken until a new stored procedure was added. This fixes that by checking to see if the table needs to be migrated whenever the stored procedures are listed or used, not just when they're added.
• 6577: Pull indexed system table functions into their own test suite.
  This was supposed to be part of https://github.com/dolthub/dolt/pull/6509 but I the push containing this commit was rejected and I didn't notice. It incorporates the testing feedback @max-hoffman suggested.

go-mysql-server

• 1969: Added a QueryEngine abstraction to engine tests to allow us to run engine tests on a running server
  Implementation is partially working. Lots of errors when used in tests, some of which are real and some of which are issues with test definitions. Will keep working on this over time, but want to get the necessary refactoring in before continuing.
• 1968: Missed cases where replicaBinlogController is needed during building
• 1967: Join aggregation projections subject to dropped refs
  Aggregation intermediate projections need to output all inputs for aliases in the target projection. The way we were tracking column source column references did not take into account that two join tables can have the same column name. Now we include the table name for alias dependency tracking.
  close https://github.com/dolthub/dolt/issues/6578
• 1965: Add error test for ambiguous column name query
  close https://github.com/dolthub/dolt/issues/6395
• 1964: Skip subquery pushdown when no subquery
  This will have a small perf impact.
• 1962: Name res cleanup, docs and inline rules
  Move bespoke rules into planbuilder. Functions are only ran for their necessary nodes. Also adds copyright headers.
  Requires some Dolt changes: https://github.com/dolthub/dolt/pull/6567
• 1955: adding catalog table function interface

vitess

• 265: upgraded YAML library
• 262: Prepared query statement round trip
  Changes to aliases and json_table to make the AST string formatters round-trip.
  Biggest change to tests is that useSelectExpressionLiteral seems to behave differently.

Closed Issues

• 6579: Stored Procedure Panic
• 6578: Schema types and collations shifting
• 6543: Fulltext and autoincrement together causes Panic
• 6541: Adding fulltext index to a column with only null values panics.

Merged PRs

dolt

• 6570: Bug fix: Correctly detect uncommitted changes for dolt_pull when roots have different feature versions
  When pulling between two Dolt databases that have different feature versions, dolt_pull() can mistakenly interpret the difference in root value hash as being uncommitted changes. This change updates dolt_pull() to use the existing actions.RootHasUncommittedChanges function and changes that function to account for differing feature version when determining if roots have uncommitted changes.
• 6537: Added a --branch parameter to Dolt global arguments on the command line.
  Addresses https://github.com/dolthub/dolt/issues/6273
  Thank you, @timsehn and @macneale4, for reviewing my previous PR. I'm creating a new PR to incorporate the changes suggested by @macneale4.
  Please let me know if anything more need to be added!
• 6514: adding query-diff cli command
  Companion PR: https://github.com/dolthub/go-mysql-server/pull/1955

go-mysql-server

• 1963: Fixed type hashing for Full-Text
  Fixes the reopened issue: https://github.com/dolthub/dolt/issues/6543
  Not all types were covered by the default case.
• 1955: adding catalog table function interface

Closed Issues

• 6273: Create flag for specifying the initial branch for a sql client.
• 6395: duplicate column names in a table alias should be prevented.
• 6543: Fulltext and autoincrement together causes Panic

Latency

Merged PRs

dolt

• 6531: Empty Servers allow dolt sql connections without arguments
  WIth this change, starting sql-server will result in creating a .dolt/sql-server.lock file in the --data-dir (CWD by default). This file will be used for two purposes:
  1. Ensuring another dolt sql-server is not started with the same data dir.
  2. Allowing the dolt sql command to connect to it. See related issue: https://github.com/dolthub/dolt/issues/6508
• 6522: update dolt_log table function to support more features
  Updates the dolt_log table function to support more features, e.g. accepting more than 2 revisions and accepting table names. This change is motivated by the ongoing CLI migration (https://github.com/dolthub/dolt/issues/3922) as the updated table function will help migrate the dolt log command.
• 6509: Have DOLT_PATCH() system table function implement an index on data change vs schema change.
  This allows for efficient queries of the form:
  SELECT * FROM DOLT_PATCH(...) WHERE diff_type = "schema"
  Dolt will only compute the schema changes and avoid computing the data changes, which may be expensive.
• 6454: go/libraries/doltcore/sqle/cluster: Add replication of user and grant changes from primary to standby replicas.

go-mysql-server

• 1963: Fixed type hashing for Full-Text
  Fixes the reopened issue: https://github.com/dolthub/dolt/issues/6543
  Not all types were covered by the default case.
• 1961: engine: Make ReadOnly a toggleable atomic.Bool.
• 1960: Fixed Full-Text NULL handling and ALTER TABLE case
  Fixes https://github.com/dolthub/dolt/issues/6540 and https://github.com/dolthub/dolt/issues/6541
• 1957: Fixed relevancy ordering for Full-Text
  Fixes https://github.com/dolthub/dolt/issues/6530
• 1952: GMS tests have to resolve defaults
  Force GMS to resolve column defaults, fix bugs exposed by additional testing. Dolt enginetests pass locally.

Closed Issues

• 6543: Fulltext and autoincrement together causes Panic

This release contains backwards incompatible changes:

• Primary key change to the dolt_status system table. Previously the primary key of this table only included the table_name column. The primary key now also contains the staged and status columns to account for the same table name appearing multiple times in dolt_status (i.e. a table that was staged and then additional unstaged changes being made).

Per Dolt’s versioning policy, this is a minor version bump because of the schema change for the dolt_status system table.

Merged PRs

dolt

• 6550: Silence branch create
  Previously dolt branch foobar HEAD would print a statement that HEAD is an invalid branch name. It would succeed though, and return an exit status of 0. So tests passed.
  This is no longer the case.
• 6542: Clean up sql command's help message
• 6474: dolt_status system table shows if merge is active when there are no conflicts
  Breaking change: Updates primary keys of dolt_status system table. We originally just had table_name in the primary key, but a table can already show up more than once in dolt_status. status and staged should also be part of the primary key.
  This PR also adds a "merged" status row to the dolt_status system table when a schema conflict has been resolved but that table has not yet been committed, which fixes https://github.com/dolthub/dolt/issues/6473

go-mysql-server

• 1959: Bug fix: Prevent panic when reading non-existent user and system vars
  Fixes https://github.com/dolthub/dolt/issues/6546
  We have enginetests that cover both of these queries, but because we don't run them with the full wire request/response processing code, we didn't catch these bugs during testing. Happy to add more tests here if there are suggestions, but I think the right way to test both of these is to get our existing test suite running over a SQL server connection, instead of just using the internal interfaces to the engine (i.e. https://github.com/dolthub/dolt/issues/3646), which Zach started on last week.
• 1958: Don't parse queries twice
  Local profile for oltp_point_select (query with smallest time spent in execution) is 5-15% speedup. Impact on queries with longer-runtime will be smaller, proportional to the fraction of time spent in analysis vs execution.
  results here: https://github.com/dolthub/dolt/pull/6547#issuecomment-1686795603
• 1957: Fixed relevancy ordering for Full-Text
  Fixes https://github.com/dolthub/dolt/issues/6530
• 1956: Removed unused AutoIncrementGetter interface
  Related to https://github.com/dolthub/dolt/issues/6543.
  The AutoIncrementGetter was moved from the editor to the table years ago, however the interface remained. I debated deleting this during my Full-Text implementation, however decided to leave it. Now, we've encountered an error with an integrator making use of the interface, so it has been removed here and the interface was moved into the integrator since it's an internal detail now.
• 1954: Lateral join uses prepend row on RHS
• 1951: Fixed collation display in SHOW CREATE TABLE
  Originally, we didn't display the collation in SHOW CREATE TABLE when the collation was the default collation. Now, it doesn't display it if it's the same as the table collation, which mimics MySQL's behavior.
• 1938: Create interface for indexible tables in IndexedTableAccess
  Currently, only ResolvedTables are allowed to have indexes. There exists an interface, sql.IndexAddressable, which any node or table can implement in order to be a candidate for index-based optimization. But in practice, implementing that interface won't actually do anything because the IndexedTableAccess struct explicitly requires a ResolvedTable.
  This PR replaces the ResolvedTable field in IndexedTableAccess with a new interface tentatively called TableNode, although a more specific name would probably be better.
  In order for a node to be used for index-based optimization, it must implement this interface, and the table returned by the UnderlyingTable method must implement sql.IndexAddressable

Closed Issues

• 6473: dolt_status system table should show if merge is active
• 6540: Can't add a fulltext index to an existing TEXT type column.
• 6530: order by a fulltext match column does not work
• 6546: Dolt sql-server panics on unknown variable

Latency

Merged PRs

dolt

• 6544: Moved AutoIncrementGetter to an internal interface
  Fixes https://github.com/dolthub/dolt/issues/6543.
  Companion to https://github.com/dolthub/go-mysql-server/pull/1956.
  The AutoIncrementGetter was moved from the editor to the table years ago, however the GMS interface remained. We had an internal reliance on the interface, even though GMS does not use it at all, so the interface has been moved to be an internal interface only. This fixes the panic as seen in the issue.

go-mysql-server

• 1950: Honor precision for datetime and timestamp, default to 0 (no fractional seconds)

Closed Issues

• 6543: Fulltext and autoincrement together causes Panic
• 6257: Correctness Issue with Exists subquery
• 6394: Valid join query erroneously reported as ambiguous
• 5789: JOIN USING support

This release contains backwards incompatible changes:

• TIMESTAMP and DATETIME columns now honor fractional second precision (FSP), from 0 to 6 decimal places. Previous releases of Dolt ignored FSP for these types and always used 6 decimal places. The column type DATETIME or TIMESTAMP both now have an implicit FSP of 0 digits (truncated to nearest second), which matches the MySQL behavior. To get the old default of 6 decimal places of FSP, use the types DATETIME(6) or TIMESTAMP(6). Existing tables with DATETIME or TIMESTAMP columns are not affected. They store 6-digit precision times before and after this change.

Per Dolt’s versioning policy, this is a minor version bump because the same CREATE TABLE statement with a DATETIME or TIMESTAMP column that specify a fractional second precision other than 6 will now produce a different schema than before.

Merged PRs

dolt

• 6532: When checking out a branch, consider it clean if the working set differs from the head only by feature version.
  Fixes https://github.com/dolthub/dolt/issues/6524
  RootValue is backed by a flatbuffer, but access to the flatbuffer is hidden behind interfaces. We check if two RootValues are equal by checking the RootValue's hash. This makes it awkward to check to see if two RootValues differ only by a single field.
  The solution here is both simple and noninvasive: we create modified root values with the same feature version and hash them again.
  This also adds an escape hatch when creating new branches, since we know that the new branches will be clean. We only do this extra check when checking out an existing branch.
• 6529: add minver tag to YamlServerConfig
• 6526: dolt version only prints the dolt version if there is no dolt database in the directory.
  Remove "no valid database in this directory" from dolt version in the case of no dolt database found.
• 6525: Removed Full-Text tables from 'schema show'
• 6518: Support datetime and timestamp precision
  Fixes https://github.com/dolthub/dolt/issues/6503
  In addition to datetime precision support, this PR also significantly simplifies the process of deserializing and caching schemas. Previously on load, we always did two round trips to dereference table's schema hash, then re-hash the object, then serialized it into an intermediate in-memory format and back to the schema.Schema, before returning it. The new code eliminates this redundancy.

go-mysql-server

• 1950: Honor precision for datetime and timestamp, default to 0 (no fractional seconds)
• 1948: Name resolution refactor
  Accumulation of:
  • https://github.com/dolthub/go-mysql-server/pull/1927
  • https://github.com/dolthub/go-mysql-server/pull/1932
  • https://github.com/dolthub/go-mysql-server/pull/1936
  • https://github.com/dolthub/go-mysql-server/pull/1939
  • https://github.com/dolthub/go-mysql-server/pull/1943
    Name resolution is interleaved with converting AST->plan nodes. Prepared statements use ParsedQuery and AST round tripping to analyze fresh query strings for every invoke. Various bug fixes.
• 1930: leave aliases in projection
  We convert expression.Alias into expression.GetField with the name replaced in the top-level projection.
  When aliasing two different columns with the same name, we fail to properly distinguish them and end up rewriting the GetField indexes for both columns to be the same; this leads to incorrect results.
  A simple fix appears to be simply allowing the top-level projection to remain as an expression.Alias.
  This fix does not work for prepared statements in the old name resolution path.
  fix for: https://github.com/dolthub/dolt/issues/6455
• 1907: support joins on using syntax
  Relevant MySQL docs
  This PR adds an implementation for the USING clause during joins.
  It is mostly equivalent to creating a conjunction over equality statements over each column in an ON clause.
  Additionally, this properly supports NATURAL LEFT/RIGHT JOINs.
  So we now have
  • ... JOIN ... USING (...)
  • ... LEFT JOIN ... USING (...)
  • ... RIGHT JOIN ... USING (...)
  • ... NATURAL JOIN ...
  • ... NATURAL LEFT JOIN ...
  • ... NATURAL RIGHT JOIN ...
    fix for: https://github.com/dolthub/dolt/issues/5789

Closed Issues

• 6524: Dolt checkout fails incorrectly when there are working set changes
• 6503: Dolt datetime defaults to precision 6, mysql defaults to 0
• 6455: nil table RIGHT_JOIN correctness bug
• 6305: XCA Tool shows no records when Dolt is used instead of MySQL

Latency

Merged PRs

dolt

• 6523: Changing primary index to use the Rewriter function, instead of Mutate
  Ensures that the tuple descriptors use the correct types, and don't do address validation on unnecessary data.

go-mysql-server

• 1949: Revert "Merge pull request #1944 from dolthub/zachmu/timestamp"
  This reverts commit ca69015243946a0a42a0ec6a225da9c4551a0e12, reversing changes made to c11b504a18c48671c29c1494460f662e1f13f712.
• 1944: Changed datetime and timestamp types to honor precision and default to 0 digit precision
  This matches the MySQL behavior.
  Partial fix for https://github.com/dolthub/dolt/issues/6503
• 1915: ANSI_QUOTES support for go-mysql-server
  Now that our parser has support for parsing in ANSI_QUOTES mode and treating double quotes as identifier quotes, this PR starts using that support when we parse queries. The main changes in this PR are:
  • a new SqlMode type to make it easier to load SQL_MODE and inspect enabled modes.
  • tracks SQL_MODE when a procedure/trigger/view is created.
  • passes ParserOptions to the parser, to enable parsing in ANSI_QUOTES mode.
  • connects the Vitess error logger to GMS' error log, so that errors in Vitess aren't swallowed (currently if a statement in a ComPrepare message doesn't parse, GMS doesn't log anything).
    Dolt has a similar, corresponding PR: https://github.com/dolthub/dolt/pull/6465
    Related to: https://github.com/dolthub/dolt/issues/6305

vitess

• 264: Revert https://github.com/dolthub/vitess/pull/261
  Change in description breaks prepared statements that send a value with character data above the max size of a CHAR column (256 bytes).
  Fixes https://github.com/dolthub/dolthub-issues/issues/489

Closed Issues

Latency

This release contains backwards incompatible changes:

• schema changes to the dolt_schemas and dolt_procedures system tables to allow @@SQL_MODE to be stored for schema fragments, as part of support for the ANSI_QUOTES parsing mode. Existing databases will have their system tables automatically upgraded to the new schema.
• The stored procedures for dolt_clone() and dolt_remote() no longer support for the optional flags related to AWS (--aws-region, --aws-creds-type, --aws-creds-file, --aws-creds-profile). There is no longer a mechanism to set these values on a running server.

Per Dolt’s versioning policy, this is a minor version bump because of the schema changes for these two system tables. For safety, this release also updates the Dolt feature version to version 4, meaning that once a database is upgraded by using this release, interacting with that database through the dolt CLI requires version 1.11.0 or higher.

Merged PRs

dolt

• 6515: Prevent dolt sql from allowing LOAD_FILE from directories outside of working directory
• 6513: dolt sql-server: Add a system_variables: key to config.yaml, which allows setting global system variables.
  For example, this can be used to set secure_file_priv when the server starts up. That variable is non-dynamic, so it cannot be set with SET @@GLOBAL...
• 6511: Bug fix: panic in schema merge when schema contains BLOB columns
  When processing a schema merge, we need to update existing, stored data in the primary index. We weren't updating the value descriptor in the prolly map to the new version from the changed schema, which started causing a problem with BLOB/TEXT types since validation checked that the value stored in those fields was a valid address pointing outside of the table to the real data.
  Related to https://github.com/dolthub/dolt/pull/6496 – the reason that PR wasn't able to cleanly repro the exact failure was because the merge was getting short circuited, since the right and left sides weren't both making changes, so essentially the merge was getting fast-forwarded instead.
• 6506: Drop the aws flags from the dolt_remote() procedure
  Apparently there are no tests for these flags because the tests don't blow up if I remove them. We still allow the flags in the CLI context though.
• 6505: Create SECURITY.md
• 6485: Fixed Full-Text merge bug, removed pseudo-index tables from view
  This does two primary things. First, it removes the pseudo-index tables from dolt status, the status table, merge, and diff outputs. This allows them to match the other index tables, which don't show up in those views anyway. Second, it fixes a bug during merge where the pseudo-index tables are the only ones that have been renamed. The setup is very unlikely, but it's been fixed nonetheless.
• 6465: ANSI_QUOTES support
  Adds support for honoring the ANSI_QUOTES SQL mode. When this SQL mode is enabled, anything in double quotes is assumed to be an identifier, and not a string literal.
  Corresponding GMS PR: https://github.com/dolthub/go-mysql-server/pull/1915
  Fixes: https://github.com/dolthub/dolt/issues/6305

go-mysql-server

• 1942: Full-Text Fixes Pt. 3
• 1941: Bug fix for JSON_ARRAY function with binary arguments
  Binary args are now treated as character strings
  Also added a testing path to ScriptTests that let you inject Vitess bindvars to exercise more server logic
  Fixes https://github.com/dolthub/go-mysql-server/issues/1855
• 1940: Add SECURITY.md.
• 1937: Remove do-nothing logic from pushdown.go
  This code is for an optimization that "pushes" filters deeper into the tree so that they're adjacent to the tables they modify.
  I've simplified the logic in two ways:
  • Removed handling of IndexedTableAccess nodes. This analysis pass used to run after these nodes were generated, but now runs before, so this code path will never be hit. Removing this logic makes it easier to make future changes to IndexedTableAccess
  • Removed the withTable subfunction, which never actually does anything, because the only table it will attempt to assign to a ResolvedTable is the table already on the node. This was likely leftover from a previous cleanup.
• 1935: Allow timestamps when encoding json
  As reported on discord, Nautobot, through DJango, puts time stamp data into a json object. This fails because:

  db> select JSON_OBJECT("a", Now());
  unsupported type: time.Time
  

  This change enables the encoding of a time stamp into a string.

vitess

• 264: Revert https://github.com/dolthub/vitess/pull/261
  Change in description breaks prepared statements that send a value with character data above the max size of a CHAR column (256 bytes).
  Fixes https://github.com/dolthub/dolthub-issues/issues/489
• 263: Add SECURITY.md.
• 261: Preserving bind var type for sqltypes.Char params
  When a caller executes a prepared statement, they send the parameters as well as parameter type information to the sql-server. Vitess populates bindvars type information that go-mysql-server uses to execute the query. Vitess is currently converting many SQL types to VARBINARY, including CHAR, which makes it look like binary data was sent and not a char string, like the caller indicated.
  This change stops converting sqltypes.Char bind var type info into VARBINARY, which enables go-mysql-server to see that a char string was passed. Without this type information, go-mysql-server doesn't seem able to differentiate between legitimate binary data sent by the client versus a character string. Since these types need to be handled differently, and we can't assume that all VARBINARY types can be converted into strings, it seems like we need to respect the SQL type the caller indicated for the bind var and pass that up to integrators.
  It seems like this change could be helpful for other SQL types, too, but I wanted to start with just sqltypes.Char to see if this approach causes any other issues.
  Fixes: https://github.com/dolthub/go-mysql-server/issues/1855
  Related go-mysql-server PR: https://github.com/dolthub/go-mysql-server/pull/1919
  Related Dolt PR: https://github.com/dolthub/dolt/pull/6441

Closed Issues

• 6305: XCA Tool shows no records when Dolt is used instead of MySQL
• 6508: Specify database name on running docker image dolt-sql-server
• 6498: Can't get hosted replication to work. And got a Panic after trying for a while.
• 6489: json_table prepared errors

Merged PRs

dolt

• 6492: Make gen-zsh command write to stdout
• 6487: feat: adding zsh plugin
  This isn't perfect, but it's a starting point. What would be better:
  • Create a separate repo for just this script (dolthub/zsh) for easier installation
  • Change gen-zsh to output the script to stdout like gh, op, and other popular tools
    You can use this via a for loop in zinit like:

  zinit wait lucid for \
  'https://github.com/iloveitaly/dolt/blob/zsh-plugin/dolt.plugin.zsh'
  

go-mysql-server

• 1935: Allow timestamps when encoding json
  As reported on discord, Nautobot, through DJango, puts time stamp data into a json object. This fails because:

  db> select JSON_OBJECT("a", Now());
  unsupported type: time.Time
  

  This change enables the encoding of a time stamp into a string.
• 1933: Fixed case sensitivity bugs in various statements
• 1928: Generated stored columns prototype
  Looking for feedback. The approach is a little bit wonky: it uses the same code paths as column defaults since they behave so similarly, but this has some weird consequences:
  • For generated columns, we fill in the column.DefaultValue field with the Generated expression
  • In various places, we now have to consider whether to use either column.DefaultValue, or column.Generated
    Overall I think I'm favorable on this approach, but keeping the two values more separate might be better for maintainability, not sure. Thoughts?

Closed Issues

• 6489: json_table prepared errors

This release contains backwards incompatible changes:

• Bug Fix: Error on merges that produce schema conflicts when @@autocommit is enabled – A bug in SQL transaction @@autocommit support was not throwing an error when @@autocommit was enabled and a call to dolt_merge() returned schema conflicts.

Per Dolt’s versioning policy, this is a minor version bump because calls to dolt_merge() that previously completed without error (and reported schema conflicts), will now return an error when @@autocommit is enabled (the default setting).

Merged PRs

dolt

• 6483: Fix case sensitivity bugs in some index statements
  Fixes https://github.com/dolthub/dolt/issues/6479
• 6481: Use explicit column names for dolt_status
• 6470: Bug Fix: Error on merges that produce schema conflicts when @@autocommit is enabled
  If @@autocommit is enabled, then a SQL commit will be created after every SQL statement. If a dolt_merge() operation results in data conflicts, then we return an error, revert the working set, and prevent the SQL commit. Schema conflicts should have the same behavior when @@autocommit is enabled, but they were not getting caught by our transaction logic.
  Fixes https://github.com/dolthub/dolt/issues/6469
• 6468: updates global arg descriptions to be more verbose
  Restores more verbose descriptions for global arguments.
• 6467: update profile doc string
  Updates profile doc string to reflect correct dolt profile add behavior.
• 6466: adds new profile event from ld
  Adds new EventType for the new dolt profile cli command

go-mysql-server

• 1933: Fixed case sensitivity bugs in various statements
• 1929: Add support for json_contains_path()
• 1926: Use EvaluateCondition for conditions in join statements
  Fixes https://github.com/dolthub/dolt/issues/6412
  Use the given EvaluateCondition util function to determine whether a join condition is satisfied, which accounts for truthy integer values as mentioned in the linked issue.
• 1925: Fix load data check constraint indexing bug
  fix show tests

Closed Issues

• 6479: Show/Drop Index no longer working on most tables
• 6469: Merging a branch with a data vs schema conflict does not have the same behavior with @@autocommit on
• 6412: Incorrect output for LEFT JOINS with integer-valued filter.
• 6461: dolt pull from another remote sometimes downloads something despite no new commits being there
• 6391: Querying dolt_diff_table errors out when table column has shrunk

This release contains backwards incompatible changes:

• The dolt_diff_<tablename> system table will log a SQL session warning and truncate historical values that are too large to fit in a modified schema. For example, if a varchar(100) column is changed to varchar(10), historical values larger than 10 characters will not be displayed in the dolt_diff_<tablename> system table and instead SQL warnings are logged and viewable with SHOW WARNINGS;. This change reverts a previous change (https://github.com/dolthub/dolt/pull/6399) that changed the types of the dolt_diff_<tablename> system table, and was incorrectly released as a new patch version, instead of a minor version.

Per Dolt’s versioning policy, this is a minor version bump because column types for the dolt_diff_<tablename> system table have changed.

Merged PRs

dolt

• 6459: Log a SQL warning when historical values don't fit in dolt_diff_<tablename>'s schema
  We previously made a change (https://github.com/dolthub/dolt/pull/6399) to automatically widen all columns in the dolt_diff_<tablename> system table to their widest setting (e.g. varchar(10) → TEXT), so that any historical values could still be displayed, even if the table's schema has been changed to a more restrictive/narrow type. This had an unintended side effect of changing data types for customers using the dolt_diff_<tablename> system table, even if they didn't have any schema changes in their table's history.
  Since then, I've investigated a few alternate approaches of how to handle this case. One idea was to search through the commit history, look for schema changes, and use the widest historical type for each column. This adds extra latency to all queries against the dolt_diff_<tablename> tables, since the commit graph has to be traversed an extra time before any results can be returned. It also turns out to be fairly involved to compare types – we don't have any APIs for that (yet) and there are some edge cases such as if a Decimal type is changed multiple times with various precision and scale that aren't all valid together.
  As that code got more complicated, I thought a simpler approach might be better to start with for handling this edge case where a historical value cannot be converted to the current table's schema. This PR applies the same behavior when a value can't be coerced to a type (e.g. trying to coerce a float to a geometry type) – it will truncate the value to NULL in the table and log a SQL warning in the session.
  This means that some historical values will be displayed as NULLs in the dolt_diff_<tablename> system table, which is still an improvement over the previous behavior where they would cause the query to error out and not return any result set. It also means that the column types will be the same types as on the current schema, making it easier for customer to know what type to expect in responses.
  We may still want to make this more sophisticated in the future, but this felt like a good tradeoff for now given that this edge case has not been a significant problem for customers so far.
• 6457: Create / alter table statements involving auto_increment columns now work
• 6440: Auto-merge schema changes that widen varchar columns
  Fixes https://github.com/dolthub/dolt/issues/6261
• 6438: Implemented --prune for fetch command procedure
  dolt fetch --prune deletes any local remote refs that are not present on the remote being fetched. This is simpler than the equivalent git command, as it only operates on branches, not other kinds of refs.
  Fixes https://github.com/dolthub/dolt/issues/6413
• 6390: Adds new --profile global arg
  Adds support for new --profile global arg which can group and apply a specified set of global args for the current command. Includes a new --profile flag and a new CLI command dolt profile to manage and list user profiles.

go-mysql-server

• 1925: Fix load data check constraint indexing bug
  fix show tests
• 1924: Makes several new kinds of alter table statements involving auto_increment columns work correctly
  Also: makes several kinds of ALTER TABLE statements with multiple clauses more lenient in their error checking than MySQL. These statements will now succeed instead of being rejected.
  Fixes https://github.com/dolthub/dolt/issues/6218
• 1922: Render enum/set variables as strings after they are set
  Fixes https://github.com/dolthub/dolt/issues/6370
  Ensure that when getting all session variables, if any of them is of an enum/set type, we convert them back to their string value. The SET command would amend the value in the session variables map to its numerical representation, which we need to translate back.
• 1921: Use a custom iterator when building HashJoin Lookups.
  Previously we were gathering all the child rows into a CachedResults node, and then iterating over the cached results to build the lookup. Cutting out the CachedResults and filling the lookup as we iterate simplifies the plan tree and also prevents a potentially large slice allocation.
  Benchmarking this showed no measurable impact on runtime, suggesting that the time taken populating/iterating over the cache was negligible. But the hash join used in the benchmark allocated 20% less memory.
• 1920: Allow AntiHashJoins to distinguish between empty and non-empty secondary tables.
  AntiHashJoin and LeftOuterHashExcludeNullsJoin should always evaluate at least one secondary row if the secondary table is nonempty. This guarentees the same behavior as MySQL for where x not in (...) expressions when x is nullable.
  Some joins (AntiJoins and some joins converted from AntiJoins) have an "excludes NULL" property, which means that if a filter condition on the join evaluates to NULL, the corresponding primary row can't be included in the result set.
  This means that if the primary row has a NULL value used in a filter, we need to know whether or not the secondary table has at least one row. Using a HashJoin makes that impossible without special handling.
  We had a test designed to catch this, but the test wasn't actually using a HashJoin, so this was missed. This PR also fixes the test to use a hash join.
• 1919: JSON_ARRAY correctly handles CHAR bindvar params
  The JSON_ARRAY function wasn't able to handle a []byte literal expression in a bind var. I also noticed that although our script tests can specify Bindings, they weren't getting used by our prepared query runners. I also fixed that, so that I could add a new test for this issue.
  Fixes: https://github.com/dolthub/go-mysql-server/issues/1855
  Related Vitess change: https://github.com/dolthub/vitess/pull/261
  Dolt CI Checks: https://github.com/dolthub/dolt/pull/6441
• 1917: More name res tests
  Fix JSON_TABLE and stored procedures on new name resolution path.
  edit: Added on duplicate update rewrite
• 1914: More Full-Text fixes
  More bugs found and fixed for FULLTEXT indexes. The ordering bug is fairly major, as it could require a drop/readd of the index, so I want to get this out now rather than waiting until I've gotten more tests/fixes in.

vitess

• 261: Preserving bind var type for sqltypes.Char params
  When a caller executes a prepared statement, they send the parameters as well as parameter type information to the sql-server. Vitess populates bindvars type information that go-mysql-server uses to execute the query. Vitess is currently converting many SQL types to VARBINARY, including CHAR, which makes it look like binary data was sent and not a char string, like the caller indicated.
  This change stops converting sqltypes.Char bind var type info into VARBINARY, which enables go-mysql-server to see that a char string was passed. Without this type information, go-mysql-server doesn't seem able to differentiate between legitimate binary data sent by the client versus a character string. Since these types need to be handled differently, and we can't assume that all VARBINARY types can be converted into strings, it seems like we need to respect the SQL type the caller indicated for the bind var and pass that up to integrators.
  It seems like this change could be helpful for other SQL types, too, but I wanted to start with just sqltypes.Char to see if this approach causes any other issues.
  Fixes: https://github.com/dolthub/go-mysql-server/issues/1855
  Related go-mysql-server PR: https://github.com/dolthub/go-mysql-server/pull/1919
  Related Dolt PR: https://github.com/dolthub/dolt/pull/6441
• 260: Rename MultiAlterDDL to AlterTable and return it for all ALTER TABLE statements
  This normalization makes it easier to handle building a plan

Closed Issues

• 6424: HASH_JOIN probe table copying perf
• 4130: Support LATERAL derived tables
• 6218: Can't add a new auto_increment column with a single alter statement
• 6370: SHOW VARIABLES should render enum/set values
• 6298: Slow Range Join
• 6261: Automatically merge changes that enlarge varchar fields
• 6413: Feature Request: analog of git fetch --prune to clean old branches
• 6429: Connect to a managed doltlab instance remote from local dolt repository
• 1855: GORM JSONArrayQuery.Contains isn't compatible

Merged PRs

dolt

• 6434: Fix for adding a unique column
• 6349: Removing old schema merge guard
  Before we added support for schema merge conflict reporting, we had a guard in place that would error out for schema changes in merges. This code prevented schema merges that Dolt couldn't handle, but it didn't provide much visibility into the exact schema conflicts. The new schema merge conflict reporting code is able to handle these same cases and provides better visibility into exactly what the schema conflicts are. We've already removed some of the checks from this guard, but there were still a few left in place. This PR completely removes the old schema merge guard code.
  The code left in this guard was still protecting against a couple of cases that are now fixed and covered with tests:
  • Collation changes – we weren't properly handling merging changes to a table's default collation. Fixed and tested now.
  • New, non-nullable columns with no default value – new columns that are non-nullable and don't have a default value can't be auto merged since we don't know what value to safely use in the new column for any existing rows, so this now throws an error with instructions on how to fix.

go-mysql-server

• 1914: More Full-Text fixes
  More bugs found and fixed for FULLTEXT indexes. The ordering bug is fairly major, as it could require a drop/readd of the index, so I want to get this out now rather than waiting until I've gotten more tests/fixes in.
• 1913: Adjust range heap coster to ensure we don't choose it over a strong lookup join.
  This improves the performance for a client query by ensuring that we don't us a HeapRangeJoin when there's a good LookupJoin candidate, but we do use a HeapRangeJoin when there isn't.
  In larger queries, the time spent generating and evaluating candidate rows dominates. Using seqIOCostFactor feels like a bit of a misnomer, but cpuCostFactor is too low to reflect this behavior, and this is what costInnerJoin uses.
• 1912: Normalized handling of ALTER TABLE statements
  This PR normalizes the handling of all ALTER TABLE statements to use the same code path. Previously some used convertDDL instead.
  Also changes DDL processing to generate multiple plan nodes for statements that need them. Currently tested with add column .. unique, but the same approach should also work with auto_increment and other clauses.
  Relies on changes in https://github.com/dolthub/vitess/pull/260
• 1909: Show variables panic
• 1908: Some Full-Text fixes
  Adds a few additional tests, along with fixes. This is primarily for Sakila compatibility. More tests and fixes are forthcoming, and will arrive in a second PR.
• 1906: Fix ErrBinaryCollation in queries with []byte argument
  Binding arguments of type []byte have been converted to string literals with the default collation (utf8mb4_0900_bin) by types.CreateStringWithDefaults().
  I think types.CreateBinary() should be used, which generates literals with binary collation.
• 1905: When estimating the cardinality of an outer join, bound the cardinality below.
  For instance, left outer joins can never have a smaller cardinality than their left child. This should let us get more accurate estimates on the cardinality of outer joins and choose better plans.
• 1904: sql/mysql_db: Rework how we store and access table data for users, grants and replica_source_info.
  In general, we keep the same structure of storing the data in structs, keyed in multimaps, and having converters for going to and from sql.Rows. We change the following important things:
  1. We more explicitly model a multimap and an indexed set of entities with multiple keyers.
  2. We add read and write locking around edits and reads of the data table.
  3. We explicitly do not expose the raw indexed set or multimap from the MySQLDb instance itself. The unprincipled access of the various *Data instances in the old implementation was somewhat problematic.
• 1880: Add "Sliding Range Join" execution plan
  (The original PR was accidentally merged. I fixed the history but there doesn't seem to be a way to "unmerge" the PR)
  This is the draft implementation of the "Sliding Range Join" execution plan. This allows for more performant joins when the join condition checks that the column on one table is within a range specified by two columns on the other table.
• 1871: Full-Text Indexes
  This is a partial implementation of Full-Text indexes. There is still quite a bit to finish on the GMS side (as can be seen from the copious amount of TODOs), but this shows the broad strokes of how it's implemented, along with most of the "difficult" design choices being implemented. The major choice that has not yet been finalized is how to deal with FTS_DOC_ID, as it's an AUTO_INCREMENT column in MySQL, but that would not play well with Dolt merging. I already have ideas on how to handle that (taking into account remotes, etc.), but that would come from a later PR.
  https://docs.google.com/document/d/1nGyYg461AhxQjFLzhEEj01XMz0VaTBaBaA44WNu0fc4/edit
  Quite a few things have changed from the initial design doc, mostly based on feedback during the meeting, however some of it was post-meeting. There are three tables instead of 1: Config (stores table-specific information shared across all indexes), WordToPos (maps words to an ID and position, not fully used in the default search), and Count (used to calculate relevancy, also not fully used in the default search). I was planning on converting MATCH ... AGAINST ... to a join between the tables, which would work when fetching results, but MATCH ... AGAINST ... may also be used as a result, which necessitated writing all of the functionality anyway, so the join plan was dropped.
  Last thing to mention, is that I'm pretty sure that Full-Text indexes actually do a full table scan. It seems weird, but AFAICT the indexes are used to quickly calculate relevancy for each search mode. It seems that, for overly large tables, the search time increases even when other index operations continue to operate nearly instantaneously.
  I've tagged two people for review to make it a bit easier. Of course, feel free to take a look at more if you desire.
  @reltuk The sql/fulltext/fulltext.go file is an expansion of the file you've previously reviewed (all still kept to a single file for now). To complement it and see how it'll be implemented on the Dolt side, you can look at memory/table.go. Dolt's table editor will be similar, and the merge paths will only use the FulltextEditor, which special logic to interface with it from those paths.
  @max-hoffman Take a look at the analyzer changes, along with the sql/plan/ddl.go file. You'll probably need to reference sql/fulltext/fulltext.go as well.
• 1869: Migrate most enginetests for new name resolution
  This doubles most of the enginetests to add versions with new name resolution. As a result testing takes ~2x as long, temporarily. Gets majority of those enginetests working with a couple bigger exceptions:
  • plans projections are subtly different in a way that should be optimized but is probably not priority
  • stored procedures need their custom resolution logic ported
  • on duplicate update expressions are buggy, going to rewrite those for new format
  • skipping one derived table alias test, where we do not have expression memoization or lateral joins to help us execute a resolved plan (related https://github.com/dolthub/dolt/issues/6407)
  • many tests throw "column not found" instead of "table not found" errors. I tried to bookkeep those with Skips, but the skipped suites may accumulate other differences in the meantime.
  • I'll need to revert our prepared statement logic before the final switch
  • Various validators work a bit differently, might end up skipping some error tests to get the final switch in sooner
    Other suites:
  • TestJSONTableScripts_Experimental -- json_table still broken
  • TestRenameColumn_Exp -- error test has different error
    A couple other discoveries:
  • We have to hallucinate unknown table, column, and procedure names in trigger bodies on CREATE, and only fail on execution
  • Column default expressions appear to be resolved at execution time
  • Alter statements are only resolved at execution time
  • The precedence for ASOF in nested views and procedure calls is a bit hairy
  • json_array w/ and w/o distinct appears to be untested
  • ORDER BY in UNION seems pretty flaky and lightly tested, we resolve names from the left
    Dolt PR: https://github.com/dolthub/dolt/pull/6414

vitess

• 260: Rename MultiAlterDDL to AlterTable and return it for all ALTER TABLE statements
  This normalization makes it easier to handle building a plan
• 259: Bug fix for round tripping parsed cast function calls
  When formatting a parsed cast node back into a SQL string, we were using the form CAST(<arg>, <arg>), but that won't roundtrip back to MySQL. It needs to be either CONVERT(<arg>, <arg>) or CAST(<arg> as <arg).
• 258: Adding new keywords to the non-reserved list so they don't need to be…
  … quoted
• 257: Support for all temporal query forms in SQL 2011
  These are the parser changes necessary to support https://github.com/dolthub/dolt/issues/6353
  In addition to the 3 forms of temporal query in SQL 2011 (AS OF, FROM .. TO, BETWEEN), this PR also implements support for the 2 extensions added by SQLServer, ALL and CONTAINED IN.
  https://learn.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables?view=sql-server-ver16
  Additionally, VERSION is now a synonym for SYSTEM_TIME, a Dolt pseudo-extension.
• 256: Support ANSI_QUOTES parsing mode
  The ANSI_QUOTES SQL mode changes the behavior of the double quote character. By default in MySQL, ANSI_QUOTES is not enabled and the double quote character is used to quote string literals. When ANSI_QUOTES is enabled, the double quote character may only quote identifiers. The ANSI_QUOTES mode does not change the behavior for backtick quote chars (they always quote identifiers) or single quote chars (they always quote string literals).
  MySQL Reference Docs for ANSI_QUOTES
  Related to: https://github.com/dolthub/dolt/issues/6305 (This is the first step towards supporting ANSI_QUOTES mode in Dolt/GMS)
  GMS PR https://github.com/dolthub/go-mysql-server/pull/1896 stubs out the new Handler interface function, but actually using ANSI_QUOTES mode in Dolt or GMS won't be possible until a few more changes to Dolt/GMS.

Closed Issues

Latency

Merged PRs

dolt

• 6417: Coercing check constraint expressions to boolean results
  Fixes https://github.com/dolthub/dolt/issues/6411
• 6415: Allow sql-client to get the username from the config file
  Previously the --user option was forced for sql-client which is overly strict since users can provide it in a config.yaml file.
• 6404: Don't panic when attempting to publish metrics if config is unparsable
  Currently if you have an unparsable config_global.json, we print a message say as much, then produce a panic attempting to use the config to print metrics. This just drops the attempt to produce the metrics.

Closed Issues

• 6411: Check constraint evaluation in merge should coerce values to true/false

Latency

Merged PRs

dolt

• 6409: Bug fix for state working sets in replication
  Fixes https://github.com/dolthub/dolt/issues/6400
• 6408: Skip test on Windows CI for now
  There's something strange happening with Windows under CI, so I'm skipping this for now.
• 6399: Bug fix: dolt_diff_<tablename> correctly shows historical values after narrowing types
  The dolt_diff_<tablename> system table uses the schema @ HEAD of the table being diff'ed. If the schema of the table has changed over time and has narrowed column types (e.g. varchar(100) → varchar(10)), then some historical values may not be valid against the table's current schema, which causes an error to be returned and no result set. This PR fixes that by changing the schema of dolt_diff_<tablename> to widen all of the column types to their widest size in the same family (e.g. varchar(100) → TEXT).
  Fixes https://github.com/dolthub/dolt/issues/6391

Closed Issues

• 6400: remote replication incorrect working set on connection
• 6353: Implement VERSIONS BETWEEN '<commit>' and '<commit>' and other ref forms
• 4933: dolt log output contains color information
• 6371: Error with dolt_commit_diff subquery
• 6391: Querying dolt_diff_table errors out when table column has shrunk
• 6403: Cannot create another user with all privileges on a database
• 2987: Feature Request: Support FULLTEXT KEY

This is the first release that supports Full-Text Indexes.

Merged PRs

dolt

• 6398: Implemented Full-Text indexes
  Technically functioning, although not the most ergonomic implementation. I need add a bit of user convenience and additional checking, such as removing the additional tables from dolt status, automatically adding the additional tables when the parent is added via dolt add, adding a user-facing write barrier on the tables, etc. Also need quite a few more tests, but the basics are in via the engine tests and very, very few bats tests.
  I'm definitely going to expand the testing in the future, as well as implementing the aforementioned features and more. But for now, this should at least allow us to release unless there are major problems that I've not caught.

go-mysql-server

• 1871: Full-Text Indexes
  This is a partial implementation of Full-Text indexes. There is still quite a bit to finish on the GMS side (as can be seen from the copious amount of TODOs), but this shows the broad strokes of how it's implemented, along with most of the "difficult" design choices being implemented. The major choice that has not yet been finalized is how to deal with FTS_DOC_ID, as it's an AUTO_INCREMENT column in MySQL, but that would not play well with Dolt merging. I already have ideas on how to handle that (taking into account remotes, etc.), but that would come from a later PR.
  https://docs.google.com/document/d/1nGyYg461AhxQjFLzhEEj01XMz0VaTBaBaA44WNu0fc4/edit
  Quite a few things have changed from the initial design doc, mostly based on feedback during the meeting, however some of it was post-meeting. There are three tables instead of 1: Config (stores table-specific information shared across all indexes), WordToPos (maps words to an ID and position, not fully used in the default search), and Count (used to calculate relevancy, also not fully used in the default search). I was planning on converting MATCH ... AGAINST ... to a join between the tables, which would work when fetching results, but MATCH ... AGAINST ... may also be used as a result, which necessitated writing all of the functionality anyway, so the join plan was dropped.
  Last thing to mention, is that I'm pretty sure that Full-Text indexes actually do a full table scan. It seems weird, but AFAICT the indexes are used to quickly calculate relevancy for each search mode. It seems that, for overly large tables, the search time increases even when other index operations continue to operate nearly instantaneously.
  I've tagged two people for review to make it a bit easier. Of course, feel free to take a look at more if you desire.
  @reltuk The sql/fulltext/fulltext.go file is an expansion of the file you've previously reviewed (all still kept to a single file for now). To complement it and see how it'll be implemented on the Dolt side, you can look at memory/table.go. Dolt's table editor will be similar, and the merge paths will only use the FulltextEditor, which special logic to interface with it from those paths.
  @max-hoffman Take a look at the analyzer changes, along with the sql/plan/ddl.go file. You'll probably need to reference sql/fulltext/fulltext.go as well.
• 1863: Bump google.golang.org/grpc from 1.37.0 to 1.53.0
  Bumps google.golang.org/grpc from 1.37.0 to 1.53.0.

Closed Issues

• 6382: mysql_server may panic when performing a join query on dolt_history_$tablename and dolt_commits tables.
• 1902: Valid join query erroneously reported as ambiguous
• 1881: duplicate column names in a table alias should be prevented.
• 1753: "SELECT * FROM tbl" has incorrect schema when table has default values.
• 1745: Allow table names with special characters when MySql allows them

Latency

Merged PRs

dolt

• 6389: fix panic in dolt_history_<table>
  The dolt_history_... table derives its schema from the base table + 3 additional columns historyTableSchema().
  However, when we have a projection over the table, the schema returned from HistoryTable.Schema() is trimmed to only the columns we look at. This is problematic when we're join planning as we look over all indexes and panic when we find an index over a column that isn't in the schema; in this case, it's the Primary key.
  By having HistoryTable implement PrimaryKeySchema, we can return a schema with all the columns, and the index picking portion of join planning works.
  Fix for: https://github.com/dolthub/dolt/issues/6382

go-mysql-server

• 1896: Stubbing out new Handler function to prep for supporting ANSI_QUOTES SQL mode
  PR https://github.com/dolthub/vitess/pull/256 adds support at the Vitess layer for parsing SQL statements using ANSI_QUOTES SQL mode, where double quotes are used as identifier quotes, and not string literal quotes. This change updates the implementations of mysql.Handler for the new function needed to parse prepared statement commands with ANSI_QUOTES mode. A future PR will enable the ability to turn on the ANSI_QUOTES support in Vitess.
• 1892: Fix panic for WHERE filter over SHOW VARIABLES
  Fix for: https://github.com/dolthub/dolt/issues/6379
• 1891: Left pad odd length hex strings
  Fixes https://github.com/dolthub/dolt/issues/6351
  If hex string v is of odd length, prepend an "0" before passing it to Go's DecodeString.
  I've changed the implementation to use DecodeString, but if keeping this as a byte array is preferred, I'll switch back.
• 1890: Fix panic when selecting unsigned number columns via driver
  Calling the reflect.Value.Int() on an unsigned typed value will panic.
  The reflect.Value.Uint() must be used.
• 1889: Recost LookupJoins to remove special casing for indexes where every column is used in a filter.
  The only joins that are affected by this recosting are joins that use an index where every column in the index has a key expression, but we can't prove that lookups are constant.
  We currently special case this, giving these a slightly elevated priority over joins that don't use every column in the index (and we can't prove that lookups are constant.)
  So for example a lookup with one key expression on an index with one column ends up being preferred over say, a lookup with three key expressions on an index with four columns, even though it looks like in practice the latter is a more efficient join. Removing this special casing resulted in improved plans in specific client queries.
  Most of the changed plans in this PR are workarounds for https://github.com/dolthub/go-mysql-server/issues/1893, https://github.com/dolthub/go-mysql-server/issues/1894, and https://github.com/dolthub/go-mysql-server/issues/1895. A few are actually better plans than we were previously generating.
• 1882: Add dynamic privilege CLONE_ADMIN

vitess

• 256: Support ANSI_QUOTES parsing mode
  The ANSI_QUOTES SQL mode changes the behavior of the double quote character. By default in MySQL, ANSI_QUOTES is not enabled and the double quote character is used to quote string literals. When ANSI_QUOTES is enabled, the double quote character may only quote identifiers. The ANSI_QUOTES mode does not change the behavior for backtick quote chars (they always quote identifiers) or single quote chars (they always quote string literals).
  MySQL Reference Docs for ANSI_QUOTES
  Related to: https://github.com/dolthub/dolt/issues/6305 (This is the first step towards supporting ANSI_QUOTES mode in Dolt/GMS)
  GMS PR https://github.com/dolthub/go-mysql-server/pull/1896 stubs out the new Handler interface function, but actually using ANSI_QUOTES mode in Dolt or GMS won't be possible until a few more changes to Dolt/GMS.
• 250: support LATERAL syntax
  syntax for: https://github.com/dolthub/dolt/issues/6194

Closed Issues

Merged PRs

dolt

• 6378: Fix misleading documentation for --host flag
• 6375: Bug fix: Commits created with cherry-pick should only have one parent commit
  When a cherry-pick requires conflict resolution, we set MergeState metadata in the working set to mark that a merge is in progress. dolt commit sees this metadata and incorrectly records the cherry-picked commit as a parent of the new, created commit. This PR fixes this, so that commits created by cherry-pick always have only one parent. To do this, we record an extra flag in MergeState that the current merge is a cherry-pick.
• 6374: adds support for db/branch syntax with --use-db global arg
  Adds support for db/branch syntax with --use-db global arg.
• 6373: Support clone_admin grants for authenticating to remotesapi
  Fixes https://github.com/dolthub/dolt/issues/6354
  GMS PR: https://github.com/dolthub/go-mysql-server/pull/1882
• 6372: Fixed backup to require a dolt directory
  Fixes https://github.com/dolthub/dolt/issues/6365

go-mysql-server

• 1887: have plan.EmptyTable implement sql.DeletableTable
  We throw an error when trying to delete from EmptyTable.
  This error is partially caused by https://github.com/dolthub/go-mysql-server/pull/1885
  The error didn't show up in GMS as there are Exchange nodes that appear in dolt side that don't appear here.
• 1886: Apply index optimization to Offset nodes and drop 0 Offsets
  This PR applies the index sort optimization to plans that look like LIMIT(OFFSET(SORT())).
  Additionally, this PR drops OFFSET nodes that start at 0, as they don't change the output.
  Fix for: https://github.com/dolthub/dolt/issues/6347
• 1885: have plan.EmptyTable implement sql.Updatable interface
  In order to not throw table doesn't support UPDATE error, the EmptyTable node should implement the Updatable interface.
  Additionally, there are other private noop iterators to prevent nil panics.
  Fix for: https://github.com/dolthub/dolt/issues/5397
• 1884: Fix data race in processlist map
  Do a deep copy of maps and any nested maps to prevent concurrent read/write access.
  Fix for: https://github.com/dolthub/dolt/issues/6332
• 1882: Add dynamic privilege CLONE_ADMIN

Closed Issues

• 6351: Left pad hex literals that aren't even lengths
• 6379: Panic due to invalid cast
• 6354: Support giving SQL users grants to remotesapi
• 6347: Query with OFFSET does not use Index
• 5397: error updating table with impossible where clause
• 6365: segmentation fault while try to create a local backup in a dolthub/dolt-sql-server:1.7.5 docker image