Merged PRs

dolt

• 6068: Add support for parquet as an output format for dolt sql
  Adds support for outputting the results of queries as parquet data. For example, dolt sql -r parquet -q "select * from mytable;" > out.parquet will select all rows from mytable, output them to stdout as parquet data, and redirect that output to the out.parquet file.

go-mysql-server

• 1804: Reworked implicit PK handling for referenced foreign keys
  Builds on https://github.com/dolthub/go-mysql-server/pull/1798, introduces an interface which exposes internal (and implicit) primary keys on indexes. Not all integrators will have the functionality, so it's an optional interface to expand compatibility. Also quite a bit simpler.
• 1803: Added "utf8mb3_czech_ci" collation, fixed missing collation check for enum/set
  Fixes https://github.com/dolthub/go-mysql-server/issues/1801
  Adds the requested collation, and fixes the panic. The panic came from an oversight when checking for a collation's implementation. enum and set use the collation during type creation, which occurs before we've verified the collation's implementation. The other string types do not use the collation during type creation, so we return the appropriate error as a result.
• 1802: Update README.md to sync with _example/main.go
• 1799: show all indexes, and prevent creating indexes named 'primary'
  We had some logic in SHOW CREATE TABLE to prevent PRIMARY KEYs from showing up twice in because they are within tables IndexCollection. This logic relied on checking if all the columns in the index were part of the Primary Key. MySQL allows and shows SECONDARY INDEXES that are identical to the primary key. It appears to differentiate them by naming the PK index "PRIMARY". Additionally, MySQL prevents users from creating SECONDARY INDEXES named "primary".
  MySQL names PK index Primary
  fix for: https://github.com/dolthub/dolt/issues/6049
• 1798: include primary key in index mapping for foreign keys
  MySQL does some behind the scenes magic and appends the primary key columns to secondary indexes.
  This creates not obvious prefixes on secondary indexes for foreign keys.
  A plus is that we save on creating secondary indexes when we don't need to.
  Note: it appears this special prefix matching applies when looking for a secondary index on the referenced table, but not on the child table itself.
  Fix for: https://github.com/dolthub/dolt/issues/6038
• 1796: Added a method to SystemVariable to let them be compared using their underlying types, used to this to fix least / greatest not working with system vars
  Fixes https://github.com/dolthub/dolt/issues/6022
• 1779: Add extra filters to AntiJoin to guarentee correct behavior around NULLs.
  This is a correctness fix: generating AntiJoins is not currently equivalent to MySql if either column being used in the not in expression contains NULL.
  This will break a lot of regression tests. If this doesn't break Turbine's tests, we should submit this while we work on a fix.

vitess

• 241: Walking sub-nodes for SHOW TABLE statements
  When preparing a SHOW TABLES statement with a bound variable in the filter clause (e.g. SHOW TABLES FROM mydb WHERE Tables_in_mydb = ?;) GMS and Vitess were identifying the bound variable parameters differently and causing the SQL client on the other end to panic. Vitess code in conn.go walks the parsed tree and looks for SQLVal instances to identify the parameters and then returns that metadata over the SQL connection. The SHOW TABLES statement above fails because the sqlparser AST wasn't including all the members of SHOW TABLES node in the walk. This case is a little tricky to test directly in go-mysql-server, because it only repros in a running sql-server when running over a Vitess conn.
  The GMS and Vitess layers are both calculating bind variable metadata, with two different techniques, and whenever they get out of sync, we will see issues like this that only appear when running over a SQL connection. Longer term, we may want consider allowing GMS to return its bind variable metadata and avoid Vitess needing to re-calculate it, if we see more instances of this problem.
  Fixes: https://github.com/dolthub/go-mysql-server/issues/1793
• 238: Fix for charset introducers in default values
  Fixes https://github.com/dolthub/dolt/issues/5970 by adding an additional default expression rule that handles charset introducers.

Closed Issues

• 6038: Missing index for foreign key error during CREATE TABLE
• 6049: Created index missing in SHOW CREATE TABLE and dolt dump output
• 1801: Creating an enum column with collation utf8_czech_ci causes panic
• 1793: PrepareStatement got incorrect paramsCount in response packet from mock mysql server.

Latency

Merged PRs

dolt

• 6047: Missed a Dolt environment variable if in bats CI definition
• 6043: SQL Server integration tests only on Dolt defaults
• 6042: Go test on only Dolt defaults (__DOLT__ format with chunk journal on)
• 6041: Fix for bug when evaluating check constraints during merge
  Fixes check constraint evaluation during merge when a table has a primary key that does not appear as the first columns in the table schema.
• 6040: Only bats on Dolt defaults during CI (__DOLT__ format with chunk journal on)
• 6039: Storage optimization: Optimize GetMany operations when table files are very large.
  Previously we would perform a linear scan of the table file index, matching up queried chunks with their location in the table file. This changes it to use a binary search, which is the right tradeoff for large table files and small(-ish) query sizes.
• 6036: add php mysql integration test
  adds an integration test to verify support for php mysql clients
  fixes: https://github.com/dolthub/dolt/issues/5813
• 6033: Changed replication configuration errors to log warnings, rather than stop the server from starting
  Fixes https://github.com/dolthub/dolt/issues/6019

go-mysql-server

• 1796: Added a method to SystemVariable to let them be compared using their underlying types, used to this to fix least / greatest not working with system vars
  Fixes https://github.com/dolthub/dolt/issues/6022

Closed Issues

• 6019: Must set @@dolt_replicate_heads before @@dolt_read_replica_remote in SQL to get a read replica started. Error message is bad.
• 5813: MySQL Client Integration Tests for PHP PDO
• 6022: Running LEAST with system variables throws an error
• 5607: SQL users act as committer, not user who started the server

Latency

This release contains a behavior change to use the SQL user name for the Dolt commit author, when Dolt is operating in a SQL mode (i.e. when calling dolt_commit() through dolt sql or through a SQL connection to a dolt sql-server).

Per Dolt's versioning policy, this may require users to update code that accesses the committer field of the dolt_log table. Thus, the minor (major.minor.patch) version bump.

Merged PRs

dolt

• 6026: Remove __LD_1__ from CI
  This is an initial pass at removing integration tests that test the old data format or the migration to the new format.
  The most important thing is the changes to the GitHub action yaml files. We no longer create actions where the default format is LD_1.
  This PR also removes some individual tests that test the old format, but not all of them, only the really low hanging fruit to start. Removing all of these tests will likely be a gradual approach as we edit bats tests.
• 6025: Use the SQL user as the Dolt commit author when operating in a SQL context
  When operating in SQL mode (e.g. dolt sql or connected to a dolt sql-server), the Dolt commit author is now recorded as the SQL user who created the Dolt commit, instead of using the user info configured through dolt config. Committing from the CLI continues to use the dolt config configured user info.
  Fixes https://github.com/dolthub/dolt/issues/5607
• 6014: Allow @@dolt_transaction_commit to be set through sql-server config
  Dolt sql-server YAML configuration can now specify dolt_transaction_commit in the behavior section to automatically set the global value of @@dolt_transaction_commit for a sql-server.
  Prereq for https://github.com/dolthub/hosted-issues/issues/99

go-mysql-server

• 1795: qualify json_table columns
  Likely due to improved aliasing code, it's simple to qualify columns for json_table.
  This PR
  1. adds skipped tests for currently unsupported functionality for json_tables
  • for ordinality
  • nested paths
  • default
  • error
  2. adds prepared tests for existing json table script and query tests
  3. reorganizes json tests
     MySQL docs for missing functionality: https://dev.mysql.com/doc/refman/8.0/en/json-table-functions.html
     Fix for: https://github.com/dolthub/dolt/issues/6004
• 1789: foreign key index match should be on prefix not subset
  Turns out the logic for using an existing secondary index for foreign keys was matching on a subset instead of a prefix.
  For example:

  fks> create table parent (fk1 int, fk2 int, primary key (fk1, fk2));
  fks> create table child (id int unique, fk1 int, fk2, primary key (fk2, fk1, id), foreign key (fk1, fk2) references parent (fk1, fk2));
  

  This should produce a child table that creates a new secondary index over the columns (fk1, fk2).
  But, we don't do that and instead reference some other index (not sure what), causing errors when calling dolt constraints verify --all
  This PR ensures that we only pick an existing secondary index as the underlying index for a foreign key if that index is a prefix for the foreign key.
• 1788: Divide pushdown
  Pushdown currently performs two sets of transforms.
  1. Push filters as low in the tree as possible:

  Filter(x=1) -> Join(Tablescan(x), Tablescan(y), x=y)
  =>
  Join(Filter(x=1)->Tablescan(x), Tablescan(y), x=y)
  

  2. Convert filters into static index scans:

  Filter(x=1)->Tablescan(x)
  =>
  Indexscan(x, x in [1,1])
  

  There are now two rules, pushFilters and generateIndexScans. Running pushFilters
  before join planning lets join planning make better use of functional dependencies.
  Also:
  • Small rule reorganizations
  • Small correctness changes exposed by refactor
  • Remove pushdown tests
  • Filter pushdown appears to work in more cases, see plan changes
  • sql.FilteredTable interface is now unused, marked as deprecated but did not delete related functions

Closed Issues

• 5607: SQL users act as committer, not user who started the server
• 6006: JSON_EXTRACT() has trouble parsing an expression with *
• 6004: JSON_TABLE can't find column when using aliases
• 4330: Replication variables too sensitive to definition order
• 1749: [Feature Request] Support for JSON operator

Latency

Merged PRs

dolt

• 6018: Optimize COMMIT performance for large databases.
  A bottleneck in sanity checks that we run when we add new chunks to the database could make commit performance slow down as the database increased in size. On some benchmarks we ran, a typical slowdown might be ~100ms per 100GBs of database size. This optimizes the code paths involved to use more efficient algorithms and to use a scan resistant LRU cache.
• 6013: Fix typo in sql-server cli docs
• 6003: Add new implementation of ParseGlobalArgs
  The main benefit of this new implementation is that it won't incorrectly parse situations where a global arg value (such as username) matches a dolt subcommand. (This would have been especially problematic when we got around to converting dolt admin.)
  This also adds a nifty little burndown chart of commands that still need to be converted from using dEnv to using CliContext, in order to determine whether to generate the CliContext for a command. It'll be extra satisfying to remove elements from that list as we convert them.
• 5986: create rust mysql client integration test
  creates an integration test to verify support for rust mysql clients
  fixes: https://github.com/dolthub/dolt/issues/5622
• 5957: Add support for dolt table import -a
  I ended up going with @zachmu's suggestion of making this a new operation type like -u, -c, and -r instead of a separate flag after I realized that -c and -r already disallow overwriting fields and it didn't really make sense to add a flag that could only be used with -u.
  This is less ambitious than what was outlined in https://github.com/dolthub/dolt/issues/5936. Notably, I made a big deal in the issue about how you can import the same file multiple times or files that contain previously imported rows without issue. This was a mostly a pet feature that I erroneously assumed would be simple to implement. Given that it wasn't a requirement, after some exploration I ultimately decided to drop it.
  This should be enough to unblock the customer; we talked about implementing a --strict flag too but I don't think it's necessary for their use case.

Closed Issues

• 5622: Add a Rust MySQL Connector test

Latency

Merged PRs

dolt

• 6000: Evaluate check constraints during merge
  Adds support for evaluating check constraint expressions on a table during a merge.
  Fixes https://github.com/dolthub/dolt/issues/3462
• 5998: Add docstring to ConnectionQueryist.

go-mysql-server

• 1790: Reverting @@system_time_zone support for returning system timezone
  I'm not sure yet why this change causes Dolt's MySQL Connector J integ test to fail, but after testing commits yesterday and testing this fix this morning, this is definitely causing some problem. In CI, but not locally, the MySQL Connector J integ test is failing with a generic/empty SQLException each time, immediately after executing the first query. Dolt logs look fine, but there's obviously something happening that the MySQL Connector J can't deal with. I have a few more repro ideas that I'll follow up with, but wanted to get the GMS -> Dolt pipeline cleared up first.
  Until then, this PR reverts @@system_time_zone from returning the actual timezone in use by the runtime and OS.
  This change was able to get the Dolt MySQL Connector integ suite passing again: https://github.com/dolthub/dolt/actions/runs/5059419161/jobs/9081665231
• 1789: foreign key index match should be on prefix not subset
  Turns out the logic for using an existing secondary index for foreign keys was matching on a subset instead of a prefix.
  For example:

  fks> create table parent (fk1 int, fk2 int, primary key (fk1, fk2));
  fks> create table child (id int unique, fk1 int, fk2, primary key (fk2, fk1, id), foreign key (fk1, fk2) references parent (fk1, fk2));
  

  This should produce a child table that creates a new secondary index over the columns (fk1, fk2).
  But, we don't do that and instead reference some other index (not sure what), causing errors when calling dolt constraints verify --all
  This PR ensures that we only pick an existing secondary index as the underlying index for a foreign key if that index is a prefix for the foreign key.
• 1785: Copy MySQL information_schema.st_spatial_reference_systems table
  This PR essentially copies the information_schema.st_spatial_reference_systems table from MySQL.
  Small refactor to move constants and help function to types package from functions/spatial package.
  Additionally, changes show create table statement to print SRIDs inside a MySQL special comment.
  Example:

  tmp> show create table t2;
  +-------+------------------------------------------------------------------+
  | Table | Create Table                                                     |
  +-------+------------------------------------------------------------------+
  | t2    | CREATE TABLE `t2` (                                              |
  |       |   `p` point /*!80003 SRID 2000 */,                               |
  |       |   `l` linestring /*!80003 SRID 2001 */                           |
  |       | ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_bin |
  +-------+------------------------------------------------------------------+
  1 row in set (0.00 sec)
  

  Partially Addresses this: https://github.com/dolthub/dolt/issues/5973
• 1781: Small aggregation perf improvements
  • We used to make a new error object for every cache.Get() miss. Use shared object instead.
  • Add reuse pool for collation weight buffers rather than making thousands of [4]byte.
    before:

vitess

• 238: Fix for charset introducers in default values
  Fixes https://github.com/dolthub/dolt/issues/5970 by adding an additional default expression rule that handles charset introducers.
• 237: Small fix for DEFAULT CHARSET when creating tables
  Super small fix for https://github.com/dolthub/dolt/issues/5749. We were referencing the wrong variable.
• 233: support parsing ALTER EVENT statements
  Supports parsing ALTER EVENT statements

Closed Issues

• 3462: dolt merge silently violates check constraints
• 1771: utf8mb3_tolower_ci not implemented

Latency

Merged PRs

dolt

• 5985: LOAD DATA bats tests improvements
  A customer ran into some issues using LOAD DATA with Hosted Dolt. Added some additional testing in response. Pleasantly surprised with our compatibility. Adds skipped tests for #5982 and #5983.
• 5966: Smart routing for sql command execution against running local servers
  When dolt sql commands are executed against a database which is in used by a sql server, connect to it through the sql port. Currently this does not support authenticated transport. If the root user is still active, we'll use that be default. Otherwise, the --user flag can be used to specify a user, but no authentication token is supported. That will come in the next round of changes.
  A step towards: https://github.com/dolthub/dolt/issues/3922
• 5961: Use UTC timestamp in date comparison test.
  Allows tests to run on laptops which aren't in UTC.
• 5951: Handle unresolved/resolved foreign keys when determining equality
  When foreign keys are unresolved, the data on which columns they point to in the child table and the parent table is stored differently, and requires extra work to "resolve" the columns before we can test for equality. We currently do not always calculate equality correctly, which can result in confusing diff output (e.g. https://github.com/dolthub/dolt/issues/5934).
  This PR adds a new Equals method on ForeignKey that takes in enough data to resolve FK column references, if needed, and test equality for any two foreign keys.
• 5949: go: sqle: cluster: commithook: Periodically heartbeat to a standby when we are primary. This allows replication_lag on the standby to more accurately reflect the possible drift locally.
• 5943: Don't panic when in detached head state.
  Fixes https://github.com/dolthub/dolt/issues/5871
  Many commands currently panic when in a detached head state, such as commit (https://github.com/dolthub/dolt/issues/5744), checkout (https://github.com/dolthub/dolt/issues/5839), and many others. This turns those panics into errors.
• 5940: go: sqle: cluster: Implement dolt_cluster_ack_writes_timeout_secs, a way to block COMMIT ack until the commit is replicated to cluster standbys.
  Setting this system variable to a non-zero value on a primary replica in a sql-server cluster will cause dolt to block a SQL client performing a commit until that client's commit is fully replicated to the replicas.
  If there is a timeout, a session warning is placed on the SQL connection.

go-mysql-server

• 1777: Add SRID 3857 to list of "supported" SRIDs
  The ogr2ogr tool performs some information gathering queries beforehand, which change the queries it generates later on to populate MySQL tables.
  When running ogr2ogr -f MySQL "MYSQL:ogr,user=root,port=3307,host=127.0.0.1" "data.gpkg", ogr2ogr sends these queries

  SELECT VERSION();
  SHOW TABLES;
  SELECT SRS_ID FROM INFORMATION_SCHEMA.ST_SPATIAL_REFERENCE_SYSTEMS WHERE ORGANIZATION = 'EPSG' AND ORGANIZATION_COORDSYS_ID = 3857;
  SELECT SRS_ID FROM INFORMATION_SCHEMA.ST_SPATIAL_REFERENCE_SYSTEMS WHERE DEFINITION = 'PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.25722356
  3,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER[
  "false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]'
  

  However, Dolt's information_schema.st_spatial_reference_system only contains SRID 0 and 4326, as a result both queries ogr2ogr makes return Empty Set.
  Consequently, this causes ogr2ogr to generate the create table queries like so

  CREATE TABLE `aeropuertos` ( fid INT UNIQUE NOT NULL AUTO_INCREMENT, geom GEOMETRY NOT NULL)
  

  When it should be

  CREATE TABLE `aeropuertos` ( fid INT UNIQUE NOT NULL AUTO_INCREMENT, geom GEOMETRY NOT NULL /*!80003 SRID 3857 */)
  

  On initial insert, this causes ogr2ogr to insert with SRID 0
  On further inserts/appends, this causes ogr2ogr to just not fill in SRID, leaving it as -2 aka not initialized
  This PR adds SRID 3857 to dolt's information_schema.st_spatial_reference_system.
  Now, the ogr2ogr command loudly fails with a useful error:

  ERROR 1: MySQL error message:unsupported feature: unsupported SRID value Description: CREATE TABLE `aeropuertos` (    fid INT UNIQUE NOT NULL AUTO_INCREMENT,    geom GEOMETRY NOT NULL /*!80003 SRID 3857 */)
  ERROR 1: Terminating translation prematurely after failed
  translation of layer aeropuertos (use -skipfailures to skip errors)
  

  Addresses these issues:
  • https://github.com/dolthub/dolt/issues/5960
  • https://github.com/dolthub/dolt/issues/5959
  • https://github.com/dolthub/dolt/issues/5931
• 1776: Returning @@system_time_zone based on OS time zone
  The @@system_time_zone global system variable is a read-only system variable that shows the timezone that the server is running in. MySQL and Golang both load the system timezone from the OS, typically using the TZ env var or /etc/timezone. This change exposes that system timezone information through the @@system_time_zone variable, which was previously hardcoded to UTC.
  Because timezone settings can change while the system is running (thanks daylight savings time!), we need to check with the runtime for the system timezone whenever it is requested. I gave SystemVariable the ability to include a function that gets executed to return the sys var value and moved the uptime sys var over to that, too.
• 1775: Error for out of range SRIDs
  This PR fixes an error inconsistency between Dolt and MySQL.
  For SRIDs that are out of range [0, MAX_UINT32], MySQL throws a ERROR: 1690: SRID value is out of range in '<func_name>'.
  For SRIDS that are in range, but just don't exist, MySQL throws a ERROR: 3548: There's no spatial reference system with SRID <invalid srid>.
  Dolt used to only throw ERROR 3548, and would incorrectly report the negative numbers as overflowed positive uint32s
  partial fix for: https://github.com/dolthub/dolt/issues/5948
• 1738: ICU Regex Implementation
  To try and prevent memory leaks, I'm having the regex only work under a callback. The idea is that we'll do all of our matches under the callback, using a node placed by an analyzer rule. I think this approach should work, and it'll expand to any other functions within the regex that need to hold memory that will be freed later.
  For now though, this portion works, and I have a small test showing such.

Closed Issues

• 5991: Compatibility issue with sqlalchemy table class, table manipulation
• 4427: REGEXP_LIKE collation support
• 5871: Many commands fail in detached head by trying to access nil pointer Working Set
• 5960: Support basic functional CRSs (-1 and -2)
• 5973: Agnostic Geometries from CRS
• 5970: charset introducer doesn't appear to work with default column values
• 5749: Unable to create table with default charset of utf8mb3
• 5932: Buggy table after fails to append data to a spatial table
• 5931: Can't append data to spatial tables with unsupported CRS
• 5948: CRS type should be signed
• 5825: Slow ANTI_JOINs
• 1771: utf8mb3_tolower_ci not implemented

Latency

Release 1.0.1 contained a backwards incompatible interface change to the CLI. This change is a precursor to making the Dolt CLI work more naturally with a running Dolt SQL Server. The --data-dir argument (and similar) moved from a command argument to a dolt argument.

ie. dolt sql --data-dir becomes dolt --data-dir sql

Consult the PR for more details.

Per our new version number rules, this should have been a minor version bump instead of a patch release. This minor version bump rectifies the mistake somewhat.

Merged PRs

dolt

• 5952: Print global arguments for 'dolt --help' but not 'dolt'
  Update the Dolt CLI To print the global flags when the --help flag is provided. New look:

  $ dolt --help
  Valid commands for dolt are
  init - Create an empty Dolt data repository.
  status - Show the working tree status.
  add - Add table changes to the list of staged table changes.
  diff - Diff a table.
  reset - Remove table changes from the list of staged table changes.
  clean - Remove untracked tables from working set.
  commit - Record changes to the repository.
  sql - Run a SQL query against tables in repository.
  sql-server - Start a MySQL-compatible server.
  sql-client - Starts a built-in MySQL client.
  log - Show commit logs.
  show - Show information about a specific commit.
  branch - Create, list, edit, delete branches.
  checkout - Checkout a branch or overwrite a table from HEAD.
  merge - Merge a branch.
  conflicts - Commands for viewing and resolving merge conflicts.
  cherry-pick - Apply the changes introduced by an existing commit.
  revert - Undo the changes introduced in a commit.
  clone - Clone from a remote data repository.
  fetch - Update the database from a remote data repository.
  pull - Fetch from a dolt remote data repository and merge.
  push - Push to a dolt remote.
  config - Dolt configuration.
  remote - Manage set of tracked repositories.
  backup - Manage a set of server backups.
  login - Login to a dolt remote host.
  creds - Commands for managing credentials.
  ls - List tables in the working set.
  schema - Commands for showing and importing table schemas.
  table - Commands for copying, renaming, deleting, and exporting tables.
  tag - Create, list, delete tags.
  blame - Show what revision and author last modified each row of a table.
  constraints - Commands for handling constraints.
  migrate - Executes a database migration to use the latest Dolt data format.
  read-tables - Fetch table(s) at a specific commit into a new dolt repo
  gc - Cleans up unreferenced data from the repository.
  filter-branch - Edits the commit history using the provided query.
  merge-base - Find the common ancestor of two commits.
  version - Displays the current Dolt cli version.
  dump - Export all tables in the working set into a file.
  docs - Commands for working with Dolt documents.
  stash - Stash the changes in a dirty working directory away.
  Dolt subcommands are in transition to using the flags listed below as global flags.
  The sql subcommand is currently the only command that uses these flags.  All other commands will ignore them.
  usage: dolt <--data-dir=<path>> subcommand <subcommand arguments>
  Specific dolt options
  -u <user>, --user=<user>
  Defines the local superuser (defaults to `root`). If the specified user exists, will take on permissions of that user.
  --data-dir=<directory>
  Defines a directory whose subdirectories should all be dolt data repositories accessible as independent databases within. Defaults to the current directory.
  --doltcfg-dir=<directory>
  Defines a directory that contains configuration files for dolt. Defaults to `$data-dir/.doltcfg`. Will only be created if there is a change to configuration settings.
  --privilege-file=<privilege file>
  Path to a file to load and store users and grants. Defaults to `$doltcfg-dir/privileges.db`. Will only be created if there is a change to privileges.
  --branch-control-file=<branch control file>
  Path to a file to load and store branch control permissions. Defaults to `$doltcfg-dir/branch_control.db`. Will only be created if there is a change to branch control permissions.
  

go-mysql-server

• 1774: Added utf8mb3_tolower_ci
• 1773: Add plan option for AntiJoins to be LeftJoin
  This PR adds a plan option to convert ANTI_JOINs to LEFT_JOINs with an additional filter condition.
  In some cases, the LEFT_JOIN performs better than the ANTI_JOIN.
  Additionally, this PR adds a new join hint for LEFT_OUTER_LOOKUP_JOINs.
  Fix for: https://github.com/dolthub/dolt/issues/5825
• 1767: Timezone improvements
  This PR starts handling some timezone differences between GMS and MySQL:
  • The NOW() and CURTIME() functions now respect the sessions's current time zone
  • Allows CONVERT_TZ to mix use of timezone names (e.g. UTC) and timezone offset strings (e.g. +00:00)
  • Adds some skipped tests for timestamp input/output session timezone conversion
• 1738: ICU Regex Implementation
  To try and prevent memory leaks, I'm having the regex only work under a callback. The idea is that we'll do all of our matches under the callback, using a node placed by an analyzer rule. I think this approach should work, and it'll expand to any other functions within the regex that need to hold memory that will be freed later.
  For now though, this portion works, and I have a small test showing such.

Closed Issues

• 1771: utf8mb3_tolower_ci not implemented

Latency

Merged PRs

dolt

• 5923: Prepend table name to auto-generated constraint check names.
  Fixes https://github.com/dolthub/dolt/issues/5915
• 5922: Changing schema equality to not consider tag when comparing columns
  Two tables can have identical schemas, but different column tags if the columns were created in different orders (e.g. a different sequence of alters to get to the same schema). When we try to merge those two tables, we see that there is not a common ancestor, so we require the schemas to be exactly identical in order to merge the data. This results in a confusing merge error for customers because we tell them the schemas are different, but they look exactly the same (except for the different column tags).
  This change removes column tags from equality checks when testing equality of two schemas and enables the case above to merge cleanly.
• 5913: Bh/replication metrics
  Adds some replication metrics to the prometheus metrics published by dolt.
• 5907: go/doltcore/merge: model NOT NULL constraint violations
  …olations table

go-mysql-server

• 1770: Union offset+limit bug
  Unions were dropping limit, applying offset as limit.
• 1768: CREATE LIKE statements should preserve check-constraints on the new table.
• 1767: Timezone improvements
  This PR starts handling some timezone differences between GMS and MySQL:
  • The NOW() and CURTIME() functions now respect the sessions's current time zone
  • Allows CONVERT_TZ to mix use of timezone names (e.g. UTC) and timezone offset strings (e.g. +00:00)
  • Adds some skipped tests for timestamp input/output session timezone conversion

Closed Issues

• 5915: Generated names for checks are not guarenteed to be unique.
• 5072: CREATE TABLE LIKE does not replicate CHECK constraints
• 5917: Problems uploading data with ogr2ogr (spatial data)
• 5906: Dolt is especially slow executing queries from within a loop
• 3963: Eliminate column tags, use column name for identity in all cases
• 5531: Foreign Keys can be dropped in a merge of branches that both resolve foreign keys

Latency

Merged PRs

dolt

• 5921: go: doltdb: hooksdatabase.go: Execute commit hooks in parallel so that synchronous commit hooks do not serialize on each other.
• 5919: sql-server: dolt_log_level system variable allows reading the current log level for the server and setting a new one.
  Written values do not persist across server restarts.
• 5918: Remove cry for help from dolt sql --help
  Fixes https://github.com/dolthub/dolt/issues/5916
  Fixes https://github.com/dolthub/dolt/issues/5920
• 5902: go/store/datas/pull/clone.go: Fix to clone to interact with sql-server remotes better.
  Before this fix, |dolt clone| against a sql-server remote can fail with a confusing error message if the sql-server has a chunk journal. Adding the chunk journal to our destination ChunkStore causes us to update the root hash, which the Clone code was not expecting.
  This change updates the Clone code to look for the possible update to the destination's root hash and to not try to set the root hash on the ChunkStore if it has already been done by the code handling the chunk journal.
• 5899: go: sqle: dolt_gc.go: Disable CALL dolt_gc() on a standby replica, where it is not currently safe. Shallow GC is still available.
• 5895: add support for ALTER EVENT
  Depends on https://github.com/dolthub/go-mysql-server/pull/1754
• 5894: batsee --output dir
  An inevitable change to make the output directory of batsee configurable. Refactored the configuration struct to make changes like this a little easier.
• 5892: Fix binlog connection re-establishment when connection closes between messages
  This PR changes our handling of io.EOF to trigger our connection re-establishment logic. Previously, only io.UnexpectedEOF was triggering a connection re-establishment.
  We found this by digging into some flakiness in the TestBinlogReplicationAutoReconnect test:
  • https://github.com/dolthub/dolt/actions/runs/4888594518/jobs/8726401205
  • https://github.com/dolthub/dolt/actions/runs/4888594518/jobs/8726401205
  • https://github.com/dolthub/dolt/actions/runs/4888566463/jobs/8726344999
    This test is a little more complex than the other binlog replication tests – it uses Toxiproxy to simulate a dying connection to test the automatic reconnection feature.
• 5890: normalize windows file paths better
  Depending on if the windows file paths start with file:///C:... or file://C:..., the url package would parse these slightly differently. As a result, the environment would be stored in the singleton under one path, and retrieved with a different one, causing errors.
  fix for: https://github.com/dolthub/dolt/issues/5780
• 5885: go/libraries/doltcore/env/actions: remotes.go: Have fetch pull all the branch heads at once.
  Also avoids round-tripping to the remote to load a commit whose value we do not need yet.
• 5884: Refactor SQLEngine instantiation into the Queryist interface
  SQL Command arguments which pertain to instantiating a local SqlEngine instance are now global arguments which come before the sql subcommand. This is a breaking change for users of the sql command who pass in the following arguments:
  • --data-dir
  • --user
  • --doltcfg-dir
  • --privilege-file
  • --branch-control-file
    Step on the path to: https://github.com/dolthub/dolt/issues/3922
• 5881: Remove overzealous validation check.
  This check appears to be redundant, in order to fail early and return a more helpful error message. But it's not helpful if it's wrong... which it is. It errors fast on inputs that should be correct (such as auto-increment columns), and it not only fails on the example from https://github.com/dolthub/dolt/issues/5855, but it would prevent us from implementing a permanent fix.
  Removing the check, all tests still pass, because invalid imports are still detected by the remaining checks.
• 5874: go/doltcore/merge: use unresolved-aware fk matching method in Foreign…
  …KeysMerge
  fix for #5531
• 5870: Improve dolt_ignore pattern rules
  This PR does two things:
  • introduces % as an alias for *, since % is the SQL wildcard
  • adds an additional check for dolt_ignore rules that are equivalent but have different ignored values.
• 5868: go through primary keys backwards
  This PR implements an iterator to go over Primary Keys backwards, allowing us to order by descending quickly.
  Companion PR: https://github.com/dolthub/go-mysql-server/pull/1742
  Fix for: https://github.com/dolthub/dolt/issues/5812
• 5856: fix empty string import for ENUM and SET types
• 5844: Mention official Arch Linux packaging in README
  I've been maintaining the AUR recipe for dolt since 2021 and updated it for almost all the releases since 0.28.4. I've now migrated the packaging from the AUR to prebuilt packages in the official Arch Linux repos.

go-mysql-server

• 1770: Union offset+limit bug
  Unions were dropping limit, applying offset as limit.
• 1763: prevent large varbinary column from being created
  We don't enforce column size limit for varbinary, causing panics.
  fix for: https://github.com/dolthub/dolt/issues/5059
• 1762: Add support for ALTER TABLE <table> MODIFY COLUMN <col> <type> UNIQUE
  This is a MySql syntax that is effectively syntactic sugar for

  ALTER TABLE <table> MODIFY COLUMN <col> <type>;
  ALTER TABLE <table> ADD UNIQUE INDEX `<col>` (col);
  

• 1760: sql/core.go: SystemVariable: Add a NotifyChanged field which can be used to be notified when the variable value changes.
• 1759: server/server.go: Log a message when we start accepting connections.
• 1756: Allow all built-in functions to be used in column default value expressions
  We currently maintain an allow list of functions that can be used in column default value expressions. MySQL has changed what they support in column default value expressions over time and the current support allows all built-in functions to be used.
  This PR removes our allow list so that any function registered as a built-in function can be used in a default expression.
  Fixes https://github.com/dolthub/dolt/issues/5887
• 1754: support ALTER EVENT statement
  Support for ALTER EVENT statements excluding case for moving events across databases using RENAME TO clause.
  Depends on https://github.com/dolthub/vitess/pull/233
• 1744: Adding tests for altering keyless tables
  Added tests to cover basic keyless table column alterations, to test Dolt change in https://github.com/dolthub/dolt/pull/5867
• 1742: convert sort over pks with index
  This PR fixes some issues with the implementation of the rule replaceSortPk. Namely, the rule now works with both TableAliases, ColumnAliases, and SubqueriyAliases.
  Additionally, the rule is able to be applied when sorting in Descending order.
  Overally, this means queries like select * from t order by pk desc limit 1 will just be a Limit over an IndexTabledAccess
  Fix for: https://github.com/dolthub/dolt/issues/5812

vitess

• 235: Add support for INSERT INTO <table> VALUE ...;
  Adds support for VALUE as a synonym of VALUES in INSERT INTO <table> VALUES ..., to match MySQL's syntax.
  Fixes https://github.com/dolthub/go-mysql-server/issues/1750
• 234: go/netutil/conn.go: Avoid panicing when ConnWithTimeouts has a Set{,}Deadline method called.
  Clients does not expect setting deadlines on connections to panic. In particular, the standard library's TLS implementation adopts an existing net.Conn and will call SetWriteDeadline on it in certain cases.
  It makes more sense to allow the deadlines to be managed by the client when they see fit. This changes the behavior to simply forward the deadlines along as soon as the client code has shown an interest in managing the deadlines.
• 233: support parsing ALTER EVENT statements
  Supports parsing ALTER EVENT statements

Closed Issues

• 3604: Set log_level with a system variable
• 5920: dolt stash --help causes SIGSEV
• 5916: dolt sql --help displays "Help" instead of helpstring
• 4599: Dropping a table then reimporting with inferred schema creates new tags unexpectedly
• 4527: dolt revert violates NOT NULL constraint
• 5673: out of range error on aggregate query
• 5028: can't add a unique key using alter table $tbl modify $col
• 5059: panic on large varbinary insert
• 5812: Slow select order by limit
• 5693: Comparison type overflows
• 5875: Subquery expression -> union->recursiveCTE expression missing finalization rules
• 5807: ENUMs treat empty strings "" as if they were on CSV import, resulting in error
• 5845: Memory leak
• 5848: dolt table import requires an empty column for AUTO_INCREMENT columns
• 5887: Error should be "function m5hash does not exists" not "function m5hash on column id is not valid for usage in a default value
• 5780: dolt sql-server fails if databases is configured in yaml
• 1750: [Feature Request] Support for INSERT INTO ... VALUE

Latency

Dolt's First Stable Production Release

The world's first and only version controlled SQL database is ready for your production workloads. If you've been waiting to try Dolt for it to be stable, fast and compatible, now is the time to try it.

Dolt 1.0 means four things:

1. Forward Storage Compatibility
2. Production Performance
3. MySQL Compatibility
4. Stable Version Control Interface

Read more about it in our release blog

Merged PRs

dolt

• 5882: Apply column defaults when merging in new columns
  Currently, when merging in a new column and updating existing rows for the new column, any column default value for the new column is ignored. This can result in inconsistent data that customers do not expect (e.g. https://github.com/dolthub/dolt/issues/4456).
  This PR adds initial support for applying column defaults when merging in a new column. Only literal default values are supported in this PR. Supporting column references and function references will require more work to properly resolve those references through the analyzer's code. If an unsupported column default value is present, this change detects that we cannot apply it and errors out the merge, instead of dropping the default value and leaving the new column null.
  Fixes https://github.com/dolthub/dolt/issues/4456 (which specifically only needs support for literal default values)

go-mysql-server

• 1751: Unions finalize subquery expressions
  The SUBQUERY -> UNION -> SUBQUERY_EXPR pattern was subject to an edge case where the outer SUBQUERY disabled recursively finalizing the inner SUBQUERY_EXPR. The intervening UNION node blocked the outer reach, but failed to re-enable finalizeSubquery during finalizeUnion. This PR edits the finalizeUnionSelector to explicitly re-enable finalize nested subqueries.
  This PR also touches hoistOutOfScopeFilters for the same edge case.
  Issue: https://github.com/dolthub/dolt/issues/5875
• 1744: Adding tests for altering keyless tables
  Added tests to cover basic keyless table column alterations, to test Dolt change in https://github.com/dolthub/dolt/pull/5867

Closed Issues

• 5888: sql-server has no way to dynamically set the log level.
• 4456: Default Value not applied to existing rows during merge of schema changes

Latency

Merged PRs

dolt

• 5889: Bug fix for dolt_checkout() on a cluster server
• 5886: go/libraries/doltcore/ref: Remove Stash from HeadRefTypes. It was there by accident.

Closed Issues

Merged PRs

dolt

• 5878: Branch Control fixes and improvements
  There was a bug where, under very specific conditions, we could get the branch control table to have a nil map when there shouldn't be one, which would throw a panic. This situation also seemed to cause a bug where you could "infinitely" delete branches without them actually being removed, and our duplicate key detection failed to work in these cases. The root cause of both of those issues have been fixed.
  In addition, if a customer wanted to reject a user from using a particular branch, but allow others, then it was very, very verbose to implement this functionality. I've instead modified our branch control system to take the longest match (which will be the most specific one due to our folding of expressions), and also added an explicit "read" permission, to force some (or all) users to only have read permissions even if there is a global allowance. This also matches the namespace table, which already operated on a longest-match principle.
• 5867: Support adding columns to keyless tables
  Fixes: https://github.com/dolthub/dolt/issues/5846
  New tests in GMS: https://github.com/dolthub/go-mysql-server/pull/1744

Closed Issues

• 5846: schema alterations not supported for keyless tables

Merged PRs

dolt

• 5865: Drop the formatting flag from the engine
  Small step to untangling shell and query engine. https://github.com/dolthub/dolt/issues/3922
  Existing tests cover behavior. This is a fairly straight forward refactor.
• 5863: Testing dolt commit in branch control bats tests
  This case was not tested and a customer was having issues so improved our tests
• 5823: Two code clean ups
  • More the format flag out of the sql engine. It's only used by the sql command, and causes confusion
  • Drop the Dbddl code path. It's not longer required.

go-mysql-server

• 1751: Unions finalize subquery expressions
  The SUBQUERY -> UNION -> SUBQUERY_EXPR pattern was subject to an edge case where the outer SUBQUERY disabled recursively finalizing the inner SUBQUERY_EXPR. The intervening UNION node blocked the outer reach, but failed to re-enable finalizeSubquery during finalizeUnion. This PR edits the finalizeUnionSelector to explicitly re-enable finalize nested subqueries.
  This PR also touches hoistOutOfScopeFilters for the same edge case.
  Issue: https://github.com/dolthub/dolt/issues/5875
• 1747: Added latin1_spanish_ci and latin1_danish_ci collations
  Fixes https://github.com/dolthub/dolt/issues/5866
• 1746: update json_set support for edge cases
  updated logic of json_set to support more input edge cases
• 1743: Optimize division with integers by using floats internally instead of decimal.Decimal
  Fix for https://github.com/dolthub/dolt/issues/5832
  Seems unlikely to break anything in Dolt, but I went ahead and ran Dolt CI tests: https://github.com/dolthub/dolt/pull/5857

vitess

• 234: go/netutil/conn.go: Avoid panicing when ConnWithTimeouts has a Set{,}Deadline method called.
  Clients does not expect setting deadlines on connections to panic. In particular, the standard library's TLS implementation adopts an existing net.Conn and will call SetWriteDeadline on it in certain cases.
  It makes more sense to allow the deadlines to be managed by the client when they see fit. This changes the behavior to simply forward the deadlines along as soon as the client code has shown an interest in managing the deadlines.
• 231: support 'show events' statement parsing
  • Added support for SHOW EVENTS statement parsing
  • Added support for SHOW CREATE EVENT statement parsing
  • Removed FULL option from SHOW TRIGGERS as it's not supported in MySQL.

Closed Issues

• 5019: Feature request: Optimize initialization for large databases
• 5877: PHP Client drops connection on empty call dolt_commit
• 5839: SIGSEGV when attempting to create a new branch while in detached head state.
• 5744: Panic if trying to create a commit on a "use"d tag
• 4570: Three-way merge does not handle type changes
• 5866: Collation latin1_spanish_ci not implemented
• 5568: limit in subquery pushed above where clause
• 5659: Implement SQL function find_in_set
• 5808: Database won't start after import
• 5183: One-time "optimistic lock failed on database Root update" on read replicas after dolt upgrade

Latency

Merged PRs

dolt

• 5864: go/libraries/doltcore/env/actions/remotes.go: Don't put chunks.TableFile instances into a map, since implementations can have embedded closures.
  Just hash the TableFiles by their IDs.
  This fixes a crash on backup restore when restoring from a backup which: (1) uses nbs.NomsBlockStore (for example, gs://, or file://) and (2) transiting any given TableFile takes > 500ms. This is most likely to happen with gs:// remotes.

Closed Issues

• 5680: Add JSON_SET support
• 5832: decimal division (/) is slow

Latency

Merged PRs

dolt

• 5860: go: env/actions: remotes.go: In SyncRoots, if we are syncing to an empty destination repository, use pull.Clone instead of pull.Pull.
  pull.Clone uses the TableFileStore interface to transit whole table files without needing to follow chunk references or do any reconciliation with the destination database regarding what it already has. It is much faster against every time of remote.
• 5853: dolt diff shouldn't print ignored tables.
  Fixes #5841
• 5843: BATS Executor Exemplar
  A utility command which runs bats test in parallel.

  $ batsee --help
  NAME
  batsee - Run the Bats Tests concurrently
  SYNOPSIS
  batsee [-t <threads>] [--skip-slow] [--max-time <time>] [--retries <retries>] [--only test1,test2,...]
  DESCRIPTION
  From within the integration-test/bats directory, run the bats tests concurrently.
  Output for each test is written to a file in the batsee_output directory.
  Example:  batsee -t 42 --max-time 1h15m -r 2 --only types.bats,foreign-keys.bats
  OPTIONS
  -t <threads>, --threads=<threads>
  Number of tests to execute in parallel. Defaults to 12
  -s, --skip-slow
  Skip slow tests. This is a static list of test we know are slow, may grow stale.
  --max-time=<duration>
  Maximum time to run tests. Defaults to 30m
  --only
  Only run the specified test, or tests (comma separated)
  -r <retries>, --retries=<retries>
  Number of times to retry a failed test. Defaults to 1
  

• 5842: Output for dolt status should say "tables", not "files"
  Fixes https://github.com/dolthub/dolt/issues/5840
• 5830: Automatically resolve additive enum/set changes in merges
  This PR adds a new type compatibility checker for resolving compatible type changes during a merge. The intent is to be able to automatically resolve merges where a column's type has changed in a compatible way, starting with adding a new value at the end of an enum or a set.
  I considered a few places to put this logic. The TypeInfo interface seemed like a logical place, but I wasn't completely comfortable with every type having to know the implementation details for each storage layer's implementation. There are also only two types that we're implementing this logic for initially (sets and enums), so editing all types seemed like overkill right now. I figured it would be a good start to get the initial logic together here, then iterate on expanding it or moving it around.

go-mysql-server

• 1743: Optimize division with integers by using floats internally instead of decimal.Decimal
  Fix for https://github.com/dolthub/dolt/issues/5832
  Seems unlikely to break anything in Dolt, but I went ahead and ran Dolt CI tests: https://github.com/dolthub/dolt/pull/5857
• 1741: allow renaming views with RENAME TABLE statement
  This is the same PR as https://github.com/dolthub/go-mysql-server/pull/1712 (it was reverted)
  • allows renaming views using RENAME TABLE statement
  • fails on renaming views using ALTER TABLE ... RENAME ... statement

Closed Issues

• 5841: Newly created tables show up in diff even if they are ignored.
• 5819: Error describing multi column indexes
• 5840: dolt status says files not tables

Merged PRs

dolt

• 5852: go/libraries/doltcore/sqle: database_provider.go: When cloning a remote into a sql-server, always run the InitDatabaseHook on it.
  Failing to run the hook means that the cloned database is not appropriately configured for replication, for example.
  Fixes: #5850
• 5847: support ViewDatabase for clusterdb
• 5838: Adding a mysql connector integ test for the Ruby mysql2 library
  Issue https://github.com/dolthub/dolt/issues/5834 uncovered that we weren't testing what seems to be the recommended Ruby MySQL connector library (mysql2), so this PR adds a connector integration test for it.
• 5831: prevent panics on chain alters
  Partial fix for: https://github.com/dolthub/dolt/issues/5786
  Companion PR: https://github.com/dolthub/go-mysql-server/pull/1736

go-mysql-server

• 1741: allow renaming views with RENAME TABLE statement
  This is the same PR as https://github.com/dolthub/go-mysql-server/pull/1712 (it was reverted)
  • allows renaming views using RENAME TABLE statement
  • fails on renaming views using ALTER TABLE ... RENAME ... statement
• 1739: Populating the Decimal property in Field response metadata
  Fixes https://github.com/dolthub/dolt/issues/5834
  In addition to the unit tests here, I'm also working on a Dolt PR to update the mysql connector library integration tests to test the Ruby mysql2 library.
• 1736: order multi-alter statements
  When running multi-alter statements, MySQL reorders the alters.
  This PR adds a sort function to organize these statements; the precedence order is currently:
  1. RENAME COLUMN
  2. DROP COLUMN
  3. ADD COLUMN
  4. ALTER INDEX
     Note: When this is supposed to work, it does, but when it doesn't our error messages are different than the ones returned by MySQL. There are existing alter tests that also return the incorrect error, but I don't think this is worth fixing right now.
     I also unskipped one BrokenScriptTest and fix the expected results for the others.
     Companion PR: https://github.com/dolthub/dolt/pull/5831

Closed Issues

• 5855: Trigger requires an empty column in order to set new row value with dolt table import
• 5850: CALL dolt_clone() does not configure the cloned database for replication.
• 5786: Dolt panics on a chained alter of adding a column and adding a unique key to the same column
• 5822: Long JSON query truncated
• 5834: ruby mysql2 gem reads field types incorrectly

Latency

Merged PRs

dolt

• 5828: go/store/nbs: reduce startup latency for chunk journal
  https://github.com/dolthub/dolt/pull/5393 added a "journal index" to accelerate chunk journal bootstrapping. Unfortunately it never worked! Bugs in the journal index bootstrapping process caused sanity checks to fail and the journal index was subsequently dropped. The chunk journal itself was still valid and the bugs had no effect on correctness. However, without a valid journal index we were forced to always take the slow-path on startup.

go-mysql-server

• 1739: Populating the Decimal property in Field response metadata
  Fixes https://github.com/dolthub/dolt/issues/5834
  In addition to the unit tests here, I'm also working on a Dolt PR to update the mysql connector library integration tests to test the Ruby mysql2 library.
• 1737: Coalesce.Type() needs to handle type types.Null
  Coalesce.Type() only checks if its arguments to have type nil. But NULL constants have type types.Null, and we need to check for that too.
• 1735: only hash expression.UnresolvedColumn for OnDuplicateExpressions
  Special logic is used to qualify columns for InsertInto.OnDuplicateExprs. Since qualify uses transform.OneNodeExprWithNode it will try to qualify all expressions (Literals, Tuples, etc). This change makes it so that we only try to qualify Columns.
  fix for: https://github.com/dolthub/dolt/issues/5799
• 1728: hashjoin indexing
  When we have plan of the following format:

  InSubquery
  ...
  CrossJoin
  Left: Table
  Right: SubqueryAlias
  OuterScopeVisibility: true
  ...
  HashJoin
  HashLookup
  source: ...
  target: TableAlias
  ...
  

  The indexes we assign to GetFields during analysis don't align with the indexes of the actual columns in each row during execution time. This is a result of StripNode, PrependNode, and the nested Joins with SubqueryAlias.
  This error wasn't caught sooner as the incorrect indexes are too low, so they never threw IndexOutOfBounds errors and just returned potentially incorrect results instead.
  The fix was to correct these indexes at analysis time.
  Firstly, SubqueryAlias nodes with OuterScopeVisibilty = true inside joins need to see the left sibling node (in addition to the parent nodes). So Scope was modified to include some new fields, specifically for sibling nodes. Additionally, the file finalizeSubquery was changed to track the parent as well, so we could detect when we're analyzing a SubqueryAlias on the right side of a join, and add the left child to the scope.
  Additionally, pushdownFilters was modified to not undo all the changes to the Analyzer for HashLookups.
  At runtime, the PrependRow nodes cache the rows outside the InSubquery, while the buildJoinIter for CrossJoin would include both the outside and the left row. This meant that some parts of the inner HashJoin would receive extra columns while others didn't. The fix here was to alter the scope.InJoin depending on which parts of HashJoin we were building.
  Lastly, to have these changes not break for PreparedStatements, we just needed to not redo finalizeUnions in postPrepared, as we don't replan joins in postPrepared, so we don't know if we're in a join or not, and the correct indexes are set in prePrepared.
  Along the way, we discovered a query that panics, but the cause is different than the purpose of this fix, and it panicked the same way before these changes, so it is left as a skipped test.
  Fix for: https://github.com/dolthub/dolt/issues/5714
• 1726: subquery indexing tests

Closed Issues

• 5826: Coalesce returning wrong value
• 5799: Constraints with tuples cause panic
• 3967: Add an equivalent to .gitignore.

Latency

Merged PRs

dolt

• 5809: Add dolt_ignore system table.
  Some notes:
  1. The implementation of dolt_ignore is slightly more complicated than the implementation of existing writable persisted system tables (like dolt_procedures and dolt_docs), but I think it's better this way.
     For context, tables like dolt_procedures exist as DoltTables stored in the user's db alongside user created tables. The user can read and write from these tables, but cannot create, drop, or alter them. CLI commands reference these tables and create them if they don't already exist. This works, but it means that the logic that governs these tables is spread out across the different commands that interact with them, and the user can't interact with them until running a command that creates them.
     dolt_ignore works slightly differently: it's backed by a DoltTable, but it also has its own sql.Table implementation. This allows us to define behavior for when the backing table hasn't been created yet: reads return an empty index, and writes create the table. As far as the mysql engine is concerned, this table always exists. So for the commands that care about dolt_table, the logic can be simpler.
  2. I'm not entirely happy with the implementation: I end up defining the schema twice, once as an sql.Schema, and once as a schema.Schema. I couldn't find an easy way to convert between them. If there's discrepancies between the two definitions, it could cause issues.
  3. dolt_ignore only works with FORMAT_DOLT at the moment. If using FORMAT_LD, the table will be ignored by dolt add and dolt_stash. There's no reason it can't work with FORMAT_LD, but the logic for getting a RowIter from an Index is slightly different and I didn't invest the time to figure it out. This is currently only documented in a comment, but can be documented more widely if needed.
  4. dolt_ignore only affects the staging and stashing of untracked files. This is consistent with git's behavior. If a table that has already been added, dolt_ignore will not prevent further changes to it. It also won't prevent a staged table from being renamed, either to or from an ignored table name.

Closed Issues

Latency

Merged PRs

dolt

• 5818: Transaction isolation for branch-qualified databases
  As well as substantially address the issues about concurrent access to dolt_branches called out in https://github.com/dolthub/dolt/issues/5784
  As a consequence, the dolt_branch procedure now implicitly commits any current transaction.

• 5815: Prevent merges when either side has multiple indexes covering the same set of columns
  When merging multiple indexes that cover the same set of columns (even if they are different types, such as a unique index and a prefix index), there is a race condition because of how our index support assumes the column tag set will be unique for every index. This can cause the wrong index to be matched and the wrong merge decisions to be made.
  I tried to come up with another way to opportunistically match indexes across the sides of a merge in PR https://github.com/dolthub/dolt/pull/5797, but I still found more edge cases and the extra complexity didn't seem worth it. It seemed better to have Dolt be vocal about the issue and let users resolve it, instead of trying to be clever when we can't merge that situation correctly every time.

• 5814: go/libraries/doltcore/env/multi_repo_env.go: Always use the GRPCDialProvider from the bootstrap env for the MultiRepoEnv RemoteDialProvider.
  Fixes #5811.

• 5806: Report schema conflicts for merges with column type changes
  As a first step towards being able to auto-resolve column type changes during a merge, I added a check for type compatibility (which always returns incompatible for DOLT storage format, unless the TypeInfos are exactly the same). This also results in schema conflict metadata being reported.
  I also added some tests for a few classes of type changes and reorganized the existing tests a bit.

• 5803: Fix dolt creds check with specified endpoint
  Using dolt creds check --endpoint <endpoint> --creds <creds> was failing with an audience error because we were using the wrong parsing logic to get the host for the endpoint. This fixes that and adds a log if the endpoint cannot be parsed

  % dolt creds check --endpoint doltremoteapi.awsdev.ld-corp.com:443 --creds b33diq4ht3t73hnp5bofekvsg9lk36ou2gto74f7ot02g38r70mg
  Calling...
  Endpoint: doltremoteapi.awsdev.ld-corp.com:443
  Key: b33diq4ht3t73hnp5bofekvsg9lk36ou2gto74f7ot02g38r70mg
  error: calling doltremoteapi with credentials.
  cause: rpc error: code = Unauthenticated desc = jwt_token validation failed: square/go-jose/jwt: validation failed, invalid audience claim (aud)
  

• 5800: go/doltcore/{sqle, merge}: Update sqlutil.ParseCreateTable to fix test gaps in schema merge suite

• 5798: --data-dir flag enabled before the sql subcommand
  The complexity of creating global arguments is being broken into multiple pieces. This step includes the breaking up of the top level dolt command set to enable global argument parsing and initialization of the CliContext. At present, the sql subcommand is the only one getting this treatment, but it will be extended with additional commands in the future.
  This change doesn't change much that is visible to the user, with the exception of the --data-dir flag being enabled before the sql subcommand.
  Step 4 of N for: https://github.com/dolthub/dolt/issues/3922

• 5795: Correct busted interface change for CliContext

  TIL that interface pointers in golang are not useful. This changes that, and add in a couple calls in command.go which were neglected in the first refactor.
  Step 3 of N: https://github.com/dolthub/dolt/issues/3922

• 5790: Bug fix and test for index merge bug that didn't detect incompatible index types
  The equality functions on Index weren't considering prefixLengths or isSpatial, which resulted in a merge bug where dolt would get confused and match two indexes as identical when they really weren't.

• 5788: Adding basic support for conflict validation in new schema merge test framework
  This also pointed out an issue in the framework where any secondary index and check constraint definitions are silently swallowed.

• 5787: Run schema merge tests in both directions
  Running the ThreeWayMergeWithSchemaChangeTestScripts tests in both directions has been pretty helpful to catch a few bugs, so I took a stab at making testSchemaMerge run merge tests in both directions.
  There shouldn't be any changes to the test cases, but it does help remove duplication in the test cases so that they are a little easier to maintain.
  (I see the Check Formatting CI check is failing here, too... I'll push another commit to reformat those and see if that makes it happy.)

• 5783: dolt show supports displaying non-commit objects.
  PR is still a draft: it needs additional documentation and tests.
  dolt show now accepts a hash of any object in the datastore, not just commits. It will also accept the special strings WORKING and STAGED.
  Some object types (currently just commits) have two representations: a "pretty" format which is more human readable, and a "non-pretty" format which displayed as JSON-like structured data. dolt show prefers the pretty format when possible, but this can be suppressed with --no-pretty.
  Further work: add support for --raw, which outputs the raw binary object. (noms show supports this but I don't know how much use it gets.)

• 5779: Add the CliContext parameter to Command.Exec
  Empty interface passed as last argument to Exec. Refactor by goland.
  NoOp change.

• 5777: go.mod: Bump go-mysql-server, migrate to github.com/dolthub/flatbuffers/v23/go.

• 5773: Fixed a bug where foreign key changes in the working set weren't carried forward during dolt checkout
  Fixes #1310

• 5772: Schema conflict detection edge cases
  https://github.com/dolthub/dolt/issues/2973 pointed out an edge case where a schema conflict wasn't getting reported. That uncovered some other edge cases. Adding those the current code was feeling messy, so I refactored our column merge logic a bit to try and better organize it. We now have mostly separate steps for: 1) calculating the mapping between left/right/anc columns, 2) detecting conflicts based on that mapping, and 3) iterating through the mapping and deciding which column to accept in the merge.
  I'm still not super happy with the conflictKind and the error messages that describe the schema conflicts, but I thought it would make more sense to see the schema merge conflict resolution UX you're working on and then clean those up in a separate pass.
  Fixes: https://github.com/dolthub/dolt/issues/2973

• 5771: Follow-up to validating extra command line arguments.
  A follow-up to https://github.com/dolthub/dolt/pull/5762 addressing comments in the original PR.

• 5768: Add Server Runtime details to the sql-server.lock file
  Add runtime information for the server to the sql-server.lock file. We track three pieces of information:

  1. The PID. As before.
  2. The port of the server, if there is one. You can grab the lock as a cli process, which has no port.
  3. A secret. Currently implemented as a UUID. This will be used to ensure that the connecting through the server is only performed by user's and processes which have read access to the server.
     In addition, this change modifies the sql-server.lock file to be only readable by the owner.
     Step (1) of: https://github.com/dolthub/dolt/issues/3922.

• 5767: go/doltcore/{doltdb, sqle}: dolt_schema_conflicts system table
  …m table

• 5764: Update setup instructions for bats tests
  Bats tests also require that remotesrv be installed.
  I added instructions for that to the README.

• 5762: Add error when commands have extra arguments.
  Rolling this "Add error when commands have extra arguments." again.
  dolt sql-client inherits its arg parser from dolt sql-server, which didn't play nice with the original PR's restriction on sql-server's positional arguments. I relaxed the arg parser and added an extra verification check into sql-server itself.

• 5761: Revert "Add error when commands have extra arguments."
  Reverts dolthub/dolt#5753
  I mistakenly thought that a failing test was flaky. Time to revert the change, investigate, and make sure I don't do anything like this again.

• 5760: Fix error message in remote command

• 5753: Add error when commands have extra arguments.
  Multiple commands would previously silently ignore extra positional arguments. This happened most frequently with commands that aren't supposed to take any positional arguments, but also occurred with some others, most notably merge, which caused https://github.com/dolthub/dolt/issues/5729.
  This fixes that by requiring users of ArgParser to specify how many positional arguments they expect by calling either NewArgParserWithMaxArguments(int) or NewArgParserWithVariableArguments(). The error message can be customized by assigning to ArgParser.TooManyArgsError. The goal is that by requiring developers to be explicit about this, it will help avoid similar issues in the future.
  This adds BATS tests for dolt merge in order to test that the new functionality works. I'm not going to add tests for every other command because there's like 80, but new commands should probably have a test for too many positional args.

• 5732: Fixed bug in dolt checkout with uncommitted changes on another branch
  Fixes #5725

• 5655: Add BATS test that dolt log correctly tags HEAD commits.
  Regression test for https://github.com/dolthub/dolt/issues/5558

• 5626: Schema merge support for adding/dropping/renaming/reordering columns
  First pass at getting schema merge working for column adds/drops/renames/reorders. Includes support for:

  • migrating existing data in a primary index to the merged schema
  • mapping changes from the right side, to the merged schema and merging them into primary and secondary indexes, as well as conflict and constraint violation detection.

go-mysql-server

• 1731: sql/parse: expose utility functions to convert parsed index and check constraint definitions
• 1728: hashjoin indexing
  When we have plan of the following format:

  InSubquery
  ...
  CrossJoin
  Left: Table
  Right: SubqueryAlias
  OuterScopeVisibility: true
  ...
  HashJoin
  HashLookup
  source: ...
  target: TableAlias
  ...
  

  The indexes we assign to GetFields during analysis don't align with the indexes of the actual columns in each row during execution time. This is a result of StripNode, PrependNode, and the nested Joins with SubqueryAlias.
  This error wasn't caught sooner as the incorrect indexes are too low, so they never threw IndexOutOfBounds errors and just returned potentially incorrect results instead.
  The fix was to correct these indexes at analysis time.
  Firstly, SubqueryAlias nodes with OuterScopeVisibilty = true inside joins need to see the left sibling node (in addition to the parent nodes). So Scope was modified to include some new fields, specifically for sibling nodes. Additionally, the file finalizeSubquery was changed to track the parent as well, so we could detect when we're analyzing a SubqueryAlias on the right side of a join, and add the left child to the scope.
  Additionally, pushdownFilters was modified to not undo all the changes to the Analyzer for HashLookups.
  At runtime, the PrependRow nodes cache the rows outside the InSubquery, while the buildJoinIter for CrossJoin would include both the outside and the left row. This meant that some parts of the inner HashJoin would receive extra columns while others didn't. The fix here was to alter the scope.InJoin depending on which parts of HashJoin we were building.
  Lastly, to have these changes not break for PreparedStatements, we just needed to not redo finalizeUnions in postPrepared, as we don't replan joins in postPrepared, so we don't know if we're in a join or not, and the correct indexes are set in prePrepared.
  Along the way, we discovered a query that panics, but the cause is different than the purpose of this fix, and it panicked the same way before these changes, so it is left as a skipped test.
  Fix for: https://github.com/dolthub/dolt/issues/5714
• 1727: implement json_set function
  implements the json_set function with a few edge cases outstanding
  fixes: https://github.com/dolthub/dolt/issues/5680
• 1726: subquery indexing tests
• 1724: Fix README.md
  Replace spaces with tabs in code indentation.
• 1722: go.mod: Use dolthub/flatbuffers/v23 instead of google/flatbuffers.
• 1721: go.mod: Move oliveagle/jsonpath -> dolthub/jsonpath.
• 1720: memory: extract rangeFilterExpr into expression package
  rangeFilterExpr contains a complex set of logic to build a sql expression given the list of sql ranges and the list of expressions on the index.
  Extract the majority of this function into NewRangeFilterExpr in the expression package. Replace the Or and And helper functions with JoinOr and JoinAnd.
  Update the call in the memory/ package to call the new expression function.
  Fix JoinOr and JoinAnd to check if the expressions are nil.
  Remove ineffective nil check in JoinAnd.
  Move nil checks into NewOr and NewAnd.
  Add comment explaining parameters to NewFilterRangeExpr.
• 1719: Added serving tray and bowtie
• 1717: Added a mascot image
• 1715: implement find_in_set
  MySQL docs: https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_find-in-set
  Note: Collations priority isn't exactly the same as MySQL
  fix for: https://github.com/dolthub/dolt/issues/5659
• 1713: Changing MaxTextResponseByteLength() to respect character_set_results
  The Type.MaxTextResponseByteLength() function was not honoring the character_set_results system variable, which resulted in Dolt sending different response length metadata from MySQL when the value of character_set_results is not the same as the type's charset.
  This change adds a *sql.Context parameter to the MaxTextResponseByteLength(ctx) function, so that the correct value can be pulled out of the session when the response metadata is sent through the handler.
  Fixes: https://github.com/dolthub/dolt/issues/5735
  Related Dolt change: https://github.com/dolthub/dolt/pull/5752/

Closed Issues

• 5714: dolt results not equal to MySQL results with multiple nested CTEs
• 5793: Improve error messaging when multiple processes connect to db concurrently.
• 5811: Using the dolt_clone sql procedure to clone a private database has weird behavior
• 3820: N-gram databases for other-language Wikipedias
• 5549: Getting "out of range for tinyint" error from dolt
• 5680: Add JSON_SET support
• 4547: Dolt says constraint violation on merge when one does not exist
• 5683: Unexpected NULL return values with false filter or empty set
• 2973: Merging Modified and Deleted Column should be a Conflict
• 5723: "Format PR" CI Workflow fails on PRs that come from forks.
• 1310: Foreign Keys don't travel with working set on dolt checkout
• 5729: call dolt_merge('branch1', 'branch2') should either merge branch1 and branch2 or fail not silently ignore the branch2 argument.
• 5725: Changes on multiple branches from sql-server has bad interaction with the CLI
• 5558: dolt log incorrectly identifies HEAD
• 5758: Typo in error msg
• 5755: Getting Started Docs sql-client missing user parameter
• 5735: Pasting into Datagrip TEXT field puts null
• 1533: convert_tz is not working as expected with @@GLOBAL.time_zone

Merged PRs

dolt

• 5750: Higher Res image for video in README
• 5745: Add explainer video to README
• 5742: Update uses of go-mysql-server error objects to point to new package.
  The dolt side of https://github.com/dolthub/dolt/issues/5656.
  This is needed for dolt to build against go-mysql-server at HEAD.
• 5717: support EVENT storage handling
  Depends on GMS PR: https://github.com/dolthub/go-mysql-server/pull/1701
  The description of the GMS PR applies here as well that this PR implements events storage only (non-versioned metadata such as last_executed is not implemented yet). The event execution is not implemented yet.
  The events are versioned and are stored on all branches. This means that checking out a new branch will have the copy of existing events of the starting point branch. The events versioned metadata is stored in dolt_events system table.

go-mysql-server

• 1716: Revert "allow renaming views with RENAME TABLE statement (#1712)"
  This reverts commit dac7262c9b7904b931f9792ac8ce3eb52f6f9ccf from PR https://github.com/dolthub/go-mysql-server/pull/1712
  The PR above added a ViewDatabase implementation for PrivilegedDatabase, which is causing some Dolt cluster integration tests to fail. Temporarily pulling this commit out so we can get other GMS changes through and can debug the Dolt test failures separately.
• 1713: Changing MaxTextResponseByteLength() to respect character_set_results
  The Type.MaxTextResponseByteLength() function was not honoring the character_set_results system variable, which resulted in Dolt sending different response length metadata from MySQL when the value of character_set_results is not the same as the type's charset.
  This change adds a *sql.Context parameter to the MaxTextResponseByteLength(ctx) function, so that the correct value can be pulled out of the session when the response metadata is sent through the handler.
  Fixes: https://github.com/dolthub/dolt/issues/5735
  Related Dolt change: https://github.com/dolthub/dolt/pull/5752/
• 1712: allow renaming views with RENAME TABLE statement
  • Added renaming of views with RENAME TABLE ... TO ... statement
  • Added ViewDatabase implementation for PrivilegedDatabase
    TODO: ALTER TABLE ... RENAME ... should fail for renaming of views. Currently, vitess parses both the statements into the same node, which makes GMS parser not be able to detect the difference.
    Should return error: ERROR 1347 (HY000): 'mydb.myview' is not BASE TABLE
• 1711: update stored procedure timestamps to be accurate
  Currently when we load non-built-in stored procedures, we re-analyze the sql string which also re-creates all the metadata. This means that the Created At and Modified At fields can re-set every time the procedure is loaded in, instead of showing the correct timestamp. This change updates the loaded in procedure with the correct timestamps.
  fixes: https://github.com/dolthub/dolt/issues/3081
• 1710: Detect invalid uses of * and window functions in queries.
  This PR is the GMS side of the fix for https://github.com/dolthub/dolt/issues/5656.
  Preventing panics from invalid window functions is easy: replace the panic with returning a new kind of error.
  The invalid *s were trickier. I added an additional analysis rule that runs immediately after resolving function names. It checks for any uses of "*" in a subexpression (so, not just "SELECT *") that aren't used inside a COUNT, COUNTDISTINCT, or JSONARRAY function.
  It's possible that there's other places where *s are allowed that we need to account for. It's also possible that there may be some other disallowed uses of * that will pass this and still cause a panic.
• 1708: Allow uppercase collations
  Fixes https://github.com/dolthub/dolt/issues/5699
  We were comparing the raw strings, which failed when they were anything but lowercase. Now we just force all strings to lowercase.
• 1701: support EVENT storage handling
  This PR implements:
  • CREATE EVENT statement
  • DROP EVENT statement
  • SHOW EVENTS statement
  • SHOW CREATE EVENT statement.
    This PR handles events storage only. The event execution logic is not implemented yet. No events will be run and/or dropped if the execution time is passed or dropped if 'ends' time is passed and 'not preserve' is defined. Created events will be only stored in the database and can be queried to see details using SHOW statements.
• 1663: Return column size in datetime type
  During testing go-mysql-server with Trino and I have noticed that it does work when I have either timestamp or datetime. I have asked them https://github.com/trinodb/trino/issues/16575 whether it's ok for them to implement default column size like they did for Phoenix connector, but they did not want to add it. I still think it should be fine without column size, since most of the clients (except the ones which are written in java) accept date time without column size.
  In order to be fully compatible about it for now I have added 0 as column size for timestamp and datetime

Closed Issues

• 5592: binlog replication doesn't use correct user
• 5711: dolt JSON has different ordering vs MySQL JSON
• 5656: Panics while executing queries
• 5510: dolt command --help should use a pager
• 5338: dolt status reports deleting and creating table when run against active server
• 5260: Lots of dangling connections can potentially hang dolt sql-server
• 4978: Error starting sql-server: invalid ref format: refs/heads/ when encountering a database with no main branch.
• 4291: dolt sql interactive shell can't see changes created in another shell if it is running
• 5743: Feature: auto-create branches on connect
• 5425: Add support for dolt show
• 5719: dolt dolt_constraint_violations doesn't show all violations on merge for composite unique keys
• 5699: Collation description in CREATE TABLE context fails when uppercased
• 3834: Show Variables Where Is An Unsupported Feature
• 1657: panic if project imports vitess.io/[email protected]

Latency