This is a patch release containing small features, bug fixes, and performance improvements.

• Faster query analysis
• Support for describe table as of
• Support for GEOMETRY type
• Better indexing for joins involving a cross join
• Bug fix for bindvars in prepared statements
• Bug fix for views in information_schema tables

Merged PRs

dolt

• 3128: GMS bump for transform refactor
• 3121: Adding tests to validate as of support for describe table.
  Enables Dolt to support describe table as of <commit> or show columns from table as of <commit>.
  Depends on:
  • Vitess support: https://github.com/dolthub/vitess/pull/150
  • Go-mysql-server support: https://github.com/dolthub/go-mysql-server/pull/923

go-mysql-server

• 930: Editing test data to work around failures with new Dolt storage format
  Without changing the test data, the new Dolt storage format doesn't recognize that there is any change and it errors out with "nothing to commit" during testdata setup.
  Editing the test data allows the testdata setup to complete successfully on the old Dolt storage format and on the new Dolt storage format. TestVersionedQueries is now configured to be skipped on the new Dolt storage format.
• 928: Updating versioned query tests based on behavior differences in Dolt harness
  When running the versioned query tests with Dolt harness there are two behavior differences that cause these tests to fail. To fix those, we need to:
  1. explicitly delete old table data and
  2. temporarily remove non-existent version test since Dolt reports a no branch error, while Memory reports a no table error.
• 927: Prevent Creation of Unique Keys
  Small fix for preventing the creation of FULLTEXT keys as well.
  TODO: There should also be a change in Dolt to prevent altering and modifying columns to use FULLTEXT keys
• 926: Have VIEWS show up in information_schema.columns
  Fix for: https://github.com/dolthub/dolt/issues/3109
  VIEWS now show up, but the our current implementation of ViewDefinition makes it difficult to match MySQL capabilities.
• 924: InsertInto Bindvars
  InsertInto is special in that [Source] is not a child sql.Node,
  so [Source] needs to be traversed separately to touch every expression
  in a DFS search.
  This fixes a bug in that DFS search missing the non-Source node
  children.
• 923: Adding support for an as of expression in show columns from table
  Passing through as of expression in show columns statements.
  Depends on Vitess support in: https://github.com/dolthub/vitess/pull/150
• 920: handle more time formats
  fix for https://github.com/dolthub/dolt/issues/3102
• 918: Multi Alter Add/Drop Columns
  This pr does the following
  1. Modifies Add/Drop/Rename/Modify columns to pull the relevant table fresh from the database
  2. Adds error checks in validate_create_table to make sure that invalid columns are detected at analyze time
     This pr does not:
  3. Address multi-alter ddls and edge cases with default values. I found a ton of small errors while working through edge cases here that I think there is a good amount of work separate from the immediate multi-alter ddl needs. This pr is enough to get us unblocked on panics that used occur when mass add/drop pairs were created.
• 914: Adding Geometry type
  Adds geometry type for GMS side.
  Includes case for geometry type for functions:
  • st_swaxy
  • st_srid
  • st_dimension
  • st_aswkt
  • st_aswkb
• 913: Apply IndexJoins with CrossJoin children
• 867: TransformUp is now sensitive to tree modifications
  TransformUp and related node/expression DFS helper functions expect the
  visit functions to return an additional boolean parameter indicating
  whether the visit changed the node:

  type TransformNodeFunc func(Node) (Node, bool, error)
  type TransformExprFunc func(Expression) (Expression, bool, error)
  type Transformer func(TransformContext) (sql.Node, bool, error)
  

  TransformUp's implementation uses the modification information to avoid
  re-creating a node with identical children:

  BenchmarkTransformOld
  BenchmarkTransformOld-12          	  396544	      2782 ns/op	    3000 B/op	      51 allocs/op
  BenchmarkTransformOldNoEdit
  BenchmarkTransformOldNoEdit-12    	  407797	      2731 ns/op	    2936 B/op	      50 allocs/op
  BenchmarkTransformNew
  BenchmarkTransformNew-12          	 4584258	       254.1 ns/op	      96 B/op	       5 allocs/op
  BenchmarkTransformNewNoEdit
  BenchmarkTransformNewNoEdit-12       	 4782098	       237.8 ns/op	      96 B/op	       5 allocs/op
  

  We use plan.InspectUp when possible, and then plan.TransformUp
  where possible, resorting to the more expensive plan.TransformUpCtx
  and plan.TransformUpCtxSchema only when necessary.

vitess

• 150: Parser support for describe table as of and show columns from table as of
  Adding parser support for describe <table> as of <asof> and show columns from <table> as of <asof>
• 144: Fix of View, Trigger, Procedure position params when used in MySQL special comments

Closed Issues

• 3135: dolt dump fails on first table
• 3047: Triggers are not displayed on TablePlus after it was added
• 3109: information_schema.columns table should have column info from views
• 2587: There can be only one auto_increment column and it must be defined as a key
• 3079: DROP PROCEDURE statement not supported with database specified in the query
• 916: RightIndexedJoin commutativity correctness

This is patch release, containing bug fixes and performance improvements.

• Better retries for push and pull
• Allow system variables related to replication to be set globally, or on the session
• Performance fix for covering indexes on the primary key
• Better schema validation for auto_increment columns
• Better support for ALTER TABLE statements that include multiple clauses
• Bug fix for dropping creating or dropping triggers or stored procedures in other databases
• Bug fix for the output schema of show tables statement
• Trigger creation times in information_schema now supported
• Bug fix for incorrect results from some indexed joins
• show create procedure now supported
• show function status now supported
• Bug fix for CREATE VIEW statements inside MySQL special comments

Merged PRs

dolt

• 3108: DoltChunkStore: Table file uploads now retry both the GetUploadLocations call and the PUT call
  For each table file upload, we now call GetUploadLocations for that a single table file, and upload it in the same retriable
  operation.
• 3106: Make dolt_replicate_heads and dolt_replicate_all_heads both Global and Session variable
  Fix for https://github.com/dolthub/dolt/issues/3104
• 3098: Enable the primary key to act as a proper covering index
  This pr ensures that the primary key can appropriately be used as a covering index when selected.
• 3095: go/libraries/doltcore: Fix bad upload table file retries
  Fixes #2183
• 3090: go/libraries/doltcore/schema: Added auto increment validation to schema serialization
• 3083: Build Dolt against GMS to test multi-alter DDL changes
  Bump GMS to fit in multi-alter ddl changes.
• 3080: go/libraries/doltcore/dbfactory: Thread memory quota provider through store layer to onHeapTableIndex
  This PR allows creators of noms block stores to manage memory resources related to them.
• 3032: go/libraries/doltcore/sqle: Refactored Auto Increment
  companion PR
• 2998: Added clarifying test for AS OF behavior in view definitions

go-mysql-server

• 918: Multi Alter Add/Drop Columns
  This pr does the following
  1. Modifies Add/Drop/Rename/Modify columns to pull the relevant table fresh from the database
  2. Adds error checks in validate_create_table to make sure that invalid columns are detected at analyze time
     This pr does not:
  3. Address multi-alter ddls and edge cases with default values. I found a ton of small errors while working through edge cases here that I think there is a good amount of work separate from the immediate multi-alter ddl needs. This pr is enough to get us unblocked on panics that used occur when mass add/drop pairs were created.
• 915: fix create/drop TRIGGER parsing to support database name specific query
• 910: Ensure that the local index map uses the computed id
  We should use the computed id of index as key for the index map of the memory table. Previously we were using the given index name which would conflict when multiple indexes were assigned the empty string.
• 909: fix create/drop procedure parsing to support database name specific query
• 907: Add db name to the column name for show tables statement
  • Database name is now included in column name in SHOW TABLES statement
  • Fixed cardinality column of information_schema.statistics table returned wrong type before
• 906: Store the creation time for trigger statements
  Will have changes to Dolt side to store this for Dolt Databases.
  MySQL has nanosecond precision, which we wont have here.
  I mimicked stored procedure behavior creation time for mysqlshim: https://github.com/dolthub/go-mysql-server/blob/main/enginetest/mysqlshim/database.go#L169
• 904: IndexedJoins mark handled filters
  Applying an indexed join in applyJoinPlan should apply equality
  lookup expressions, but currently do not. pushdown mistakenly
  marked those expressions as handled and eliminated necessary filters.
  This PR prevents IndexedJoinAccess with nil lookups from marking
  filters as handled.
  A future improvement could the pushdown range logic
  to opportunistically apply lookups in applyJoinPlan, and let
  pushdown mark those as handled. This would improve perf by both
  removing both a table scan and filter from some index joins.
• 903: support show create procedure
• 898: Improve MultiAlters error handling with Alter Primary Key, Alter Index, Alter Default, Alter AutoIncrement
  This pr does the following
  1. Improves MultiAlter DDL analyzer by checking for failure modes in adding/dropping pks, adding/dropping indexes, adding/dropping defaults
  2. Pushes the table resolution of AlterPk, AlterIndex, AlterDefault,Alter AutoIncrement at execution time. This is due to the fact that the node should know whether columns were added or dropped in previous steps of a MultiAlter DDL.
  3. Adds a small bug fix with how columns were being dropped in the validate_create_table
  4. This pr does not include changes to multialters with ADD/DROP constraints due to additional complexity.
• 890: support SHOW FUNCTION STATUS functionality
  SHOW FUNCTION STATUS result is queried from INFORMATION_SCHEMA.ROUTINES table
• 886: Auto Increment Refactor

vitess

• 149: add db name specification to create/drop TRIGGER statements
• 148: add db name specification to create/drop procedure statements
  Database name is case sensitive but procedure name is not, so TableIdent and ColIdent structs are used for defining db and procedure names, respectively.
• 147: Have SHOW CREATE PROCEDURE <procedure_name> parse
  TODO: need changes in GMS to handle show create procedure logic.
• 144: Fix of View, Trigger, Procedure position params when used in MySQL special comments

Closed Issues

• 2980: Dolt Panic when creating view
• 3102: Truncated time strings are not parsed correctly
• 3075: Bad foreign key error message
• 3104: dolt_replicate_heads should be available as a global var
• 3071: dolt errors and crashes when trying to use DBeaver
• 2183: unknown push error When pushing a branch with a single commit to DoltHub
• 3065: WHERE statement has no effect when joining multiple tables
• 2325: Support Compound Primary Key Change Statements
• 3046: Adding/changing PK through TablePlus display fails
• 2999: Dolt uses current view definition even with AS OF
• 2804: Support SHOW FUNCTION STATUS syntax
• 3048: Procedures do not show on Tableplus display
• 894: when is version 1.0 api standard set?
• 900: Does go-mysql-server support create database and table from SQL file?
• 901: ERROR 1105 (HY000): Error: index already exists

This is a patch release containing new features and bug fixes.

Features:

• A new system table function, dolt_diff(). Read more about it here
• SHOW CREATE PROCEDURE is now supported

Bugs fixes:

• Error in connecting to a commit hash revision when in read replica mode
• Incorrect handling of foreign key creation on tables with spatial columns
• Incorrect handling of foreign key creation for tables with multi-column primary keys
• CHARACTER SET tokens in CREATE TABLE statements now parse correctly
• Incorrectly dropped WHERE clauses in certain indexed joins
• Various information_schema tables now contain more accurate information
• Columns and table names that begin with numbers but have at least one letter (123a) are now valid

Merged PRs

dolt

• 3055: Bug fix for connecting to a particular commit hash in a read replica
  Fixes https://github.com/dolthub/dolt/issues/3056
• 3054: /{.github,go}: require go 1.18
• 3053: Simplify RebuildIndex
  Two improvements with index rewriting:
  1. Brian's TupleFactory optimizations were being skipped, and 70% of
     the runtime was spent building newBinaryNomsWriter instances for tuple
     construction. Removing this restores index inserts as the only memory
     pressure.
  2. Brian's BulkEditAccumulator lets the final Map sort skip
     work materializing intermediate prolly trees.
     On a 2000 row table, fix (1) reduces the runtime by 50% (80ms -> 40ms),
     and fix (2) reduces the runtime by another 40% (40ms -> 10ms)%, for a
     cumulative ~90% improvement.
• 3042: Fixed two bugs in foreign key creation
  Fixed https://github.com/dolthub/dolt/issues/3023
  Fixes https://github.com/dolthub/dolt/issues/3028
• 3041: go/cmd/dolt: Fix fetch printing newline for each progress update on Windows
  Fixes: https://github.com/dolthub/dolt/issues/2941
  The above reported bug is caused by EphemeralPrinter. It's related to how the underlying library of uilive clears terminal lines on Windows. uilive will only clear terminal lines successfully if it detects that the given io.Writer is a terminal. The package does this by checking if the io.Writer has a Fd function and using the returned file descriptor for its istty check.
  When color.Output is used as the io.Writer, uilive's terminal check fails since that does not declare Fd. The problem is that color.Output does use the initial os.StdOut which is in fact the terminal...
  This was fixed by wrapping color.Output with a struct that reports the correct file descriptor.
  This PR also improves the display output of ephemeral printer by controlling when uilive flushes its output.
• 3038: go.mod,proto/third_party: Bump protobuf dependencies to migrate from github.com/golang/protobuf to google.golang.org/protobuf.
• 3036: unskip bats test for CREATE DB ... CHARACTER SET ... statement
• 3035: go/libraries/sqle: Added integration tests for "Database Revisions", simplified DoltDatabseProvider
• 3030: Unblock the failing MySQL FDW integration test
  This pr:
  1. Switches to an older version of MySQL FDW due to #3029
  2. Adds a comment in the c api to test an additional function
  3. Adds logic in the mysql bats client tests that shuts the down the postgres server in the teardown. Will be useful in case new versions of the wrapper cause another breakage.
• 3025: .github/workflows: Added timeout for MySQL client integration tests
• 3021: go/libraries/doltcore/sqle/dsess: Refactor Dolt Session update methods
  This PR alters the life cycle of WriteSessions within a dolt session. WriteSessions now have a single working set ref throughout their existence, and assert that their working set ref is not changed by set or update calls.
  To switch branches/working set, the Session discards the existing (empty) WriteSession, and creates a new one
• 2913: dolt_diff system table function support
  Adds support for a new dolt_diff system table function and changes the behavior of the existing dolt_commit_diff_$tablename and dolt_history_$tablename system tables to no longer disambiguate column names with tag suffixes.
  The new dolt_diff system table function enables callers to use the exact to and from schemas in a two-way diff of table data. Using this sample data input, the following examples show what the dolt_diff table function returns in different cases.
  General example:

  > select * from dolt_diff("foo", @Commit1, @Commit9);
  +------+------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | to_c | to_b | to_pk | to_commit                        | to_commit_date                    | from_a | from_b | from_pk | from_commit                      | from_commit_date                  | diff_type |
  +------+------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | jkl  | foo  | 1     | dap9j5he1296trqe5dp4j2n9lut9tuaa | 2022-03-16 20:13:05.878 +0000 UTC | foo    | bar    | 1       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  | NULL | baz  | 2     | dap9j5he1296trqe5dp4j2n9lut9tuaa | 2022-03-16 20:13:05.878 +0000 UTC | baz    | bash   | 2       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  | five | four | 3     | dap9j5he1296trqe5dp4j2n9lut9tuaa | 2022-03-16 20:13:05.878 +0000 UTC | NULL   | NULL   | NULL    | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | added     |
  +------+------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  

  Diff across a column drop:

  > select * from dolt_diff("foo", @Commit1, @Commit2);
  +------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | to_a | to_pk | to_commit                        | to_commit_date                    | from_a | from_b | from_pk | from_commit                      | from_commit_date                  | diff_type |
  +------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | foo  | 1     | frfpd9k00s417gi8b88067l5oavkfpt5 | 2022-03-16 20:13:05.821 +0000 UTC | foo    | bar    | 1       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  | baz  | 2     | frfpd9k00s417gi8b88067l5oavkfpt5 | 2022-03-16 20:13:05.821 +0000 UTC | baz    | bash   | 2       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  +------+-------+----------------------------------+-----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  

  Diff across a column drop and rename:

  > select * from dolt_diff("foo", @Commit1, @Commit3);
  +------+-------+----------------------------------+----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | to_b | to_pk | to_commit                        | to_commit_date                   | from_a | from_b | from_pk | from_commit                      | from_commit_date                  | diff_type |
  +------+-------+----------------------------------+----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  | foo  | 1     | tnipgs33r3kd0pt823u0uu5ek4n39dpf | 2022-03-16 20:13:05.83 +0000 UTC | foo    | bar    | 1       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  | baz  | 2     | tnipgs33r3kd0pt823u0uu5ek4n39dpf | 2022-03-16 20:13:05.83 +0000 UTC | baz    | bash   | 2       | diq7v3vlqpjb21imb3bqv04spob7l0p2 | 2022-03-16 20:13:05.814 +0000 UTC | modified  |
  +------+-------+----------------------------------+----------------------------------+--------+--------+---------+----------------------------------+-----------------------------------+-----------+
  

  Depends on:
  • https://github.com/dolthub/vitess/pull/131
  • https://github.com/dolthub/go-mysql-server/pull/838
  • https://github.com/dolthub/go-mysql-server/pull/885

go-mysql-server

• 904: IndexedJoins mark handled filters
  Applying an indexed join in applyJoinPlan should apply equality
  lookup expressions, but currently do not. pushdown mistakenly
  marked those expressions as handled and eliminated necessary filters.
  This PR prevents IndexedJoinAccess with nil lookups from marking
  filters as handled.
  A future improvement could the pushdown range logic
  to opportunistically apply lookups in applyJoinPlan, and let
  pushdown mark those as handled. This would improve perf by both
  removing both a table scan and filter from some index joins.
• 899: run query on info schema routines table for SHOW PROCEDURE STATUS
  Also fixes SHOW PROCEDURE STATUS to show all procedures in all databases.
  Removes show_procedure_status.go file (removing the showProcedureStatus node)
• 897: Code sample update to remove missing Auth package
  Updated code sample in README and _example/main.go to remove the missing Auth package.
  Reported in: https://github.com/dolthub/go-mysql-server/issues/888
• 896: adding 1 second to trigger creation time
  Fix for this issue: https://github.com/dolthub/dolt/issues/3047
  Triggers weren't showing up in TablePlus because the creation time for triggers is always 0 in unix time, which is out of TablePlus supported timestamp range (for some reason). So this PR just makes the trigger creation time 1 second after the start of all time, which TablePlus considers valid.
• 895: fix SHOW INDEXES FROM otherdb.tab fails as database is initialized incorrectly in parsing
  Before Show Index statement was initializing the Database incorrectly as it would set it to either empty or current database. This was an issue for using the statement for database that user is not currently on.
  This PR also fixes couple of info schema table inconsistencies against Mysql.
• 893: /go.mod: require go 1.18
• 887: fix non_unique column value definition in information_schema.statistics table
  fix non_unique being assigned incorrect value. Verified with Mysql docs, "0 if the index cannot contain duplicates (which .IsUnique() is true), 1 if it can."
• 885: Ensure CheckPrivileges runs on resolved table functions
  While testing privilege checks for the new dolt_diff table function, I realized that CheckPrivileges was being run on UnresolvedTableFunction and not the actual resolved table function. Since the dolt_diff table function pulls data from underlying tables, it needs to be resolved before CheckPrivileges runs to ensure we can accurately check privileges.
• 884: added warnings for some syntaxes of ALTER TABLE and CREATE DATABASE statements
  • Added a warning for using CHARACTER SET, COLLATE or ENCRYPTION syntaxes when creating database. The database is still created.
  • Added a warning for using DISABLE KEYS or ENABLE KEYS options for alter table statement. Nothing changes for the table being altered.
  • Added a system variable that mysqldump uses, innodb_stats_auto_recalc
• 881: More tests for alter table bugs
  1. Split PkOrdinals tests into DDL and DML (new)
  2. New script test that triggers the primary key / secondary key ordinal mismatch
  3. Add ctx.Done() checks to a few key node iterators before operating on child rows (may have missed some, but tried to hit the table, index, and edit iterators)
     companion PR: https://github.com/dolthub/dolt/pull/3013

vitess

• 147: Have SHOW CREATE PROCEDURE <procedure_name> parse
  TODO: need changes in GMS to handle show create procedure logic.
• 146: /go.mod: require go 1.18
• 143: Allow Columns to Start with Numbers...kinda
  Recognizes identifiers that start with numbers, unless they are immediately after a .
  TODO: table names that start with numbers do parse, but there is a regex blocks these from parsing in dolt, so will fix in future PR.
• 142: Add DISABLE | ENABLE KEYS syntax for ALTER TABLE statement
  Added parsing of DISABLE KEYS and ENABLE KEYS options for ALTER TABLE statement
  Updated checking Index action keyword to be case insensitive
• 140: support CHARSET, COLLATE and ENCRYPTION syntax in CREATE DATABASE statement
  added parsing of CHARACTER SET, COLLATE and ENCRYPTION options for CREATE DATABASE statement
• 139: Revert "Merge pull request #137 from dolthub/james/special-comments"
  This reverts commit 19d17da876ea71be47f41092d8ab420885b9ba18, reversing
  changes made to 0bb5262822024a793f65040a77430ebe0624058e.

Closed Issues

• 3044: Renaming column fails on TablePlus as it generates query with DEFAULT value specified.
• 2982: Cannot add a column that starts with a number
• 3056: use db/<hash> breaks when server replication enabled
• 2941: Dolt pull/push progress doesn't clear last reported progress on Windows
• 3028: UNIQUE constraint inappropriately created during foreign key creation
• 3023: Foreign key on a table with spatial type column triggers a spurious error cannot create an index over spatial type columns
• 2463: Missing features in dolt log
• 2971: CHARACTER SET in CREATE DATABASE statement results in SQL parsing error
• 888: The _example/main.go compile error.

This is a patch release, containing bug fixes:

• Corrupted secondary indexes after dropping and re-adding a table's primary keys in a different order
• LIKE expressions with an OR returned incorrect results
• Better support for unquoted keyword identifiers in parser
• Made the db_working, db_staged, db_head system variables read-only, deprecate detached head mode
• Various DDL statements now parse correctly
• Hex numbers not evaluating correctly in some expressions
• Fixed various console display bugs during long-running operations like dolt pull

Merged PRs

dolt

• 3013: Apply PK ordinals to secondary index pk tag ordering
  The original bug was found by stabs: https://github.com/dolthub/dolt/issues/2991
  Refer to the GMS PR with a script test for testing the condition that triggers the bug: https://github.com/dolthub/go-mysql-server/pull/881
  A summary of the bug is: Secondary indexes were caching old pk orderings. Schema changes created an inconsistency between secondary indexes and the primary keys they referred to, causing errors in index lookups. This only impacts DDL statements involving a new pk ordering and preexisting secondary indexes.
  There are several layers of indirection between pk source row and secondary index row. When creating an index row, a higher layer of code reads and formats keys in the correct index ordering (for example, reading schema order (x,y,z) -> (y,z,x)). However, we deconstruct and rebuild that row before inserting values into a secondary index. The second constructor drops the ordering in favor of a hashmap of tags, and uses an index embedded tag ordering pks to recreate the row.
  This fix extends schema.SetPkOrdinals to override the preexisting ordering of pks with an indexCollection.SetPks method.
• 3007: go/libraries/doltcore/sqle/writer: Working Set refactor
• 3002: Andy/temp table refactor
  redo of #2993
• 3001: libraries/doltcore/dtestutils/testcommands: fix dropped errors
  This fixes several err variables that were set but then unhandled, as well as an os.Chdir() that looked a little risky without one.
• 2996: Add ORM Bats testing utilities
  This PR setups the basic bats setup for ORMs. Once we have an ORM passing end to end we can expose this to the CI with the relevant workflows.
• 2994: Change transactions per second to sql_transactions_per_second
  Fixes the column name of the result set for a particular benchmarking query.
• 2986: Update queries to include tps as well as more sophisticated percentil…
  This pr fixes some of the queries produced by the TPCC automation. It also adds a query to return TPS
• 2967: Add Ephemeral Printer
  EphemeralPrinter is tool that you can use to print temporary line(s) to the console. Every time Display is called, the previously written lines are cleared, and the new lines are flushed to the output.
  Fixes #2964
• 2966: go/libraries/doltcore/sqle: Remove "Detached Head" mode
  This PR removes support for "detached-head" mode in the SQL context and modifies surrounding functionality to restrict state changes in the SQL context. The central change here is that all SQL writes must be made against a working set and all writes must be made transactionally. With the exception of read-only sessions, all SQL sessions will have an associated working set.
  Currently, SQL sessions can modify their state through special session variables:
  • myDatabase_head_ref: essentially a branch change, changes the current head and also switches to the associated working set.
  • myDatabase_head: similar to the _head_ref variable, but more general in that any commit can be used as a new head. Setting this variable puts the session into detached-head mode. Writes to this session are not transactionally written to a working set. Instead, they are written as free-floating values (eligible to be GC'd) and commits made in detached-head mode are "dangling" commits: they are not associated with a branch (and also eligible to be GC'd). A typical pattern for using this mode is to set the database head, make some writes and commits, and land this work by writing into the dolt_branches table.
  • myDatabase_working: setting this variable only affects the working set of session, essentially force-setting it to some other state.
    This PR restricts the use of these variables in the following way:
  • myDatabase_head_ref: if the new value is a branch or working set ref, the session will switch working sets, other refs will error.
  • myDatabase_head: the new value will be resolved to a commit. If the commit corresponds to a single branch, the session will switch to the associated working set. If the commit cannot be resolved or doesn't correspond to a unique branch, an error will be returned. edit: this variable is now read-only
  • myDatabase_working: similar to _head, if the new value can be resolved to a unique working set, the session will switch to it, otherwise it will error. edit: this variable is now read-only
• 2935: merge constraints more intelligently
  Merging new checks defined on separate branches used to just drop both checks, and somehow NOT NULL constraints too. This fixes that.
  Uncertain behavior:
  • merging constraint and conflicting data is successful
  • modifying constraint and conflicting data is successful

go-mysql-server

• 881: More tests for alter table bugs
  1. Split PkOrdinals tests into DDL and DML (new)
  2. New script test that triggers the primary key / secondary key ordinal mismatch
  3. Add ctx.Done() checks to a few key node iterators before operating on child rows (may have missed some, but tried to hit the table, index, and edit iterators)
     companion PR: https://github.com/dolthub/dolt/pull/3013
• 878: Fix logic with OR operation and add missing nil check to Like
  MySQL has some really weird behavior where the OR of two non bools evaluates too false. Improperly handling this situation was trickling up to a LIKE panic on the new query added to queries.go

  SELECT "A" OR "A";
  +------------+
  | "A" OR "A" |
  +------------+
  |          0 |
  +------------+
  

• 873: Add support for making fulltext return unsupported feature
  This pr addresses this issue
  Cc mysql docs
• 872: sql: expose convert native go value to value
  Expose convertToValue function from memory package as ConvertToValue.
  Return an error instead of panicing if the value type is not known.
• 871: Apply bindvars to subqueries
  Cherry pick from https://github.com/dolthub/go-mysql-server/pull/795 to fix Subquery bindvars specficially.
• 869: Fixed invalid hex number type association from number to blob
  Fixes https://github.com/dolthub/dolt/issues/2968
• 861: have methods for dolt to call to parse check expression
  To properly merge check constraints, Dolt needs to partially parse their expression strings.
  This PR gives Dolt access to some helper methods, to complete this operation. Should have no functional effect on GMS.
• 849: Implemented sort node, number type, and handler support for Row2, Expr2, Type2
• 838: Table function support
  Adds support for table_functions in go-mysql-server.
  Depends on:
  • https://github.com/dolthub/vitess/pull/131

vitess

• 139: Revert "Merge pull request #137 from dolthub/james/special-comments"
  This reverts commit 19d17da876ea71be47f41092d8ab420885b9ba18, reversing
  changes made to 0bb5262822024a793f65040a77430ebe0624058e.
• 137: Handle subquery indexes correctly for special MySQL comments
  Fix for: https://github.com/dolthub/dolt/issues/2980
  When Tokenizer detects a special MySQL comment, it just creates a new Tokenizer and embeds it in the old one under the specialComment member variable.
  This "inner" Tokenizer is then used to handle all the parsing, and its results are just passed to the outer Tokenizer. However, the issue is that the lexer or whatever only reads the Position member variable from the outermost Tokenizer.
  So my proposed fix is to just copy over the Position of the inner Tokenizer to the outer one (with an offset to handle the leading /*![12345]. I think a better fix might be to just replace the old Tokenizer with the one we create for specialComment?
• 136: Initial implementation of fulltext with indexes and DDL
  This pr
  1. Adds fulltext parsing support for indexes and DDL.
  2. Formats the repo
• 135: Allowing Event Keywords to be used as column name, table name, variable name, etc
• 134: Get Closer to MySQL's Behavior for Keywords (Reserved and Non-Reserved)
  I doubt I'll be able to get all of the keywords working immediately.
  This PR focuses on getting as many of the keywords in and working, and documenting what doesn't work yet.
• 133: reserve found and allow as column safe
• 131: Table function support
  Add support for table_functions as a new type of table factor in the dolthub/vitess SQL grammar.

Closed Issues

• 2854: Syntax error when using event as column alias
• 2964: Use a helper function to handle errors returned by functions in a defer statement?
• 2968: value out of range - insert 0x... into binary
• 2977: Composite Primary Key doesn't seem to work
• 2462: Support -a in dolt tag command
• 2960: Runtime error: invalid memory address or nil pointer dereference error when cloning a Dolt repository

Merged PRs

dolt

• 2961: Fix error muting by defer and excessive memory allocation in newBSChunkSource. Fix localbs scheme using file scheme.
  Root issue is that blobstore.GetBytes returns a byte buffer whose capacity is too large. That buffer is passed to parseTableIndex which takes ownership of that buffer.
  In Dolt v0.37.0, parseTableIndex was changed to throw an error if the capacity or length of the given buffer is too large. This was done to prevent large memory usage by Dolt. Since the above buffer was too large, parseTableIndex would throw an error and the chunkSource would fail to initialize. Two other issues prevented this error from being caught.
  This bug was not caught prior to Dolt v0.37.0's release because errors returned in newBSChunkSource were being overridden by a defer. This muted any errors returned by newBSChunkSource including the error that parseTableIndex was throwing.
  Additionally, the bats tests for the localbs scheme were using the file scheme under the hood. This prevented the tests from catching this bug.
  This PR fixes the above issues and closes #2960
• 2958: Escape the given tablename in the engine_table_reader
  Fixes #2949
• 2956: store/datas: CommitOptions: Remove types references.
• 2954: performance/utils/sysbench_runner: fix dropped errors
  This fixes three dropped err variables in performance/utils/sysbench_runner.
• 2953: Bump GoMod and Trim the sql-use.expect
  ☕ An Automated Dependency Version Bump PR 👑

  Initial Changes

  The initial changes contained in this PR were produced by go geting the dependency.

  $ cd ./go
  $ go get github.com/dolthub/<dependency>/go@<commit>
  

  Before Merging

  This PR must have passing CI and a review before merging.
• 2950: Skipped bats test for updating tables with dashes in the name using dolt table import -u
  Issue #2949
• 2946: go/libraries/doltcore/dbfactory: aws.go: Support / as a character in database names for AWS remotes.
  This was previously forbidden because of how table files were stored in the
  destination bucket, but that was changed long ago and this is now well
  supported. Our internal remotes serving infrastructure uses chunk stores
  constructed in the same way.
• 2942: Fix Pk ordinal delete bug
  Consider a table with a primary key defined in a different order than the schema:

  > create table a (x int, y int, primary key (y,x);
  > insert into a values (0,1), (2,3);
  Query OK, 1 row affected
  

  Deletes were matching rows in the index and correctly tracking the affected row count, but generating intermediate schema-order key tuples incompatible with the table PK order. The incompatible keys failed to match the source keys in the roundtrip as the edits were applied to the table:

  > delete from a where x = 0;
  Query OK, 1 row affected
  > delete from a where x = 0;
  Query OK, 1 row affected
  > select * from a where x = 0:
  +---+---+
  | x | y |
  +---+---+
  | 0 | 1 |
  | 2 | 3 |
  +---+---+
  

  i.e. we were matching (1,0) and then trying to delete (0,1).
  This fixes the intermediate row key tuple to match the pk order. Test in https://github.com/dolthub/go-mysql-server/pull/854.
• 2940: Grammar fixes
• 2938: go/libraries/doltcore/sqle: Remove some dependencies on pkg types
  This PR
  • refactors the dolt_schemas and dolt_procedures tables to remove direct types.Map dependencies.
  • refactors table to remove direct types.Map dependencies.
  • modifies DoltIndex to only push down filters for the current format, not for prolly indexes.
  • skips unsupported engine tests for the new format so that the engine suite can be run without panicking
• 2930: Prevent Multiple Not-NULL Constraints
  Removed code where we would add an additional NOT NULL constraint when dropping primary keys.
  Should be impossible to add multiple NOT NULL constraints to a single column, as there are checks right before anywhere we append a NOT NULL for if there already is NOT NULL constraint.
  Added a method in schema marshalling that'll remove all duplicate not null constraints.
• 2929: go/store/datas: Begin work for moving top-of-DAG chunks to flatbuffers messages.
  This starts on the changes necessary to implement __DOLT_1__ format with flatbuffers messages representing top-of-DAG entities like StoreRoot, WorkingSet, Commit, Tag, RootValue, etc. This particular PR creates machinery to move StoreRoot, WorkingSet and Tag to simple flatbuffer message representations. This PR includes:
  • Move CommitMeta, TagMeta and WorkingSetMeta to store/datas package.
  • Move accessors in doltcore/doltdb of WorkingSet and Tag heads to use new HeadTag and HeadWorkingSet methods on the Dataset interface instead.
  • Add a new __DOLT_DEV__ format which uses these top-of-DAG flatbuffer messages.
  • Add serial.StoreRoot message which stores an un-chunked RefMap of roots.
  • Add serial.Tag message which stores the tag metadata and the hash of the commit.
  • Add serial.WorkingSet message which stores the working set metadata, the hash of the current root value, hash of the staged root value and hash of the merge state.
  • Add a types.SerialMessageKind and types.SerialMessage type, which is implemented like InlineBlob but can also introspect the serialized message to WalkRefs the addresses of it.
• 2924: Change HedgeStrategy interface to control nextTry durations directly
  Strategy has been renamed to HedgeStrategy. HedgeStrategy's now control the nextTry durations directly and an ExponentialHedgeStrategy has been added that can be composed with other strategies to get the previous behavior.

go-mysql-server

• 862: fix misspelled name in information_schema.innodb table
• 861: have methods for dolt to call to parse check expression
  To properly merge check constraints, Dolt needs to partially parse their expression strings.
  This PR gives Dolt access to some helper methods, to complete this operation. Should have no functional effect on GMS.
• 860: Add all non INNODB information_schema tables.
  This PR adds a bunch of information_schema tables that were previously missing. All are initialized with an empty row iter.
• 859: add all information_schema.innodb_* tables as empty tables
• 856: Only skip updating defaults when added column has no default value AND is nullable
  Fix for bad PR #855 which fails the following Dolt test:
  http://github.com/dolthub/dolt/blob/51987ab84ef9d07b3c6ceae3ab65b9ca65ab55bd/go/libraries/doltcore/sqle/sqlddl_test.go#L482-L482
  Verified this PR passes Dolt tests.
  Also, removes TestAddAndDropColumn test which will be replaced with an appropriate bat test on Dolt.
  Related to https://github.com/dolthub/dolt/issues/2939
• 855: Don't apply defaults to all rows when the added column has no default
  Related to https://github.com/dolthub/dolt/issues/2939
  When a user was working with a large table, adding a column with no default caused an OOM crash. The crash is a result of the ADD COLUMN sql statement causing every row to be modified in the table. This is not-expected and a bug. We expect adding a column with no default to only modify the schema and not any Dolt row.
  The fix is to skip applying defaults to table if the added column has no default. We can call this a "fast" add column since only the schema has to be modified.
  In the future, we may also add a "fast" drop column where only the schema is modified. Currently, we modify every row for a DROP COLUMN statement. In the "fast" drop scenario, it might be important to ensure values aren't retained across ADD COLUMN and DROP COLUMN statements with the same column name. This PR also adds tests to ensure this.
• 854: Pk ordinals delete test
• 853: removing routine_definition column from information_schema.routines table in enginetest as it gives different values in GMS and Dolt
  The value in routine_definition comes out different:
  • in Dolt, Exchange(parallelism=3)\n └─ Project(6)\n └─ Table(dual)\n
  • in GMS, Project(6)\n └─ Table(dual)\n
• 852: add information_schema.routines table
• 850: Added sql.FilteredIndex to allow integrators to specify index filter pushdowns
• 848: Add FLUSH PRIVILEGES support

vitess

• 133: reserve found and allow as column safe
• 132: Add FLUSH as a command and its options

Closed Issues

• 2960: Runtime error: invalid memory address or nil pointer dereference error when cloning a Dolt repository
• 2859: Support FLUSH PRIVILEGES syntax
• 2949: dolt table import cannot handle table names with - in them
• 2879: SELECT 1 as found breaks
• 2462: Support -a in dolt tag command
• 2858: Implement the information_schema.routines table
• 2927: Columns can have more than one not-null constraint in storage
• 2939: Error altering table - high memory usage and process is killed
• 2919: 3-way schema merge creates duplicate columns
• 2890: Keyless Tables Do Not Support Rollback Semantics
• 1390: document the format of s3 remote strings

Merged PRs

dolt

• 2934: integration-tests/transactions: Added Concurrent Transaction Tests
• 2933: go/utils/remotesrv: use io.Copy in the http file server to write table file data to http.ResponseWriter

  Problem

  This PR fixes an issue when pulling chunks from Dolt remotesrv. The symptom is a throughput below minimum allowable error. The problem is that the dolt client can't handle large http chunks without failing to meet its minimum required download throughput.
  Dolt remotesrv hosts a simple http file server that can be used to write new table files and read existing table files. When remotesrv receives a request to read a file it will currently read all of the requested range into memory. It then, writes the
  entire range to http.ResponseWriter in a single Write call.
  This causes a very large http chunk (see chunked transfer encoding) to be sent over the wire. When the Dolt client tries to read the http response using Read, it takes a long time to return. Due to the way the Dolt client measures Read throughput, if a single Read takes long enough, the minimum required throughput will not be met.

  Solution

  We fixed this issue by using io.Copy to read from the file and simultaneously write to the http.ResponseWriter. io.Copy produces small enough writes to the response writer. This reduces the number of bytes read during a singular Read call on the Dolt client and allows the throughput measurement to be accurate.
  We could have fixed this problem by changing the way the minimum required throughput was measured, but the above change reduces the memory overhead of remotesrv and uses an idiomatic Golang pattern.

  Why doesn't this issue appear when pulling chunks from S3?

  In Golang, http.Response.Body maximum Read sizes are determined by the Content-Length and whether chunked transfer encoding is being used. The exact semantics can be investigated by reading the sources here:
  • net/http parseTransferEncoding
  • net/http chunkedReader
  • net/http chunkedWriter
    Since S3 returns a Content-Length and does not use chunked transfer encoding each Read call returns in a small enough size.
• 2925: add parquet to dump options
• 2923: Column union works like a set union
• 2912: SQL dump now has --batch option for batched INSERTS
• 2901: Edit Stats display in Import and fix stats computation
  This pr
  1. Fixes the formatting of stats and allows commas to be printed
  2. Fixes an error with stats computation where many queries were getting marked as modifications.
  3. Bumps GMS to add in a fix with dropped insert ignore errors
• 2900: Added skipped bats for dolt version requiring write perms
• 2895: Allow Rollback on Keyless Tables
  Delete all pending changes when an error occurs.

go-mysql-server

• 851: add collation_character_set_applicability table in infoSchema
• 850: Added sql.FilteredIndex to allow integrators to specify index filter pushdowns
• 846: analyzer: Collect all range expressions for multi-column indexes
• 845: Fix InTuple optimizer bug
  Last week we fixed a dropped optimizer error return that was masking a
  tuple bug. We expected a slice of interfaces, but single element tuples
  return one value. We catch this in most circumstances, but missed the
  multicolumn case triggered by an AND expression. The error was passing
  logictests because we still return correct results even if we fail to
  push a filter into a table scan.
• 842: /.github/workflows/bump-dependency.yaml: pin go version
• 841: Bump vitess while automatic bumping is broken
• 840: /.github/workflows/bump-dependency.yaml: add GOOS env
• 839: server: Refactor handler to use buffered channel
• 837: Add IGNORE behavior for CHECK CONSTRAINTS and on duplicate key
  Fixes an issue where INSERT IGNORE was not correctly ignoring check constraints
• 836: fix set serialization
• 833: Fixing a race condition by no longer updating the UserVar's type during Eval
  The type was being set when retrieved in Eval for completeness, but it shouldn't be needed.
  When used as a value, the analyzer was already using the NewUserVarWithType method that explicitly loaded and set the type. The NewUserVar method (which doesn't set type) should only be used when the user var is the left hand side of a set var operation.
• 832: add processlist table to information_schema
  The result is the same as current SHOW PROCESSLIST query result
• 829: Added additional privilege tests
  Added new testing structure for privileges I'm calling "quick tests". A test can take up 6 lines rather than 18, which is a big difference when there's 100 of them. Plus they're quicker to write this way. They're the same is normal tests though.
• 825: add information_schema.statistics table
  Cardinality is currently set to total number of row in the table instead of an estimate number of unique values in the index

vitess

• 132: Add FLUSH as a command and its options
• 130: add PROCESSLIST to non-reserved-keyword list
• 129: Adding scripts to test parsing behavior of MySQL
• 128: /go/bucketpool: Optimized findBucket
• 127: Add testcases for exercising keywords and remove some create user tests
  This PR:
  1. Allows the COMMENT_KEYWORD to be alias by selects/
  2. Removes some usages of aliased keywords in the select queries
  3. Adds skipped tests that exercises the validity of reserved and non reserved keywords through a series of tests
• 126: go/mysql: Add buffering to prepared statement result sets

Closed Issues

• 2865: dolt dump and dolt diff --sql should feature a --batch flag
• 2919: 3-way schema merge creates duplicate columns
• 1843: Bad error message when database is unselected
• 2615: dolt table import -u <table> shows incorrect processing information for us-housing-prices
• 2630: Support flag that skips over printing skipped rows in import
• 2814: Syntax error when use 'Comment' as column alias
• 2310: [Feature Request/Bug] Add processlist to information_schema table
• 2856: Add details to information_schema.statistics table.
• 2890: Keyless Tables Do Not Support Rollback Semantics
• 2897: can not exit after execute dolt login
• 2692: SHOW CREATE TABLE does not respect primary key ordinals

This is a patch release. It fixes a bug in displaying the primary keys of a table via SHOW CREATE TABLE

Merged PRs

dolt

• 2830: /go/store/prolly: Add Prolly Node array with subtree sizes
  • adds new field to prolly tree nodes to store a varint-encoded array of subtree cardinalities
  • alters tree chunker to track subtree cardinalities during tree construction
  • splits out node construction logic into nodeBuilder
    the varint encoding scheme should probably be replaced by something more performant like streaming vbyte

go-mysql-server

• 833: Fixing a race condition by no longer updating the UserVar's type during Eval
  The type was being set when retrieved in Eval for completeness, but it shouldn't be needed.
  When used as a value, the analyzer was already using the NewUserVarWithType method that explicitly loaded and set the type. The NewUserVar method (which doesn't set type) should only be used when the user var is the left hand side of a set var operation.
• 828: Correct primary key column output for show create table
  Used the column order before, instead of the key order
• 823: sql: Added destination buffer parameter to Type.SQL()
• 819: fix BETWEEN function of missing inclusive logic
  BETWEEN() does inclusive check including checks with NULL value

Closed Issues

• 2869: Hanging/memory leak on importing a single row to a specific branch
• 815: Index error lost in parent call

This is a patch release, containing minor features, bug fixes and performance improvements.

It adds the following features:

• dolt_diff system table now indicates whether a table in a diff had data changes, schema changes, or both
• dolt_diff_table_name system tables have simpler schemas in the presence of historical schema changes to the table
• SHOW STATUS LIKE now parses
• Faster table scans when using an index
• Named windows can now be used

It addresses the following bugs:

• Slow foreign key check logic when no value was changed
• NULL values in dolt dump
• Duplicate indexes can now be created
• Clone and pull commands now correctly use "downloading" language, better status output
• Bug fix for BETWEEN expressions with a NULL bound
• DDL statements that failed when no database was selected now work

Merged PRs

dolt

• 2888: don't loop through all fks, if change doesn't impact any fk columns
• 2875: fix null value being exported for any types into .sql file
  the issue covered all typeInfo types where null values were not getting exported correctly.
  the fix is to add NULL whenever the value is nil and cannot be converted into sql types.
• 2867: go/store/nbs: Add OverwriteStoreManifest to nbs package
  In some cases, it may be desirable to add tables files to the manifest without opening the new table files as chunkSources. For large clone operations, it can be problematic to load the table file chunkSources as this causes their indices to be loaded into memory. In order to reduce the memory overhead of a clone, this PR adds OverwriteStoreManifest.
  OverwriteStoreManifest adds table files to the manifest file and persists it. It will not call Rebase, unlike UpdateManifest.
  Care should be taken when using OverwriteStoreManifest as old table files will not be garbage collected. Overwriting the manifest multiple times may allow the store to grow arbitrarily large in size.
• 2864: doltcore/sqle: Optimize Delete
  Fetching and updating auto increment values is relatively expensive given the current implementation. Conditionally skipping this step makes deleted 2x faster
• 2857: Removing use of SuperSchema from Dolt_Blame system view.
  Quick change to remove the SuperSchema reference from Dolt_Blame system view, now that we have migrated Dolt_DIff_$tablename off of SuperSchema's tag-suffixed column names.
• 2851: libraries/utils/editor: fix dropped error
  This fixes a dropped err variable in libraries/utils/editor.
• 2848: unskip dropTable bats test
• 2844: go/store/prolly: Faster table scans
• 2843: Expand DOLT_DIFF system table to indicate the type of change
  This change adds two new boolean columns to the unscoped DOLT_DIFF system table to enable customers to determine if a change to a table was a schema change or a data change (or both).
  Example Usage:

  > create table x (a int primary key, b int, c int);
  > create table y (a int primary key, b int, c int);
  > insert into x values (1, 2, 3), (2, 3, 4);
  > select DOLT_COMMIT('-am', 'Creating tables x and y');
  > select * from dolt_diff;
  +----------------------------------+------------+-----------+-------------------------+----------------------------------+-------------------------+-------------+---------------+
  | commit_hash                      | table_name | committer | email                   | date                             | message                 | data_change | schema_change |
  +----------------------------------+------------+-----------+-------------------------+----------------------------------+-------------------------+-------------+---------------+
  | 1blnkur3m1hla2a4got9t513982a4ad1 | x          | jfulghum  | [email protected] | 2022-02-21 14:39:37.01 -0800 PST | Creating tables x and y | true        | true          |
  | 1blnkur3m1hla2a4got9t513982a4ad1 | y          | jfulghum  | [email protected] | 2022-02-21 14:39:37.01 -0800 PST | Creating tables x and y | false       | true          |
  +----------------------------------+------------+-----------+-------------------------+----------------------------------+-------------------------+-------------+---------------+
  

  Resolves: https://github.com/dolthub/dolt/issues/2834
• 2840: go/dolt/cmd: Use download language when dolt pulls from a remote
  Fixes #2798
  For downloads (pulling chunks from a remote db to a local db), dolt will now report the number of tables files written to disk as Files Written. It will no longer report an Upload Rate for a download as that is the speed at which the file is copied from a temp directory.
  For uploads (pulling chunks from a local db to a remote db), dolt will now report the number of table files created during the pull process as Files Created and the number of these tables files that have been uploaded as Files Uploaded.
• 2835: go/store/val: Remove NULL bitmask from new tuple format
• 2833: store/types: fix dropped error
  This fixes a dropped err variable in the store/types package.
• 2832: Simplify DOLT_DIFF_$TABLE system tables by removing tag suffixes
  When the DOLT_DIFF_$TABLE system tables show schema history with column name conflicts, the column names are disambiguated by adding their unique tags as suffixes. This makes it difficult to work with these system tables. This change simplifies the output by restricting the output schema to be based on the current table schema and avoids any column name conflicts.
  Resolves: https://github.com/dolthub/dolt/issues/1910 (for dolt_diff_$table, but does not change `dolt_commit_diff_$table yet)
• 2831: go/store/val: Added fast field access for fixed size tuple fields
• 2829: unskip fixed tests
  • un-skipped some tests that are resolved
  • fixed some test typos
  • added test for add foreign key constraint bug
• 2828: Remove index filters
• 2826: Add the ignore skipped row param
  Adds a flag to skip rows that are printed by --continue.
• 2825: Removing Duplicate Index Constraint
• 2822: go/cmd/dolt: Improve progress reporting of clone
  Previously, it was difficult to understand the progress of dolt clone, especially when you are cloning a db with a small number of large table files. dolt clone now lists which table files are being concurrently downloaded and shows the progress and download rate for each.
  An example:
  
  @reltuk Brought up the concern for compatibility between the new uilive dep and github.com/faith/color. In my testing there doesn't seem to be any issues:
  
• 2812: Modify Benchmark Configuration for End to End support of the TPCC benchmark
• 2789: fix dolt log pager panics with ctrl+c on Windows
  added catching Interrupts for dolt log as it panics with Ctrl+C on Windows with more command

go-mysql-server

• 827: fix ROLLBACK; fail with no database selected
  MySQL lets ROLLBACK when no database is selected
• 826: Don't call ProcessList.Update on every row spooled in a result set by default
• 824: analyzer doesn't error for invalid value
  re: https://github.com/dolthub/go-mysql-server/issues/815
  Originally we were failing to handle an analyzer error. Catching that error exposed a case where we want to manually prevent that error path from bubbling up.
• 820: Add uint types to LiteralToInt
  Re: issue https://github.com/dolthub/go-mysql-server/issues/799
  This database and this query were parsing 128 as a uint8, which my type switch missed:

  $ dolt clone https://doltremoteapi.dolthub.com/post-no-preference/stocks
  $ cd stocks
  $ dolt sql
  stocks> select date, act_symbol, avg(close) OVER (PARTITION BY act_symbol ORDER BY date ROWS BETWEEN 128 PRECEDING AND CURRENT ROW) AS ma200 FROM ohlcv WHERE act_symbol='AAPL' having date = '2022-02-11';
  offset must be a non-negative integer; found: 128
  

  After fix:

  select date, act_symbol, avg(close) OVER (PARTITION BY act_symbol ORDER BY date ROWS BETWEEN 128 PRECEDING AND CURRENT ROW) AS ma200 FROM ohlcv WHERE act_symbol='AAPL' having date = '2022-02-11';
  +-------------------------------+------------+--------------------+
  | date                          | act_symbol | ma200              |
  +-------------------------------+------------+--------------------+
  | 2022-02-11 00:00:00 +0000 UTC | AAPL       | 158.29837209302272 |
  +-------------------------------+------------+--------------------+
  

  I haven't been able to create a testing database with the same type parsing behavior yet. Something about the stocks database or that specific query is yielding a types.Value with value=128 and type=sql.Uint8.
• 819: fix BETWEEN function of missing inclusive logic
  BETWEEN() does inclusive check including checks with NULL value
• 817: Changing UserVar to return its correct type
  Changing UserVar to return its correct type, when used as a value in an expression.
  Resolves: https://github.com/dolthub/go-mysql-server/issues/790
• 816: /go.{mod,sum}: bump vitess
• 813: /sql/analyzer: Added feature flag for single threaded execution
• 811: fix DropTable enginetest
  fix DropTableTest in enginetest, was causing failed test in Dolt
• 810: Extending script queries to allow easier testing of warning messages
  This change lets you specify the expected number of warnings and an expected substring from test queries. It also tidies up some of the logic around running tests with expected warnings.
  Example test syntax this allows you to write:

  Assertions: []enginetest.ScriptTestAssertion{
  {
  Query:                           "select * from dolt_diff_t;",
  ExpectedWarning:                 1105,
  ExpectedWarningsCount:           4,
  ExpectedWarningMessageSubstring: "unable to coerce value from field",
  SkipResultsCheck:			true,
  },
  },
  

  Needed for: https://github.com/dolthub/dolt/pull/2832
• 809: query plan test for duplicate indexes
• 806: Deduplicate IndexLookup filters
  Opportunistically eliminate filter expressions when
  sql.IndexedTableAccess is used for a table scan. The original logic
  respecting join scoping for applying indexes is unaffected, this simply
  connects out current filter elimination logic with the expressions used
  for range scan.
  This requires integrators implementing sql.IndexLookup to only return
  rows selected by the lookup filter.
  Changes were made to ranges and datetime filters To allow integrators
  to comply to this filtering standard. GMS can now distinguish between
  NULL and infinity. Tests are added for IS NULL and IS NOT NULL handling
  for single and multi column expressions. Engine datetimes are not truncated
  to support date index lookups with the same precision as MySQL.
  We currently fail to eliminate filter inversions. More work is needed
  to compare inverted filters to their range scan representation.
• 804: Revert "addColumn"
  This reverts commit f64b71875501714d69ed9b80c38e2564467d2794.
• 803: Add NamedWindows Node
  This moves window name validation, resolving, and embedding out of
  parsing and into the analyzer.
• 802: Add vitess to go mod to support SHOW STATUS LIKE
• 801: Added external functions for persisting Grant Tables
  This adds the functions necessary to be able to easily persist the data from Grant Tables as JSON for an integrator.
• 794: Fix drop table from different database
  Added DropTable case in getTransactionDatabase
  Added tests to check dropping table in out of scope database
  Changed ddlNode to CurDatabase that is update with every table being dropped
  Changed taking array of tables as string type to plan.UnresolvedTable type
  Updated resolve-table rule in analyzer to support DropTable filtering out non-existent tables.
• 789: fix alter table columns in different database
  Removed ddlNode and added tests for AddColumn , DropColumn , RenameColumn , DropColumn that check updates with different database tables than current one.
  Added cases in engine.go for updating the transaction database for some plans including above and CreateIndex , AlterIndex

vitess

• 126: go/mysql: Add buffering to prepared statement result sets
• 121: Add show like status
• 120: Named Windows
  Add yacc syntax for window names. WindowDef is used with either its Name
  field in a WINDOW clause, or NameRef in an OVER clause. NameRef is OVER
  is not mandatory, but Name in WINDOW is.
  This PR adds precedence rules to avoid shift/reduce conflicts between
  the WINDOW name field, which can be any string, and RANGE, PARTITION,
  ROWS, and RANGE should be prioritized for other rules if they appear as
  the first token in a window_spec.

Closed Issues

• 2860: Bad error with rollback on unselected database
• 2874: dolt dump - Error with dumping
• 2855: Panic on information_schema.tables
• 2674: UPDATE in trigger statements don't work with multiple rows
• 2729: Allow duplicate indexes to be created
• 2834: Expand DOLT_DIFF system table to indicate the type of change
• 2824: Unexpected --continue behaviour for dolt table import -u
• 2783: Support `SHOW SESSION STATUS LIKE...' syntax.
• 2798: Dolt fetch, pull, etc. use "upload" language
• 1910: Schema changes for dolt_diff_$tablename and dolt_commit_diff_$tablename
• 2779: Panic on SIGINT during commit log
• 2598: Dolt ignores schema name in DROP statements
• 2764: Some DDL statements can't be chained in a single session but work when applied in multiple sessions
• 2821: New flag to suppress skipped rows when using --continue flag with dolt table import
• 2753: cli arguments should optionally override config values
• 2802: Every SQL Commit is a Dolt Commit
• 815: Index error lost in parent call
• 790: String variables aren't evaluated correctly in where clauses
• 799: Aggregate Partition Window Rows beyond 127
• 146: regex-metrics can't be initialized
• 171: Two foreign keys in same CREATE TABLE statements triggers error
• 355: Question/Feature Request: How can I increase the parallelism of expression evaluation?
• 654: fatal error: concurrent map write read
• 705: error: can't modify database databaseProvider
• 709: create table with unnamed key got error ERROR 1105 (HY000): Error: index already exists
• 191: Support for VALUES in INSERT ... ON DUPLICATE KEY UPDATE
• 250: unsupported syntax: create database mapping
• 313: How can I custom my log.
• 481: Anyone working on the replication?

Merged PRs

dolt

• 2813: Config users are ignored if privilege users are present
  Config users were always being added. This has been fixed.

Closed Issues

This is a major feature release.

• Dolt now supports users and roles, with the ability to grant and revoke privileges on databases and tables.
• dolt log now supports the --oneline and --decorate options.
• @@dolt_transaction_commit now works as a global as well as a session variable.

Merged PRs

dolt

• 2811: Aaron/database remove map from datasets
• 2810: Added privilege persistence
  This adds the ability for users and roles to be persisted, through a YAML option or a CLI argument, to a JSON file.
• 2807: Making dolt_transaction_commit a global variable
• 2801: go/store/nbs: reduce onHeapTableIndex memory overhead
  Changes the way onHeapTableIndex is implemented. Instead of parsing the given byte buffer into new prefixes, ordinals, lengths, and suffixes slices, we now compute directly on the given buffer taking ownership of it.
  Creating an onHeapTableIndex before this PR:
  • Put the table file index bytes at the end of a byte slice and pass it to parseTableIndex
  • Footer is read to determine chunkCount
  • Byte slice is used to compute offsets from the lengths. Then parsed prefixes, ordinals, lengths, offsets, and suffixes are stored as new slices on the heap.
  • Byte slice reference is dropped
    Creating an onHeapTableIndex after this PR:
  • Put the table file index bytes at the end of a byte slice and pass it to parseTableIndex
  • Footer is read to determine chunkCount and chunkCount is used to determine total size of the index. The total size is used to slice away any additional bytes given at the front of the buffer. onHeapTableIndex retains a slice of just the index data (without the footer).
  • New space is allocated to store offsets. OffsetsReader is used to convert the provided chunk lengths into chunk offsets.
  • Byte slice reference is retained
    As a result of this change, mmapTableReader was broken. Since the new index takes ownership of the buffer, as soon as mmapTableReader unmapped the index file from memory, a segfault was thrown. mmapTableReader has instead been changed to copy index bytes into the heap.
• 2793: go/store/{geometry, prolly, val}: Remove SQL dependencies from and cleanup pkg val
  • lots of shuffling in go/store/val/codec.go, tried to make it easier to read
  • improved (de)serialization of time.Time
  • added pkg geometry to share serialization logic between types and val
  • added (de)serialization of geometry types for the new format
  • moved SQL dependencies out of go/store/val/ in to sqle/index
  • added val.SlicedBuffer and used it to cache pointers in prolly.Node
• 2788: Add errors to tableIndex interface funcs
  In preparation for a disk tableIndex, I altered the tableIndex functions to return errors and fixed any code that needs to return errors.
  The Ordinals() and Prefixes() functions have also been changed to return errors, but for TOW I'm not sure we can return an entire slice of all ordinals or prefixes without blowing up.
• 2781: Implementing missing git log features for dolt log
  • --oneline option
  • show refs and tags next to commits
  • --decorate option
  • short
  • long
  • auto
  • no
  • show tags next to commits
• 2778: go/store/prolly: 3-way Merge for Prolly Trees
• 2774: Add Generational Chunk Store for Noms Show
  Edits the default chunk store for noms as the GenerationaCS
• 2773: go/store/datas: Remove ValueReadWriter from Database interface; make Database interface smaller.
  Moving VRW out of the Database interface means users need to keep a VRW and a Database reference in sync and call the Database once the correct VRW has the data in it.
• 2762: Add the TPCC benchmark and ensure that it runs correctly in Docker automation
  1. This PR creates a runner that executes against the TPCC benchmark. It can be run locally with the sample config provided
  2. This PR edits the SysbenchDocker file to run this benchmark so the CI smoketest for all benchmarking executes correctly
• 2752: libraries/doltcore/doltdb: fix dropped errors
  This fixes a dropped code error and multiple test errors in libraries/doltcore/doltdb.

go-mysql-server

• 797: Added privilege checks for every currently supported statement
  This adds privileges for every supported statement.
• 796: Framework for RowIter2 and other new data representation interfaces
  • Implement Table2 for memory tables
  • Several low-level node types converted to Node2
  • Special top-level iterator to switch iteration type based on whether the tree is all Node2 or not

Closed Issues

• 2806: @@dolt_transaction_commit can't be used as GLOBAL var, only session
• 2805: Blake2 godep breaks build on m1 mac

This is a feature release.

A new system table, dolt_diff, that contains which tables changed between two commits, is now supported.

A new dolt_blame system view which contains who last edited each row of a table is now supported.

The default behavior for SQL COMMIT statements in the presence of merge conflicts has changed. They are now allowed by default.

Named windows are now supported.

It also includes bug fixes:

• dolt login ignores interrupt signal
• Bug for --set-upstream on push
• Buggy behavior involving subqueries in triggers

Merged PRs

dolt

• 2797: add bats test for different db stmts
  Added tests for using different database than current for ALTER TABLE - CREATE INDEX and ADD CONSTRAINT.
  Added tests for ADD CONSTRAINT FOREIGN KEY query should be failing, but currently does not.
• 2786: window enginetests
• 2784: add bats test for DROP TABLE for different database
  added bats test for "DROP TABLE otherdb.table;" statement
• 2782: Fixing schema ordering to unbreak tests
  I added the table_name column as part of the primary key, then changed the order, not realizing that would mess up the test data.
• 2780: go/cmd/dolt/commands: Added dolt inspect utility for investigating table file indexes
• 2777: Fix auto increment imports
  Adds functionality and tests that allow dolt table import to correctly import auto increment data.
• 2776: removing feature flag
• 2775: Fix Dolt Login unable to exit on SIGINT
  TODO:
  • should also quit on error request cancelled
    SIGINT is what cancels request, so whole thing should just quit anyway.
• 2772: Adds IndexTransformer to nbs which transforms a table file index with lengths to one with offsets
  IndexTransformer is an io.Reader that takes an io.Reader of the table file index. It specifically transforms the lengths into offsets.
• 2771: Bug fix for an index out of bounds panic in DOLT_HISTORY_
  Panic occurs when selecting from DOLT_HISTORY_<TABLE> using a filter expression and the underlying table that doesn't have a primary key.
• 2769: New DOLT_DIFF system table
  Adds support for a new DOLT_DIFF system table that is not scoped to a specific user table; for each commit, this table includes one row for every table changed in that commit.
  Sample Output:

  > SELECT * FROM DOLT_DIFF;
  +----------------------------------+-----------+-------------------------+-----------------------------------+-------------------------+------------+
  | commit_hash                      | committer | email                   | date                              | message                 | table_name |
  +----------------------------------+-----------+-------------------------+-----------------------------------+-------------------------+------------+
  | edr1ichcj8vpve9lofv31e7taiajv3uu | jfulghum  | [email protected]       | 2022-02-07 13:39:47.717 -0800 PST | Creating tables z       | z          |
  | 91ff8so7alcnuiq0qa1qq9e7o58i7jdb | jfulghum  | [email protected]       | 2022-02-07 13:39:26.143 -0800 PST | Creating tables x and y | x          |
  | 91ff8so7alcnuiq0qa1qq9e7o58i7jdb | jfulghum  | [email protected]       | 2022-02-07 13:39:26.143 -0800 PST | Creating tables x and y | y          |
  +----------------------------------+-----------+-------------------------+-----------------------------------+-------------------------+------------+
  

  An unscoped DOLT_DIFF felt more appropriate than an unscoped DOLT_HISTORY table since users will likely use DOLT_DIFF, then use a scoped DOLT_DIFF_<$TABLE> to look at changes in each table. DOLT_HISTORY_<$TABLE> also has semantics of showing the full history at each commit, where this table is intended to show which tables changed (not which tables existed at a commit).
  Resolves: https://github.com/dolthub/dolt/issues/2740
• 2767: go/store/prolly: Size validation for prolly tree nodes, Speedup unit tests for pkg prolly
• 2766: go/store/prolly: diffs for new prolly tree
• 2754: DOLT_BLAME system view support
  Initial support for the new DOLT_BLAME system view.
  Resolves: https://github.com/dolthub/dolt/issues/2265
• 2750: Fix --set-upstream bug when feature branch already exists on remote
  We were missing a test for push --set-upstream where
  the feature branch had already been pushed from another
  clone.
• 2742: By default allow transactions to commit working sets with conflicts. Also fix DOLT_MERGE --abort
  This pr introduces a new session variable that allows for transactions by default commit working sets with conflicts.
  It also adds a fix for DOLT_MERGE abort which was broken before.
• 2731: go/libraries/doltcore/sqle: Added support for secondary prolly indexes

go-mysql-server

• 793: Add type checking to auto increment
• 792: Named Windows
  Build WINDOW clause definitions from AST into plan.Window nodes.
  Coversion currently happens in parser. This breaks the separation of
  concerns between parsing and semantic validation to avoid creating new node
  types to differentiate between Over windows and Window clause windows.
  A proper refactor probably requires changing the the parse/exec
  representations for all related nodes to carry info from the parser
  further before semantic validation.
• 791: Database and table name visibility affected by privileges
  This limits the visibility of database and table names depending on the privileges associated with an account (whether directly on via an active role). In addition, all engine tests now run with grant tables enabled. Of course not all statements are checking for their respective privilege, but all of the other logic paths are being exercised (especially the new PrivilegedDatabase and ...Provider constructs).
  I removed some driver implementation as it was literally broken (errors in the file on main) and was obviously not relied on by anything (not even tests), so instead of fixing it for the changes I made, I just removed it altogether.
• 786: Added all static global, database, and table privileges
  This adds all static global, database, and table level privileges. They parse and persist but do not yet restrict on anything. Future PR will dive into the full set of privilege checking.
  One major point to note (which will probably bring about confusion otherwise) is that these new tables are implemented as essentially a "view" on top of the mysql.user table. This will significantly help when it comes privilege checking, especially in the context of determining active roles and such. To complement this, PrivilegeSet was almost entirely rewritten. The related files are data_editor_view.go and grant_table_shim.go.
  With the original direction, I could have 10 different table queries and 10 sets of privilege data to manage with only 2 roles. This way, I'll only have 3 queries and 3 sets of privilege data in the same scenario.
• 785: No longer mark subqueries in triggers as cacheable
  Resolves issue https://github.com/dolthub/dolt/issues/2674
• 783: recursive CTEs
  RecursiveCte node impl with tests for joins, group by, and subquery
  aliasing.
  RecursiveCte's are not supported in DML statements. Nested recursive
  CTE's and nested subqueries are not supported yet. Aliasing is not
  neatly split down the middle on either side for RCte unions; only the
  dual table can have alias clashes on either side of the CTE for now.
  Cyclical recursion will timeout at depth 20. This will likely need a
  rewrite with better global aliasing to support infinitely nestable
  RecursiveCte's.
  Indexed table lookups are not applied to RecursiveCte tables currently
  because of a bug in the memory impl.
• 782: migrate invalidArgument in time.go to errors.go

vitess

• 119: partitionBy and overBy optional, one mandatory with frame def
• 118: Added remaining privileges
  This adds the remaining privileges.
• 117: recursive CTEs

Closed Issues

• 2621: dolt login is unquittable
• 2785: Support: is there a query to extract file content on a specific date?
• 2265: dolt blame in SQL interface
• 2740: Unspecified table dolt diff and dolt_history should show the history of all the tables per commit
• 2682: Some DDL statements without a database selected don't work
• 2758: No TLS on doltlab.com
• 2649: Expression parsing bug where parser can't find a column that exists
• 2719: [Feature Request] Support with recursive

This is a feature release, adding initial support for spatial types. Read more here:

https://www.dolthub.com/blog/2022-02-09-spatial-types/

Also newly supported are recursive common table expressions (WITH RECURISVE).

This release also addresses a number of bugs:

• dolt incorrectly dropped a not null constraint when dropping a primary key
• Several DDL statements now work with no current database selected
• Bug fix in JSON_CONTAINS
• Bug fix for NOT BETWEEN expressions

Merged PRs

dolt

• 2747: Update README to mention DoltLab
• 2746: libraries/doltcore/table/editor: fix dropped test errors
  This fixes two dropped test err variables in libraries/doltcore/table/editor.
• 2745: Retain not null constraint on primary key columns when dropping primary key
  • Fixes adding primary keys using db.table syntax and unskipped bats test for it
  • Fixes issue of making all columns NOT NULL when adding pk with alter table
    (With the above issue, dropping pk bats test was not catching retaining of null constraint)
  • Fixes retaining NOT NULL constraint when dropping pk with alter table
• 2743: /go/store/prolly: flatbuffers-based prolly tree
  • replaces Node with flatbuffers-based mapNode
  • temporarily removes map-count accounting from Map
• 2741: go/go.mod: Bump gopsutil. Gets rid of deprecated warning on macOS.
• 2737: /go.mod: run go mod tidy
• 2736: Skipped bats test for alter table db.table add primary key
• 2733: manually go get gms main
• 2728: Fixing BATS test for table check constraint validation errors
• 2718: prevent creating and altering table with spatial types as primary key
• 2716: go/store/prolly: Prefix Range Scans for Prolly Trees
  Expands range-scan functionality for pkg prolly to include ranges defined on a prefix of key fields. This is necessary to support partial indexes and secondary index in Dolt. Various refactors were made to support this change:
  • removed support for "reverse" ranges, which aren't used anywhere
  • refactored range iters for prolly.Map to use a nodeCursor to track the end of the range
• 2715: go/store/datas: Decouple pull/push/clone from store/types, and move it into its own package.
• 2714: BATS tests for Spatial Types
• 2710: remove stmt conversion
• 2702: Correct ff/no-ff functionality for Dolt_Merge() and add tests for DOLT_MERGE when autocommit is off

go-mysql-server

• 784: fix alter table add pk
  Added case for getting non current database name in transaction for AlterPk
• 783: recursive CTEs
  RecursiveCte node impl with tests for joins, group by, and subquery
  aliasing.
  RecursiveCte's are not supported in DML statements. Nested recursive
  CTE's and nested subqueries are not supported yet. Aliasing is not
  neatly split down the middle on either side for RCte unions; only the
  dual table can have alias clashes on either side of the CTE for now.
  Cyclical recursion will timeout at depth 20. This will likely need a
  rewrite with better global aliasing to support infinitely nestable
  RecursiveCte's.
  Indexed table lookups are not applied to RecursiveCte tables currently
  because of a bug in the memory impl.
• 781: Expose Typ field in sql.RangeColumnExpr
• 780: Expose enginetest.CreateIndex() for testing
• 779: fix json_contains error handling
  Fixed tests for "Fix JSON Extract on struct" PR
• 778: Properly implemented SHOW GRANTS and SHOW PRIVILEGES
  SHOW GRANTS was a shell that always returned the same result, and it has now been properly implemented (without USING since active roles aren't in yet). Also added SHOW PRIVILEGES.
• 777: Add PeerGrouper, simplify count and rank window funcs.
  Count has a default range framer now, matching MySQL and
  Postgres's behavior. Move custom peer grouping logic into
  a formal framer to simplify ranking agg functions, which
  should simplify other rank funcs in the future.
• 775: add groupby column check
  Any alias column that is non alias table column will not have table name assigned.
• 774: Prevent invalidating table check constraints when altering table schema
  Updated logic for dropping and renaming columns to ensure that no table checks are referenced.
  Further improvements should eventually be made to ensure that table check constraints aren't modified by other schema alterations, such as modifying a column's type.
• 773: Added roles, revoking, and dropping users and roles
  This adds roles, along with implementing the accompanying statements and functionality. In addition, we may now revoke privileges and roles, and also drop users and roles.
• 772: ST_LATITUDE and ST_LONGITUDE functions
• 771: RANGE window framing with rangeFrameBase
  Support RANGE window frames, adds unit tests for framing
  and enginetests for queries.
  Count needs to be rewritten to support RANGES. Better validation
  for range expression/bound matching would be helpful.
• 770: fix not between function and or logics
  Fixed handling NOT BETWEEN function was incorrect
  Added missing logic for OR operator
• 761: ST_SWAPXY function
• 760: ST_DIMENSION function
  TODO: missing cases for multi geometry types

vitess

• 116: tools.go hack.
• 115: go.mod: Bump to go 1.17.

Closed Issues

• 2683: [Request] Bring back JSON/ add YAML schema export
• 2713: dolt push: invalid or corrupt table file
• 2705: Strange behavior of DOLT_MERGE()
• 2679: SQL commit should not be required after a fast forward dolt_merge() with autocommit off
• 2711: [Feature Request] Oh my Zsh support for local dolt folders

This is a feature release that introduces support for the full range of window expressions in aggregate functions.

Other issues:

• Fixed a bug where CHECK constraints were sometimes dropped when altering a table

Merged PRs

dolt

• 2695: Bug fix for dropped table check constraints when altering a table.
  Resolves: https://github.com/dolthub/dolt/issues/2672
  Confirmed that alterschema tests now cover all schema properties.

go-mysql-server

• 771: RANGE window framing with rangeFrameBase
  Support RANGE window frames, adds unit tests for framing
  and enginetests for queries.
  Count needs to be rewritten to support RANGES. Better validation
  for range expression/bound matching would be helpful.
• 763: Basic privilege granting and checking
  This adds an extremely basic implementation of privilege checking, along with the ability to grant a subset of privileges. In addition, connections carry along their authentication information, and direct queries on the engine (through use of the *sql.Context) also carry auth information for later privilege checking.

Closed Issues

• 2708: CHECK CONSTRAINTS silently dropped when you add a primary key to a table
• 2672: Adding column default drops CHECK constraints
• 2713: dolt push: invalid or corrupt table file

This is a patch release. It addressees the following issues:

• Corrected docs for schema export / import commands
• Fixed several fatal bugs involving column defaults

Merged PRs

dolt

• 2709: Bats test refactoring for CHECK constraints
  Factored out bats tests for check constraints into their own file. Added skipped test for alter removing constraints.
• 2689: libraries/doltcore/merge: fix dropped error
  This fixes a dropped error in libraries/doltcore/merge.
• 2681: Minor updates to schema export/import command documentation

go-mysql-server

• 769: Fixed a bug (just introduced) in resolving default column expressions
  Unbreaks dolt
  The reason that this bug didn't break the in-memory implementation is it keeps its column defaults reified as Expressions, so it doesn't really use most of the related analyzer machinery
• 762: Refactored how grant tables are stored
  This refactors a lot of previously-written code to more manageably handle future modifications to the grant tables, and to ease their use and reduce potential development mistakes.

Closed Issues

• 2703: can we deploy my own dolthub likes gitlab, or gerrit in my server?

This is a patch release. It addresses several bugs:

• Problems with column defaults causing errors and panics
• Support for window frame definitions (ROWS PRECEDING and ROWS FOLLOWING)
• Fix for certain GROUP BY expressions involving aliases
• Fix for JSON_OBJECT()

Additionally, --batch mode is now disabled by default for dolt sql when piping SQL scripts to the dolt command. Use --batch to enable it when running SQL import scripts. Also new is the --file argument to dolt sql, useful for environment where it is difficult to redirect STDIN.

Merged PRs

dolt

• 2699: Add CreateTagOnDB and DeleteTagsOnDB
  Closes #2693
• 2696: Register server the metrics with prometheus so they get published
• 2694: /go/libraries/doltcore/{doltdb/durable, sqle}: Added Format_7_18 to NomsBinFormat switch statements
  Format_7_18 is still used in ld in some cases
• 2685: Disable --batch mode for dolt sql by default
  --batch mode for dolt sql has issues in when it flushes to disk which make it incompatible with some SQL scripts. It's now only enabled when asked for explicitly.
  As part of this work, also made multiple statements in a single -q block work by default (used to require --batch or --disable-batch, the latter of which is retired)
  Finally, added a --file option to dolt sql, useful for environments where it's hard to redirect STDIN.
• 2677: fix vertical format printing issue
  row numbering was incorrect
• 2611: Spatial types actually stored as spatial types
  The ReadPoint, ReadLine, and ReadPolygon methods just use ReadString and convert to correct types.
  Able to call st_methods on column types now.
• 2609: Andy/prolly indexes
  This PR adds a second implementation of durable.Index and attempts to refactor pkg sqle to use durable.Index in place of types.Map. This is a major refactor in the SQL engine with the aim of supporting both the existing storage format and the new-and-improved™ storage format. As a measure of progress roughly 70% of the engine tests are passing against sqle while using the new format.
  Various changes were made in pkg val in order to support the SQL types/encodings need for sysbench. Some of these encodings are experimental and will change in the future.

go-mysql-server

• 769: Fixed a bug (just introduced) in resolving default column expressions
  Unbreaks dolt
  The reason that this bug didn't break the in-memory implementation is it keeps its column defaults reified as Expressions, so it doesn't really use most of the related analyzer machinery
• 768: go.mod,.github: Bump to 1.17.
• 766: Fix panics and errors introduced by recent changes to column default resolution
  This will fix https://github.com/dolthub/dolt/issues/2659 when picked up by dolt
• 764: Window frame support
  This PR adds sql.WindowFrame and sql.WindowFramers to represent
  window frames from AST -> plan -> exec.
  Every unique and valid combination of window frame
  Start and End bounds has a plan node (sql.WindowFrame)
  and a framer (sql.WindowFramer) pair. The frames, framers,
  and constructor functions that map from AST -> plan and
  plan -> exec are generated by optgen.
• 759: Revert "add ability to override logger (#758)"
  This reverts commit 2a4f3ea458b57c97bc3f1b70d697e103f62be6ef.
• 758: add ability to override logger
• 756: Adding support for non-string key types in JSON_OBJECT
  Resolves: https://github.com/dolthub/dolt/issues/2626
• 755: sql/expression/function: json_{array,object}.go: Fix to handle JSONValue subexpressions as well.
• 749: GroupBy normalization rule maintains parent Projection dependencies
  We flatten GroupBy aggregations to isolation agg functions, but in the process lose nested column dependencies. The downstream qualifyColumns rule was erroring, but without passthrough projections binary expressions like Arithemtic(Sum(x.i), y.i) will fail at execution time without a full set of input dependencies.
  For this query:`

  select sum(x.i) + y.i from mytable as x, mytable as y where x.i = y.i GROUP BY x.i
  

  We correctly identify that the GroupBy node has one primary aggregation, Sum, and project the Arithmetic separately.

  GroupBy -> Sum(x.i) + y.i -> ... TableScan (x,y)
  =>
  Project (Arithmetic(sum(x.i) + (y.i)))-> GroupBy(Sum(x.i)) -> ...
  

  The Project node fails downstream trying to lookup the y.i dependency we discarded in the transform. This PR adds dependencies back to cover the new parent Project for GroupBy flattening.

  GroupBy -> Sum(x.i) + y.i -> ... TableScan (x,y)
  =>
  Project (Arithmetic(sum(x.i) + (y.i)))-> GroupBy(Sum(x.i), y.i) -> ...
  

• 737: Adding functions st_asgeojson and st_geomfromgeojson
• 732: Allow conversion of zero values of other types to date / timestamp values
  This is for compatibility with postgres dumps and other tools that expect it

vitess

• 115: go.mod: Bump to go 1.17.
• 114: Remove a bunch of unused stuff.
• 113: .github,go: Cleanup workflows. Fix all tests to work under -race. Fix fragile prototext assertions in tests.
• 112: go,go/vt/proto: Regenerate pb message implementations, upgrade to APIv2.
• 111: go/vt/sqlparser: Get rid of partially parsed statements and {,ddl_}skip_to_end.

Closed Issues

• 2693: actions.CreateTag and actions.DeleteTag can't be used from dolthubapi
• 2658: dolt can't commit changes without a database selected
• 2680: --disable-batch flag required in dolt sql to use autocommit off
• 2659: Analyzer panic in resolveColumnDefaults when adding a new column or running describe

This is a patch release. It has some small bug fixes and feature releases:

• Fixed problems with merging schemas
• Fixed logging in sql-server going to the wrong file
• Missing docs
• JSON_ARRAY() function

Merged PRs

dolt

• 2686: redirect logrus standard log to cli.CliErr
• 2678: Skipped bats test for fast forward merge with autocommit off
• 2666: Skipped bats test for creating tables using the database.table syntax
  Fails if used in the same session as creating the database.
• 2663: go/libraries/doltcore/sqle: TableToRowIter: Export a method for doltgres to easily get a full table scan.
• 2662: Add ordinal to merge schema after merge
  Still need to add test cases here
• 2654: Removed Parse from typeinfo, some type conversion fixes
  Back when GetTypeConverter was added, Parse was deprecated. This PR now fully removes it. In addition, a few bugs regarding type conversions were fixed, a new test enforcing the inclusion of all types in GetTypeConverter was added, and a few more comments were added to clarify the different string type implementations.
• 2652: dump_docs.go now includes verifyConstraintsDocs
  dump_docs doesn't dump the docs for verify constraints, which means they're not visible in the online documentation. Hopefully this fixes that.
  After building I checked that it worked locally and seemed to produce the right output.
  Context: this is my first PR and my first time editing go code. Entering the great unknown here. :-)
• 2646: store/datas: fix dropped error
  This fixes a dropped err variable in the store/datas package.

go-mysql-server

• 755: sql/expression/function: json_{array,object}.go: Fix to handle JSONValue subexpressions as well.
• 753: go.mod: Bump vitess, pick up parser simplifications.
• 746: Window exec uses framing iterator and support agg functions in windows
  plan.Window is converted into new WindowIter backed by the framing setup.
  Codegen unary agg function nodes, including executable and unit testing
  harness. Rewrite window aggregation functions for framing.
  More unit tests that will make it easier to debug WindowBlock/Iter
  Doc comments for the new WindowIter exec stack.
• 745: Added password validation for logging in
  Now when creating a user, you can specify a password and log in according to the password given. Denies all incorrect password or invalid users. Also added a killswitch in the grant tables to disable this functionality since it's not yet ready.
• 741: add JSON_ARRAY() function
• 723: Apply Defaults in the AddColumn node
  This pr
  1. Adds Defaults in the Add Column Node
  2. Adds a nullability check in the Update node

vitess

• 112: go,go/vt/proto: Regenerate pb message implementations, upgrade to APIv2.
• 110: add serial supoort
  add serial support https://dev.mysql.com/doc/refman/8.0/en/numeric-type-syntax.html#:~:text=SERIAL
• 107: go/vt/sqlparser: Remove PartialDDL support.
  There's no need to support partial DDL in dolt.

Closed Issues

• 2670: Error 1105 when trying to switch branches with auto-commit turned off
• 2626: Error invalid type: INT when selecting from view
• 2628: AS OF queries do not support TIMESTAMP
• 2627: Incorrect behaviour of nested JSON_OBJECT functions
• 797: Dolt import should cast float strings that can be rounded to ints
• 2533: Support JSON_ARRAY() function

This is a patch release. It adds better logging for sql-server.

Merged PRs

dolt

• 2653: std out and err redirects
  Allow for stdout and stderr to be redirected to a file via a command line argument. When running outside of a terminal window it can be really inconvenient to redirect io. These parameters were added to make that easier.
• 2551: /go/gen/fb/serial: Flatbuffers Serialization Format
  Proposal for a new serialization format based on Flatbuffers. Flatbuffers has a number of features that make it appealing as a serialization/storage format for Dolt:
  • Forward/Backward compatibility for format evolution
  • Client library code generation for many languages
  • Stable binary format
  • Zero-copy deserialization, good performance
  • (g)RPC support
    However, it's not completely free. Serialized leaf nodes for indexes are roughly 1.6% bigger than the current candidate format in prolly/Node.go. Serialization overhead for a leaf node with 200 key-value tuple pairs:

  key:   (int32)
  value: (int32, int32, int32)
  data        3200 = 800 + 2400
  offsets     796  = sizeof(uint16) * (199 + 199)
  metadata    11   = TupleFormat * 2, tree_count, tree_level
  flatbuffers 65   = (1.6% overhead)
  total size  4072
  

  This overhead estimate is after "externalizing" the format for tuples, meaning that key/value tuples in Prolly tree nodes are stored in raw byte buffers and cannot be parsed/deserialized with Flatbuffers alone. Encoding each tuple in an individual Flatbuffers struct or table would be prohibitively expensive in both space and time.

Closed Issues

• 2617: JSON_OBJECT in a CASE expression doesn't work
• 2599: Ambiguous column name when it is not

Merged PRs

dolt

• 2639: Fixed bug with unresolved foreign keys being resolved in transactions
  The logic that handled foreign key merging during transaction commits did not take unresolved foreign keys into consideration, therefore errors were produced. This fixes that bug.
• 2634: go/libraries/doltcore/sqle: database_provider: Fix up unlocked read of p.databases map.
• 2580: Modify Add Column to apply defaults in engine
  This pr removes the direct row map access in addcolumn.go when adding a new column and pushes the logic to the engine. The engine uses the Updater interface to apply the default value.

go-mysql-server

• 744: /enginetest: Add a primary key to 'specialtable'
• 743: Fixes pushdownGroupByAliases bug with repeated projection column name.
  The analyzer rule stored column names in a hash map to mirror deps on
  either side of the GroupBy and Project child node. The return expressions
  were recreated without the table info, which is fine for most cases but
  breaks when there are deps from two tables with the same column name.
  This maintains table metadata for *expression.UnresolvedColumn
  throughout the rule, so that the output projections maintain table
  identity.
• 740: one more reentrant lock removal
• 739: /sql/analyzer: Remove reentrant locking from Catalog
• 723: Apply Defaults in the AddColumn node
  This pr
  1. Adds Defaults in the Add Column Node
  2. Adds a nullability check in the Update node

Closed Issues

• 2557: Dolt CLI locks table export file for a longer while

Merged PRs

dolt

• 2623: Fixed UUID type inference
  This change reorders the type inference rules to attempt to match a UUID before a timestamp. This is necessary because many valid UUIDs will parse as MySQL time strings as well.
• 2616: sysbench changes
• 2612: get the index for a key within a map
• 2607: benchmark/perf_tools: Added local sysbench script
• 2605: store/type: fix dropped test errors
  This fixes dropped test errors in store/type and its subpackage edits.
• 2601: Rewrite portion of the import path to better support partials
  This pr addresses #2586 and adds additional test cases for partial updates
• 2597: Lazily load column defaults
• 2593: add \G vertical query format
• 2572: Simplify dolt dump to use the sql engine and purge old data mover code
  This pr
  1. Rewrites dolt dump to use the sql engine
  2. Removes table_data_loc and DataMover abstractions
  3. Factors out a data mover pipeline used by export and dump.

go-mysql-server

• 739: /sql/analyzer: Remove reentrant locking from Catalog
• 736: Fixed bug for JSON values in CASE expressions
• 735: server/context.go: getOrCreateSession: Use a safer .mu.Unlock construct in case s.NewSession panics.
• 734: Fixed example in README
  Fix mysql command line mismatch the example code
• 733: add JSON_MERGE_PRESERVE() function
• 731: Revert groupby change for partitionless agg functions
  GroupBy can't switch it's execution runtime to the same as Window without a significant perf hit for select count(primary_key) from table. We could simplify group by iter/its aggregation functions now that it's just a for loop off a child iter.
• 729: Lazy evaluation for column defaults
• 728: Adding SRID and Axis Options for WKT and WKB functions
• 727: Ensures that check constraints cant be added to table that violate th…
  …e rule
• 726: add int uint conversion
• 724: GroupBy window framing refactor
  This PR adds:
  • design for window framing
  • new agg functions for window frames
  • GroupBy refactor using framing (frame = partition)
    Aggregations will have two execution paths while I am swapping out the Window execution layer:
  • In GroupBy node, agg functions will use window frame setup
  • In Window nodes, agg functions will continue to use AggregationBuffer
    Other:
  • The aggregation package is a bit of a mess right now. I will delete all of the old aggregation and window code during the window refactor.
  • I was 50/50 on whether or not we should use a special iterator for groupBy. It was not too hard to extend the windowBlockIterator, which should drop-in work for regular windows.
• 722: Bug fix for using TRIM function in CHECK constraints
• 721: Adding Support for SRID
  Adds srid field to existing spatial types, and st_srid function: https://dev.mysql.com/doc/refman/8.0/en/gis-general-property-functions.html#function_st-srid.
  Will add SRID option to wkb and wkt function in separate PR.
• 718: Adding Well Known Binary Conversion Functions
  Functions:
  • st_asbinary
  • st_aswkb
  • st_pointfromwkb
  • st_linefromwkb
  • st_polyfromwkb
  • st_geomfromwkb
• 711: Adding Well Known Text Conversions
  These are functions that allow a user to create spatial types using the Well Known Text (WKT) format.
  Functions:
  • point_from_text
  • line_from_text
  • poly_from_text
  • geom_from_text
  • as_text
  • as_wkt
• 710: Adding functions ST_X and ST_Y
  These are spatial type functions that work specifically with point.
• 704: Basic CREATE USER implementation
  This adds the minimum for the CREATE USER statement to work, in that we have an in-memory representation of the grant tables, which may be accessed either directly (by the mysql database) or through the user statements (of which we only have CREATE USER for now).
• 695: Add SQL func time_format
  Add SQL func time_format
  Also took the time to refactor time parsing to match MySQL behavior closer, even its weirdest parts.
  Fixes a bug in Time.Convert that led single-digit micro-second times to be parsed as tenth of seconds, e.g. 14:00:00.00007 was parsed as 14:00:00.7.
  Note: I'm trying to achieve minimum compatibility with Metabase's MySQL connector, which required some functions (see #694)
• 694: Add Locate SQL function
  Add Locate SQL function.
  I'm trying to achieve minimum compatibility with Metabase's MySQL connector, and it requires a couple new functions, so you'll see a few more PRs from me.
  Reference: https://www.w3schools.com/sql/func_mysql_locate.asp

Closed Issues

• 2535: Support JSON_MERGE_PRESERVE() function
• 1576: Add vertical format \G to DoltSQL shell queries.
• 2586: Possible dolt table import -u regression in v0.35.3

Merged PRs

dolt

• 2592: sql-server: Fix regression: Respect autocommit config.
  In the migration to *SqlEngine in 0.34.4 to 0.34.5, the server config's respect for the default value for autocommit regressed. In 0.34.5 - 0.35.3, autocommit is always turned on by default for incoming connections, regardless of the setting in the config.yaml file or the flag --no-auto-commit being passed at the command line.
• 2584: /go/store/types/format.go: Added NomsBinFormat for new storage format
• 2578: /go/doltcore/{doltdb, sqle}: Extracted durable.Index interface for types.Map
  This PR adds the durable.Index interface to abstract types.Map out of primary and secondary indexes. Currently, this interface has limited functionality, and in almost all cases will need to be unwrapped to be useful. This is the first step is a large refactor to decouple Index logic from a Noms-specific implementation.

Closed Issues

• 2589: Does dolt support stateless interaction with remote?
• 2588: REGEXP_REPLACE() has a read-X in the docs