Bot releases are hidden (Show)
Published by oscarbatori over 4 years ago
We are excited to announce the release of Dolt 0.17.1.
In this lease we extend the index support we discussed in the prior release to UNIQUE
indexes. Our type inference now also supports DATE
, TIME
, and DATETIME
types.
TableEditor
under doltdb
, similar to the IndexEditor
. I renamed the old editor to sqlTableEditor
, and it's now using TableEditor
underneath. It's just to comply with the SQL interfaces.TableEditor
thread-safe. Because of this, IndexEditor
also received the same changes. I verified that there was a lack of data races using go test -race
, and also ran the concurrency tests with thousands more iterations and edits (they're lower now so that we're not spending an hour in Jenkins). Just to verify that my tests also worked, I made temporary intentional changes that would allow for data races, and the tests caught them, and go test -race
also caught them.dolt table import
and dolt table cp
both use the TableEditor
now. Due to the concerns mentioned in the previous paragraph, I decided to benchmark a few metrics to get an idea of the performance impact of these two major changes.Before: 67.074s
After: 67.363s
The result is close enough that it's within the margin of error, so it's safe to say that new TableEditor
is just as performant as the old implementation.table import
is a different story though. This time, I imported a 10,000,000 row .psv
, with the results being:
Before: 53.521s
After: 107.816s
It takes roughly twice as long now, which is slower, but not tragically so. Primarily, the difference in speed is that the old code just wrote directly to the edit accumulator, where as the TableEditor
does a ton more bookkeeping, being a generalized editor.table import
& schema import
select 1/0 from dual
table_editor.go
into its own file and built upon it. I've also changed everything that modifies indexes, from rebuilding the indexes to the SQL path, to make use of the new index editor.select 1/0 from dual
NULL
if that's preferable.Published by oscarbatori over 4 years ago
We are excited to announce the release of Dolt 0.17.0!
Dolt now supports the main index statements, that is CREATE INDEX
, DROP INDEX
, and all ALTER TABLE
commands involving an index. Some less common types of index types that are not supported are PARTIAL
, UNIQUE
, SPATIAL
, and FULLTEXT
. Finally, SHOW INDEX
is not yet supported.
As usual, please file an issue if you find any bugs, or have a feature request.
SELECT
<
, <=
, >
, >=
, and BETWEEN
all now use indexes. Previously, only =
used indexes or primary keys, so this should be a pretty huge win.allstarfull
)const
definitions and need to be consistent across Dolt versions. Using iota
within const definitions can lead to unexpected results:
const (
name = "Andy"
mood = "ugh"
a = iota + 1
b
c
)
func main() {
fmt.Println(a)
fmt.Println(b)
fmt.Println(c)
}
3
4
5
These unit tests lock these constants in place.Published by oscarbatori over 4 years ago
nightly
and releases
now push to master
. (regressions
) has been merged into masterPublished by oscarbatori over 4 years ago
NAME
dolt sql-server - Start a MySQL-compatible server.
SYNOPSIS
dolt sql-server --config <file>
dolt sql-server [-H <host>] [-P <port>] [-u <user>] [-p <password>] [-t <timeout>] [-l <loglevel>] [--multi-db-dir <directory>] [-r]
DESCRIPTION
By default, starts a MySQL-compatible server which allows only one user connection ata time to the dolt repository in the current directory. Any edits made through this server will be automatically reflected
in the working set. This behavior can be modified using a yaml configuration file passed to the server via --config <file>, or by using the supported switches and flags to configure the server directly on
the command line (If --config <file> is provided all other command line arguments are ignored).
This is an example yaml configuration file showing all supported items and their default values:
log_level: info
behavior:
read_only: false
autocommit: true
user:
name: root
password: ""
listener:
host: localhost
port: 3306
max_connections: 1
read_timeout_millis: 30000
write_timeout_millis: 30000
databases: []
SUPPORTED CONFIG FILE FIELDS:
log_level - Level of logging provided. Options are: `trace', `debug`, `info`, `warning`, `error`, and `fatal`.
behavior.read_only - If true database modification is disabled.
behavior.autocommit - If true write queries will automatically alter the working set. When working with autocommit enabled it is highly recommended that listener.max_connections be set to 1 as concurrency
issues will arise otherwise.
user.name - The username that connections should use for authentication.
user.password - The password that connections should use for authentication.
listener.host - The host address that the server will run on. This may be `localhost` or an IPv4 or IPv6 address.
listener.port - The port that the server should listen on.
listener.max_connections - The number of simultaneous connections that the server will accept.
listener.read_timeout_millis - The number of milliseconds that the server will wait for a read operation.
listener.write_timeout_millis - The number of milliseconds that the server will wait for a write operation.
databases - a list of dolt data repositories to make available as SQL databases. If databases is missing or empty then the working directory must be a valid dolt data repository which will be made available
as a SQL database
databases[i].path - A path to a dolt data repository.
databases[i].name - The name that the database corresponding to the given path should be referenced via SQL.
If a config file is not provided many of these settings may be configured on the command line.
OPTIONS
--config=<file>
When provided configuration is taken from the yaml config file and all command line parameters are ignored.
-H <Host address>, --host=<Host address>
Defines the host address that the server will run on (default `localhost`)
-P <Port>, --port=<Port>
Defines the port that the server will run on (default `3306`)
-u <User>, --user=<User>
Defines the server user (default `root`)
-p <Password>, --password=<Password>
Defines the server password (default ``)
-t <Connection timeout>, --timeout=<Connection timeout>
Defines the timeout, in seconds, used for connections
A value of `0` represents an infinite timeout (default `30000`)
-r, --readonly
Disables modification of the database
-l <Log level>, --loglevel=<Log level>
Defines the level of logging provided
Options are: `trace', `debug`, `info`, `warning`, `error`, `fatal` (default `info`)
--multi-db-dir=<directory>
Defines a directory whose subdirectories should all be dolt data repositories accessible as independent databases.
--no-auto-commit
When provided sessions will not automatically commit their changes to the working set. Anything not manually committed will be lost.
Published by oscarbatori over 4 years ago
We are pleased to release a couple of bug fixes and a new configuration feature.
In. particular:
dolt_diff_<table>
tables is now fixedsqllogictest-tester
jenkins job. whenever we run the releases
job, we update branch regressions-tip-latest
to the latest release results and also update regressions-tip-previous
to be what regressions-tip-latest
was...Published by oscarbatori over 4 years ago
This release contains a bug fix to the SQL server implementation that may cause the server to have issues with starting.
Published by oscarbatori over 4 years ago
Dolt 0.16.0 is a very exciting release. It contains an important change to how we store columns, as well as a host of exciting features. The change to how we store columns does require users to migrate their repositories when they upgrade. We will provide some background, as well as the (very simple) migration procedure, before discussing features in the release.
We are absolutely committed to making this as painless as possible for our users, as a consequence don't hesitate to shoot us a note at [email protected] if you face any difficulty with the migration, or just need to discuss it further due to the sensitive nature of your data.
Dolt uses an integer value called a tag to identify columns. This contrived example illustrates that:
$ dolt init
$ cat > text.csv
name,id
novak,1
$ dolt table import -c --pk=id peeps text.csv
CREATE TABLE `peeps` (
`name` LONGTEXT COMMENT 'tag:0',
`id` LONGTEXT NOT NULL COMMENT 'tag:1',
PRIMARY KEY (`id`)
);
Versions of Dolt prior to this release only required tag uniqueness per table in a given commit, not across tables and across every commit.. This caused issues when diffing and merging between commits where column tags had been reused. We decided to bite the bullet and make the fix. Going forward, all column tags will be unique across all tables and history.
Existing Dolt repositories must be migrated to be used with Dolt client 0.16.0 and later. Running a command with the new client on an old repository will result in an error message prompting you to migrate the repository. The migration process is heavily tested, deterministic, and makes no changes to underlying data. It merely fixes the format to satisfy the requirements of new versions. . After upgrading Dolt, run a single command in your repository to migrate:
$ dolt migrate
Once that's complete you should be done. If you have Dolt data living at a remote, and collaborators, there's just one additional step. The first user to upgrade will need to run:
$ dolt migrate --push
This will force push the migrated branches. Subsequent collaborators will have to then run:
$ dolt migrate --pull
This will sync their migrated repo with the migrated remote, and preserve any local changes they have, applying them on top.
We are committed to making our SQL implementation as close as possible 100% correct, and this release represents a big step towards that goal. The improvements include:
SHOW CREATE VIEW
now works, so views can be inspected in the canonical mannerSHOW TABLES
statementsDECIMAL
, TIME
, SET
, ENUM
, NCHAR
, NVARCHAR
, and aliases to many moredolt sql-server
and dolt sql
now support accessing multiple dolt repositories in a single SQL session. Each repository is exposed as a database. See databases with SHOW DATABASES
, and select one to query with USE $database
. Joins across repositories are supported. Start dolt sql
or dolt sql-server
with the new --multi-db-dir
argument, which must name a directory containing dolt repositories to expose in queries.dolt sql-server
now supports writes, which means that the working set will be updated by UPDATE, INSERT, DELETE
and other statements which change data. The SQL server was previously read-only. This fixes https://github.com/liquidata-inc/dolt/issues/549. Important caveat: only one concurrent connection is allowed to prevent data races. Support for better concurrency is being tracked in https://github.com/liquidata-inc/dolt/issues/579
user()
, left()
, if()
are now supported, motivated by getting Dolt working with DataGripdolt sql -x
optioninformation_schema
databasedolt sql -r json
As well as making our SQL implementation compliant with MySQL, we are also committed to implementing all VCS operations available on the command line available via the SQL interface.
dolt_branch
system table where the list of branches on a repo is surfaced in SQLWe have now fixed AWS S3 remotes, so if you want to use your own S3 infrastructure as a backend, you can do that. See the dolt remote
CLI documentation for details. While we love to see data and users on DoltHub, we are committed to making Dolt and open standard, and that means being useful to the broadest possible audience.
As well as fixing the issue with remotes, we fixed a number of other bugs:
setup
function, it's equivalent to manually putting a skip on every test. Then whenever we fix it, we can just delete it in one place.dolt checkout <branch>
or dolt checkout -b <branch>
, your local docs would be overwritten by the working root (your changes vanish).dolt checkout <branch>
, given there are no conflicts. Since your "working set" for docs is technically on the filesystem, and not on the roots, it was getting wiped. Now i'm pulling the unstagedDocDiffs from the filesystem, and excluding those when it comes time to saveDocsOnCheckout
.BIT(1)
or TINYINT/BOOLEAN
) then it will parse TRUE
and FALSE
(and other variations) as their respective integers. This restores a behavior that was removed when the new type system was introduced, and all of the code paths were coalesced.NOT NULL
set.rowconv.NewJoiner
in diffRows
Published by oscarbatori over 4 years ago
We are excited to announce the release of Dolt 0.15.2.
In our last release, 0.15.1 we highlighted the ability to query AS OF
a branch or commit. In this release we expand this functionality by allowing uses to query AS OF
a timestamp. This represents a version of the syntax familiar to users of other SQL databases with versioned data support. Thus we allow users to treat the underling commit graph as either wall clock time or relative to a commit.
Dolt SQL now supports TIMESTAMP
and DATETIME
functions, which can be used to construct a time object of the given type. DATETIME('2020-03-01')
and DATETIME()
will return the given time or the current time, respectively.
We continue to make enhancements to our underlying SQL engine, and other assorted bug fixes.
As usual, let us know if you have any thoughts by filing an issue!
checkout
command.checkout
to work when branch name and table name are identical?dolt sql -x <saved_query_name>
dolt sql --list-saved
Changes
dolt sql -s <name> -q <query>
now saves the query with id = namePublished by oscarbatori over 4 years ago
We are excited to announce the the release of Dolt 0.15.1.
We now support AS OF
queries, similar to the Microsoft SQL Server implementation described here. Timestamps are not yet supported for AS OF
expressions, but users can use branch names, commit hashes, and other commits specs:
SELECT name FROM persons AS OF 'add-new-employees'
SELECT * FROM persons AS OF 'hvbsl13cbi03ptft78k0pgtkgpd68ehj'
SELECT * FROM persons AS OF 'HEAD~'
These queries will retrieve rows from the table as it existed at the named revision. Different tables in a join can use different AS OF
clauses:
SELECT name FROM persons as of 'HEAD~' NATURAL JOIN addresses AS OF 'HEAD~2'
Elsewhere we continued to improve the performance and correctness of SQL, expand test coverage, and fix bugs.
testcommands
package, see go/libraries/doltcore/dtestutils/testcommands/command.go..mdx
files, JSX version of .md
).md
files for building docs).mdx
Published by oscarbatori over 4 years ago
We are excited to announce the release of Dolt 0.15.0.
Previously Dolt had a much narrower type system than MySQL. For ease of use reasons, we just mapped types that we did not support to their "super type", for example using the previous Dolt release:
doltsql> CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE, PRIMARY KEY (name));
doltsql> describe pet;
+---------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+----------+------+-----+---------+-------+
| name | LONGTEXT | NO | PRI | | |
| owner | LONGTEXT | YES | | | |
| species | LONGTEXT | YES | | | |
| sex | LONGTEXT | YES | | | |
| birth | DATETIME | YES | | | |
| death | DATETIME | YES | | | |
+---------+----------+------+-----+---------+-------+
Using this release of Dolt, we can see that richer type choices are respected:
doltsql> CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE, PRIMARY KEY (name));
doltsql> describe pet;
+---------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+-------------+------+-----+---------+-------+
| name | VARCHAR(20) | NO | PRI | | |
| owner | VARCHAR(20) | YES | | | |
| species | VARCHAR(20) | YES | | | |
| sex | CHAR(1) | YES | | | |
| birth | DATE | YES | | | |
| death | DATE | YES | | | |
+---------+-------------+------+-----+---------+-------+
We hope this makes it easier for users to use Dolt in the context of their existing data infrastructure. This change is backward compatible. So old versions of Dolt can read repos written by new versions, and vice versa. That said, you should upgrade to the latest and greatest!
We also now support unions, a powerful tool for tabulating the results of analyses produced by different queries:
doltsql> (select 1 as this) union (select 2 as this);
+------+
| this |
+------+
| 1 |
| 2 |
+------+
We also improved the performance of the Dolt log table, as well as making our usual assortment of bug fixes and improvements.
Published by oscarbatori over 4 years ago
We are pleased to announce the release of Dolt version 0.14.0.
The major new feature in this release is the query catalog, which is implemented via adding additional options to the dolt sql
command. Users can now pass --save
and --message
options to SQL queries to save and version them at the repo level:
-s <saved query name>, --save=<saved query name>
Used with --query, save the query to the query catalog with the name provided. Saved queries can be examined in the dolt_query_catalog system table.
-m <saved query description>, --message=<saved query description>
Used with --query and --save, saves the query with the descriptive message given. See also --name
This will allow users to document their data with versioned SQL queries. In the future we hope to make this a validation tool.
We continue to make progress on our SQL implementation, both in terms of correctness and performance. For example, GREATEST()
now supports timestamps. We are prioritizing correctness over performance, though we are making gains in performance.
As usual, if you see anything amiss, please don't hesitate to file an issue.
Published by oscarbatori over 4 years ago
The set of changes for this version are:
As usual, please let us know if you have any questions or concerns.
dolt creds use
to select a credential.CreateBlob
and MustCreateBlob
to CreateBinary
and MustCreateBinary
because those names make more sense.Published by oscarbatori over 4 years ago
We are releasing a patch fix to Dolt, as the 0.13.0 release contained a bug. The new feature for creating license and read me documents on a repository contained a bug such that when cloning a repository using dolt clone
, the documents were not updated. This patch ensures that functionality works correctly.
Since this is a patch of a recent release, see 0.13.0 release notes for details about the new features recently introduced.
Published by oscarbatori over 4 years ago
We are excited to announce the release of Dolt 0.13.0, hot on the heels of relaunching DoltHub.
It's now incredibly easy to install Dolt, so if you haven't tried it yet, you can now obtain a copy with a single command, and start playing with datasets:
$ curl -L https://github.com/liquidata-inc/dolt/releases/latest/download/install.sh | bash
The installer script works on Mac and Linux. For Windows, download the MSI installer below.
We released a blog post detailing some exciting new functionality for surfacing versioning data in Dolt. This is the first of a set of features that will eventually expose all the Git-like internals of Dolt to SQL, and facilitate automated use of Dolt by allowing users to define their default choices inside SQL statements.
dolt log
command via a SQL queryWe now allow users to create License and Readme documents as part of their Dolt repository, these appear as LICENSE.md
and README.md
files in the root of your repo. Edit them with the text editor of your choice, then add them to a commit with dolt add
, the same as a table. Their contents are versioned alongside your tables' data. License and Readme files will soon be visible on DoltHub for repositories that provide them. Allowing users to specify the terms on which data is available is an important step towards creating a vibrant data-sharing community.
Our SQL implementation now supports persistent views, taking us closer to having a fully functioning SQL engine. Create a view using the standard SQL syntax:
CREATE VIEW myview AS SELECT col1, col2 FROM mytable
Then query it like any other table:
SELECT * FROM myview
We made performance enhancements to SQL, including supporting indexed joins on a table's primary key columns. This should make the engine usable for joins on the primary key columns of two tables. Additional improvements in join performance are in the works. We also fixed assorted bugs and made performance improvements in other areas of the SQL engine.
sequenceIterator
for the use case when sequenceCurosor
is simply accessing elements in its sequence (ie MapIterator
, SetIterator
, and ListIterator
)sequenceIterator
designed by @reltuk to batch chunk fetching from the ValueStore
. In use cases such as DoltHub where chunk fetching IO is slow, this will dramatically accelerate performance.doltdb/root_val_test.go
env
package to []doltdb.DocDetails
from []*doltdb.DocDetails
. You'll see some reformatting to accommodate this change.checkout <doc>
checkout <branch>
merge <branch>
(one scenario is still buggy, need help identifying solution)dolt status
to indicate that the doc needs to be added and committed to finish merging. Right now it appears the doc is getting added to the working root.dolt sql
commanddolt diff
dolt diff <doc>
will land in a subsequent PR.rm LICENSE.md
dolt diff
diff --dolt a/LICENSE.md b/LICENSE.md
deleted doc
- new license
Adding docs that don't already exist on the staged root:
touch README.md
touch LICENSE.md
dolt diff
diff --dolt a/README.md b/README.md
added doc
diff --dolt a/LICENSE.md b/LICENSE.md
added doc
Modifying a doc that has already been committed:
dolt diff
diff --dolt a/LICENSE.md b/LICENSE.md
--- a/LICENSE.md
+++ b/LICENSE.md
this is my license
+
+ How to use this repository:
+ Step 1)....
dolt reset --hard
, with testsdolt reset --soft
, dolt reset .
, dolt reset
, dolt reset <doc>
, with testsdoltcore/env/dolt_docs
to more general usedoltcore/env/environment.go
to help stage and unstage docsdolt add .
for dolt_docs, with updated dolt status
and bats testsDECIMAL
ENUM
SET
GEOMETRY
GEOMETRYCOLLECTION
LINESTRING
MULTILINESTRING
POINT
MULTIPOINT
POLYGON
MULTIPOLYGON
The first three mentioned are the last ones I'm going to implement for now, as the GEOMETRY
types are very, very complex. However, DECIMAL
proved to be a bit tougher than expected, and I didn't want to delay getting this PR out since I'll be gone over the entire Thanksgiving holiday (technically I'm on it now but I promised a PR by the 25th). Besides the ones listed above, every other type is in this PR in some capacity (NCHAR
and friends are missing, but that type can be replicated as it's just a charset/collation shortcut).Type
interface that returned an attribute map, but decided against it since I wanted to be able to add more than raw attributes, and a map of functions just seemed too complex for the tradeoff.hebrew
character set, then they can just reference CharacterSet_hebrew
, rather than defining their own variable with custom_var, err := ParseCharacterSet("hebrew")
.JSON
, but implementing it right may take 1-2 full weeks, which I don't feel is worth it right now.BOOLEAN
as an int8
. But that's how MySQL does it, so it's probably for the best.Int8
can be a global type because it is fully self-contained in its information. Varchar
, OTOH, requires a length to be meaningful.utf8mb4
as a character set (which mimics Go's string type implementation to the best of my knowledge), and changing it to something else doesn't change any actual behavior. That can come later because that's A LOT to do...TIME
type is weird, in that the minimum precision is microseconds, so I chose to base the entire type around that. All of the parseable variations are present as well. I was initially going to use time.Duration
from Go, but it actually didn't suit the requirements. Working with it turned out to be more difficult than just using microseconds. For example, the only way to make a duration is to parse a string of a specific format, which would be parsing a string, then recreating a different string to parse, which is too much.YEAR
is the most useless type I've ever encountered in anything, but hey it's included and it's fully supported now, hooray...NOT NULL
according to the MySQL documentation.Delete
then Insert
way instead of adding a Replacer
interface. The insert-only functionality in the event of no primary key isn't what the "expected" behavior would be in my opinion, and since we don't have a way to check from this library, I think we should just do it this way instead. On the other side, if we have a Replacer
interface, then we can't ensure that the implementer will properly choose if it will an Insert-only. I'd rather enforce a behavior that will be applicable and correct in the most obvious case, and it reduces the necessary code from implementers too since they won't need to duplicate any trigger functionality or anything.Published by oscarbatori almost 5 years ago
We are excited to announce the release of Dolt 0.12.0!
We have our first open-source committer to the Dolt project! Thanks to @namdnguyen for providing a helpful fix to our documentation. We are hoping this will be the first of many open-source contributions to Dolt.
As discussed in this blog post, we use sqllogictest to test our SQL implementation's logical correctness. It contains 5 million tests! This release marks a huge jump in compliance, with our implementation now hitting 89%, up from well under 50% just a few weeks ago.
--diff-where
command allows the user to add a predicate on the table being diff'd to reduce the surface area of the diff output and drill into specific data of interest.
When a user commits data, a timestamp is associated with the commit. By allowing Dolt users to customize the timestamp we allow the user to create an implicit bi-temporal database (based on commit time) while maintaining the ordinal integrity of the commit graph for querying history and reasoning about the sequence of updates.
Using the SQL diff command, that is dolt diff -q
or dolt diff --sql
, users can produce SQL output that will transform one branch into another. In other words this command will produce the difference, in data and schema transformations, between two refspecs in the commit log of Dolt repository.
As usual, this release also contains bug fixes and performance improvements. Please create an issue if you have any questions or find a bug.
--- a/test @ 4uvb6bb3p7dqudnuidh9oh4ccsehik7n
+++ b/test @ 2tl4quv92ot0jg4v3ai204rld00trbo4
CREATE TABLE test (
`pk` BIGINT NOT NULL COMMENT 'tag:0'
- `c1` BIGINT COMMENT 'tag:1'
`c2` BIGINT COMMENT 'tag:2'
`c3` BIGINT COMMENT 'tag:3'
`c4` BIGINT COMMENT 'tag:4'
`c5` BIGINT COMMENT 'tag:5'
< PRIMARY KEY (`pk`, `c1`)
> PRIMARY KEY (`pk`)
);
Also add the pk contstraint so it shows when it is not changing:
--- a/test @ idfqe6c5s2i9ohihkk4r4tj70tf3l8c7
+++ b/test @ 2tl4quv92ot0jg4v3ai204rld00trbo4
CREATE TABLE test (
`pk` BIGINT NOT NULL COMMENT 'tag:0'
`c1` BIGINT COMMENT 'tag:1'
`c2` BIGINT COMMENT 'tag:2'
< `c3` BIGINT COMMENT 'tag:3'
> `newColName3` BIGINT COMMENT 'tag:3'
`c4` BIGINT COMMENT 'tag:4'
`c5` BIGINT COMMENT 'tag:5'
PRIMARY KEY (`pk`, `c1`)
);
dolt
. Looking for feedback on approach before I head too far down this path, if it is suboptimal.int
and string
and only accounts for .csv
style test data formats, but I'd like to make my data generation functions robust enough to be able to account for all file formats that dolt supports and all noms types...dolf diff --sql
output. Supports:
go.mod
.Published by oscarbatori almost 5 years ago
We are excited to announce the release of Dolt 0.11.0.
We implemented a dolt log table, thus making our first attempt to surface dolt version control concepts in SQL by surfacing commit data. This will allow users to leverage commit data in an automated setting via SQL. Clone a public repo to see how it works:
$ dolt clone Liquidata/ip-to-country
$ cd ip-to-country
$ dolt sql -q "select * from dolt_log"
$ dolt sql -q "select committer,date from dolt_log order by date desc"
+-------------+--------------------------------+
| committer | date |
+-------------+--------------------------------+
| Tim Sehn | Wed Sep 25 12:30:43 -0400 2019 |
| Tim Sehn | Wed Sep 18 18:27:02 -0400 2019 |
.
.
.
We added support for DATETIME
data type in SQL. This is a major milestone in achieving compatibility with existing RDBMS solutions.
We continue to rapidly improve our SQL implementation. On the performance side some degenerate cases of query performance saw large improvements. We also resolved some issues where update statements had to be "over parenthesized", with the parser now matching the standard.
We support null values in CSV files that are imported via the command line, as well as minor bug fixes under the hood.
If you find any bugs, or have any questions or feature requests, please create an issue and we will take a look.
value_decoder.go
, as opposed to going through the table indirection. This is about a 30% perf improvement for linear scans on skipValue()
, which is worth the duplication here.UUID
and InlineBlob
here for now.sqle/types/tests
), so that's why they're in there. In particular, the cycle was that sqle
imports sqle/types
, and the tests rely on (and thus must import) sqle
, causing the cycle.-q
pathway. That will be a different/future PR though.noms
relies on periods specifically pretty heavily. Seems to be excluded from the regex below by design, since they build some types on the expectation that a branch name or ref
contain a period.noms
layer and this variable (go/store/datas/dataset.go):
// DatasetRe is a regexp that matches a legal Dataset name anywhere within the
// target string.
var DatasetRe = regexp.MustCompile(`[a-zA-Z0-9\-_/]+`)
acts as the regex source of "truth" for branch names/ dataset look ups, and I believe more..
to this string in order to parse the string later and correctly create it's Path
types...noms
Path
delimiters to be a different character, but the changes go pretty deep and start breaking a lot of things. Happy to continue down that course in order to support periods in branch names, but it might take me a bit of time change everything. I'm also not sure what character should replace the period... asterisk? Anyway, this PR seemed like low hanging fruit fix to resolve the panic at least.DoltToSQLType
hardcoded map from the original SQL implementation. I didn't change it everywhere (it's used heavily in the old SQL code that isn't even being called anymore), but it's changed where it matters. Added a new interface function and changed the printing code to be a bit more consistent (we were mixing uppercase with lowercase).BIGINT
for sql.Int64
, as int
parses in MySQL to a 32-bit integer, which isn't correct. Essentially made it so that if you took the CREATE
statement exactly as-is and exported your data to a bunch of inserts and ran it in MySQL then it wouldn't error out, as it previously would have.SqlTypeInit
and add the struct to sqlTypeInitializers
and you've got a type that works in SQL now!Published by mjesuele almost 5 years ago
We are excited to announce the latest release of Dolt, which includes a new feature, substantial improvements to existing features, and a new Windows installer.
Dolt now has a blame
command, which provides row audit functionality familiar to Git users. (Blame for individual cells is in the works.) We have a deep dive on the implementation of dolt blame on our blog, so definitely check that out if you're interested.
One of the long-term goals of Dolt is to provide a database with fine-grained audit capabilities to support hygienic management of valuable human-scale data, and this feature is a huge step towards realizing that vision.
One of our major goals for the product is full SQL compliance; this release contains steps towards achieving that. In particular, the following commands are now supported:
CREATE TABLE
DROP TABLE
INSERT VALUES
& INSERT SET
(no IGNORE
or ON DUPLICATE KEY UPDATE
support yet, also no INSERT SELECT
support yet)UPDATE
(Single table, no IGNORE
support yet)REPLACE VALUES
and REPLACE SET
(no REPLACE SELECT
support yet)As well as making progress against our goal of full compliance, we also created a test suite that will help validate our SQL implementation. Check out the test suite and harness, and the related blog post. This is an important step in creating a fully transparent mechanism for our progress against our compliance goal.
We also fixed some bugs and made some performance improvements.
We now support schema inference from a CSV file. This is a convenience function to make importing a CSV with a correct schema easier. The command is best understood by looking at the help
details:
$ dolt schema import --help
NAME
dolt schema import - Creates a new table with an inferred schema.
SYNOPSIS
dolt schema import [--create|--replace] [--force] [--dry-run] [--lower|--upper] [--keep-types] [--file-type <type>] [--float-threshold] [--map <mapping-file>] [--delim <delimiter>]--pks <field>,... <table> <file>
We now provide both 32- and 64-bit MSI packages for easy installation of Dolt on Windows. These may be used instead of manually extracting the archives (which are now provided in .zip
format instead of .tar.gz
). Please let us know if you encounter any issues.
Various bug fixes and enhancements, and also improvements to dolt clone
which had a problematic race condition.
As always, bug reports, feedback, and feature requests are very much appreciated. We hope you enjoy using Dolt!
$ dolt blame lunch-places
+--------------------+----------------------------------------------------+-----------------+------------------------------+----------------------------------+
| NAME | COMMIT MSG | AUTHOR | TIME | COMMIT |
+--------------------+----------------------------------------------------+-----------------+------------------------------+----------------------------------+
| Boa | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Chipotle | lunch-places: Added Chipotle | katie mcculloch | Thu Aug 29 11:38:00 PDT 2019 | m2jbro89ou8g6rv71rs7q9f3jsmjuk1d |
| Sidecar | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Wendy's | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Bangkok West Thai | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Jamba Juice | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Kazu Nori | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| McDonald's | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Sunnin | change rating | bheni | Thu Apr 4 15:43:00 PDT 2019 | 137qgvrsve1u458briekqar5f7iiqq2j |
| Bruxie | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Espresso Cielo | added Espresso Cielo | Matt Jesuele | Wed Jul 10 12:20:39 PDT 2019 | 314hls5ncucpol2qfdphf923s21luk16 |
| Seasalt Fish Grill | fixed ratings | bheni | Thu Apr 4 14:07:36 PDT 2019 | rqpd7ga1nic3jmc54h44qa05i8124vsp |
| Starbucks | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Tocaya | update tocaya rating | bheni | Thu Jun 6 17:22:24 PDT 2019 | qi331vjgoavqpi5am334cji1gmhlkdv5 |
| Sake House | fixed ratings | bheni | Thu Apr 4 14:07:36 PDT 2019 | rqpd7ga1nic3jmc54h44qa05i8124vsp |
| Swingers | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Art's Table | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Bay Cities | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Benny's Tacos | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Bibibop | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Curious Palate | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
| Meat on Ocean | Had an unhandled schema merge conflict which I ch… | Tim Sehn | Fri Mar 22 12:21:59 PDT 2019 | ffabndafbp64r393ttghused8siip77j |
+--------------------+----------------------------------------------------+-----------------+------------------------------+----------------------------------+
types
change might be simpler than I first thought! I probably felt it was harder just because finding all of these locations was a major pain...getCommitSpec
in that.\\''
properly, and now it does. Had to add 4 backslashes on the test because bash does its own escaping before passing it to dolt."[<commit>] [<table>...]",
"--export <table> <file>",
"--add-column [--default <default_value>] [--not-null] [--tag <tag-number>] <table> <name> <type>",
"--rename-column <table> <old> <new>",
"--drop-column <table> <column>",
These have all been split into separate commands:Published by reltuk about 5 years ago
Contained in this release
DROP
, UPDATE
, INSERT
If you find any bugs, have a feature request, or an interesting use-case, please raise an issue.
$ dolt diff --summary rnfm50gmumlettuebt2latmer617ni3t
diff --dolt a/sentences b/sentences
--- a/sentences @ gd1v6fsc04k5676c105d046m04hla3ia
+++ b/sentences @ 2ttci8id13mijhv8u94qlioqegh7lgpo
7,800,102 Rows Unmodified (99.99%)
15,030 Rows Added (0.19%)
108 Rows Deleted (0.00%)
960 Rows Modified (0.01%)
1,888 Cells Modified (0.00%)
(7,801,170 Entries vs 7,816,092 Entries)
diff --dolt a/translations b/translations
--- a/translations @ p2355o6clst8ssvr9jha2bfgqbrstkmm
+++ b/translations @ 62ri8lmohbhs1mc01m9o4rbvj6rbl8ee
5,856,845 Rows Unmodified (90.91%)
468,173 Rows Added (7.27%)
578,242 Rows Deleted (8.98%)
7,626 Rows Modified (0.12%)
7,626 Cells Modified (0.06%)
(6,442,713 Entries vs 6,332,494 Entries)
Fixes #77sql.go
file. I know at first you mentioned keeping them there for reference, but they're not being used at all at this point, and they're still in git history if we want to look at them again in the future for some reason. It's clutter at this point.go-mysql-server
. The code looks intentional, in that converting strings to ints will return 0 if the string is not parsable. I'll file it as a non-conforming bug on their end, but for now I'm skipping the test.ChunkStore#GetMany
and then go off and process. Clients largely try to control the memory overhead and pipeline depth by passing in a buffered channel of an appropriate size. The expectation is that the implementation of GetMany
will have an amount of data in flight at any give in time that is in some reasonable way proportional to the channel size.getManyAtOffsetsWithReadFunc
so that we only have 4 concurrent dispatched reads per table_reader
instance at a time.tableReader
s), and not depend on the number of tableReader
s which happen to back the chunk store.GetManyCompressed
on every chunk address and copies their contents to /dev/null
. It was run on a ~10GB (compressed) data set:$ /usr/bin/time -l -- go run test.go
...
MemStats: Sys: 16628128568
161.29 real 67.29 user 456.38 sys
5106425856 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
10805008 page reclaims
23881 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
8 signals received
652686 voluntary context switches
21071339 involuntary context switches
After:
$ /usr/bin/time -l -- go run test.go
...
MemStats: Sys: 4590759160
32.17 real 30.53 user 29.62 sys
4561879040 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
1228770 page reclaims
67100 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
14 signals received
456898 voluntary context switches
2954503 involuntary context switches
On these runs, sys time, wallclock time, vm page reclaims and virtual memory used are all improved pretty substantially.Delete
and Insert
functions that we already have. The previous delete would ignore a delete on a non-existent row, so I just changed it to throw the correct error if the row does not exist so that REPLACE
works properly now (else it will always say a REPLACE
did both a delete & insert).clone
as the implementation is different. Some things in the events package have been refactored to prevent circular dependencies. Adding StandardAttributes
will help me generate the info for my new metrics.git log master..feature
. Useful for displaying the commit logPublished by reltuk about 5 years ago
We have released version 0.98 of Dolt, which as you probably know is now open source. A quick reminder that you can freely host the awesome public data you put in Dolt at DoltHub.
This release contains performance improvements, and bug fixes but no major new features. Please let me know if you have any questions.
$ dolt commit -m "commit ints"
After:
$ dolt commit -m "commit ints"
commit 3cvbeh6bn94hlhfaig5pa65peiribrhn
Author: Matt Jesuele <[email protected]>
Date: Mon Aug 26 19:10:17 -0700 2019
commit ints
dolthub.com
as the default, which is incorrect. I've changed it to the appropriate environment constant so that it also properly updates when we change from our beta domain.go.mod
, as we have git history and don't quite need the comments.Published by reltuk about 5 years ago
The first public binary release of Dolt, git for data.