Bot releases are hidden (Show)
Published by oscarbatori almost 4 years ago
We are pleased to announce Dolt 0.22.7.
This release focuses on bug fixes, and performance improvements in SQL. In particular delivered huge performance improvements in our SQL implementation. You can find the scope of these performance improvements detailed on our benchmarks page.
printStagedDiffs
always returned 0, even when there were diffs not staged. This return was also causing it to print in printStatus
"nothing to commit, working tree clean". This was not the case.printStagedDiffs
to return the number of the staged tables plus the number of staged docs instead. This prevents it from entering the if statement with the print also.table.SqlTableReader
as a replacement for directly reading from table maps. Used it to replace types.MapIterator
in sqle.doltTableRowIter
rowconv -> pipeline
to pipeline -> rowconv
Published by oscarbatori almost 4 years ago
-a
flag to dolt commit
and DOLT_COMMIT
. It stages all tables.Published by oscarbatori almost 4 years ago
Published by oscarbatori almost 4 years ago
Published by oscarbatori almost 4 years ago
version()
is used to emulate the target MySQL version, I've added dolt_version()
so that one may specifically query the dolt version.Published by oscarbatori almost 4 years ago
We are excited to announce the minor version release of Dolt 0.22.0.
We continue to expand the SQL tables that surface information about the commit graph, in this release we added:
dolt_commits
dolt_commit_ancestors
dolt_commit_diffs_<table>
We added support for prepared statements to our SQL server.
dolt ls --all
dolthubapi
uses NewCreatingWriter
which breaks with Andy's recent changes (it's being used in dolthubapi here)Published by oscarbatori almost 4 years ago
dolthubapi
for the import table api. I extracted some logic so that I can use InferSchema
and MoveDataToRoot
to root to reduce some of the duplicationsmaster
, however the workflow only works when the PR is based against master
. Therefore this PR does not target the aforementioned PR's branch.CREATE INDEX
memory usage on.table_editor.go
to periodically flush the contents once some arbitrary amount of operations have been performed.Published by oscarbatori almost 4 years ago
We are excited to announce the release of Dolt 0.21.1, a patch release with functionality and performance improvements.
A significant new aspect of the Dolt release process will be providing SQL benchmarks. You can read a blog about our approach to benchmarking using sysbench here, and you can find the benchmarking tools here. By way of example the benchmarks for this release were created with the following command:
./run_benchmarks.sh bulk_insert oscarbatori v0.21.0 v0.22.1
This produced the following result, which we host on DoltHub:
nil
error after that.onDone
when the rowTrackingIter is Closed.onDone
is called. ChildPublished by oscarbatori about 4 years ago
We are excited to announce the release of Dolt 0.21.0. This release contains a host of exciting features, as well as our usual blend of bug fixes and performance improvements.
As a result of our own internal data collaboration projects, we realized that a squash
command for condensing change sets as a consideration for collaborators was an essential tool. This is now in Dolt.
A user highlighted that Dolt didn't work with NFS mounted drives due to the way it was interacting with the filesystem. We have now fixed this.
We now have a dolt gc
command for cleaning up unreferenced data. This was requested by several users as a space saving mechanism in production settings.
We continue to aggressively pursue performance improvements, most notably a huge improvement in full table scans.
sysbench
toolingAs we detailed in a blogpost yesterday we have created a tooling to provide our development team and contributors with a simple way to measure SQL performance. For example, to compare a arbitrary commit to the current working set (to test whether changes introduce expected performance benefits):
$ ./run_benchmarks.sh bulk_insert <username> 19f9e571d033374ceae2ad5c277b9cfe905cdd66
This will build Dolt at the appropriate commits, spin up containers with sysbench
, and execute the benchmarks.
An open source contributor provided several fixes to our CLI documentation, which we have gratefully merged.
We have fixed Google Cloud Platform remotes motivated by a bug report from a user experimenting with Dolt.
Database
from its root chunk and coping all reachable chunks to a new set of NBS tables.NomsBlockStore
garbage collection and protect against NBS
corruption. Storing gcGen
in the manifest will support deleting chunks from an NBS
in a safe way. NBS
instances that see a different gcGen
than they saw when they last read the manifest will error and require clients to re-attempt their write.NBS
will now have three forms of write errors (not including IO errors or other kinds of unexpected errors):
nbs.errOptimisticLockFailedTables
: Another writer landed a manifest update since the last time we read the manifest. The root chunk is unchanged and the set of chunks referenced in the manifest is either the same or has strictly grown. Therefore the NBS can handle this by rebasing on the new set of tables in the manifest and re-attempting to add the same set of novel tables.nbs.errOptimisticLockFailedRoot
: Another writer landed a manifest update that includes a new root chunk. The set of chunks referenced in the manifest is either the same or has strictly grown, but it is not know which chunk are reachable from the new root chunk. The NBS has to pass this value to the client and let them decide. If the client is a datas.database
it will attempt to rebase, read the head of the dataset it is committing to, and execute its mergePolicy
(Dolt passes a noop mergePolicy
).chunks.ErrGCGenerationExpired
: This is similar to a moved root chunk, but with no guarantees about what chunks remain in the ChunkStore. Any information from CS.Has(ctx, chunk)
is now stale. Writers must rewrite all data to the chunkstore.${output,,}
is only supported on Bash 4+. I switched to using tr
so I could run the tests locally.Published by oscarbatori about 4 years ago
We are excited to announce the release of Dolt 0.20.2, including a minor version bump as we introduce a new feature SQL triggers.
SQL triggers are SQL snippets that can be executed every time a row is inserted. Here is a simple example taken from the blog post announcing the feature:
$ dolt sql
> create table a (x int primary key);
> create table b (y int primary key);
> create trigger adds_one before insert on a for each row set new.x = new.x + 1;
> insert into a values (1), (3);
Query OK, 2 rows affected
trigger_blog> select * from a;
+---+
| x |
+---+
| 2 |
| 4 |
+---+
Any legal SQL statement can be executed as a trigger, here we just defined a simple increment.
SET new.x = foo
expressions working for triggers. This required totally rewriting how we were handling setting system variable as well, since these two kinds of statements are equivalent in the parser.Published by oscarbatori about 4 years ago
set new.x = blah
as part of a BEFORE INSERT trigger. Need to rewrite the SET handling parser logic for that.Published by oscarbatori about 4 years ago
enginetests
were due to fixing bugs that were found from testing.engine.ApplyDefaults
method.Published by oscarbatori about 4 years ago
We are excited to announce the release of a minor version of Dolt, going from 0.18.4 to 0.19.0, prompted by the addition of tags, modeled on Git, and read-tables
, a command that provides a form of shallow clone operation that clones only the data at a given commit or branch on a remote.
We are excited about Dolt as a data distribution format, and we want to provide tools for folks distributing data to robustly version their releases. Tags enable data publishers to signal discrete data releases to their users. Users can diff between two tags to compare changes.
dolt read-tables
In discussions with some of our users we found that they were excited about using Dolt as a format for collaborating on data, but needed to then use that data in automated settings where it would be unacceptable to clone the whole history on every clone, for example ephemeral compute resources that use the data for automated jobs. We provided read-tables
as a way to obtain the Dolt database at a given commit, or branch.
As usual we include in this release bug fixes and performance improvements. In this release we particularly emphasized fixes to go-mysql-server
that is becoming an increasingly important interface for many of our users.
$ cd mysql-client-tests
$ docker build -t mysql-tests -f MySQLDockerfile .
$ docker run -it --rm mysql-tests:latest /bin/bash
root@e1fbe3c4579f:/mysql-client-tests# bats mysql-client-tests.bats
✓ python mysql.connector client
✓ python pymysql client
✓ mysql-connector-java client
✓ node mysql client
4 tests, 0 failures
root@e1fbe3c4579f:/mysql-client-tests# exit
doltdb/Commit.go
only work because they match the constants heredoltdb.parentListField
and datas.ParentListField
which should cause the list of parents to never be found
Published by oscarbatori about 4 years ago
We are releasing Dolt 0.18.3 with a host of bug fixes and performance improvements. In particular:
SHOW TABLES
now supports AS OF
syntax, which can point at a timestamp or branchdolt tag
. Basic functionality to create, list, and delete tags. For now the command is hidden. At the Noms layer, tags are implemented as commits. Still todo:
Published by oscarbatori about 4 years ago
We have released Dolt 0.18.2 with bug fixes, performance improvements (mainly around memory usage), and improvements to the way we handle merges.
diff
support for foreign keys and indexestableEditAccumulator
seems to be the only place, and it is discarded after it goes through flushEditAccumulator
, so that doesn't seem to be it. However, setting its fields to nil
fixes the leak, indicating that there is somewhere holding on to old tableEditAccumulator
s. I also checked if async.ActionExecutor
was the culprit, but manual debugging and testing showed that it wasn't holding on to any references once they went through the work
method. This isn't a true fix, as there is still technically a leak for whatever is holding on to the tableEditAccumulator
, but it's a leak of mere kilobytes per hour rather than megabytes per minute.pprof
from before.Published by oscarbatori over 4 years ago
We are pleased to announce a patch release with bug fixes and performance improvements, specifically:
SHOW CREATE
statements from Dolt because the datatypes came back upper case (valid SQL, but breaks a hashtable look up in SQL Alchemy metadata parsing)dolt_history_<table>
tables, which had broken commit filteringdolt commit
will currently go ahead and create the merge commit even if there are unstaged conflicts in one of the merged tables. This is not the behavior we want.dolt commit
also fails if there are any conflicts in any of the tables in the working set.go/utils/repofmt/checkfmt.sh
... That script uses goimports
(go install golang.org/x/tools/cmd/goimports
) to check the import order of the file... Not really sure why this works.run
step in the workflow...dolt log
. The intended migration is thatparentsList
in a commit if it is present, andparents
Set if it is not.parents
Set anymore.parents
Set whenparentsList
is not available.doltdb.CommitSpec
to carry the original input in the refs
case, instead of trying to add a prefix or anything else. Instead, doltdb.Resolve()
takes the current HEAD
ref, and fully resolves the CommitSpec
itself.NewCommitSpec
. In particular, there was a lot of NewCommitSpec("HEAD", "some-branch-name-that-is-not-CWB")
in order to get NewCommitSpec("some-branch-name-that-is-not-CWB", "")
. This collapsed all such uses to the same syntax. "HEAD"
is only used for resolving the CWB now.HEAD
, I've not bothered to always thread the CWBRef
to the Resolve
call.ld
Dolt usage, but I will follow up there when this lands in master
.Published by oscarbatori over 4 years ago
We are excited to announce the release of significant enhancements to Dolt's SQL implementation, as well as a new feature, prompting a minor version bump to 0.18.0.
Dolt's diff
command now supports a -q
flag which will evaluate the results of a query at two commits and compare them:
dolt diff -q <query>
This gives users a richer programmatic interface into analyzing the impact of changes through time by showing them the exact changes in, potentially complex, queries at various points in the commit graph. For example, consider the National Vulnerabilities Database on DoltHub:
%dolt diff p30hoseurm9qfl7jhb9t4l7jfnshk6v9 -q 'select floor(impact_score), count(*) from cve group by floor(impact_score)'
+-----+-------------------------+----------+
| | FLOOR(CVE.impact_score) | COUNT(*) |
+-----+-------------------------+----------+
| < | <NULL> | 8468 |
| > | <NULL> | 8428 |
| < | 1 | 3125 |
| > | 1 | 3124 |
| < | 2 | 41465 |
| > | 2 | 41668 |
| < | 3 | 14324 |
| > | 3 | 14318 |
| < | 4 | 5385 |
| > | 4 | 5418 |
| < | 5 | 22039 |
| > | 5 | 22034 |
| < | 6 | 33798 |
| > | 6 | 33890 |
| < | 9 | 282 |
| > | 9 | 283 |
| < | 10 | 16798 |
| > | 10 | 16864 |
+-----+-------------------------+----------+
This shows the changes in the counts of various vulnerabilities across a range of impact score buckets. We are excited to continue enhancing the richness of the programmatic interfaces we provide into Dolt's commit graph to facilitate more robust automated interactions.
Many of the major use-cases identified for Dolt involve the need to sync with existing RDBMS systems. With this in mind we are committed to moving towards total SQL compatibility as quickly as possible. A major piece of this is foreign keys. Many application servers use foreign keys to maintain valid structure around their data, and not supporting those constructs would make it tough to replicate an existing table structure that used them without manual schema manipulation.
In this release we now support inter-table foreign keys. We do not yet support intra-table foreign keys (i.e. a constraint on a table against the same table), which is less common, though allowed in the standard. We will support such constraints in a future release.
We added support for the list of functions below. You can see an index of functions that we support relative to the MySQL standard in our docs.
GET_LOCK
IS_FREE_LOCK
IS_USED_LOCK
RELEASE_LOCK
RELEASE_ALL_LOCKS
ASCII
BIN
BIT_LENGTH
SIGN
UCASE
UNHEX
ACOS
ASIN
ATAN
CRC32
COS
COT
HEX
DEGREES
RADIANS
SIN
TAN
CURDATE
CURRENT_DATE
CURRENT_TIME
CURRENT_TIMESTAMP
CURTIME
DAYNAME
MICROSECOND
MONTHNAME
TIME_TO_SEC
WEEKOFYEAR
DATE_FORMAT
WEEK
bats
tests2020/06/25 21:04:02 Error running `git merge-base remotes/origin/db/ci-github-actions-check-repo remotes/origin/master` to find merge parent: exit status 128
exit status 1
the exit status 128
is from //go/utils/checkcommiters/main.go
... im thinking could be the go version? i had to use 1.13
here to get past the error that occurred when i used go version ^1.13
regarding the -mod=readonly
flag....linux
and macos
... need to fix the broken tests that have specific dependency issuesdolt
on PRscommit --force
.query-diff
. The methodology is:
RootValue
from & toSortFields
Published by oscarbatori over 4 years ago
A patch release containing bug fixes and performance improvements, in particular we are focused on reducing the number of skipped BATS tests.
dolt schema export
. Other commands like dolt show tables
or dolt diff --sql
still output tags. This is easy to change on case by case basis.