Breaking changes

• UAST UDFs now returns data in a specific format, in a BLOB column.

• You must regenerate indexes to use this version. pilosalib index driver is now called pilosa, and the old pilosa driver does not exists anymore.

Bugfixing

• *: pushdown projections properly in the files table
• remove oniguruma support from prebuilt binaries
• gitbase: close repositories in squash table
• pilosalib: fix index usage with parallelism
• sql/plan: make sure ResolvedTable closes row iterators
• vendor: upgrade go-git, fix closed file in pack iter
• gitbase: remove warnings on symbolic references
• function: fix race condition getting UASTs
• sql/analyzer: check non-parallelizable nodes under QueryProcess
• sql/analyzer: execute parallelize as last rule
• rule: do not error when parts of joins are not squashable
• gitbase: close siva FS after use
• If deferredColumns is an alias then resolve it.
• Fix get/set from global table.
• build: disable static linking and oniguruma
• rule: fix unability to check conditions of natural joins

New features

• A new way to serialize UASTs
• uast_children new UDF
• uast_extract new UDF
• *: implement show full processlist
• *: implement session configuration
• sql/index: remove old pilosa driver and rename pilosalib to pilosa
• gitbase: implement sql.PartitionCounter in all tables
• *: correctly handle process management using processlist
• sql/plan: make create index sync by default
• sql/analyzer: resolve aggregate aliases in groupby grouping
• Upgrade mysql and vitess.
• build: only tag latest in docker when version is not prerelease.
• build: make static build with oniguruma enabled
• internal/function: modify uast_extract to return a flattened array
• sql: make table resolution case insensitive
• Support ON/OFF for SET autocommit
• Update go-mysql-server to do not force timeouts
• Add timeout on the configuration

Performance improvements

• Update pilosa to the latest version
• Add partitions to process several repositories at the same time
• Add UAST cache
• Add language cache
• Compile gitbase using oniguruma instead regular regex expressions to make LANGUAGE function faster
• One cache for repository pool.
• gitbase: use KeepDescriptors go-git option
• *: make SquashedTable not implement sql.Node

Documentation

• Make some snippets copyable
• docs: fix how to install oniguruma on linux
• docs: fix uast function descriptions to reflect the new uast serialization
• docs: explain how uast and uast_mode functions serialize data
• docs: add GITBASE_CACHESIZE_MB to the documentation
• docs: add note about oniguruma support in static binary
• docs: guide on how to optimize queries
• docs: add code example on how to manage serialized uast data
• docs: fix optional dependencies
• docs: Add instructions how to run in docker with bblfsh
• docs: improve uast functions documentation

Bugfixing

• Update go-mysql-server and pilosa dependencies

Bugfixing

• build: disable static linking and oniguruma
• rule: fix unability to check conditions of natural joins

Documentation

• docs: guide on how to optimize queries
• docs: add code example on how to manage serialized uast data
• docs: fix optional dependencies
• docs: Add instructions how to run in docker with bblfsh
• docs: improve uast functions documentation

• Update go-mysql-server to do not force timeouts
• Add timeout on the configuration

Breaking changes

With this version is not possible to use old UAST serialization anymore.

New features

• Upgrade mysql and vitess.
• build: only tag latest in docker when version is not prerelease.
• build: make static build with oniguruma enabled
• internal/function: modify uast_extract to return a flattened array
• sql: make table resolution case insensitive
• Support ON/OFF for SET autocommit

Bugfixing

• rule: do not error when parts of joins are not squashable
• gitbase: close siva FS after use
• If deferredColumns is an alias then resolve it.
• Fix get/set from global table.

Documentation

• docs: explain how uast and uast_mode functions serialize data
• docs: add GITBASE_CACHESIZE_MB to the documentation
• docs: add note about oniguruma support in static binary

Breaking changes

You must need to regenerate indexes to use this version. pilosalib index driver is now called pilosa, and the old pilosa driver does not exists anymore.

New features

• sql/index: remove old pilosa driver and rename pilosalib to pilosa
• gitbase: implement sql.PartitionCounter in all tables
• *: correctly handle process management using processlist
• sql/plan: make create index sync by default
• sql/analyzer: resolve aggregate aliases in groupby grouping

Bugfixing

• vendor: upgrade go-git, fix closed file in pack iter
• gitbase: remove warnings on symbolic references
• function: fix race condition getting UASTs
• sql/analyzer: check non-parallelizable nodes under QueryProcess
• sql/analyzer: execute parallelize as last rule

Documentation

• docs: fix uast function descriptions to reflect the new uast serialization

New features

• *: implement show full processlist
• *: implement session configuration

Bugfixing

• *: pushdown projections properly in the files table
• remove oniguruma support from prebuilt binaries
• gitbase: close repositories in squash table
• pilosalib: fix index usage with parallelism
• sql/plan: make sure ResolvedTable closes row iterators

Performance improvements

• One cache for repository pool.
• gitbase: use KeepDescriptors go-git option
• *: make SquashedTable not implement sql.Node

Documentation

• docs: fix how to install oniguruma on linux

Breaking changes

UAST UDFs now returns data in a specific format, in a BLOB column. To make gitbase able to return the old JSON format, you need to add --old-uast-serialization on gitbase execution. This format and the flag will be removed in future versions.

New features

• A new way to serialize UASTs
• uast_children new UDF
• uast_extract new UDF

Performance improvements

• Update pilosa to the latest version
• Add partitions to process several repositories at the same time
• Add UAST cache
• Add language cache
• Compile gitbase using oniguruma instead regular regex expressions to make LANGUAGE function faster

Documentation

• Make some snippets copyable

This version introduces an incompatible change processing UASTs. Now the format of UAST returned by the UDF uast is semantic. Previously the format was annotated

A new UDF uast_mode was added to get UASTs in a specific format (semantic, annotated or native). Example:

uast_mode('annotated', blob_content, language(tree_entry_name, blob_content)), '//*[@roleIdentifier]')

New Features

• Upgrade bblfsh client and add uast_mode function #439
• Pilosa index driver as library (pilosalib)
• sql: add support for show indexes
• vendor: upgrade bblfsh client-go
• Dockerfile: update to use --directories flag
• cmd/gitbase: load siva files and git indistinctly
• *: rename ref_commits index to history_index
• gitbase: add a more descriptive error connecting to bblfsh
• internal/rule,gitbase: use refs indexes in squash tables
• internal/rule: display indexed tables in squash
• gitbase: do not warn on not found HEAD
• Now, DESCRIBE statement return used indexed ids on queries

Performance improvements

• Optimize struct layout #437
• Add regression testing queries #431
• Do not register pilosa driver if no connection.
• *: add siva files to the repository pool with an ID
• Update to new go-git version (~ 40%-60% speed performance, 60% less memory)
• sql/plan: more efficient implementation of GroupBy

Bugfixing

• Solved a problem parsing USING in CREATE INDEX
• gitbase: do not skip repo when it has no head in squash
• internal/function: recover from panics in uast functions
• Fix "NOT expressions with non existing value"
• fix panic when bmp is nil
• Fix merging indexes when row is nil.
• Add regression queries

Documentation

• Documentation enhancenments #434, #435
• docs: update references to go-mysql-server

This version introduces an incompatible change processing UASTs. Now the format of UAST returned by the UDF uast is semantic. Previously the format was annotated

A new UDF uast_mode was added to get UASTs in a specific format (semantic, annotated or native). Example:

uast_mode('annotated', blob_content, language(tree_entry_name, blob_content)), '//*[@roleIdentifier]')

Improvements

• Documentation enhancenments #434, #435
• Optimize struct layout #437
• Add regression testing queries #431
• Remove unused code #445
• Upgrade bblfsh client and add uast_mode function #439

Bugfixing

• Solved a problem parsing USING in CREATE INDEX

New Features

• internal/rule,gitbase: use refs indexes in squash tables
• internal/rule: display indexed tables in squash
• gitbase: do not warn on not found HEAD
• Now, DESCRIBE statement return used indexed ids on queries

Performance improvements

• Update to new go-git version (~ 40%-60% speed performance, 60% less memory)
• sql/plan: more efficient implementation of GroupBy

Bugfixing

• Add regression queries

New Features

• Pilosa index driver as library (pilosalib)
• sql: add support for show indexes
• vendor: upgrade bblfsh client-go
• Dockerfile: update to use --directories flag
• cmd/gitbase: load siva files and git indistinctly
• *: rename ref_commits index to history_index
• gitbase: add a more descriptive error connecting to bblfsh

Performance improvements

• Do not register pilosa driver if no connection.
• *: add siva files to the repository pool with an ID

Bugfixing

• gitbase: do not skip repo when it has no head in squash
• internal/function: recover from panics in uast functions
• Fix "NOT expressions with non existing value"
• fix panic when bmp is nil
• Fix merging indexes when row is nil.

Documentation

• docs: update references to go-mysql-server

New Features

• sql/index/pilosa: implement negation indexes
• sql/analyzer: use indexes with 'NOT IN' expressions
• sql/plan: disallow indexes on blob and json columns
• add version() udf
• sql/expression/function/aggregation: allow conversion in AVG
• sql/analyzer: resolve ascend and descend indexes
• sqle: log query in debug

Performance improvements

• gitbase: compress keys using zlib
• sql/analyzer: add a rule to evaluate filters ahead of time

Bugfixing

• internal/rule: fix projections when squash joins
• cmd/gitbase: fix version and build info
• cmd/gitbase: remove -h flag as shortcut for --host
• gitbase: make squash iterators cancellable
• gitbase: fix bugs handling git errors
• sql/analyzer: remove unnecessary qualification of star
• sql/analyzer: fix resolveNaturalJoins rule to work properly with table aliases
• sql/analyzer: add a rule to reorder aggregations inside projections
• sql/parse: fix parsing when there are preceding spaces

Documentation

• docs: Add troubleshooting info about connecting with MySQL client 8.x
• docs: Add indexes section into the summary

New Features

• sql/driver/pilosa: implement AscendIndex and DescendIndex
• plan/index: Add tracing to index creation
• sql/(index,plan): add logs for index creation
• sql/function/aggregation: add SUM aggregation function
• internal/rule: make SquashedTable tree more explanatory
• vendor: upgrade bblfsh client-go
• cmd/gitbase/command: make pilosa endpoint and index directory configurable through env vars
• *: implement commit_files table
• Documentation: https://docs.sourced.tech/gitbase/
• cmd/gitbase: add readonly flag
• cmd/server: add trace parameters to enable jaeger tracing
• sql/index: improved index tracing
• functions: Add CONCAT(...) function.
• sql/index: improved index tracing
• sql/plan: allow synchronous index creations

Performance improvements

• index/pilosa: batch boltdb and pilosa writes
• sql/analyzer: rule to move join conditions to filters
• Do not use full path in plain repository IDs
• internal/function: do not make requests to bblfsh for unsupported languages
• *: use indexes in the topmost squashed table
• cmd/gitbase: make squash rule active by default
• gitbase: custom encode and decode of index keys to save space
• index/pilosa: speedup index iterator

Bugfixing

• sql/analyzer: rule to resolve sort based on dependencies
• sql/parse: fix order by with group by
• sql/analyzer: improve error message when an alias is defined and used in the same projection
• sql/analyzer: fix star resolution when there are natural joins involved
• sql/(plan,index): force index deletion on save failure
• travis: build and upload macOS releases
• internal/rule: fix squashjoins rule to squash projections properly too
• gitbase: ignore .DS_Store files in darwin
• internal/rule: fix indexes of filters in squash
• gitbase: fix tree entry index iteration when object is unpacked
• Fix drop index panic
• sql/function: make time functions not return errors on conversion
• index/pilosa: do not error when key is not found
• sql/analyzer: fix columns in the resolveNaturalJoins rule when a Natural Join is transformed
• internal/function: don't fail if the uast function can't parse a blob

Documentation

• docs: add commit_files table to the schema documentation
• docs: add create index examples
• docs: new examples
• Add GITBASE_READONLY to docs
• docs: add indexes section

New Features

• cmd/gitbase: add readonly flag
• cmd/server: add trace parameters to enable jaeger tracing
• sql/index: improved index tracing
• functions: Add CONCAT(...) function.

Performance improvements

• gitbase: custom encode and decode of index keys to save space
• index/pilosa: speedup index iterator

Bugfixing

• gitbase: ignore .DS_Store files in darwin
• internal/rule: fix indexes of filters in squash
• gitbase: fix tree entry index iteration when object is unpacked
• Fix drop index panic
• sql/function: make time functions not return errors on conversion

Documentation

• docs: add commit_files table to the schema documentation
• docs: add create index examples

New Features

• sql/driver/pilosa: implement AscendIndex and DescendIndex
• plan/index: Add tracing to index creation
• sql/(index,plan): add logs for index creation
• sql/function/aggregation: add SUM aggregation function
• internal/rule: make SquashedTable tree more explanatory
• vendor: upgrade bblfsh client-go
• cmd/gitbase/command: make pilosa endpoint and index directory configurable through env vars
• *: implement commit_files table
• Documentation: https://docs.sourced.tech/gitbase/

Performance improvements

• index/pilosa: batch boltdb and pilosa writes
• sql/analyzer: rule to move join conditions to filters
• Do not use full path in plain repository IDs
• internal/function: do not make requests to bblfsh for unsupported languages
• *: use indexes in the topmost squashed table
• cmd/gitbase: make squash rule active by default

Bugfixing

• sql/analyzer: rule to resolve sort based on dependencies
• sql/parse: fix order by with group by
• sql/analyzer: improve error message when an alias is defined and used in the same projection
• sql/analyzer: fix star resolution when there are natural joins involved
• sql/(plan,index): force index deletion on save failure
• travis: build and upload macOS releases
• internal/rule: fix squashjoins rule to squash projections properly too

New Features

• gitbase: implement sql.Indexable on all tables
• *: implement indexable interfaces on basic and relation tables
• cmd/gitbase: register pilosa index on server startup
• Make repo dir configurable in Dockerfile
• *: remove commit_has_blob, squash rule for commit_blobs
• *: remove commit_has_tree, add squashes for commit_trees
• *: remove history_idx, squash rules for ref_commits
• *: add squashes for joining repos with commits, entries and blobs
• gitbase: implement commit_blobs table
• Add commit_trees table
• gitbase: implement ref_commits table
• cmd/gitbase: glob support for adding repositories
• *: rename columns for easier natural joins, add repository_id

Bugfixing

• make it clear that the server is now running
• gitbase: implement files table with pushdown
• internal/function: language should return nil for unknown languages

Bugfixing

• Fix incorrect error handling in commitsIter.New
• Ignore missing HEAD references

New Features

• Update enry dependency to v1.6.4
• Add bblfsh integration
• Relicense to Apache 2.0
• Skip git errors in iterators if GITBASE_SKIP_GIT_ERRORS is set
• Add commit parents to commits table
• Add support for siva files
• Support multiple siva and git folders
• Add Dockerfile

Performance improvements

• Don`t read blob content if it's not in the projection
• Skip hash references that do not point to commits
• Improve history_idx and commit_has_tree performance with an LRU cache