database-lab-engine

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

APACHE-2.0 License

Stars
2.1K
Committers
26
View Code on GitHub Visit Website View on X
Ecosystems: PostgreSQL

Bot releases are hidden (Show)

database-lab-engine - DBLab Engine v3.5.0 Latest Release

Published by vitabaks 11 months ago

New features

Improvements and fixes

Internal

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

Full diff

Diff between versions v3.4.0 and v3.5.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.4.0...v3.5.0

Where to start / get help

Provide feedback, contribute

We greatly value your feedback. Connect with us via:

Additional resources where you can get insights about DBLab and Postgres:

Interested in giving back to the project? Here's how you can make an impact:

database-lab-engine - DBLab Engine v3.4.0

Published by NikolayS about 1 year ago

In this update:

New name: DBLab Engine

The new name for the Database Lab Engine is "DBLab Engine". Updates are currently underway across our materials to reflect this change. To align with this change, we have introduced specific domains for the product: dblab.dev and dblab.sh. For ease of access, we have established the following short URLs:

New contributors

🙇 Many thanks to our first-time contributors: @mbobin, @TheLiamGuy, @gromsterus, @ibash, @byx01, @davidsvenson.

By the numbers

Thank you for the support!

New DBLab SE installer

We've expanded the installation options for DBLab SE. Besides its presence in the AWS Marketplace, you can now seamlessly install DBLab SE directly from the Postgres.ai Console. This setup is entirely automated and can be used anywhere:

  • For those who have existing machines, we support the "BYOM" (Bring Your Own Machine) method
  • If you're utilizing AWS, GCP, DigitalOcean, or Hetzner Cloud, the installer handles resource provisioning—including VMs, disks, and more—automatically.

Check out the docs.

Migration notes

For those considering an upgrade, we recommend executing a fresh installation of DBLab. At present, we don't have automation in place for upgrades. If you need to update an existing DBLab installation, it's essential to stop and remove the current containers. Then, initiate a fresh start of the DBLab container using the 3.4.0 tag. More details can be found in the docs. Before initiation, ensure you've adjusted the DBLab configuration as outlined below.

Postgres images

There is a new method for running clones with customized port and socket directories. Please note that this is NOT backward compatible with previous versions of the Postgres images that DBLab utilized. It is crucial to ensure the dockerImage options utilize tags with the -0.3.0 suffix or later, such as postgresai/extended-postgres:15-0.3.0 (details can be found in this MR).

Global telemetry

The global.telemetry subsection has been deprecated and removed. The control to activate/deactivate telemetry has been relocated to the platform section:

platform:
  url: "https://postgres.ai/api/general"
  enableTelemetry: true

Platform

From version v3.4.0 onward, the platform section is enabled by default.

If either the accessToken or orgKey keys aren't empty, ensure the url parameter is specified; otherwise, the DBLab instance will not initialize. By default, neither accessToken nor orgKey are defined when in case of self-managed installations.

platform:
  url: "https://postgres.ai/api/general"
  enableTelemetry: true
#  orgKey: "org_key"
#  accessToken: "platform_access_token"

Docker mounts

Please omit the following Docker mount directives during container startup, as they are no longer necessary:

    --volume /sys/kernel/debug:/sys/kernel/debug:rw \
    --volume /lib/modules:/lib/modules:ro \
    --volume /proc:/host_proc:ro \

Changelog

New features

Improvements and fixes

Internal changes

Where to start / get help

Provide feedback, contribute

We greatly value your feedback. Connect with us via:

Additional resources where you can hear more information about DBLab and Postgres:

Interested in giving back to the project? Here's how you can make an impact:

Diff between versions v3.3.0 and v3.4.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.3.0...v3.4.0

database-lab-engine - Database Lab Engine v3.3.0

Published by NikolayS almost 2 years ago

New features

Improvements and fixes

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

Diff between versions v3.2.0 and v3.3.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.2.0...v3.3.0

database-lab-engine - Database Lab Engine v3.2.0

Published by NikolayS almost 2 years ago

Release v3.2.0 delivers a good number of new features and improvements. DLE UI is extended with two new tabs, "Logs" and "Configuration", to improve the user experience when configuring or reconfiguring DLE for managed Postgres services such as AWS RDS and Google Cloud SQL:

This allows users to see PGDATA initialization errors when setting up DLE, then adjust the configuration and retry initialization right in the UI – without any need to connect to the DLE machine using SSH.

The demo DLE instance can be found here: https://demo.aws.postgres.ai:446/instance (token: demo_token).

Postgres 15 is already supported!

PostgreSQL 15 is already supported by DLE and included in the list of default images, with many extensions shipped with it: https://postgres.ai/docs/database-lab/supported-databases.

AWS Marketplace – the easiest way to get DLE for any Postgres database

DLE in AWS Marketplace has now General Availability (GA) status, it has already been updated to version 3.2.0: https://aws.amazon.com/marketplace/pp/prodview-wlmm2satykuec, offering the Standard license, with support, embedded monitoring, simplified installation, and access to a wider variety of supported Postgres images (including extensions not available by default). An instance with DLE delivers "dev/test databases as a service" and levels up all dev&test processes. For example, on an r5.xlarge instance (4 vCPUs, 32 GiB RAM) with 200 GiB disk space, it is possible to run up to 30 database clones simultaneously (and even more, if shared_buffers is adjusted to lower values) spending as low as $360 per month – less than for a single regular RDS clone of the same size.

New contributor

🙇 The development team greets a new contributor: @Adrinlol. This release wouldn't be possible without other contributors: @agneum, @vitabaks, @DmitryFomin1, @ane4ka, @NikolayS, @denis256, @yuridevx.

What's new

New features

Improvements and fixes

  • Redesign the Retrieval service: !549
  • Mounting of socket directory for non-guest virtualization roles: !557 (@denis256)
  • Add a link to the guide "PostgreSQL configuration in clones": !559
  • Add session manager to CF template: !560
  • Add maintenance_work_mem for Logical mode: !567
  • Report the type of DLE subscription in UI: !563
  • Detect if DLE is running in a Docker container (this resolves issues of running DLE on KVM): !571, !603
  • Apply the configuration and then start Postgres on the restore stage, to reduce the number of restarts needed: !518, !547 (@yuridevx), !558, !572
  • Fix the creation of a restricted user: !586
  • Build CE UI Docker image after merging into the master branch: !590
  • Adjust container readiness health checks: !597, !602
  • Remove deprecated eslint analyzers: !605
  • Increase the number of parallel jobs in logical mode from 2/2 to 4/4 – usually, many more CPU cores are available: !611

Diff showing all changes

Diff between versions v3.1.2 and v3.2.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.2...v3.2.0.

Migration from DLE v3.1.x and configuration notes

There are no special actions that are required to migrate from DLE 3.1.x to 3.2.0: you need to stop the DLE container, change the image, and start a new one using the new image. All snapshots should remain available, and if, at stopping time, some clones were present, they should be automatically recreated by DLE.

Note: if you have an older version, check the previous release notes as well: https://gitlab.com/postgres-ai/database-lab/-/releases.

Configuration directory

The config directory, by default, is ~/.dblab/engine/configs and the corresponding mounted volume must be writable:

  ...
  --volume ~/.dblab/engine/configs:/home/dblab/configs
  ...

Logs directory

Make sure that ~/.dblab/engine/logs is mounted as a volume to the DLE container:

  ...
  --volume ~/.dblab/engine/configs:/home/dblab/configs \
  --volume ~/.dblab/engine/meta:/home/dblab/meta \
  --volume ~/.dblab/engine/logs:/home/dblab/logs \
  ...

Alternatively, you can mount the entire directory (including configs and meta – but make sure that these subdirectories exist, DLE doesn't create them yet):

  ...
  --volume ~/.dblab/engine:/home/dblab
  ...

Query preprocessing

The queryPreprocessing section supports inline SQL now. Examples can be found here: !576#examples

Configuration UI

Because the configuration can be viewed and edited in the UI, make sure:

  • secret values are masked (e.g., verificationToken, Platform accessToken, you can find all masking rules here: !608#masking-rules)
  • DLE configuration matches the reference configuration files in structure and naming; it is not recommended to have custom names and sections

Where to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

database-lab-engine - v3.1.2

Published by NikolayS over 2 years ago

Database Lab Engine v3.1.2

DLE v3.1.2 delivers multiple bug fixes and improvements.

🙇 The development team greets new contributors: @yuridevx, @denis256.

New features and logic adjustments

Improvements and fixes

Contributor's experience

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

Diff between versions v3.1.1 and v3.1.2: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.1...v3.1.2

database-lab-engine - v3.1.1

Published by NikolayS over 2 years ago

Database Lab Engine v3.1.1

DLE v3.1.1 delivers multiple bug fixes and improvements.

The development team greets a new contributor: @Alexand.

Improvements and fixes

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

Diff between versions v3.1.0 and v3.1.1: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.0...v3.1.1

database-lab-engine - Database Lab Engine v3.1.0

Published by NikolayS over 2 years ago

Database Lab Engine v3.1.0

DLE 3.1 has many improvements and several new features, such as native support of pgBackRest and the ability to configure the timezone when using CLI.

In this release, the development team has focused on the Database Lab Engine community, making it easier to get help or contribute. The team greets all new contributors: @Nikolay Devxx, @asotolongo, @Tanya301, @denis-boost, @pietervincken, @ane4ka

Community news

The README.md has been significantly improved. It now has four translations, with more coming in the future. Many thanks to our README translation contributors! It helps engineers worldwide understand DLE concepts and improve the development experience by building powerful dev&test environments.

Migration notes

⚠️ Prior to version 3.1, the default ZFS version DLE's Docker images were using was 0.8.x. For DLE 3.1, the default Docker image (postgresai/dblab-server:3.1.0) uses ZFS 2.1.2. If you need to upgrade DLE from a previous version that worked with a ZFS pool 0.8.x (default for previous DLE versions), consider the following options:

  • Option 1: upgrade your system to use ZFS 2.1, optionally upgrade your pool (zpool upgrade dblab_pool), and then upgrade DLE to use the default image postgresai/dblab-server:3.1.0
  • Option 2: postpone the ZFS upgrade, stay on ZFS 0.8, and upgrade DLE to version 3.1 using a special image, postgresai/dblab-server:3.1.0-zfs0.8

Changelog

New features

Improvements and fixes

Internal

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

If you want to help the project grow, consider various ways of contributing:

Diff between versions v3.0.0 and v3.1.0-rc.1: https://github.com/postgres-ai/database-lab-engine/compare/v3.0.0...v3.1.0-rc.1

database-lab-engine - Database Lab Engine v3.0.3

Published by NikolayS over 2 years ago

DLE 3.0.3 is a backward-compatibility update to ensure that the DLE image is compatible with existing datasets (ZFS 0.8.*). In DLE 3.0.2, in the Docker image ZFS version was prematurely switched to 2.*. The release 3.0.3 mitigates it. There are plans to switch to ZFS 2.* in the future major versions of DLE.

The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository on GitLab or GitHub:

Changelog

Improvements and fixes

Internal

Migration notes

Make sure the DLE running command contains the environment variable --env DOCKER_API_VERSION="1.41".
The maximum supported Docker API version is 1.41, but the client version (1.42) is too new.

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions v3.0.2 and v3.0.3: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.2...v3.0.3

database-lab-engine - Database Lab Engine v3.0.2

Published by NikolayS over 2 years ago

DLE 3.0.2 is a security update, mitigating multiple CVEs reported by various security inspection tools. There are no known critical vulnerabilities that are fixed or are still unfixed in this release but upgrading at the earliest convenience is recommended.

The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository on GitLab or GitHub:

Changelog

Internal

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions v3.0.1 and v3.0.2: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.1...v3.0.2

database-lab-engine - Database Lab Engine v3.0.1

Published by NikolayS over 2 years ago

The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.

Changelog

Improvements and fixes

Internal

How to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions v3.0.0 and v3.0.1: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.0...v3.0.1

database-lab-engine - Database Lab Engine v3.0.0

Published by NikolayS almost 3 years ago

Database Lab Engine (DLE) is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their very own database provisioned in seconds without extra costs.

DLE v3.0.0 includes more than 100 individual changes. This is the most advanced open-source software ever released that empowers development, testing, and troubleshooting environments for fast-growing projects. The use of Database Lab Engine 3.0 provides a competitive advantage to companies via implementing the "Shift-left testing" approach in software development.

Read the blog post about this release: https://postgres.ai/blog/20211221-dle-3-0-0-brings-ui-and-persistent-clones

Among major changes:

  • UI included to the core, it allows working with a single DLE instance,
  • persistent clones: clones now survive DLE (or VM) restart,
  • for the "logical" data provisioning mode: the ability to switch reset clone's state using a snapshot from different pool/dataset,
  • better logging, easier configuration,
  • improvements for the cases with multiple DLEs are running a single machine,
  • PostgreSQL 14 support.

Please note that upgrading from an older version requires some actions from the DLE administrator (see "Migration notes" below).

The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.

Changelog

New features

Improvements and fixes

Internal

Migration notes

The Database Lab configuration contains a new section, embeddedUI, to run UI application that allows observing the current state of the instance and performing basic clone operations:

embeddedUI:
  enabled: true
  dockerImage: "postgresai/ce-ui:latest"
  host: ""
  port: 2346

Note, this application is delivered as a separate Docker container ("Embedded UI") and is controlled by DLE.

Starting with version v3.0.0, DLE collects non-personally identifiable telemetry data.
The Engine uses anonymous usage data to assist users and improve product quality.

it is possible to manage the configuration in the section global:

global:
  telemetry:
    enabled: true
    url: "https://postgres.ai/api/general"
    ...

Please leave it enabled if possible – this will contribute to Database Lab development.
See more details on the Telemetry page in the docs

Now it is available to select a specific pool to work with, use a new option selectedPool in the poolManager section

poolManager:
  selectedPool: "dblab_pool"
  ...

Starting with version v3.0.0, Database Lab Engine can be running with an empty verification token, which is not recommended by default.
In such a case DLE warns about empty token in instance logs and API will not require any credentials. Be sure that this option is configured correctly

In order to make the management of service containers easier, special labels have been added to them:

- `dblab_instance_id`: <dle_instance_id>
- `dblab_engine_name`: <dle_container_name>

For example, there is a snippet of the docker inspection response of a sync instance:

"Labels": {
    "dblab_control": "dblab_sync",
    "dblab_instance_id": "c64mvrvku3086hlp2e6g",
    "dblab_engine_name": "dblab_server",
    "maintainer": "postgres.ai"
}

See more examples of configuration files in the Database Lab repository

The Docker image of DLE v3.0.0: postgresai/dblab-server:3.0.0

Launch command example:

sudo docker run \
  --name dblab_server \
  --label dblab_control \
  --privileged \
  --publish 127.0.0.1:2345:2345 \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume /var/lib/dblab:/var/lib/dblab/:rshared \
  --volume ~/.dblab/engine/configs:/home/dblab/configs:ro \
  --volume ~/.dblab/engine/meta:/home/dblab/meta \
  --env DOCKER_API_VERSION=1.39 \
  --detach \
  --restart on-failure \
  postgresai/dblab-server:3.0.0

If you have problems or questions, please contact our communities for help: https://postgres.ai/docs/questions-and-answers#where-to-get-help

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions 2.5.0 and v3.0.0: https://gitlab.com/postgres-ai/database-lab/-/compare/2.5.0...v3.0.0

database-lab-engine - DLE v3.0.0 Release Candidate 2

Published by NikolayS almost 3 years ago

The second release candidate of DLE version 3.0.0, a major release with many changes and improvements. See the release notes for RC1: https://gitlab.com/postgres-ai/database-lab/-/tags/v3.0.0-rc.1

Compared to RC1, this version fixes multiple bugs, points on logical inconsistency, and issues based on users' feedback during RC1 testing:

Migration notes

Upgrading from version 2.5.x and earlier require some actions from the DLE administrator; see the migration notes for v3.0.0-rc.1: https://gitlab.com/postgres-ai/database-lab/-/tags/v3.0.0-rc.1#migration-notes

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions v3.0.0-rc.1 and v3.0.0-rc.2: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.0-rc.1...v3.0.0-rc.2

database-lab-engine - DLE v3.0.0 Release Candidate 1

Published by NikolayS almost 3 years ago

DLE v3.0.0-rc.1 includes several major changes and many improvements. Most notable changes are:

  • persistent clones: clones now survive DLE (or VM) restart,
  • GUI included to the core, it allows working with a single DLE instance,
  • PostgreSQL 14 support.

Please note that upgrading from an older version requires some actions from the DLE administrator (see "Migration notes" below).

The list below has "WIP" status. The final list for the 3.0 major version release may include some changes.

The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.

Changelog

New features

Improvements and fixes

Internal

Migration notes

The Database Lab configuration contains a new section localUI to run UI application that allows observing the current state of the instance and performing basic clone operations:

localUI:
  enabled: true
  dockerImage: "postgresai/basic-ui:latest"
  host: ""
  port: 2346

Note, this application is delivered as a separate Docker container (Local UI) and running by the Database Lab Engine.

Since v3.0.0-rc.1, DLE collects non-personally identifiable telemetry data.
The Engine uses anonymous usage data to assist users and improve product quality.

it is possible to manage the configuration in the section global:

global:
  telemetry:
    enabled: true
    url: "https://postgres.ai/api/general"
    ...

Please leave it enabled if possible – this will contribute to Database Lab development.
See more details on the Telemetry page in the docs

Now it is available to select a specific pool to work with, use a new option selectedPool in the poolManager section

poolManager:
  selectedPool: "dblab_pool"
  ...

Since version v3.0.0-rc.1, Database Lab Engine can be running with an empty verification token, which is not recommended by default.
In such a case DLE warns about empty token in instance logs and API will not require any credentials. Be sure that this option is configured correctly

In order to make the management of service containers easier, special labels have been added to them:

- `dblab_instance_id`: <dle_instance_id>
- `dblab_engine_name`: <dle_container_name>

For example, there is a snippet of the docker inspection response of a sync instance:

"Labels": {
    "dblab_control": "dblab_sync",
    "dblab_instance_id": "c64mvrvku3086hlp2e6g",
    "dblab_engine_name": "dblab_server",
    "maintainer": "postgres.ai"
}

See more examples of configuration files in the Database Lab repository

The Docker image of DLE 3.0.0-rc.1: registry.gitlab.com/postgres-ai/database-lab/dblab-server:test-3-0-0-rc-1

Launch command example:

sudo docker run \
  --name dblab_server \
  --label dblab_control \
  --privileged \
  --publish 127.0.0.1:2345:2345 \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume /var/lib/dblab:/var/lib/dblab/:rshared \
  --volume ~/.dblab/engine/configs:/home/dblab/configs:ro \
  --volume ~/.dblab/engine/meta:/home/dblab/meta \
  --env DOCKER_API_VERSION=1.39 \
  --detach \
  --restart on-failure \
  registry.gitlab.com/postgres-ai/database-lab/dblab-server:test-3-0-0-rc-1

If you have problems or questions, please contact our communities for help: https://postgres.ai/docs/questions-and-answers#where-to-get-help

Request for feedback

Any feedback would be greatly appreciated:

Compare versions

Diff between versions 2.5.0 and v3.0.0-rc.1: https://gitlab.com/postgres-ai/database-lab/-/compare/2.5.0...v3.0.0-rc.1

Package Rankings
Top 8.17% on Proxy.golang.org
Badges
Extracted from project README
Contributor Covenant
Related Projects
postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating ...

04 Jun 2019 1,301
mathesar

Web application providing an intuitive user experience to databases.

17 Mar 2021 2,235
awesome-postgres

A curated list of awesome PostgreSQL software, libraries, tools and resources, inspired by awesom...

02 Aug 2015 9,603
from-oracle-to-postgresql

Yet another transition guide for developers

21 Nov 2022 3
database-lab

DBLab Engine enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empow...

23 Oct 2019 335
yuniql

Free and open source schema versioning and database migration made natively with .NET/6. NEW THIS...

06 Oct 2019 418
questdb

An open source time-series database for fast ingest and SQL queries

28 Apr 2014 13,423
logto

🧑‍🚀 The better identity infrastructure for developers and the open-source alternative to Auth0.

19 Jun 2021 8,500