DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.
APACHE-2.0 License
Bot releases are visible (Hide)
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Diff between versions v3.4.0 and v3.5.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.4.0...v3.5.0
demo-token
)We greatly value your feedback. Connect with us via:
Additional resources where you can get insights about DBLab and Postgres:
Interested in giving back to the project? Here's how you can make an impact:
Published by NikolayS about 1 year ago
In this update:
The new name for the Database Lab Engine is "DBLab Engine". Updates are currently underway across our materials to reflect this change. To align with this change, we have introduced specific domains for the product: dblab.dev
and dblab.sh
. For ease of access, we have established the following short URLs:
curl -sSL dblab.sh | bash
)demo-token
)demo-token
)🙇 Many thanks to our first-time contributors: @mbobin, @TheLiamGuy, @gromsterus, @ibash, @byx01, @davidsvenson.
Thank you for the support!
We've expanded the installation options for DBLab SE. Besides its presence in the AWS Marketplace, you can now seamlessly install DBLab SE directly from the Postgres.ai Console. This setup is entirely automated and can be used anywhere:
Check out the docs.
For those considering an upgrade, we recommend executing a fresh installation of DBLab. At present, we don't have automation in place for upgrades. If you need to update an existing DBLab installation, it's essential to stop and remove the current containers. Then, initiate a fresh start of the DBLab container using the 3.4.0
tag. More details can be found in the docs. Before initiation, ensure you've adjusted the DBLab configuration as outlined below.
There is a new method for running clones with customized port and socket directories. Please note that this is NOT backward compatible with previous versions of the Postgres images that DBLab utilized. It is crucial to ensure the dockerImage
options utilize tags with the -0.3.0
suffix or later, such as postgresai/extended-postgres:15-0.3.0
(details can be found in this MR).
The global.telemetry
subsection has been deprecated and removed. The control to activate/deactivate telemetry has been relocated to the platform
section:
platform:
url: "https://postgres.ai/api/general"
enableTelemetry: true
From version v3.4.0
onward, the platform
section is enabled by default.
If either the accessToken
or orgKey
keys aren't empty, ensure the url
parameter is specified; otherwise, the DBLab instance will not initialize. By default, neither accessToken
nor orgKey
are defined when in case of self-managed installations.
platform:
url: "https://postgres.ai/api/general"
enableTelemetry: true
# orgKey: "org_key"
# accessToken: "platform_access_token"
Please omit the following Docker mount directives during container startup, as they are no longer necessary:
--volume /sys/kernel/debug:/sys/kernel/debug:rw \
--volume /lib/modules:/lib/modules:ro \
--volume /proc:/host_proc:ro \
ignoreErrors
in subsections logicalDump
and logicalRestore
to allow not to interrupt the process of dump/restore in case of errors - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/699, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/700, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/730
skipPolicies
in subsection logicalRestore
to allow to skip policies (CREATE POLICY
) during restore process, to avoid errors like policy "xxx" for table "yyy" already exists
: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/769
cloneAccessAddresses
in section provision
to control port publishing for clone containers; now limited to 127.0.0.1
by default; support multiple addresses and IPv6: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/786
9.6-0.3.0
... 15-0.3.0
; ⚠️ older tags won't work with DBLab 3.4.xpgvector
extension added to/status
with proper messages - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/696
portPool
inclusive (previously, it was mistakenly exclusive, leading to unexpected behavior): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/785
Error response from daemon: Duplicate mount point: /var/lib/dblab/dblab_pool/dataset_1/dump/
: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/791
trusted.gpg.d
because apt-key
is deprecated - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/701
demo-token
)We greatly value your feedback. Connect with us via:
Additional resources where you can hear more information about DBLab and Postgres:
Interested in giving back to the project? Here's how you can make an impact:
Diff between versions v3.3.0 and v3.4.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.3.0...v3.4.0
Published by NikolayS almost 2 years ago
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Diff between versions v3.2.0 and v3.3.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.2.0...v3.3.0
Published by NikolayS almost 2 years ago
Release v3.2.0 delivers a good number of new features and improvements. DLE UI is extended with two new tabs, "Logs" and "Configuration", to improve the user experience when configuring or reconfiguring DLE for managed Postgres services such as AWS RDS and Google Cloud SQL:
This allows users to see PGDATA initialization errors when setting up DLE, then adjust the configuration and retry initialization right in the UI – without any need to connect to the DLE machine using SSH.
The demo DLE instance can be found here: https://demo.aws.postgres.ai:446/instance (token: demo_token
).
PostgreSQL 15 is already supported by DLE and included in the list of default images, with many extensions shipped with it: https://postgres.ai/docs/database-lab/supported-databases.
DLE in AWS Marketplace has now General Availability (GA) status, it has already been updated to version 3.2.0: https://aws.amazon.com/marketplace/pp/prodview-wlmm2satykuec, offering the Standard license, with support, embedded monitoring, simplified installation, and access to a wider variety of supported Postgres images (including extensions not available by default). An instance with DLE delivers "dev/test databases as a service" and levels up all dev&test processes. For example, on an r5.xlarge instance (4 vCPUs, 32 GiB RAM) with 200 GiB disk space, it is possible to run up to 30 database clones simultaneously (and even more, if shared_buffers
is adjusted to lower values) spending as low as $360 per month – less than for a single regular RDS clone of the same size.
🙇 The development team greets a new contributor: @Adrinlol. This release wouldn't be possible without other contributors: @agneum, @vitabaks, @DmitryFomin1, @ane4ka, @NikolayS, @denis256, @yuridevx.
queryPreprocessing
in logicalRestore
!576
queryPreprocessing
(available in sections logicalRestore
, logicalSnapshot
, physicalSnapshot
) !576
excludeTables
– allow excluding certain tables for data retrieval in logical mode: !536, https://gitlab.com/postgres-ai/database-lab/-/issues/348 (was also backpatched to 3.1.2)maintenance_work_mem
for Logical mode: !567
Diff between versions v3.1.2 and v3.2.0: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.2...v3.2.0.
There are no special actions that are required to migrate from DLE 3.1.x to 3.2.0: you need to stop the DLE container, change the image, and start a new one using the new image. All snapshots should remain available, and if, at stopping time, some clones were present, they should be automatically recreated by DLE.
Note: if you have an older version, check the previous release notes as well: https://gitlab.com/postgres-ai/database-lab/-/releases.
The config directory, by default, is ~/.dblab/engine/configs
and the corresponding mounted volume must be writable:
...
--volume ~/.dblab/engine/configs:/home/dblab/configs
...
Make sure that ~/.dblab/engine/logs
is mounted as a volume to the DLE container:
...
--volume ~/.dblab/engine/configs:/home/dblab/configs \
--volume ~/.dblab/engine/meta:/home/dblab/meta \
--volume ~/.dblab/engine/logs:/home/dblab/logs \
...
Alternatively, you can mount the entire directory (including configs
and meta
– but make sure that these subdirectories exist, DLE doesn't create them yet):
...
--volume ~/.dblab/engine:/home/dblab
...
The queryPreprocessing
section supports inline SQL now. Examples can be found here: !576#examples
Because the configuration can be viewed and edited in the UI, make sure:
verificationToken
, Platform accessToken
, you can find all masking rules here: !608#masking-rules)To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Published by NikolayS over 2 years ago
DLE v3.1.2 delivers multiple bug fixes and improvements.
🙇 The development team greets new contributors: @yuridevx, @denis256.
excludeTables
– exclude certain tables for data retrieval in logical mode: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/536, https://gitlab.com/postgres-ai/database-lab/-/issues/348
--restricted
flag https://gitlab.com/postgres-ai/database-lab/-/merge_requests/532, https://gitlab.com/postgres-ai/database-lab/-/issues/356
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Diff between versions v3.1.1 and v3.1.2: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.1...v3.1.2
Published by NikolayS over 2 years ago
DLE v3.1.1 delivers multiple bug fixes and improvements.
The development team greets a new contributor: @Alexand.
shared_preload_libraries
- https://gitlab.com/postgres-ai/database-lab/-/merge_requests/522
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Diff between versions v3.1.0 and v3.1.1: https://github.com/postgres-ai/database-lab-engine/compare/v3.1.0...v3.1.1
Published by NikolayS over 2 years ago
DLE 3.1 has many improvements and several new features, such as native support of pgBackRest and the ability to configure the timezone when using CLI.
In this release, the development team has focused on the Database Lab Engine community, making it easier to get help or contribute. The team greets all new contributors: @Nikolay Devxx, @asotolongo, @Tanya301, @denis-boost, @pietervincken, @ane4ka
The README.md has been significantly improved. It now has four translations, with more coming in the future. Many thanks to our README translation contributors! It helps engineers worldwide understand DLE concepts and improve the development experience by building powerful dev&test environments.
⚠️ Prior to version 3.1, the default ZFS version DLE's Docker images were using was 0.8.x. For DLE 3.1, the default Docker image (postgresai/dblab-server:3.1.0
) uses ZFS 2.1.2. If you need to upgrade DLE from a previous version that worked with a ZFS pool 0.8.x (default for previous DLE versions), consider the following options:
zpool upgrade dblab_pool
), and then upgrade DLE to use the default image postgresai/dblab-server:3.1.0
postgresai/dblab-server:3.1.0-zfs0.8
/snapshot
API calls which could take many seconds in the past https://gitlab.com/postgres-ai/database-lab/-/merge_requests/426
dataStateAt
is determined - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/356
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback will be greatly appreciated:
If you want to help the project grow, consider various ways of contributing:
Diff between versions v3.0.0 and v3.1.0-rc.1: https://github.com/postgres-ai/database-lab-engine/compare/v3.0.0...v3.1.0-rc.1
Published by NikolayS over 2 years ago
DLE 3.0.3 is a backward-compatibility update to ensure that the DLE image is compatible with existing datasets (ZFS 0.8.*). In DLE 3.0.2, in the Docker image ZFS version was prematurely switched to 2.*. The release 3.0.3 mitigates it. There are plans to switch to ZFS 2.* in the future major versions of DLE.
The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository on GitLab or GitHub:
Make sure the DLE running command contains the environment variable --env DOCKER_API_VERSION="1.41"
.
The maximum supported Docker API version is 1.41, but the client version (1.42) is too new.
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback would be greatly appreciated:
Diff between versions v3.0.2 and v3.0.3: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.2...v3.0.3
Published by NikolayS over 2 years ago
DLE 3.0.2 is a security update, mitigating multiple CVEs reported by various security inspection tools. There are no known critical vulnerabilities that are fixed or are still unfixed in this release but upgrading at the earliest convenience is recommended.
The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository on GitLab or GitHub:
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback would be greatly appreciated:
Diff between versions v3.0.1 and v3.0.2: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.1...v3.0.2
Published by NikolayS over 2 years ago
The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.
To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.
Any feedback would be greatly appreciated:
Diff between versions v3.0.0 and v3.0.1: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.0...v3.0.1
Published by NikolayS almost 3 years ago
Database Lab Engine (DLE) is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their very own database provisioned in seconds without extra costs.
DLE v3.0.0 includes more than 100 individual changes. This is the most advanced open-source software ever released that empowers development, testing, and troubleshooting environments for fast-growing projects. The use of Database Lab Engine 3.0 provides a competitive advantage to companies via implementing the "Shift-left testing" approach in software development.
Read the blog post about this release: https://postgres.ai/blog/20211221-dle-3-0-0-brings-ui-and-persistent-clones
Among major changes:
Please note that upgrading from an older version requires some actions from the DLE administrator (see "Migration notes" below).
The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.
reset
command - https://gitlab.com/postgres-ai/database-lab/-/issues/291
dumpLocation
directory before taking a dump - https://gitlab.com/postgres-ai/database-lab/-/issues/294
v
while comparing Git tags to run CI jobs - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/382
The Database Lab configuration contains a new section, embeddedUI
, to run UI application that allows observing the current state of the instance and performing basic clone operations:
embeddedUI:
enabled: true
dockerImage: "postgresai/ce-ui:latest"
host: ""
port: 2346
Note, this application is delivered as a separate Docker container ("Embedded UI") and is controlled by DLE.
Starting with version v3.0.0, DLE collects non-personally identifiable telemetry data.
The Engine uses anonymous usage data to assist users and improve product quality.
it is possible to manage the configuration in the section global
:
global:
telemetry:
enabled: true
url: "https://postgres.ai/api/general"
...
Please leave it enabled if possible – this will contribute to Database Lab development.
See more details on the Telemetry page in the docs
Now it is available to select a specific pool to work with, use a new option selectedPool
in the poolManager
section
poolManager:
selectedPool: "dblab_pool"
...
Starting with version v3.0.0, Database Lab Engine can be running with an empty verification token, which is not recommended by default.
In such a case DLE warns about empty token in instance logs and API will not require any credentials. Be sure that this option is configured correctly
In order to make the management of service containers easier, special labels have been added to them:
- `dblab_instance_id`: <dle_instance_id>
- `dblab_engine_name`: <dle_container_name>
For example, there is a snippet of the docker inspection response of a sync instance:
"Labels": {
"dblab_control": "dblab_sync",
"dblab_instance_id": "c64mvrvku3086hlp2e6g",
"dblab_engine_name": "dblab_server",
"maintainer": "postgres.ai"
}
See more examples of configuration files in the Database Lab repository
The Docker image of DLE v3.0.0: postgresai/dblab-server:3.0.0
Launch command example:
sudo docker run \
--name dblab_server \
--label dblab_control \
--privileged \
--publish 127.0.0.1:2345:2345 \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /var/lib/dblab:/var/lib/dblab/:rshared \
--volume ~/.dblab/engine/configs:/home/dblab/configs:ro \
--volume ~/.dblab/engine/meta:/home/dblab/meta \
--env DOCKER_API_VERSION=1.39 \
--detach \
--restart on-failure \
postgresai/dblab-server:3.0.0
If you have problems or questions, please contact our communities for help: https://postgres.ai/docs/questions-and-answers#where-to-get-help
Any feedback would be greatly appreciated:
Diff between versions 2.5.0 and v3.0.0: https://gitlab.com/postgres-ai/database-lab/-/compare/2.5.0...v3.0.0
Published by NikolayS almost 3 years ago
The second release candidate of DLE version 3.0.0, a major release with many changes and improvements. See the release notes for RC1: https://gitlab.com/postgres-ai/database-lab/-/tags/v3.0.0-rc.1
Compared to RC1, this version fixes multiple bugs, points on logical inconsistency, and issues based on users' feedback during RC1 testing:
host
parameter from the server configuration since it is always running inside a Docker container https://gitlab.com/postgres-ai/database-lab/-/merge_requests/388
verificationToken
in configs https://gitlab.com/postgres-ai/database-lab/-/merge_requests/388
instance_id
file to the metadata directory to avoid conflicts when multiple DLEs are running on the same machine https://gitlab.com/postgres-ai/database-lab/-/merge_requests/396
v
https://gitlab.com/postgres-ai/database-lab/-/merge_requests/382
Upgrading from version 2.5.x and earlier require some actions from the DLE administrator; see the migration notes for v3.0.0-rc.1
: https://gitlab.com/postgres-ai/database-lab/-/tags/v3.0.0-rc.1#migration-notes
Any feedback would be greatly appreciated:
Diff between versions v3.0.0-rc.1 and v3.0.0-rc.2: https://gitlab.com/postgres-ai/database-lab/-/compare/v3.0.0-rc.1...v3.0.0-rc.2
Published by NikolayS almost 3 years ago
DLE v3.0.0-rc.1 includes several major changes and many improvements. Most notable changes are:
Please note that upgrading from an older version requires some actions from the DLE administrator (see "Migration notes" below).
The list below has "WIP" status. The final list for the 3.0 major version release may include some changes.
The development team asks everyone to participate in testing and report any bugs or improvement ideas in the DLE repository: https://gitlab.com/postgres-ai/database-lab.
reset
command - https://gitlab.com/postgres-ai/database-lab/-/issues/291
dumpLocation
directory before taking a dump - https://gitlab.com/postgres-ai/database-lab/-/issues/294
The Database Lab configuration contains a new section localUI
to run UI application that allows observing the current state of the instance and performing basic clone operations:
localUI:
enabled: true
dockerImage: "postgresai/basic-ui:latest"
host: ""
port: 2346
Note, this application is delivered as a separate Docker container (Local UI) and running by the Database Lab Engine.
Since v3.0.0-rc.1, DLE collects non-personally identifiable telemetry data.
The Engine uses anonymous usage data to assist users and improve product quality.
it is possible to manage the configuration in the section global
:
global:
telemetry:
enabled: true
url: "https://postgres.ai/api/general"
...
Please leave it enabled if possible – this will contribute to Database Lab development.
See more details on the Telemetry page in the docs
Now it is available to select a specific pool to work with, use a new option selectedPool
in the poolManager
section
poolManager:
selectedPool: "dblab_pool"
...
Since version v3.0.0-rc.1, Database Lab Engine can be running with an empty verification token, which is not recommended by default.
In such a case DLE warns about empty token in instance logs and API will not require any credentials. Be sure that this option is configured correctly
In order to make the management of service containers easier, special labels have been added to them:
- `dblab_instance_id`: <dle_instance_id>
- `dblab_engine_name`: <dle_container_name>
For example, there is a snippet of the docker inspection response of a sync instance:
"Labels": {
"dblab_control": "dblab_sync",
"dblab_instance_id": "c64mvrvku3086hlp2e6g",
"dblab_engine_name": "dblab_server",
"maintainer": "postgres.ai"
}
See more examples of configuration files in the Database Lab repository
The Docker image of DLE 3.0.0-rc.1: registry.gitlab.com/postgres-ai/database-lab/dblab-server:test-3-0-0-rc-1
Launch command example:
sudo docker run \
--name dblab_server \
--label dblab_control \
--privileged \
--publish 127.0.0.1:2345:2345 \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /var/lib/dblab:/var/lib/dblab/:rshared \
--volume ~/.dblab/engine/configs:/home/dblab/configs:ro \
--volume ~/.dblab/engine/meta:/home/dblab/meta \
--env DOCKER_API_VERSION=1.39 \
--detach \
--restart on-failure \
registry.gitlab.com/postgres-ai/database-lab/dblab-server:test-3-0-0-rc-1
If you have problems or questions, please contact our communities for help: https://postgres.ai/docs/questions-and-answers#where-to-get-help
Any feedback would be greatly appreciated:
Diff between versions 2.5.0 and v3.0.0-rc.1: https://gitlab.com/postgres-ai/database-lab/-/compare/2.5.0...v3.0.0-rc.1