A template for PostgreSQL High Availability with Etcd, Consul, ZooKeeper, or Kubernetes
MIT License
Bot releases are hidden (Show)
Published by CyberDem0n almost 7 years ago
Published by CyberDem0n almost 7 years ago
This version adds support for using Kubernetes as a DCS, allowing to run Patroni as a cloud-native agent in Kubernetes without any additional deployments of Etcd, Zookeeper or Consul.
https://github.com/zalando/patroni/blob/master/docs/releases.rst#version-14
Published by CyberDem0n almost 7 years ago
Published by alexeyklyukin about 8 years ago
This release improves management of Patroni cluster by bring in pause mode, improves maintenance with scheduled and conditional restarts, makes Patroni interaction with Etcd or Zookeeper more resilient and greatly enhances patronictl.
Upgrade notice
When upgrading from releases below 1.0 read about changing of credentials and configuration format at 1.0 release notes.
Pause mode
Previously, one had to send SIGKILL signal to Patroni to stop it without terminating PostgreSQL. The new pause mode detaches Patroni from PostgreSQL cluster-wide without terminating Patroni. It is similar to the maintenance mode in Pacemaker. Patroni is still responsible for updating member and leader keys in DCS, but it will not start, stop or restart PostgreSQL server in the process. There are a few exceptions, for instance, manual failovers, reinitializes and restarts are still allowed. You can read a detailed description of this feature.
In addition, patronictl supports new pause
and resume
commands to toggle the pause mode.
Scheduled and conditional restarts
This change enhances Patroni restarts by adding a couple of conditions that can be verified in order to do the restart. Among the conditions are restarting when PostgreSQL role is either a master or a replica, checking the PostgreSQL version number or restarting only when restart is necessary in order to apply configuration changes.
It is now possible to schedule a restart in the future. Only one scheduled restart per node is supported. It is possible to clear the scheduled restart if it is not needed anymore. A combination of scheduled and conditional restarts is supported, making it possible, for instance, to scheduled minor PostgreSQL upgrades in the night, restarting only the instances that are running the outdated minor version without adding postgres-specific logic to administration scripts.
patronictl restart supports several new options. There is also patronictl flush command to clean the scheduled actions.
Robust DCS interaction
Originally, ping_timeout and connect_timeout values were calculated from the negotiated session timeout. Patroni loop_wait was not taken into account. As
a result, a single retry could take more time than the session timeout, forcing Patroni to release the lock and demote.
This change set ping and connect timeout to half of the value of loop_wait, speeding up detection of connection issues and leaving enough time to retry the connection attempt before loosing the lock.
Postpone updating the Etcd topology known to the client until after the original request. When retrieving the cluster topology, implement the retry timeouts depending on the known number of nodes in the Etcd cluster. This makes our client prefer to get the results of the request to having the up-to-date list of nodes.
Both changes make Patroni connections to DCS more robust in the face of network issues.
Patronictl, monitoring and configuration
Previously, there was no reliable way to query Patroni about PostgreSQL instances that fail to stream changes (for instance, due to connection issues). This change exposes the contents of pg_stat_replication via the /patroni endpoint.
Add a command to create cluster structure in Etcd. The cluster is created with user-specified sysid and leader, and both leader and member keys are made persistent. This command is useful to create so-called master-less configurations, where Patroni cluster consisting of only replicas replicate from the external master node that is unaware of Patroni. Subsequently, one
may remove the leader key, promoting one of the Patroni nodes and replacing
the original master with the Patroni-based HA cluster.
bin_dir
to locate PostgreSQL binaries (Ants Aasma)It is useful to be able to specify the location of PostgreSQL binaries explicitly when Linux distros that support installing multiple PostgreSQL versions at the same time.
custom_conf
of (Alejandro Martínez)Allows for custom configuration file paths, which will be unmanaged by Patroni, details:
https://github.com/zalando/patroni/blob/master/docs/SETTINGS.rst#postgresql
Bug fixes and code improvements
Make sure that Patroni understand 2-digits version numbers when doing conditional restarts based on the PostgreSQL version.
Use the dedicated python module instead of traversing directories manually in order to find DCS modules.
Previously, Patroni did not call any callbacks when attaching to the already running node with the correct role. Since callbacks are often used to route
client connections that could result in the failure to register the running
node in the connection routing scheme. With this fix, Patroni calls on_start
callback even when attaching to the already running node.
Avoid dropping active physical replication slots on master. PostgreSQL cannot
drop such slots anyway. This change makes possible to run non-Patroni managed
replicas/consumers on the master.
Forces Patroni to close all former connections when PostgreSQL node is started. Avoids the trap of reusing former connections if postmaster was killed with SIGKILL.
Make sure that standby names that do not comply with the slot naming rules don't cause the slot creation and standby startup to fail. Replace the dashes in the slot names with underscores and all other characters not allowed in slot names with their unicode codepoints.
Published by alexeyklyukin over 8 years ago
This release introduces the global dynamic configuration that allows dynamic changes of the PostgreSQL and Patroni configuration parameters for the entire HA cluster. It also delivers numerous bugfixes.
Upgrade notice
When upgrading from v0.90 or below, always upgrade all replicas before the master. Since we don't store replication credentials in DCS anymore, an old replica won't be able to connect to the new master.
Dynamic Configuration
Introduce new REST API endpoint /config to provide PostgreSQL and Patroni configuration parameters that should be set globally for the entire HA cluster (master and all the replicas). Those parameters are set in DCS and in many cases can be applied without disrupting PostgreSQL or Patroni. Patroni sets a special flag called "pending restart" visible via the API when some of the values require the PostgreSQL restart. In that case, restart should be issued manually via the API.
Patroni SIGHUP or POST to /reload will make it re-read the configuration file.
See the dynamic configuration for the details on which parameters can be changed and the order of processing difference configuration sources.
The configuration file format has changed since the v0.90. Patroni is still compatible with the old configuration files, but in order to take advantage of the bootstrap parameters one needs to change it. Users are encourage to update them by referring to the documentation.
More flexible configuration*
Introduce database
and config_base_name
configuration parameters. Among others, it makes possible to run Patroni with PipelineDB and other PostgreSQL forks.
Those include the scope, the node name and the namespace, as well as the secrets and makes it easier to run Patroni in a dynamic environment, i.e. Kubernetes Please, refer to the supported environment variables for further details.
This allows patronictl to take advantage of environment-based configuration.
This simplifies identification and configuration of synchronous replication for a given node.
Stability, security and usability improvements
This change reduces the amount of noise generated by Patroni API health checks during the lengthy initialization of this node from the backup.
Avoid running pg_rewind if the source cluster is not the master.
In addition, avoid removing the data directory on an unsuccessful rewind, unless the new parameter remove_data_directory_on_rewind_failure is set to true. By default it is false.
Previously, Patroni always used the replication credentials from the Postgres URL in DCS. That is now changed to take the credentials from the patroni configuration. The secrets (replication username and password) and no longer exposed in DCS.
Demote now runs totally asynchronously without blocking the DCS interactions.
This allows patronictl to issue "protected" requests, i.e. restart or reinitialize, when Patroni is configured to require authorization on those.
Avoids the issues of Patroni not stopping properly when receiving the SIGTERM
Generates and dynamically changes haproxy configuration from the patroni state in the DCS using confide
Makes the callback calls more reliable, particularly in the cluster stop case. In addition, introduce the pg_ctl_timeout
option to set the timeout for the start, stop and restart calls via the pg_ctl
.
Make retries more predictable and robust.
Reduce the connection timeouts to make Zookeeper connection attempts more frequent.
This releases adds support for Consul, includes a new noloadbalance tag, changes the behavior of the clonefrom tag, improves pg_rewind handling and improves patronictl control program.
Consul support
Patroni runs against Consul, in addition to Etcd and Zookeeper. the connection parameters can be configured in the YAML file.
New and improved tags
This tag makes Patroni always return that the replica is not available to the load balancer.
Previously, a node name had to be supplied to the clonefrom, forcing a tagged replica to clone from the specific node. The new implementation makes clonefrom a boolean tag: if it is set to true, the replica becomes a candidate for other replicas to clone from it. When multiple candidates are present, the replicas picks one randomly.
Stability and security improvements
Removes some spurious error messages, improves the stability of the failover, addresses some corner cases with reading data from DCS, shutdown, demote and reattaching of the former leader.
Previously, when stopping Patroni, systemd also sent a signal to PostgreSQL. Since Patroni also tried to stop PostgreSQL by itself, it resulted in sending to different shutdown requests (the smart shutdown, followed by the fast shutdown). That resulted in replicas disconnecting too early and a former master not being able to rejoin after demote. Fix by Jan with prior research by Alexander.
Previously, we only called pg_rewind if the former master had crashed. Change this to always run pg_rewind for the former master as long as pg_rewind is present in the system. This fixes the case when the master is shut down before the replicas managed to get the latest changes (i.e. during the "smart" shutdown).
Both unit and acceptance tests run automatically against Etcd, Zookeeper and Consul on each commit or pull-request.
This prevents a possibility of reading system environment variables by connecting to the PostgreSQL cluster managed by Patroni.
Configuration and control changes
patronictl can use the same configuration file as Patroni itself.
This simplifies generating configuration for Patroni automatically, or merging a single configuration from different sources.
Enables support for DCSs other than Etcd in patronictl.
This release adds support for cascading replication and simplifies Patroni management by providing scheduled failovers. One may use older versions of Patroni (in particular, 0.78) combined with this one in order to migrate to the new release. Note that the scheduled failover and cascading replication related features will only work with Patroni 0.80 and above.
Cascading replication
Add support for the replicatefrom and clonefrom tags for the patroni node (Oleksii Kliukin).
The tag replicatefrom allows a replica to use an arbitrary node a source, not necessary the master. The clonefrom does the same for the initial backup. Together, they enable Patroni to fully support cascading replication.
Add support for running replication methods to initialize the replica even without a running replication connection (Oleksii).
This is useful in order to create replicas from the snapshots stored on S3 or FTP. A replication method that does not require a running replication connection should supply no_master: true in the yaml configuration. Those scripts will still be called in order if the replication connection is present.
Patronictl, API and DCS improvements
Implement scheduled failovers (Feike Steenbergen).
Failovers can be scheduled to happen at a certain time in the future, using either patronictl, or API calls.
Add support for dbuser and password parameters in patronictl (Feike).
Add PostgreSQL version to the health check output (Feike).
Improve Zookeeper support in patronictl (Oleksandr Shulgin)
Migrate to python-etcd 0.43 (Alexander Kukushkin)
Configuration
Tests
Add acceptance tests with behave in order to check real-world scenarios of running Patroni (Alexander, Oleksii).
The tests can be launched manually using the behave command. They are also launched automatically for pull requests and after commits.