Crunchy Data is pleased to announce the release of PostgreSQL Operator 4.3.0 Beta 3. We encourage you to download it and try it out.

Changes Since Beta 2:

Breaking Changes

• pgo-installer is renamed to pgo-deployer
• The examples files that pgo-deployer have been renamed, i.e. deploy.yml
• The names of the scheduled backups are shortened to use the pattern <clusterName>-<backupType>-sch-backup
• The --cpu flags on pgo create cluster, pgo update cluster, pgo create pgbouncer, pgo update pgbouncer, as well as the --pgbackrest-cpu and --pgbouncer-cpu flags on the pgo create cluster and pgo update cluster commands now set both the Resource Request and Limit for CPU

Features

Automatic PostgreSQL Operator Upgrade Process

One of the biggest challenges to using a newer version of the PostgreSQL
Operator was upgrading from an older version.

This release introduces the ability to upgrade from an older version of the
Operator (as early as 4.1.0) to the newest version (4.3.0) using the
pgo upgrade command.

External WAL Volume

An optimization used for improving PostgreSQL performance related to file system
usage is to have the PostgreSQL write-ahead logs (WAL) written to a different
mounted volume than other parts of the PostgreSQL system, such as the data
directory.

To support this, the PostgreSQL Operator nows supports the ability to specify an
external volume for writing the PostgreSQL write-head log (WAL) during cluster
creation, which carries through to replicas and clones. When not specified, the
WAL resides within the PGDATA directory and volume, which is the present
behavior.

To create a PostgreSQL cluster to use an external volume, one can use the
--wal-storage-config flag at cluster creation time to select the storage
configuration to use, e.g.

pgo create cluster --wal-storage-config=nfsstorage hippo

Additionally, it is also possible to specify the size of the WAL storage on all
newly created clusters. When in use, the size of the volume can be overridden
per-cluster. This is specified with the --wal-storage-size flag, i.e.

pgo create cluster --wal-storage-config=nfsstorage --wal-storage-size=10Gi hippo

This implementation does not define the WAL volume in any deployment templates
because the volume name and mount path are constant.

Improved Custom Configuration for PostgreSQL Clusters

The ability to customize the configuration for a PostgreSQL cluster with the
PostgreSQL Operator can now be easily modified by making changes directlry to
the ConfigMap that is created with each PostgreSQL cluster. The ConfigMap, which
follows the pattern <clusterName>-pgha-config (e.g. hippo-pgha-config for
pgo create cluster hippo), manages the user-facing configuration settings
available for a PostgreSQL cluster, and when modified, it will automatically
synchronize the settings across all primaries and replicas in a PostgreSQL
cluster.

Presently, the ConfigMap can be edited using the kubectl edit cm command, and
future iterations will add functionality to the PostgreSQL Operator to make this
process easier.

Other Features

• A script called client-setup.sh is now included to set up the client libraries after the install process from pgo-deployer is run
• The pgo-deployer now deploys the metrics stack (Prometheus, Grafana)
• One can specify the "image prefix" (e.g. crunchydata) for the containers that are deployed by the PostgreSQL Operator. This adds two fields to the pgcluster CRD: CCPImagePrefix and PGOImagePrefix. These are also available on pgo create cluster with the --ccp-image-prefix and --pgo-image-prefix flags respectively
• The pgBouncer deployment can be scaled. This can be done from the pgo create cluster --pgbouncer-replicas, pgo create pgbouncer --replicas, and pgo update pgbouncer --replicas flags. The default number of replicas is "1", which deploys one pgBouncer Pod. This and the other pgBouncer CRD attributes can also be set declaratively in a pgcluster CR.

Fixes

• The default PostgreSQL memory request was lowered to 128Mi to avoid crashes, particularly in test environments. This is actually set in the installers so it's easily configurable, and the "default of last resort" is still 512Mi to be consistent with the PostgreSQL defaults.
• pgo-deployer can now work with external container registries as well as image pull secrets
• The ConfigMap and ClusterRoleBinding for the pgo-deployer container are now cleaned up
• The stanza-create Job now waits for both the PostgreSQL cluster and the pgBackRest repository to be ready before executing
• The Grafana and Prometheus Pods no longer crash on GKE
• pgo clone --metrics now enables the metrics on the cloned cluster. This fixes a regression with the feature
• Ensure commands that execute SQL directly on a container (e.g. pgo show user) work if PostgreSQL is deployed to a nonstandard port

Crunchy Data announces the release of the 'PostgreSQL Operator 4.3.0 Beta 2](https://www.crunchydata.com/developers/download-postgres/containers/postgres-operator).

Changes since Beta 1:

Breaking Changes

• "Limit" resource parameters are no longer set on the containers, in particular, the PostgreSQL container, due to undesired behavior stemming from the host machine OOM killer. Further details can be found in the original pull request.
• Added DefaultInstanceMemory, DefaultBackrestMemory, and DefaultPgBouncerMemory options to the pgo.yaml configuration to allow for the setting of default memory requests for PostgreSQL instances, the pgBackRest repository, and pgBouncer instances respectively.
• If unset by either the PostgreSQL Operator configuration or one-off, the default memory resource requests for the following applications are:
  • PostgreSQL: 512Mi
  • pgBackRest: 48Mi
  • pgBouncer: 24Mi
• Remove the Default...ContainerResources set of parameters from the pgo.yaml configuration file.
• Remove the PreferredFailoverNode feature, as it had already been effictively removed.
• Remove explicit rm calls when cleaning up PostgreSQL clusters. This behavior is left to the storage provsioner that one deploys with their PostgreSQL instances

Features

Elimination of ClusterRole Requirement for the PostgreSQL Operator

PostgreSQL Operator 4.0 introduced the ability to manage PostgreSQL clusters
across multiple Kubernetes Namespaces. PostgreSQL Operator 4.1 built on this
functionality by allowing users to dynamically control which Namespaces it
managed as well as the PostgreSQL clusters deployed to them. In order to
leverage this feature, one must grant a ClusterRole
level permission via a ServiceAccount to the PostgreSQL Operator.

There are a lot of deployment environments for the PostgreSQL Operator that only
need for it to exists within a single namespace and as such, granting
cluster-wide privileges is superfluous, and in many cases, undesirable. As such,
it should be possible to deploy the PostgreSQL Operator to a single namespace
without requiring a ClusterRole.

To do this, but maintain the aforementioned Namespace functionality for those
who require it, PostgreSQL Operator 4.3 introduces the ability to opt into
deploying it with minimum required ClusterRole privileges and in turn, the
ability to deploy the PostgreSQL Operator without a ClusterRole. To do so, the
PostgreSQL Operator introduces the concept of "namespace operating mode" which
lets one select the type deployment to create. The namespace mode is set at the
install time for the PostgreSQL Operator, and files into one of three options:

• dynamic: This is the default. This enables full dynamic Namespace
  management capabilities, in which the PostgreSQL Operator can create, delete and
  update any Namespaces within the Kubernetes cluster, while then also having the
  ability to create the Roles, Role Bindings and Service Accounts within those
  Namespaces for normal operations. The PostgreSQL Operator can also listen for
  Namespace events and create or remove controllers for various Namespaces as
  changes are made to Namespaces from Kubernetes and the PostgreSQL Operator's
  management.

• readonly: In this mode, the PostgreSQL Operator is able to listen for
  namespace events within the Kubernetetes cluster, and then manage controllers
  as Namespaces are added, updated or deleted. While this still requires a
  ClusterRole, the permissions mirror those of a "read-only" environment, and as
  such the PostgreSQL Operator is unable to create, delete or update Namespaces
  itself nor create RBAC that it requires in any of those Namespaces. Therefore,
  while in readonly, mode namespaces must be preconfigured with the proper RBAC
  as the PostgreSQL Operator cannot create the RBAC itself.

• disabled: Use this mode if you do not want to deploy the PostgreSQL Operator
  with any ClusterRole privileges, especially if you are only deploying the
  PostgreSQL Operator to a single namespace. This disables any Namespace
  management capabilities within the PostgreSQL Operator and will simply attempt
  to work with the target Namespaces specified during installation. If no target
  Namespaces are specified, then the Operator will be configured to work within
  the namespace in which it is deployed. As with the readonly mode, while in
  this mode, Namespaces must be pre-configured with the proper RBAC, since the
  PostgreSQL Operator cannot create the RBAC itself.

Based on the installer you use, the variables to set this mode are either named:

• PGO_NAMESPACE_MODE
• namespace_mode

pgo-installer Improvements

The new pgo-installer container can now be run directly from kubectl, i.e.

kubectl apply -f /path/to/install.yml

Other New Features

• pgo create cluster now supports the --pgbackrest-storage-config flag to specify specific a specific storage configuration for the pgBackRest repository in a cluster
• pgo create cluster now supports the --pgbackrest-cpu, --pgbackrest-memory, --pgbouncer-cpu, and --pgbouncer-memory resource request flags for the pgBackRest repository and the pgBouncer instances respectively.
• pgo create pgbouncer now supports the --cpu, and --memory resource request flags for requesting container resources for the pgBouncer instances
• pgo update cluster now supports the --pgbackrest-cpu, --pgbackrest-memory resource request flags for modifying container resources for the pgBackRest repository.
• pgo update pgbouncer now supports the --cpu, and --memory resource request flags for requesting container resources for the pgBouncer instances.
• Specify a different S3 Certificate Authority (CA) with pgo create cluster by using the --pgbackrest-s3-ca-secret flag, which refers to an existing Secret that contains a key called aws-s3-ca.crt that contains the CA. Reported by Aurelien Marie @(aurelienmarie)

Fixes

• Default the recovery action to "promote" when performing a "point-in-time-recovery" (PITR), which will ensure that a PITR process completes.
• Fix performance degradation discovered in certain environments that use the dynamic Namespace feature. Reported by (@dadez)
• Remove backoffLimit from Jobs that can be retried, which is most of them. Reported by Leo Khomenko (@lkhomenk)

Crunchy Data announces the release of the PostgreSQL Operator 4.3.0 Beta 1.

Details on the new major features, as well as other features and fixes, proceed
below. While we aim to have our betas feature complete, new features and
bug fixes will be introduced in subsequent beta release of PostgreSQL
Operator 4.3.

We also aim to have quarterly release. Due to some disruptions related to
ongoing world events, we are delaying the release but still aiming to make it
available by in mid-April. This will give us some time to get a few more
features in and tighten up existing functionality. We hope that the wait will be
worth it for you.

With that said, we did want to make the beta available sooner so you could begin
evaluating it for your purposes.

The PostgreSQL Operator 4.3.0 release includes the following software versions upgrades:

• The PostgreSQL containers now use versions 12.2, 11.7, 10.12, 9.6.17, and 9.5.21
  • This now includes support for using the JIT compilation feature introduced
    in PostgreSQL 11
• PostgreSQL containers now support PL/Python3
• pgBackRest is now at version 2.24
• Patroni is now at version 1.6.4
• postgres_exporter is now at version 0.7.0

The PostgreSQL Operator is released in conjunction with the Crunchy Container Suite.

PostgreSQL Operator is tested with Kubernetes 1.13 - 1.17, OpenShift 3.11+,
Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+.

Major Features

• Standby Clusters + Multi-Kubernetes Deployments
• Set custom PVC sizes for PostgreSQL clusters on creation and clone
• Tablespaces
• All Operator commands now support TLS-only PostgreSQL workflows

Standby Clusters + Multi-Kubernetes Deployments

A key component of building database architectures that can ensure continuity of
operations is to be able to have the database available across multiple data
centers. In Kubernetes, this would mean being able to have the PostgreSQL
Operator be able to have the PostgreSQL Operator run in multiple Kubernetes
clusters, have PostgreSQL clusters exist in these Kubernetes clusters, and only
ensure the "standby" deployment is promoted in the event of an outage or planned
switchover.

As of this release, the PostgreSQL Operator now supports standby PostgreSQL
clusters that can be deployed across namespaces or other Kubernetes or
Kubernetes-enabled clusters (e.g. OpenShift). This is accomplished by leveraging
the PostgreSQL Operator's support for
pgBackRest
and leveraging an intermediary, i.e. S3, to provide the ability for the standby
cluster to read in the PostgreSQL archives and replicate the data. This allows a
user to quickly promote a standby PostgreSQL cluster in the event that the
primary cluster suffers downtime (e.g. data center outage), for planned
switchovers such as Kubernetes cluster maintenance or moving a PostgreSQL
workload from one data center to another.

To support standby clusters, there are several new flags available on
pgo create cluster that are required to set up a new standby cluster. These
include:

• --standby: If set, creates the PostgreSQL cluster as a standby cluster
• --pgbackrest-repo-path: Allows the user to override the pgBackRest
  repository path for a cluster. While this setting can now be utilized when
  creating any cluster, it is typically required for the creation of standby
  clusters as the repository path will need to match that of the primary cluster.
• --password-superuser: When creating a standby cluster, allows the user to
  specify a password for the superuser that matches the superuser account in the
  cluster the standby is replicating from
• --password-replication: When creating a standby cluster, allows the user to
  specify a password for the replication user that matches the superuser account
  in the cluster the standby is replicating from

Note that the --password flag must be used to ensure the password of the main
PostgreSQL user account matches that of the primary PostgreSQL cluster, if you
are using Kubernetes to manage the user's password.

For example, if you have a cluster named hippo and wanted to create a standby
cluster called hippo and assuming the S3 credentials are using the defaults
provided to the PostgreSQL Operator, you could execute a command similar to:

pgo create cluster hippo-standby --standby \
  --pgbackrest-repo-path=/backrestrepo/hippo-backrest-shared-repo
  --password-superuser=superhippo
  --password-replication=replicahippo

To shutdown the primary cluster (if you can), you can execute a command similar
to:

pgo update cluster hippo --shutdown

To promote the standby cluster to be able to accept write traffic, you can
execute the following command:

pgo update cluster hippo-standby --promote-standby

To convert the old primary cluster into a standby cluster, you can execute the
following command:

pgo update cluster hippo --enable-standby

Once the old primary is converted to a standby cluster, you can bring it online
with the following command:

pgo update cluster hippo --startup

For information on the architecture and how to
set up a standby PostgreSQL cluster, please visit url

At present, streaming replication between the primary and standby clusters are
not supported, but the PostgreSQL instances within each cluster do support
streaming replication.

Customize PVC Size on PostgreSQL cluster Creation & Clone

The PostgreSQL Operator provides the ability to set customization for how large
the a PVC can be via the "storage config" options available in the PostgreSQL
Operator configuration file (aka pgo.yaml). While these provide a baseline
level of customizability, it is often important to be able to set the size
of the PVC that a PostgreSQL cluster should use at cluster creation time. In
other words, users should be able to choose exactly how large they want their
PostgreSQL PVCs to be.

PostgreSQL Operator 4.3 introduces the ability to set the PVC sizes for the
PostgreSQL cluster, the pgBackRest repository for the PostgreSQL cluster, and
the PVC size for each tablespace at cluster creation time. Additionally,
this behavior has been extended to the clone functionality as well, which is
helpful when trying to resize a PostgreSQL cluster. Here is some information on
the flags that have been added:

pgo create cluster

--pvc-size - sets the PVC size for the PostgreSQL data directory
--pgbackrest-pvc-size - sets the PVC size for the PostgreSQL pgBackRest
repository

For tablespaces, one can use the pvcsize option to set the PVC size for that
tablespace.

pgo clone cluster

--pvc-size - sets the PVC size for the PostgreSQL data directory for the newly
created cluster
--pgbackrest-pvc-size - sets the PVC size for the PostgreSQL pgBackRest
repository for the newly created cluster

Tablespaces

Tablespaces can be used to spread out PostgreSQL workloads across
multiple volumes, which can be used for a variety of use cases:

• Partitioning larger data sets
• Putting data onto archival systems
• Utilizing hardware (or a storage class) for a particular database
  object, e.g. an index

and more.

Tablespaces can be created via the pgo create cluster command using
the --tablespace flag. The arguments to --tablespace can be passed
in using one of several key/value pairs, including:

• name (required) - the name of the tablespace
• storageconfig (required) - the storage configuration to use for the tablespace
• pvcsize - if specified, the size of the PVC. Defaults to the PVC size in the storage configuration

Each value is separated by a :, for example:

pgo create cluster hacluster --tablespace=name=ts:storageconfig=nfsstorage

All tablespaces are mounted in the /tablespaces directory. The
PostgreSQL Operator manages the mount points and persistent volume
claims (PVCs) for the tablespaces, and ensures they are available
throughout all of the PostgreSQL lifecycle operations, including:

• Provisioning
• Backup & Restore
• High-Availability, Failover, Healing
• Clone

etc.

One additional value is added to the pgcluster CRD:

• TablespaceMounts: a map of the name of the tablespace and its
  associated storage.

Tablespaces are automatically created in the PostgreSQL cluster. You can
access them as soon as the cluster is initialized. For example, using
the tablespace created above, you could create a table on the tablespace
ts with the following SQL:

CREATE TABLE (id int) TABLESPACE ts;

Tablespaces can also be added to existing PostgreSQL clusters by using the
pgo update cluster command. The syntax is similar to that of creating a
PostgreSQL cluster with a tablespace, i.e.:

pgo update cluster hacluster --tablespace=name=ts2:storageconfig=nfsstorage

As additional volumes need to be mounted to the Deployments, this action can
cause downtime, though it expectation is that the downtime is brief.

Based on usage, future work will look to making this
more flexible. Dropping tablespaces can be tricky as no objects must
exist on a tablespace in order for PostgreSQL to drop it (i.e. there is
no DROP TABLESPACE .. CASCADE command).

Enhanced pgo df

pgo df provides information on the disk utilization of a PostgreSQL cluster,
and previously, this was not reporting accurate numbers. The new pgo df looks
at each PVC that is mounted to each PostgreSQL instance in a cluster, including
the PVCs for tablespaces, and computers the overall utilization. Even better,
the data is returned in a structured format for easy scraping. This
implementation also leverages Golang concurrency to help compute the results
quickly.

Enhanced pgBouncer Integration

The pgBouncer integration was completely rewritten to support the TLS-only
operations via the PostgreSQL Operator. While most of the work was internal,
you should now see a much more stable pgBouncer experience. Additionally, a few
new commands were added:

• pgo show pgbouncer shows information about a pgBouncer deployment
• pgo update pgbouncer --rotate-password allows one to rotate the service
  account password for pgBouncer

Rewritten pgo User Management commands

The user management commands were rewritten to support the TLS only workflow.
These commands now return additional information about a user when actions are
taken. Several new flags have been added too, including the option to view all
output in JSON. Other flags include:

• pgo update user --rotate-password to automatically rotate the password
• pgo update user --disable-login which disables the ability for a PostgreSQL user to login
• pgo update user --enable-login which enables the ability for a PostgreSQL user to login
• pgo update user --valid-always which sets a password to always be valid, i.e. it has no
  expiration
• pgo show user does not show system accounts by default now, but can be made
  to show the system accounts by using pgo show user --show-system-accounts

A major change as well is that the default password expiration function is now
defaulted to be unlimited (i.e. never expires) which aligns with typical
PostgreSQL workflows.

Breaking Changes

• pgo create cluster will now set the default database name to be the name of the cluster. For example, pgo create cluster hippo would create the initial database named hippo.
• The Database configuration parameter in pgo.yaml (db_name in the Ansible inventory) is now set to "" by default.
• the --password/-w flag for pgo create cluster now only sets the password for the regular user account that is created, not all of the system accounts (e.g. the postgres superuser).
• A default postgres-ha.yaml file is no longer is no longer created by the Operator for every PostgreSQL cluster.
• The pgbackups.crunchydata.com, deprecated since 4.2.0, has now been completely removed, along with any code that interfaced with it.
• Remove --series flag from pgo create cluster command. This affects API calls more than actual usage of the pgo client.
• pgo benchmark, pgo show benchmark, pgo delete benchmark are removed. PostgreSQL benchmarks with pgbench can still be executed using the crunchy-pgbench container.
• pgo ls is removed.
• The API that is used by pgo create cluster now returns its contents in JSON. The output now includes information about the user that is created.
• The API that is used by pgo show backup now returns its contents in JSON. The output view of pgo show backup remains the same.

Features

• Several additions to pgo create cluster around PostgreSQL users and databases, including:
  • --cpu flag that sets the amount of CPU to use for the PostgreSQL instances in the cluster
    ---database / -d flag that sets the name of the initial database created.
  • --memory flag that sets the amount of memory to use for the PostgreSQL instances in the cluster
  • --user / -u flag that sets the PostgreSQL username for the standard database user
  • --password-length sets the length of the password that should be generated, if --password is not set.
  • --show-system-accounts returns the credentials of the system accounts (e.g. the postgres superuser) along with the credentials for the standard database user
• Added the PodAntiAffinityPgBackRest and PodAntiAffinityPgBouncer to the pgo.yaml configuration file to set specific Pod anti-affinity rules for pgBackRest and pgBouncer Pods that are deployed along with PostgreSQL clusters that are managed by the Operator. The default for pgBackRest and pgBouncer is to use the value that is set in PodAntiAffinity.
• pgo create cluster now supports the --pod-anti-affinity-pgbackrest and --pod-anti-affinity-pgbouncer flags to specifically overwrite the pgBackRest repository and pgBouncer Pod anti-affinity rules on a specific PostgreSQL cluster deployment, which overrides any values present in PodAntiAffinityPgBackRest and PodAntiAffinityPgBouncer respectfully. The default for pgBackRest and pgBouncer is to use the value for pod anti-affinity that is used for the PostgreSQL instances in the cluster,
• pgo clone now supports the --enable-metrics flag, which will deploy the monitoring sidecar along with the newly cloned PostgreSQL cluster.
• The pgBackRest repository now uses ED25519 SSH key pairs.
• Add the --enable-autofail flag to pgo update to make it clear how the autofailover mechanism can be re-enabled for a PostgreSQL cluster.

Changes

• POSIX shared memory is now used for the PostgreSQL Deployments.
• The number of unsupported pgBackRest flags on the deny list has been reduced.
• The liveness and readiness probes for a PostgreSQL cluster now reference the /opt/cpm/bin/health
• wal_level is now defaulted to logical to enable logical replication
• archive_timeout is now a default setting in the crunchy-postgres-ha and crunchy-postgres-ha-gis containers and is set to 60
• ArchiveTimeout, LogStatement, LogMinDurationStatement are removed from pgo.yaml, as these can be customized either via a custom postgresql.conf file or postgres-ha.yaml file
• Quoted identifiers for the database name and user name in bootstrap scripts for the PostgreSQL containers
• The PostgreSQL Operator now logs its timestamps using RFC3339 formatting as implemented by Go
• SSH key pairs are no longer created as part of the Operator installation process. This was a legacy behavior that had not been removed
• The pv/create-pv-nfs.sh has been modified to create persistent volumes with their own directories on the NFS filesystems. This better mimics production environments. The older version of the script still exists as pv/create-pv-nfs-legacy.sh
• Load pgBackRest S3 credentials into environmental variables as Kubernetes Secrets, to avoid revealing their contents in Kubernetes commands or in logs
• Update how the pgBackRest and pgMonitor pamareters are loaded into Deployment templates to no longer use JSON fragments
• Remove using expenv in the add-targeted-namespace.sh script

Fixes

• Ensure PostgreSQL clusters can be successfully restored via pgo restore after 'pgo scaledown' is executed
• Allow the original primary to be removed with pgo scaledown after it is failed over
• Report errors in a SQL policy at the time pgo apply is executed, which was the previous behavior. Reported by José Joye (@jose-joye)
• pgo scaledown now works on a failed primary that has recovered and became a replica
• Ensure all replicas are listed out via the --query flag in pgo scaledown and pgo failover. This now follows the pattern outlined by the Kubernetes safe random string generator
• The pgo-rmdata Job will not fail if a PostgreSQL cluster has not been properly initialized
• Fixed a separate pgo-rmdata crash related to an improper SecurityContext
• Allow the standard PostgreSQL user created with the Operator to be able to create and manage objects within its own user schema. Reported by Nicolas HAHN (@hahnn)
• Honor the value of "PasswordLength" when it is set in the pgo.yaml file for password generation. The default is now set at 24
• Do not log pgBackRest environmental variables to the Kubernetes logs
• By default, exclude using the trusted OS certificate authority store for the Windows pgo client.
• Update the pgo-client imagePullPolicy to be IfNotPresent, which is the default for all of the managed containers across the project
• Set UsePAM yes in the sshd_config file to fix an issue with using SSHD in newer versions of Docker
• Only add Operator labels to a managed namespace if the namespace already exists when executing the add-targeted-namespace.sh script

• Ensure all available backup types are displayed in the pgo schedule are listed (full, incr, diff)
• Ensure schedule tasks create with pgo create schedule are deleted when pgo delete cluster is called
• Fix the missing readiness/liveness probes used to check the status of the apiserver and event containers in the postgres-operator pod
• Remove misleading error messages from the logs that were caused by the readiness/liveness probes on the apiserver and event containers in the postgres-operator pod
• Fix a race condition where the pgo-rmdata job could fail when doing its final pass on deleting PVCs. This went unnoticed as it was the final task to occur

Crunchy Data announces the release of the PostgreSQL Operator 4.2.2 on February, 18, 2020.

The PostgreSQL Operator 4.2.2 release provides bug fixes and continued support to the 4.2 release line.

This release includes updates for several packages supported by the PostgreSQL Operator, including:

• The PostgreSQL containers now use versions 12.2, 11.7, 10.12, 9.6.17, and 9.5.21
• The PostgreSQL containers now support PL/Python3
• Patroni is updated to version 1.6.4

The PostgreSQL Operator is released in conjunction with the Crunchy Container Suite.

PostgreSQL Operator is tested with Kubernetes 1.13+, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+.

Changes since 4.2.1

• Added the --enable-autofail flag to pgo update to make it clear how the auto-failover mechanism can be re-enabled for a PostgreSQL cluster.
• Remove using expenv in the add-targeted-namespace.sh script

Fixes since 4.2.1

• Ensure PostgreSQL clusters can be successfully restored via pgo restore after 'pgo scaledown' is executed
• Ensure all replicas are listed out via the --query flag in pgo scaledown and pgo failover. This now follows the pattern outlined by the Kubernetes safe random string generator (#1247)
• Honor the value of "PasswordLength" when it is set in the pgo.yaml file for password generation. The default is now set at 24
• Set UsePAM yes in the sshd_config file to fix an issue with using SSHD in newer versions of Docker
• The backup task listed in the pgtask CRD is now only deleted if one already exists
• Ensure that a successful "rmdata" Job does not delete all cluster pgtasks listed in the CRD after a successful run
• Only add Operator labels to a managed namespace if the namespace already exists when executing the add-targeted-namespace.sh script
• Remove logging of PostgreSQL user credentials in the PostgreSQL Operator logs
• Consolidation of the Dockerfiles for RHEL7/UBI7 builds
• Several fixes to the documentation (#1233)

Crunchy Data announces the release of the PostgreSQL Operator 4.2.1 on January, 16, 2020.

The PostgreSQL Operator 4.2.1 provides bug fixes and continued support to the 4.2 release line.

The PostgreSQL Operator is released in conjunction with the Crunchy Container Suite.

PostgreSQL Operator is tested with Kubernetes 1.13+, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+.

Fixes

• Ensure Pod labels are updated after failover (#1218)
• Fix for scheduled tasks to continue executing even after pgo delete schedule is called (#1215)
• Ensure pgo restore carries through the --node-label to the new primary (#1206)
• Fix for displaying incorrect replica names with the --query flag on pgo scaledown/pgo failover after a failover occurred
• Fix for default CA exclusion for the Windows-based pgo client
• Fix a race condition where the pgo-rmdata job could fail when doing its final pass on deleting PVCs. This went unnoticed as it was the final task to occur
• Fix image pull policy for the pgo-client container to match the project default (IfNotPresent)
• Update the "Create CRD Example" to reference the crunchy-postgres-ha container
• Update comments used for the API documentation generation via Swagger
• Update the directions for running the PostgreSQL Operator from the GCP Marketplace deployer
• Updated OLM installer example and added generation script
• Update the "cron" package to v3.0.1

Crunchy Data announces the release of the PostgreSQL Operator 4.2.0 on December, 31, 2019.

The focus of the 4.2.0 release of the PostgreSQL Operator was on the resiliency and uptime of the PostgreSQL clusters that the PostgreSQL Operator manages, with an emphasis on high-availability and removing the Operator from being a single-point-of-failure in the HA process. This release introduces support for a distributed-consensus based high-availability approach using Kubernetes distributed consensus store as the backing, which, in other words, allows for the PostgreSQL clusters to manage their own availability and not the PostgreSQL Operator. This is accomplished by leveraging the open source high-availability framework Patroni as well as the open source, high-performant PostgreSQL disaster recovery management tool pgBackRest.

To accomplish this, we have introduced a new container called crunchy-postgres-ha (and for geospatial workloads, crunchy-postgres-gis-ha). If you are upgrading from an older version of the PostgreSQL Operator, you will need to modify your installation to use these containers.

Included in the PostgreSQL Operator 4.2.0 introduces the following new features:

• An improved PostgreSQL HA (high-availability) solution using distributed consensus that is backed by Kubernetes. This includes:
  • Elimination of the PostgreSQL Operator as the arbiter that decides when a cluster should fail over
  • Support for Pod anti-affinity, which indicates to Kubernetes schedule pods (e.g. PostgreSQL instances) on separate nodes
  • Failed primaries now automatically heal, which significantly reduces the time in which they can rejoin the cluster.
  • Introduction of synchronous replication for workloads that are sensitive to transaction loss (with a tradeoff of performance and potentially availability)
• Standardization of physical backups and restores on pgBackRest, with native support for pg_basebackup removed.
• Introduction of the ability to clone PostgreSQL clusters using the pgo clone command. This feature copies the pgBackRest repository from a cluster and creates a new, single instance primary as its own cluster.
• Allow one to use their own certificate authority (CA) when interfacing with the Operator API, and to specify usage of the CA from the pgo command-line interface (CLI)

The container building process has been optimized, with build speed ups reported to be 70% faster.

The Postgres Operator 4.2.0 release also includes the following software versions upgrades:

• The PostgreSQL containers now use versions 12.1, 11.6, 10.11, 9.6.16, and 9.5.20.
• pgBackRest is upgraded to use version 2.20
• pgBouncer is upgraded to use version 1.12
• Patroni uses version 1.6.3

PostgreSQL Operator is tested with Kubernetes 1.13 - 1.15, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+. We have taken steps to ensure the PostgreSQL Operator is compatible with Kubernetes 1.16+, but did not test thoroughly on it for this release. Cursory testing indicates that the PostgreSQL Operator is compatible with Kubernetes 1.16 and beyond, but we advise that you run your own tests.

Major Features

High-Availability & Disaster Recovery

PostgreSQL Operator 4.2.0 makes significant enhancements to the high-availability and disaster recovery capabilities of the PostgreSQL Operator by moving to a distributed-consensus based model for maintaining availability, standardizing around pgBackRest for backups and restores, and removing the Operator itself as a single-point-of-failure in relation to PostgreSQL cluster resiliency.

As the high-availability environment introduced by PostgreSQL Operator 4.2.0 is now the default, setting up a HA cluster is as easy as:

pgo create cluster hacluster
pgo scale hacluster --replica-count=2

If you wish to disable high-availability for a cluster, you can use the following command:

pgo create cluster boringcluster --disable-autofail

New Required HA PostgreSQL Containers: crunchy-postgres-ha and crunchy-postgres-gis-ha

Using the PostgreSQL Operator 4.2.0 requires replacing your crunchy-postgres and crunchy-postgres-gis containers with the crunchy-postgres-ha and crunchy-postgres-gis-ha containres respectively. The underlying PostgreSQL installations in the container remain the same but are now optimized for Kubernetes environments to provide the new high-availability functionality.

A major change to this container is that the PostgreSQL process is now managed by Patroni. This allows a PostgreSQL cluster that is deployed by the PostgreSQL Operator to manage its own uptime and availability, to elect a new leader in the event of a downtime scenario, and to automatically heal after a failover event.

Upgrading to these new containers is as simple as modifying your CRD ccpimage parameter to use crunchy-postgres-ha to use the HA enabled containers. Please see our upgrade instructions to select your preferred upgrade strategy.

pgBackRest Standardization

pgBackRest is now the only backup and restore method supported by the PostgreSQL Operator. This has allowed for the following features:

• Faster creation of new replicas when a scale up request is made
• Automatic healing of PostgreSQL instances after a failover event, leveraging the pgBackRest delta restore feature. This allows for a significantly shorter healing process
• The ability to clone PostgreSQL clusters

As part of this standardization, one change to note is that after a PostgreSQL cluster is created, the PostgreSQL Operator will schedule a full backup of the cluster. This is to ensure that a new replica can be created from a pgBackRest backup. If this initial backup fails, no new replicas will be provisioned.

When upgrading from an earlier version, please ensure that you have at least one pgBackRest full backup in your backup repository.

Pod Anti-Affinity

PostgreSQL Operator 4.2.0 adds support for Kubernetes pod anti-affinity, which provides guidance on how Kubernetes should schedule pods relative to each other. This is helpful in high-availability architectures to ensure that PostgreSQL pods are spread out in order to withstand node failures. For example, in a setup with two PostgreSQL instances, you would not want both instances scheduled to the same node: if that node goes down or becomes unreachable, then your cluster will be unavailable!

The way the PostgreSQL Operator uses pod anti-affinity is that it tries to ensure that none of the managed pods within the same cluster are scheduled to the same node. These include:

• Any PostgreSQL instances
• The pod that manages pgBackRest repository
• If deployed, any pgBouncer pods

This helps improve the likelihood that a cluster can remain up even if a downtime event occurs.

There are three options available for pod anti-affinity:

• preferred: Kubernetes will try to schedule any pods within a PostgreSQL cluster to different nodes, but in the event it must schedule two pods on the same node, it will. This is the default option
• required: Kubernetes will schedule pods within a PostgreSQL cluster to different nodes, but in the event it cannot schedule a pod to a different node, it will not schedule the pod until a different node is available. While this guarantees that no pod will share the same node, it can also lead to downtime events as well.
• disabled: Pod anti-affinity is not used.

These options can be combined with the existing node affinity functionality (--node-label) to group the scheduling of pods to particular node labels!

Synchronous Replication

PostgreSQL Operator 4.2 introduces support for synchronous replication by leveraging the "synchronous mode" functionality provided by Patroni. Synchronous replication is useful for workloads that are sensitive to losing transactions, as PostgreSQL will not consider a transaction to be committed until it is committed to all synchronous replicas connected to a primary. This provides a higher guarantee of data consistency and, when a healthy synchronous replica is present, a guarantee of the most up-to-date data during a failover event.

This comes at a cost of performance as PostgreSQL: as PostgreSQL has to wait for a transaction to be committed on all synchronous replicas, a connected client will have to wait longer than if the transaction only had to be committed on the primary (which is how asynchronous replication works). Additionally, there is a potential impact to availability: if a synchronous replica crashes, any writes to the primary will be blocked until a replica is promoted to become a new synchronous replica of the primary.

You can enable synchronous replication by using the --sync-replication flag with the pgo create command.

Updated pgo CLI Flags

• pgo create now has a CLI flag for pod anti-affinity called --pod-anti-affinity, which accepts the values required, preferred, and disabled
• pgo create --sync-replication specifies to create a PostgreSQL HA cluster with synchronous replication

Global Configuration

To support high-availability there are some new settings that you can manage from your pgo.yaml file:

• DisableAutofail - when set to true, this will disable the new HA functionality in any newly created PostgreSQL clusters. By default, this is false.
• DisableReplicaStartFailReinit - when set to true, this will disable attempting to re-provision a PostgreSQL replica when it is stuck in a "start failed" state. By default, this false.
• PodAntiAffinity - Determines the type of pod anti-affinity rules to apply to the pods within a newly PostgreSQL cluster. If set to required, pods within a PostgreSQL cluster must be scheduled on different nodes, otherwise a pod will fail to schedule. If set to preferred, Kubernetes will make a best effort to schedule pods of the same PostgreSQL cluster on different nodes. If set to disabled, this feature is disabled. By default, this is preferred.
• SyncReplication - If set to true, enables synchronous replication in newly created PostgreSQL clusters. Default to false.

pgo clone

PostgreSQL Operator 4.2.0 introduces the ability to clone the data from one PostgreSQL cluster into a brand new PostgreSQL cluster. The command to do so is simple:

pgo clone oldcluster newcluster

After the command is executed, the PostgreSQL Operator checks to see if a) the oldcluster exists and b) the newcluster does not exist. If both of these conditions hold, the PostgreSQL Operator creates two new PVCs the match the specs of the oldcluster PostgreSQL data PVC (PrimaryStorage) and its pgBackRest repository PVC (BackrestStorage).

If these PVCs are successfully created, the PostgreSQL Operator will copy the contents of the pgBackRest repository from the oldcluster to the one setup for the newcluster by means of a Kubernetes Job that is running rsync provided by the pgo-backrest-repo-sync container. We are able to do this because all changes to the pgBackRest repository are atomic.

If this successfully completes, the PostgreSQL Operator then runs a pgBackRest restore job to restore the PostgreSQL cluster. On a successful restore, the new PostgreSQL cluster is then scheduled and runs in recovery mode until it reaches a consistent state, and then comes online as a brand new cluster

To optimize the time it takes to restore for a clone, we recommend taking a backup of the cluster you want to clone. You can do this with the pgo backup command, and choose if you want to take a full, differential, or incremental backup.

Future work will be focused on additional options, such as being able to clone a PostgreSQL cluster to a particular point-in-time (so long as the backup is available to support it) and supporting other pgo create flags.

Schedule Backups With Retention Policies

While the PostgreSQL Operator has had the ability to schedule full, incremental, and differential pgBackRest backups for awhile, it has not been possible to set the retention policy on these backups. Backup retention policies allow users to manage their backup storage whle maintaining enough backups to be able to recover to a specific point-in-time, or perform forensic work on data in a particular state.

For example, one can schedule a full backup to occur nightly at midnight and keep up to 21 full backups (e.g. a 21 day retention policy):

pgo create schedule mycluster --schedule="0 0 * * *" --schedule-type="pgbackrest" --pgbackrest-backup-type=full --schedule-opts="--repo1-retention-full=21"

Breaking Changes

Feature Removals

• Physical backups using pg_basebackup are no longer supported. Any command-line option that references using this method has been removed. The API endpoints where one can specify a pg_basebackup remain, but will be removed in a future release (likely the next one).
• Removed the pgo-lspvc container. This container was used with the pgo show pvc and performed searches on the mounted filesystem. This would cause issues both on environments that could not support a PVC being mounted multiple times, and for underlying volumes that contained numerous files. Now, pgo show pvc only lists the PVCs associated with the PostgreSQL clusters managed by the PostgreSQL Operator.
• Native support for pgpool has been removed.

Command Line (pgo)

pgo create cluster

• The --pgbackrest option is removed as it is no longer needed. pgBackRest is enabled by default

pgo delete cluster

The default behavior for pgo delete cluster has changed so that all backups and PostgreSQL data are deleted by default.

To keep a backup after a cluster is deleted, one can use the --keep-backups flag with pgo delete cluster, and to keep the PostgreSQL data directory, one can specify the --keep-data flag. There is a plan to remove the --keep-data flag in a future release, though this has not been determined yet.

The -b, --delete-backups, -d, and --delete-data flags are all deprecated and will be removed in the next release.

pgo scaledown

With this release, pgo scaledown will delete the PostgreSQL data directory of the replica by default. To keep the PostgreSQL directory after the replica has scaled down, one can use the --keep-data flag.

pgo test

pgo test is optimized to provide faster results about the availability of a PostgreSQL cluster. Instead of attempting to make PostgreSQL connections to each PostgreSQL instance with each user, pgo test now checks the availability of the service endpoints for each PostgreSQL cluster as well as the output of the PostgreSQL readiness checks, which check the connectivity of a PostgreSQL cluster.

Both the API and the output of pgo test are modified for this optimization.

Additional apiserver Changes

• An authorization failure in the apiserver (i.e. not having the correct RBAC permission for a pgouser) will return a status code of 403 instead of 401
• The pgorole permissions now support the "*" permission to specify all pgorole RBAC permissions are granted to a pgouser. Users upgrading from an earlier version should note this change if they want to assign their users to access new features.

Additional Features

pgo (Operator CLI)

• Support the pgBackRest options for backup retention, including --repo1-retention-full, --repo1-retention-diff, --repo1-retention-archive, --repo1-retention-archive-type, which can be added in the --backup-opts flag in the pgo backup command. For example:

# create a pgBackRest incremental backup with one full backup being retained and two differential backups being retained, along with incremental backups associated with each
pgo backup mycluster --backup-type="pgbackrest" --backup-opts="--type=incr --repo1-retention-diff=2 --repo1-retention-full=1"

# create a pgBackRest full backup where 2 other full backups are retained, with WAL archive retained for full and differential backups
pgo backup mycluster --backup-opts="--type=full --repo1-retention-full=2 --repo1-retention-archive=4 --repo1-retention-archive-type=diff"

• Allow for users to define S3 credentials and other options for pgBackRest backups on a per-cluster basis, in addition to leveraging the globally provided templates. This introduces the following flags on the
  pgo create cluster command:

  • --pgbackrest-s3-bucket - specifics the AWS S3 bucket that should be utilized
  • --pgbackrest-s3-endpoint specifies the S3 endpoint that should be utilized
  • --pgbackrest-s3-key - specifies the AWS S3 key that should be utilized
  • --pgbackrest-s3-key-secret- specifies the AWS S3 key secret that should be utilized
  • --pgbackrest-s3-region - specifies the AWS S3 region that should be utilized

• Add the --disable-tls flag to the pgo command-line client, as to be compatible with the Operator API server that is deployed with DISABLE_TLS enabled.

• Improved output for the pgo scaledown --query and and pgo failover --query commands, including providing easy-to-understand results on replication lag

• Containerized pgo via the pgo-client container. This can be installed from the Ansible installer using the pgo_client_container_install flag, and it installs into the same namespace as the PostgreSQL Operator. You can connect to the container via kubectl exec and execute pgo commands!

Builds

• Refactor the Dockerfiles to rely on a "base" definition for ease of management and to ensure consistent builds across all container images during a full make
• Selecting which image builder to use is now argument based using the IMGBUILDER environmental variable. Default is buildah
• Optimize yum clean invocation to occur on same line as the RUN, which leads to smaller image builds.

Installation

• Add the pgo_noauth_routes (Ansible) / NOAUTH_ROUTES (Bash) configuration variables to disable TLS/BasicAuth authentication on particular API routes. This defaults to '/health'
• Add the pgo_tls_ca_store Ansible / TLS_CA_TRUST (Bash) configuration variables to specify a PEM encoded list of trusted certificate authorities (CA) for the Operator to use when authenticating API requests over TLS
• Add the pgo_add_os_ca_store / ADD_OS_TRUSTSTORE (Bash) to specify to use the trusted CA that is provided by the operating system. Defaults to true

Configuration

• Enable individual ConfigMap files to be customized without having to upload every single ConfigMap file available in pgo-config. Patch by Conor Quin (@Conor-Quin)
• Add EXCLUDE_OS_TRUST environmental variable that allows the pgo client to specify that it does not want to use the trusted certificate authorities (CA) provided by the operating system.

Miscellaneous

• Migrated Kubernetes API groups using API version extensions/v1beta1 to their respective updated API groups/versions. This improves compatibility with Kubernetes 1.16 and beyond. Original patch by Lucas Bickel (@hairmare)
• Add a Kubernetes Service Account to every Pod that is managed by the PostgreSQL Operator
• Add support for private repositories using imagePullSecret. This can be configured during installation by setting the pgo_image_pull_secret and pgo_image_pull_secret_manifest in the inventory file using Ansible installer, or with the PGO_IMAGE_PULL_SECRET and PGO_IMAGE_PULL_SECRET_MANIFEST environmental variables using the Bash installer. The "pull secret" is the name of the pull secret, whereas the manifest is what is used to define the secret
• The pgorole permissions now support the "*" permission to specify all pgorole RBAC permissions are granted to a pgouser
• Policies that are executed using pgo apply and pgo create cluster --policies are now executed over a UNIX socket directly on the Pod of the primary PostgreSQL instance. Reported by @yuanlinios
• A new sidecar, crunchyadm, is available for running managemnet commands on a PostgreSQL cluster. As this is experimental, this feature is disabled by default.

Fixes

• Update the YAML library to v2.2.4 to mitigate issues presented in CVE-2019-11253
• Specify the pgbackrest user by its ID number (2000) in the backrest-repo container image so that containers instantiated with the runAsNonRoot option enabled recognize the pgbackrest user as non-root.
• Ensure any Kubernetes Secret associated with a PostgreSQL backup is deleted when the --delete-backups flag is specified on pgo delete cluster
• The pgBouncer pod can now support connecting to databases that are added after a PostgreSQL cluster is deployed
• Remove misleading error messages from the logs that were caused by the readiness/liveness probes on the apiserver and event containers in the postgres-operator pod
• Several fixes to the cleanup of a PostgreSQL cluster after a deletion event (e.g. pgo delete cluster) to ensure data is safely removed. This includes ensuring schedules managed by pgo schedule are removed, as well as PostgreSQL cluster and backup data
• Skip the HTTP Basic Authorization check if the BasicAuth parameter in pgo.yaml is set to false
• Ensure all available backup types are displayed in the pgo schedule are listed (full, incr, diff)
• Ensure schedule tasks create with pgo create schedule are deleted when pgo delete cluster is called
• Fix the missing readiness/liveness probes used to check the status of the apiserver and event containers in the postgres-operator pod
• Remove misleading error messages from the logs that were caused by the readiness/liveness probes on the apiserver and event containers in the postgres-operator pod
• Fix a race condition where the pgo-rmdata job could fail when doing its final pass on deleting PVCs. This became noticeable after adding in the task to clean up any configmaps that a PostgreSQL cluster relied on
• Improved logging around authorization failures in the apiserver

Crunchy Data announces the release of Postgres Operator 4.1.1 on November, 22, 2019.

Postgres Operator 4.1.1 provide bug fixes and continued support to Postgres Operator 4.1 as well as continued compatibility with newer versions of PostgreSQL.

The PostgreSQL Operator is released in conjunction with the Crunchy Container Suite.

The Postgres Operator 4.1.1 release includes the following software versions upgrades:

• The PostgreSQL now uses versions 12.1, 11.6, 10.11, 9.6.16, and 9.5.20.

Postgres Operator is tested with Kubernetes 1.13 - 1.15, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+. At present, Postgres Operator 4.1 is not compatible with Kubernetes 1.16.

Fixes

• Add the --disable-tls flag to the pgo command-line client, as to be compatible with the Operator API server that is deployed with DISABLE_TLS enabled. This is backported due to this functionality being missed in the 4.1 release.
• Update the YAML library to v2.2.4 to mitigate issues presented in CVE-2019-11253
• Specify the pgbackrest user by its ID number (2000) in the backrest-repo container image so that containers instantiated with the runAsNonRoot option enabled recognize the pgbackrest user as non-root.
• Ensure any Kubernetes Secret associated with a PostgreSQL backup is deleted when the --delete-backups flag is specified on pgo delete cluster
• Enable individual ConfigMap files to be customized without having to upload every single ConfigMap file available in pgo-config. Patch by Conor Quin (@Conor-Quin)
• Skip the HTTP Basic Authorization check if the BasicAuth parameter in pgo.yaml is set to false

Crunchy Data announces the preview release of the PostgreSQL Operator 4.2 by making available a 4.2.0-alpha.1. Please see our contribution guidelines on how the use and limitations of an alpha release.

The principal work in PostgreSQL Operator 4.2 focuses on the provisioning, high-availability, and disaster recovery systems employed by the Operator. Please note the following things in this beta:

• PostgreSQL Operator 4.2 now only uses the crunchy-postgres-ha and crunchy-postgres-gis-ha containers.
• High-availability is now facilitated via the Patroni framework and provides a distributed consensus solution that removes the PostgreSQL Operator as a single-point-of-failure
• Backups are now standardized on the pgBackRest backup and disaster recovery management utility

We encourage you to test and report your feedback.

Additional Features

pgo (Operator CLI)

• Add the --disable-tls flag to the pgo command-line client, as to be compatible with the Operator API server that is deployed with DISABLE_TLS enabled.

Builds

• Refactor the Dockerfiles to rely on a "base" definition for ease of management and to ensure consistent builds across all container images during a full make
• Selecting which image builder to use is now argument based using the IMGBUILDER environmental variable. Default is buildah
• Optimize yum clean invocation to occur on same line as the RUN, which leads to smaller image builds.

Configuration

• Enable individual ConfigMap files to be customized without having to upload every single ConfigMap file available in pgo-config. Patch by Conor Quin (@Conor-Quin)

Fixes

• Update the YAML library to v2.2.4 to mitigate issues presented in CVE-2019-11253
• Specify the pgbackrest user by its ID number (2000) in the backrest-repo container image so that containers instantiated with the runAsNonRoot option enabled recognize the pgbackrest user as non-root.
• Ensure any Kubernetes Secret associated with a PostgreSQL backup is deleted when the --delete-backups flag is specified on pgo delete cluster

Crunchy Data announces the release of Postgres Operator 4.1 on October 15, 2019.

In addition to new features, such as dynamic namespace manage by the Operator and the ability to subscribe to a stream of lifecycle events that occur with PostgreSQL clusters, this release adds many new features and bug fixes.

The Postgres Operator 4.1 release also includes the following software versions upgrades:

• The PostgreSQL now uses versions 11.5, 10.10, 9.6.15, and 9.5.19. The PostgreSQL container now includes support for PL/Python.
• pgBackRest is now 2.17
• pgMonitor now uses version 3.2

To build Postgres Operator 4.1, you will need to utilize buildah version 1.9.0 and above.

Postgres Operator is tested with Kubernetes 1.13 - 1.15, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+. At present, Postgres Operator 4.1 is not compatible with Kubernetes 1.16.

Major Features

Dynamic Namespace Management

Postgres Operator 4.1 introduces the ability to dynamically management Kubernetes namespaces from the Postgres Operator itself. Kubernetes namespaces provide the ability to isolate workloads within a Kubernetes cluster, which can provide additional security and privacy amongst users.

The previous version of the Postgres Operator allowed users to add Kubernetes namespaces to which the Postgres Operator could deploy and manage clusters. Postgres Operator 4.1 expands on this ability by allowing the Operator to dynamically add and remove namespaces using the pgo create namespace and pgo delete namespace commands.

This allows for several different deployment patterns for PostgreSQL clusters, including:

• Deploying PostgreSQL clusters within a single namespace for an individual user
• Deploying a PostgreSQL cluster into its own namespace

Note that deleting a namespace in Kubernetes deletes all of the objects that reside within that namespace, including active PostgreSQL clusters. Ensure that you wish to delete everything inside a namespace before executing pgo delete namespace.

This has also lead to a change in terms of how role-based access control (RBAC) is handled. Traditionally, RBAC permissions we added to the ClusterRole objects, but in order to support dynamic namespace management, the RBAC has been moved to the Role objects.

If you would like to use the dynamic namespace feature Kubernetes 1.11 and OpenShift 3.11, you will also need to utilize the add-targeted-namespace.sh script that is bundled with Postgres Operator 4.1. To add a namespace to dynamically to your Postgres Operator deployment in Kubernetes 1.11, you first need to create the namespace with kubectl (e.g. kubectl create namespace yournamespace) and then run the add-targeted-namespace.sh script (./add-targeted-namespace.sh yournamespace).

Lifecycle Events

Postgres Operator 4.1 now provides PostgreSQL lifecycle events that occur during the operation of a cluster. Lifecycle events include things such as when a cluster is provisioned, a replica is added, a backup is taken, a cluster fails over, etc. Each deployed PostgreSQL cluster managed by the PostgreSQL Operator will report back to the Operator around these lifecycle events via the NSQ distributed messaging platform.

You can subscribe to lifecycle events by topic using the pgo watch command. For subscribe to all events for clusters under management, you can run pgo watch alltopic. Eventing can be disabled using the DISABLE_EVENTING environmental variable within the postgres-operator deployment.

For more information, please read the Eventing section of the documentation.

Breaking Changes

Containers

• The node_exporter container is no longer shipped with the PostgreSQL Operator. A detailed explanation of how node-style metrics are handled is available in the "Additional Features" section.

API

• The pgo update cluster API endpoint now uses a HTTP POST instead of GET
• The user management endpoints (e.g. pgo create user) now use a HTTP POST instead of a GET.

Command-line interface

• Removed the -db flag from pgo create user and pgo update user
• Removed --update-password flag from the pgo user command

Installation

• Changed the Ansible installer to use uninstall and uninstall-metrics tags instead of deprovision and deprovision-metrics respectively

Builds

• The Makefile now uses buildah for building the containers instead of Docker. The PostgreSQL Operator can be built with buildah v1.9.0 and above.

Additional Features

General PostgreSQL Operator Features

• PostgreSQL Operator users and roles can now be dynamically managed (i.e. pgouser and pgorole)

• Readiness probes have been added to the apiserver and scheduler and is now included in the new event container. The scheduler uses a special heartbeat task to provide its readiness.

• Added the DISABLE_TLS environmental variable for apiserver, which allows the API server to run over HTTP.

• Added the NOAUTH_ROUTES environmental variable for apiserver, which allows useres to bypass TLS authentication on certain routes (e.g. /health). At present, only /health can be used in this variable.

• Services ports for the postgres_exporter and pgBadger are now templated so a user can now customize them beyond the defaults.

PostgreSQL Upgrade Management

• The process to perform a minor upgrade on a PostgreSQL deployment was modified in order to minimize downtime. Now, when a pgo upgrade cluster command is run, the PostgreSQL Operator will upgrade all the replicas to the desired version of PostgreSQL before upgrading the primary container. If autofail is enabled, the PostgreSQL Operator will failover to a pod that is already updated to a newer version, which minimizes downtime and allows the cluster to upgrade to the desired, updated version of PostgreSQL.

• pgo upgrade now supports the --all flag, which will upgrade every single PostgreSQL cluster managed by the PostgreSQL Operator (i.e. pgo upgrade --all)

PostgreSQL User Management

• All user passwords are now loaded in from Kubernetes Secrets.
• pgo create user --managed now supports any acceptable password for PostgreSQL
• Improved error message for calling the equivalent pgo show user command when interfacing with the API directly and there are no clusters found for th euser.

Monitoring

• Updated the Grafana dashboards to use those found in pgMonitor v3.2
• The crunchy-collect container now connects to PostgreSQL using a password that is stored in a Kubernetes secret
• Introduced support for collecting host-style metrics via the cAdvisor installations that are installed and running on each Kubelet. This requires for the ClusterRole to have the nodes and nodes/metrics resources granted to it.

Logging

• Updated logging to provide additional details of where issues occurred, including file and line number where the issue originated.

Installation

• The Ansible installer uninstall tag now has the option of preserving portions of the previous installation
• The Ansible installer supports NFS and hostpath storage options
• The Ansible installer can now set the fsgroup for the metrics tag
• The Ansible installer now has the same configuration options as the bash installer
• The Ansible installer now supports a separate RBAC installation
• Add a custom security context constraint (SCC) to the Ansible and bash installers that is applied to pods created by the Operator. This makes it possible to customize the control permissions for the PostgreSQL cluster pods managed by the Operator

Fixes

• Fixed a bug where testuser was always created even if the username was modified in the pgo.yaml
• Fixed the --expired flag for pgo show user to show the number of days until a user's password expires
• Fixed the workflow for pgo benchmark jobs to show completion
• Modify the create a cluster via a custom resource definition (CRD) to use pgBackRest
• Fixed an issue with the pgpool label when a pg_dump is performed by calling the REST API
• Fixed the pgo load example, which previous used a hardcoded namespace. This has changed with the support of dynamic namespaces.

Crunchy Data announces the release of Postgres Operator 3.5.5 on October 15, 2019.

The release of Postgres Operator 3.5.5 is a maintenance release that provides fixes to Operator functionality as well as updates to the supported versions of PostgreSQL, including:

• The PostgreSQL containers now use versions 11.5, 10.10, 9.6.15, and 9.5.19. The PostgreSQL container now includes support for PL/Python.
• pgBackRest is now 2.17

Postgres Operator is tested with Kubernetes 1.13 - 1.15, OpenShift 3.11+, Google Kubernetes Engine (GKE), and VMware Enterprise PKS 1.3+. At present, Postgres Operator 3.5 is not compatible with Kubernetes 1.16.

Fixes

• Support for request queueing for the Pgcluster and Pgtask CRD controllers
• Repair Helm Chart for Postgres Operator 3.5 (thanks to PowerStateFailure).

This is a minor release for the Postgres Operator and includes the following:

• Fixed an issue with backrest stanza not creating occasionally.
• #804 Fixed an issue with pgbadger target

This is a minor release for the Postgres Operator and includes the following:

• This release updates the version of PostgreSQL.
• Added validation error if --autofail flag is set and user attempts to perform a manual failover.
• Fixed an issue with --delete-data hanging when a cluster was created with --pgool.
• Fixed an issue with pgo.yaml pgbackrest default not being used when creating a cluster.
• Shorten -backrest-shared-repo pvc name to -pgbr-repo as in some cases generated name was over the size limit and causes issues.

Crunchy Data is pleased to release PostgreSQL Operator 4.0.0

Crunchy PostgreSQL Operator extends Kubernetes to support the creation, configuration and management of PostgreSQL clusters at scale. When combined with the Crunchy PostgreSQL Container Suite, the Crunchy PostgreSQL Operator provides an open source software solution for PostgreSQL scaling, high-availability, disaster recovery, monitoring, and more.

The Crunchy PostgreSQL Operator 4.0.0 supports the Crunchy PostgreSQL Container Suite version 2.4.0+. Prior versions of the Crunchy PostgreSQL Container Suite are not supported by the Crunchy PostgreSQL Operator 4.0.0.

Key user facing features added to the Crunchy PostgreSQL Operator 4.0.0 include:

-- Support for deploying the Crunchy PostgreSQL Operator into it's own namespace,

-- Support for deploying PostgreSQL Clusters into multiple namespaces being watched by a single Crunchy PostgreSQL Operator

-- Support for the ability to perform pgbackrest backups to Amazon S3

-- Inclusion of Ansible installation playbooks for eased installation

-- Crunchy PostgreSQL Operator 4.0.0 supports the Operator Lifecycle Management (OLM) and will be made available the operatorhub.io catalog shortly.

-- Updates and enhancements to the Crunchy PostgreSQL Operator documentation

This release incorporates significant changes relative to the PostgreSQL Operator 3.5.x.

In particular, the changes to how namespaces are supported are material. For additional information, please see the Installation and Design documentation for details on how namespaces are supported and configured:

https://access.crunchydata.com/documentation/postgres-operator/4.0.0/gettingstarted/design/designoverview/

Instructions for upgrading from PostgreSQL Operator version 3.5.x to 4.0.0 can be found in the documentation here:

https://access.crunchydata.com/documentation/postgres-operator/4.0.0/upgrade/

The Crunchy PostgreSQL Operator 4.0.0 does not yet provide a helm chart based installation. Crunchy Data is evaluating inclusion of a helm chart for subsequent releases and welcomes feedback on the desirability of helm chart based installations.

---

Additional changes and features incorporated within Crunchy PostgreSQL Operator 4.0.0 include:

-- Revised RBAC rules to better support OLM

-- Revised default configuration to better support OLM

-- Configuration files under conf/postgres-operator/cluster/1 were moved to conf/postgres-operator

-- Namespaces were added to server side pgouser file to indicate to which namespaces a pgo user has access

-- A default deployment/install of the Operator was changed to better reflect how multiple namespaces are supported, see the Installation documentation for details

-- Removed the -n shorthand flag for the --no-prompt flag, -n is now the shorthand for the --namespace flag to better match kubectl flags

-- pgo.yaml - removed Pgo.LSPVCTemplate and Pgo.LoadTemplate settings, these have moved to the normal configuration location

-- Removed the volume pgo-config references within the deployment.json, this means that the various Operator containers do not mount the pgo-config ConfigMap directly any longer, instead it is looked for upon initialization internally, if found, it is used for Operator configuration as normal, if not found, the default baked-in version of configuration is used, this supports OLM and an easier more automatic means of deploying the Operator where a user just wants to test it out before doing significant configuration customization (e.g. OLM/Openshift console deployment)

-- pgo-apiserver TLS keys now detected at runtime and not mounted, if a pgo.tls Secret is not found, a self-signed TLS credential will be generated and the Secret created by the Operator

-- CRD group names has changed from client-go.k8s.io to crunchydata.com to be consistent with OLM standards and better identify PostgreSQL Operator resources

-- Environment variables were renamed to standardize using PGO as the prefix instead of some which were CO

-- Updated Deployment version from v1beta1 to v1

-- pgo show namespace command which shows the namespaces currently being watched by the operator and which ones are accessible from your pgo user account

-- Namespace command line flag for pgo commands to specify which namespace to inspect or create resources into, this flag value overrides the PGO_NAMESPACE environment variable if set by the pgo CLI user

-- pgo benchmark command lets you create and view pgbench reports default 'baked-in' configuration, see Design documentation for details

-- Default installer supports installing Operator multiple times on a single Kubernetes cluster

-- New pgo ls command, lets you list files inside the postgres container

-- pgbasebackup backup paths are now maintained in the pgbackup CRD for a given PG cluster, this allows users to view historical backup-paths using the pgo show backup command.

-- pgbasebackup was refactored to be more in line with pgbackrest and pgdump commands.

-- Additional validation added to backup-opts to ensure the options are valid with the type of backup being performed.

-- Added liveness and readiness probes to the Operator deployment example

-- Changed pgo CLI to look at PGOUSER environment variable first in order of searching for the credentials file

-- Removed the primary=true/false label as its redundant

-- Removed the pgpool and pgbouncer labels from the Postgres Pods, stored in the pgcluster CRD instead.

-- quickstart.sh script was removed

-- Removed the pgupgrade CRD, pgo show upgrade and pgo delete upgrade commands.

-- PGOUSERNAME and PGOUSERPASS environment variables support added for pgo CLI for use cases where storing username and password in a file is not possible or preferable

-- Changes to REST API including:

-- REST API change: /clusters/{name} GET changes to /showclusters POST

-- REST API change: /policies/{name} GET changes to /showpolices POST

-- REST API change: /pvc/{name} GET changes to /showpvc POST

-- REST API change: /deleteclusters/{name} GET changes to /deleteclusters POST

-- REST API change: /deletepolicies/{name} GET changes to /deletepolicies POST

-- REST API change: /clusters/test/{name} GET changes to /testclusters POST

-- Changes to pgo syntax including:

-- pgo show cluster all command syntax is changed to pgo show cluster --all

-- pgo show policy all command syntax is changed to pgo show policy --all

-- pgo show pvc all command syntax is changed to pgo show pvc --all

-- pgo delete cluster all command syntax is changed to pgo delete cluser --all

-- pgo delete policy all command syntax is changed to pgo delete policy --all

-- pgo test all command syntax is changed to pgo test --all

This release provides the following updates:

PostgreSQL minor version updates
pgBackrest version updated from 2.10 to 2.13
Bug fix for replicas not connecting back to the Primary after fail-over if both Primaries get started at the same time.