spark-operator

This release contains a number of fixes and enhancements. Details about the changes since the last release can be found in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/compare/v1beta2-1.2.1-3.0.0...v1beta2-1.2.2-3.0.0.

This release includes the following notable enhancements and bug fixes:

• Added support for custom request resources in the Volcano scheduler integration: #966.
• Extended ingress URL format for the Spark driver UI: #1008.
• Added support for proxy users: #1022.
• Added support for container-level security context: #1023.
• Added support for spark-submit packages and repositories: #1040.

This is the first release based on the new Spark 3.0.0 release. Changes in this release include support for some new configuration options and enhancements introduced in Spark 3.0.0 that are relevant to the operator. Major commits in this release include #954 and #976. The minor version is bumped to reflect that it is based on Spark 3.0.0.

This is a bug fix release based on Spark 2.4.5. Notable fixes/enhancements include:

• #935: added the container port for exposing Spark metrics in Prometheus data format.
• #914: support customization of Spark driver UI port and ingress configuration.
• #898: fixed how default values of certain fields get applied.
• #876: fix for #868.
• #867: fix for termination time when using sidecars.
• #852: added a new metric on job start latency.
• #847: fix around use of metricsProperties and hasPrometheusConfigFile.
• Other fixes and refactoring.

This is a bug fix release, which includes a fix in #832 that caused issues reported in #795 and #826.

This release is based on Spark 2.4.5 and API version v1beta2, and includes a number of bug fixes and enhancements.

This is a patch release and contains important bug fixes in #645, #647, and #649.

This is the first release with the new v1beta2 version of the API (#578). This release also includes an upgrade to Spark 2.4.4 (#625) for both the operator image and Spark images for the various language bindings. This release includes the following enhancements and bug fixes:

• Support for high-availability mode with leader election (#518, #575).
• Allow PodSecurityContext to be overridden (#528).
• Spark pod mutating webhook enhancements (#543).
• A new ResourceQuota enforcement webhook (#544 ).
• Switch to using controller-gen to generate/update CRD yamls (#578).
• Integration with Volcano for batch scheduling (#567, #585, #599, #606).
• TTL support for SparkApplications (#615).
• Bug fixes.
• Documentation enhancements.

This release includes the following enhancements and fixes:

• Support for adding sidecar containers into the driver and executor pods (#478).
• Support for requesting GPUs (#509).
• Support for specifying hostNetwork for the driver and executor pods (#510).
• Support for specifying node selectors at pod level (#504).
• Support for specifying DNS config options for the driver and executor pods (#519).
• Changed the way driver pod status is checked (#506). The operator now sets the name of the driver pod while submitting an application to run and uses the name to get the driver pod instead of listing the Spark pods and filter out the driver pod.
• Changed the type of the driver UI service from NodePort to ClusterIP (#520).
• Other fixes and documentation enhancements.

This release contains an import bug fix for an issue related to application spec update and restart handling. Specifically it contains:

• Fixed a bug related to application spec update and restart handling: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/446.
• Added a submission ID used to group resources that belong to the same application run: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/446.

This release includes the following changes:

• The mutation webhook now gets SparkApplication objects and use their specs to figure out what to mutate to Spark driver and executor Pods. This replaced the original approach of using special annotations to carry needed mutations. This makes it much easier to support future mutations.
• Added support for specifying PodSecurityContext for the driver and/or executor Pods.
• Pinned the Kubernetes dependencies to a specific version instead of a release branch.
• Other bug fixes.

This beta release includes the following changes:

• Various bug and documentation fixes.
• Better jsonschema validation.

This is the first beta release of the operator after switching to use the v1beta1 version of the APIs.

This is an alpha release with the v1alpha1 version of the CRDs. This release includes the following changes:

• Fixes to allow running multiple instances of the operator with each one for a different namespace.
• Support for OpenShift clusters.
• Fixes to make the scheduling logic for ScheduledSparkApplications more robust.
• Other fixes and enhancements.

This is an alpha release with the v1alpha1 version of the CRDs. This release includes the following changes:

• Upgraded to Spark 2.4.0.
• Major refactoring to the SparkApplication controller logic around application/driver/executor state tracking and management.
• Optional ingress creation for Spark driver UI.
• Basic end-to-end tests.
• Other bug fixes and enhancements.

This release includes the following list of features and changes:

• Helm Chart for the Spark Operator.
• Support for creating a one-shot SparkApplication from a ScheduledSparkApplication.
• Support for tolerations.
• Upgraded to golang 1.11.
• Bug fixes.

This is an alpha release with API version v1alpha1 for Spark 2.3.x and up (including the Spark master branch or 2.4.0-SNAPSHOT). It includes the following list of features and changes:

• Support for exposing application-level (exposed and served by the operator) as well as driver and executor metrics to Prometheus.
• Support for automatically uploading local application dependencies to S3 in sparkctl.
• New mutating admission webhook that replaced the old initializer for pod customization.
• Support for using pod affinity/anti-affinity, mounting arbitrary volumes to the driver/executors, using image pull secret (requires a Spark image built from the Spark master).
• PySpark support (requires a Spark image built from the Spark master).
• Documentation enhancements.

This release version works with Spark 2.3.x. Major changes since the last release include the following:

• Native cron support via the ScheduledSparkApplication CRD.
• Bug fixes.
• Documentation enhancements.

This is the first release of the Spark Operator with API version v1alpha1. This release version works with Spark 2.3.x.