vcluster

Service Mapping

vcluster now supports mapping services between host and virtual cluster. You can specify which services from the host cluster should be available inside the vcluster and which services inside the vcluster should be synced with the host cluster. You can configure this in the helm chart via the new section mapServices:

mapServices:
  # Services that should get mapped from the
  # virtual cluster to the host cluster.
  # vcluster will make sure to sync the service
  # ip to the host cluster automatically as soon
  # as the service exists.
  fromVirtual:
  - from: my-virtual-namespace/my-virtual-service
    to: my-host-service
  # Same as from virtual, but instead sync services
  # from the host cluster into the virtual cluster.
  # If the namespace does not exist, vcluster will
  # also create the namespace for the service.
  fromHost:
  - from: my-host-namespace/my-host-service
    to: my-virtual-namespace/my-virtual-service

For more information, please take a look at the vcluster docs

Init Manifests

vcluster now supports creation with manifests that will be applied as soon as the vcluster has started. This can be useful to configure and deploy virtual cluster with certain resources that are then deployed into vcluster itself. You can configure these manifests inside the helm values:

init:
  manifests: |-
    apiVersion: v1
    kind: Service
    ...
    ---
    apiVersion: v1
    kind: ConfigMap
    ...

vcluster Scheduler

vcluster now supports running a scheduler inside the virtual cluster. This is especially useful if you need to label and taint nodes within the vcluster and do not want to label or taint the actual host nodes. The scheduler can be enabled via:

sync:
  nodes:
    enabled: true
    syncAllNodes: true # or use nodeSelector 
    enableScheduler: true

This will tell vcluster to now start the scheduler inside vcluster and will only sync pods that have a node assigned. For more information, please take a look at the vcluster docs.

Changes

• cli: Warn if the patch version is passed in kubernetes version flag (#414)
• cli: New vcluster version command
• syncer: vcluster will now rewrite certain ingress annotations
• syncer: vcluster will now set the annotation cluster-autoscaler.kubernetes.io/daemonset-pod on pods that belong to a daemon set inside the virtual cluster
• syncer: Fixed an issue where vcluster would not migrate managed endpoints to Kubernetes managed endpoints correctly
• syncer: Fixed an issue with service type change from ExternalName type to other types
• syncer: Fixed an issue where default/kubernetes EndpointSlices content should match default/kubernetes Endpoints - add the missing ports
• syncer: Syncer should not set pods nodeSelector when the scheduler is used.
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: vcluster will now sync service selectors by default for better compatibility and reduced permission surface of vcluster instead of syncing all endpoints. vcluster will still sync endpoints for services that have no selector
• syncer: Fixed an issue where vcluster would not sync the increased persistent volume size back to the virtual cluster
• syncer: Fixed an issue where vcluster wouldn't sync ClusterIP to LoadBalancer change correctly
• syncer: vcluster will now recreate a PVC inside the virtual cluster if there is a corresponding host PVC with a different volume name
• syncer: Fixed an issue where default/kubernetes Endpoints object was referencing incorrect IPs in k8s and eks flavors.
• syncer: vcluster will now try to reapply the kube config secret periodically
• syncer: Fixed an issue where vcluster would try to bind a virtual pod multiple times to a node
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: Update vcluster default go runtime to 1.18
• syncer: Fixed an issue where vcluster now will be able to pull images from a private registry.
• chart: coredns is service type and external IP can be now easily overridden through helm values (thanks @olljanat)
• chart: Add isolation.namespace
• chart: Enable CoreDNS multi-replica deployments via .coredns.replicas helm value
• chart: Add RFC6598 address space to network policy (thanks @desponda)

vcluster Scheduler

vcluster now supports running a scheduler inside the virtual cluster. This is especially useful if you need to label and taint nodes within the vcluster and do not want to label or taint the actual host nodes. The scheduler can be enabled via:

sync:
  nodes:
    enabled: true
    syncAllNodes: true # or use nodeSelector 
    enableScheduler: true

This will tell vcluster to now start the scheduler inside vcluster and will only sync pods that have a node assigned.

Changes

• cli: Warn if the patch version is passed in kubernetes version flag (#414)
• cli: New vcluster version command
• syncer: vcluster will now rewrite certain ingress annotations
• syncer: vcluster will now set the annotation cluster-autoscaler.kubernetes.io/daemonset-pod on pods that belong to a daemon set inside the virtual cluster
• syncer: Fixed an issue where vcluster would not migrate managed endpoints to Kubernetes managed endpoints correctly
• syncer: Fixed an issue with service type change from ExternalName type to other types
• syncer: Fixed an issue where default/kubernetes EndpointSlices content should match default/kubernetes Endpoints - add the missing ports
• syncer: Syncer should not set pods nodeSelector when the scheduler is used.
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: vcluster will now sync service selectors by default for better compatibility and reduced permission surface of vcluster instead of syncing all endpoints. vcluster will still sync endpoints for services that have no selector
• syncer: Fixed an issue where vcluster would not sync the increased persistent volume size back to the virtual cluster
• syncer: Fixed an issue where vcluster wouldn't sync ClusterIP to LoadBalancer change correctly
• syncer: vcluster will now recreate a PVC inside the virtual cluster if there is a corresponding host PVC with a different volume name
• syncer: Fixed an issue where default/kubernetes Endpoints object was referencing incorrect IPs in k8s and eks flavors.
• syncer: vcluster will now try to reapply the kube config secret periodically
• syncer: Fixed an issue where vcluster would try to bind a virtual pod multiple times to a node
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: Update vcluster default go runtime to 1.18
• syncer: Fixed an issue where vcluster now will be able to pull images from a private registry.
• chart: coredns is service type and external IP can be now easily overridden through helm values (thanks @olljanat)
• chart: Add isolation.namespace
• chart: Enable CoreDNS multi-replica deployments via .coredns.replicas helm value
• chart: Add RFC6598 address space to network policy (thanks @desponda)

vcluster Scheduler

vcluster now supports running a scheduler inside the virtual cluster. This is especially useful if you need to label and taint nodes within the vcluster and do not want to label or taint the actual host nodes. The scheduler can be enabled via:

sync:
  nodes:
    enabled: true
    syncAllNodes: true # or use nodeSelector 
    enableScheduler: true

This will tell vcluster to now start the scheduler inside vcluster and will only sync pods that have a node assigned.

Changes

• cli: Warn if the patch version is passed in kubernetes version flag (#414)
• cli: New vcluster version command
• syncer: vcluster will now set the annotation cluster-autoscaler.kubernetes.io/daemonset-pod on pods that belong to a daemon set inside the virtual cluster
• syncer: Fixed an issue where vcluster would not migrate managed endpoints to Kubernetes managed endpoints correctly
• syncer: Fixed an issue with service type change from ExternalName type to other types
• syncer: Fixed an issue where default/kubernetes EndpointSlices content should match default/kubernetes Endpoints - add the missing ports
• syncer: Syncer should not set pods nodeSelector when the scheduler is used.
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: vcluster will now sync service selectors by default for better compatibility and reduced permission surface of vcluster instead of syncing all endpoints. vcluster will still sync endpoints for services that have no selector
• syncer: Fixed an issue where vcluster would not sync the increased persistent volume size back to the virtual cluster
• syncer: Fixed an issue where vcluster wouldn't sync ClusterIP to LoadBalancer change correctly
• syncer: vcluster will now recreate a PVC inside the virtual cluster if there is a corresponding host PVC with a different volume name
• syncer: Fixed an issue where default/kubernetes Endpoints object was referencing incorrect IPs in k8s and eks flavors.
• syncer: vcluster will now try to reapply the kube config secret periodically
• syncer: Fixed an issue where vcluster would try to bind a virtual pod multiple times to a node
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: Update vcluster default go runtime to 1.18
• syncer: Fixed an issue where vcluster now will be able to pull images from a private registry.
• chart: coredns is service type and external IP can be now easily overridden through helm values (thanks @olljanat)
• chart: Add isolation.namespace
• chart: Enable CoreDNS multi-replica deployments via .coredns.replicas helm value
• chart: Add RFC6598 address space to network policy (thanks @desponda)

vcluster Scheduler

vcluster now supports running a scheduler inside the virtual cluster. This is especially useful if you need to label and taint nodes within the vcluster and do not want to label or taint the actual host nodes. The scheduler can be enabled via:

sync:
  nodes:
    enabled: true
    syncAllNodes: true # or use nodeSelector 
    enableScheduler: true

This will tell vcluster to now start the scheduler inside vcluster and will only sync pods that have a node assigned.

Changes

• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: vcluster will now sync service selectors by default for better compatibility and reduced permission surface of vcluster instead of syncing all endpoints. vcluster will still sync endpoints for services that have no selector
• syncer: Fixed an issue where vcluster would not sync the increased persistent volume size back to the virtual cluster
• syncer: Fixed an issue where vcluster wouldn't sync ClusterIP to LoadBalancer change correctly
• syncer: vcluster will now recreate a PVC inside the virtual cluster if there is a corresponding host PVC with a different volume name
• syncer: Fixed an issue where default/kubernetes Endpoints object was referencing incorrect IPs in k8s and eks flavors.
• syncer: vcluster will now try to reapply the kube config secret periodically
• syncer: Fixed an issue where vcluster would try to bind a virtual pod multiple times to a node
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: Update vcluster default go runtime to 1.18
• chart: Add isolation.namespace
• chart: Enable CoreDNS multi-replica deployments via .coredns.replicas helm value
• chart: Add RFC6598 address space to network policy (thanks @desponda)

Changes

• syncer: New syncer flag --sync-service-selector to let vcluster sync service selectors instead of endpoints. This has the advantage that it requires less permissions than the default, but does not work for services without a selector or leader election endpoints. (#281)
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• chart: Add RFC6598 address space to network policy (thanks @desponda)

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Old behaviour can restored by enabling the legacy storageclasses sync with:

sync:
  legacy-storageclasses:
    enabled: true

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Old behaviour can restored by enabling the legacy storageclasses sync with:

sync:
  legacy-storageclasses:
    enabled: true

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• chart: coredns is now part of the vcluster chart and dns configurations can be easily overridden through helm values
• syncer: Now correctly rewrites PVC metrics as well
• syncer: The statefulset init container now has default resources and runs as non root
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

!! Breaking Changes !!

vcluster now includes the coredns manifests directly in the helm chart. If you are overriding the path /manifests/coredns inside the syncer with your custom manifests, you'll need to disable coredns configmap via values.yaml:

coredns:
  enabled: false

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• syncer: Now correctly rewrites PVC metrics as well
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. (Coming soon) vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• syncer: Now correctly rewrites PVC metrics as well
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

Isolated Mode

Creating a secure multi tenancy environment is hard and vcluster is already able to provide an isolated control plane in a Kubernetes cluster, however actual vcluster workload isolation is currently still up to the users themselves to figure out. With v0.7.0 we want to introduce a new vcluster feature that automatically creates common defaults for workload isolation that can be enabled via the --isolate flag in vcluster create or through the helm value isolation.enabled: true. This feature imposes a couple of restrictions on vcluster workloads to make sure they do not break out of their virtual environment:

1. vcluster enforces a Pod Security Standard on syncer level, which means that for example pods that try to run as privileged container or mount a host path will not be synced to the host cluster. Valid options are baseline (default in isolated mode) and restricted.
2. vcluster deploys a resource quota as well as limit range alongside the vcluster itself. This allows restricting resource consumption of vcluster workloads. If enabled sane defaults for those 2 resources are chosen.
3. (Coming soon) vcluster deploys a network policy alongside itself that will restrict access of vcluster workloads to other pods in the host cluster.

Please take a look at the isolated mode docs for more information.

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• syncer: Adds --sync-label flag to sync untranslated labels onto physical resources (thanks @bschwenn)
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

Changed StorageClasses Syncer

vcluster will now sync storage classes from the virtual cluster to the host cluster if sync of storage classes is enabled. This will replace the current behaviour where storage classes where only synced from host to virtual cluster.

We decided to replace the existing behaviour, because creating storage classes is a valid use case as long as the CSI driver is installed within the host cluster, but certain parameters for the CSI driver should get changed through a storage class. It also makes sense to not sync created storage classes from the host cluster anymore as this is not required to schedule persistent volume claims and currently just has informational purposes.

This is somewhat a breaking change as vclusters that currently have sync of storage classes enabled would now behave differently moving forward as changes to the host cluster storage classes are not propagated anymore. However migration should work as expected, as created storage classes within vcluster that mirrored host cluster storage classes before would just get created in the host cluster under a different name.

Added ServiceAccounts sync support

Added a new service account syncer that makes it possible to sync service accounts from the vcluster to the host cluster with certain annotations and labels. This is useful for features such as IAM Roles for ServiceAccounts where the service account needs a certain annotation to give AWS permissions to a pod

Other Changes

• chart: EKS chart now accepts .sync.* values as documented
• syncer: Fixed an issue where syncing the service spec.loadBalancerSourceRanges was not possible

Plugins

Plugins are a feature to extend the capabilities of vcluster. They allow you to add custom functionality, such as:

1. Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
2. Syncing custom resources from or to the virtual cluster
3. Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
4. Manage resources and applications inside the host or virtual cluster
5. Enforcing certain restrictions on synced resources or extending the existing syncers of vcluster
6. Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.

For more information, please take a look at the vcluster docs.

Pause and Resume vclusters

vcluster is now able to pause and resume. Pausing a vcluster means to temporarily scale down the vcluster and delete all its created workloads on the host cluster. This can be useful to save computing resources used by vcluster workloads in the host cluster.

For more information please checkout the vcluster docs

Directly execute a command with vcluster connect in same shell

vcluster now allows command execution with vcluster context in command vcluster connect. For example:

# Retrieve vcluster namespaces
vcluster connect test -n test -- kubectl get ns

# New shell with vcluster kube context
vcluster connect test -n test -- bash

Expiring kube configs & automatic service account creation

vcluster is now able to automatically create service account tokens for generated kube configs, which allow you to easily create kube configs for other vcluster users that should not be cluster admin. For example:

# Create a kube config for a cluster viewer
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view

# OR: create a kube config for a cluster admin
vcluster connect my-vcluster -n my-vcluster --service-account admin --cluster-role cluster-admin

# OR: create a kube config that expires after an hour
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view --token-expiration 3600

This makes it also possible to use vcluster more easily without ingresses that require ssl passthrough. For more information please checkout the vcluster access docs and vcluster ingress docs

Support for VolumeSnapshots

vcluster now supports syncing of volume snapshots between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  volumesnapshots:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Support for PodDisruptionBudgets

vcluster now suports syncing of pod disruption budgets between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  poddisruptionbudgets:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Deprecations:

• .rbac.clusterRole.create, .rbac.role.extended - both helm values will be removed in a future version of vcluster. Their function is replaced by the new .sync.* helm values, which will ensure that minimal necessary RBAC role and clusterrole is created based on the resources that will be synced by vcluster.
• .rbac.role.create helm value will be removed in future version of vcluster and minimal standard role will always be created.
• --create-cluster-role flag of the vcluster create CLI command is deprecated for the same reasons as the .rbac.clusterRole.create helm value, as described above.

Other Changes

• cli: New flag --insecure for vcluster connect to create a kube config with insecure-skip-tls-verify
• cli: vcluster create can now use urls as value for -f flags
• cli: New command vcluster get service-cidr to print the current clusters service cidr
• cli: Notify the user about incompatible workdir file when running vcluster create
• cli: vcluster connect will now use a random port locally to avoid port conflicts if no --local-port flag is specified.
• cli: Added shortcut -s for global flag --silent
• cli: New eks distro (thanks @rothgar)
• syncer: Removed core dns tolerations (thanks @mtougeron)
• syncer: Make initial kube config secret creation non-fatal
• syncer: If phyiscal and virtual pod have a node assigned and they differ, delete the physical pod
• syncer: Sync service nodePort from host cluster instead of virtual cluster
• syncer: Only write coredns manifests to local file if environment variable "DEBUG" is "true"
• syncer: Use vcluster service as owner instead of deployment / statefulset, because we can set the controller: true in the owner reference for it, which allows node draining (drain #273)
• syncer: Fixed an issue where webhooks were not correctly resolved in vcluster
• syncer: Start controllers after api server started
• syncer: Fixed syncer cache race condition after Service creation
• syncer: Improved node / pod syncer logic to make it more robust
• syncer: New --toleration flag to add tolerations automatically to each pod (#330 thanks @kuuji)
• syncer: the --sync flag can now be passed to the syncer multiple times, and all the values will be combined. Disabling sync of a certain resource with a --sync=-resource flag still takes precedence over any enabling --sync=resource flags that might follow.
• chart: Fixed a problem where k3s would not startup correctly on cgroupsv2 nodes (#264)
• chart: new .sync.* values have been added to control which resources are being synced, and which permissions are given to vcluster via RBAC role and cluster role. This way the RBAC permissions are controlled on a more granular level, and the old .rbac helm values are deprecated. Using the .sync.RESOURCE.enabled values is now the recommended way to enable/disable which resources are synced. See docs for usage examples - https://www.vcluster.com/docs/architecture/synced-resources
• chart: the .sync.nodes.syncAllNodes, .sync.nodes.nodeSelector and .sync.nodes.syncNodeChanges values have been added for easier control of node syncing behavior via helm charts and more precise RBAC permissions controll. See docs for usage examples - https://www.vcluster.com/docs/architecture/nodes . Direct use of the --sync-all-nodes, --node-selector and --enforce-node-selector syncer args is not recommended because the associated RBAC permissions may be missing.
• chart: add support for externalIPs & externalTrafficPolicy (thanks @log1cb0mb)
• other: vcluster now uses cosign to sign its images and binaries (#220). Thanks @developer-guy for your contribution!
• other: For each release there will be a new vcluster-images.txt which holds all the needed images by vcluster. In addition, we include two scripts to download and push the needed images automatically

Plugins

Plugins are a feature to extend the capabilities of vcluster. They allow you to add custom functionality, such as:

1. Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
2. Syncing custom resources from or to the virtual cluster
3. Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
4. Manage resources and applications inside the host or virtual cluster
5. Enforcing certain restrictions on synced resources or extending the existing syncers of vcluster
6. Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.

For more information, please take a look at the vcluster docs.

Pause and Resume vclusters

vcluster is now able to pause and resume. Pausing a vcluster means to temporarily scale down the vcluster and delete all its created workloads on the host cluster. This can be useful to save computing resources used by vcluster workloads in the host cluster.

For more information please checkout the vcluster docs

Directly execute a command with vcluster connect in same shell

vcluster now allows command execution with vcluster context in command vcluster connect. For example:

# Retrieve vcluster namespaces
vcluster connect test -n test -- kubectl get ns

# New shell with vcluster kube context
vcluster connect test -n test -- bash

Expiring kube configs & automatic service account creation

vcluster is now able to automatically create service account tokens for generated kube configs, which allow you to easily create kube configs for other vcluster users that should not be cluster admin. For example:

# Create a kube config for a cluster viewer
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view

# OR: create a kube config for a cluster admin
vcluster connect my-vcluster -n my-vcluster --service-account admin --cluster-role cluster-admin

# OR: create a kube config that expires after an hour
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view --token-expiration 3600

This makes it also possible to use vcluster more easily without ingresses that require ssl passthrough. For more information please checkout the vcluster access docs and vcluster ingress docs

Support for VolumeSnapshots

vcluster now supports syncing of volume snapshots between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  volumesnapshots:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Support for PodDisruptionBudgets

vcluster now suports syncing of pod disruption budgets between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  poddisruptionbudgets:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Deprecations:

• .rbac.clusterRole.create, .rbac.role.extended - both helm values will be removed in a future version of vcluster. Their function is replaced by the new .sync.* helm values, which will ensure that minimal necessary RBAC role and clusterrole is created based on the resources that will be synced by vcluster.
• .rbac.role.create helm value will be removed in future version of vcluster and minimal standard role will always be created.
• --create-cluster-role flag of the vcluster create CLI command is deprecated for the same reasons as the .rbac.clusterRole.create helm value, as described above.

Other Changes

• cli: New flag --insecure for vcluster connect to create a kube config with insecure-skip-tls-verify
• cli: vcluster create can now use urls as value for -f flags
• cli: New command vcluster get service-cidr to print the current clusters service cidr
• cli: Notify the user about incompatible workdir file when running vcluster create
• cli: vcluster connect will now use a random port locally to avoid port conflicts if no --local-port flag is specified.
• cli: Added shortcut -s for global flag --silent
• cli: New eks distro (thanks @rothgar)
• syncer: Removed core dns tolerations (thanks @mtougeron)
• syncer: Make initial kube config secret creation non-fatal
• syncer: If phyiscal and virtual pod have a node assigned and they differ, delete the physical pod
• syncer: Sync service nodePort from host cluster instead of virtual cluster
• syncer: Only write coredns manifests to local file if environment variable "DEBUG" is "true"
• syncer: Use vcluster service as owner instead of deployment / statefulset, because we can set the controller: true in the owner reference for it, which allows node draining (drain #273)
• syncer: Fixed an issue where webhooks were not correctly resolved in vcluster
• syncer: Start controllers after api server started
• syncer: Fixed syncer cache race condition after Service creation
• syncer: Improved node / pod syncer logic to make it more robust
• syncer: New --toleration flag to add tolerations automatically to each pod (#330 thanks @kuuji)
• syncer: the --sync flag can now be passed to the syncer multiple times, and all the values will be combined. Disabling sync of a certain resource with a --sync=-resource flag still takes precedence over any enabling --sync=resource flags that might follow.
• chart: Fixed a problem where k3s would not startup correctly on cgroupsv2 nodes (#264)
• chart: new .sync.* values have been added to control which resources are being synced, and which permissions are given to vcluster via RBAC role and cluster role. This way the RBAC permissions are controlled on a more granular level, and the old .rbac helm values are deprecated. Using the .sync.RESOURCE.enabled values is now the recommended way to enable/disable which resources are synced. See docs for usage examples - https://www.vcluster.com/docs/architecture/synced-resources
• chart: the .sync.nodes.syncAllNodes, .sync.nodes.nodeSelector and .sync.nodes.syncNodeChanges values have been added for easier control of node syncing behavior via helm charts and more precise RBAC permissions controll. See docs for usage examples - https://www.vcluster.com/docs/architecture/nodes . Direct use of the --sync-all-nodes, --node-selector and --enforce-node-selector syncer args is not recommended because the associated RBAC permissions may be missing.
• chart: add support for externalIPs & externalTrafficPolicy (thanks @log1cb0mb)
• other: vcluster now uses cosign to sign its images and binaries (#220). Thanks @developer-guy for your contribution!
• other: For each release there will be a new vcluster-images.txt which holds all the needed images by vcluster. In addition, we include two scripts to download and push the needed images automatically

Plugins

Plugins are a feature to extend the capabilities of vcluster. They allow you to add custom functionality, such as:

1. Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
2. Syncing custom resources from or to the virtual cluster
3. Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
4. Manage resources and applications inside the host or virtual cluster
5. Enforcing certain restrictions on synced resources or extending the existing syncers of vcluster
6. Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.

For more information, please take a look at the vcluster docs.

Pause and Resume vclusters

vcluster is now able to pause and resume. Pausing a vcluster means to temporarily scale down the vcluster and delete all its created workloads on the host cluster. This can be useful to save computing resources used by vcluster workloads in the host cluster.

For more information please checkout the vcluster docs

Directly execute a command with vcluster connect in same shell

vcluster now allows command execution with vcluster context in command vcluster connect. For example:

# Retrieve vcluster namespaces
vcluster connect test -n test -- kubectl get ns

# New shell with vcluster kube context
vcluster connect test -n test -- bash

Expiring kube configs & automatic service account creation

vcluster is now able to automatically create service account tokens for generated kube configs, which allow you to easily create kube configs for other vcluster users that should not be cluster admin. For example:

# Create a kube config for a cluster viewer
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view

# OR: create a kube config for a cluster admin
vcluster connect my-vcluster -n my-vcluster --service-account admin --cluster-role cluster-admin

# OR: create a kube config that expires after an hour
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view --token-expiration 3600

This makes it also possible to use vcluster more easily without ingresses that require ssl passthrough. For more information please checkout the vcluster access docs and vcluster ingress docs

Support for VolumeSnapshots

vcluster now supports syncing of volume snapshots between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  volumesnapshots:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Support for PodDisruptionBudgets

vcluster now suports syncing of pod disruption budgets between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  poddisruptionbudgets:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Deprecations:

• .rbac.clusterRole.create, .rbac.role.extended - both helm values will be removed in a future version of vcluster. Their function is replaced by the new .sync.* helm values, which will ensure that minimal necessary RBAC role and clusterrole is created based on the resources that will be synced by vcluster.
• .rbac.role.create helm value will be removed in future version of vcluster and minimal standard role will always be created.
• --create-cluster-role flag of the vcluster create CLI command is deprecated for the same reasons as the .rbac.clusterRole.create helm value, as described above.

Other Changes

• cli: New flag --insecure for vcluster connect to create a kube config with insecure-skip-tls-verify
• cli: vcluster create can now use urls as value for -f flags
• cli: New command vcluster get service-cidr to print the current clusters service cidr
• cli: Notify the user about incompatible workdir file when running vcluster create
• cli: vcluster connect will now use a random port locally to avoid port conflicts if no --local-port flag is specified.
• cli: Added shortcut -s for global flag --silent
• cli: New eks distro (thanks @rothgar)
• syncer: Removed core dns tolerations (thanks @mtougeron)
• syncer: Make initial kube config secret creation non-fatal
• syncer: If phyiscal and virtual pod have a node assigned and they differ, delete the physical pod
• syncer: Sync service nodePort from host cluster instead of virtual cluster
• syncer: Only write coredns manifests to local file if environment variable "DEBUG" is "true"
• syncer: Use vcluster service as owner instead of deployment / statefulset, because we can set the controller: true in the owner reference for it, which allows node draining (drain #273)
• syncer: Fixed an issue where webhooks were not correctly resolved in vcluster
• syncer: Start controllers after api server started
• syncer: Fixed syncer cache race condition after Service creation
• syncer: Improved node / pod syncer logic to make it more robust
• syncer: New --toleration flag to add tolerations automatically to each pod (#330 thanks @kuuji)
• syncer: the --sync flag can now be passed to the syncer multiple times, and all the values will be combined. Disabling sync of a certain resource with a --sync=-resource flag still takes precedence over any enabling --sync=resource flags that might follow.
• chart: Fixed a problem where k3s would not startup correctly on cgroupsv2 nodes (#264)
• chart: new .sync.* values have been added to control which resources are being synced, and which permissions are given to vcluster via RBAC role and cluster role. This way the RBAC permissions are controlled on a more granular level, and the old .rbac helm values are deprecated. Using the .sync.RESOURCE.enabled values is now the recommended way to enable/disable which resources are synced. See docs for usage examples - https://www.vcluster.com/docs/architecture/synced-resources
• chart: the .sync.nodes.syncAllNodes, .sync.nodes.nodeSelector and .sync.nodes.syncNodeChanges values have been added for easier control of node syncing behavior via helm charts and more precise RBAC permissions controll. See docs for usage examples - https://www.vcluster.com/docs/architecture/nodes . Direct use of the --sync-all-nodes, --node-selector and --enforce-node-selector syncer args is not recommended because the associated RBAC permissions may be missing.
• chart: add support for externalIPs & externalTrafficPolicy (thanks @log1cb0mb)
• other: vcluster now uses cosign to sign its images and binaries (#220). Thanks @developer-guy for your contribution!
• other: For each release there will be a new vcluster-images.txt which holds all the needed images by vcluster. In addition, we include two scripts to download and push the needed images automatically

Plugins

Plugins are a feature to extend the capabilities of vcluster. They allow you to add custom functionality, such as:

1. Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
2. Syncing custom resources from or to the virtual cluster
3. Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
4. Manage resources and applications inside the host or virtual cluster
5. Enforcing certain restrictions on synced resources or extending the existing syncers of vcluster
6. Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.

For more information, please take a look at the vcluster docs.

Pause and Resume vclusters

vcluster is now able to pause and resume. Pausing a vcluster means to temporarily scale down the vcluster and delete all its created workloads on the host cluster. This can be useful to save computing resources used by vcluster workloads in the host cluster.

For more information please checkout the vcluster docs

Directly execute a command with vcluster connect in same shell

vcluster now allows command execution with vcluster context in command vcluster connect. For example:

# Retrieve vcluster namespaces
vcluster connect test -n test -- kubectl get ns

# New shell with vcluster kube context
vcluster connect test -n test -- bash

Expiring kube configs & automatic service account creation

vcluster is now able to automatically create service account tokens for generated kube configs, which allow you to easily create kube configs for other vcluster users that should not be cluster admin. For example:

# Create a kube config for a cluster viewer
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view

# OR: create a kube config for a cluster admin
vcluster connect my-vcluster -n my-vcluster --service-account admin --cluster-role cluster-admin

# OR: create a kube config that expires after an hour
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view --token-expiration 3600

This makes it also possible to use vcluster more easily without ingresses that require ssl passthrough. For more information please checkout the vcluster access docs and vcluster ingress docs

Support for VolumeSnapshots

vcluster now supports syncing of volume snapshots between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  volumesnapshots:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Support for PodDisruptionBudgets

vcluster now suports syncing of pod disruption budgets between the host and virtual cluster, that can be enabled via a values.yaml:

sync:
  poddisruptionbudgets:
    enabled: true

and then used via:

vcluster create ... -f values.yaml

Deprecations:

• .rbac.clusterRole.create, .rbac.role.extended - both helm values will be removed in a future version of vcluster. Their function is replaced by the new .sync.* helm values, which will ensure that minimal necessary RBAC role and clusterrole is created based on the resources that will be synced by vcluster.
• .rbac.role.create helm value will be removed in future version of vcluster and minimal standard role will always be created.
• --create-cluster-role flag of the vcluster create CLI command is deprecated for the same reasons as the .rbac.clusterRole.create helm value, as described above.

Other Changes

• cli: New flag --insecure for vcluster connect to create a kube config with insecure-skip-tls-verify
• cli: vcluster create can now use urls as value for -f flags
• cli: New command vcluster get service-cidr to print the current clusters service cidr
• cli: Notify the user about incompatible workdir file when running vcluster create
• cli: vcluster connect will now use a random port locally to avoid port conflicts if no --local-port flag is specified.
• cli: Added shortcut -s for global flag --silent
• syncer: Make initial kube config secret creation non-fatal
• syncer: If phyiscal and virtual pod have a node assigned and they differ, delete the physical pod
• syncer: Sync service nodePort from host cluster instead of virtual cluster
• syncer: Only write coredns manifests to local file if environment variable "DEBUG" is "true"
• syncer: Use vcluster service as owner instead of deployment / statefulset, because we can set the controller: true in the owner reference for it, which allows node draining (drain #273)
• syncer: Fixed an issue where webhooks were not correctly resolved in vcluster
• syncer: Start controllers after api server started
• syncer: Fixed syncer cache race condition after Service creation
• syncer: Improved node / pod syncer logic to make it more robust
• syncer: New --toleration flag to add tolerations automatically to each pod (#330 thanks @kuuji)
• syncer: the --sync flag can now be passed to the syncer multiple times, and all the values will be combined. Disabling sync of a certain resource with a --sync=-resource flag still takes precedence over any enabling --sync=resource flags that might follow.
• chart: Fixed a problem where k3s would not startup correctly on cgroupsv2 nodes (#264)
• chart: new .sync.* values have been added to control which resources are being synced, and which permissions are given to vcluster via RBAC role and cluster role. This way the RBAC permissions are controlled on a more granular level, and the old .rbac helm values are deprecated. Using the .sync.RESOURCE.enabled values is now the recommended way to enable/disable which resources are synced. See docs for usage examples - https://www.vcluster.com/docs/architecture/synced-resources
• chart: the .sync.nodes.syncAllNodes, .sync.nodes.nodeSelector and .sync.nodes.syncNodeChanges values have been added for easier control of node syncing behavior via helm charts and more precise RBAC permissions controll. See docs for usage examples - https://www.vcluster.com/docs/architecture/nodes . Direct use of the --sync-all-nodes, --node-selector and --enforce-node-selector syncer args is not recommended because the associated RBAC permissions may be missing.
• chart: add support for externalIPs & externalTrafficPolicy (thanks @log1cb0mb)
• other: vcluster now uses cosign to sign its images and binaries (#220). Thanks @developer-guy for your contribution!
• other: For each release there will be a new vcluster-images.txt which holds all the needed images by vcluster. In addition, we include two scripts to download and push the needed images automatically

Plugins

Plugins are a feature to extend the capabilities of vcluster. They allow you to add custom functionality, such as:

1. Syncing specific resources from or to the virtual clusters, including cluster scoped resources like cluster roles
2. Syncing custom resources from or to the virtual cluster
3. Deploying resources on virtual cluster startup, such as CRDs, applications, etc.
4. Manage resources and applications inside the host or virtual cluster
5. Enforcing certain restrictions on synced resources or extending the existing syncers of vcluster
6. Any other operator use case that could benefit from having access to the virtual cluster and the host cluster simultaneously.

For more information, please take a look at the vcluster docs.

Pause and Resume vclusters

vcluster is now able to pause and resume. Pausing a vcluster means to temporarily scale down the vcluster and delete all its created workloads on the host cluster. This can be useful to save computing resources used by vcluster workloads in the host cluster.

For more information please checkout the vcluster docs

Directly execute a command with vcluster connect in same shell

vcluster now allows command execution with vcluster context in command vcluster connect. For example:

# Retrieve vcluster namespaces
vcluster connect test -n test -- kubectl get ns

# New shell with vcluster kube context
vcluster connect test -n test -- bash

Expiring kube configs & automatic service account creation

vcluster is now able to automatically create service account tokens for generated kube configs, which allow you to easily create kube configs for other vcluster users that should not be cluster admin. For example:

# Create a kube config for a cluster viewer
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view

# OR: create a kube config for a cluster admin
vcluster connect my-vcluster -n my-vcluster --service-account admin --cluster-role cluster-admin

# OR: create a kube config that expires after an hour
vcluster connect my-vcluster -n my-vcluster --service-account viewer --cluster-role view --token-expiration 3600

This makes it also possible to use vcluster more easily without ingresses that require ssl passthrough. For more information please checkout the vcluster access docs and vcluster ingress docs

Support for VolumeSnapshots

vcluster now supports syncing of volume snapshots between the host and virtual cluster, that can be enabled via a values.yaml:

rbac:
  clusterRole:
    enabled: true
  role:
    extended: true

syncer:
  extraArgs:
  - --sync=volumesnapshots

and then used via:

vcluster create ... -f values.yaml

Support for PodDisruptionBudgets

vcluster now suports syncing of pod disruption budgets between the host and virtual cluster, that can be enabled via a values.yaml:

rbac:
  role:
    extended: true

syncer:
  extraArgs:
  - --sync=poddisruptionbudgets

and then used via:

vcluster create ... -f values.yaml

Other Changes

• cli: New flag --insecure for vcluster connect to create a kube config with insecure-skip-tls-verify
• cli: vcluster create can now use urls as value for -f flags
• cli: New command vcluster get service-cidr to print the current clusters service cidr
• cli: Notify the user about incompatible workdir file when running vcluster create
• cli: vcluster connect will now use a random port locally to avoid port conflicts if no --local-port flag is specified.
• cli: Added shortcut -s for global flag --silent
• syncer: Make initial kube config secret creation non-fatal
• syncer: If phyiscal and virtual pod have a node assigned and they differ, delete the physical pod
• syncer: Sync service nodePort from host cluster instead of virtual cluster
• syncer: Only write coredns manifests to local file if environment variable "DEBUG" is "true"
• syncer: Use vcluster service as owner instead of deployment / statefulset, because we can set the controller: true in the owner reference for it, which allows node draining (drain #273)
• syncer: Fixed an issue where webhooks were not correctly resolved in vcluster
• syncer: Start controllers after api server started
• syncer: Fixed syncer cache race condition after Service creation
• syncer: Improved node / pod syncer logic to make it more robust
• chart: Fixed a problem where k3s would not startup correctly on cgroupsv2 nodes (#264)
• other: vcluster now uses cosign to sign its images and binaries (#220). Thanks @developer-guy for your contribution!
• other: For each release there will be a new vcluster-images.txt which holds all the needed images by vcluster. In addition, we include two scripts to download and push the needed images automatically