vcluster

Deploy Helm Charts within vcluster on startup

vcluster is now able to deploy helm charts defined in the chart values inside the virtual cluster on startup. For more information, please take a look at the vcluster docs

Other Changes

• chart: Fixed an issue where k8s distro was allowing Node authorization which is not needed. (thanks @olljanat)
• chart: Fixed an issue where vcluster reserves a lot more resources that it actually needs. (thanks @olljanat)
• chart: Added isolation mode to EKS distro.
• chart: Fixed an issue where pre-install job was not able to run in namespace where Restricted Pod Security Standard is enforced because of missing seccompProfile. (thanks @olljanat)
• syncer: Fixed an issue where vcluster synced pods could end with an - in hostname
• syncer: Fixed an issue where vcluster serviceNames where not correctly translated in alb ingresses
• cli: Added kube-config-context-name flag for create command as well as syncer container
• cli: Added auto completion support for the fish shell
• cli: Fixed an issue where vcluster connect was trying to deploy a proxy pod even though a minikube VM driver was used.

Changes

• chart: Fixed an issue where upgrading the k8s chart wasn't possible
• cli: Fixed an issue where slow k8s backends would fail vcluster creation
• cli: Fixed an issue where vcluster disconnect would falsely error if the vcluster was already deleted
• cli: vcluster cli will now deploy vcluster with enabled node sync on local k8s distributions
• syncer: Fixed an issue where k3s-based vcluster would excessively print error messages such as "Failed to connect to proxy" and "Remotedialer proxy error"

Changes

• chart: Fixed an issue where upgrading the k8s chart wasn't possible
• cli: Fixed an issue where slow k8s backends would fail vcluster creation
• cli: Fixed an issue where vcluster disconnect would falsely error if the vcluster was already deleted
• cli: vcluster cli will now deploy vcluster with enabled node sync on local k8s distributions
• syncer: Fixed an issue where k3s-based vcluster would excessively print error messages such as "Failed to connect to proxy" and "Remotedialer proxy error"

Changes

• chart: Added helm chart READMEs
• chart: Added the plugin name from the plugin.yaml to the environment of the plugin container
• syncer: Fixed an issue where vcluster where vcluster would emit unnecessary events in the case of non-root

Changes

• chart: Added helm chart READMEs
• chart: Added the plugin name from the plugin.yaml to the environment of the plugin container
• syncer: Fixed an issue where vcluster where vcluster would emit unnecessary events in the case of non-root

!! BREAKING CHANGES !!

• vcluster create now automatically does a connect by default. Can be disabled via vcluster create --connect=false
• vcluster connect now by default switches the current kube context to the vcluster context. Can be disabled via vcluster connect --update-current=false
• vcluster delete now by default deletes the namespace of the vcluster if it was created by vcluster before. Can be disabled via vcluster delete --auto-delete-namespace=false

Greatly improved vcluster cli experience

In this release we mainly focused on improving the CLI experience for vcluster. On most local kubernetes distributions (minikube, docker-desktop, k3d, kind) you are now able to start vcluster without port-forwarding which makes it a lot easier to get started and use vcluster for local k8s development. It's also great news if you consider using vcluster in your CI/CD pipelines as you can now use KIND and vcluster together without having to run vcluster connect in the background anymore.

So the current vcluster create && vcluster connect flow in a separate terminal simplifies to:

# Start and connect a new vcluster in a single command for local Kubernetes distros
vcluster create my-vcluster

# Then immediately access the vcluster with regular kubectl commands
kubectl get ns

# Disconnect or delete the vcluster when you are done to switch back
# to the host context
vcluster disconnect

Changes

• cli: New flag --kube-config-context-name for vcluster connect
• cli: Using vcluster connect with port-forwarding will now switch back the context on CTRL+C

!! BREAKING CHANGES !!

• vcluster create now automatically does a connect by default. Can be disabled via vcluster create --connect=false
• vcluster connect now by default switches the current kube context to the vcluster context. Can be disabled via vcluster connect --update-current=false
• vcluster delete now by default deletes the namespace of the vcluster if it was created by vcluster before. Can be disabled via vcluster delete --auto-delete-namespace=false

Greatly improved vcluster cli experience

In this release we focused on improving the CLI experience for vcluster. On most local kubernetes distributions (minikube, docker-desktop, k3d, kind) you are now able to start vcluster without port-forwarding which makes it a lot easier to get started and use vcluster for local k8s development. It's also great news if you consider using vcluster in your CI/CD pipelines as you can now use KIND and vcluster together without having to run vcluster connect in the background anymore.

So the current vcluster create && vcluster connect flow in a separate terminal simplifies to:

# Start and connect a new vcluster in a single command for local Kubernetes distros
vcluster create my-vcluster

# Then immediately access the vcluster with regular kubectl commands
kubectl get ns

# Disconnect or delete the vcluster when you are done to switch back
# to the host context
vcluster disconnect

Changes

• cli: New flag --kube-config-context-name for vcluster connect

!! BREAKING CHANGES !!

• vcluster create now automatically does a connect by default. Can be disabled via vcluster create --connect=false
• vcluster connect now by default switches the current kube context to the vcluster context. Can be disabled via vcluster connect --update-current=false
• vcluster delete now by default deletes the namespace of the vcluster if it was created by vcluster before. Can be disabled via vcluster delete --auto-delete-namespace=false

Greatly improved vcluster cli experience

In this release we focused on improving the CLI experience for vcluster. On most local kubernetes distributions (minikube, docker-desktop, k3d, kind) you are now able to start vcluster without port-forwarding which makes it a lot easier to get started and use vcluster for local k8s development. It's also great news if you consider using vcluster in your CI/CD pipelines as you can now use KIND and vcluster together without having to run vcluster connect in the background anymore.

So the current vcluster create && vcluster connect flow in a separate terminal simplifies to:

# Start and connect a new vcluster in a single command for local Kubernetes distros
vcluster create my-vcluster

# Then immediately access the vcluster with regular kubectl commands
kubectl get ns

# Disconnect or delete the vcluster when you are done to switch back
# to the host context
vcluster disconnect

Changes

• cli: New flag --kube-config-context-name for vcluster connect

Changes

• cli: New kube-config-context-name for vcluster connect to specify a custom kube context name
• syncer: When DEBUG env var is true, syncer profiling information will be served via built-in API server on the /debug/pprof/ path.
• syncer: If vcluster loses track of a physical pod (for example if vcluster was paused and restarted) and that physical pod was deleted, vcluster will also delete it upon restart
• syncer: Fixed an issue where plugin hooks wouldn't be registered correctly initially

Changes

• syncer: When DEBUG env var is true, syncer profiling information will be served via built-in API server on the /debug/pprof/ path.
• syncer: If vcluster loses track of a physical pod (for example if vcluster was paused and restarted) and that physical pod was deleted, vcluster will also delete it upon restart
• syncer: Fixed an issue where plugin hooks wouldn't be registered correctly initially

Changes

• syncer: vcluster will now print the modification made on update sync to console in debug mode

Plugin Hooks & Improvements

Plugins are now able to intercept and mutate incoming and outgoing objects from the syncer. This makes it possible to adjust syncer behaviour without the need to replace the complete syncer. Furthermore you can now label objects with vcluster.loft.sh/controlled-by to signal vcluster it should not reconcile this object and another party is responsible for handling it.

Other Changes

• cli: vcluster list command now has status column showing state of the vclusters. Now paused vclusters will also be shown in list output
• syncer: Enhanced the vcluster syncer to discover the correct value of the service CIDR range automatically
• syncer: Fixed an issue where vcluster 0.8 and above do not support k8s versions which use discovery.k8s.io/v1beta1 instead of discovery.k8s.io/v1 e.g rancher/k3s:v1.20.15-k3s1 (thanks @Shaked)
• syncer: Updated k8s dependencies to v1.24
• syncer: Fixed an issue where vcluster would unnecessarily update the coredns configmap
• syncer: Fixed an issue where vcluster wouldn't apply init manifests correctly if a target namespace was specified
• syncer: Fixed an issue where enabling the scheduler would not allow provisioning of pvc with a storage class that uses WaitForFirstConsumer
• syncer: vcluster will now fail to start if storage classes sync and legacy storage classes sync are enabled at the same time
• syncer: vcluster now supports 1.24 in k3s and k8s clusters
• syncer: vcluster now uses an index to remember what secrets belong to which pod. This should greatly improve perforamance for large vclusters as not every secrets needs to reiterate over every pod.
• chart: k8s & eks now deploys a regular job instead of using an pre-install hook
• chart: k8s & eks job now uses non root by default
• chart: supports coredns resources in helm chart
• chart: Improved helm chart notes
• chart: vcluster now supports priorityClassName option through helm chart

Plugin Hooks & Improvements

Plugins are now able to intercept and mutate incoming and outgoing objects from the syncer. This makes it possible to adjust syncer behaviour without the need to replace the complete syncer. Furthermore you can now label objects with vcluster.loft.sh/controlled-by to signal vcluster it should not reconcile this object and another party is responsible for handling it.

Other Changes

• cli: vcluster list command now has status column showing state of the vclusters. Now paused vclusters will also be shown in list output
• syncer: Enhanced the vcluster syncer to discover the correct value of the service CIDR range automatically
• syncer: Fixed an issue where vcluster 0.8 and above do not support k8s versions which use discovery.k8s.io/v1beta1 instead of discovery.k8s.io/v1 e.g rancher/k3s:v1.20.15-k3s1 (thanks @Shaked)
• syncer: Updated k8s dependencies to v1.24
• syncer: Fixed an issue where vcluster would unnecessarily update the coredns configmap
• syncer: Fixed an issue where vcluster wouldn't apply init manifests correctly if a target namespace was specified
• syncer: Fixed an issue where enabling the scheduler would not allow provisioning of pvc with a storage class that uses WaitForFirstConsumer
• syncer: vcluster will now fail to start if storage classes sync and legacy storage classes sync are enabled at the same time
• syncer: vcluster now supports 1.24 in k3s and k8s clusters
• syncer: vcluster now uses an index to remember what secrets belong to which pod. This should greatly improve perforamance for large vclusters as not every secrets needs to reiterate over every pod.
• chart: k8s & eks now deploys a regular job instead of using an pre-install hook
• chart: k8s & eks job now uses non root by default
• chart: supports coredns resources in helm chart
• chart: Improved helm chart notes
• chart: vcluster now supports priorityClassName option through helm chart

Plugin Hooks & Improvements

Plugins are now able to intercept and mutate incoming and outgoing objects from the syncer. This makes it possible to adjust syncer behaviour without the need to replace the complete syncer. Furthermore you can now label objects with vcluster.loft.sh/controlled-by to signal vcluster it should not reconcile this object and another party is responsible for handling it.

Other Changes

• cli: vcluster list command now has status column showing state of the vclusters. Now paused vclusters will also be shown in list output
• syncer: Enhanced the vcluster syncer to discover the correct value of the service CIDR range automatically
• syncer: Fixed an issue where vcluster 0.8 and above do not support k8s versions which use discovery.k8s.io/v1beta1 instead of discovery.k8s.io/v1 e.g rancher/k3s:v1.20.15-k3s1 (thanks @Shaked)
• syncer: Updated k8s dependencies to v1.24
• syncer: Fixed an issue where vcluster wouldn't apply init manifests correctly if a target namespace was specified
• syncer: Fixed an issue where enabling the scheduler would not allow provisioning of pvc with a storage class that uses WaitForFirstConsumer
• syncer: vcluster will now fail to start if storage classes sync and legacy storage classes sync are enabled at the same time
• syncer: vcluster now supports 1.24 in k3s and k8s clusters
• syncer: vcluster now uses an index to remember what secrets belong to which pod. This should greatly improve perforamance for large vclusters as not every secrets needs to reiterate over every pod.
• chart: k8s & eks now deploys a regular job instead of using an pre-install hook
• chart: k8s & eks job now uses non root by default
• chart: supports coredns resources in helm chart
• chart: Improved helm chart notes
• chart: vcluster now supports priorityClassName option through helm chart

Plugin Hooks & Improvements

Plugins are now able to intercept and mutate incoming and outgoing objects from the syncer. This makes it possible to adjust syncer behaviour without the need to replace the complete syncer. Furthermore you can now label objects with vcluster.loft.sh/controlled-by to signal vcluster it should not reconcile this object and another party is responsible for handling it.

Other Changes

• cli: vcluster list command now has status column showing state of the vclusters. Now paused vclusters will also be shown in list output
• syncer: Enhanced the vcluster syncer to discover the correct value of the service CIDR range automatically
• syncer: Fixed an issue where vcluster 0.8 and above do not support k8s versions which use discovery.k8s.io/v1beta1 instead of discovery.k8s.io/v1 e.g rancher/k3s:v1.20.15-k3s1 (thanks @Shaked)
• syncer: Updated k8s dependencies to v1.24
• syncer: Fixed an issue where vcluster wouldn't apply init manifests correctly if a target namespace was specified
• syncer: Fixed an issue where enabling the scheduler would not allow provisioning of pvc with a storage class that uses WaitForFirstConsumer
• syncer: vcluster will now fail to start if storage classes sync and legacy storage classes sync are enabled at the same time
• syncer: vcluster now supports 1.24 in k3s and k8s clusters
• syncer: vcluster now uses an index to remember what secrets belong to which pod. This should greatly improve perforamance for large vclusters as not every secrets needs to reiterate over every pod.
• chart: supports coredns resources in helm chart
• chart: Improved helm chart notes
• chart: vcluster now supports priorityClassName option through helm chart

Plugin Hooks & Improvements

Plugins are now able to intercept and mutate incoming and outgoing objects from the syncer. This makes it possible to adjust syncer behaviour without the need to replace the complete syncer. Furthermore you can now label objects with vcluster.loft.sh/controlled-by to signal vcluster it should not reconcile this object and another party is responsible for handling it.

Other Changes

• syncer: Updated k8s dependencies to v1.24
• syncer: Fixed an issue where enabling the scheduler would not allow provisioning of pvc with a storage class that uses WaitForFirstConsumer
• chart: Improved helm chart notes
• chart: vcluster now supports priorityClassName option through helm chart

Changes

• syncer: Fixed an issue where syncer would panic while applying init manifests
• chart: Fixed an issue where service account and image pull secrets were not correctly applied in k8s distro
• chart: Allows vcluster isolation NetworkPolicy to be used in a host cluster that uses 6443 port for the API server

Changes

• syncer: Fixed an issue where syncer would panic while applying init manifests
• chart: Fixed an issue where service account and image pull secrets were not correctly applied in k8s distro
• chart: Allows vcluster isolation NetworkPolicy to be used in a host cluster that uses 6443 port for the API server

Changes

• chart: Fixed an issue where service account and image pull secrets were not correctly applied in k8s distro
• chart: Allows vcluster isolation NetworkPolicy to be used in a host cluster that uses 6443 port for the API server

Service Mapping

vcluster now supports mapping services between host and virtual cluster. You can specify which services from the host cluster should be available inside the vcluster and which services inside the vcluster should be synced with the host cluster. You can configure this in the helm chart via the new section mapServices:

mapServices:
  # Services that should get mapped from the
  # virtual cluster to the host cluster.
  # vcluster will make sure to sync the service
  # ip to the host cluster automatically as soon
  # as the service exists.
  fromVirtual:
  - from: my-virtual-namespace/my-virtual-service
    to: my-host-service
  # Same as from virtual, but instead sync services
  # from the host cluster into the virtual cluster.
  # If the namespace does not exist, vcluster will
  # also create the namespace for the service.
  fromHost:
  - from: my-host-namespace/my-host-service
    to: my-virtual-namespace/my-virtual-service

For more information, please take a look at the vcluster docs

Init Manifests

vcluster now supports creation with manifests that will be applied as soon as the vcluster has started. This can be useful to configure and deploy virtual cluster with certain resources that are then deployed into vcluster itself. You can configure these manifests inside the helm values:

init:
  manifests: |-
    apiVersion: v1
    kind: Service
    ...
    ---
    apiVersion: v1
    kind: ConfigMap
    ...

vcluster Scheduler

vcluster now supports running a scheduler inside the virtual cluster. This is especially useful if you need to label and taint nodes within the vcluster and do not want to label or taint the actual host nodes. The scheduler can be enabled via:

sync:
  nodes:
    enabled: true
    syncAllNodes: true # or use nodeSelector 
    enableScheduler: true

This will tell vcluster to now start the scheduler inside vcluster and will only sync pods that have a node assigned. For more information, please take a look at the vcluster docs.

Changes

• cli: Warn if the patch version is passed in kubernetes version flag (#414)
• cli: New vcluster version command
• syncer: vcluster will now rewrite certain ingress annotations
• syncer: vcluster will now set the annotation cluster-autoscaler.kubernetes.io/daemonset-pod on pods that belong to a daemon set inside the virtual cluster
• syncer: Fixed an issue where vcluster would not migrate managed endpoints to Kubernetes managed endpoints correctly
• syncer: Fixed an issue with service type change from ExternalName type to other types
• syncer: Fixed an issue where default/kubernetes EndpointSlices content should match default/kubernetes Endpoints - add the missing ports
• syncer: Syncer should not set pods nodeSelector when the scheduler is used.
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: vcluster will now sync service selectors by default for better compatibility and reduced permission surface of vcluster instead of syncing all endpoints. vcluster will still sync endpoints for services that have no selector
• syncer: Fixed an issue where vcluster would not sync the increased persistent volume size back to the virtual cluster
• syncer: Fixed an issue where vcluster wouldn't sync ClusterIP to LoadBalancer change correctly
• syncer: vcluster will now recreate a PVC inside the virtual cluster if there is a corresponding host PVC with a different volume name
• syncer: Fixed an issue where default/kubernetes Endpoints object was referencing incorrect IPs in k8s and eks flavors.
• syncer: vcluster will now try to reapply the kube config secret periodically
• syncer: Fixed an issue where vcluster would try to bind a virtual pod multiple times to a node
• syncer: Use the vcluster service label selector for creating a fake kubelet service instead of finding out the pod label selector
• syncer: Update vcluster default go runtime to 1.18
• syncer: Fixed an issue where vcluster now will be able to pull images from a private registry.
• chart: coredns is service type and external IP can be now easily overridden through helm values (thanks @olljanat)
• chart: Add isolation.namespace
• chart: Enable CoreDNS multi-replica deployments via .coredns.replicas helm value
• chart: Add RFC6598 address space to network policy (thanks @desponda)