Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
APACHE-2.0 License
Bot releases are hidden (Show)
Published by RainbowMango about 2 years ago
karmadactl
: Fixed a panic issue when retrieving resources from an unknown cluster(karmadactl get xxx --cluster=not-exist
). (#2171, #2201, @my-git9)karmadactl
: Fix Kubernetes v1.24 can not be joined issue. (#1972, @zgfh)karmada-controller-manager
: Fixed Argo CD can not assess Deployment health status issue. (#2256, @xuqianjins)karmada-controller-manager
: Fixed Argo CD can not assess StatefulSet/DaemonSet health status issue. (#2264, @RainbowMango)karmada-search
: Fixed can not query a single namespace issue. (#2274, @luoMonkeyKing)karmada-search
: Fixed panic issue when dumps error info. (#2333, @AllenZMC)Helm Chart
: Fixed misconfigured MutatingWebhookConfiguration
and added missing APIService
configuration for karmada-aggregated-apiserver
. (#2420, @zhixian82)Published by RainbowMango about 2 years ago
karmadactl
: Fix Kubernetes v1.24 can not be joined issue. (#2199, @zgfh)Helm Chart
: Fixed misconfigured MutatingWebhookConfiguration
and added missing APIService
configuration for karmada-aggregated-apiserver
. (#2421, @zhixian82)Published by RainbowMango about 2 years ago
karmadactl
: Fixed Kubernetes v1.24 can not be joined issue. (#2200, @zgfh)Helm Chart
: Fixed misconfigured MutatingWebhookConfiguration
and added missing APIService
configuration for karmada-aggregated-apiserver
. (#2422, @zhixian82)Published by RainbowMango over 2 years ago
karmada-search
: Fixed panic when the resource annotation is nil. (#1939, @XiShanYongYe-Chang )karmada-search
: Fixed a panic issue comparing uncomparable type cache.ResourceEventHandlerFuncs
. (#1971, @liys87x )karmadactl
: Removed --dry-run
flag from describe
, exec
and log
commands. (#2036, @wlp1153468871 )karmada-controller-manager
: Fixed Job
status might be incorrectly marked as Completed
issue. (#2007, @Garrybest )karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2127, @AllenZMC )Published by RainbowMango over 2 years ago
karmadactl
: Fixed namespace can not be customized issue. (#1827, @likakuli )karmadactl
: fixed karmadactl can not taint while karmada control plane config is not located on default path. (#1838, @wuyingjun-lucky )karmada-controller-manager
: Fixed Job
status might be incorrectly marked as Completed
issue. (#2010, @Garrybest )karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2126, @AllenZMC )Published by RainbowMango over 2 years ago
karmadactl
: Fixed namespace can not be customized issue. (#1826, @likakuli)karmadactl
: fixed can not taint while karmada control plane config is not located on default path. (#1837, @wuyingjun-lucky)karmada-controller-manager
: Fixed Job
status might be incorrectly marked as Completed
issue. (#2011, @Garrybest)karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2133, @AllenZMC)Published by kevin-wangzefeng over 2 years ago
A new component karmada-descheduler
was introduced, for rebalancing the scheduling decisions over time.
One example use case is: it helps evict pending replicas (Pods) from resource-starved clusters so that karmada-scheduler
can "reschedule" these replicas (Pods) to a cluster with sufficient resources.
For more details please refer to Descheduler user guide.
(Feature contributor: @Garrybest)
By leveraging the newly added spread-by-region
constraint, users are now able to deploy workloads aross regions
, e.g. people may want their workloads always running on different regions for HA purposes.
We also introduced two plugins to karmada-scheduler
, which add to accurate scheduling.
ClusterLocality
is a scoring plugin that favors clusters already assigned.SpreadConstraint
is a filter plugin that filters clusters as per spread constraints.(Feature contributors: @huone1, @gf457832386)
We are also in the progress of enhancing the multi-cluster failover mechanism. Part of the work has been included in this release.
For example:
--cluster-failure-threshold
) has been added to both karmada-controller-manager
and karmada-agent
, which specifies the cluster failure threshold (defaults to 30s). A cluster will be considered not-ready
only when it stays unhealthy longer than supposed.--failover-eviction-timeout
) has been added to karmada-controller-manager
, which specifies the grace period of eviction (defaults to 5 minutes). If a cluster stays not-ready
longer than supposed, the controller taints the cluster. (Note: The taint is essentially the eviction order and the implementation is planned for the next release.)(Feature contributors: @Garrybest, @dddddai)
The Aggregated API
was initially introduced in Release 1.0, which allows users to access clusters through Karmada by a single aggregated API endpoint. By leveraging this feature, we introduced a lot of interesting features to karmadactl
and kubectl-karmada
.
get
sub-command now supports clusters both in push
and pull
mode.# karmadactl get deployment -n default
NAME CLUSTER READY UP-TO-DATE AVAILABLE AGE ADOPTION
nginx member1 2/2 2 2 33h N
nginx member2 1/1 1 1 4m38s Y
podinfo member3 2/2 2 2 27h N
logs
command prints the container logs in a specific cluster.# ./karmadactl logs nginx-6799fc88d8-9mpxn -c nginx -C member1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
...
watch
and exec
commands to karmadactl
, in addition to get
and logs
. They all use the aggregated API.(Feature contributor: @lonelyCZ)
alpha
)The newly introduced karmada-search
caches resources in clusters and allows users to search for resources without directly touching real clusters.
# kubectl get --raw /apis/search.karmada.io/v1alpha1/search/cache/apis/apps/v1/deployments
{
"apiVersion": "v1",
"kind": "List",
"metadata": {},
"items": [{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"cluster.karmada.io/name": "member1",
},
}
},
]
}
The karmada-search
also supports syncing cached resources to backend stores like Elasticsearch or OpenSearch. By leveraging the search engine, you can perform full-text searches with all desired features, by field, and by indice; rank results by score, sort results by field, and aggregate results.
(Feature contributors: @huntsman-li, @liys87x)
Introduced InterpretStatus
for the Resource Interpreter Webhook
framework, which enables customized resource status collection.
Karmada can thereby learn how to collect status for your resources, especially custom resources. For example, a custom resource may have many status fields and only Karmada can collect only those you want.
Refer to [Customizing Resource Interpreter][https://github.com/karmada-io/karmada/blob/master/docs/userguide/customizing-resource-interpreter.md] for more details.
(Feature contributor: @XiShanYongYe-Chang)
Benefiting from the Kubernetes native APIs, Karmada can easily integrate the Kubernetes ecosystem. The following components are verified by the Karmada community:
Kyverno
: policy engine. Refer to working with kyverno for more details.Gatekeeper
: another policy engine. Refer to working with gatekeeper for more details.fluxcd
: GitOps tooling for helm chart. Refer to working with fluxcd for more details.(Feature contributors: @Poor12, @learner0810)
karmadactl
: Fixed the cluster joining failures in the case of legacy secrets. (@zgfh, #1306)karmadactl
: Fixed the issue that you cannot use the '-v 6' log level. (@zgfh, #1426)karmadactl
: Fixed the issue that the --namespace
flag of init
command did not work. (@sayaoailun, #1416)karmadactl
: Allowed namespaces to be customized. (@sayaoailun, #1449)karmadactl
: Fixed the init
failure due to data path not clean. (@prodanlabs, #1455)karmadactl
: Fixed the init
failure to read the KUBECONFIG environment variable. (@lonelyCZ, #1437)karmadactl
: Fixed the init
command failure to select the default release version. (@prodanlabs, #1456)karmadactl
: Fixed the issue that the karmada-system
namespace already exists when deploying karmada-agent
. (@hanweisen, #1604)karmadactl
: Fixed the issue that the karmada-controller-manager args did not honor customized namespaces.` (@prodanlabs, #1683)karmadactl
: Fixed a panic due to nil annotation when promoting
resources to Karmada.` (@duanmengkk, #1759)karmadactl
: Fixed the promote
command failure to migrate cluster-scoped resources. (@duanmengkk, #1766)karmadactl
: fixed the karmadactl taint failure while the karmada control plane config is not located in the default path. (@wuyingjun-lucky, #1825)helm-chart
: Fixed the karmada-agent installation failure due to the lack of permission. (@AllenZMC, #1457)helm-chart
: Fixed the issue that version constraints skip pre-releases. (@pigletfly, #1444)karmada-controller-manager
: Fixed the issue that ResourceBinding may hinder en-queue in the case of schedule failures. (@mrlihanbo, #1499)karmada-controller-manager
: Fixed the panic when the interpreter webhook returns nil patch. (@CharlesQQ, #1584)karmada-controller-manager
: Fixed the RB/CRB controller failure to aggregate status in the case of work condition update. (@mrlihanbo, #1513)karmada-aggregate-apiserver
: Fixed timeout issue when requesting cluster/proxy
with options -w
or logs -f
from karmadactl get
. (@XiShanYongYe-Chang, #1620)karmada-aggregate-apiserver
: Fixed exec failed: error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (@pangsq, #1632)karmada-controller-manager
: Introduced several flags to specify controller's concurrent capacities(--rate-limiter-base-delay
, --rate-limiter-max-delay
, --rate-limiter-qps
, --rate-limiter-bucket-size
). (@pigletfly, #1399)karmada-controller-manager
: The klog flags now have been grouped for better readability. (@RainbowMango, #1468)karmada-controller-manager
: Fixed the FullyApplied
condition of ResourceBinding/ClusterResourceBinding
mislabeling issue in the case of non-scheduling. (@huone1, #1512)karmada-controller-manager
: Added default AggregateStatus
webhook for DaemonSet
and StatefulSet
. (@Poor12, #1586)karmada-controller-manager
: OverridePolicy
with empty ResourceSelector
will be considered to match all resources just like nil. (@likakuli, #1706)karmada-controller-manager
: Introduced --failover-eviction-timeout
to specify the grace period of eviction. Tants(cluster.karmada.io/not-ready
or cluster.karmada.io/unreachable
) will be set on unhealthy clusters after the period. (@Garrybest, #1781)karmada-controller-manager/karmada-agent
: Introduced --cluster-failure-threshold
flag to specify cluster failure threshold. (@dddddai, #1829)karmada-scheduler
: Workloads can now be rescheduled after the cluster is unregistered. (@huone1, #1383)karmada-scheduler
: The klog flags now have been grouped for better readability. (@jameszhangyukun, #1491)karmada-scheduler
: Added a scoring plugin ClusterLocality
to favor clusters already requested. (@huone1, #1334)karmada-scheduler
: Introduced filter plugin SpreadConstraint
to filter clusters that do not meet the spread constraints. (@gf457832386, #1570)karmada-scheduler
: Supported spread constraints by region strategy. (@huone1, #1646)karmada-webhook
: Introduced --tls-cert-file-name
and --tls-private-key-file-name
flags to specify the server certificate and private key. (@mrlihanbo, #1464)karmada-agent
: The klog flags now have been grouped for better readability. (@lonelyCZ, #1389)karmada-agent
: Introduced several flags to specify controller's concurrent capacities(--rate-limiter-base-delay
, --rate-limiter-max-delay
, --rate-limiter-qps
, --rate-limiter-bucket-size
). (@dddddai, #1505)karmada-scheduler-estimator
: The klog flags now have been grouped for better readability. (@AllenZMC, #1493)karmadactl
: Introduced --context
flag to specify the context name to use. (@lonelyCZ, #1748)karmadactl
: Introduced --kube-image-mirror-country
and --kube-image-registry
flags to init
subcommand for Chinese mainland users. (@wuyingjun-lucky, #1764)karmadactl: Introduced
deinit sub-command to uninstall Karmada.
(@prodanlabs, #1337)alpine
has been promoted to v3.15.1
. (@RainbowMango, #1519)karmada-controller-manager
: The hpa
controller is disabled by default now. (@Poor12, #1580)karmada-aggregated-apiserver
: The deprecated flags --karmada-config
and --master
in v1.1 have been removed from the codebase. (@AllenZMC, #1834)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by RainbowMango over 2 years ago
karmadactl: fixed the
karmada-systemnamespace already existing issue when deploying
karmada-agent` issue. (#1608, @hanweisen)karmadactl: Fixed karmada-controller-manager args not honor customized namespace issue.
(#1689, @prodanlabs)karmada-controller-manager
: Fixed ResourceBinding maybe prevents en-queue in case of schedule failure. (#1507, @mrlihanbo)karmada-controller-manager
: Fixed the FullyApplied
condition of ResourceBinding/ClusterResourceBinding
mislabeling issue in case of non-scheduled. (#1517, @huone1)karmada-controller-manager
: Fixed RB/CRB controller can't aggregate status in case of work condition update issue. (#1523, @mrlihanbo)karmada-controller-manager
: Fixed panic in case of interpreter webhook returns nil patch. (#1592, @CharlesQQ)karmada-aggregate-apiserver
: Fixed timeout issue when request cluster/proxy
with options -w
or logs -f
from karmadactl get
. (#1630, @XiShanYongYe-Chang)karmada-aggregate-apiserver
: Fixed exec failed: error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (#1642, @XiShanYongYe-Chang)alpine
has been promoted to v3.15.1
. (#1583, @RainbowMango)Published by RainbowMango over 2 years ago
karmadactl: fixed the
karmada-systemnamespace already existing issue when deploying
karmada-agent` issue. (#1609, @hanweisen)karmadactl: Fixed karmada-controller-manager args not honor customized namespace issue.
(#1690, @prodanlabs)karmada-controller-manager
: Fixed the FullyApplied
condition of ResourceBinding/ClusterResourceBinding
mislabeling issue in case of non-scheduled. (#1518, @huone1)karmada-controller-manager
: Fixed RB/CRB controller can't aggregate status in case of work condition update issue. (#1524, @mrlihanbo)karmada-controller-manager
: Fixed panic in case of interpreter webhook returns nil patch. (#1591, @CharlesQQ)karmada-aggregate-apiserver
: Fixed timeout issue when request cluster/proxy
with options -w
or logs -f
from karmadactl get
. (#1631, @XiShanYongYe-Chang)karmada-aggregate-apiserver
: Fixed exec failed: error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (#1641, @pangsq)alpine
has been promoted to v3.15.1
. (#1582, @RainbowMango)Published by RainbowMango over 2 years ago
karmadactl
: Fixed --namespace
flag of init
command not work issue. (#1452, @sayaoailun)karmadactl
: Fixed init
failure due to data path not clean issue. (#1473, @prodanlabs)karmadactl
: Fixed init
can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)karmadactl
: Fixed init
can not select default release version issue. (#1496, @prodanlabs)helm chart
: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)Published by RainbowMango over 2 years ago
karmadactl
: Fixed --namespace
flag of init
command not work issue. (#1452, @sayaoailun)karmadactl
: Fixed init
failure due to data path not clean issue. (#1473, @prodanlabs)karmadactl
: Fixed init
can not select default release version issue. (#1495, @prodanlabs)karmadactl
: Fixed init
can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)helm chart
: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)karmada-controller-manager
: Fixed a bug where resource binding is not created occasionally. (#1384, @dddddai)Published by kevin-wangzefeng over 2 years ago
The newly introduced MultiClusterIngress API exposes HTTP and HTTPS routes that target multi-cluster services within the Karmada control plane. The specification of MultiClusterIngress
is compatible with Kubernetes Ingress.
Traffic routing is controlled by rules defined on the MultiClusterIngress resource, an MultiClusterIngress controller is responsible for fulfilling the ingress. The Multi-Cluster-Nginx Ingress Controller is one of the MultiClusterIngress controller implementations maintained by the community.
(Feature contributors: @GitHubxsy @XiShanYongYe-Chang)
The newly introduced FederatedResourceQuota provides constraints that limit total resource consumption per namespace across all clusters
. It can limit the number of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.
(Feature contributors: @RainbowMango @XiShanYongYe-Chang)
The default number of reconciling workers has been enlarged and configurable. A larger number of workers means higher responsiveness but heavier CPU and network load. The number of concurrent workers could be configured by the flags introduced to karmada-controller-manager
and karmada-agent
.
Flags introduced to karmada-controller-manager
:
Flags introduced to karmada-agent
:
(Feature contributor: @pigletfly)
Introduced AggregateStatus
support for the Resource Interpreter Webhook
framework, which enables customized resource status aggregating.
Introduced InterpreterOperationInterpretDependency
support for the Resource Interpreter Webhook
framework, which enables propagating workload's dependencies automatically.
Refer to Customizing Resource Interpreter for more details.
(Feature contributors: @iawia002 @mrlihanbo)
karmadactl
and kubectl-karmada
: Fixed that init
cannot update the APIService
. (@prodanlabs, #1207)karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake (should be Normal
but not Warning
). (@Garrybest, #1267)karmada-controller-manager
and karmada-agent
: Fixed that resync slows down reconciliation. (@Garrybest, #1265)karmada-controller-manager
/karmada-agent
: Fixed continually updating cluster status due to unordered apiEnablements. (@pigletfly, #1304)karmada-controller-manager
: Fixed that Replicas set by OverridePolicy will be reset by the ReviseReplica interpreterhook. (@likakuli, #1352)karmada-controller-manager
: Fixed that ResourceBinding couldn't be created in a corner case. (@dddddai, #1368)karmada-scheduler
: Fixed inaccuracy in requested resources in the case that pod limits are specified but requests are not. (@Garrybest, #1225)karmada-scheduler
: Fixed spreadconstraints[i].MaxGroups is invalidated in some scenarios. (@huone1, #1324)karmadactl
: Introduced --tls-min-version
flag to specify the minimum TLS version. (@carlory, #1278)karmadactl
: Improved the get
command to show more useful information. (@lonelyCZ, #1270)karmada-controller-manager
/karmada-agent
: Introduced --resync-period
flag to specify reflector resync period (defaults to 0, meaning no resync). (@Garrybest, #1261)karmada-controller-manager
: Introduced --metrics-bind-address
flag to specify the customized address for metrics. (@pigletfly, #1341)karmada-webhook
: Introduced --metrics-bind-address and --health-probe-bind-address flags. (@mrlihanbo, #1346)karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake (should be Normal
but not Warning
). (@Garrybest, #1267).spec.targetCluster
and spec.overriders
have been deprecated in favor of spec.overrideRules
. (@RainbowMango #1238)karmada-aggregate-apiserver
: Deprecated --master
and --karmada-config
flags. Please use --kubeconfig
instead. (@carlory, #1336)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by RainbowMango over 2 years ago
karmadactl
and kubectl-karmada
: Fixed init
can not update the APIService
issue. (#1207, @prodanlabs )karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake(should be Normal
but Warning
). (#1267, @Garrybest )karmada-controller-manager
and karmada-agent
: Fixed resync slow down reconcile issue. (#1265, @Garrybest )Published by kevin-wangzefeng almost 3 years ago
The newly introduced karmada-aggregated-apiserver
component aggregates all registered clusters and allows users to access member clusters through Karmada by the proxy endpoint, e.g.
- Retrieve `Node` from `member1`: /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes
- Retrieve `Pod` from `member2`: /apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api/v1/namespaces/default/pods
Please refer to user guide for more details.
(Feature contributor: @kevin-wangzefeng @GitHubxsy @XiShanYongYe-Chang @mrlihanbo @jrkeen @prodanlabs @carlory @RainbowMango)
Legacy workloads running in Kubernetes now can be promoted to Karmada smoothly without container restart.
In favor of promote
commands added to Karmada CLI, any kind of Kubernetes resources can be promoted to Karmada easily, e.g.
# Promote deployment(default/nginx) from cluster1 to Karmada
kubectl karmada promote deployment nginx -n default -c cluster1
(Feature contributor: @lonelyCZ @iawia002 @dddddai)
Benefiting from the Kubernetes native API support, Karmada can easily integrate the single cluster ecosystem for multi-cluster, multi-cloud purpose. The following components have been verified by the Karmada community:
argo-cd
: refer to working with argo-cd
Flux
: refer to propagating helm charts with flux
Istio
: refer to working with Istio
Filebeat
: refer to working with Filebeat
Submariner
: refer to working with Submariner
Velero
: refer to working with Velero
Prometheus
: refer to working with Prometheus
(Feature contributor: @lfbear @learner0810 @zirain @Rains6 @gy95 @XiShanYongYe-Chang )
By leverage of the new-introduced RuleWithCluster
fields to OverridePolicy
and ClusterOverridePolicy
, users are now able to define override policies with a single policy for specified workloads.
(Feature contributor: @iawia002 @lfbear @RainbowMango @lonelyCZ @jameszhangyukun )
Introduced init
command to Karmada CLI
. Users are now able to install Karmada by a single command.
Please refer to Installing Karmada for more details.
(Feature contributor: @prodanlabs @lonelyCZ @jrkeen )
Now all controllers provided by Karmada work as plug-ins. Users are now able to turn off any of them from the default enabled list.
See --controllers
flag of karmada-controller-manager
and karmada-agent
for more details.
(Feature contributor: @snowplayfire @iawia002 @jameszhangyukun )
Introduced ReviseReplica
support for the Resource Interpreter Webhook
framework, which enables scheduling all customized workloads just like Kubernetes native ones.
Refer to Resource Interpreter Webhook Proposal for more design details.
(Feature contributor: @iawia002)
karmada-controller-manager
: Fixed the issue that the annotation of resource template cannot be updated. (@mrlihanbo #1012)karmada-controller-manager
: Fixed the issue of generating binding reference key. (@JarHMJ #1003)karmada-controller-manager
: Fixed the inefficiency of en-queue failed task issue. (@Garrybest #1068)Karmada CLI
: Introduced --cluster-provider
flag to join
command to specify provider of joining cluster. (@2hangchen #1025)Karmada CLI
: Introduced taint
command to set taints for clusters. (@lonelyCZ #889)Karmada CLI
: The Applied
condition of Work
and Scheduled/FullyApplied
of ResourceBinding
are available for kubectl get
. (@lonelyCZ #1110)karmada-controller-manager
: The cluster discovery feature now supports v1beta1
of cluster-api
. (@iawia002 #1029)karmada-controller-manager
: The Job
's startTime
and completionTime
now available at resource template. (@Garrybest #1034)karmada-controller-manager
: introduced --controllers
flag to enable or disable controllers. (@snowplayfire #1083)karmada-controller-manager
: Support retain ownerReference
from observed objects. (@snowplayfire #1116)karmada-controller-manager
and karmada-agent
: Introduced cluster-cache-sync-timeout
flag to specify the time waiting for cache sync. (@snowplayfire #1112)karmada-scheduler-estimator
: Introduced /metrics
endpoint to emit metrics. (@Garrybest #1030)ApplyPolicy
and ScheduleBinding
events for resource template. (@mrlihanbo #1070)ReplicaSchedulingPolicy
API deprecated at v0.9.0 now has been removed in favor of ReplicaScheduling
of PropagationPolicy
. (@iawia002 #1161)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by RainbowMango almost 3 years ago
Published by kevin-wangzefeng almost 3 years ago
The newly introduced Resource Interpreter Webhook
framework allows users to implement their own CRD plugins that will be consulted at all parts of propagation process. With this feature, CRDs and CRs will be propagated just like Kubernetes native resources, which means all scheduling primitives also support custom resources. An example as well as some helpful utilities are provided to help users better understand how this framework works.
Refer to Proposal for more details.
(Feature contributor: @RainbowMango, @XiShanYongYe-Chang, @gy95)
Introduced dynamicWeight
primitive to PropagationPolicy
and ClusterPropagationPolicy
. With this feature, replicas could be divided by a dynamic weight list, and the weight of each cluster will be calculated based on the available replicas during scheduling.
This feature can balance the cluster's utilization significantly. (#841)
Introduced Job
schedule (divide) support. A Job
that desires many replicas now could be divided into many clusters just like Deployment
.
This feature makes it possible to run huge Jobs across small clusters. (#898)
(Feature contributor: @Garrybest )
After workloads (e.g. Deployments) are propagated to member clusters, users may also want to get the overall workload status across many clusters, especially the status of each pod
. In this release, a get
subcommand was introduced to the kubectl-karmada
. With this command, user are now able get all kinds of resources deployed in member clusters from the Karmada control plane.
For example (get deployment
and pods
across clusters):
$ kubectl karmada get deployment
NAME CLUSTER READY UP-TO-DATE AVAILABLE AGE ADOPTION
nginx member2 1/1 1 1 19m Y
nginx member1 1/1 1 1 19m Y
$ kubectl karmada get pods
NAME CLUSTER READY STATUS RESTARTS AGE
nginx-6799fc88d8-vzdvt member1 1/1 Running 0 31m
nginx-6799fc88d8-l55kk member2 1/1 Running 0 31m
(Feature contributor: @lfbear @QAQ-rookie)
resourcebinding.karmada.io/namespace
, resourcebinding.karmada.io/name
, clusterresourcebinding.karmada.io/name
) which were previously added on the Work object now have been moved to annotations. (@XiShanYongYe-Chang, #752)SyncFailed
and SyncSucceed
) to the Work object. (@wawa0210, #800)Scheduled
) to the ResourceBinding and ClusterResourceBinding
. (@dddddai, #823)CreateExecutionNamespaceFailed
and RemoveExecutionNamespaceFailed
) to the Cluster object. (@pigletfly, #749)workqueue_adds_total
, workqueue_depth
, workqueue_longest_running_processor_seconds
, workqueue_queue_duration_seconds_bucket
) for karmada-agent
and karmada-controller-manager
. (@Garrybest, #831)FullyApplied
) to the ResourceBinding and ClusterResourceBinding
. (@lonelyCZ, #825)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by kevin-wangzefeng about 3 years ago
Upgrading support
Users are now able to upgrade from the previous version smoothly. With the multiple version feature of CRD, objects with different schemas can be automatically converted between versions. Karmada uses the semantic versioning and will provide workarounds for inevitable breaking changes.
In this release, the ResourceBining
and ClusterResourceBinding
promote to v1alpha2
and the previous v1alpha1
version is still available for one more release. With the upgrading instruction, the previous version of Karmada can promote smoothly.
(Feature contributor: @RainbowMango )
Introduced karmada-scheduler-estimator to facilitate end-to-end scheduling accuracy
Karmada scheduler aims to assign workload to clusters according to constraints and available resources of each member cluster. The kube-scheduler
working on each cluster takes the responsibility to assign Pods to Nodes.
Even though Karmada has the capacity to reschedule failure workload between member clusters, but the community still commits lots of effort to improve the accuracy of the end-to-end scheduling.
The karmada-scheduler-estimator
is the effective assistant of karmada-scheduler
, it provides prediction-based scheduling decisions that can significantly improve the scheduling efficiency and avoid the wave of rescheduling among clusters. Note that this feature is implemented as a pluggable add-on. For the instructions please refer to scheduler estimator guideline.
(Feature contributor: @Garrybest )
Maintainability improvements
A bunch of significant maintainability improvements were added to this release, including:
Simplified Karmada installation with helm chart.
(Feature contributor: @algebra2k @jrkeen )
Provided metrics to observe scheduler status, the metrics API now served at /metrics
of karmada-scheduler
.
With these metrics, users are now able to evaluate the scheduler's performance and identify the bottlenecks.
(Feature contributor: @qianjun1993 )
Provided events to Karmada API objects as supplemental information to debug problems.
(Feature contributor: @pigletfly )
ResourceBinding
/ClusterResourceBinding
won't be deleted after associate PropagationPolicy
/ClusterPropagationPolicy
is removed and is still available until resource template
is removed.(@qianjun1993, #601)--leader-elect-resource-namespace
which is used to specify the namespace of election object to components karmada-controller-manager/
karmada-scheduler/
karmada-agent`. (@XiShanYongYe-Chang #698)ReplicaSchedulingPolicy
has been deprecated and will be removed from the following release. The feature now has been integrated into ReplicaScheduling.kubectl-karmada
commands as the extensions for kubectl
. (@XiShanYongYe-Chang #686)karmada-controller-manager
introduced a version
command to represent version information. (@RainbowMango #717 )karmada-scheduler
/karmada-webhook
/karmada-agent
/karmada-scheduler-estimator
introduced a version
command to represent version information. (@lonelyCZ #719 )Submariner
to connect the network between member clusters. (@XiShanYongYe-Chang #737 )karmada-scheduler
to monitor scheduler performance. (@qianjun1993 #747)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by kevin-wangzefeng about 3 years ago
Automatic cluster discovery with cluster-api
For users who are using cluster-api (sigs.k8s.io/cluster-api), Karmada is now able to automatically discover & join clusters when provisioned, and unjoin them in case of destroyed.
Note that this features is implemented as a built-in plugin. To enbale it, simply indicate the following to flags in karmada-controller-manager
config:
--cluster-api-kubeconfig string Path to the cluster-api management cluster kubeconfig file.
--cluster-api-context string Name of the cluster context in cluster-api management cluster kubeconfig file.
(Feature contributor: @XiShanYongYe-Chang )
Introduced CommandOverrider and ArgsOverrider to simplify commands customization per cluster
For multi-cluster applications, it's quite common to set different arguments when running on different clusters or environments.
In this release, two overrider plugins: CommandOverrider
and ArgsOverrider
are introduced, based on industry best practices. These two handy tools allow users to declare complex declarations and avoid configuration mistakes.
Workload types supported now are: Deployment
, ReplicaSet
, DaemonSet
, StatefulSet
and Pod
, more types including CRDs will be supported in later releases.
(Feature contributor: @lfbear @betaincao )
Better integration support with Kubernetes ecosystem
The Kubernetes native APIs support and patterns to run cloud-native applications of Karmada
make it quite easy to quickly integrate with other projects in the Kubernetes ecosystem.
In release, several useful features that will help Karmada
work seamlessly with other systems.
ResourceBinding
and ClusterResourceBinding
now supports present the applied
status. (@pigletfly #595)Job
, Service
, and Ingress
. (@mrlihanbo #609)cordon
and uncordon
commands to mark a cluster schedulable and un-schedulable. (#464, @algebra2k )--skipped-propagating-namespaces
flag to skip resources in certain namespaces from propagating. (#533, @pigletfly )Karmada
or member cluster's kube-apiserver. (#611, @Garrybest )
Mac OS
. (#538, @lfbear )Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by kevin-wangzefeng over 3 years ago
Support multi-cluster service discovery
In many cases, a Kubernetes user may want to split their deployments across multiple clusters, but still retain mutual dependencies between workloads running in those clusters.
Users are now able to export
and import
services between clusters with Multi-Cluster Service API (MCS-API). (@XiShanYongYe-Chang)
Support more precise cluster status management
Besides reporting cluster status, the cluster status controller
now also renews the lease
. The newly introduced cluster monitor
monitors the lease
and will mark cluster ready status to unknown
in case of cluster status controller
not working. (@Garrybest)
Support replica scheduling based on cluster resources
In some scenarios, users want to divide
the replicas in a deployment
to multiple clusters if a single cluster doesn't have sufficient resources.
Users now able to declare the replica scheduling preference by the new field ReplicaDivisionPreference
in PropagationPolicy
and ClusterPropagationPolicy
. (@qianjun1993)
Support more convenient APIs to divide replicas by weight list
Users now able to declare cluster weight by ReplicaDivisionPreference
in PropagationPolicy
and ClusterPropagationPolicy
, with the preference Weighted
, the scheduler will divide replicas according to the WeightPreference
. (@qianjun1993)
This feature is designed to replace the standalone ReplicaSchedulingPolicy
API in the future.
--karmada-context
flag to indicate the cluster context in karmada kubeconfig file. (#415, @mrlihanbo)--cluster-lease-duration
and --cluster-lease-renew-interval-fraction
flags to specify the lease expiration period and renew interval fraction. (#421, @pigletfly)--skipped-propagating-apis
flag to skip the resources from propagating. (#345, @pigletfly)hack/deploy-karmada.sh
and hack/deploy-karmada-agent.sh
scripts support install Karmada components on both Kind
clusters and standalone clusters. (#458, @lfbear)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
Published by kevin-wangzefeng over 3 years ago
Support syncing with member cluster behind proxy
In some scenarios where certain clusters may not be directly connected from the Internet, such as:
By setting proxy-url
in the kubeconfig
when registering member clusters, Karmada will talk to member clusters through indicated proxy. (#307, @liufen90)
Introduced ImageOverrider for simplifying image replacement
In most scenarios where clusters are running in different cloud or data centers, the workload requires a different image registry. ImageOverrider
is a handy tool to override images for a workload before they are propagated to clusters. (#370, @XiShanYongYe-Chang)
Support scheduling based on cluster taint toleration
Karmada-scheduler
now reflects taints on member clusters and tolerations defined in PropagationPolicy
and ClusterPropagationPolicy
when scheduling resources. (#320, @mrlihanbo)
Support scheduling based on cluster topology
Karmada-scheduler
now supports scheduling resources according to the topology information(cluster/provider/region/zone)
defined in cluster
objects. (#357, @mrlihanbo)
hack/remote-up-karmada.sh
to install Karmada on a specified Kubernetes as host. (#367, @lfbear)version
command to show the version it is built from. Try it on by command: # karmadactl version
. (#285, @algebra2k)ResourceBinding
objects for namespace-scoped resource template. (#315, @vincent-pli)leader-elect
command line flag to enable or disable leadership election. (#321, @pigletfly)Work
objects name now consist of the resource template's .metada.name
, .metada.kind
and .metadata.namespace
. (#359, @Garrybest)Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)