kube-router

Changelog

82410c14 Add support for BGP path prepending through node annotations (#476)

This release has a critical fix to network policy implementation and an enhancement to network policy semantics also included several minor fixes.

Thanks to contributions from @johanot @andor44 @lubinsz

Changelog

57f4eea2 Implemented the use of both namespaceSelector and podSelector in network policy peers (#475) (#479)
99341199 Fix nwplcy re-sync issue (#477) (#478)
3a09fdac enable build on Arm platform (#445)
17f92dea Health 1 (#463)
58da2d41 Fix for network policy connection refused issue (#461) (#471)
7c21815b Report delay metrics as seconds, not nanos (#465)

We are excited to release a new version of kube-router. Kube-router now can be used with other CNI's like Flannel/Weave to advertise service VIP's to routers in your network fabric.

Set below flag to false

--enable-cni                       Enable CNI plugin. Disable if you want to use kube-router features alongside another CNI plugin. (default true)

and enable one the flags as per your requirement.

--advertise-cluster-ip             Add Cluster IP of the service to the RIB so that it gets advertises to the BGP peers.
--advertise-external-ip            Add External IP of service to the RIB so that it gets advertised to the BGP peers.
--advertise-loadbalancer-ip        Add LoadbBalancer IP of service status as set by the LB provider to the RIB so that it gets advertised to the BGP peers.
--advertise-pod-cidr               Add Node's POD cidr to the RIB so that it gets advertised to the BGP peers. (default true)

This release also fixes a regression due to which DSR functionality was not usable.

Changelog

327a46d5 fix race condition issues with health checks (#460)
5c6a24d4 Fix NPE when performing cleanup() (#458)
1c7866cd Allow CNI plugin to be disabled (#443)
5e4ca292 set iBGP export policies only if its enabled (#453)
08095487 closes https://github.com/cloudnativelabs/kube-router/issues/413 (#457)
6887ce71 add proper message when reading pod CIDR from cni conf file (#450)
d63c23a5 proxy: Fix ineffassign error (#447)

This is a bug-fix release which has two critical bug fixes.

8aa43242 fixes a routing issue where a node missing route entries to talk to pods on other nodes in different subnets.

380a4763 fixes an issue where nodes exchanges routes for pod CIDR's immediately as and when a new node comes up.

Changelog

abfb705b services: correct check for inactive service endpoints (#430)
380a4763 update export polices onNodeUpdate so the routes are exchanged with new node immediatley (#441)
8aa43242 adds explicit check for existing tunnel to avoid partial matches (#442)
54210686 Print output of ip route command as string. (#439)
0538a2a9 perform clean-up of external ip from custom route table for external ip only if the table is not empty (#437)

This release v0.2.0-beta.5 is mainly a bug fix release and includes some critical bug fixes.

• Fixes issue where there is delay of 5 min (default sync period) before external peers get advertised with service VIP's
• On service delete, VIP's associated with service were not withdrawn ealirer.

With the above fixes, external BGP peers should see service VIP's advertised by the kube-router pods instantly and also on service delete routes are withdrwan.

• Also included are patches that control source IP used for the traffic send over tunnel interface. Always node IP is used as source IP address when sending traffic over IPIP tunnel interfaces

• Fixes regression in network policies where is namespace selector in network policy was not honoured

Thanks to @dlamotte for the fixes to handle cleanups of service VIP associated with kube-dummy-if when service are deleted.

Changelog

6d86656f fix wrong use of advertiseVIPs where withdrawVIPs is required (#436)
b0733cb5 update clusteripprefixset so that BGP export policies allow advertising the service VIP (#435)
725bff6b use node ip as source when accessing service VIP's from the node (#433)
359ab1d9 explicilty specify source IP to use when send traffic over tunnels (#428)
2f39f983 cleanup routing table 79 (external IPs) (#431)
0a9b164c Update kubeadm iptables cleanup command (#424)
09b2f13e fix the wrong lister used (#422)
05b702a9 [WIP] docs cleanup (#418)
dfca917c proxy: cleanup stale IPs on kube-dummy-if (#417)
e13b7714 Add DigitalOcean Sponsorship (#414)
27d51cdd add docs/index.md for mkdocs (#416)

Changelog

Ability to enable/disable node advertising its pod CIDR to external BGP peers (#408) - @murali-reddy
split routing controllers to smaller modules by function (#406) - @murali-reddy
break controller package to independent packages (#405) - @murali-reddy
move health, metrics to sepearate packages (#404) - @murali-reddy
update gobgp to d31262d which fixes critical bug with BGP graceful restarts (#403) - @andrewsykim

Full Commit History

89aef2c7 go-releaser: update binary path
c0441624 only push new docker image on release
682c4944 remove release task in travis CI (#409)
41332a18 Ability to enable/disable node advertising its pod CIDR to external BGP peers (#408)
23d43622 split routing controllers to smaller modules by function (#406)
05bec8b3 break controller package to independent packages (#405)
1a0bfa2d move health, metrics to sepearate packages (#404)
6eece2d7 update gobgp to d31262d for BGP graceful restart fixes (#403)

Kube-router v0.2.0-beta.3 release is mainly follow-up release to the internal code refactoring done in v0.2.0-beta.2 and v0.2.0-beta.1

Breaking Change

Kube-router internal started using SharedInformerFactory which has a resync period. When resync period value great that 0 is specified, informer in go-client replays all the cached object, resulting in unnessary updates and processing. Since kube-router controllers already has periodc full sync periodic replay of objects is not required. Hence the flag --config-sync-period is deprecated.

Major Changes

af9875d4777e51fd8e595f67858122ca6de98f29 Replace glide depenedency management with dep
71d16bf4d6d68054bbe3417b68019a606a7e227b Code is restructured to confirm to standard go projects format
041c05570abb106af996ef373bbd2754dfc2eaa9 Optimizations to improve the periodic sync's done by the controllers
a1ecedf8022a71e095e76d8814a05e56971e8a8b Some more optimization to perform full-sync in the controllers only when needed
8c746b2cbc8026717d534236358033007fd7c9b0 Make GoBGP server listen only on node IP

Changelog

• only sync internal peers for OnAdd or OnDelete events
• ignore update events for endpoints used for leader election
• setting annotation kube-router.io/service.local on a Service should result in the same behavior as setting externalTrafficPolicy=Local for route advertisement

Full Commit History

945a13cd skip node OnUpdate events (#391)
6ba2e947 ignore update events for endpoints used for leader election (#390)
dd5d2faf Osxcompat 1 (#377)
28f10ff2 add local endpoints check if annotation kube-router.io/service.local is set (#388)

Changelog

• Add service/endpoints handlers so BGP routes are added/removed as soon as service/endpoints updates occur (@andrewsykim)
• Networkpolicies should account for pods that are not given a pod IP yet (@xanonid)
• Refactor to use shared informers (@andrewsykim)
• Fix bug where default ASN is used for iBGP peering when cluster ASN is set (@andrewsykim)
• Support advertising service.Status.LoadBalancer.Ingress IPs with flag --advertise-loadbalancer-ip (@jjo)
• Add delay for AWS EC2 operations and only make API calls if kube-router has the necessary IAM roles (@murali-reddy)
• Add unit tests Test_addExportPolicies (@andrewsykim)
• Allow advertise pod CIDR to be set using a node annotation kube-router.io/pod-cidr, this will override the pod CIDR specified in node.Spec.PodCIDR(@andrewsykim)
• Withdraw VIP routes if service.Spec.ExternalTrafficPolicy=Local and a node does not contain healthy endpoints for a service (@TvL2386)

Full Change History

1fcf5db2 add CHANGELOG (#387)
ab08c311 add service/endpoint event handler for routes controller (#384)
e94bf3df Fix networkpolicies if there are unscheduled pod, log errors (#378) (#379)
3763b200 refactor to use shared informers (#373)
ed0dc390 [jjo] added network_services_controller ginkgo testing (#348)
4fd5cb81 update command line options documentation (#372)
ed341871 Fixed the /var/run/docker.sock permissions in the DSR example (#368)
398ba8bd Fixed a typo (#367)
2a4ffd67 fix bug where default ASN is used for iBGP peering when cluster ASN (#362)
5436a731 glide update, to get pull latest dependencies (#365)
559304fe fix --advertise-loadbalancer-ips for LoadBalancer w/o IPs (#355)
696d7014 clear state of global service/endpoint watcher in Test_addExportPolicies (#353)
bf5c11ff [jjo] support advertising status.loadBalancer.ingress IPs via flag (#354)
35482c51 add 1s deley between ec2 API calls, and check if node has ec2 IAM (#351)
f708b14e fix UT test case (#352)
6a8e707c allow pod CIDR to be set using node annotations (#345)
25923bea Test_addExportPolicies (#333)
035a9a81 withdraw ClusterIP bgp route for externalTrafficPolicy=Local when there are no local endpoints (#347)
3956ad70 Update README.md
acb3dd45 use constants for all annotations (#346)
faead6db Dockefile: use ADD instead of COPY and rearrange (#340)

We are excited to bring new release with awesome new features and bug-fixes. Thanks to all users who provided valuable feedback and reported issues.

New features

• Kube-router now has support for BGP route-reflector functionality. Big shout out to @ut0mt8 for contributing route reflector feature. Read the docs how you can use route-reflector functionality.
• Kube-router now has support for HostPort support. Please read the docs to use HostPort functionality.
• Kube-router now complements the existing support for externalTrafficPolicy=Local with node advertising the service VIP's to BGP peer routers only if it has atlease one endpoint correspondng to service.

Changelog

9b6bf95 limit disableSrcDstCheck performed on OnNodeUpdate() (#327)
d72c410 close docker client after use (#343)
d7d0223 [WIP] support for Hostport (#335)
c7ce398 [jjo] add arm cross-compile support (#336)
4902a0c route advertisement should account for services using externalTrafficPolicy=Local (#334)
4e768b1 Feature Route Reflector (#325)
df6b375 AWS: perform disable src-dst check only if node has necessary IAM. Also introduce delay (#319)
31402a5 fix Pod Toolbox documentation issue (#318)
f68372e Update network_policy_controller.go (#315)
8c21111 Adding --version / -V support to print version information (#312)
2147b09 Dockerfile: Updated to alpine:3.7. (#306)

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

We are happy to announce the second release candidate for kube-router 0.1.0

In this release we have focused on introducing instrumentation to monitor metrics and health in kube-router for ease of operations, some minor bug fixes, adjustments to the logging output as well as introduced the BGP graceful restart function to enable zero downtime upgrades in supported environments.

Action required

If you previously used the metrics on port 8080 and wish to keep using this port you must set it at kube-router start using --metrics-port as the default now is 0, disabled

Logging

A lot of the output from kube-router has had logging levels added to prevent spamming the logs with normal operations and debug output. To run kube-router in verbose mode for debugging please add --v=3 to the start options.

Metrics

We have added a sample Grafana dashboard & the possibility to change what port and path the metrics are exposed at. You can read more about the functionality here

Health checks

kube-router now comes with togglable health endpoint to monitor the status of your kube-router deployment. Read more about the functionality and how to enable it here

Local services

Support for service.spec externalTrafficPolicy=Local has been added. It takes precedence over kube-router.io/service.local annotation. Read more about externalTrafficPolicy here

BGP graceful restart

Support has been added in this version for graceful BGP restarts. To enable it start kube-router with --bgp-graceful-restart. When peering with global peers make sure they support this mode before enabling

Notable Bug Fixes

• A regression was introduced in RC1 where GoBGP would panic when adding nodes in some cases. This is fixed in this version by #296
• Conntrack cleanup would log errors when cleaning up old conntrack recods if 0 flows was present, fixed by #305

Changelog

a466ace4 Nsc conntrack fix (#305)
e25c1741 support service.spec externalTrafficPolicy=Local. Takes precedence over kube-router.io/service.local annotation. (#303)
617c7736 code cleanup (#301)
33698908 reverted rollingUpdate strategy (#300)
a480a518 Merge pull request #298 from roffe/healthcheck
daefc818 updated help text
b1f34049 fixed bool values for if http health is enabled
38f45a4a chnged max unavail to 1 according to #171
deba3fa2 mutex on stats
311d7265 cleanup health checks daemonsets
bb49531d liveness checks
2c3510d0 Docs
23f43135 Updated daemonsets to include health checks
258cf2c1 health docs
f38dc79d Docs for healthz
f2da4459 added startup delay before healthchecks starts
cf7c66ee Adde flag so health controller knows if HTTP is requested or not
7066d4dd change so no http listener starts if port is not in valid range
e53aef28 more work on healthchecks
77cb340c missing bracket
4f8f83d6 metrics controller ticker
12aec998 added heartbeats
3eb5461f adding output messages
3469cc18 fixed healthport var
e6b01d54 added health controller
f1cb675d Update MAINTAINER.md
193776c5 prevent calling gobgp AddNeighbour call before GoBGP server is properly started (#296)
f3e7aced Metrics + Logging update (#294)
1492f0b2 small update to the metrics documentation
60f67d89 Metrics updates (#292)
edda2b14 Added more metrics and small bugfix + more docs (#291)
e36759e0 Documentation updates (#289)
c8c19e4a Added cli config for changing prometheus port & path (#288)
42f71777 Updated generic cluster docs & daemonset (#284)
8fe06539 Small docs fixes for setting up a generic cluster (#283)
1edee933 Documentation how to install kube-router on generic clusters (#281)
e51dfbff support node local service without SNAT, so the pod can see client origin IP if the request from external IP (#277)
aeaa7034 Fix typo in document (#276)
78588fc3 tunnel interface names cannot be longer than 15 characters (#274)
5155c8da Update README.md
862b4101 Make kubeadm install docs clearer (#270)
c07df49f unit tests for syncInternalPeers() (#268)
0b739e4d project status update (#269)

Automated with GoReleaser
Built with go version go1.9.3 linux/amd64

NOTE: We have found a regression in this release. We are working on fixing the issue in the next release. Please use v.0.0.20 release artifacts.

We are glad to release first release candidate for Beta release. Since first version of the project, we spent more than 8 months to stabilizing the code and fine tuning the solution as per the users feedback. Thanks to all users we are instrumental in shaping the project.

Changelog

50dcc32d Unit Tests for Node Update Events (#265)
4eca430b route controller unit tests (#264)
542680c6 skip build/push if user is not cloudnativelabs since it cannot access dockerhub creds (#263)
e27847f8 package utils unit tests and CI fixes (#261)
f8221099 Prevent slice representing ipset growing forver resulting in excessive memory usage (#260)
94a2ec7e Flush conntrack entry when UDP service endpoint is deleted (#259)
8ce5e4fe Fix potential issues in DSR due to Golang limitations in switching network namespaces and add verbose logging (#258)
1e1f8b4e Update README.md
d473f83a Update README.md
76ea8c01 fix TCP vs 6 string comparision in publishMetrics (#257)
22f05e99 service prometheus metrics (#255)
d8764411 Update README.md

Automated with GoReleaser
Built with go version go1.8.5 linux/amd64

Changelog

e5bd063e maintainers
d13339c6 IPVS sync services performnce improvement (#253)
38e30821 take-out kube-proxy IPVS status and add DSR k8s 1.9 has IPVS Beta, but hey Kube-router has DSR :) now
c4830aa4 enable Pprof with option --enable-pprof (#245)
fcd1a1aa Enable support for eBGP Multihop for external peers (#244)
5323b4b8 Fix --hairpin-mode=true (#243)
628111c7 so long gitter
de3e178b Update README.md
2685367a add --enable-ibgp flag (#227)
bca42210 --bgp-graceful-restart flag should apply for global peers (#226)
7c9db2c7 sync with iBGP peers only after RIB is polulated first
0dabd7cf Support BGP Graceful Restart (#220)
97fb3423 explicilty create kube-bridge interface. Earlier when pod is launched on the node (#225)
75bdfa2a Fix --nodeport-bind-all-ip option so that it works (#217)
cc2cad3a Update kubeadm.md (#216)

Automated with GoReleaser
Built with go version go1.8.5 linux/amd64

Changelog

28c5dd20 use table id instead of table name for custom routing tables (#215)
5783c30f Fix typo in doc (#212)

Automated with GoReleaser
Built with go version go1.8.5 linux/amd64

We are very excited to bring the new release of Kube-router packed with features.

• Support for external IP. Service proxy in kube-router now supports external IP's configured on the service.
• Support for advertising external IP's to the configured BGP peers
• Support for DSR (Direct Server Return).

Please read the blog to understand how these building blocks can be used for highly available and scalable ingress for bare-metal clusters

• Support for egress network policies
• Support for CIDR in network policies
• Support to change IPVS scheduling method through annotations

Changelog

1a647b69 updating documentation
b1800948 set priority for custom routing tables so that, FWMARK packets are delivered locally and for the rest of the traffic policy routing sets endpoints (dev kube-bridge) as next hop for external IP's
5f425ebb Fixes direct server return on exteranl IP's (#210)
6a3fadae Direct Server Return Fixes (#209)
77f45e35 Use service annotations to choose IPVS scheduling method (#207)
6d432681 support for direct server return (#204)
4ca0afa8 Support for advertising service external IP to be configured BGP peers (#203)
ba7697b7 Support for Services with externalIPs (#201)
84741b6a network policy: use 'addrtype' module and --src-type=LOCAL to match local traffic
665e6676 Fix to avoid re-adding existing BGP export policy, and policy assignment (#200)
50173e3b Support for 32 bit ASN numbers specifically the range 4200000000 - 4294967294 as specified by RFC 6996 (#196)
9eb1848d Network policy: always permit traffic from local node to the pods irrespective of network policies (#195)
ea459a32 Merge pull request #193 from cloudnativelabs/ntwk-policy-cidr-selector
928deedf Support CIDR in network policy ingress and egress rules
c3e8c111 Fix missing quotes in upgrading examples
2b910b2a Merge pull request #186 from cloudnativelabs/103-egress-network-policy
45cb0873 Network policy egress support
7c489a6e Update DaemonSet manifests (#188)
aa48cf2d gofmt (#187)
492cd1b4 Merge pull request #184 from cloudnativelabs/178-client-go-5
0010425b bump up client-go to release-5.0 and vendor kubernetes/api

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

This release brings with it a number of fixes for regressions introduced in the last release. Users are encouraged to upgrade as soon as possible, especially if you experienced issues with kube-router recently. A big thanks to @hjdr4 and @dimm0 for reporting issues and helping to test our fixes. Also a shoutout to our latest contributor @ryarnyah for ipset contributions.

BREAKING CHANGES
Please take some time to read the shiny new upgrade guide before proceeding. This release includes changes to the CLI flags and Annotations used for hairpin-mode and BGP peering.

Changelog

342ea5a Prevent masquerading pod -> NodeIP traffic (#174) critical bug
5f58cd2 Fix conditional so only matched pods are firewalled (#183) critical bug
a5a446a Use forward domain notation for annotations breaking change
3debbfa Cleanup ipset create function + restore (#177)
c125778 Fix make vagrant-image-update with rsynced folder
c714a80 CI: Make travis build PR# branches
c8902cc Add docs to cover upgrading kube-router (#179)
2ac6861 Documentation: Fix BGP peer example
fc86d2e Merge pull request #169 from ryarnyah/fix/clean-ipset
3431a18 Delete ipset dependency + delete unused ipsets
d6a15ec Fix link to developer's guide
e19f2a6 BGP peer password auth, consistent configurations (#164)

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

We're happy to say that kube-router is picking up steam! Big thanks to all the new users who are taking it for a spin and reporting bugs and showing us how they use kube-router's unique features. This release is a real doozy, and we look forward to the imminent beta release!

Notable Bug Fixes

d433fa5 Fix IPVS persistence flag (#160)
62900c7 NodePort service listens on all ip/interfaces in case of kube-proxy. Currently kube-router listens only on node IP. This fix introduces flag nodeport-bindon-all-ip with which you can have kube-proxy like behaviour. If not specified only nodeIP will be open for connections.

Enhancements

d6ea740 Dynamic gobgpd listener config, fix leaked listener (#156)
ca97d0d Adding ability to disable IP-in-IP tunnelining for cross node pod-to-pod connectivity where nodes are in different subnet. With tunneling disabled its expected that default gateway has learned the pod CIDR's allocated for all the nodes and can route the pod-to-pod traffic across nodes in different subnets

Other Commits

6ea240f Fix image refered in kubeadm daemonsets to release/latesr versions
f6a5e23 fixing gofmt, go_vet, gocyclo, golint errors

DaemonSet Upgrade Steps

image: cloudnativelabs/kube-router[:latest] + imagePullPolicy: Always

These are the default options in our manifests. In this scenario you just need to delete the pods so they are recreated. This will not interrupt cluster services/traffic.

kubectl -n kube-system delete pods -l k8s-app=kube-router

image: cloudnativelabs/kube-router:vX.X.X

If your image option is pinned to a specific version, you can upgrade using kubectl set image.

kubectl -n kube-system set image ds/kube-router kube-router=cloudnativelabs/kube-router:v0.0.16
kubectl -n kube-system delete pods -l k8s-app=kube-router

image: cloudnativelabs/kube-router-git:master

Already running the latest version huh? We like your style.

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

This patch release has critical fix that affects distros with strict reverse path filtering. When nodes span across multiple subnets, Kube-router uses ip-in-ip tunneling. Fix in this release ensures traffic originates and leaves on same tunnel interface.

DaemonSet Upgrade Steps

image: cloudnativelabs/kube-router[:latest]

imagePullPolicy: Always

These are the default options in our manifests. In this scenario you just need to delete the pods so they are recreated. This will not interrupt cluster services/traffic.

kubectl -n kube-system delete pods -l k8s-app=kube-router

image: cloudnativelabs/kube-router:vX.X.X

If your image option is pinned to a specific version, you can upgrade using kubectl set image.

kubectl -n kube-system set image ds/kube-router kube-router=cloudnativelabs/kube-router:v0.0.15
kubectl -n kube-system delete pods -l k8s-app=kube-router

Changelog

c9c6107 Merge pull request #143 from cloudnativelabs/pbr_tunnels
7b9eedc Setup policy-based routing so that traffic arriving on a tunnel interface leaves on same tunnel interface irrespective of rp_filter value
22330ab Merge pull request #146 from cloudnativelabs/goreportcard
ed935a4 fix gofmt, go_vet, misspells
b171f22 Update README.md

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

Changelog

2cd54fc Merge pull request #142 from cloudnativelabs/br_netfilter
ff079d2 Enable netfilter for bridge, requirement for all CNI that use bridge https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64

Changelog

5951f55 Adding architecture details
89105e8 Merge pull request #134 from Thoro/mac-build-environment
bcb5622 Merge pull request #133 from Thoro/consolidate-stop-channel
cb0888b Add build-image folder, which allows to build kube-router on mac os
9bd4bb8 Consolidate multiple stop channels into one
72fd0d2 re-organize documentation
547b17d Merge pull request #131 from cloudnativelabs/112-ntwk-plcy
1af19cb NetworkPolicy: Fix e2e test failures

Automated with GoReleaser
Built with go version go1.8.3 linux/amd64