Bot releases are hidden (Show)
Published by derailed almost 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
New this release, we've added preliminary sanitizers for the following RBAC resources: clusterrole, clusterrolebinding, role and rolebinding. The sanitizers will now check if these resource are indeed in use on your clusters.
We've revamped the way excludes worked. Big thanks and credits goes to Dirk Jablonski for the push! So you can now excludes some sanitizers based not only on the resource name and type but also based on the sanitization codes. ie exclude all pod freds as long as they have missing probes (Code=102) but flag any other issues. This I think will make Popeye a bit more flexible.
NOTE: You will need to revamp your spinachYAML files as the format changed!!
Here is an example:
popeye:
# Excludes define rules to exempt resources from sanitization
excludes:
# NOTE!! excludes now use the full singular resource kind ie pod and not po or pods.
pod:
# Excludes all pods named fred unless the sanitizer reports any different codes from 102 or 106
- name: rx:fred
codes:
- 102
- 106
Please keep in mind the paint is still fresh here and I could have totally hosed some stuff in the process. If so reach out for your issues/prs button.
Thank you all for your great suggestions, fixes, patience and kindness!!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed almost 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
In this drop, we've cleaned up a few code duds and addressed a bit of debt.
Thanks to an awesome contribution by dardanel, Popeye can now report sanitization issues as Prometheus metrics. Thus, you will have the ability to run Popeye in cluster as a job and push sanitization metrics back to the prometheus mothership. How cool is that? As it stands these will just be reported as raw counts and thus you won't have sanitization details but you can leverage Prometheus AlertManager to trigger your clusters investigation based on these reports.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed about 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Maintenance release bugs and cleanup.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed about 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Maintenance release bugs and cleanup.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed about 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Oops! Broke the scorer ;(
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed about 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
I am super excited about this drop and hope you will be too! Lot's of changes and features but also more opportunities for breakage. So please proceed with caution and please do file issues so we can all gain from the improvements.
Until now Popeye did not really handle any kind of sanitizer run histories. We've added a --save
option that allows sanitizer runs to be persisted to disk.
# Perform a cluster blee sanitization and persists results to disk.
popeye -A --save
Some folks had requested a junit flavored output for integration with CI/CD tools like Jenkins. To this end, we've provided a new formatter to output sanitizer reports as Junit flavored XML.
In order to enable the report, use the following argument:
popeye -o junit
NOTE: This is an experimental feature and subject to change based on users feedback!
We've refactored the sanitizer report to now include sanitizer codes. Each report section have a different set of codes depending on the sanitization checks. For instance, code POP-106 No resource defined
will now be indicated in the report. We will document the various codes, their meanings and resolutions once we've got a chance to vet the changes and make sure we're all happy with the new reports!
On this note, and an interesting side effect, you can now change the code severity level in your spinach
config file. There has been some reports, voicing a need to change the message severity based on your cluster policies. That said, I would warn against it, as the end goal here is to come up with a set of standard best practices across all clusters. The reason we' ve decided to open this up a bit was so that we can zero in as a community for clusters best practices. So I will ask, that if you do feel the urge to modify a sanitizer code severity, you file an issue so that we can discuss as a group and come up with the best directives so we can all endup with a winner. This is a total backdoor for improving your clusters score without changing any manifests...
Here is a sample spinach.yml config to override a code severity:
# Severities: Ok: 0, Info: 1, Warn: 2, Error: 3
popeye:
codes:
206:
severity: 2 # Set severity level to Warn vs Info if No pod DisruptionBudget is set.
In this drop we've also added a few security rules as sanitizer checks. This is just the beginning of a long journey but you should start seeing a few security checks in your reports.
As a results Popeye will notify if the following conditions are true on your clusters:
default
ServiceAccountWe're going to be more active in this area in the next few drops so please let us know which checks might be most useful so we can prioritize accordingly.
In this release we've added a few new resources to the sanitization pass. Some checks are still primitive we will improve on that soon.
Sadly, we're are still having issues deploying Popeye as a snap ;( Though we're hopeful these will be resolved soon, we've decided to offer a brewed version of Popeye as an alternate for our Linux friends.
brew install derailed/popeye/popeye
Saving the best for last! As you might be aware K8s 1.6 release is going to remove some resource api group version in the schema. Cluster admins/operators are going to need to not only change their application manifests but also update their applications dependencies. This is going to most likely cause some disturbance in the force. No worries Popeye has your back!
In this drop, we've added some very basic checks for potential use of the deprecated APIs. Since Popeye looks at a live cluster and what is actually deployed and running, the sanitizers will alert you of potential deprecation problems before you update your entire Kubernetes cluster to 1.6.
Popeye sanitizers will warn you on deprecated resource api groups on the following:
NOTE! It is possible that Popeye might not cover 100% of the cases as Helm charts or operators implementation might bypass the basic checks Popeye is relying on to determine a resource api group version.
We hope you will find these features useful and timely in helping in the migration.
I think that's a wrap for this drop. Please be mindful that a lot of code changes happened here and some breakage might occur. Please help us zero in and file issues should you experience incorrect reports. Thank you!!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Popeye is designed to report sanitization on a live cluster. As such when a cluster is mainly idle, the over allocation report may yield false positives. To this end, we've added a --over-allocs
option to the CLI to opt-in over allocations reports. By default this option will be off, hence no over cpu/memory allocations will be reported. This now gives you an option to report allocation based on cluster load.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Bug and Maintenance release.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Added check for pbs. The sanitizer will report usage and possible misconfiguration if PodDiscruptionBudgets are available on the cluster.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
Bugs and clean up...
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
If you dig this tool, please make some noise on social! @kitesurfer
BREAKING CHANGE!
As of this release the spinach.yml format has changed slightly. There is now a new exludes
section that allows one to exclude any Kubernetes resources from the sanitizer run. A resource is identified by a resource kind and a fully qualified resource name ie namespace/resource_name
. For example a pod named fred-1234 in namespace blee FQN will be blee/fred-1234
. This provides for differentiating fred/p1
and blee/p1
. For cluster wide resources, FQN=name
. Exclude rules can have either a straight string match or a regular expression. In the later case the regular expression must be indicated using the rx:
prefix.
NOTE! Please thread carefully here with your regex as more resources than expected may get excluded from the report via a loose regex rule. When your cluster resources change, this could lead to rendering sanitization sub-optimal. Once in a while it might be a good idea to run Popeye Config less
to make sure you're trapping any new issues with your clusters...
Here is an example spinach file as it stands in this release:
popeye:
allocations:
cpu:
over: 200
under: 50
memory:
over: 200
under: 50
# New excludes section now provides for excluding any resources scanned by Poppeye.
excludes:
# Exclude any configmaps within namespace fred that ends with a version#
configmap:
- rx:fred*\.v\d+
# Exclude kube-system + any namespace the start with either kube or istio
namespace:
- kube-public
- rx:kube
- rx:istio
# Exclude node named n1 from the scan.
node:
- n1
# Exclude any pods that start with nginx or contains -telemetry
pod:
- rx:nginx
- rx:.*-telemetry
# Exclude any service containing -dash in their name.
service:
- rx:*-dash
# Node...
node:
limits:
cpu: 90
memory: 80
# Pod...
pod:
limits:
cpu: 80
memory: 75
restarts: 3
NOTE: Malformed regex issues will be surfaced in the logs! Please use
popeye version
for logs location.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
The exclude section of the yaml now supports regular expresions. In order to designate a regular expression matcher your exclude must start with rx:
. Here are some examples:
exclude:
# Exclude pod named blee.
- blee
# Exclude all pod name that start with nginx.
- rx:nginx
# Exclude all pod that contain -duh ie blee-duh and fred-duh.
- rx:.*-duh
NOTE: Malformed regex issues will be surfaced in the logs! Please use
popeye version
for logs location.
In my speed up excitements, I've spaced checking for clusters that don't currently support metrics. This yield to an npe ;(. This should now be resolved. Sorry about this waffle thin disruption in the force!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Added a caching layer to improve sanitization report generation. This is a first pass of many but looks like 2X improvement over previous release. Yeah!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Lost of work happened under the hood in this release. Mainly refactoring, bugs and cleanup items. If you notice any breakage from the previous release, please file an issue so we can improve Popeye. Thank you!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Bugs and cleanup!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Fixed up a few sanitizer messages.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Once again with feelings...
Dedicating this release, in honor of my beloved dog Nikita who passed away yesterday ;(
Added Sanitizer reports for the following resources:
Popeye will now scan for configuration and usage issues that may arise from these resources.
Ever wondered how much cluster capacity you actually need? Or which resource scaling may cause your cluster to surpass it's capacity? Fear not my friends! In this release, we introduce Capacitor
. We've added metrics monitoring to the sanitizer reports. Capacitor checks your resources (provided they are set!) for potential over/under allocation based on reported metrics. Additionally, Popeye's capacitor checks your HorizontalPodAutoscalers and pre-computes resource allocations based on max replicas. Thus you can be warned when there is a potential for your clusters to either reach or surpass their capacity.
Mind you, this is very much still experimental, so procceed with caution!
Added support for YAML and JSON output via -o
CLI parameter.
NOTE! Jurassic mode, though still in full effect, has been moved to
-o jurassic
As of this release, Popeye has been dockerized. You can now run Popeye directly on
your clusters either as a single shot or part of a cronjob. Please checkout the README and the k8s directory for more info about that.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Dedicating this release, in honor of my beloved dog Nikita who passed away yesterday ;(
Added Sanitizer reports for the following resources:
Popeye will now scan for configuration and usage issues that may arise from these resources.
Ever wondered how much cluster capacity you actually need? Or which resource scaling may cause your cluster to surpass it's capacity? Fear not my friends! In this release, we introduce Capacitor
. We've added metrics monitoring to the sanitizer reports. Capacitor checks your resources (provided they are set!) for potential over/under allocation based on reported metrics. Additionally, Popeye's capacitor checks your HorizontalPodAutoscalers and pre-computes resource allocations based on max replicas. Thus you can be warned when there is a potential for your clusters to either reach or surpass their capacity.
Mind you, this is very much still experimental, so procceed with caution!
Added support for YAML and JSON output via -o
CLI parameter.
NOTE! Jurassic mode, though still in full effect, has been moved to
-o jurassic
As of this release, Popeye has been dockerized. You can now run Popeye directly on
your clusters either as a single shot or part of a cronjob. Please checkout the README and the k8s directory for more info about that.
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0
Published by derailed over 5 years ago
Thank you so much for your support and suggestions to make Popeye better!!
Also if you dig this tool, please make some noise on social! @kitesurfer
Added Sanitizer reports for ConfigMap and Secrets. Popeye now scans your clusters for potential dead cm or secret resources and there associated data keys.
Added --jurassic|-j aka NeanderTerm mode, for terminals that don't support dazzling emojis and smashing colors.
Went thru a pretty significant refactor on this drop, so please be on the lookout for potential disturbance in the force and report any issues you may encounter. Thank you!!
© 2019 Imhotep Software LLC. All materials licensed under Apache v2.0