popeye - v0.6.0

Published by derailed almost 5 years ago

Release v0.6.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Popeye's got your RBAC!

New this release, we've added preliminary sanitizers for the following RBAC resources: clusterrole, clusterrolebinding, role and rolebinding. The sanitizers will now check if these resource are indeed in use on your clusters.

Excludes are OUT??

We've revamped the way excludes worked. Big thanks and credits goes to Dirk Jablonski for the push! So you can now excludes some sanitizers based not only on the resource name and type but also based on the sanitization codes. ie exclude all pod freds as long as they have missing probes (Code=102) but flag any other issues. This I think will make Popeye a bit more flexible.

NOTE: You will need to revamp your spinachYAML files as the format changed!!

Here is an example:

popeye:
  # Excludes define rules to exempt resources from sanitization
  excludes:
    # NOTE!! excludes now use the full singular resource kind ie pod and not po or pods.
    pod:
      # Excludes all pods named fred unless the sanitizer reports any different codes from 102 or 106
      - name: rx:fred
        codes:
        - 102
        - 106

Please keep in mind the paint is still fresh here and I could have totally hosed some stuff in the process. If so reach out for your issues/prs button.

Thank you all for your great suggestions, fixes, patience and kindness!!

Resolved Bugs

popeye - v0.5.0

Published by derailed almost 5 years ago

Release v0.5.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

In this drop, we've cleaned up a few code duds and addressed a bit of debt.

Prometheus Report

Thanks to an awesome contribution by dardanel, Popeye can now report sanitization issues as Prometheus metrics. Thus, you will have the ability to run Popeye in cluster as a job and push sanitization metrics back to the prometheus mothership. How cool is that? As it stands these will just be reported as raw counts and thus you won't have sanitization details but you can leverage Prometheus AlertManager to trigger your clusters investigation based on these reports.

Resolved Bugs

Issue #58 Thank you @renan!!

popeye - v0.4.3

Published by derailed about 5 years ago

Release v0.4.3

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Maintenance release bugs and cleanup.

Resolved Bugs

Wrong Popeye version in Docker builds Issue #48
Incorrect ServiceAccount reporting Issue #49

popeye - v0.4.2

Published by derailed about 5 years ago

Release v0.4.2

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Maintenance release bugs and cleanup.

Resolved Bugs

Issue #44

popeye - v0.4.1

Published by derailed about 5 years ago

Release v0.4.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Oops! Broke the scorer ;(

Resolved Bugs

popeye - v0.4.0

Published by derailed about 5 years ago

Release v0.4.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

I am super excited about this drop and hope you will be too! Lot's of changes and features but also more opportunities for breakage. So please proceed with caution and please do file issues so we can all gain from the improvements.

Spinach Boosts Memory?

Until now Popeye did not really handle any kind of sanitizer run histories. We've added a --save option that allows sanitizer runs to be persisted to disk.

# Perform a cluster blee sanitization and persists results to disk.
popeye -A  --save

Junit Output

Some folks had requested a junit flavored output for integration with CI/CD tools like Jenkins. To this end, we've provided a new formatter to output sanitizer reports as Junit flavored XML.

In order to enable the report, use the following argument:

popeye -o junit

NOTE: This is an experimental feature and subject to change based on users feedback!

Codes

We've refactored the sanitizer report to now include sanitizer codes. Each report section have a different set of codes depending on the sanitization checks. For instance, code POP-106 No resource defined will now be indicated in the report. We will document the various codes, their meanings and resolutions once we've got a chance to vet the changes and make sure we're all happy with the new reports!

On this note, and an interesting side effect, you can now change the code severity level in your spinach config file. There has been some reports, voicing a need to change the message severity based on your cluster policies. That said, I would warn against it, as the end goal here is to come up with a set of standard best practices across all clusters. The reason we' ve decided to open this up a bit was so that we can zero in as a community for clusters best practices. So I will ask, that if you do feel the urge to modify a sanitizer code severity, you file an issue so that we can discuss as a group and come up with the best directives so we can all endup with a winner. This is a total backdoor for improving your clusters score without changing any manifests...

Here is a sample spinach.yml config to override a code severity:

# Severities: Ok: 0, Info: 1, Warn: 2, Error: 3
popeye:
  codes:
    206:
     severity: 2 # Set severity level to Warn vs Info if No pod DisruptionBudget is set.

Security Now!

In this drop we've also added a few security rules as sanitizer checks. This is just the beginning of a long journey but you should start seeing a few security checks in your reports.

As a results Popeye will notify if the following conditions are true on your clusters:

Running Pods using the default ServiceAccount
Running containers as root
Warning about mounting API server certs on pods.

We're going to be more active in this area in the next few drops so please let us know which checks might be most useful so we can prioritize accordingly.

Mo' Resources

In this release we've added a few new resources to the sanitization pass. Some checks are still primitive we will improve on that soon.

DaemonSet
ReplicaSet
Ingress
PodSecurityPolicy
NetworkPolicy

Linux Brewed!

Sadly, we're are still having issues deploying Popeye as a snap ;( Though we're hopeful these will be resolved soon, we've decided to offer a brewed version of Popeye as an alternate for our Linux friends.

brew install derailed/popeye/popeye

1.6 Deprecations

Saving the best for last! As you might be aware K8s 1.6 release is going to remove some resource api group version in the schema. Cluster admins/operators are going to need to not only change their application manifests but also update their applications dependencies. This is going to most likely cause some disturbance in the force. No worries Popeye has your back!

In this drop, we've added some very basic checks for potential use of the deprecated APIs. Since Popeye looks at a live cluster and what is actually deployed and running, the sanitizers will alert you of potential deprecation problems before you update your entire Kubernetes cluster to 1.6.

Popeye sanitizers will warn you on deprecated resource api groups on the following:

extensions/v1beta1 or apps/v1beta1 or apps/v1beta2 for DaemonSet, Deployment, StatefulSet, ReplicaSet
extensions/v1beta1.Ingress
extensions/v1beta1.PodSecurityPolicy
extensions/v1beta1.NetworkPolicy

NOTE! It is possible that Popeye might not cover 100% of the cases as Helm charts or operators implementation might bypass the basic checks Popeye is relying on to determine a resource api group version.

We hope you will find these features useful and timely in helping in the migration.

I think that's a wrap for this drop. Please be mindful that a lot of code changes happened here and some breakage might occur. Please help us zero in and file issues should you experience incorrect reports. Thank you!!

Resolved Bugs

popeye - v0.3.13

Published by derailed over 5 years ago

Release v0.3.13

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Add over-allocs flag

Popeye is designed to report sanitization on a live cluster. As such when a cluster is mainly idle, the over allocation report may yield false positives. To this end, we've added a --over-allocs option to the CLI to opt-in over allocations reports. By default this option will be off, hence no over cpu/memory allocations will be reported. This now gives you an option to report allocation based on cluster load.

Resolved Bugs

Issue #39

popeye - v0.3.12

Published by derailed over 5 years ago

Release v0.3.12

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Bug and Maintenance release.

Resolved Bugs

Issue #38

popeye - v0.3.11

Published by derailed over 5 years ago

Release v0.3.11

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Resolved Bugs

Issue #37

popeye - v0.3.10

Published by derailed over 5 years ago

Release v0.3.10

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

PodDisruptionBugdet

Added check for pbs. The sanitizer will report usage and possible misconfiguration if PodDiscruptionBudgets are available on the cluster.

Resolved Bugs

popeye - v0.3.8

Published by derailed over 5 years ago

Release v0.3.8

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Bugs and clean up...

Resolved Bugs

Issue #33

popeye - v0.3.7

Published by derailed over 5 years ago

Release v0.3.7

Notes

Thank you so much for your support and suggestions to make Popeye better!!

If you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Spinach Config Reloaded!

BREAKING CHANGE!

As of this release the spinach.yml format has changed slightly. There is now a new exludes section that allows one to exclude any Kubernetes resources from the sanitizer run. A resource is identified by a resource kind and a fully qualified resource name ie namespace/resource_name. For example a pod named fred-1234 in namespace blee FQN will be blee/fred-1234. This provides for differentiating fred/p1 and blee/p1. For cluster wide resources, FQN=name. Exclude rules can have either a straight string match or a regular expression. In the later case the regular expression must be indicated using the rx: prefix.

NOTE! Please thread carefully here with your regex as more resources than expected may get excluded from the report via a loose regex rule. When your cluster resources change, this could lead to rendering sanitization sub-optimal. Once in a while it might be a good idea to run Popeye Config less to make sure you're trapping any new issues with your clusters...

Here is an example spinach file as it stands in this release:

popeye:
  allocations:
    cpu:
      over: 200
      under: 50
    memory:
      over: 200
      under: 50

  # New excludes section now provides for excluding any resources scanned by Poppeye.
  excludes:
    # Exclude any configmaps within namespace fred that ends with a version#
    configmap:
      - rx:fred*\.v\d+
    # Exclude kube-system + any namespace the start with either kube or istio
    namespace:
      - kube-public
      - rx:kube
      - rx:istio
    # Exclude node named n1 from the scan.
    node:
      - n1
    # Exclude any pods that start with nginx or contains -telemetry
    pod:
      - rx:nginx
      - rx:.*-telemetry
    # Exclude any service containing -dash in their name.
    service:
      - rx:*-dash

  # Node...
  node:
    limits:
      cpu:    90
      memory: 80

  # Pod...
  pod:
    limits:
      cpu:    80
      memory: 75
    restarts: 3

NOTE: Malformed regex issues will be surfaced in the logs! Please use popeye version for logs location.

Resolved Bugs

popeye - v0.3.6

Published by derailed over 5 years ago

Release v0.3.6

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Spinach Exclude

The exclude section of the yaml now supports regular expresions. In order to designate a regular expression matcher your exclude must start with rx:. Here are some examples:

exclude:
  # Exclude pod named blee.
  - blee
  # Exclude all pod name that start with nginx.
  - rx:nginx
  # Exclude all pod that contain -duh ie blee-duh and fred-duh.
  - rx:.*-duh

NOTE: Malformed regex issues will be surfaced in the logs! Please use popeye version for logs location.

Performance part Duh

In my speed up excitements, I've spaced checking for clusters that don't currently support metrics. This yield to an npe ;(. This should now be resolved. Sorry about this waffle thin disruption in the force!

Resolved Bugs

popeye - v0.3.5

Published by derailed over 5 years ago

Release v0.3.5

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Perfomance Pass

Added a caching layer to improve sanitization report generation. This is a first pass of many but looks like 2X improvement over previous release. Yeah!

Resolved Bugs

popeye - v0.3.4

Published by derailed over 5 years ago

Release v0.3.4

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Spring Cleaning!

Lost of work happened under the hood in this release. Mainly refactoring, bugs and cleanup items. If you notice any breakage from the previous release, please file an issue so we can improve Popeye. Thank you!

Resolved Bugs

Issue #25

popeye - v0.3.3

Published by derailed over 5 years ago

Release v0.3.3

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Bugs and cleanup!

Resolved Bugs

Issue #24

popeye - v0.3.2

Published by derailed over 5 years ago

Release v0.3.2

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Cleaning up

Fixed up a few sanitizer messages.

Resolved Bugs

popeye - v0.3.1

Published by derailed over 5 years ago

Release v0.3.1

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Once again with feelings...

Nikita

Dedicating this release, in honor of my beloved dog Nikita who passed away yesterday ;(

New Sanitizers

Added Sanitizer reports for the following resources:

Deployment
StatefulSet
HorizontalPodAutoscaler
PersistentVolume
PersistentVolumeClaim

Popeye will now scan for configuration and usage issues that may arise from these resources.

WARNING! Capacitors are Charged Up!

Ever wondered how much cluster capacity you actually need? Or which resource scaling may cause your cluster to surpass it's capacity? Fear not my friends! In this release, we introduce Capacitor. We've added metrics monitoring to the sanitizer reports. Capacitor checks your resources (provided they are set!) for potential over/under allocation based on reported metrics. Additionally, Popeye's capacitor checks your HorizontalPodAutoscalers and pre-computes resource allocations based on max replicas. Thus you can be warned when there is a potential for your clusters to either reach or surpass their capacity.

Mind you, this is very much still experimental, so procceed with caution!

Report Formats

Added support for YAML and JSON output via -o CLI parameter.

NOTE! Jurassic mode, though still in full effect, has been moved to -o jurassic

Popeye Does Docker

As of this release, Popeye has been dockerized. You can now run Popeye directly on
your clusters either as a single shot or part of a cronjob. Please checkout the README and the k8s directory for more info about that.

Resolved Bugs

Issue #22

popeye - v0.3.0

Published by derailed over 5 years ago

Release v0.3.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

Nikita

Dedicating this release, in honor of my beloved dog Nikita who passed away yesterday ;(

New Sanitizers

Added Sanitizer reports for the following resources:

Deployment
StatefulSet
HorizontalPodAutoscaler
PersistentVolume
PersistentVolumeClaim

Popeye will now scan for configuration and usage issues that may arise from these resources.

WARNING! Capacitors are Charged Up!

Ever wondered how much cluster capacity you actually need? Or which resource scaling may cause your cluster to surpass it's capacity? Fear not my friends! In this release, we introduce Capacitor. We've added metrics monitoring to the sanitizer reports. Capacitor checks your resources (provided they are set!) for potential over/under allocation based on reported metrics. Additionally, Popeye's capacitor checks your HorizontalPodAutoscalers and pre-computes resource allocations based on max replicas. Thus you can be warned when there is a potential for your clusters to either reach or surpass their capacity.

Mind you, this is very much still experimental, so procceed with caution!

Report Formats

Added support for YAML and JSON output via -o CLI parameter.

NOTE! Jurassic mode, though still in full effect, has been moved to -o jurassic

Popeye Does Docker

As of this release, Popeye has been dockerized. You can now run Popeye directly on
your clusters either as a single shot or part of a cronjob. Please checkout the README and the k8s directory for more info about that.

Resolved Bugs

Issue #22

popeye - v0.2.0

Published by derailed over 5 years ago

Release v0.2.0

Notes

Thank you so much for your support and suggestions to make Popeye better!!

Also if you dig this tool, please make some noise on social! @kitesurfer

Change Logs

ConfigMap and Secrets Support

Added Sanitizer reports for ConfigMap and Secrets. Popeye now scans your clusters for potential dead cm or secret resources and there associated data keys.

Jurassic World Support

Added --jurassic|-j aka NeanderTerm mode, for terminals that don't support dazzling emojis and smashing colors.

Refactored!

Went thru a pretty significant refactor on this drop, so please be on the lookout for potential disturbance in the force and report any issues you may encounter. Thank you!!