harvey

• Adds support for Python 3.12
• Bumps all dependencies
• Adds a healthcheck to the nginx Docker container
• Returns system_lock bool on retrieving locks
• Prints deployment log to the error logger if log_level is set to DEBUG
• Raises a HarveyError on deployment fail instead of sys.exit(1), removes redundant sys.exit(0) on success
  • Prints complete error instead of unhelpful "deployment failed" message
• Captures Slack errors via Sentry but no longer raises an exception to ensure builds continue even if there is a problem with message delivery
• Adds py-call-uwsgi-fork-hooks flag to assist threading with uwsgi and sentry (closes #83)

• Swaps the deprecated slackclient for the new slack_sdk, no user impact expected
• Locks max Python version to 3.11 as uwsgi is not yet compatible with the new Python 3.12

• Fixes a bug that couldn't deserialize subprocess output correctly due to ascii encoding (closes #81)

• Fixes a bug where page_size for the /projects endpoint wasn't respected

• Drops support for Python 3.7
• Adds USE_HTTPS_AUTH option to use HTTPS URLs instead of the default SSH URLs
• Use docker compose version invocation on startup to verify version instead of older docker-compose --version now that v1 is deprecated
• Unify the version of the released package and the version in the config under a single variable (this has drifted often in the past)
• Removes the raise_error parameter of kill_deployment since killing a deployment will inherently raise an unhandled exception when we exit with a status of 1.
  • Corrects the status of succeed_deployment from 1 to 0
• Properly expands harvey_path
• .env files are now sourced from both the relative root of the project and the $HARVEY_PATH to allow for greater flexibility
• Bump dependencies
• Corrects and clarifies all documentation
• Drops ability to install from Pip (no longer release to PyPi) since the invocation hasn't worked for some time and needs local commands

• Adds runtime to each deployment attempt's data structure so you can track that over time
• Threads via Harvey are now named (after a project) and can be retrieved via the new /threads endpoint
• Fixes a bug where subprocess errors would get clobbered, now output from subprocesses should be printed as a string instead of bytes or being stripped out
• Overhauls the deployment logs to show more or less information depending on if you are running in debug mode or not
• Unifies message structure when sending to Slack, stdout/stderr, and storing to database
• Properly sorts attempts in desc order by timestamp so clients don't need to
• Fix a bug where deploys were miscategorized as success when they were actually a failure

• Stores webhooks to the database so we can use them later for things like redeploying or reference
• Adds a new /projects/<project_name>/redeploy endpoint that allows you to redeploy a project with the local webhook data
• Fixes the long-running and multi-bug issues related to workers, connections, and timeouts stopping harvey from running
  • Simplifies the uwsgi worker config greatly in the hopes to fix thread locking issues (closes #72)
  • Fixes connections getting refused after ~24 hours of uptime due to using the http socket instead of the uwsgi socket
  • Patches segfault on macOS by not using proxies
  • Adjusts various timeouts and limits across the board to assist with edge-case connection and errors related to the server
• Unifies git_timeout and deploy_timeout to new operation_timeout with a default of 300 seconds.
  • Lowers Docker API timeout from 30 seconds to 10 seconds
• Deployments now store the log, timestamp and status keys inside an attempts array allowing for multiple saved records of each attempt of a deploy. This is helpful when a commit is redeployed later ensuring that the information from every attempt at deploying a specific commit are retained. Previously, you would only have the most recent details available because the log, status, and timestamp were overridden on each new deploy of the same commit. There is still a timestamp at the root level of deployments that will update to the most recent attempt's timestamp (closes #74)
• Overhauls logging
  • Adjusts log sizes from 200kb to 2mb
  • Logs now delete on a uWSGI cron each day if they are older than 14 days
  • We no longer log harvey and uwsgi logs separately since they were both going into the uwsgi logs (closes #78)
• Bumps all dependencies

• Adds a try/catch block to the spawned thread in an attempt to kill off failed deployments
• Moves database logic to adopt the repos pattern to separate their logic from the rest of the service
• Adds a script to fail In-Progress deployments
• Changes the verbose webhook body debug logger to an info logger that only says which repos the webhook originated from
• Log 500 error messages correctly when an endpoint is hit (previously some errors weren't ever bubbling up and were suppressed or lost)
• Don't throw an error when a lock cannot be looked up for a project, log instead (allows for first-time deploys)
• Various other bug fixes and improvements

• API has more unified error handling for 5xx error types
• Introduced new HarveyError and cleaned up various error handling throughout the app. All messages should now properly get logged when a deployment gets killed, errors raised when Sentries need to be triggered, and various log levels were corrected
• Healthchecks of containers now check when the container was started to ensure that not only are they running, but that they restarted within the last 60 seconds as a part of the deploy
  • Fixed a bug where containers may not properly recreate when their configs or images don't differ from the last deploy
• If a project's lock status cannot be determined, we now kill the process instead of continuing and logging only to ensure we don't steamroll a previous deployment
• Harvey can now distinguish between a system lock and a user lock allowing for user-locked deployments to stay that way even if a deployment fails
• Wraps store_deployment_details in a try/except to log out errors with saving details to the DB
• Drops default deployment timeout from 30 minutes to 10 minutes
• Various other improvements and fixes

• Bumps uwsgi from 2.0.20 to 2.0.21 unlocking Python 3.10 and 3.11 support
• Bumps nginx version
• Changes from process/threads concurrency to dynamic worker concurrency with uwsgi
• Added timeouts and worker kill commands so Harvey will canabolize itself instead of the OS in the case of long running workers or too many requests
• Adds total_count to collection responses

• Switches from sha1 webhook secrets from GitHub to sha256 for increased security and finally adds unit tests for the validate_webhook_secret function

• Swaps gunicorn for uwsgi for better performance. This switch also fixes the concurrency deadlock in production
• Harvey now runs as a daemon which should drastically improve performance
• Drops default processes running from 4 to 2
• Adds a check to ensure the deployment_type passed was valid and fails if not

• Changes name of the PyPI project from harvey-ci to harvey-cd to better match the nature of the application
• Consolidates all sqlite database tables into a single database file for better performance and disk savings (to properly migrate to using this new database structure, please run the utils/consolidate_sqlite_databases.py script which will migrate and consolidate the old database(s) content to the single unified file)

Breaking Changes

• Renamed all occurances of pipeline to deployment to unify the language used throughout the project and avoid confusion
• The pipeline key has been renamed to deployment_type in the webhook data. A default of deploy has been set where previously there was no default
• The /pipelines/start endpoint is now called /deploy
• All other /pipeline endpoints have been renamed /deployment

Features

• Adds a simple authentication system where endpoints can be protected with the same API key as the webhook secret (uses basic auth)
• Added new /locks, /locks/{project_name}, /projects/{project_name}/lock, and /projects/{project_name}/unlock endpoints
• Adds sentry integration
• Performance improvements for sorting & paginating large lists of deployments and locks
• Enforces Docker Compose v2 availability by checking during the app bootstrap process

Bug Fixes

• Fixes a bug where retrieving a lock status may return the wrong value
• Fixes a bug that kills builds when a lock cannot be found (eg: new projects)

Chores

• Moved all lock functionality to its own module

• Overhauled the pipeline logs to provide more details that are easier to read
• Changed the required harvey.json file for each project to .harvey.yml (expected fields/values remain the same)
• Sets the default pagination limit down to 20 from 100, adds PAGINATION_LIMIT env var to allow customization
• Added new HARVEY_PATH, GIT_TIMEOUT, DEPLPOY_TIMEOUT, and DEPLOY_ON_TAG env vars for customization
• Better syncing of the version string for the release and what appears in logs
• Better error handling around 404s at the API level
• Emojis will now be converted to a fallback string representation in pipeline logs while they will remain emojis in Slack messages
• Various bug fixes and refactoring improvements

• Added logging with configurable log level
• Use new docker compose invocation over older docker-compose
• Various code refactor including bug fixes and optimizations, and improved code coverage
  • Cleaned up various dead env variables and documentation
• No longer log progress bars of Docker image builds
• Gracefully handles not being able to find a container or list of containers
• Shallow clone repos to 1 commit instead of 10
• Retry pipelines that fail due to local, uncommitted changes present when pulling the repo (closes #38)
• Adds a locking mechanism to deployments when a pipeline is in-flight to avoid crashing Docker when multiple docker compose commands are run simultaneously for a single project (closes #58)
• Now saves projects to ~/harvey/projects instead of locally to harvey/projects
• Now saves project_logs to ~/harvey/project_logs instead of locally to harvey/logs
• Adds a check to ensure there is a docker-compose.yml file present
  • Now supports yaml files ending in both yml or yaml
• Overhauled the API to allow retrieval of projects in addition to pipelines. We now also store more metadata about each object including a timestamp, the full log, the project name, commit, status, etc
• Fixed various path bugs that wouldn't resolve properly on Windows (used os.path.join())
• Changed all datetime/timestamp fields to use utc time

• Adds the ability to pass Harvey configuration data in a data field in the webhook instead of the default harvey.json configuration file kept in the repo (either are now options)
• Configuration data now properly gets validated (pipeline key and existence)
• Added emojis to the healthcheck messages via Slack
• The pipelines and stages modules were consolidated into pipelines now that the testing functionality has been removed
• Removed the filter webhook functionality as it was prohibitively expensive to do correctly due to the vast number of IPs to guard against
• Reworked how we pulled json data from webhooks to be more straightforward
• We now use the docker Python SDK instead of hitting raw socket endpoints (closes #49)
• Adds gunicorn for production deployments instead of the development Flask server
• Refactors invocations of subprocesses to not use the shell, no longer change directories but invoke commands from within the context they require
  • Changes the compose key to prod_compose which now accepts a boolean. We no longer support custom flags to the compose endpoint
• Removes deprecated /compose endpoint which has been replaced with the /start endpoint
• Adds a new healthcheck key on the config which accepts an array of container names to check for when deploying. Harvey will attempt to run healthchecks against these containers and retry a few times if they are not yet running. The pipeline will only show as success if we can only get a good healthcheck from the list of containers provided. This new option now allows healthchecks to be run against all containers in the stack instead of the base app alone (great for adding databases, caches, and other containers you may have)
• Various bug fixes

Breaking Changes

• Strips away all non-compose logic - Harvey is now exclusively a Docker Compose deployment platform (closes #9, closes #15, closes #19, closes #20, closes #22, closes #24, closes #28, closes #44)
• Strips away the non-functioning testing framework so that we can focus on deployments instead of the full CI/CD landscape
• Removes the poorly integrated APP_MODE env variable now that FILTER_WEBHOOKS can be toggled

Other Changes

• Added a 404 error handler to the API
• Rebasing repositories now doesn't assume the branch to rebase against is titled main
• Cleaned up all documentation to reflect the breaking changes

• Corrected a container naming bug that could make compose containers fail their healthchecks due to mismatched container names
• Small enhancements and cleanup throughout the project

• Corrects encoding of test logs, binary data should no longer show in the output (closes #4)
• Overhauls the webhook flow
  • Users can now configure their own comma separated list of ALLOWED_BRANCHES
  • Previously a webhook secret was required; now, Harvey will run Pipelines without a webhook secret (bypassing decoding and validation of the previously non-existent secret) if there is no WEBHOOK_SECRET variable set
  • Additional refactor surrounding how we validate webhook secrets
• Fixed the container healthcheck when using the compose workflow - this was accomplished by making the compose and non-compose container names uniform
• Bumps dependencies (Flask 1 to Flask 2) and now requires Python 3.7. Also removes the mock library in favor of the builtin unittest.mock library
• Various code refactors and bug fixes