autopkg

Automating packaging and software distribution on macOS.

OTHER License

Stars
1.3K

Bot releases are hidden (Show)

autopkg - AutoPkg 2.7.3 Latest Release

Published by homebysix 5 months ago

2.7.3 (May 15, 2024)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.2..v2.7.3

autopkg - AutoPkg 3.0 RC3 Test Release

Published by nmcspadden 9 months ago

Much has changed, internally!

Since AutoPkg 3.0 RC2, I've rewritten the entire Recipe loading logic. This is currently stored in the dev_new_recipe_logic branch, and is being tested for merging into the dev branch here in https://github.com/autopkg/autopkg/pull/926. Since this is not yet merged into dev, GitHub's automatic comparison probably won't do the right thing and generate proper comparison notes.

For now, I am keeping track of the major issues people are experiencing on this Trello board: https://trello.com/b/NlGjpVJm/autopkg-30
(I will also probably move that somewhere else in the future to reduce the spread of information, which is already fragmented enough).

I have successfully tested the rewritten recipe logic on a bunch of recipes, but I need a lot more data, especially from people whose setups don't look like mine. AutoPkg's beauty is that it's flexible enough to run in all kinds of environments and all kinds of ways, and my curse is that I'm largely insulated from the effects of changes on those - so I need all of you to test it for me and tell what's not working.

The ultimate goal here, of course, is to get the new Recipe Map and the new Recipe loading logic finalized so that I can start tackling the major blockers identified in issues already - where it's not loading recipes the way it did previously. The release of AutoPkg 3.0 should essentially be a big performance improvement without any major changes to the way people use AutoPkg (something I haven't always been particularly good at).

Please, test, and generate Discussions or file Issues for things that are blocking you so I can track them and we can work on them accordingly.

Known Issues

  • Recipe trust verification doesn't do anything right now. It doesn't verify, and it doesn't fail - every recipe passes regardless. This is being implemented still.
  • (Dev only) Most of the unit tests were disabled as I've split up the monolithic autopkglib into smaller modules, and have not yet moved the tests around accordingly. Many more need to be written.
  • There's a significant performance hit for the one time the recipe map is generated, until we think of a better way to manage git hashing.
  • The behavior of the GITHUB_TOKEN_PATH isn't great. It works in the specific context of Github-related processors, but it's not usable outside of that. This is being re-implemented.
  • The behavior of preferences when you combine both JSON/Plist external files along with the built-in macOS preferences domain does not at all work the way you'd expect. Some preferences are just silently used from macOS prefs, some are used from the external file, but it almost certainly will be a surprise either way. For this test release, I recommend you rely on either/or - delete your existing macOS preferences and only use a prefs file, or go the other way and don't use an external preference file. This is something I intend to fix before 3.0 release.
  • This release expects you to have already installed AutoPkg on the machine. I have not tested it on a fresh install at all.

What's Changed

Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC2...v3.0.0RC3
See also https://github.com/autopkg/autopkg/pull/926 for the full change in the recipe logic rewrite.

autopkg - AutoPkg 3.0 RC2

Published by nmcspadden about 1 year ago

Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC1...v3.0.0RC2

ISSUES:

autopkg - AutoPkg 3.0 RC1

Published by nmcspadden about 1 year ago

Given the massive number of fundamental changes in AutoPkg behavior, we have decided to promote this version bump to 3.0.0.

Full Changelog from 2.7.2: https://github.com/autopkg/autopkg/compare/master...v3.0.0RC1

What's new in 3.0 RC1 compared to 2.8.1 RC6:

Complete rewrite of GitHub integrations

The Github integrations were previously based on curling the GitHub API directly. For various reasons, that's become both unwieldy and unreliable, especially with how much GitHub has applied rate limiting to API calls from unauthenticated sources. The entire stack has been rewritten to rely on the PyGithub module instead.

autopkglib.github now provides a utility class GitHubSession that will store information about a given repo (as well as the AutoPkg repo and orgs itself by default), and can be used as an interface for working with the PyGithub module. Note that the PyGithub module does require installation of another module, so you will need to use this RC's version of Python (which includes it), or do a pip install on your own, in order to support the new functionality.

GitHubReleasesInfoProvider processor has also been completely rewritten to use the new module and class.

As described in the previous 2.8.1 RC6's notes:
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.

Known issues: the current implementation of the code does not do any rate limit checking or backoff. If you are denied access to the API calls because of rate limits or lack of access, you will get a Python traceback (which is still similar to the behavior from before, but now the error does not come from curl). I hope to add a little bit better of an experience for API rate limiting issues in a future version of AutoPkg.

Previous RC notable items:

Recipe Map

AutoPkg no longer searches through the folders on disk every time it looks for a recipe or processor. Instead, it now builds a recipe map (stored on disk as a JSON file) that tells AutoPkg where to find recipes by shortname and identifier.
The recipe map contains four top level keys:

  • identifiers
  • overrides
  • overrides-identifiers
  • shortnames

Each top level key contains a list of key/value pairs corresponding to the full path of a recipe file. Example:

 "identifiers": {
    "com.github.autopkg.AutoPkg1GitMaster": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AutoPkg/AutoPkg1GitMaster.pkg.recipe",
...
  "shortnames": {
    ...
    "Adium.download": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/Adium/Adium.download.recipe",
...

This recipe map is automatically built whenever you add or delete a repo, or create a new recipe. The recipe map automatically maps everything in your RECIPE_SEARCH_DIRS preferences. The map is consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).

For existing clients, simply re-adding an existing repo will build your map: autopkg repo-add recipes

IMPORTANT NOTE: Searching for recipes on disk no longer happens - only the recipe map is consulted! If you experience issues with locating recipes, please file a Github Issue with steps to reproduce the problem.

autopkglib separation

This has no effect on AutoPkg usage whatsoever, but it's an important note for anyone familiar with the code. Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.

autopkglib.common contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs has all the Preference objects and getting/saving preferences code.

WARNING: if you have any custom processors that loaded functions from autopkglib, such as log or is_mac(), they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed. If you are a custom processor author and are not sure how to address this, please contact the AutoPkg maintainers and/or ask in Slack for advice.

More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once the logic is fully rewritten.

autopkg - AutoPkg 2.8.1 RC6

Published by nmcspadden about 1 year ago

https://github.com/autopkg/autopkg/compare/v2.8.1RC5...dev?expand=1

Recipe Map Changes

The Recipe map is now consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).
The Recipe map is built when you add or remove a repo, or if you generate a new recipe.

If you do not have a recipe map, or the recipe map is invalid when it is needed, autopkg now tells you what to do next (add or remove a repo to fix it). In the future, I'll add a preference or a command line argument to automatically rebuild vs. failing. (In a CI environment, you probably want it to fail rather than to attempt to build one, as it may have incomplete state at that point. In a regular desktop environment, you probably want it to automatically build and proceed).

autopkglib separation

Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.

autopkglib.common contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs has all the Preference objects and getting/saving preferences code.

WARNING: if you have any custom processors that loaded functions from autopkglib, such as log or is_mac(), they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed.

More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once I fix the logic everywhere.

GitHub integration and token usage

The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.

The GitHub logic is being completely rewritten in a different branch, so I've only barely touched it here, and there may be things that break. I don't intend to spend any more effort on the existing GitHub code except to keep it from breaking as I touch things around it. Once this dev branch is done, I'll merge in those changes and work on that next. In the meantime, if you do have existing issues with anything GitHub-related specifically, please file an Issue and I'll make sure that is tested against the new branch.

autopkg - AutoPkg 2.8.1 RC5

Published by nmcspadden about 1 year ago

The recipe map is now created automatically when you use any autopkg verb aside from "help" or "version". This also handles situations where we attempted to read an invalid recipe map (such as would be created in between RC3 and RC4) without rebuilding one first.

This makes the autopkg recipe search feature much more seamless, and is a lot closer to how the existing recipe location mechanism works in the release version (2.7.2).

Fixes #874.

autopkg - AutoPkg 2.8.1 RC4

Published by nmcspadden about 1 year ago

  • AutoPkg can now locate and run overrides by their identifiers. The recipe map now contains both override shortnames and override identifiers to map to the full file paths.
autopkg - AutoPkg 2.8.1 RC3

Published by nmcspadden about 1 year ago

Fixes:

  • Recipe map now correctly rebuilds after adding a new repo (https://github.com/autopkg/autopkg/issues/868)
  • Unit tests now run on the dev branch first
  • Fixed some various lint issues
  • Fixed an issue where you could you not add or remove recipes if RECIPE_SEARCH_DIRS was missing from the preferences; it now populates the default folders.

Known Issues:

  • Preferences store relative/local paths inconsistently. Some are stored in preferences as ~/Library/AutoPkg, others convert to an absolute path upon preference storage like /Users/nmscpadden/Library/AutoPkg.
  • You cannot run a recipe not in the recipe map by using a shortname in the local directory. You must use the absolute path to a recipe not in the recipe map.
  • AutoPkg search is still wonky; this will be addressed in the next release.
autopkg - AutoPkg 2.8.1 RC2

Published by nmcspadden about 1 year ago

Fixes:

autopkg - AutoPkg 2.8.0 RC1

Published by nmcspadden about 1 year ago

New features:

  • The recipe map branch has now been merged in. This means that AutoPkg will build a recipe map (JSON on disk) of all repos and recipes when you add or delete repos, and use that for searching for recipes. This drastically speeds up recipe search, especially if you have lots of repos checked out. The old on-disk search method is a fallback, but will be removed in the future.
  • The recipe map feature was merged in after a long hiatus, and I can't currently guarantee everything was merged correctly. Please test this and let us know if there are any problems or bugs.
  • MunkiImporter now creates pkginfo files with a supported architecture in the filename if and only if there is exactly one supported architecture (i.e. "SomePackage-1.0.0-x86_64.pkginfo"). This will make it much easier to distinguish multiple architecture pkginfos for the same package. A similar PR was merged for Munki (https://github.com/munki/munki/pull/1185).

Fixes:

autopkg - AutoPkg 2.7.2

Published by gregneagle almost 2 years ago

2.7.2 (December 07, 2022)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.1..v2.7.2

autopkg - AutoPkg 2.7.1

Published by gregneagle almost 2 years ago

2.7.1 (December 06, 2022)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7..v2.7.1

autopkg - AutoPkg 2.7

Published by nmcspadden about 2 years ago

2.7.0 (Aug 22, 2022)

Python 3.10.6

AutoPkg now uses Python 3.10.6, and PyObjc 8.5. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.

Automated tests on GitHub

Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!

Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.

Other Changes

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.0...v2.5

autopkg - AutoPkg 2.6.0 Beta

Published by nmcspadden over 2 years ago

2.6.0 (April 18, 2022)

Python 3.10

AutoPkg now uses Python 3.10, and PyObjc 8.2. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.

Automated tests on GitHub

Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!

Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.

Other Changes

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.5.0...v2.6.0Beta1

autopkg - AutoPkg 2.5.0 Beta

Published by gregneagle over 2 years ago

2.5.0 (April 08, 2022)

FIXES:

  • Resolved a bug preventing autopkg repo-update and autopkg repo-delete operations on local file paths (#724; fixes #723 and lindegroup/autopkgr#666)
  • Unarchiver now explicitly clears out archive_format variable when finishing execution #777
  • Xattr usage on Linux should no longer throw errors #778
  • Add a new asset_url output variable to GithubReleasesInfoProvider #779
  • Prevent MunkiImporter duplicating imports when multiple architectures. This now lets MunkiImporter correctly support Apple Silicon/Universal versions of existing software in the repo. #781
autopkg - AutoPkg 2.4.1

Published by nmcspadden over 2 years ago

The main new feature here is that MunkiImporter now allows importing of multiple "flavors" of a given version of software, differing by what architectures they support. In other words, you can now have a recipe import Foo-2.0 (Intel) and another recipe import Foo-2.0 (Apple Silicon) and MunkiImporter will actually import both.

What's Changed

New Contributors

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1

autopkg - AutoPkg 2.4.1 Beta

Published by nmcspadden almost 3 years ago

Re-release of 2.3.2 beta with one single logic fix preventing the recipe map from generating, and the correct versioning.

What's Changed

New Contributors

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1-b

What's Changed

New Contributors

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1-b

autopkg - AutoPkg 2.3.2 Beta

Published by nmcspadden almost 3 years ago

2.3.2 (December 20, 2021)

I've created a new pre-release version of AutoPkg based on the dev_recipe_map branch. This branch has a radical redesign of the recipe loading logic that dramatically speeds up recipe locating.

Instead of searching for recipes or processors by traversing the file system every time we need to go find something, we instead generate a static map file of all repos and recipes on-disk. This static map (cleverly titled "recipe_map.json") searches the file system during certain operations (adding/removing repos, or adding new recipes, or adding overrides), and stores a list of all recipes in two ways:

  1. A list of all identifiers, with the absolute paths to the recipe as the values;
  2. A list of all recipe shortnames ("GoogleChrome.download") with the absolute paths to the recipes as values

Whenever. you do any operation in AutoPkg, it consults this recipe map only and knows where to go next. This has reduced load times for running a recipe from multiple seconds (which scales higher based on how many recipes you have on-disk) to fractions of a second.

Graham Pugh provided this sample output to show the very stark difference with this branch:
Master:

% autopkg run -v GoogleChrome.jamf
**load_recipe time: 22.676719694
**verify_parent_trust time: 114.754227371
**process_cli_overrides time: 114.754611623
**verify time: 172.079330759
**process time: 216.01272573499998

With the new dev version:

./autopkg run -v GoogleChrome.jamf
**load_recipe time: 0.03708735200000002
**verify_parent_trust time: 0.696461484
**process_cli_overrides time: 0.696903272
**verify time: 0.711556611
**process time: 44.696209888

22 seconds -> 0.03 seconds to load recipes.

FUTURE DIRECTION
So, what's next for this?

I want to replace just about all Github Search API calls with static maps instead. Rather than relying on the API to give us information sometimes, what if we took this static map idea a step further? The AutoPkg Github Repo itself could store a static mapping of all recipes and all repos across the org, and clients would simply fetch that static map when trying to search for recipes.

To enhance and combine this functionality with the recipe map, we'd also need to change the override to contain a bit more info. Right now, the override just contains the chain of parents and their hashes as was generated at the time, but what if it also referenced the recipe map to also store the list of all things the recipe would ever need to execute successfully? If we had a full repo map as well as a local recipe map, we could easily triangulate exactly where all the resources required to run a recipe are located, and then fetch them if we don't have them. If we store that information in the override itself, then CI environments that are ephemeral would have all the info they need to run any recipe contained within the override itself, rather than having to make a lot of guesses or assumptions.

CONTEXT
Right now, AutoPkg interacts with GitHub in a few ways:

  1. You use autopkg search, which generally does what it says;
  2. You use autopkg info -p, which tries to search for the parent repos of a recipe and fetch all of them;

The problem is that the Github search API occasionally just.. doesn't. This API is rate limited really heavily, and is especially harsh for large organizations that have lots of outbound traffic from one set of IPs. This means that if a large organization is talking to GitHub API often, you could be rate limited just by sheer volume of traffic. When the API rate limits you, it doesn't return useful results to AutoPkg.

In a CI environment, if you rely on autopkg info -p to automatically pull your repos, this means that occasionally Github just doesn't give you anything. So AutoPkg will fail to pull its parent repos for recipe chains properly, and that means that recipes just occasionally fail for no reason. Trying it again usually just works, without making any changes. At Facebook/Meta, where I use this feature heavily, I see this very frequently.

So frequently, in fact, that it actually reduces the overall reliability of the automatic parent fetching feature.

I still believe that its intended goal is a good one: to avoid having to maintain a hardcoded list of repos to check out in a CI environment (or any environment). I think it makes sense for Autopkg to be able to dynamically figure out what repos you need, and then go get them. The evidence simply shows that we just can't rely on Github's API search for that.

I'd love to hear people's thoughts and see people's test results.

autopkg - AutoPkg 2.3.1

Published by homebysix over 3 years ago

2.3.1 (March 03, 2021)

FIXES:

autopkg - AutoPkg 2.3

Published by homebysix over 3 years ago

2.3 (March 01, 2021)

NEW FEATURES:

AutoPkg now supports recipes in yaml format (#698, example recipe). Yaml recipes tend to be more human-readable than plist recipes, especially for those who don't work with plists on a daily basis.

AutoPkg can produce new recipes in yaml format using autopkg new-recipe SomeCoolApp.pkg.recipe.yaml and make overrides in yaml format using autopkg make-override --format=yaml SomeCoolApp.pkg. Searching for public yaml recipes on GitHub is also possible using autopkg search.

NOTES FOR RECIPE AUTHORS:

  • Because yaml recipes will require AutoPkg 2.3 or later in order to run, and because some members of the AutoPkg community may still be using AutoPkg 1.x, recipe authors are encouraged to be conservative and keep existing public recipes in their current format for a while.
  • If you have both plist and yaml recipes for the same app in your repo, you may experience unexpected behavior now that AutoPkg detects and uses yaml recipes.

OTHER CHANGES FROM 2.2:

  • Added support for internal GitHub URLs (#649)
  • autopkg make-override no longer creates override for deprecated recipes by default (#685)
  • Typo fixed in the recipe template created by autopkg new-recipe
  • Fixed a bug causing autopkg repo-add and autopkg repo-delete to fail for repos in GitHub organizations with non-alphanumeric characters in their names (#712, #715)
  • CodeSignatureVerifier warns when certain incorrect input variables are detected
  • MunkiImporter now uses consistent pkginfo matching logic (#671)
  • Minor edits to help text
  • Improvements to Versioner processor (#600)
  • Help is now shown for autopkg list-processors --help, matching behavior of most other verbs (#717)
  • The output of autopkg list-recipes --plist is now text instead of binary (this matches previous behavior in AutoPkg 1.x)
  • More output when using autopkg repo-add and autopkg repo-delete (#704)
  • Fixed a bug in MunkiImporter that caused incorrect uninstaller_item_location path (#702)
  • Building a foundation for long term expansion of platform support (#648, #651, #653, #656, #658, #666, #670)

KNOWN ISSUES:

  • #710 is currently affecting some autopkg search results (regardless of whether the recipes are plist or yaml)