Automating packaging and software distribution on macOS.
OTHER License
Bot releases are hidden (Show)
prefetch_filename
on macOS 14.5. (https://github.com/autopkg/autopkg/pull/939)Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.2..v2.7.3
Published by nmcspadden 9 months ago
Since AutoPkg 3.0 RC2, I've rewritten the entire Recipe loading logic. This is currently stored in the dev_new_recipe_logic
branch, and is being tested for merging into the dev
branch here in https://github.com/autopkg/autopkg/pull/926. Since this is not yet merged into dev
, GitHub's automatic comparison probably won't do the right thing and generate proper comparison notes.
For now, I am keeping track of the major issues people are experiencing on this Trello board: https://trello.com/b/NlGjpVJm/autopkg-30
(I will also probably move that somewhere else in the future to reduce the spread of information, which is already fragmented enough).
I have successfully tested the rewritten recipe logic on a bunch of recipes, but I need a lot more data, especially from people whose setups don't look like mine. AutoPkg's beauty is that it's flexible enough to run in all kinds of environments and all kinds of ways, and my curse is that I'm largely insulated from the effects of changes on those - so I need all of you to test it for me and tell what's not working.
The ultimate goal here, of course, is to get the new Recipe Map and the new Recipe loading logic finalized so that I can start tackling the major blockers identified in issues already - where it's not loading recipes the way it did previously. The release of AutoPkg 3.0 should essentially be a big performance improvement without any major changes to the way people use AutoPkg (something I haven't always been particularly good at).
Please, test, and generate Discussions or file Issues for things that are blocking you so I can track them and we can work on them accordingly.
Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC2...v3.0.0RC3
See also https://github.com/autopkg/autopkg/pull/926 for the full change in the recipe logic rewrite.
Published by nmcspadden about 1 year ago
Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC1...v3.0.0RC2
ISSUES:
Published by nmcspadden about 1 year ago
Given the massive number of fundamental changes in AutoPkg behavior, we have decided to promote this version bump to 3.0.0.
Full Changelog from 2.7.2: https://github.com/autopkg/autopkg/compare/master...v3.0.0RC1
The Github integrations were previously based on curling the GitHub API directly. For various reasons, that's become both unwieldy and unreliable, especially with how much GitHub has applied rate limiting to API calls from unauthenticated sources. The entire stack has been rewritten to rely on the PyGithub module instead.
autopkglib.github
now provides a utility class GitHubSession
that will store information about a given repo (as well as the AutoPkg repo and orgs itself by default), and can be used as an interface for working with the PyGithub module. Note that the PyGithub module does require installation of another module, so you will need to use this RC's version of Python (which includes it), or do a pip install on your own, in order to support the new functionality.
GitHubReleasesInfoProvider processor has also been completely rewritten to use the new module and class.
As described in the previous 2.8.1 RC6's notes:
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH
. The old GITHUB_TOKEN
preference key is no longer used anywhere. You must now store your GH token in a flat text file.
Known issues: the current implementation of the code does not do any rate limit checking or backoff. If you are denied access to the API calls because of rate limits or lack of access, you will get a Python traceback (which is still similar to the behavior from before, but now the error does not come from curl). I hope to add a little bit better of an experience for API rate limiting issues in a future version of AutoPkg.
AutoPkg no longer searches through the folders on disk every time it looks for a recipe or processor. Instead, it now builds a recipe map (stored on disk as a JSON file) that tells AutoPkg where to find recipes by shortname and identifier.
The recipe map contains four top level keys:
Each top level key contains a list of key/value pairs corresponding to the full path of a recipe file. Example:
"identifiers": {
"com.github.autopkg.AutoPkg1GitMaster": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AutoPkg/AutoPkg1GitMaster.pkg.recipe",
...
"shortnames": {
...
"Adium.download": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/Adium/Adium.download.recipe",
...
This recipe map is automatically built whenever you add or delete a repo, or create a new recipe. The recipe map automatically maps everything in your RECIPE_SEARCH_DIRS preferences. The map is consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).
For existing clients, simply re-adding an existing repo will build your map: autopkg repo-add recipes
IMPORTANT NOTE: Searching for recipes on disk no longer happens - only the recipe map is consulted! If you experience issues with locating recipes, please file a Github Issue with steps to reproduce the problem.
This has no effect on AutoPkg usage whatsoever, but it's an important note for anyone familiar with the code. Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.
autopkglib.common
contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs
has all the Preference objects and getting/saving preferences code.
WARNING: if you have any custom processors that loaded functions from autopkglib, such as log
or is_mac()
, they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed. If you are a custom processor author and are not sure how to address this, please contact the AutoPkg maintainers and/or ask in Slack for advice.
More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once the logic is fully rewritten.
Published by nmcspadden about 1 year ago
https://github.com/autopkg/autopkg/compare/v2.8.1RC5...dev?expand=1
The Recipe map is now consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).
The Recipe map is built when you add or remove a repo, or if you generate a new recipe.
If you do not have a recipe map, or the recipe map is invalid when it is needed, autopkg now tells you what to do next (add or remove a repo to fix it). In the future, I'll add a preference or a command line argument to automatically rebuild vs. failing. (In a CI environment, you probably want it to fail rather than to attempt to build one, as it may have incomplete state at that point. In a regular desktop environment, you probably want it to automatically build and proceed).
Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.
autopkglib.common
contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs
has all the Preference objects and getting/saving preferences code.
WARNING: if you have any custom processors that loaded functions from autopkglib, such as log
or is_mac()
, they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed.
More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once I fix the logic everywhere.
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.
The GitHub logic is being completely rewritten in a different branch, so I've only barely touched it here, and there may be things that break. I don't intend to spend any more effort on the existing GitHub code except to keep it from breaking as I touch things around it. Once this dev branch is done, I'll merge in those changes and work on that next. In the meantime, if you do have existing issues with anything GitHub-related specifically, please file an Issue and I'll make sure that is tested against the new branch.
Published by nmcspadden about 1 year ago
The recipe map is now created automatically when you use any autopkg verb aside from "help" or "version". This also handles situations where we attempted to read an invalid recipe map (such as would be created in between RC3 and RC4) without rebuilding one first.
This makes the autopkg recipe search feature much more seamless, and is a lot closer to how the existing recipe location mechanism works in the release version (2.7.2).
Fixes #874.
Published by nmcspadden about 1 year ago
Published by nmcspadden about 1 year ago
Fixes:
Known Issues:
~/Library/AutoPkg
, others convert to an absolute path upon preference storage like /Users/nmscpadden/Library/AutoPkg
.Published by nmcspadden about 1 year ago
Fixes:
Published by nmcspadden about 1 year ago
New features:
Fixes:
Published by gregneagle almost 2 years ago
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.1..v2.7.2
Published by gregneagle almost 2 years ago
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7..v2.7.1
Published by nmcspadden about 2 years ago
AutoPkg now uses Python 3.10.6, and PyObjc 8.5. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.
Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!
Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.0...v2.5
Published by nmcspadden over 2 years ago
AutoPkg now uses Python 3.10, and PyObjc 8.2. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.
Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!
Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.5.0...v2.6.0Beta1
Published by gregneagle over 2 years ago
FIXES:
autopkg repo-update
and autopkg repo-delete
operations on local file paths (#724; fixes #723 and lindegroup/autopkgr#666)archive_format
variable when finishing execution #777
asset_url
output variable to GithubReleasesInfoProvider #779
Published by nmcspadden over 2 years ago
The main new feature here is that MunkiImporter now allows importing of multiple "flavors" of a given version of software, differing by what architectures they support. In other words, you can now have a recipe import Foo-2.0 (Intel) and another recipe import Foo-2.0 (Apple Silicon) and MunkiImporter will actually import both.
Copier
Processor. by @Frzk in https://github.com/autopkg/autopkg/pull/711
asset_url
output variable to GitHubReleasesInfoProvider by @haircut in https://github.com/autopkg/autopkg/pull/779
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1
Published by nmcspadden almost 3 years ago
Re-release of 2.3.2 beta with one single logic fix preventing the recipe map from generating, and the correct versioning.
Copier
Processor. by @Frzk in https://github.com/autopkg/autopkg/pull/711
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1-b
Copier
Processor. by @Frzk in https://github.com/autopkg/autopkg/pull/711
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.3.1...v2.4.1-b
Published by nmcspadden almost 3 years ago
I've created a new pre-release version of AutoPkg based on the dev_recipe_map branch. This branch has a radical redesign of the recipe loading logic that dramatically speeds up recipe locating.
Instead of searching for recipes or processors by traversing the file system every time we need to go find something, we instead generate a static map file of all repos and recipes on-disk. This static map (cleverly titled "recipe_map.json") searches the file system during certain operations (adding/removing repos, or adding new recipes, or adding overrides), and stores a list of all recipes in two ways:
Whenever. you do any operation in AutoPkg, it consults this recipe map only and knows where to go next. This has reduced load times for running a recipe from multiple seconds (which scales higher based on how many recipes you have on-disk) to fractions of a second.
Graham Pugh provided this sample output to show the very stark difference with this branch:
Master:
% autopkg run -v GoogleChrome.jamf
**load_recipe time: 22.676719694
**verify_parent_trust time: 114.754227371
**process_cli_overrides time: 114.754611623
**verify time: 172.079330759
**process time: 216.01272573499998
With the new dev version:
./autopkg run -v GoogleChrome.jamf
**load_recipe time: 0.03708735200000002
**verify_parent_trust time: 0.696461484
**process_cli_overrides time: 0.696903272
**verify time: 0.711556611
**process time: 44.696209888
22 seconds -> 0.03 seconds to load recipes.
FUTURE DIRECTION
So, what's next for this?
I want to replace just about all Github Search API calls with static maps instead. Rather than relying on the API to give us information sometimes, what if we took this static map idea a step further? The AutoPkg Github Repo itself could store a static mapping of all recipes and all repos across the org, and clients would simply fetch that static map when trying to search for recipes.
To enhance and combine this functionality with the recipe map, we'd also need to change the override to contain a bit more info. Right now, the override just contains the chain of parents and their hashes as was generated at the time, but what if it also referenced the recipe map to also store the list of all things the recipe would ever need to execute successfully? If we had a full repo map as well as a local recipe map, we could easily triangulate exactly where all the resources required to run a recipe are located, and then fetch them if we don't have them. If we store that information in the override itself, then CI environments that are ephemeral would have all the info they need to run any recipe contained within the override itself, rather than having to make a lot of guesses or assumptions.
CONTEXT
Right now, AutoPkg interacts with GitHub in a few ways:
autopkg info -p
, which tries to search for the parent repos of a recipe and fetch all of them;The problem is that the Github search API occasionally just.. doesn't. This API is rate limited really heavily, and is especially harsh for large organizations that have lots of outbound traffic from one set of IPs. This means that if a large organization is talking to GitHub API often, you could be rate limited just by sheer volume of traffic. When the API rate limits you, it doesn't return useful results to AutoPkg.
In a CI environment, if you rely on autopkg info -p
to automatically pull your repos, this means that occasionally Github just doesn't give you anything. So AutoPkg will fail to pull its parent repos for recipe chains properly, and that means that recipes just occasionally fail for no reason. Trying it again usually just works, without making any changes. At Facebook/Meta, where I use this feature heavily, I see this very frequently.
So frequently, in fact, that it actually reduces the overall reliability of the automatic parent fetching feature.
I still believe that its intended goal is a good one: to avoid having to maintain a hardcoded list of repos to check out in a CI environment (or any environment). I think it makes sense for Autopkg to be able to dynamically figure out what repos you need, and then go get them. The evidence simply shows that we just can't rely on Github's API search for that.
I'd love to hear people's thoughts and see people's test results.
Published by homebysix over 3 years ago
FIXES:
autopkg repo-update
and autopkg repo-delete
operations on local file paths (#724; fixes #723 and lindegroup/autopkgr#666)Published by homebysix over 3 years ago
NEW FEATURES:
AutoPkg now supports recipes in yaml format (#698, example recipe). Yaml recipes tend to be more human-readable than plist recipes, especially for those who don't work with plists on a daily basis.
AutoPkg can produce new recipes in yaml format using autopkg new-recipe SomeCoolApp.pkg.recipe.yaml
and make overrides in yaml format using autopkg make-override --format=yaml SomeCoolApp.pkg
. Searching for public yaml recipes on GitHub is also possible using autopkg search
.
NOTES FOR RECIPE AUTHORS:
OTHER CHANGES FROM 2.2:
autopkg make-override
no longer creates override for deprecated recipes by default (#685)autopkg new-recipe
autopkg repo-add
and autopkg repo-delete
to fail for repos in GitHub organizations with non-alphanumeric characters in their names (#712, #715)autopkg list-processors --help
, matching behavior of most other verbs (#717)autopkg list-recipes --plist
is now text instead of binary (this matches previous behavior in AutoPkg 1.x)autopkg repo-add
and autopkg repo-delete
(#704)uninstaller_item_location
path (#702)KNOWN ISSUES:
autopkg search
results (regardless of whether the recipes are plist or yaml)