cloud-init

Cloud-init release 24.2 is now available

The 24.2 release:

• spanned about 4 months
• had 39 contributors
• fixed 53 issues

Highlights:

• feat: Add support for FTP and FTP over TLS (#4834)
• feature(schema): add networkv2 schema (#4892)
• feat: Add new distro 'azurelinux' for Microsoft Azure Linux. (#4931)
• feat: Use NetworkManager renderer by default in RHEL family
• feat(ca_certs): Add AlmaLinux OS and CloudLinux OS support (#5264)
• feat(sysconfig): Add DNS from interface config to resolv.conf (#5401)
• feat(alpine): add support for Busybox adduser/addgroup (#5176)
• feat(disk_setup): Add support for nvme devices (#5263)

Breaking Changes:

• fix(rhel)!: Fix network boot order in upstream cloud-init (#5089)

Note for downstream image creators:

• Added support in cloudinit.features.DEPRECATION_INFO_BOUNDARY allowing stable downstream images to pin the original MAJOR.MINOR version of cloud-init released on that image. This avoids introduction of new deprecation messages (and potential exit 2 from cloud-init status) across cloud-init version upgrades.

• fix(ec2): Correctly identify netplan renderer (#5361)

• fix(ec2): Ensure metadata exists before configuring PBR (#5287)
• fix: Check renderer for netplan-specific code (#5321)
• test: Fix failing test_ec2.py test (#5324)

Cloud-init bug fix release against 24.1.

• fix(dhcpcd): Make lease parsing more robust (#5129)
• net/dhcp: raise InvalidDHCPLeaseFileError on error parsing dhcpcd lease
  (#5128) [Chris Patterson]
• fix: Fix runtime file locations for cloud-init (#4820)
• net/dhcp: bump dhcpcd timeout to 300s (#5127) [Chris Patterson]
• net: Warn when interface rename fails
• ephemeral(dhcpcd): Set dhcpcd interface down
• test: Remove side effects from tests (#5074)
• refactor: Import log module rather than functions (#5074)

• fix: Always use single datasource if specified (#5098)
• fix: Allow caret at the end of apt package (#5099)

• test: Don't assume ordering of ThreadPoolExecutor submissions (#5052)
• refactor(ec2): simplify convert_ec2_metadata_network_config
• tests: drop CiTestCase and convert to pytest
• bug(tests): mock reads of host's /sys/class/net via get_sys_class_path
• fix: Fix breaking changes in package install (#5069)
• fix: Undeprecate 'network' in schema route definition (#5072)
• fix(ec2): fix ipv6 policy routing
• fix: document and add 'accept-ra' to network schema (#5060)
• bug(maas): register the correct DatasourceMAASLocal in init-local
  (#5068) (LP: #2057763)

• fix: Include DataSourceCloudStack attribute in unpickle test (#5039)
• bug(vmware): initialize new DataSourceVMware attributes at unpickle (#5021)
• fix(apt): Don't warn on apt 822 source format (#5028)
• fix: Add "broadcast" to network v1 schema (#5034)
• pro: honor but warn on custom ubuntu_advantage in /etc/cloud/cloud.cfg (#5030)
• net/dhcp: handle timeouts for dhcpcd (#5022)
• fix: Make wait_for_url respect explicit arguments
• bug(wait_for_url): when exceptions occur url is unset, use url_exc
• test: Fix scaleway retry assumptions
• fix: Make DataSourceOracle more resilient to early network issues (#5025)
• tests: Fix wsl test (#5008)

Cloud-init release 24.1 is now available

Summary

The 24.1 release:

• spanned about 5 months
• had 27 contributors
• welcomed commits from 6 new contributors
• fixed 5 Launchpad issues
• fixed 28 Github issues

Highlights

• added support for WSL (Windows Subsystem for Linux)
• added support for dhcpcd
• added experimental support for Python 3.13
• improved behavior of cloud-init status
• ec2: added support for multi NIC / IP instances
• many improvements for cross-distro compatiblity

Breaking Changes

• made dhcpcd the default dhcp client
• removed Ubuntu's ordering dependency on snapd.seeded
• stopped adding network v2 DNS to global DNS
• dropped support for dsa host keys

See the full changelog for more details.

Highlights:

• fix(nocloud): smbios datasource definition

• fix: Handle systemctl when dbus not ready (#4842) (LP: #2046483)

Highlights:

• fix: Handle invalid user configuration gracefully (#4797) (LP: #2051147)

Highlights:

• fix(status): Handle systemctl commands when dbus not ready

General:

• Add shellcheck to CI
• util: Remove function abs_join()
• Add support for cloud-init "degraded" state
• status.json: Don't override detail key with error condition
• tools/render-template: Make yaml loading opt-in, fix setup.py
• boothook: allow stdout/stderr to emit to cloud-init-output.log
• cloud-init-generator: Various performance optimizations
• systemd: Standardize cloud-init systemd enablement
• benchmark: benchmark cloud-init-generator independent of ds-identify
• Automatically linkcheck in CI
• cmd: Don't write json status files for non-boot stages
• apport: collect ubuntu-pro logs if ubuntu-advantage.log present
• sources: do not override datasource detection if None is in list
• clean: add a new option to clean generated config files
• pep-594: drop deprecated pipes module import

Distributions:

• Debian: Set Debian's default locale to be c.UTF-8
• Gentoo: upstream gentoo patch
• OpenBSD: Update build-on-openbsd dependencies
• SUSE: Add suse

AliYun: Allow aliyun ds to fetch data in init-local

Azure:

• report failure to eject as error instead of debug
• Check for stale pps data from IMDS
• imds: remove limit for connection errors if route present
• workaround to disable reporting IMDS failures on Azure Stack
• errors: include http code in reason for IMDS failure
• report failure to host if ephemeral DHCP secondary NIC
• imds: ensure new errors are logged immediately when retrying
• check for primary interface when performing DHCP
• imds: retry on 429 errors for reprovisiondata
• add option to enable/disable secondary ip config
• Ignore system volume information folder while scanning
  for files in the ntfs resource disk

MAAS: Add datasource to init-local timeframe

NWCS: Update DataSourceNWCS.py

VMware: Fall back to vmtoolsd if vmware-rpctool errs

ConfigModules:

• cc_apt_configure:
  • add deb822 support for default sources file
  • install software-properties-common when absent but needed
  • Install gnupg if gpg not found
  • kill dirmngr/gpg-agent without gpgconf dependency
• cc_apt_pipelining: Update docs, deprecate options
• cc_resizefs: Add bcachefs resize support
• cc_package_update_upgrade_install:
  • Allow installing snaps via package_update_upgrade_install module
• cc_ubuntu_advantage:
  • do not rely on uaclient.messages module
  • shift CLI command from ua to pro for all interactions
• cc_set_hostname, cc_update_hostname: check for create_hostname_file
  key before writing /etc/hostname

Network:

• ephemeral: Handle link up failure for both ipv4 and ipv6
• warn invalid cfg add /run/cloud-init/network-config
• remove the word "on instance boot" from cloud-init generated config

Schema:

• non-root fallback to default paths on perm errors
• annotation path for invalid top-level keys
• top-level additionalProperties: false
• network-config optional network key. route uses oneOf
• add cloud_init_deepest_matches for best error message
• add network-config support to schema subcommand
• cloud-init schema in early boot or in dev environ
• annotation of nested dicts lists in schema marks

DsIdentify:

• exit 2 on disabled state from marker or cmdline
• Allow disable service and override environment
• match Azure datasource's ds_detect() behavior
• Use grep for faster parsing of cloud config in ds-identify

Logging:

• Standardize logging output to stderr
• Don't configure NullHandler
• Standardize use of cloudinit's logging module
• Log PPID for better debuggin
• Remove logging of PPID path
• style: Make cloudinit.log functions use snake case
• Export warning logs to status.json

Docs:

• man: Document cloud-init error codes
• Add summit digest/trip report to docs
• Clarify boothook docs
• Describe disabling cloud-init using an environment variable
• Tidy up contributing docs
• Add cloud-init overview/introduction
• vmware: Update contents relevant to disable_vmware_customization
• summit-notes: add 2023 notes for reference in mailinglist/discourse

Cloud-init bug fix release 23.3.2:

• Fix pip-managed ansible on pip < 23.0.1 (#4403)

Cloud-init bug fix release 23.3.2:

• Revert "ds-identify/CloudStack: $DS_MAYBE if vm running on vmware/xen (#4281)"
  (#4511) (LP: #2039453)

• Remove python deprecated crypt dependency in favor of
  passlib.hash.bcrypt where present. crypt was used by by NetBSD and
  Azure datasource. Plan is to eventually deprecate passlib dependency too.
• network: support busybox micro DHCP client (udhcpc) alternative to
  deprecated isc-dhcp-client.
• Clouds:
  • Ensure Akamai(and Linode) datasource included in default discovery
  • Ec2: support decoding double base64 encoded user-data
  • GCE: improve fallback nic selection ens4 over eth0 as primary/default NIC
• Distributions:
  • debian/ubuntu: no longer use localectl to set keymap (LP: #2030788)
  • systemd v253++: cloud-init status --wait won't block on spawned daemons
  • FreeBSD: Ephemeral networking fixes for BSD since /run is preserved
  • FreeBSD introduce ds-identify for early datasource detection
• Modules:
  • users: support doas/opendoas rules as alternative to sudo
  • PEP 668: pip-installed ansible to support break-system-packages installs
• tooling: New file /run/cloud-init/combined-cloud-config.json to read all
  merged user-data/config without invoking python.

Cloud-init bug fix release 23.2.2:

• Fix NoCloud kernel commandline key parsing (#4273) (Fixes: #4271)
  (LP: #2028562)
• Fix reference before assignment (#4292) (Fixes: #4288) (LP: #2028784)

p.s.: Many thanks to rustydb and anguswilliams for reporting the issues and anyone involved!

Cloud-init bug fix release 23.2.1:

• Correct nocloud-net datasource detection when ds=nocloud-net provided on
  kernel commandline. (LP: #2025180)

p.s.: Many thanks to sorinp for late breaking regression detection with release 23.2 brought up in IRC.

Distributions:

- Alpine: update locale file, use os-release PRETTY_NAME
- FreeBSD:
  * user account locking
  * growpart resize root partition and grow using growfs onestart
  * better identify MBR slices
- RedHat:
  * Drop IBM refresh_rmc_and_interfaces config module only
     applicable on RHEL7
  * Fedora: Enable CA handling
- OpenSUSE:
  * Enable SUSE based distros for ca handling
  * Remove sysvinit files

clouds

- Azure:
  * retry fetching metadata up to 300 seconds
  * introduce identity module
  * add networking check for all source PPS
  * improved error reporting, for hosts and DHCP errors
  * report success to host and introduce kvp module
- GCE: activate network discovery on every boot
- OpenStack: honor the DNS servers associated with a network
- Oracle: prefer system_cfg over ds network config source
- DataSourceScaleway: upcoming IPv6 support
- NoCloud:
   * Use seedfrom protocol to determine mode
   * fix kernel commandline semi-colon delimited args
   * support `ci.ds=` kernel cmdline key for all datasources

ConfigModules:

- ntp: add 'peers' and 'allow' directives
- cc_grub_dpkg: Added UEFI support

Network:

- DHCP: Refactor dhcp client code for deprecated isc-dclient
- fix netstate getway keyerror for iproutes without gateway
- resolv_conf: Allow > 3 nameservers
- NetworkManager:
  * set higher autoconnect priority value 120 for cloud-init
  * add method for ipv6 static IP configuration
  * generate ipv6 stateful dhcp config at par with sysconfig
- sysconfig: prefer sysconfig when NM ifcfg-rh plugin installed
- macs: ignore duplicate MAC for devs with driver driver qmi_wwan

Security:

- do not create dsa and ed25519 host keys when crypto FIPS
- Make user/vendor data sensitive and remove log permissions

Schema:

- Make user/vendor data sensitive and remove log permissions
- users: schema permit empty list to indicate create no users
- validation of jinja template user-data

Docs:

- update network configuration path links
- Document use of `ip route append` to add routes
- Update kernel command line docs

Security release.

Make user/vendor data sensitive and remove log permissions

Because user data and vendor data may contain sensitive information,
this commit ensures that any user data or vendor data written to
instance-data.json gets redacted and is only available to root user.

Also, modify the permissions of cloud-init.log to be 640, so that
sensitive data leaked to the log isn't world readable.
Additionally, remove the logging of user data and vendor data to
cloud-init.log from the Vultr datasource.

LP: #2013967
CVE: CVE-2023-1786