yggdrasil-go

Fixed

• A bug which caused startup problems on Windows and FreeBSD should be fixed
• Resolved some minor link state and listener management bugs during shutdown

Added

• WebSocket support for peerings, by using the new ws:// scheme in Listen and Peers
  • Additionally, the wss:// scheme can be used to connect to a WebSocket peer behind a HTTPS reverse proxy

Changed

• On Linux, the TUN adapter now uses vectorised reads/writes where possible, which should reduce the amount of CPU time spent on syscalls and potentially improve throughput
• Link error handling has been improved and various link error messages have been rewritten to be clearer
• Upgrade dependencies

Fixed

• Multiple multicast connections to the same remote machine should now work correctly
  • You may get two connections in some cases, one inbound and one outbound, this is known and will not cause problems
• Running as a Windows service should be more reliable with service startup and shutdown bugs fixed

• Go 1.21 is now required to build Yggdrasil

Added

• The getPeers endpoint now reports the RTT/latency of directly connected peers

Changed

• The tree parent selection algorithm now prefers the lowest latency peers instead of the most stable
• Session key exchange logic has been changed to improve throughput and reduce occasional jitter

Fixed

• Bloom filter hashing now works correctly on big-endian architectures
• Incorrect buffer pool usage has been fixed, reducing memory allocations
• The multicast beacon interval now backs off correctly, reducing the number of beacons sent
• A denial-of-service vulnerability in the QUIC library has been fixed with a dependency update

Added

• A new peer option ?maxbackoff=X has been added to control the maximum backoff time for a given peer, supports duration values like 5m, 1h etc

Changed

• The maximum backoff period for failing peer connections has been reduced to just over 1 hour, compared to 4.5 hours before
• The getPeers endpoint now sorts peers in a more stable fashion
• Upgrade dependencies

Fixed

• A bug where QUIC listeners could stop listening for incoming connections unexpectedly has been fixed
• The priority tiebreak between multiple peerings to the same node has been fixed
• Peer connection ordering is no longer sensitive to poor system time resolution
• The admin socket now verifies the length of input public keys
• The PPROFLISTEN environment variable has been fixed and now starts the pprof listener correctly
• A panic in getPeers has been fixed when using abstract UNIX sockets on Linux

Fixed

• Fixed a crash that could happen when calculating the size of bloom filters during encoding

Fixed

• Fixed a data race from buffered pathfinder traffic
• Fix a bug where the next-hop selection may not take shortcuts through treespace
• Backoffs are now reset correctly when a successful handshake is completed
• Backoffs will no longer exceed roughly 4.5 hours when peers are down for a long time
• The -normaliseconf option will now work correctly with PrivateKeyPath
• Improved the reliability of QUIC peering setup by disabling 0-RTT

Added

• New -publickey command line option that prints the derived public key from a configuration file
• Support for connecting to TLS peers via SOCKS with the new sockstls:// link schema

Changed

• Stabilise tree parent selection algorithm
• Improved logging when the TUN interface fails to set up

Fixed

• Fixed a panic that could occur when a connection reaches an inconsistent error state
• The admin socket will now report more peering handshake error conditions in getPeers
• Yggdrasil will no longer panic at startup when duplicate peers are configured
• The build script will no longer incorrectly import LDFLAGS from the environment

Fixed

• Fix the Debian package so that upgrades are handled more smoothly

NOTE: If you are using the Debian package and upgrading an existing installation, you may need to update your AdminListen line in your configuration file to unix:///var/run/yggdrasil/yggdrasil.sock or delete the AdminListen line altogether.

Added

• Authenticated peering handshake with optional password, i.e.
  • For listeners: tls://[::]:12345?password=123456abcdef
  • For peers: tls://a.b.c.d:12345?password=123456abcdef
  • For multicast interfaces with the new Password option in each MulticastInterfaces section
  • Maximum password length is 64 characters
• QUIC support for peerings, by using the new quic:// scheme in Listen and Peers
  • This has not been extensively tested and may perform worse than TCP or TLS peers
• The private key can now be stored in PEM format separately to the main configuration file with the new PrivateKeyPath configuration file option
  • Use the -exportkey flag to export the key to a file from an existing config

Changed

• New routing scheme, which is backwards incompatible with previous versions of Yggdrasil
  • The wire protocol version number, exchanged as part of the peer setup handshake, has been increased to 0.5
  • Nodes running this new version will not be able to peer with earlier versions of Yggdrasil
  • A DHT is no longer used to map public keys and routes through treespace
  • Bloom filters are used to track on-tree links and nodes reachable via that link
  • Nodes now gossip separate per-link information which is tracked in CRDT structures, forcing local consistency and preventing unnecessary flapping when a route to the root node has changed or is broken
  • Greedy routing is once again used instead of source routing
  • Per-link keepalives have been replaced with periodic acknowledgements, reducing idle bandwidth
• The link handshake and multicast beacon formats have been revised for better future extensibility
• The link code has been refactored for more robust tracking of peering states
  • As a result, the admin socket is now able to report information about configured peerings that are down
  • Reconnect intervals are now tracked separately for each configured peer with exponential backoffs

Removed

• Yggdrasil will no longer request BBR congestion control for TCP and TLS peerings on Linux

Fixed

• Restored removePeer admin socket endpoint
• Fixed the RetryPeersNow API call for mobile

Fixed

• A bug which could result in high CPU usage after a network interface change has been fixed
• TLS listeners no longer require a TLS client certificate, as it is not necessary
• A panic in the mobile wrapper has been fixed when getting peers JSON

Added

• Authenticated peering handshake with optional password, i.e.
  • For listeners: tls://[::]:12345?password=123456abcdef
  • For peers: tls://a.b.c.d:12345?password=123456abcdef
  • For multicast interfaces with the new Password option in each MulticastInterfaces section
  • Maximum password length is 64 characters
• QUIC support for peerings, by using the new quic:// scheme in Listen and Peers
  • This has not been extensively tested and may perform worse than TCP or TLS peers
• The private key can now be stored in PEM format separately to the main configuration file with the new PrivateKeyPath configuration file option
  • Use the -exportkey flag to export the key to a file from an existing config

Changed

• New routing scheme, which is backwards incompatible with previous versions of Yggdrasil
  • The wire protocol version number, exchanged as part of the peer setup handshake, has been increased to 0.5
  • Nodes running this new version will not be able to peer with earlier versions of Yggdrasil
  • A DHT is no longer used to map public keys and routes through treespace
  • Bloom filters are used to track on-tree links and nodes reachable via that link
  • Nodes now gossip separate per-link information which is tracked in CRDT structures, forcing local consistency and preventing unnecessary flapping when a route to the root node has changed or is broken
  • Greedy routing is once again used instead of source routing
  • Per-link keepalives have been replaced with periodic acknowledgements, reducing idle bandwidth
• The link handshake and multicast beacon formats have been revised for better future extensibility
• The link code has been refactored for more robust tracking of peering states
  • As a result, the admin socket is now able to report information about configured peerings that are down
  • Reconnect intervals are now tracked separately for each configured peer with exponential backoffs

Removed

• Yggdrasil will no longer request BBR congestion control for TCP and TLS peerings on Linux

Added

• Dropped outbound peerings will now try to reconnect after a single second, rather than waiting up to 60 seconds for the normal peer timer

Changed

• Session encryption keys are now rotated at most once per minute, which reduces CPU usage and improves throughput on fast low latency links
• Buffers are now reused in the session encryption handler, which improves session throughput and reduces memory allocations
• Buffers are now reused in the router for DHT and path traffic, which improves overall routing throughput and reduces memory allocations

Fixed

• A bug in the admin socket where requests fail unless arguments is specified has been fixed
• Certificates on TLS listeners will no longer expire after a year
• The -address and -subnet command line options now return a useful warning when no configuration is specified

Added

• Support for prioritising multiple peerings to the same node has been added, useful for nodes with multiple network interfaces
  • The priority can be configured by specifying ?priority=X in a Peers or Listen URI, or by specifying Priority within a MulticastInterfaces configuration entry
  • Priorities are values between 0 and 254 (default is 0), lower numbers are prioritised and nodes will automatically negotiate the higher of the two values

Changed

• On Linux, SO_REUSEADDR is now used on the multicast port instead of SO_REUSEPORT, which should allow processes running under different users to run simultaneously

Fixed

• Adding peers using the InterfacePeers configuration option should now work correctly again
• Multiple connections from the same remote IP address will no longer be incorrectly dropped
• The admin socket will no longer incorrectly claim TCP connections as TLS
• A panic that could occur when calling GetPeers while a peering link is being set up has been fixed

Added

• Support for peering over UNIX sockets is now available, by configuring Listen and peering URIs in the unix:///path/to/socket.sock format

Changed

• yggdrasilctl has been refactored and now has cleaner output
• It is now possible to addPeer and removePeer using the admin socket again
• The getSessions admin socket call reports number of bytes received and transmitted again
• The link setup code has been refactored, making it easier to support new peering types in the future
• Yggdrasil now maintains configuration internally, rather than relying on a shared and potentially mutable structure

Fixed

• Tracking information about expired root nodes has been fixed, which should hopefully resolve issues with reparenting and connection failures when the root node disappears
• A bug in the mobile framework code which caused a crash on Android when multicast failed to set up has been fixed
• Yggdrasil should now shut down gracefully and clean up correctly when running as a Windows service

Fixed

• ICMPv6 "Packet Too Big" payload size has been increased, which should fix Path MTU Discovery (PMTUD) when two nodes have different IfMTU values configured
• A crash has been fixed when handling debug packet responses
• yggdrasilctl getSelf should now report coordinates correctly again

Changed

• Go 1.17 is now required to build Yggdrasil

Added

• bytes_sent, bytes_recvd and uptime have been added to getPeers
• Clearer logging when connections are rejected due to incompatible peer versions

Fixed

• Latency-based parent selection tiebreak is now reliable on platforms even with low timer resolution
• Tree distance calculation offsets have been corrected

This is an emergency release to correct build issues introduced in v0.4.1 and includes no other changes. Please see the Version 0.4.1 release notes for more information about the contents of this release.

Fixed

• Reverted a dependency update which resulted in problems building with Go 1.16 and running on Windows

Added

• TLS peerings now support Server Name Indication (SNI)
  • The SNI is sent automatically if the peering URI contains a DNS name
  • A custom SNI can be specified by adding the ?sni=domain.com parameter to the peering URI
• A new ipv6rwc API package now implements the IPv6-specific logic separate from the tun package

Fixed

• A crash when calculating the partial public key for very high IPv6 addresses has been fixed
• A crash due to a concurrent map write has been fixed
• A crash due to missing TUN configuration has been fixed
• A race condition in the keystore code has been fixed

Added

• New routing scheme, which is backwards incompatible with previous versions of Yggdrasil
  • The wire protocol version number, exchanged as part of the peer setup handshake, has been increased to 0.4
  • Nodes running this new version will not be able to peer with earlier versions of Yggdrasil
  • Please note that the network may be temporarily unstable while infrastructure is being upgraded to the new release
• TLS connections now use public key pinning
  • If no public key was already pinned, then the public key received as part of the TLS handshake is pinned to the connection
  • The public key received as part of the handshake is checked against the pinned keys, and if no match is found, the connection is rejected

Changed

• IP addresses are now derived from ed25519 public (signing) keys
  • Previously, addresses were derived from a hash of X25519 (Diffie-Hellman) keys
  • Importantly, this means that all internal IPv6 addresses will change with this release — this will affect anyone running public services or relying on Yggdrasil for remote access
• It is now recommended to peer over TLS
  • Link-local peers from multicast peer discovery will now connect over TLS, with the key from the multicast beacon pinned to the connection
  • socks:// peers now expect the destination endpoint to be a tls:// listener, instead of a tcp:// listener
• Multicast peer discovery is now more configurable
  • There are separate configuration options to control if beacons are sent, what port to listen on for incoming connections (if sending beacons), and whether or not to listen for beacons from other nodes (and open connections when receiving a beacon)
  • Each configuration entry in the list specifies a regular expression to match against interface names
  • If an interface matches multiple regex in the list, it will use the settings for the first entry in the list that it matches with
• The session and routing code has been entirely redesigned and rewritten
  • This is still an early work-in-progress, so the code hasn't been as well tested or optimized as the old code base — please bear with us for these next few releases as we work through any bugs or issues
  • Generally speaking, we expect to see reduced bandwidth use and improved reliability with the new design, especially in cases where nodes move around or change peerings frequently
  • Cryptographic sessions no longer use a single shared (ephemeral) secret for the entire life of the session. Keys are now rotated regularly for ongoing sessions (currently rotated at least once per round trip exchange of traffic, subject to change in future releases)
  • Source routing has been added. Under normal circumstances, this is what is used to forward session traffic (e.g. the user's IPv6 traffic)
  • DHT-based routing has been added. This is used when the sender does not know a source route to the destination. Forwarding through the DHT is less efficient, but the only information that it requires the sender to know is the destination node's (static) key. This is primarily used during the key exchange at session setup, or as a temporary fallback when a source route fails due to changes in the network
  • The new DHT design is no longer RPC-based, does not support crawling and does not inherently allow nodes to look up the owner of an arbitrary key. Responding to lookups is now implemented at the application level and a response is only sent if the destination key matches the node's /128 IP or /64 prefix
  • The greedy routing scheme, used to forward all traffic in previous releases, is now only used for protocol traffic (i.e. DHT setup and source route discovery)
  • The routing logic now lives in a standalone library. You are encouraged not to use it, as it's still considered pre-alpha, but it's available for those who want to experiment with the new routing algorithm in other contexts
  • Session MTUs may be slightly lower now, in order to accommodate large packet headers if required
• Many of the admin functions available over yggdrasilctl have been changed or removed as part of rewrites to the code
  • Several remote debug functions have been added temporarily, to allow for crawling and census gathering during the transition to the new version, but we intend to remove this at some point in the (possibly distant) future
  • The list of available functions will likely be expanded in future releases
• The configuration file format has been updated in response to the changed/removed features

Removed

• Tunnel routing (a.k.a. crypto-key routing or "CKR") has been removed
  • It was far too easy to accidentally break routing altogether by capturing the route to peers with the TUN adapter
  • We recommend tunnelling an existing standard over Yggdrasil instead (e.g. ip6gre, ip6gretap or other similar encapsulations, using Yggdrasil IPv6 addresses as the tunnel endpoints)
  • All TunnelRouting configuration options will no longer take effect
• Session firewall has been removed
  • This was never a true firewall — it didn't behave like a stateful IP firewall, often allowed return traffic unexpectedly and was simply a way to prevent a node from being flooded with unwanted sessions, so the name could be misleading and usually lead to a false sense of security
  • Due to design changes, the new code needs to address the possible memory exhaustion attacks in other ways and a single configurable list no longer makes sense
  • Users who want a firewall or other packet filter mechansim should configure something supported by their OS instead (e.g. ip6tables)
  • All SessionFirewall configuration options will no longer take effect
• SIGHUP handling to reload the configuration at runtime has been removed
  • It was not obvious which parts of the configuration could be reloaded at runtime, and which required the application to be killed and restarted to take effect
  • Reloading the config without restarting was also a delicate and bug-prone process, and was distracting from more important developments
  • SIGHUP will be handled normally (i.e. by exiting)
• cmd/yggrasilsim has been removed, and is unlikely to return to this repository