yggdrasil-go

This is the fourth, and hopefully final, release candidate build for Yggdrasil v0.4.

THIS BUILD CONTAINS A PROTOCOL BREAK AND IS NOT COMPATIBLE WITH PEERS RUNNING EARLIER RC OR NON-RC VERSIONS. MAKE SURE TO BACK UP YOUR CONFIGURATION FILE BEFORE INSTALLING!

Please note that your IPv6 address/subnet will change. Also note that tunnel routing and the session firewall are no longer supported.

Changes from RC3:

• Protocol version increased — will not connect to nodes running RC3 or lower anymore, but should connect to nodes running the eventual final v0.4 release version
• Multicast packet format changed
• MulticastInterfaces configuration option has a new format — sending and receiving multicast beacons are now configurable separately
• SOCKS connections now connect to TLS listeners
• Protocol bugfixes

This is the third release candidate build for Yggdrasil v0.4.

THIS BUILD CONTAINS A PROTOCOL BREAK AND IS NOT COMPATIBLE WITH PEERS RUNNING EARLIER NON-RC VERSIONS. MAKE SURE TO BACK UP YOUR CONFIGURATION FILE BEFORE INSTALLING!

Please note that your IPv6 address/subnet will change. Also note that tunnel routing and the session firewall are no longer supported.

Changes from RC2:

• Memory leak fixed
• Multicast peerings are now TLS by default
• TLS sessions now pin keys to the certificate if no other key is specified

This is the second release candidate build for Yggdrasil v0.4.

THIS BUILD CONTAINS A PROTOCOL BREAK AND IS NOT COMPATIBLE WITH PEERS RUNNING EARLIER NON-RC VERSIONS. MAKE SURE TO BACK UP YOUR CONFIGURATION FILE BEFORE INSTALLING!

Please note that your IPv6 address/subnet will change. Also note that tunnel routing and the session firewall are no longer supported.

Changes from RC1:

• Peer public keys are now pinned with ?key= instead of ?ed25519=
• Peer connections are dialled using a cancellable context
• Dependencies updated

This is the first release candidate build for Yggdrasil v0.4.

THIS BUILD CONTAINS A PROTOCOL BREAK AND IS NOT COMPATIBLE WITH PEERS RUNNING EARLIER NON-RC VERSIONS. MAKE SURE TO BACK UP YOUR CONFIGURATION FILE BEFORE INSTALLING!

Please note that your IPv6 address/subnet will change. Also note that tunnel routing and the session firewall are no longer supported.

Added

• New simulation code under cmd/yggdrasilsim (work-in-progress)

Changed

• Multi-threading in the switch
  • Swich lookups happen independently for each (incoming) peer connection, instead of being funneled to a single dedicated switch worker
  • Packets are queued for each (outgoing) peer connection, instead of being handled by a single dedicated switch worker
• Queue logic rewritten
  • Heap structure per peer that traffic is routed to, with one FIFO queue per traffic flow
  • The total size of each heap is configured automatically (we basically queue packets until we think we're blocked on a socket write)
  • When adding to a full heap, the oldest packet from the largest queue is dropped
  • Packets are popped from the queue in FIFO order (oldest packet from among all queues in the heap) to prevent packet reordering at the session level
• Removed global sync.Pool of []byte
  • Local sync.Pools are used in the hot loops, but not exported, to avoid memory corruption if libraries are reused by other projects
  • This may increase allocations (and slightly reduce speed in CPU-bound benchmarks) when interacting with the tun/tap device, but traffic forwarded at the switch layer should be unaffected
• Upgrade dependencies
• Upgrade build to Go 1.16

Fixed

• Fixed a bug where the connection listener could exit prematurely due to resoruce exhaustion (if e.g. too many connections were opened)
• Fixed DefaultIfName for OpenBSD (/dev/tun0 -> tun0)
• Fixed an issue where a peer could sometimes never be added to the switch
• Fixed a goroutine leak that could occur if a peer with an open connection continued to spam additional connection attempts

Added

• Support for pinning remote public keys in peering strings has been added, e.g.
  • By signing public key: tcp://host:port?ed25519=key
  • By encryption public key: tcp://host:port?curve25519=key
  • By both: tcp://host:port?ed25519=key&curve25519=key
  • By multiple, in case of DNS round-robin or similar: tcp://host:port?curve25519=key&curve25519=key&ed25519=key&ed25519=key
• Some checks to prevent Yggdrasil-over-Yggdrasil peerings have been added
• Added support for SOCKS proxy authentication, e.g. socks://user@password:host/...

Fixed

• Some bugs in the multicast code that could cause unnecessary CPU usage have been fixed
• A possible multicast deadlock on macOS when enumerating interfaces has been fixed
• A deadlock in the connection code has been fixed
• Updated HJSON dependency that caused some build problems

Changed

• DisconnectPeer and RemovePeer have been separated and implemented properly now
• Less nodes are stored in the DHT now, reducing ambient network traffic and possible instability
• Default config file for FreeBSD is now at /usr/local/etc/yggdrasil.conf instead of /etc/yggdrasil.conf

Fixed

• Fixes a memory leak that may occur if packets are incorrectly never removed from a switch queue

Changed

• Make DHT searches a bit more reliable by tracking the 16 most recently visited nodes

Added

• Support for the Wireguard TUN driver, which now replaces Water and provides far better support and performance on Windows
• Windows .msi installer files are now supported (bundling the Wireguard TUN driver)
• NodeInfo code is now actorised, should be more reliable
• The DHT now tries to store the two closest nodes in either direction instead of one, such that if a node goes offline, the replacement is already known
• The Yggdrasil API now supports dialing a remote node using the public key instead of the Node ID

Changed

• The -loglevel command line parameter is now cumulative and automatically includes all levels below the one specified
• DHT search code has been significantly simplified and processes rumoured nodes in parallel, speeding up search time
• DHT search results are now sorted
• The systemd service now handles configuration generation in a different unit
• The Yggdrasil API now returns public keys instead of node IDs when querying for local and remote addresses

Fixed

• The multicast code no longer panics when shutting down the node
• A potential OOB error when calculating IPv4 flow labels (when tunnel routing is enabled) has been fixed
• A bug resulting in incorrect idle notifications in the switch should now be fixed
• MTUs are now using a common datatype throughout the codebase

Removed

• TAP mode has been removed entirely, since it is no longer supported with the Wireguard TUN package. Please note that if you are using TAP mode, you may need to revise your config!
• NetBSD support has been removed until the Wireguard TUN package supports NetBSD

Added

• New API functions SetMaximumSessionMTU and GetMaximumSessionMTU
• New command line parameters -address and -subnet for getting the address/subnet from the config file, for use with -useconffile or -useconf
• A warning is now produced in the Yggdrasil output at startup when the MTU in the config is invalid or has been adjusted for some reason

Changed

• On Linux, outgoing InterfacePeers connections now use SO_BINDTODEVICE to prefer an outgoing interface
• The genkeys utility is now in cmd rather than misc

Fixed

• A data race condition has been fixed when updating session coordinates
• A crash when shutting down when no multicast interfaces are configured has been fixed
• A deadlock when calling AddPeer multiple times has been fixed
• A typo in the systemd unit file (for some Linux packages) has been fixed
• The NodeInfo and admin socket now report unknown correctly when no build name/version is available in the environment at build time
• The MTU calculation now correctly accounts for ethernet headers when running in TAP mode

Added

• Support for TLS listeners and peers has been added, allowing the use of tls://host:port in Peers, InterfacePeers and Listen configuration settings - this allows hiding Yggdrasil peerings inside regular TLS connections

Changed

• Go 1.13 or later is now required for building Yggdrasil
• Some exported API functions have been updated to work with standard Go interfaces:
  • net.Conn instead of yggdrasil.Conn
  • net.Dialer (the interface it would satisfy if it wasn't a concrete type) instead of yggdrasil.Dialer
  • net.Listener instead of yggdrasil.Listener
• Session metadata is now updated correctly when a search completes for a node to which we already have an open session
• Multicast module reloading behaviour has been improved

Fixed

• An incorrectly held mutex in the crypto-key routing code has been fixed
• Multicast module no longer opens a listener socket if no multicast interfaces are configured

Added

• The core library now includes several unit tests for peering and yggdrasil.Conn connections

Changed

• On recent Linux kernels, Yggdrasil will now set the tcp_congestion_control algorithm used for its own TCP sockets to BBR, which reduces latency under load
• The systemd service configuration in contrib (and, by extension, some of our packages) now attemps to load the tun module, in case TUN/TAP support is available but not loaded, and it restricts Yggdrasil to the CAP_NET_ADMIN capability for managing the TUN/TAP adapter, rather than letting it do whatever the (typically root) user can do

Fixed

• The yggdrasil.Conn.RemoteAddr() function no longer blocks, fixing a deadlock when CKR is used while under heavy load

Added

• Yggdrasil will now complain more verbosely when a peer URI is incorrectly formatted
• Soft-shutdown methods have been added, allowing a node to shut down gracefully when terminated
• New multicast interval logic which sends multicast beacons more often when Yggdrasil is first started to increase the chance of finding nearby nodes quickly after startup

Changed

• The switch now buffers packets more eagerly in an attempt to give the best link a chance to send, which appears to reduce packet reordering when crossing aggregate sets of peerings
• Substantial amounts of the codebase have been refactored to use the actor model, which should substantially reduce the chance of deadlocks
• Nonce tracking in sessions has been modified so that memory usage is reduced whilst still only allowing duplicate packets within a small window
• Soft-reconfiguration support has been simplified using new actor functions
• The garbage collector threshold has been adjusted for mobile builds
• The maximum queue size is now managed exclusively by the switch rather than by the core

Fixed

• The broken hjson-go dependency which affected builds of the previous version has now been resolved in the module manifest
• Some minor memory leaks in the switch have been fixed, which improves memory usage on mobile builds
• A memory leak in the add-peer loop has been fixed
• The admin socket now reports the correct URI strings for SOCKS peers in getPeers
• A race condition when dialling a remote node by both the node address and routed prefix simultaneously has been fixed
• A race condition between the router and the dial code resulting in a panic has been fixed
• A panic which could occur when the TUN/TAP interface disappears (e.g. during soft-shutdown) has been fixed
• A bug in the semantic versioning script which accompanies Yggdrasil for builds has been fixed
• A panic which could occur when the TUN/TAP interface reads an undersized/corrupted packet has been fixed

Changed

• Yggdrasil can now send multiple packets from the switch at once, which results in improved throughput with smaller packets or lower MTUs
• Performance has been slightly improved by not allocating cancellations where not necessary
• Crypto-key routing options have been renamed for clarity
  • IPv4Sources is now named IPv4LocalSubnets
  • IPv6Sources is now named IPv6LocalSubnets
  • IPv4Destinations is now named IPv4RemoteSubnets
  • IPv6Destinations is now named IPv6RemoteSubnets
  • The old option names will continue to be accepted by the configuration parser for now but may not be indefinitely
• When presented with multiple paths between two nodes, the switch now prefers the most recently used port when possible instead of the least recently used, helping to reduce packet reordering
• New nonce tracking should help to reduce the number of packets dropped as a result of multiple/aggregate paths or congestion control in the switch

Fixed

• Security vulnerability: Address verification was not strict enough, which could result in a malicious session sending traffic with unexpected or spoofed source or destination addresses which Yggdrasil could fail to reject
  • Versions 0.3.6 and 0.3.7 are vulnerable - users of these versions should upgrade as soon as possible
  • Versions 0.3.5 and earlier are not affected
• A deadlock was fixed in the session code which could result in Yggdrasil failing to pass traffic after some time

Changed

• The switch should now forward packets along a single path more consistently in cases where congestion is low and multiple equal-length paths exist, which should improve stability and result in fewer out-of-order packets
• Sessions should now be more tolerant of out-of-order packets, by replacing a bitmask with a variable sized heap+map structure to track recently received nonces, which should reduce the number of packets dropped due to reordering when multiple paths are used or multiple independent flows are transmitted through the same session
• The admin socket can no longer return a dotfile representation of the known parts of the network, this could be rebuilt by clients using information from getSwitchPeers,getDHT and getSessions

Fixed

• A number of significant performance regressions introduced in version 0.3.6 have been fixed, resulting in better performance
• Flow labels are now used to prioritise traffic flows again correctly
• In low-traffic scenarios where there are multiple peerings between a pair of nodes, Yggdrasil now prefers the most active peering instead of the least active, helping to reduce packet reordering
• The Listen statement, when configured as a string rather than an array, will now be parsed correctly
• The admin socket now returns coords as a correct array of unsigned 64-bit integers, rather than the internal representation
• The admin socket now returns box_pub_key in string format again
• Sessions no longer leak/block when no listener (e.g. TUN/TAP) is configured
• Incoming session connections no longer block when a session already exists, which results in less leaked goroutines
• Flooded sessions will no longer block other sessions
• Searches are now cleaned up properly and a couple of edge-cases with duplicate searches have been fixed
• A number of minor allocation and pointer fixes

Added

• Yggdrasil now has a public API with interfaces such as yggdrasil.ConnDialer, yggdrasil.ConnListener and yggdrasil.Conn for using Yggdrasil as a transport directly within applications
• Session gatekeeper functions, part of the API, which can be used to control whether to allow or reject incoming or outgoing sessions dynamically (compared to the previous fixed whitelist/blacklist approach)
• Support for logging to files or syslog (where supported)
• Platform defaults now include the ability to set sane defaults for multicast interfaces

Changed

• Following a massive refactoring exercise, Yggdrasil's codebase has now been broken out into modules
• Core node functionality in the yggdrasil package with a public API
  • This allows Yggdrasil to be integrated directly into other applications and used as a transport
  • IP-specific code has now been moved out of the core yggdrasil package, making Yggdrasil effectively protocol-agnostic
• Multicast peer discovery functionality is now in the multicast package
• Admin socket functionality is now in the admin package and uses the Yggdrasil public API
• TUN/TAP, ICMPv6 and all IP-specific functionality is now in the tuntap package
• PPROF debug output is now sent to stderr instead of stdout
• Node IPv6 addresses on macOS are now configured as secured
• Upstream dependency references have been updated, which includes a number of fixes in the Water library

Fixed

• Multicast discovery is no longer disabled if the nominated interfaces aren't available on the system yet, e.g. during boot
• Multicast interfaces are now re-evaluated more frequently so that Yggdrasil doesn't need to be restarted to use interfaces that have become available since startup
• Admin socket error cases are now handled better
• Various fixes in the TUN/TAP module, particularly surrounding Windows platform support
• Invalid keys will now cause the node to fail to start, rather than starting but silently not working as before
• Session MTUs are now always calculated correctly, in some cases they were incorrectly defaulting to 1280 before
• Multiple searches now don't take place for a single connection
• Concurrency bugs fixed
• Fixed a number of bugs in the ICMPv6 neighbor solicitation in the TUN/TAP code
• A case where peers weren't always added correctly if one or more peers were unreachable has been fixed
• Searches which include the local node are now handled correctly
• Lots of small bug tweaks and clean-ups throughout the codebase

Fixed

• The AllowedEncryptionPublicKeys option has now been fixed to handle incoming connections properly and no longer blocks outgoing connections (this was broken in v0.3.4)
• Multicast TCP listeners will now be stopped correctly when the link-local address on the interface changes or disappears altogether

Added

• Support for multiple listeners (although currently only TCP listeners are supported)
• New multicast behaviour where each multicast interface is given it's own link-local listener and does not depend on the Listen configuration
• Blocking detection in the switch to avoid parenting a blocked peer
• Support for adding and removing listeners and multicast interfaces when reloading configuration during runtime
• Yggdrasil will now attempt to clean up UNIX admin sockets on startup if left behind by a previous crash
• Admin socket getTunnelRouting and setTunnelRouting calls for enabling and disabling crypto-key routing during runtime
• On macOS, Yggdrasil will now try to wake up AWDL on start-up when awdl0 is a configured multicast interface, to keep it awake after system sleep, and to stop waking it when no longer needed
• Added LinkLocalTCPPort option for controlling the port number that link-local TCP listeners will listen on by default when setting up MulticastInterfaces (a node restart is currently required for changes to LinkLocalTCPPort to take effect - it cannot be updated by reloading config during runtime)

Changed

• The Listen configuration statement is now an array instead of a string
• The Listen configuration statement should now conform to the same formatting as peers with the protocol prefix, e.g. tcp://[::]:0
• Session workers are now non-blocking
• Multicast interval is now fixed at every 15 seconds and network interfaces are reevaluated for eligibility on each interval (where before the interval depended upon the number of configured multicast interfaces and evaluation only took place at startup)
• Dead connections are now closed in the link handler as opposed to the switch
• Peer forwarding is now prioritised instead of randomised

Fixed

• Admin socket getTunTap call now returns properly instead of claiming no interface is enabled in all cases
• Handling of getRoutes etc in yggdrasilctl is now working
• Local interface names are no longer leaked in multicast packets
• Link-local TCP connections, particularly those initiated because of multicast beacons, are now always correctly scoped for the target interface
• Yggdrasil now correctly responds to multicast interfaces going up and down during runtime