Highlights

This new release brings many new and exciting features and improvements including:

• New build mode to hide the non-glibc symbols and libstdc++
• ZFS code extracted out of the kernel in form of a dynamically linked library libsolaris.so
• Build mode intended to tailor the kernel to a set of specific device drivers - driver profiles
• Mechanism to build a version of the kernel with a subset of glibc symbols needed to support a specific application
• Experimental lazy stack support
• Major improvements to building and running ZFS images
• Major improvements to the aarch64 port like dedicated exception stack and support to handle system calls and signals
• Support new system calls
• Support the SO_REUSE_PORT socket option
• Minimal rnetlink support
• Various VFS enhancements

Enhancements and Improvements in Detail

• Hide most symbols and standard C++ library
  • hide most symbols using -fvisibility=hidden and expose the public ones using (__visibility__("default")) - 2df86521c874b6a0eaf7755f17870b8130b5ceb9, 317d259ab5b0b49a1a114bc837147746e471abc9
  • add lists of exported symbols
  • add option to build kernel with most symbols hidden - af2d371a61f6ab1eb5a066a0c3e93230faf6611c
  • enable garbage collection of code and data when hiding symbols - 730fd82ef76d7f4d201fe60960c31fa8d3766371
• Improve modularity
  • extract ZFS code as external dynamic library libsolaris.so - d7a627484b34aa8254d8fe444972aa18fe007ca7
  • httpserver-monitoring-api: stop using kernel internal C++ API - 891bfcd06918474cfa3aca5c5329ef8248d3a939
  • add new C-wrappers to expose the module API - 182499cb6fee68c2ac6fb6d37f84773f79b26f4c
• Driver Profiles
  • support driver profiles - ffb91b1ef9fd15e340ab603de2ea0969d5273552
• App-specific kernel
  • support app version script - d19ccb1cde100ab4a5c8e6db9a0d69560cabbd04
• Enhance networking stack
  • socket: implement the Linux flavor of SO_REUSE_PORT option - 985d4d767ab968d22c9ea0732a07a9399320369b
  • routecache: handle the case when sent to non-loopback address - a0251df210a6739830bd37e52211268f5672a27c
• Aarch64 enhancements
  • move kernel to 63rd GB of virtual memory - 850ce0e32c2bc444ef1a56695e9c19776b5c5411
  • handle system calls - 5666619470c2b051c5e03a8659609605be55d4b4
  • handle exceptions on dedicated stack - a6fbc361adf239dbfd50c35b4202e1f680c81fb8
  • force TLB flush when mprotect changes permission - fd2435925a9b8b680cf38f254c057bdc2064a4ba
  • implement signal handler - d8d271914ea846ace1763cc2e960e47d7342a4ee
  • fix zfs support - ea9cb449b516e0f209a8bd9049218cd487062f07
• Improve building and running ZFS images
  • new zfs_builder to create ZFS images - f935ca14ff2c46b5cbcc36ff35f1a44b279983be
  • vfs: support mounting ZFS from non-root partition - 9cc49b385637f3e5233d869a9c9428bfbc1d4222
  • support building rofs+zfs image and second ZFS disk - f6e09c0d95018ee525ed592a471c5a929b055f11
  • allow mounting and building on host - 6aa1053c3e38fb9f4f2cc3b55a3eed8873fba08a
• New syscalls
  • add getgid, getuid, lseek and statfs - b9c3210c1548d9d65491766510e868d923f73cca
  • implement getcwd
  • implement getdents64 - 6716ad91bd32762624935cd31d57eea406563e81
• Netlink support
  • minimal Linux rtnetlink support - 3f4960276b92a256976850be2887b2ec055ee239
  • my other patches to fix bugs and improve - f8c1f96cdad1d494f60f0e7fe81c8c10d2f57a9a, 73d566405b78cca3cae059bc943c7c3860e7162e, 91609107b1105601db63c351abe876d443dd23ee, aed28862b8ca134cfbad37c2d25b6dcb8e737dad, ad0c9d8d1b1a2b6f0ccd9f661e3e33f41ae89c63, 508b73c29eaff2909d2bc4d6438250456dff9b4f
• VFS improvements
  • implement symlinkat - 504b1bb8dd0fb6f952da782840ee82a432ec0f60
  • fill gaps in unlinkat - b4512b7f4fd6e2bcfc359f19b14e0d9cb11b19da
  • implement renameat - 1097dfa069fa0cdc62339943c52ed84e3e9dfe1b
• Sysfs improvements
  • add new OSv specific pseudo file /sys/osv/memory/linear_maps - 2847b2a0449d01b129627e6a96b2005094b5461a
• Libc improvements
  • implement _dl_find_object() - dfa73ee35f7ecc3f604ca094b37602d71b212c59
  • use musl implementation of getifaddrs and if_nameindex - e6ab059dd2f735ac895ebf70e820c4f58e358555
  • fix bugs in if_nametoindex/if_indextoname
• Lazy stack support
  • inline assembly to pre-fault stack - f5684d9c3f4f8d20a64605cfe66fd51771754256
  • do nothing in kernel threads and on populated stack - fabacc49db5efe26c8c9e93a466ee6fb02fda151
  • ensure next stack page statically - 9b1e9b0daf70b902c9afda7ee1cf5d3be0c09e22
  • ensure next stack page dynamically if preemption enabled - 7c9c4c370602307077500da235f5c672fb7db2f8
  • ensure next stack page conditionally if interrupts and preemption enabled - 52505124c3912a37a9d81e59805c93963575a5d0
  • prevent deadlock when taking vma_list_mutex for write - 9fb4574076fdd5617ba451885daaf18c39f3ad21
  • activate lazy stack in pthreads - d6aacabe73e2d40cdaef6dc8455786d4a57bbd57
  • new tracepoint for stack pre-faults - c8cdbde1790aa35065036169643a254e605d2445
• Support building on Fedora 36
• Scripts/tooling improvements
  • support flame graphs - 2c342c578f61cb96267455732abd3f8555c05df3
  • lua: change build process to download artifacts from lua binaries - 05c06bafd55e96419171147a44fb52f635e65ec9
  • trace.py: add option to show multiline backtrace - 1970fd6c40ce538e0d8d2fb03a6a2a60b8882829
• New apps/runtimes supported
  • memory: move malloc virtual address space below 0x800000000000 - 6de9f41db72cfa887140ec1a76039519a2c54910
  • support Java on aarch64
  • support SeaWeedFS

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing many patches and providing guidance for many others
• Waldemar Kozaczuk for contributing most patches

Closed Issues

• #1200 Mounting ZFS filesystem from arbitrary device and partition
• #1198 OSv fails to build on Fedora 36
• #1196 Proposal to change virtual memory layout
• #1195 aarch64: crashes when enabling tracepoint sched_switch
• #1190 Crash in the FreeBSD networking stack code caused by likely bug in reference counting logic
• #1189 aarch64: atomic_fetchadd_int() and atomic_fetchadd_long() in machine/atomic.h sometimes yield incorrect results
• #1188 Was trying to build and run the distributed file system called SeaWeedFS.
• #1185 Strange application behaviour difference between host and unikernel
• #1183 Please mention that OSv is open-source on your front page documentation
• #1182 Please host the source code of your website on github
• #1181 Please enable HTTPS on osv.io documentation
• #1180 Assertion failure when reading 2048 bytes from block device
• #1177 Test failure in tracing_smoke_test
• #1176 Gdb "osv info threads" doesn't work on Fedora 34
• #1175 Test failure on Fedora 34 because of gdb script failure
• #1172 No rules to make target 'osv.vmdk'.
• #1170 Implement Linux version of SO_REUSEPORT
• #1166 Revamp/improve building of Lua module
• #1165 creating temporary files takes a long time on Java 11
• #1160 virtio: cloud-hypervisor report warnings in virtio-code
• #1158 aarch64: tst-sampler.cc crashes sometimes in SMP mode
• #1157 aarch64: openjdk11 crashed on OSv in JIT mode
• #1156 aarch64: implement system call assembly instruction svc
• #1155 aarch64: consider implementing dedicated stacks like on x64
• #1154 aarch64: implement signal handler
• #1153 tst-mmap.cc hangs on aarch64
• #1152 tst-elf-permissions.cc fails on aarch64
• #1151 tst-sigaltstack.so fails on aarch64
• #1145 aarch64: openjdk8 crashed on OSv
• #1135 mmap() should handle collisions with areas mapped with mmu::linear_map()
• #1131 aarch64: ZFS image crashes due to a page fault caused by enabled access flag
• #1119 Missing mkdirat() function and syscall
• #1116 better handle exceptions thrown by dynamic linker
• #1110 Modularization/"Librarization" - create toolchain to optionally build custom kernel tailored to specific hypervisor or app
• #1096 Bugs in if_nametoindex/if_indextoname bug
• #1087 aarch64: dynamically map kernel code in early boot
• #1082 Build process can't find Lua libraries
• #1068 Building a full OSv image without running it
• #1009 Make ZFS optional as a shared library
• #97 Be more selective on symbols exported from the kernel

This new release brings new modern version of musl (1.1.24), greatly improved support of aarch64 architecture and many improvements of Virtio-FS including DAX window support to speed-up read I/O.

Highlights

• Musl
  • Upgraded from 0.9.12 to 1.1.24
  • Added new Bionic unit tests
• AArch64
  • Added Cadence UART driver
  • Added support to compile kernel natively on aarch64 host
  • Added support to cross-compile kernel on x64 Ubuntu and CentOS 7
  • Added support to build modules
  • Implemented mmio serial console for Firecracker
  • Added floating-point support
  • Supports ROFS
  • Greatly improved debugging
  • Fixed debug build which involved implementing context switch logic in assembly
  • Got most unit tests to pass
  • Added support of static TLS (Thread Local Storage)
  • Implemented PL031 driver
  • Added support of Virtio-FS
  • Improved SMP support (#vCPUs >= 2)
  • Tested on QEMU in TCG mode (full emulation)
  • Tested on QEMU and Firecracker with KVM on real ARM hardware - Raspberry PI 4 and Odroid N2 Plus
  • Tested many new apps:
    • nginx
    • rust examples
    • java 11
    • graalvm
    • python
    • iperf
    • ffmpeg
    • node
    • REST cli app
    • lighttpd
• Virtio-FS
  • Boots application from virtio-fs
  • Implemented DAX
  • Implemented readdir
• Added support to build with GCC 10 and 11
• VFS
  • Added mkdirat
  • Added statx
  • Added minimal unlinkat
• Hypervisors
  • Boots on Intel's cloud-hypervisor - see this Wiki for details
• Apps/runtimes
  • Latest golang (1.15.x)
  • nginx from host

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing many patches and providing guidance for many others
• Fotis Xenakis for further improving Virtio-FS support (most notably DAX window)
• Stewart Hildebrand for making OSv boot on LYNX Secure hypervisor and fixing many AAarch64-related bugs
• Waldemar Kozaczuk for contributing most patches

Commits by Author

David Smith (1):

• Fix mkdir syscall issue and implement mkdirat

David Woodhouse (1):

• x64: Parse X2APIC CPU definitions in MADT table

Fotis Xenakis (29):

• docker: update default to ubuntu 20.04
• virtio-fs: update fuse protocol header
• virtio-fs: implement FUSE_INIT map_alignment field
• virtio-fs: add basic read using the DAX window
• virtio-fs: refactor driver / fs
• virtio-fs: implement dax window manager
• virtio-fs: use multiple dax mappings in filesystem
• virtio-fs: minor driver code improvements
• virtio-fs: fix lost wakeup in driver
• virtio-fs: fix allocation failure condition
• virtio-fs: expose FUSE reply length
• virtio-fs: use FUSE_OPENDIR and FUSE_RELEASEDIR
• virtio-fs: implement readdir
• virtio-fs: add error check in fs
• virtio-fs: use exclusive device ids
• vfs: homogenize mount_rofs_rootfs and mount_zfs_rootfs
• loader: add rootfs type command line option
• loader: add support for booting from virtio-fs
• scripts/build: don't exit when exporting files
• virtio: fix return type of get_driver_features()
• virtio-fs: set the VIRTIO_F_VERSION_1 feature bit
• scripts: add kernel command-line options to run.py
• scripts: enhance virtiofsd functionality in run.py
• virtio-fs: update fuse protocol header
• virtio-fs: make DAX manager testable
• tests: add DAX manager white-box unit tests
• loader: extend root fs auto-discovery
• fs: support d_off in readdir for all filesystems
• scripts: add virtio-fs DAX option to run.py

Inacio Klassmann (1):

• fixed NameError: name 'file' is not defined

Matthew Kenigsberg (1):

• Handle newline in /etc/os-release

Nadav Har'El (23):

• libosv.so: fix warnings on truncated symbol names
• memset_base(): avoid warnings in stricter C++ compiler
• lzloader: fix memset() implementation
• setup.py: support Fedora 32
• Add new mechanism for function aliases - outside original source file
• syscall_to_function.h: simplify, and avoid C++20 extension
• scripts/setup.py: simplify Fedora's Lua requirement
• musl: fix __stdio_read.c to conform to new Musl
• Add test for fread() hang when reading from a directory
• fix compatibility features.h for Fedora 34
• Fix __cxa_finalize() signature
• bsd/sha2.c: fix mismatched signatures
• Fix warnings about misleading indentation
• xen: fix overread and overwrite bug
• Makefile: avoid warning in one Musl source file
• fs/vfs/main.cc: fix gcc 11 build error
• core/trace.cc: free aligned_alloc() with free()
• sched: free aligned_alloc() with free()
• core/mmu.cc: silence warning
• add __libc_single_threaded symbol
• scripts/setup.py: add Fedora 34
• tests: fix warning in tst-epoll.cc
• Implement statx() function

Punit Agrawal (3):

• scripts: Update script to download RPM packages
• Update boost library detection logic for cross compiling
• scripts/build: Simplify libgcc_s.so.1 detection

Robin Gögge (1):

• scripts: fix setup.py for Ubuntu versions

Samuel Laberge (3):

• Make getpid() return non-zero number
• Fix kill tests for non-zero PID
• Reject executables with non-matching machine type

Shawn Barber (1):

• Modifying times function to gracefully handle being passed a null pointer

Stewart Hildebrand (19):

• build: set arch in arg-parsing loop
• Add script to download aarch64 cross compiler toolchain
• aarch64: re-introduce dtb relocation
• aarch64: add function to get PSCI method from device tree
• aarch64: use PSCI method from device tree
• Introduce Cadence UART driver
• Cadence: initialize from device tree
• Cadence: add barrier
• aarch64: move fpu_state_save/load to entry.S
• boost: add missing library
• setup.py: add readelf/grep
• aarch64: populate image size in header
• scripts/boost: update URL
• aarch64: Support downloading packages on Ubuntu 18.04 x64
• aarch64/boot.S: fix preboot pagetables
• build: use CROSS_PREFIX if defined
• virtio-mmio: add NULL check in register_mmio_devices
• aarch64: use proper data size for __loader_argc
• Cadence: add device tree parsing function

Waldemar Kozaczuk (201):

• Merge pull request #1107 from punitagrawal/master
• Merge pull request #1133 from codegik/fix-open-file
• lua: added missing lpeg library
• setup.py: support Ubuntu 20.04
• maven: fix "recursive expression cycle" problem
• pthreads: expose pthread_attr_getscope() symbol properly
• serial console: refactor common logic to allow re-use when implementing mmio version
• arm: add memory-mapped isa console
• arm mmio serial console: use id passed in DTB for irqid instead of a hardcoded number
• arm on firecracker: detect and enable mmio isa early console
• arm on firecracker: map first 6GB one-to-one during early boot
• arm on firecracker: clean cmdline from early console artifacts
• arm on firecracker: recognize GIC v2 on arm and skip parsing PCI if count 0
• aarch64: until MSI-X support becomes functional enable regular PCI interrupt
• scripts: use common binary partition header file to build disk images
• scripts: support running aarch64 rofs images
• aarch64: until MSI-X support becomes functional do not try to setup vqueue MSI-X vector
• debugging: fix loader.py to handle "osv runqueue" properly
• debugging: enhance loader.py to support both x64 and aarch64
• dynamic linker: fix the old bug in the logic to calculate the ELF base
• scripts/run.py: do not hardcode memory size and cpu count for aarch64
• scripts/build: refine build process for aarch64
• aarch64: support virtio mmio devices
• aarch64: parse and activate virtio mmio devices
• aarch64: support building unit tests image
• aarch64: fix memmove bug
• licenses: add newlib license
• scripts: update firecracker.py to support aarch64
• scripts: make firecracker.py default to host architecture
• tests: fix shell if/else syntax to make it work on both Ubuntu and Fedora
• scripts: enhance run.py to allow running aarch64 images with KVM enabled
• tests: do not include java tests for aarch64
• libc: revert some files with "#undef" to point back to musl
• libc: revert some files with "__restrict" to point back to musl
• libc: remove more string functions from libc/ in favor of the original musl ones
• libc: revert more files with "#undef" to point back to musl
• libc: remove unused libc/regex/glob.c
• libc: replace two more files with the musl copies
• libc: replace unistd/ttyname.c with the musl copy
• libc: replace identical headers with the symlinks to musl copies
• libc: replace weak_alias and wrapper functions with linker symbol copy mechanism
• libc: replace more include/api/ headers with their musl copies
• libc: point 3 more files from libc/ to musl/
• libc: remove unused file
• libc: replace __strndup.c by adding entry to aliases.ld
• libc: replace network/gai_strerror.cc with its identical C copy from musl
• libc: replace 5 more headers in include/api with musl copies
• fs:change uiomove() to use size_t in the signature instead of int
• libc: replace some headers libc/internal with symlinks to identical copies under musl/src/
• tests: add verbatim copies of Bionic tests for w/ctype
• tests: adapted w/ctype tests to work with boost unittest and added tst-string
• libc: replace iswctype_l and wctype_l.c with musl equivalent
• libc: drop OSv toupper_l in favor of musl copy and add alias __toupper_l
• libc: drop towlower_l and towupper_l in favor of musl copies
• libc: drop setlocale.c and duplocale.c and add __setlocale alias
• libc: replace locale/wcsxfrm*.c (from old musl) with musl copy of locale/wcsxfrm.c
• libc: replace locale/wcscoll*.c (from old musl) with musl copy of locale/wcscoll.c
• libc: replace locale/strxfrm*.c (from old musl) with musl copy of locale/strxfrm.c
• libc: replace locale/strfmon.c with the musl copy
• libc: replace locale/strcoll*.c (from old musl) with musl copy of locale/strcoll.c
• libc: replaced 6 files in /libc/stdio/ that uses syscall instruction with musl copies and macros
• tests: fix compilation of tst-wctype.cc on older versions of gcc
• libc: replace floatscan.c, intscan.c and shgetc.c with the symlinks to its almost identical copies
• libc: replace string/strsignal.c with musl copy
• tests: added bionic-based unit test to test time libc functions
• libc: copy time/__tz.c from musl as is
• libc: replaced most libc/time/ files with their musl copies
• libc: replace langinfo with musl copies
• libc: replaced wcsftime files with their almost identical musl copies
• bsd: rename __unused* macros to __bsd_unused* to avoid conflicts with musl headers
• libc: replace 8 headers in include/api with their symlinked copies
• api: replace 3 more include/api headers with musl copies
• libc: replace include/api/x64/bits/socket.h with a symlink to musl copy
• libc: replace include/api/sys/eventfd.h with symlink to musl copy
• libc: replace libc/internal/libm.h with a symlink to musl original
• libc: replace include/api/aarch64/bits/mman.h with a symlink to musl copy
• libc: drop 5 more headers with symlinks to musl copies
• headers: get rid of fenv.h copy
• includes: fixed fenv.h symlink to point to the right musl directory
• tests: added bionic based tst-stdio.cc unit test
• rename musl submodule to musl_0.9.12 and add new submodule musl_1.1.24
• Upgrade musl from 0.9.12 to 1.1.24
• tests: pass AARCH64_PORT_STUB macro when building aarch64 image
• tests: enhance test.py to support running unit tests on aarch64
• aarch64: implement feenableexcept/fedisableexcept/fegetexcept
• tests: add simple tst-setjmp.cc to verify setjmp/longjmp/sigsetjmp/siglongjmp functionality
• aarch64: implement sigsetjmp/siglongjmp
• aarch64: fix memory mapping initialization when booting with #vCPUs >= 2
• tests: make TLS related tests build for aarch64
• tests: Fixed dependabot junit security alert
• elf: refine logging
• aarch64: support static TLS (Thread Local Storage)
• scripts: fixed build-capstan-mpm-packages to use correct parameters with new capstan 0.5
• firecracker: upgrade to the latest 0.23.0
• virtio: removed commented out "sleep()" from virtio::fs() constructor
• replaced offensive names of variables and methods with better ones
• tests: enable tst-feexcept when running with KVM on
• build: do not run maven for java modules if nothing changes
• tests: make irrelevant build messages very quiet
• aarch64: implement dump_registers()
• aarch64: store ESR (Exception Syndrom Register) in the exception frame
• x64: fix assembly warning in vmlinux-boot64.S
• scripts: add optional mode option to manifest_from_host.sh
• java: make libfreeblpriv3.so optional for openjdk8-from-host module
• setup: support Ubuntu 20.10
• scripts: make test script work with latest Python on Fedora 33
• setup: support Fedora 33
• libc: add alias of getauxval
• build: enforce Partial RELRO for internal apps
• aarch64: properly handle unexpected synchronous exceptions
• vfs: add minimal implementation of unlinkat
• aarch64: update syscall.h with correct numbers
• kernel: enforce object files are compiled with '-fno-pie'
• makefile: added comments justifying building kernel objects with "-fno-pie"
• aarch64: synchronize data and instruction caches after loading code into memory
• travis: upgrade to newer docker base image - Ubuntu 20.10
• aarch64: do not disable rofs cache in run.py
• bootchart: do not print empty entries
• aarch64: make boot messages quieter and consistent with x86_64
• aarch64: implement processor_to_nano in armclock
• setup: install aarch64 C++ cross-compile packages for x86_64 hosts only
• aarch64: support downloading aarch64 packages on Ubuntu x64
• aarch64: support compiling kernel natively on aarch64 host
• scripts: enhance manifest_from_host.sh to work on aarch64 host
• syscalls: ignore missing SYS_mkdir for aarch64
• aarch64: enhance setup.py to install multi-arch gdb package on Ubuntu
• include: support compiling OSv on CentOS 7
• modules: add common makefile to allow cross-compiling
• tests: simplify makefile by including new modules/common.gmk
• libtools: include common.gmk
• httpserver-monitoring-api: include common.gmk
• httpserver-api: include common.gmk
• tests/tst-threadcomplete.cc: disable part testing pinning when cpu == 1
• aarch64: fix loader.py to use correct IP register
• aarch64: handle R_AARCH64_COPY relocations
• scripts/run.py: use QEMU_PATH to use arbitrary qemu executable
• tests: handle more generic CROSS_PREFIX for aarch64 builds
• aarch64: make dl_tests makefile build for aarch64
• boost: enhance main makefile to build against arbitrary boost
• boost: enhance modules/common.gmk to build against arbitrary boost
• setup: support CentOS 7
• scripts: download and build arbitrary version of boost
• scripts: allow building QEMU from sources
• scripts: make firecracker.py point to artifacts in the last
• aarch64: align up size of the kernel TLS to 64 bytes
• aarch64: fix TLS kernel variables access from apps
• aarch64: fix building ramfs images on Fedora
• aarch64: use memory barrier to force completion of PTE writes
• tests: improve makefile by using more portable case statement
• aarch64: make kernel_vm_base equal to kernel_base
• aarch64: implement reschedule_from_interrupt() in assembly
• aarch64: re-enable unit tests fixed by latest patches
• arch64: enable sampler and corresponding unit test
• tst-stdio.cc: disable slow fread_unbuffered_pathological_performance test
• aarch64: change pthread_mutexattr_t, pthread_condattr_t and pthread_barrierattr_t to unsigned int
• tst-condvar.cc: run SMP test only if #vCPUs >= 2
• aarch64: add thread_main to improve stacktraces
• aarch64: add CFI directives to the page fault and interrupt handler assembly
• loader.py: improve "osv info threads"
• aarch64: mask interrupt and disable timer in the handler
• aarch64: implement minimal PL031 support
• aarch64: enable detection of virtio-fs driver
• scripts: enhanced download_and_build_qemu.sh to build QEMU from arbitrary repo and enable virtiofsd
• aarch64: unmask clock event interrupt on secondary CPUs
• dynamic linker: properly handle STT_NOTYPE symbols
• mempool: resize l1_pool_stats once
• aarch64: handle AT_HWCAP in getauxval
• aarch64: make RCU work correctly in SMP mode
• libc: add strerrordesc_np
• syscalls: minimally implement tgkill
• libc: add brk()/sbrk() stubs
• travis: authenticate to docker hub
• java tests: upgrade commons-io to 2.7 to silence github.com warning
• travis: attempt to enable KVM
• tests: fix style issues in tst-kill.cc
• tests: fix tst-kill.cc
• scripts: fix trace.py
• aarch64: make wait_record with weak memory model
• aarch64: make queue_mpsc work with weak memory model
• aarch64: make synch_thread/synch_port work with weak memory model
• sigaction: handle null sa_sigaction as SIG_DFL
• scripts/build: default to rofs when building aarch64 images
• aarch64: enable test_net.py and test_tracing.py
• aarch64: do not probe for pci if opt_pci_disabled
• scripts: update firecracker.py to properly support aarch64
• java: add OpenJDK module for java 9 and above
• openjdk8: make it work with aarch64
• aarch64: make java-base and related modules compile
• tst-async.cc: include to silence error in GCC 11
• boost: update path of function_output_iterator to silence warnings
• futex: implement FUTEX_WAIT_BITSET
• acpi: capture ACPI root address when booting in PVH mode
• tests: disable java tests when cross-compiling to aarch64
• travis: enable running unit tests
• travis: disable running tst-time.so
• aarch64: on Fedora 34 download the aarch64 artifacts from the version 33
• vfs: harden drele() to check for null v_op and vop_inactive
• cirp: pass --travis-api-url https://api.travis-ci.com to work a round a bug in cirp plugin
• setup: add Ubuntu 21.04 distribution
• scripts: updated test.py to reflect most up-to-date state of tests on aarch64
• replace AARCH64_PORT_STUB with aarch64 and x86_64

Closed Issues

• #1148 vfs: missing statx
• #1147 futex: missing implementation of FUTEX_WAIT_BITSET
• #1146 acpi: detect the root table pointer using the address field rsdp_paddr passed during PVH boot
• #1144 Build on Fedora 34
• #1142 [firecracker][aarch64] This code only works correctly for OSV_KERNEL_VM_SHIFT = 0x40000000 or 0
• #1134 aarch64: the do_main_thread hangs in osv::rcu_flush() in SMP mode
• #1133 fixed NameError: name 'file' is not defined
• #1132 aarch64: unmask interrupts on non-primary cores
• #1130 aarch64: tst-threadcomplete.cc hangs most of the time with 2 vCPUs or more
• #1129 aarch64: support LSE instructions
• #1127 aarch64: some unit tests occasionally hang when running on QEMU in emulated mode (TCG)
• #1126 aarch64: does not build on Fedora 31
• #1125 aarch64: does not build on Ubuntu 20.04/10
• #1124 aarch64: kernel built in debug mode crashes with page faults
• #1123 aarch64: early boot hangs sometimes in SMP mode (#vCPUs >=2)
• #1122 aarch64: TLS kernel variables incorrectly accessed by applications
• #1121 aarch64: tst-pin.cc crashes with cpus >=2
• #1120 aarch64: boot fails with debug build due to TLS assert
• #1117 aarch64: virtio::register_mmio_devices() sometimes page faults on real hardware like RockPro64 or Odroid
• #1115 Boot failure in Fedora 33
• #1114 tst-stdio-rofs fails in x86_64 too
• #1113 corrupt ELF notes in loader.elf generated on Fedora 33 causes PVH boot failure
• #1112 arch_init_early_console crashes when kernel built with certain version of gcc
• #1111 x64: Parse X2APIC CPU definitions in MADT table
• #1109 Make getpid() return non-zero number
• #1107 scripts: Update script to download RPM packages
• #1106 aarch64: enhance main makefile to support building from host for aarch64
• #1105 aarch64: enhance build system to support building aarch64 versions of modules and apps
• #1104 Identify and reject executables not matching kernel machine (x64, aarch64)
• #1103 arm64: Unable to build using instructions
• #1101 aarch64: support static TLS (Thread Local Storage)
• #1100 aarch64: sometimes triggers a synchronous exception of type IL (Instruction Length) on arm64 native host
• #1099 x86_64 build error
• #1097 Clean fails
• #1095 Unable to read very large file in C++
• #1093 aarch64: OSv crashes frequently during power-off process with ROFS
• #1092 aarch64: hangs while booting on KVM with #vCPUs >= 2
• #1091 aarch64: bugs in arm-clock
• #1090 aarch64: broken memmove
• #1086 scripts: fix setup.py for Ubuntu versions
• #1085 scripts: setup.py broken for Ubuntu versions < 19.10
• #1084 Build fails to upload image on Fedora
• #1083 Assembler warnings in build on Fedora 32
• #1062 Support read-only subset of Virtio-FS file system
• #1047 Provide limited implementation of the tgkill syscall needed by Golang PIE apps
• #1003 gdb did not know about "osv syms" command.
• #991 aarch64 system call numbers are probably wrong
• #922 Can't run osv-memcached in osv
• #916 scripts/loader.py is not python3 compatible
• #895 NFS Makefile cleanups
• #813 mount-fs.so should gracefully handle the case of missing filesystem library
• #796 JVM ballooning is disabled
• #786 Add Mono (MITized) or CoreRT to OSV
• #760 syscall set_mempolicy missing
• #752 Sigar and (G)LibC
• #747 Need sigsuspend and openat64 support to run Nginx in OSv
• #717 OSv build on ARM processor
• #687 Inconsistent mix of build_env = host / external
• #675 "python" interpreter in scripts
• #661 vnode v_mode = 0100444, even though ls -al says different
• #648 When not using AWS default instance the script release-ec2.py will fail
• #646 Read-only file system
• #626 Added java "wrapper" module
• #619 external/x64: "make build_env=external" fails
• #544 Need to implement "pthread_setcancell_state"
• #524 Fault when exiting the internal cli
• #506 major() and friends macros are not Linux compatible
• #504 Cassandra gets stuck when running ycsb workload-C
• #498 build: failure because of old file modules/tests/usr.manifest
• #455 Augment tracepoint backtrace with Java backtrace
• #404 API: Use the swagger validator - to verify our swagger file is compatible with the spec
• #385 JSON friendly value for processor features in the RESTful API
• #379 Lost of network on GCE
• #312 New performance regression in ctxsw micro-benchmark
• #212 Allow running a statically-compiled Linux executable
• #201 Intel MPX support for buffer overflow detection
• #191 run "git submodule update --init" automatically
• #189 Block IO tracing and visualization
• #164 Our clock, even osv::clock::uptime, is not really monotonic
• #151 fix miscompile with ld.gold
• #102 Hang in tlb_flush()
• #90 Support per-thread standard input/output
• #84 per-module search path
• #77 Integrate Apache Portable Runtime native library with OSv
• #61 Hook the JVM dtrace to OSv shell
• #59 Investigate EC2 network disconnects
• #47 dlopen() and program::add_object() needs reference counting
• #27 Integrate VisualVM support for JVM

This new release focuses on improving documentation, build system, and testing. It also introduces Travis-based CI/CD pipeline to publish artifacts to "nightly repo". In addition, it also adds many new features/improvements:

• preliminary virtio-fs support
• QEMU microvm machine support
• numerous improvements to dynamic linker including local-exec TLS
• kernel smaller by 300 K
• new HTTP monitoring API
• less memory utilization with ROFS

Highlights

• Revised existing documentation and added a new one
  • Main README
  • Dynamic Linker
  • Memory Allocation
  • Automated Testing Framework
  • Building from host guide
• Improved build process
  • Setup Travis-based CI/CD pipeline to automatically build and publish kernel and artifacts
  • Setup process to publish OSv build docker images to docker hub
  • Cleaned up build process by eliminating the "build from externals" mode
  • Migrated python scripts to version 3
  • Removed dependency on JDK to build the kernel
• Merged IPV6 code from Spirent into a new branch
• Removed the dependency on the boost program options library from kernel and utility apps which made kernel smaller by 300K
• Created Automated Testing Framework designed to run > 30 real apps, some of them under stress using ab or wrk tools.
• Improved dynamic linker
  • Added local-exec TLS support
  • Expose musl version of libc
  • Handle RTLD_NEXT
  • Added some bionic unit tests
• Resurrected aarch64 support
  • Simple hello world app works now on QEMU and Raspberry PI 4 with KVM enabled
  • Basic support in run.py
• Added support of QEMU microvm machine
• Added clock synchronization mechanism with host on KVM
• File system improvements
  • ZFS improvements
    • Can mount and modify OSv-build drive on a host and use it back on OSv
  • NFS
    • Changed filesystem driver to be a library
  • ROFS
    • Improved memory utilization by integrating with pagecache
  • Virtio-FS
    • Added preliminary support
• Added new HTTP monitoring API module

New Apps

• WebAssembly/Wasmer examples
• Running Docker images example
• Deno
• KeyDB

Closed Issues

• #352 - Thread-local storage doesn't work in PIE
• #743 - Stop using libraries and header files from external/
• #784 - realloc crashes with "page fault outside application"
• #820 - Add ability to build read-only version of httpserver-api module for monitoring purposes
• #854 - Allow non-contiguous malloc() when memory is fragmented
• #880 - Tomcat doesn't run on openjdk8-fedora
• #916 - scripts/loader.py is not python3 compatible
• #918 - Importing ZFS volume fails on OSv
• #975 - mysql cannot working with fs=ramfs
• #976 - Fix annoying warnings while building tests
• #980 - cpiod.so is much too big
• #1014 - Make kernel not to use boost program_options library
• #1015 - OSv guest's wall clock does not follow host's wall clock
• #1040 - Boost 1.69.0 on Fedora 30 contains hidden symbols
• #1044 - Test build with Travis CI
• #1045 - Delay processing PT_NOTE segments until all PT_LOAD segments are mmapped
• #1053 - httpserver should not use boost::program_options
• #1056 - Upgrade utility scripts to Python 3
• #1056 - Virtio-net: use large buffers when VIRTIO_NET_F_MRG_RXBUF is not negotiated
• #1065 - When relocating do not lookup STB_LOCAL symbols by name
• #1066 - Ignore ELF versioning table when object looks up symbols by name in itself (self-lookup)
• #1067 - Make symbols visible to the object itself from threads created by INIT functions
• #1070 - Running Linux apps compiled and linked against musl
• #1073 - Bugs in itimer::work()
• #1074 - Cannot run YCSB with Redis on OSV
• #1077 - malloc_usable_size of malloc_large returns incorrect value
• #1078 - Eliminate libnfs external module

Commits by Author

BassMatt (1):

• Main scripts in scripts/ folder updated to use Python3

Fotis Xenakis (9):

• pci: allow 64-bit BAR offsets
• pci: add batch capability discovery
• virtio-pci: discover shared memory regions
• virtio: expose shared memory regions
• virtio-fs: minor code improvements in driver
• virtio-fs: minor code improvements in filesystem
• virtio-fs: update fuse protocol header
• virtio-fs: add driver support for the DAX window
• docker: update default distros

Mason Davis (1):

• Add bare .travis.yml for testing build

Matthew Pabst (1):

• setup: added make dependency

Nadav Har'El (11):

• scripts/manifest_from_host.sh: write errors to stderr
• build: stop including libboost_program_options
• elf.cc: if Boost libraries aren't visible to applications, don't report them
• tests: fix warning in tst-mmap.cc
• tests: fix warning in tst-symlink
• tests: fix compilation warning in tests/tst-run.cc
• stdio: fix scanf bug
• submodules: cleanly remove unused submodules
• alarm(): fix crash when alarm is rearmed from alarm handler
• scripts/setup.py: fix Fedora package to python3

Waldemar Kozaczuk (158):

• Added patch/pull requests clarification to the main README
• options: implement light and simple alternative to boost program options
• loader: replace boost program_options with light options utility functions
• cpiod: reduce cpiod size by replacing boost program options with the custom light alternative
• httpserver: replace boost program_options with light options utility functions
• commands: remove program options
• tests: Add boost_system.so from externals to test manifest
• scripts: enhance build-capstan-mpm-packages
• scripts: make run.py behave correctly regardless where called from
• scripts: enhance tests/testing.py
• scripts: add automated testing framework
• scripts: tweaked compose_and_test_selected_apps to simplify building some golang test examples
• scripts: generate meta descriptor when creating kernel into capstan repo
• Support local-exec TLS access
• scripts: improved automated test scripts to explicitly report success
• firecracker: upgrade to the latest version 0.19.0 and enhance script to use api-less mode
• aarch64: make it build again
• firecracker: improve networking setup tools
• firecracker: enhance firecracker.py to save/restore terminal
• tests: enhance automated tests scripts to run on firecracker
• virtio-net: use large buffers when VIRTIO_NET_F_MRG_RXBUF is not negotiated
• libc: added __log2f_finite() alias
• tls: make local-exec tls aligned
• tls: parameterize static tls reservation
• procfs: implement /proc/self/exe
• virtio-mmio: do not pretend OSv implements version other than 2
• power down: cause triple fault if keyboard controller method did not work on non-acpi hypervisors
• virtio: adjust virtual queues initialization to adhere to the suggestions in the spec
• scripts: modify run.py to return easier to read output --dry-run
• scripts/run.py: allow to pass block device cache mode to qemu
• scripts/run.py: support running OSv on new QEMU microvm machine
• makefile: introduce loader_options.ld as a better way to pass APP_LOCAL_EXEC_TLS_SIZE to the linker script
• makefile: move loader_options.ld one level up
• makefile: remove redundant linker script for vmlinuz.bin which causes gcc 9.2 linker error
• scripts: enhance test_app_with_test_script.py to allow running with vhost/tap networking
• httpserver-jvm-plugin: remove dependency on externals
• unit tests: replace deprecated BOOST_MESSAGE with BOOST_TEST_MESSAGE
• vfs: make fcntl() handle F_DUPFD_CLOEXEC
• tests: clean makefile to print quiet messages consistenly
• tst-fallocate: replace mktemp with mkstemp to fix compiler warnings
• tst-truncate: replace tmpnam with mkstemp to fix compiler warnings
• elf: reverse unloading of objects in needed list
• tst-fs-link: replace mktemp with mkstemp to fix compiler warnings
• tst-zfs-mount: replace mktemp with mkstemp to fix compiler warnings
• misc-fs-stress: replace mktemp with mkstemp to fix compiler warnings
• tests: replace tmpnam() with mkdtemp() in implementation of TempDir
• libtools: make gcc compiler messages quiet
• tests: stop using boost from externals
• elf: add debug statement aiming to help diagnose problems
• zfs: do not import extra pools unless requested
• zfs: default device name to vblk0.1 if vdev_path different than /dev/vblk*
• elf: when relocating do not lookup STB_LOCAL symbols by name
• kvm: keep wall clock in sync with host
• elf: do not lookup symbol twice in resolve_pltgot
• arm: moved constructor init function in preparation for next patch
• arm: do not relocate DTB, instead read it ahead of time
• elf: make symbols visible to the object itself from threads created by INIT functions
• procfs: add meminfo pseudo-file
• glibc: add explicit_bzero() function
• libc: expose it as musl
• elf: fix thread visiblity issue
• thread: make walking parent-child thread relationship safer
• elf: ignore versioning table if symbol is looked up by object itself
• git: ignore JSON Compilation Database files used by CLion IDE
• elf: separate loading segments from processing headers
• httpserver: use libyaml header from host instead of external
• externals: eliminate unused glibcbase build parameter
• externals: removed unneeded x64 git submodules - glibc.bin and glibc-testsuite
• java: move java_api and jni_helpers to java modules
• external x64: move java balloon code out of kernel into the module java-base
• external x64: replace openjdk7 module with with new openjdk8-from-host one
• external x64: remove misc.bin
• external x64: remove gcc.bin
• modules: add simple sysinfo utility
• setup: support Fedora 30 and 31
• virtio-fs: add fuse protocol header
• virtio-fs: initial implementation of read-only subset
• aarch64: fixed linker error introduced with virtio-fs changes
• tests: add bionic tests to the source tree
• tests: improve regex rules to catch failures in tst-dlfcn
• scripts: upgrade to python 3 - part 1.1
• scripts: migrate setup.py to python 3
• httpserver: properly handle chunked POST requests with body
• scripts: upgrade to python 3 - part 2
• scripts: upgrade to python 3 - part 1.2
• scripts: change firecracker.py to default to '--no-api' mode
• scripts: re-enable tcp_close_without_reading to run on firecracker
• scripts: upgrade to python 3 - part 3
• scripts: enhance run.py to allow running OSv on arm
• scripts/loader.py: make osv info callouts work with older C++ ABI
• aarch64: support JUMP_SLOT relocation in the kernel
• scripts: use strip command as set in makefile so it works properly with aarch64
• scripts: new script to download and extract arbitrary Fedora rpm
• externals: enhance setup.py to download aarch64 gcc and boost code and libraries
• scripts/setup.py: add support of Ubuntu 19.10
• memmove: fix a bug in memmove_backwards
• scripts/setup.py: refactored by moving common packages to distribution level
• scripts/setup.py: move code to download aarch64 packages to its own script
• docker: add new docker files to build OSv 'runner' images
• docker: added build hook file to allow specifying particular fedora or ubuntu version
• docker: add docker files defining base images
• scripts: remove redundant parsing of arguments in run.py
• scripts: enhance test.py to allow blacklisting tests and specifying test manifest
• scripts: fix build-capstan-mpm-packages to remove obsolete fonts module
• docker: changed builder file to git clone full
• travis CI: setup nightly publishing of OSv artifacts
• travis: clean the build tree just in case
• externals: eliminate aarch64 gcc and boost libraries from externals
• dl_tests: fix the makefile to not duplicate building some artifacts
• cloud-init: replace boost program_options with light options utility functions
• libc: make timerfd_* functions handle wall clock jumps
• libc: refine how timerfd_* functions handle wall clock jumps
• virtio: optimize number of exits when parsing capabilities
• virtio: extract common logic to map capability bar
• httpserver: use openjdk8-from-host module in tests
• fs: move nfs support out of kernel in lieu of a separate pluggable module
• tests: move nfs testing out to a separate module
• httpserver: allow compiling out features not needed for monitoring api
• httpserver: add read-only monitoring module
• httpserver: add monitoring API unit tests
• mempool: fix a bug in page_range_allocator() when handling worst case O(n) scenario
• mempool: use map_anon() for large allocations or when memory is fragmented
• httpserver: fix monitoring module Makefile to properly generate JSON cc files
• module.py: do not resolve java-base for non-java apps
• bootfs: fix harmless message about complaining libgcc_s.so.1 when uploading manifest
• build: reference source bootfs.manifest.skel to prevent rebuiding kernel everytime for ZFS build
• build cleanup: export files regardless of filesystem chosen
• make: add new uniform kernel.elf artifact intended for direct-kernel boot mode
• build: allow building kernel-less ZFS and ROFS disk images
• certs: make certificates work with apps running on firecracker
• httpserver-api: add --kernel_path parameter to allow running tests with specific kernel
• tests: allow building ZFS and ROFS versions of test mps
• scripts: do not resolve symlink targets when exporting -
• capstan: add kernel.elf and httpserver monitoring module to the list of generated artifacts
• tests: improve automated testing framework
• travis: fix build to take into account renamed unit tests artifacts
• zfs: avoid f_fsid sign extension when translating f_fsid to va_fsid
• zfs: modify fsid_guid to store extra bits to allow detecting ZFS in pagecache
• rofs: report fsid (filesystem ID) properly
• sysfs: add new pseudo-files free_page_ranges and pools to help monitor memory utilization
• pagecache: refactor to allow integration with non-ZFS filesystems
• rofs: optimize memory utilization by integrating with page cache
• travis: simplify CIRP publishing and add new artifacts
• docker: refine build docker files
• travis: fix the build
• loader: add option to mount extra filesystem
• run.py: add support of mounting directory via virtiofs device
• socket: warn about missing ancillary information handling

Wonsup Yoon (1):

• run.py: add tap option

Zhiting Zhu (4):

• add pthread_setcanceltype implementation
• Add dlsym tests
• Lookup symbol deeper
• Implement RTLD_NEXT

yuchenq (2):

• osv: fix keyboard reset, add pci reset
• osv: properly assign gsi number inside handler

This new release of OSv focuses on improving Linux compatibility and tooling aimed to make it possible to run unmodified Linux apps on OSv "as-is".

Overview

From the beginning, OSv was designed to implement a subset of Linux POSIX API superset. But until this release most Linux applications had to be re-compiled from source as shared libraries or some, like Java, rely on OSv version of /usr/bin/java wrapper to run. This meant that one could NOT run a Linux executable "as is". In other words, OSv has always been Linux-compatible at source level but not at binary level.

This release offers a breakthrough and allows running unmodified Linux position-independent executables (so-called "pies") and position-dependant executables "as-is" as long as they do not use "fork/execve" or other unsupported Linux API. It means that very often one can take a binary from Linux host and run it on OSv without having to locate the source code on the Internet and build it as a shared library.

In addition, this release makes OSv more Linux-compatible from another end - booting on a hypervisor. The previous release 0.53 made OSv kernel "look like" ELF64 uncompressed Linux kernel. The new release 0.54 has enhanced OSv loader to "look like" vmlinuz and thus allow booting on Docker's Hyperkit on OSX. The OSv loader has also been enhanced to boot as Linux ELF64 PVH/HVM loader on QEMU with --kernel option.

Highlights

Linux compatibility

• Applications
  • Enhanced getopt family of functions to work correctly with both position-independent executables and position-dependent executables in order to allow receiving program arguments
  • Enhanced dynamic linker to be capable of executing position-dependent executables
  • Mapped kernel higher in virtual memory - from 0x00200000 to 0x40200000 (2nd GiB) in order to make space for position-dependent executables
  • Added new GNU libc extensions: error(), __prognames and __progname_full
  • Added missing pseudo-files to procfs and minimal implementation of sysfs in order to support libnuma to allow programs like ffmpeg using x265 codec run on OSv "as-is"
  • Encanced /proc/self/maps to include i-node number and device ID to support GraalVM apps with isolates
  • Enhanced epoll_pwait() implementation
  • Improved dynamic linker by making it:
    • Ignore old version symbols so that new version symbols are resolved correctly instead
    • Delay resolving symbols found missing during relocate_rela() phase for certain relocation types to allow more unmodified Linux executables run on OSv
    • Handle DT_RUNPATH
• Booting
  • Added vmlinuz-compatible version of the kernel to allow OSv boot on Docker's Hyperkit
  • Enhanced loader to support PVH/HVM boot to allow OSV run on QEMU with --kernel option
  • Added support of QEMU 4.x
  • Enhanced HPET driver to support 32-bit main counter

Filesystem improvements

• VFS
  • Hardened implementation of open()/sys_open()/task_conv() to handle null path
  • Enhanced __fxstata to handle AT_SYMLINK_NOFOLLOW
• RAMFS
  • Greatly improved speed of write/append operations
  • Fixed bugs
    • Delay freeing data until i-node closed
    • Keep i-node number the same

Tools

• Added script manifest_from_host.sh to allow building images from artifacts on Linux host “as-is” without need to compile
• Added script build-capstan-mpm-packages to create capstan MPM packages
• Added Ubuntu- and Fedora-based Docker files to help create build and test environment
• Enhanced test.py to allow executing unit tests on Firecracker

Bugs and other enhancements

• Fixed sem_trywait() that for example allows Java 12 run properly on OSv
• Improved memory utilization by using memory below the kernel
• Introduced new command line suffix ! allowing to force termination of lingering threads
• Revamped building of the cli and httpserver apps to use OpenSSL 1.1 and Lua 5.3 and minimize compilation
• Tweaked OSv code to support compilation by GCC 9

Improved Documentation

• Refreshed main README
• OSv-apps
• Scripts  

Apps

• Added number of *-from-host apps that demonstrate building images out of binaries from Linux host:
  • Java
  • Python
  • Node
  • Lua
  • Ffmpeg 
• Added demo app - openjdk12-jre-from-docker that creates an image out of a Docker image
• Added demo app that demonstrates running GraalVM isolates
• Added an example of a basic mono app
• Improved support of Golang PIEs

Closed issues

• #1050 - Can't run anything with 1.01G of memory
• #1049 - tst_huge hangs with memory over 1GB.
• #1048 - VM with memory larger than 4GB doesn't boot
• #1043 - Map kernel higher in virtual memory
• #1039 - Handle new DT_RUNPATH in object::load_needed()
• #1035 - iperf3 fails with exception nested to deeply on ROFS/RamFS image
• #1034 - Build failures when build directory's pathname has a space
• #1031 - The graalvm-example fails with graalvm 1.0.0-rc13
• #1026 - golang-pie-httpserver crashes on control-C
• #1023 - Ignore missing symbols when loading objects with BIND_NOW in relocate_rela()
• #1022 - lua package requires openssl 1.0
• #1012 - Improve physical memory utilization by using memory below 2MB
• #884 - slow write/append to files on ramfs
• #689 - PIE applications using "optarg" do not work on newer gcc
• #561 - OSv failed to run a pthread application.
• #534 - imgedit.py can't always connect to qemu-nbd
• #305 - Fail to run iperf3 on osv
• #190 - Allow running a single unmodified regular (non-PIE) Linux executable
• #34 - Mono support

Commits by author

KANATSU Minoru (1):

• libc: add __explicit_bzero_chk()

Nadav Har'El (9):

• scripts/build: gracefully handle spaces in image= parameter
• build: don't fail build if pathname has space
• trace.py: fix failure on newest Python
• tracepoints: fix for compiling on gcc 9
• sched: fix gcc 9 warning
• libc: avoid weak_alias() warnings from gcc 9.
• acpi: ignore new gcc 9 warning
• imgedit.py: do not open a port to the entire world
• sched.hh: add missing include

Waldemar Kozaczuk (86):

• Added initial version of README under scripts directory
• Lowered default ZFS qcow2 image size from 10GB to to 256MB
• Add script to setup external bridge
• Refactor and enhance firecracker script
• Changed loader to print total boot time by default
• Enhanced setup.py to support Ubuntu 18.10 and 19.04
• Add GNU libc extension function error()
• Add GNU libc extension variables __progname and __progname_full
• Update nbd_client.py to support both old- and new-style handshake
• Simplify building images out of artifacts found on host filesystem
• Move getopt* files to libc folder and convert to C++
• Enhance getopt family of functions to work with PIEs
• Tweaked nbd_client.py to properly handle handshake and transport flags in new handshake protocol
• Tweak open() and sys_open() to return EFAULT error when pathname null
• elf: handle new DT_RUNPATH
• Enhanced __fxstata to handle AT_SYMLINK_NOFOLLOW
• vfs: Harden task_conv() to return EFAULT when cpath argument is null
• Added option suffix "!" to force termination of remaining application threads
• Move _post_main invocation to run_main
• Provide full implementation of epoll_pwait
• Start using memory below kernel
• Change vmlinux_entry64 to switch to protected mode and jump to start32
• Fixed indentation in xen.cc
• Move kernel to 0x40200000 address (1 GiB higher) in virtual memory
• Allow running non-PIE executables that do not collide with kernel
• Make RAMFS not to free file data when file is still opened
• Fix slow write/append to files on ramfs
• hpet: Support 32-bit counter
• mprotect: page-align len parameter instead of returning error
• procfs: populate maps file with i-node numbers
• procfs: Add device ID information to the maps file
• syscall: add getpid
• signal: tag user handler thread as an application one
• Make OSv boot as vmlinuz
• Prepare for local-exec TLS patch
• Clean boot logic from redundant passing OSV_KERNEL_BASE
• Support PVH/HVM direct kernel boot
• Enhance scripts/test.py to allow running unit tests on firecracker
• Refine confstr() to conform more closely to Linux spec
• Enhanced manifest_from_host.sh to support building apps from host and docker images
• Fixed compilation errors in modules mostly related to strlcpy
• Enhance scripts/build to allow passing arguments to modules/apps
• Remove obsolete loader.bin build artifact and related source files
• Add --help|-h option to build script to explain usage
• Remove reference to external from httpserver-api makefile
• Removed remains of externals reference from httpserver-api Makefile
• scripts: hardened manifest_from_host.hs to verify lddtree is installed on the system
• scripts: remove old Ubuntu and Fedora from setup.py; added support of Fedora 29
• scripts: fixed typo in setup.py
• Reverted some changes related to upgrading openssl that got checked in prematurely
• scripts: Enhanced manifest_from_host.sh to better support regular expressions and filter x86_64 ELF files
• Fixed compilation errors in modules httpserver-jolokia-plugin, josvsym and monitoring-agent mostly related to strlcpy
• Fixed httpserver file system integration test
• ramfs: fix arithmetic bug leading to write overflows
• scheduler: Initialize _cpu field in detached_state struct
• semaphores: fix sem_trywait
• pthreads: implement pthread_attr_getdetachstate
• pthreads: make implementation of pthread_attr_getdetachstate more correct
• java: add basic java test that does NOT use OSv wrapper
• Made maven more quiet and only show errors
• java: tweaked openjdk7 to add /usr/bin/java symlink
• setup: add pax-utils package for Ubuntu and Fedora
• Add docker files to help setup OSv build environment for Ubuntu/Fedora
• Upgrade cli, lua and httpserver-api modules to use OpenSSL 1.1 and Lua 5.3
• dynamic linker: adjust message when symbol missing
• docs: Updated main README to make it better reflect current state of OSv
• hpet: handle wrap-around with 32-bit counter
• Fix bug in arch_setup_free_memory
• memory: enforce physical free memory ranges do not start at 0
• pthreads: provide minimal implementation to handle SCHED_OTHER policy
• memory setup: ignore 0-sized e820 region
• libc: added __exp2_finite wrapper needed by newer libx265
• elf: skip old version symbols during lookup
• syscall: add set_mempolicy and sched_setaffinity needed by libnuma
• Ignore missing symbols when processing certain relocation types on load
• procfs: add minimum subset of status file intended for linuma consumption
• fs: extracted common pseudofs logic
• fs: add subset of sysfs implementation needed by numa library
• ramfs: make sure to pass absolute paths for mkbootfs.py
• ramfs: make sure i-node number stay the same after node allocation
• scripts: fix export_manifest.py to handle all symlinks properly
• scripts: enhance manifest_from_host.sh to put cmdline example for executables
• httpserver: make test images use openjdk8 as new jetty app requires it
• httpserver: enhance test script to accept different location of test image
• loader: print boot command line and expanded runscript line if applicable
• capstan: add script to automate building capstan MPM packages

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing many patches and providing guidance for many others
• Waldemar Kozaczuk for contributing most patches

This is the next release of OSv since the previous one 0.52.0 was crafted. Unlike previous one this release does deliver new exciting feature - it supports running OSv on new lightweight hypervisor firecracker which makes possible to boot OSv Read-Only FS image in as low as 5ms per bootchart. We hope next release will happen before end of second quarter of 2019.

Features Highlights

• Enhanced OSv to run on firecracker (for details please see here)
  • OSv can boot as vmlinux (64-bit ELF - please see loader.elf attached)
  • OSv does not require ACPI to run
  • Added support of virtio mmio devices
• Added support of both legacy and modern virtio 1.0 PCI devices (for details please see here)
• Further reduced minimal memory to run OSv to 18MB (25% reduction)
• Fixed critical bug affecting apps doing heavy floating point computations like ffmpeg (see issues #1019, #1020, #1010, #1018, #536)
• Implemented some missing syscalls to better support Golang apps
• Enhanced Python 3 app to support including modules
• Upgraded tomcat app to version 9
• Added support of graalvm apps

Specifics

• Refactored virtio layer to support legacy and modern PCI and mmio devices
• Implemented modern virtio PCI device
• Enhanced OSv to boot without ACPI present
• Enhanced OSv to discover SMP information by reading MP table if ACPI is not present
• Made OSv bootable as a vmlinux 64-bit ELF
• Added boot option to disable PCI enumeration
• Added virtio mmio implementation
• Made httpserver-api configuration file configurable
• Enhanced ELF dynamic loader to support position dependent shared libraries (Graal VM)
• Deleted “assigned virtio driver"
• Made memory allocation more efficient
  • Handles small memory allocations ( < 16 bytes) more space efficiently
  • Implemented more space-efficient early memory allocation scheme (reduced memory utilization by 6MB - see #270)

Closed issues

• #1030 - Test failure with Fedora Java 8
• #1029 - Debug build fails because tracepoint sections are discarded
• #1028 - scripts/firecracker.py doesn't pass input to OSv
• #1020 - Maybe need to reset fpu status word on context switch
• #1019 - Maybe need to reset fpu state after saving it
• #1018 - OSv crashes sporadically with page fault when transcoding video with ffmpeg
• #1017 - upload_manifest.py opens a port to the world, and always the same port
• #1011 - Make allocations < 16 bytes more space efficient
• #1010 - Memory access fault when OSv pounded by periodic /os/threads REST call
• #1006 - Missing function fcntl64
• #1004 - Handle (either support correctly, or forbid) non-fPIC shared objects
• #956 - Missing readlinkat()
• #536 - Cassandra with CLI crash with cassandra stress
• #270 - Wasted memory in early malloc()

List of contributors alphabetically

• Felix Yan (@felixonmars) - 1 patch
• Nadav Har'El (@nyh) - 8 patches
• Przemek Pikuła (@pshem) - 4 patches
• Waldemar Kozaczuk (@wkozaczuk )- 45 patches
• Geraldo Netto (@geraldo-netto) - 6 patches
• Vinícius Fraga (@viniciusmsfra) - 1 patch

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing many patches and providing guidance for many others
• Waldemar Kozaczuk for contributing most patches

Artifacts

This time we are only publishing kernel artifacts and capstan packages for Python 3 and REST API httpserver.

This is the next release of OSv since the previous release 0.51.0 was crafted. Though this release does not deliver any dramatic features, it brings major improvements around kernel size and memory utilization and adds support of many new apps including Python 3. We hope new releases will happen every quarter and the next one should be expected around beginning of 2019.

Features Highlights

• Smaller kernel which makes OSv use less memory and boot faster
  • reduces non-compressed kernel size from 9.2MB to 6.7MB (27% reduction) and compressed one from 6.2MB to 3.9MB (37% reduction)
  • reduces boot time by ~ 20ms
• Reduced minimal memory to run OSv to 25MB
• Reduced memory utilization for ram images (especially large ones)
• Added support for Python 3 and Golang 1.11
• Added many new apps including ffmpeg, spring boot, ratpack and akka

Specifics

• Set non-isolated mode as a default one for Java images
• Removed obscure UTF8 normalization feature to save 130 KB in kernel size
• Started merging IPv6 code from Spirent
• Relaxed ELF symbol resolution failure when BIND_NOW which should make more apps supported out of the box on OSv
• Improved bootfs to remove unnecessary copy of data
• Revived multi boot mode to support eventually booting on Hyperkit and on QEMU with qboot
• Implemented almost-in-place kernel decompression to reduce minimal memory size that provides a foundation for even less memory usage changes in the future
• Disabled '—whole-archive' linking for libgcc.a to reduce kernel size by 2.5MB

Closed issues

• #645
• #781
• #792
• #977
• #984
• #985
• #989
• #990
• #993
• #999

List of contributors alphabetically

• Charles Myers (@cmyers-spirent) - 2 patches
• Geraldo Netto - 2 patches
• Nadav Har'El (@nyh) - 13 patches
• Timmons C. Player (@DerangedMonkeyNinja) - 1 patch
• Waldemar Kozaczuk (@wkozaczuk) - 23 patches

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing many patches and providing guidance for many others
• Waldemar Kozaczuk for contributing most patches
• Timmons C. Player and Charles Myers for contributing patches providing initial IPV6 support

Artifacts

This time we are only publishing kernel and capstan packages for Python 3 and ffmpeg.

This is the first release of OSv since late 2015 when the last release 0.24.0 was crafted. Even though the development has not been as vigorous as in previous years, new release 0.51.0 delivers many new exciting features as well as fixes of many outstanding defects. We hope new releases will happen more frequently in future.

Features Highlights

• New MikelAngelo capstan by XLAB Slovenia
  • Enhanced to support packages which makes building OSv even easier than before
  • See this crash course for more details
• New filesystems
  • ROFS (Read Only File System) - for more details read cd449667b7f86721095ddf4f9f3f8b87c1c414c9 and 1f122aee15bd294a34990f81fd1133121dde87f5
  • RAMFS-only images support
  • NFS (Network File System) support
• New languages/runtimes support
  • Golang (for more details read here
  • Node.JS 6.10.3, 8.11.2
  • Python 2.7
  • Free Pascal
  • Rust
• New hypervisors support
  • XEN support for aarch64 (ARM) by Sergiy Kibrik
  • OSv can boot on HyperV
  • Fixed critical bug related to VirtualBox - 13b8acbf712fd18f5b5e39da3b4faa8d40a2e951
• New applications support
  • Nginx
  • Lighttpd
• Improved Java support
  • Simpler non-isolated run java mode - read more details here 785e57e60969e25c19c87ad1b922ad9b66428c0e and f0bb5c24d5ba00a7418b1bf5a46043ce2c12eae5
  • Support for Java 8 compact profiles, Java 9 and 10 - more details here bee9421617a4f812b49ada00e83127bb112f4499
  • Modularized Java build (19bf18f97de002af0f832bd39bc78b156ceb6371)
  • Added new Zulu JREs (look for openjdk[89]-zulu-* named apps under https://github.com/cloudius-systems/osv-apps)
• Added support of SYSCALL instruction needed by Golang and Free Pascal
  • libvdso - fast syscall compatibility library
  • Added number of SYSCALLs to support Golang
• Improved httpserver module
  • Modularization by API
  • Added HTML5 command line app
  • Configurable to serve arbitrary static content
• Cloud init improvements
  • ISO image support
  • Supports mount points
• Many pthread improvements/bugs
• ELF namespaces support - e9758b9107c28f42eacaedd466f49d343bbddeae
• Support Open MPI apps (High Performance Computing)
• Support compilation using GCC 6, 7 and 8
• Improved signals support
• Added runscript support
• Improved DHCP
• Added many missing C-lib functions
• Support for Virtlet (https://github.com/mirantis/virtlet)
• Added new docker file to build OSv kernel and capstan packages
• Added new docker file to make it easy to run selected set of apps on OSv
• Overall committed 549 patches and closed 206 issues

Acknowledgments

We want to thank all contributors to the project. But the special thanks go to:

• Nadav Har’El for contributing and reviewing most patches and providing guidance for many others
• Gregor Berginc, Justin Cinkelj and Miha Pleško (@miha-plesko) from XLAB Slovenia for greatly enhancing capstan, enhancing cloud init module and contributing many other patches to make Node.JS, Python 2.7 and nginx work on OSv.
• Benoit Canet for implementing NFS and contributing many patches to support Golang on OSv
• Sergiy Kibrik for contributing patches to make OSv run on XEN with Arm architecture
• Dor Laor for words of encouragements without which this release may have not happenned

List of contributors alphabetically

• Quentin Barnes - 2 patches
• Gregor Berginc (@gberginc) - 2 patches
• Benoit Canet (@benoit-canet) - 47 patches
• Brandon Cox (@justnoise) - 1 patch
• Hawx Chen (@HawxChen) - 1 patch
• Justin Cinkelj (@justinc1) - 77 patches
• Glauber Costa (@glommer) - 1 patch
• Shiqing Fan - 1 patch
• Dongjiu Geng - 1 patch
• Rean Griffith - 3 patches
• Nadav Har'El (@nyh) - 272 patches
• Sergiy Kibrik (@sa-kib) - 32 patches
• Waldemar Kozaczuk(@wkozaczuk) - 57 patches
• Michael Meeuwisse (@Meeuwisse) - 3 patches
• Vilmos Nebehaj - 1 patch
• Geraldo Netto (@geraldo-netto) - 13 patches
• Rick Payne (@rickpayne) - 5 patches
• Timmons C. Player (@DerangedMonkeyNinja) - 8 patches
• Christian Schwarz (@problame) - 3 patches
• Vincent Schwarzer - 1 patch
• Joel Thompson (@joelthompson) - 2 patches
• Yuri Volchkov (@yvolchkov) - 10 patches
• Iori YONEJI - 1 patch
• Madhuri Yechuri (@myechuri) - 1 patch
• Wang Yu - 2 patches
• eyujugu - 1 patch
• Hidehito Yabuuchi - 1 patch

Artifacts

This is also first time we are publishing many binary artifacts along these release notes on GitHub to download from. These binaries match exact files used by new capstan to build OSv images and go into $HOME/.capstan directory. We hope these will make much easier for newcomers to OSv to try their apps. The published artifacts include:

• osv-loader.qemu - OSv kernel
• OSv modules
  • osv.bootstrap.* - default bootstrap
  • osv.cli.* - command line app
  • osv.httpserver-api.* - REST API (backed)
  • osv.httpserver-html5-cli.* - HTML5 command line app
  • osv.httpserver-html5-gui.* - HTML5 monitoring GUI
• runtimes
  • osv.run-java.* - provides 'java' executable functionality in OSv to bootstrap Java apps
  • osv.run-go.* - tiny app to bootstrap Golang apps
  • osv.openjdk10-java-base.* - Java 10 minimal JRE repackaged to run Java apps on OSv
  • osv.node-js.* - Node 8.11.2 built as PIE shared library to run on OSv
• example apps
  • iperf
  • lighttpd
  • memcached
  • mysql
  • netperf
  • nginx
  • redis

Going forward we will be publishing kernel and possibly modules binaries if they change.