FEX

A fast usermode x86 and x86-64 emulator for Arm64 Linux

MIT License

Stars
1.8K
View Code on GitHub Visit Website
Ecosystems: Linux, C++

Bot releases are visible (Hide)

FEX - FEX-2208

Published by Sonicadvance1 about 2 years ago

Read the blog post at FEX-Emu's Site!

Some really exciting changes this month. Thunk stabilization out of the gate is a huge boon to tinkerers and a bunch of other things spread
throughout!

Thunk improvements

The amount of work to reach this point can't be understated. @neobrain has been putting forth a bunch of infrastructure
work over the past few months which hasn't been super visible to the end user. This month it culminated towards fixing a bunch of stability problems
with thunks.

One of the biggest problems ends up being when a pointer is passed between the guest and host through thunks. X11's XFree function is used for both
x86 specific pointers and AArch64 pointers. This ends up being a problem since FEX-Emu uses jemalloc for its allocator while a guest application is
highly likely to use glibc's allocator. Passing the opposite one to either will crash either. We now distinquish between a "Guest" pointer and a
"Host" pointer in our thunk of XFree, which significantly improves stability of X11 thunks.

We also now have our libGL thunk implicitly load libX11. Due to how FEX's thunks work, we don't pull in all of the library dependencies of a "real"
library like libGL. Long term FEX-Emu will likely want to link to the same libraries that the real x86-64 library would have linked to. For now, libGL
now relies on libX11 directly. This fixes an issue where libGL thunks wouldn't work at all for any game launched from Steam.

Then the big pull request that does an absolute wackload of infrastructure work to make things work, Implement signature-based thunking of function pointers. This pull request is a bit complex to explain but it allows function pointers to be marshalled
across the thunk architecture boundary safely. Allowing both x86-64 and AArch64 to call in to the other for code for whatever reason is necessary. I
would recommend reading the pull request itself because the information is quite dense there.

Then some very minor changes the fixes some edge case behaviour.

  • Make glXGetProcAddr of unknown address non-fatal
  • Make glXGetProcAddr querying itself work
  • Add some missing glX and GL functions that were missing.
    • Nearly everything supported now, just some minor things missing

The take away for this effort is that OpenGL thunking is now significantly more stable. We don't have a lot of games tested yet, but follow along at
our wiki for documenting which games support thunks under FEX-Emu.

When OpenGL or Vulkan thunks are enabled, games are dramatically sped up. It is the recommended way to play a game but we need more testing coverage
to ensure it is stable enough to use.

Fix edge-case instruction faulting behaviour

x86 has six instructions that explicitly fault in different ways. We weren't handling RIP setting correctly on some of these.

This was uncovered by running Elden Ring inside of FEX. With this bug fixed FEX-Emu can now run the game if Denuvo is disabled by renaming the
executables. Sadly it doesn't seem like Snapdragon can run this game yet.

AVX initial implementation details

@Lioncache has been working away at making this feature a reality. Newer games coming out are starting to require AVX
to run and FEX needs to support this. This is a necessary feature since we need to claim compatibility for any CPU feature that
games use. So far the list of games we found is few that require the feature but it is going to become more common. The latest generation of game
consoles will drive this feature over the next decade of releases.

This sets up the initial groundwork inside of FEX-Emu to support the feature, with instruction implementations coming in the next months.

Don't expect this to run on any of your ARM devices any time soon though. We are going to require SVE with 256-bit register width to expose it. All
current hardware either doesn't support SVE at all, or only supports a 128-bit register width. Look forward to future hardware that ships with this
feature.

FEXConfig quality of life improvements

This is cleaning up some of the rough edges inside of FEXConfig, making it easier to modify your config. Significantly less keypresses to open a
config! Perfect!

Fix SOMA again

Due to how FEX changed some of its threading logic, we broke SOMA abusing the SETXID signal. This is now resolved and the game runs under FEX-Emu
again.

Fix Static Register Allocation in signal handlers

Applications relying on signals would very likely have crashed for a while due to this regression. With this resolve these games are stable again.

FEXRootFSFetcher automation options

FEXRootFSFetcher now has command line options for automatically choosing distro image and setting it up in config. This is useful for containers
embedding FEX and wanting a fresh rootfs instead of shipping it. It also does a runtime check for any image tooling programs before executing, fixing
a spurious error message.

Fix hang in Proton from close_range syscall

Proton would sometimes hang due to a close_range syscall trying to close all file descriptors. Make FEX a bit smarter about closing FDs with this
syscall which fixed the bug.

FEXBash change PS1 description

When running FEXBash it can be confusing for new users how it is behaving. Now we show the current operating path, user, and FEXBash as a prefix.
Hopefully this can be less confusing to new users that FEXBash isn't a VM, or docker style chroot.

Support Linux 5.19 passthrough

Not much changed from FEX's perspective here. Some minor DRM changes that naturally work themselves out.

Only initialize perf map file if profiling is enabled

Sorry about filling up your /tmp folder with empty files. This has been resolved.

Add pidfd_open syscall wrapper for compiling on older Linux distributions

Thanks to @wannacu for this fix. Our testing of older Linux distributions is quite spotty since FEX-Emu only officially
supports back to glibc 2.31 distros. This adds a wrapper utility for this syscall so older versions of Linux can compile FEX. Your mileage may vary
since FEX-Emu doesn't officially support it although.

Developer specific improvements

Remove static-pie

Static pie will never work due to glibc limitations around dlopen that FEX needs for thunks. Remove the option entirely to ensure no one tries to use
it.

FMT updated to 9.0.0

Newest release is best release

Bitness of syscall handler improvements

We were setting up both 32-bit and 64-bit syscall handlers upon initialization. Now we only initialize one or the other. Saves a bit of memory and
startup time shaves some microseconds off.

Vulkan-Headers included in External

It's no longer required as a install dependency on the host. We need to handle all Vulkan signatures, not just what is available on the host.
Makes compiling Vulkan thunks slightly less painful.

Misc

  • Cortex-X1C supported in CPUID
  • Support a globally installed Config file
  • Support installing many json files from FEX Data folder
  • Disable UnitTestGenerator since it is unused
  • Assume optimizing LogManager assertion functions
  • Support executable names being picked up for wineserver.
    • Useful to see if a Windows game is doing something that our telemetry picks up
  • Removing last usage of raw IR arguments in emulation backends
    • x86_64: Migrate args over to named IR arguments
    • json_ir_generator: Remove Args() functions from IR structs
    • These make it a lot easier to see what arguments are being used and why

Raw Changes

FEX - FEX-2207

Published by Sonicadvance1 over 2 years ago

Read the blog post at FEX-Emu's Site!

This is going to be a very interesting release this month for users. Quite a large number of features landed for this release!

Automatic TSO mode migration

When FEX is running a single threaded application, we can be optimistic and disable heavy TSO-emulation related features. This significantly speeds up some single threaded applications. Once the program creates a thread then FEX will disable this optimize and clear its code cache to be safe.

EroFS rootfs image support

While FEX has supports SquashFS for a long time. We are now adding support for EroFS as well. The big advantage of EroFS is that it doesn't serialize accesses to a single thread. When you're having dozens of threads accessing the filesystem this is a real bottleneck. Low end devices would end up having a single CPU core maxed out inside of the squashfuse application while multiple threads are trying to request data.

erofsfuse solves this by allowing multi-threaded decompression that scales quite well depending on the number of file requests in flight. We can see how this scales in the following benchmark graphs.

As one can see, while erofsfuse scales quite well with multiple threads; squashfuse stays pretty much flat the entire time. The downsides to EroFS is that the compression ratio of its LZ4HC compression isn't quite as good as ZSTD, causing the rootfs to be larger. But the reduction in memory usage, and lower read amplification plus higher bandwidth is worth it. Seriously improving performance of using a rootfs over a network mapped share like some people do.

An additional problem is that the erofsfuse application requires erofs-utils version 1.5, which came out on 2022-06-13. This is really bleeding edge currently.

Never the less, FEXRootFSFetcher will now allow you to download a FEX Rootfs image with this compression format. Just ensure you have a erofsfuse installed.

FEXServer

This is a fairly significant change to how FEX-Emu operates in the background. Similar to how wine has a wineserver, FEX is now requiring a FEXServer
to always be running.

For now the FEXServer is taking over duty for rootfs image mounting and a logging server. In this future this will be expanded to also handle code
caching services and more. FEXServer will automatically start on invocation of FEX and be running in the background until all instances of FEX close.

Pressure-vessel and Proton Fixes

FEX-Emu now officially works inside of pressure-vessel. This is the tool that Steam uses for running Proton games. Thunking doesn't yet work in this case but it is coming.

If you're wanting to test proton games, make sure to sign up to the latest SteamLinuxRuntime_soldier beta in the settings and give it a go. It's not currently the speediest, but it should work.

Disable FEXServer rootfs when running under pressure-vessel

Pressure-vessel sets up an x86-64 rootfs. FEX shouldn't be using the FEXServer provided rootfs in this case.
We now detect when running inside of pressure-vessel, and disable the FEXServer RootFS

Enable Hypervisor bit

This change allows pressure-vessel to detect FEX-Emu and do FEX specific setup for games.

Fix open syscall path emulation

The open syscall is fairly rarely used so this has gone unnoticed for a while. We weren't wrapping this syscall in our filesystem emulation and was breaking applications from running. With this fixed, the latest Proton Experimental branch from Steam now works!

Support thunks in pressure-vessel

Pressure-vessel uses a bunch of environment variable overriding to replace where libraries are inside of its chroot. Support this inside of FEX. While this is a step to getting Thunks working inside of pressure vessel, it is not yet supported.

Thunks

Lots of improvements to thunks, it's hard to capture them all. There is a heavy amount of infrastructure work going on in here to make thunks more robust and stable. Starting with Vulkan and GL.

  • Work around lack of generic callback support in VK_EXT_debug_report (4771a340)
    • Disable debug report callback (751b66d4)
  • Allow building thunks on a wider range of platforms (ad6fd5ab)
  • Add fex:is_lib_loaded (88b94bef)
  • Support returning host function pointers to the guest (04a1ac96)

Fix clone3 syscall's stack pointer again

In an edge case of how FEX-Emu handles clone3, it wasn't handling the stack pointer size correctly again.
Resolving this edge case once again gets Steam's web helper working with glibc 2.34.

Fix 32-bit memory allocation range scanning

When scanning for free chunks of memory in the 32-bit range, FEX-Emu needs to use a custom allocator to ensure everything returned ends up in the lower 32-bit memory space. This fixes a bug where large allocations would never find an empty space. Fixes X-Plane 11!

Optimize file descriptor to filename mapping

It is a common occurrence that FEX needs to map an open file descriptor back to a file path. This used to take 14 system calls.
Since each system call was querying filesystem metadata these could take some time. With this optimized approach it now takes only one system call instead. Significantly lowering file IO overhead!

Enable Wine application profiles

Wine applications when they are executing typically only showed up as wine or wine-preloader to FEX-Emu.
Now we work around this issue by scanning the arguments to find the executable name, which allows application profiles to function.
Now we can easily support SonicMania.exe.json!

FEXRootFSFetcher fix to file hashing

It was discovered that this tool was hashing files incorrectly. The new version is now hashing correctly and image files have been updated to be using the new hash. Nothing to see here

Fix 32-bit DRM ioctl DRM_IOCTL_WAIT_VBLANK

This ioctl does exactly what it says on the tin. Due to a copy and paste error, this wasn't actually waiting on vblank.

Fix 32-bit ioctl structure copying

A feature of the DRM subsystem allows you to extend ioctl struct definitions safely. The kernel knows the size of the ioctl structure and if it
differs from what the userspace application passes in, then it will only copy the smaller amount of data and zero out the rest.
This allows older userspaces to safely work with newer kernels. FEX wasn't reproducing this with its ioctl emulation in some V3D ioctls, resulting in unsafe execution of ioctls. This has been resolved.

Support CLMUL Extension

This instruction is heavily used to accelerate CRC and other hashing algorithms. This perfectly matches the AArch64 instruction as well. So
implementing this was very straightforward!

Self-modifying-code frontend improvements

Allows FEX to track code pages inside our frontend decoding. This fixes some issues where code can be changed while we are decoding things in the frontend. Now FEX can detect this and throw away what it compiled.

Developer specific improvements

Check for binfmt_misc conflict before installing

To ensure building from source doesn't result in a broken configuration, cmake will now check for conflicting binfmt_misc files before installing.
How to uninstall the conflicting binfmt_misc files is specific to how the user has installed them, so it is left up to them to find out how.

Auto CI fetching

If the CI systems need an updated rootfs, the config can now be updated and they will fetch the latest.

unittests now longer forever recompiler

ASM unittests would always reglob on building which took time. This is now fixed

Fix ASAN bug in how register allocation data was allocated

This was hard to track, finally this annoying bug that has gone back and forth a bit has been resolved!

ARM64 CPU feature detection for ASM unit tests

Automatically disables some incompatible unit tests on ARM64 devices that don't support some features. No more confusing failures.

GDB integration

This allows a plugin to be loaded in GDB to show more information that we would otherwise have. Giving us both backtraces and source inside of GDB
even through the JIT. Should let debugging the JIT be that much easier.

Raw Changes

FEX - FEX-2206

Published by Sonicadvance1 over 2 years ago

Read the blog post at FEX-Emu's Site!

Quite a large amount of changes this month since we cancelled last month's release.

Steam's webhelper working again

Steam started enabling the chromium sandbox. Seccomp isn't supported in FEX-Emu so it was crashing early on.
Forcibly disable it trying to use the sandbox using an application profile.
This lets the game library be visible again, although it can take a while to appear.

Fix LRCPC and add support for LRCPC2

There was a bug in our CMPXCHG implementation that wasn't using ARM's acquire-release semantics accidentally.
Fixing this bug allowed us to reenable our TSO emulation using LRCPC.
Additionally we have added support for LRCPC2 which gives us some immediate encoded instructions to further reduce overhead.

On hardware that supports LRCPC these can result in a reasonable performance uplift.

SHA-1 and SHA-256 instructions implemented

These SHA instruction have been implemented and the CPUID bit is now exposed.
This is a GPR based implementation, an implementation using AArch64's equivalent SHA instructions will be implemented at a later time.

Self-modifying code support improvements

Many things have changed with supporting self-modifying code in a more extensive fashion.
FEX-Emu will now tracking guest allocations of executable memory and when the code has been modified, we will clear the JIT caches.
This happens for both true self-modifying code and also libraries being loaded.
Fault handling is employed to know when code is modified in memory to ensure we can tracak changes.
This is a new setting in FEXConfig called mtrack. The older syscall only tracking path is deprecated but still available for testing.

Option to emulate x87 with 64-bit float operations

Big shout out to CallumDev for implementing this long awaited feature.

A major performance problem of emulating x86 is any older game will be compiled to use the x87 extension. This is especially true for 32-bit games.
The problem with this extension is that by default it uses 80-bit floats, which AArch64 doesn't support.
We end up emulating this entire extension using a soft-float implementaiton, which while being quite accurate, is obscenely slow.

This performance hack is now available to remove a significant amount of the overhead by operating x87 instructions using 64-bit float scalar
operations instead.
This is known to be inaccurate, but most Windows games will actually be configuring the x87 unit to be lower precision than 80-bit.
Additionally most games don't actually need the extra precision that 80-bit provides, so it is usually safe to emulate it more inaccurately.

This may still have some bugs, we know at least one game that has issues that aren't explained by pure precision problems. The feature can be enabled
in FEXConfig under the Hacks tab, look for "X87 Reduced Precision"

Clone3 syscall fixed

With Glibc 2.34 released, this project has started using the clone3 syscall for creating threads.
FEX's implementation was mostly untested which resulted in all applications breaking.
Stack pointer behaviour was broken and now with this fixed, glibc 2.34 now works out of the box.

FEXRootFSFetcher don't try to continue download

FEX-Emu's CDN doesn't support continuing file downloads. Disable to not cause issues.

FEXCore: Reclaimable thread pool allocator

FEXCore now uses an intrusive pooling allocator to allow sleeping threads to give back memory to the pool.
This allows multiple threads to share a memory resource, reducing memory usage by a significant amount if an application has a bunch of sleeping
threads.

FEXBash: Set PS1 environment variable to show running under emulation

Once running FEXBash it can be hard to tell if you're running your bash terminal under emulation.
Setting PS1 to FEXBash> makes it easier to tell that the terminal is running under emulation.

FEXBash> uname -a

Linux ryanh-TR2 5.17.5 #FEX-2206 SMP Jun 4 2022 15:11:07 x86_64 x86_64 x86_64 GNU/Linux

OpcodeDispatcher: Fixes PEXTRB

Newer Unreal engine releases were generating a PEXTRB instruction that our frontend decoder was decoding incorrectly.
Typically this would result in a crash.
This fixes both Dirt 4 and Psychonauts 2.

Misc

  • CMake
    • Add support for mold
    • Add flag for defined signed overflow handling
  • Arm64: Optimize constant generation with ADRP+ADR
  • EmulatedFiles: Fixes temporary file generation flags
  • Struct Verifier: Fixes some bugs with DRM headers not getting picked up
  • Linux v5.17 and v5.18 support
  • JIT: Code relocation support
  • OpcodeDispatcher:
    • Adds support for non-temporal loadstores
    • Implements support for PAUSE instruction
  • Syscalls:
    • 32-bit mmap syscalls fixes
      • Has been broken since the start, most applications use mmap2 instead
      • Fixes Kega Fusion
  • CompileService: Removed since it is no longer required
    • We no longer try to compile in a reentrant safe fashion
  • JITSymbols: Cleaner printing of RIP relative to a file
  • Standard TODO markers for code searching
  • Some 32-bit FS/GS writing fixes
    • Not really used so didn't affect anything

Raw Changes

FEX Release FEX-2206

FEX - FEX-2204

Published by Sonicadvance1 over 2 years ago

Changes

CPUID

FEXCore

OpcodeDispatcher

JIT

FileManager

Linux

Scripts

Softfloat

TestHarnessRunner

Misc

Termux

GDBServer

Documentation

FEX - FEX-2203

Published by Sonicadvance1 over 2 years ago

Changes

FEX - FEX-2202

Published by Sonicadvance1 over 2 years ago

Changes

FEXCore

Linux

Thunks

Tools

CMake

FMT

Vixl

Unit tests

Misc

FEX - FEX-2201

Published by Sonicadvance1 almost 3 years ago

Changes

A large amount of changes this month.

  • We now support BMI2, RDTSCP, CLZero.
  • We tell the guest what ARM CPU name we are running on.
  • We expose the hybrid CPU flag flag on big.LITTLE ARM SoCs.
  • Bunches of syscall fixes, JIT compile time speed increases, usability improvements.
  • Tons more.

A new FEXRootFSFetcher tool for downloading prebuilt RootFS images.
A Python script for automatically setting up an Ubuntu PPA and RootFS.

  • See our quick start guide to see how here
  • Or see the Ubuntu PPA here

FEXCore

Arm64

  • Fixes vixl assertions around ubfm usage (5654f9a0)
  • Fixes MapSelectCC FGT flag (13087f84)

CPUID

  • Fixes another memory overflow issue (0e45b98f)
  • Fixes crash on unknown CPU (be3e3a35)
  • CPUID Expose Hybrid flag and CPU names (9394e49c)

Context

  • Take some arguments as pointer-to-const (9d439047)

Dispatcher

  • Disables unsync context message (154468c1)
  • Adds more state reconstruction to state restore (fa6f1b1d)

OpcodeDispatcher

  • Handle PDEP (0a3a270a)
  • Handle PEXT (91984003)
  • Implements RDTSCP (ee821b9c)
  • Implements CLZero instruction (5f0dfcd7)

X86Tables

  • Build Unknown op definition tables at compile time (e8670bba)
  • Speed up initialization of X86Tables (e99a23cf)

HostFeatures

  • Detect if the host CPU suports float exceptions (117cbde2)
  • Consolidates HostFeatures flags (e24eb7a7)

IR

  • Fixes sized constant mask (565d1e27)

Misc

  • Don't use Run() inside RunUntilExit() (0bf57765)
  • Fixes a memcpy overflow in processor brand (340699ca)
  • Fixes FEXCore_Base config dependency (95bd309d)
  • Implements recoverable INTO instruction (cd4269f4)
  • Reorganizes some AOT related code (d4655fbb)

Linux Emulation/Syscalls

  • Fixes pidfd_send_signal for 32-bit (ac63a2ba)
  • Finish 32-bit msqid_ds usage (98210fdf)
  • Implements the remaining 32-bit syscalls (bc120b9f)
  • Fixes semid_ds_64 definition (f2fd9d9e)
  • Fixes 32-bit shmctl (45430e95)
  • Adds the safe syscall unimplemented gap for x86-64 (e05d116b)
  • Enable warnings (75e5df54)
  • Socket
  • Resolve sign comparison mismatch cases (097f48f3)

Misc

CMake

  • Adds an OVERRIDE_VERSION option (08085998)
  • Adds a TUNE_CPU option (51466519)
  • Adds experimental libc++ option (f3a27a57)

Build fixes

  • Fixes build on Ubuntu 20.04 take 3 (8f835678)
  • Fixes build on Ubuntu 20.04 take 2 (059e8a90)
  • Fixes build on Ubuntu 20.04 (02005e71)
  • Fixes Ubuntu 20.04 compilation on AArch64 (47bd4795)

Config

  • Enables all host threads by default (edce9818)
  • Sanitize Core option (9bffaeea)

Docs

  • Adds documentation about the FEX monthly release process (a65e7a09)

GDBServer

  • Fixes long string packet encodings (88ce9b5c)

Misc

  • Stop hardcoding /usr/bin paths (1d5bd152)
  • Improves compile ability for older libraries (c691d709)
  • More AOT code movement (43dc232e)
  • FEXMountDaemon squashfs fixes. (ea73c9d7)
  • Update jemalloc for fixed libc overrides (c43af0e1)
  • Implement thunk library generation using libclang (425d9323)
  • {x32, x64}/Thread: Make use of .data() instead of .at(0) (b6499ac7)
  • Adds a new ReentrantMutex to use for FEXCore (b9c49027)
  • NetStream: Move NetBuf definition into cpp file (9860e8b7)

Tests

gvisor

  • Disables flaky test (e62cb417)

unittests

  • Fixes a missing dependency on ASM tests (b078f511)
  • Enables remaining unaligned atomic tests on ARMv8.0 (fd042476)
  • Fixes known failures (5c4112f1)

RootFS

  • Stop trying to retry rootfs after five times (a45bc759)
  • Be more robust against stale lock files (d4b31cd4)

Tools

FEXConfig

  • Removes jemalloc usage (c65be9f5)

FEXRootFSFetcher

  • Remove 0x prefix on hash (b72245a2)
  • Adds a new tool to help set up a new RootFS (72e8a997)

Scripts

  • Adds a python script that can hand hold a user through FEX install (b276750c)
FEX - FEX-2112

Published by Sonicadvance1 almost 3 years ago

Arm64

  • Reduce the chance of hanging on reentrant allocations (8ab6830f)
  • Adds an inline syscall optimization (f7468703)
  • COND_FGT should map to gt, not hi, as it is not FGTU (b45603d1)
    • Fixes steamwebhelper burning all CPU cores!

FEXCore

  • Fixes CompileService race condition on thread creation (5345f7fa)
  • Centralize alignment utility functions in one header (966272ce)
  • Fixes ERROR_AND_DIE (fbd14b65)
  • Supports guest SIGILL (6e1ea92c)
  • CPUID
    • Call handler functions directly (ffee22d3)
  • OpcodeDispatcher
    • Remove debug message (d883b4bf)
    • Implement BZHI (50c8d9ed)
    • Amend return types of BitSize helpers (c63c5fb6)
    • Implement MULX (498a845c)
    • Resolve sign mismatches in loops (ad864e0e)
    • Add helper for getting bit sizes (3a3cbf2b)
    • Handle RORX (51d21d86)
    • Implement BMI2 SARX/SHLX/SHRX (e37a0bbd)
    • Removes usage of SignalFrame stack on JITs (d81d5e21)
    • Use std::vector as the underlying container for std::stack (61ccec37)
    • Partial support for RIP adjust in signal handler (77093a7f)
    • Resolve unused variable warnings (a04cc4dc)
  • OpcodeDecoder
    • Shorten up GPR and MM base offset retrieval (59ed91c5)
    • Convert enums into enum classes where applicable (d1fa8bcf)
    • Add support for ADX (d3cd354c)
  • Frontend
    • Handle VEX RXB bits (d47182b6)
    • Mark modrm function LUT as static (c6662a46)
  • GDBStub
  • Fixes a few hangs and crashes (e027b83a)
  • Interpreter
    • Build op handler table at compile-time (63f9e0b4)
    • Mark F80CMP array as static constexpr (13ad5b66)
  • JIT
    • Eliminate redundant jump target map lookups (8ef02783)
    • Ensures signals in compileservice JIT space is handled (97e3a364)
  • JitSymbols
    • Make use of fmt (db0740be)
  • IR
    • Make some interface functions accept pointers to const (8c38f936)
    • Convert NodeID into a strong type (b93871ff)
    • Add type alias for Node IDs (1f306d66)
  • RegisterAllocationPass
    • Resolve sign comparison mismatch in CalculateNodeInterference() (597e4e1d)
    • Reduce usages of NodeIDs where applicable (19a06519)

Linux

  • Updates syscalls to 5.16 syscalls (c4d05fb8)
  • Fixes sched_getaffinity (d5bf6414)
  • Fixes MAP_32BIT error case (92ae6ae2)
  • Fixes sigprocmask with new and old set being the same address (7bb78912)
  • Passthrough 32-bit syscalls that can be (3dc938b8)
  • Emulate MAP_32BIT on mmap (081d0169)
  • SignalDelegator update signal mask on return from sigsuspend (90511ddd)
  • Updates 32-bit DRM emulation (a92233c7)

Thunks

  • Fail loudly if thunking is enabled for a library that isn't installed (b1b61608)
  • ThunksDB
    • Updates file to include local libs (e0d40dd4)

Tests

  • Enables 32-bit host runner (cabb58ad)
  • Minor cleanup (2a894bc1)
  • Various thunk library cleanups (3730a428)

Misc

  • BucketList
    • Minor API touchups (baadf0b9)
  • CMake
    • Adds option to enable time-trace compile option (919e2bc3)
  • CompileService
    • Store WorkItem instances as unique_ptr (ef772c3e)
  • ELFCodeLoader
    • Fixes BRK allocation on some ELF loading (b327daf5)
  • EnumUtils
    • Remove shift operators from helper macro (977d92b7)
  • General
    • Migrate logging over to fmt where possible (513b02fd)
  • RootFS
    • Keep rootfs around while in container (16f1ad46)
  • IR.json/json_ir_generator: Minor touchups to generated IR utilities (9b43f94f)
  • Various packaging improvements (23e583e1)

Frontend

  • Adds Socket logger and tool (6c93b6f7)
  • FEXLoader
    • Change 32-bit memory check error to warning (82ebdf52)
    • Print memory map when 32-bit intersect happens (e6d285a7)
  • Syscalls
    • Move construction of syscall name map into GetSyscallName() (0560e9be)
FEX - FEX-2111

Published by Sonicadvance1 almost 3 years ago

Core

  • Mark relevant Interpreter/JIT functions as [[nodiscard]] (43454abc)

  • IR

    • Fixes memory ops having a duplicate size field (56e5e78b)
    • Add handling for ANDN operations (34b2f93d)

  • Frontend

    • Handle VEX source operands (7e9201cf)

  • Interpreter

    • Splits ops in to separate files (e9937d9a)

  • JIT

    • Fixes asserts added to the JIT (28d084bf)

  • JITSymbols

    • Allow grouping JIT symbols by guest named regions (ff74e0a0)
    • Change over to runtime enablement of symbols (8b3c4615)

  • Arm64

    • Fixes paranoid TSO mode (8b645465)
    • Consolidate HandleSIGBUS (babb81a2)
    • Don't fall back to native (a3b39afe)
    • Make sure to spill static FPRs on guest signal (98ba0bfa)
    • Be more explicit about x87 ABI usage (bdc66a33)

  • OpcodeDispatcher

    • Implement BLSR/BLSMSK (b4a71a21)
    • Implement handling for BLSI (285ef387)
    • Handle BMI1 BEXTR (76538be0)
    • Deduplicate OpToIndex definition (031fa8a7)

CPUID

  • Adds support for hybrid flag (a9d31227)

Linux Emulation

  • Fixes execve on softlinks in rootfs (a393d660)
  • Implements virtio ioctls for 32-bit (e0343647)
  • Fixes FEXLoader argument passing (f9078f8d)
  • Follow more symlinks in emulation (09ee6d3b)
  • No longer do magic on SIGCHLD (95457bc7)
  • Fix 32-bit nanosleep always passing valid remainder (9e8af234)

Thunks

  • Fix missing libdrm include path (fa1648c6)
  • Respect DESTDIR environment variable (6cd73a67)
  • XCB Add missing header file (8dfe305a)
  • vulkan
    • Suppress compiler warnings about unknown attributes (64aa4f00)

Misc

  • Fixes environment loader not hooked up to ArgumentLoader (8f170d4a)
  • Fix left-over printf specifier in fmt log (aa1c47cd)
  • X86Tables
    • Make flag helper functions constexpr (d09706aa)
  • Telemetry
    • Adds telemetry for when an application tears (235367b6)
  • DeadContextStoreElimination
    • Fix missing printf specifier entry (b1ab252c)
  • RAPass
    • Add debug compile option to disable spill slot reuse (36612233)
  • TestHarnessRunner
    • Make argument check more strict (eb8a8bf9)
    • Convert LOGMAN_THROW_A into error log and exit (c9c35262)
  • Cmake
    • Change static-pie message to indicate compiled without it (e547f0ca)
  • FEXConfig
    • Fixes timeout in select causing 100% CPU load (8aebbbd0)
  • Allocator
    • Reserve upper 128TB of VA on 64-bit process (cae2f8ca)
FEX - FEX-2110

Published by Sonicadvance1 about 3 years ago

  • AOTIR
    • fix use after free (327c4d55)
  • Config
    • Check for container-manager and redirect (56bddd3a)
  • Core
    • Minor documentation and code splitting (fb01e8bf)
  • FEXConfig
    • Change shortcut for opening application profile (a56463a7)
  • FEXLoader
    • Flush log output (360c4a20)
  • FEXMountDaemon
    • Fixes shutdown race conditions (c9a62658)
  • FileManager
    • Allow reading real root (ba174e5c)
  • Interpreter
    • Changes basic loadstores to sized accesses (08f56540)
  • Linux
    • Implements support for POSIX message queues on 32-bit (21ff4339)
    • Implements support for timer_create (f232dceb)
    • Fixes timex definition for 32-bit syscalls (368095f9)
    • Fixes rlimit syscalls for 32-bit (e91061f7)
  • OpcodeDispatcher
    • Ensure some x87 templates get passed long constants (1fa3afd0)
  • Scripts
    • Adds a new script for extracting function definitions (9d460e80)
  • Softfloat
    • Allow forcing use of some x87 on x86 host (d004fee7)
  • StructVerifier
    • Fix struct match and minor fixes (2b757a9b)
  • Thunks
    • Vulkan thunks (b3efb1d2)
    • Support versioned libraries (0dc8e233)
    • Install a global thunksDB for our current thunks (72125c9c)
    • Adds a few missing libGL thunk functions (9a07b550)
    • Some minor X related thunk changes (e0b878f1)
    • Expands Xext thunked functions listo (073224ff)
    • Expands what asound thunking supports (00511c16)
    • Adds a new Thunks database config file (66c7fdb6)
    • Makes file searches a bit easier (a48ed376)
    • Adds init helpers with function call (98bee5a6)
    • ThunkHelpers script improvements (8d13261a)
    • Minor fixes to the config and loading (d93edb2d)
  • Misc
    • Adds support for setting host environment variables from config (d5f9f43a)
    • Build fix for ENABLE_JITSYMBOLS (591fc001)
    • RA validation (3744ec2a)
FEX - FEX-2109

Published by skmp about 3 years ago

  • Arm64

    • Fixes SRA spilling on signal (35c66429)
    • Reimplements support for binfmt_misc without update-binfmts (5d73ac32)

  • Arm64Emitter

    • Resolves some IWYU warnings (c206942b)

  • Config

    • Moves non-OS specific configuration loading to FEXCore (6da33306)

  • EmulatedFiles

    • Fixes openat for emulated files not using FDCWD (109c42a6)

  • FEXBash

    • Allow creating a bash instance easily (9140ba28)

  • FEXConfig

    • Load application config and advanced tab (3e05544e)

  • FEXCore

    • Return the ParentThread with InitCore (97f413cf)
    • Minor symbol visibility fixes (5debdf8d)
    • Use GetCursorAddress when able (8eb0df96)

  • FEXLoader

    • Fixes potential bug in log output to stdout/stderr (10922293)

  • FEXMountDaemon

    • Fixes dangling mounts problem (dd343165)
    • Fixes some minor issues (ad34cddb)
    • Early fork to deparent child (7795078f)

  • Hangover

    • Initial support for the syscall handling. (77db25fc)

  • Linux

    • Setup signal mask correctly to block signal-in-signal situations (70931bf3)
    • sigaltstack ignore SS_ONSTACK (c74620f0)
    • Handle fpstate in the signal delegator correctly (1c4503e2)
    • Fixes 32-bit interval timers (425ee98f)
    • Fix V3d and VC4 ioctl definitions (0c484ac4)
    • x86: Initial V3D and VC4 ioctl emulation (127d7c1a)
    • Fixes accidental execve escape (1c3be542)
    • Fixes 32-bit syscalls that use 64-bit values (6afc3ca1)
    • Implements support for clone with namespaces (7aae9b7e)
    • Fixes readlinkat for self (8dd41e7f)
    • Implements pivot_root syscall (cce3f365)

  • RA

    • Add max NoteCount assert (c645d868)

  • SignalDelegator

    • More splitting and cleanup (097b3ad8)

  • Misc

    • Fix unaligned CASPair on ARMv8.0 (2c02dcac)
    • Syscall fixes (befd0aa9)
    • Properly implement single CAS on ARMv8.0 (25e95855)
    • Move BucketList into it's own file (4f66d3e9)
    • Gvisor fixes (d600b34b)
    • Massive amount of IWYU cleanup (10793e89)
    • Fixes Callback interface to take a thread argument (fa224a35)
    • Fixes jemalloc library ordering (b373d0fc)
    • Proton 6.3 32-bit fixes (8099dfc8)
    • OpcodeDispatcher: Implements undocumented repne on string ops (926ddabb)
    • 32-bit wine fixes (6a08587d)
    • Updates jemalloc to fix missing alias posix_memalign (63af80fc)
    • Adds new FEXGetConfig program (43052a57)
    • Implements support for offline only telemetry (b120a8ea)
    • Rebase skmp/no sra (0be16bae)

  • unittests

    • Hotfix for older nasm (9563b5aa)

  • x86

    • Fixes siginfo_t si_addr for SIGBUS/SIGSEGV (6b878394)
FEX - FEX-2108

Published by skmp about 3 years ago

  • 64BitAllocator

    • Convert std::vector table into std::array (ec6cc5cb)

  • AArch64

    • Workaround static-pie crashing (4c431518)

  • Arm64

    • Fixes fill and spill slot offset calculation (6ac23003)

  • CPUID

    • Update TM and TM2 CPUID bits (d142d7c3)

  • CPack

    • Update package name to remove conflict (2d11df03)

  • FEXLoader

    • stderr on immediate failure (84dee03a)

  • FEXMountDaemon

    • Make squashfs mounting more robust (ff98a43f)

  • FlexBitSet

    • Use non-template type parameter for index parameters (7931dc2c)

  • GdbStub

    • Fix memory leak in GdbServerLoop() (749f1eba)

  • General

    • Remove redundant string constructions in log calls related to IR op name retrieval (80caf903)

  • Jits

    • Migrate logs over to fmt where applicable (7052d022)

  • Linux

    • Implements support for vsyscall (a5b142cf)
    • Implements rt_{tg,}sigqueueinfo (1a64c908)
    • Implements a base implementation of signalfd{4,} (2db44cf2)
    • Remove logs about parent or child stack usage (336eb9a8)

  • OpDispatcher

    • Fixes imul flags calculations (7dd70039)

  • OpcodeDispatcher

    • Split the opcode handling to multiple files (2fd41de5)

  • RCLSE

    • Fixes an assumption in RCLSE (896aa528)

  • Misc

    • Linux: Implements support for signals 32 and 33 (7e8577a5)
    • Wine fixes (de0337db)
    • Cleanup compile service on fork (a48e41e4)
    • Adds an option to stall processes on launch (03159648)
    • Adds an option to disable jemalloc from cmake (dec82219)
    • binfmt_misc: Support loading ELFs from FD and support preserve (007b1cbe)
    • Fixes bug in RAPass (b0a28e59)
    • Works around static-pie crashing (62775161)
    • Adds cmake option ENABLE_STATIC_PIE (aa767b39)
    • Fixes some issues I found when running under tsan and asan (14480f3e)
    • Fixes binfmt_misc install when not installed to /usr (04aacea5)
    • Validation Passes: Remove unused variables (7f5b3524)
    • Fixes flag setting for 8bit and 16bit LOCK ALU ops (3a0da68b)
    • Adds a convenience of named thunk configs (f4e044a5)
    • Update man page to show JSON key for config options (108081e2)
    • Fixes Host thunks lib not being able to GL (6ff9b966)
    • Allocator fixes (a24d63b5)
    • Atomic improvements (53417e53)
    • Actually fix jemalloc for FEXConfig (35a4f52d)
    • Switch some memory allocations over to mmap (1eee2dd0)
    • Update jemalloc to fix glibc override. (0d9ab9ca)
    • Fix jemalloc malloc replace (a9bb33b8)
    • Fixes old kernel defines for sockios (8f09bd92)
    • Use externals xxhash if not found installed (0a8ee430)
FEX - FEX-2107

Published by skmp over 3 years ago

Compatibility & Bug Fixes

  • Fixes bugs in unaligned atomic signal handlers
  • CPUID cleanups
  • Fixed a memory leak in Register Allocation
  • Several syscall fixes (pidfd_send_signal, arch_prctl, fcntl, send(m)msg, shmdt, recvmsg, *chown32, edge cases around time syscalls, 32/tmpfile)
  • Implement more syscalls (setfsuid32, setfsgid32, getgroups32, setgroups32, settimeofday, futimesat, utimes, 32/sigpending, 32/truncate64)
  • Deferred signal handler registration, fixes bash (pts handover)
  • Fixed handling of some rare elf files
  • Implemented 32 bit iret

Usability

  • Adds support for squashfs based rootfses
  • cpack support for debian packaging

Performance

  • Huge page support in our VA allocator

Misc

  • Reduce warnings
  • Several assorted Cleanups
  • Relocate ELF handling logic to the os frontend
  • Improve logging SNR
FEX - FEX-2106

Published by skmp over 3 years ago

Compatibility & Bug Fixes

  • Several syscall bug fixes (pselect, epool, semctl, msgctl, pidfd_getfd, poll, fadvise64, sigaction, mmap, epoll, uname, openat, ...)
  • Implemented SSE4.1

Performance

  • Multi threaded AOTGen
  • Optimized ioctl/drm marshaling

Misc

  • Several minor cleanups & refactors
FEX - FEX-2105

Published by skmp over 3 years ago

Compatibility & Bug Fixes

  • New, simpler ELF Loader
  • ioctl32 marshaling for several devices
  • Handle more cases of an application pinging self
  • binfmt_misc fixes to support AppImage
  • Several system call fixes (eventfd, eventfd2, openat, creat, trucnate, clock_nanosleep, pselect6, sendmsg, recvmsg)
  • Signal fixes
  • Better handling of /proc/self/, /proc/pid-self/
  • Implements support for CLFLUSH

Performance

  • Faster AOTIR file loading
  • AOTIR offline generation
  • Switch to xxhash from fasthash
  • IR structure optimizations
  • Adds Long Divide removal pass
  • Zero-cost asserts

Misc

  • Fixes host thread stacks ending up in lower 32-bit VA
  • Remove reliance on librt and libnuma
  • Default hidden visibility and strip symbols
  • FEX now uses jemalloc
FEX - FEX-2104

Published by skmp over 3 years ago

Compatibility & Bug Fixes

  • Disables RCPC on ARM64 JIT
  • Implements CPUID 0x8000'0005 for L1 cacheline information
  • Implements CPUID 0x8000'0006 for cacheline information
  • Implements MOVNTDQA
  • Restrict imm code motion around selects to matching sizes, fixes dav1d
  • Validate LOCK handling, add missing segment offsets
  • Add atomic logic for SecondaryALUOp
  • Adds support for locked NOT
  • Adds support for locked ADC and SBB
  • Adds a couple new 32bit syscalls
  • Fixes an edge case of 32bit cmpxchg <reg>, <reg>
  • Lock around FDToNameMap accesses (Fixes Geekbench 4 stability issues)
  • Flush context around OP_SYSCALLs, Syscalls might read it
  • SA_NOCLDSTOP only blocks CLD_CONTINUED/STOPPED/TRAPPED
  • Add missing break for UD2 in INTOp
  • Init on X87FNSAVE, fix FNINIT
  • Switches FEXCore over to pthreads implementation

Usability

  • Support for global application profiles
  • Thunks can be configured with json as an overlay
  • FEXConfig improvements
  • Adds support for Named RootFS folders in FEXConfig
  • Cleanup threads when they exit
  • Default to no logging
  • Allows installing of FEXThunks in our data directory

Documentation

  • New Readme.md & auto generated SourceOutline.md to help newcomers to the codebase
  • Man Pages

Internal restructuring

  • Unify all four dispatchers
  • Separate thread and state
  • Allow both ARM64 and X86_64 jits to be compiled at the same time
FEX - FEX-2103

Published by skmp over 3 years ago

Compatibility

  • Support for unaligned atomic memory ops
  • Thread local mman cache invalidation
  • glib 2.32 support (cpuid fixes, faccessat2)
  • cmpxchg flag fixes
  • fixed cvtt* ops to actually truncate, both for x87 and sse
  • Implemented BTC, BTR and BTS atomic variants
  • Workaround crashes around exit_group for graceful exit
  • Fixed SAR8 & SAR16 sign extension
  • Added support for x87 rounding modes
  • Added support for x87 precision modes
  • Temporary fix for x87's frem
  • Fixed select system calls to update the descriptors
  • Fixed handling of invalid ops when multiblock is enabled
  • Fixed an edge case bug in CreateElementPair
  • Fixed BRK handling
  • Fixed /proc/cpuinfo topology
  • Fixed compilation of superblocks with more than 256 spills
  • Fixed a bug in in the MUL -> SHL optimization

Performance

  • Added ir-cache that halves loading times when enabled (--aotir-capture && --aotir-load)
  • x87 ops no longer force interpreter for the entire superblock

Usability

  • uname now returns the host name
  • Added checks for python in the CMake files

Misc

  • ThunkLibs now generate asm wrappers, so they can be compiled with older GCCs
  • Asserts on unsupported atomic ops
  • Removed erratic asserts in RA
FEX - FEX-2102

Published by skmp over 3 years ago

Compatibility

  • Unaligned cmpxchg & cmpxchg8b on ARMv8.1+
  • Mutliblock now gracefully handles unsupported / invalid instructions. This makes it safe to always enable multi block
  • Several fixes for 32-bit binaries: branch handling, signal return, memory allocation, ioctl32 for x86-64 host
  • Improved BRK handling

Perfomance

  • Much reduced stuttering during JIT compilation. The JIT optimizer is now over 3x as fast.
  • Per-thread IR Caching with cached RA, for faster recovery from code cache resets

Usability

  • Defaults now to jit, multiblock, 5000 instructions.
  • CPUID returns fex version

Misc

  • Cleaned up IR printing, extended asm tests to test IR dumping & IR printing
  • Integrated gcc target tests to our CI
  • Removed many warnings

The detailed change log is available here

Related Projects
quickemu

Quickly create and run optimised Windows, macOS and Linux virtual machines

15 Mar 2020 10,152
linux-installer

Universal GNU+Linux installer script

08 Nov 2020 20
linux-lab

Docker/Qemu Based Linux Kernel Learning, Development and Testing Environment; New Linux ELF Video...

11 Jul 2016 1,150
heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible bo...

03 Aug 2016 1,402
linuxboot

The LinuxBoot project is working to enable Linux to replace your firmware on all platforms.

21 Dec 2017 857
snacklinux

Linux distribution for lazy people

10 Mar 2013 16
u-root

A fully Go userland with Linux bootloaders! u-root can create a one-binary root file system (init...

19 Nov 2015 2,430
linux-kernel-module-cheat

The perfect emulation setup to study and develop the Linux kernel v5.4.3, kernel modules, QEMU, g...

30 Jul 2016 3,947
box86

Box86 - Linux Userspace x86 Emulator with a twist, targeted at ARM Linux devices

09 Jan 2019 3,123