macOS: full support — MoltenVK auto-discovery, portability fix, and KytyPS5.app bundle (#394)

* macOS: auto-discover bundled MoltenVK on startup

On macOS the Vulkan loader comes from MoltenVK. SDL's default
probe only searches system paths, so a bundled libMoltenVK.dylib
next to the executable is invisible under the hardened runtime.

Probe it explicitly via SDL_GetBasePath() before falling back to
SDL's default search. Respect SDL_VULKAN_LIBRARY overrides and
log which loader is used. Update README to reflect that no
environment variable is required for bundled builds.

Co-Authored-By: internal-model

* window: fix Vulkan portability enumeration on macOS

MoltenVK requires VK_KHR_portability_enumeration and
VK_INSTANCE_CREATE_ENUMERATE_PORTABILITY_BIT_KHR to surface
portability devices via enumeratePhysicalDevices. Without the flag
the adapter is hidden on some driver versions, causing
"Could not find suitable device" failures.

Add the instance extension and flag when available, and add
diagnostic logs for all MoltenVK fallbacks (depthClipEnable,
colorWriteEnable, depthBounds, fragmentShaderBarycentric)
so black-screen diagnostics are actionable.

* macOS: add KytyPS5.app bundle packaging

Add a proper macOS application bundle when the launcher is built.
launcher becomes KytyPS5.app (CFBundleIdentifier org.kytyps5.kytyps5)
with KytyPS5 as the executable, and kyty_emulator is installed as a
helper inside Contents/MacOS so QCoreApplication::applicationDirPath()
finds it. MoltenVK is auto-discovered via SDL_GetBasePath() in that
directory, so no SDL_VULKAN_LIBRARY is required.

- src/launcher/Info.plist.in: minimal bundle plist (version from
  PROJECT_VERSION, LSMinimumSystemVersion 13.0).
- src/launcher/CMakeLists.txt: set MACOSX_BUNDLE, BUNDLE DESTINATION,
  install helper, re-sign helper with JIT entitlements, and invoke
  macdeployqt when available. Keep flat kyty_emulator for CLI.
- .github/workflows/build.yml: bundle MoltenVK into
  KytyPS5.app/Contents/MacOS when present, verify bundle artifacts
  (file/lipo/codesign), and handle bundle paths in the release zip.
- README.md: document bundle double-click and MoltenVK locations.

* launcher: fix bundle rpath for Qt frameworks

The bundle executable had @rpath references to Qt frameworks but no
LC_RPATH, causing dyld to abort at launch:

  Library not loaded: @rpath/QtConcurrent.framework
  Reason: no LC_RPATH's found

Set INSTALL_RPATH to @executable_path/../Frameworks and
BUILD_WITH_INSTALL_RPATH so the built and installed bundles
resolve Qt from Contents/Frameworks. Verified via otool -l
(LC_RPATH now present) and macdeployqt now succeeds without
"Cannot resolve rpath" errors.

* launcher: add Import game button

Add a prominent Import button to ConfigurationListWidget that lets
users select a game folder (containing eboot.bin) or a folder
containing multiple games via QFileDialog. Handles:

- Single game folder selected: adds its parent to game_dirs (Scan
  searches subdirectories, so parent is needed)
- Folder with multiple games: adds the selected folder
- Deduplication via PathKey/NormalizeGameDirectories, writes
  settings, rescans, and selects the imported game if found
- Shows info/warning dialogs when no games are found

UI: QToolButton with SP_DirOpenIcon and text Import beside the
existing toolbar, styled with a blue background for visibility.
Logic in ConfigurationListWidget::import_game().

* macOS: fix bundled loader and artifact packaging

* Revert "launcher: add Import game button"

This reverts commit 2f670ef95c.

---------

Co-authored-by: nmzik <Nmzik@mail.ru>

.github/workflows/build.yml

@@ -208,34 +208,66 @@ jobs:
         run: |
           archive="$RUNNER_TEMP/MoltenVK-macos.tar"
           package="$RUNNER_TEMP/MoltenVK"
+          install_dir="_Build/macos/install"
+          app_dir="$install_dir/KytyPS5.app"
+          moltenvk="$install_dir/libMoltenVK.dylib"
           curl --fail --location --retry 3 \
             --output "$archive" \
             "https://github.com/KhronosGroup/MoltenVK/releases/download/$MOLTENVK_VERSION/MoltenVK-macos.tar"
           echo "$MOLTENVK_SHA256  $archive" | shasum -a 256 --check
           tar -xf "$archive" -C "$RUNNER_TEMP"
+          test -d "$app_dir"
           install -m 755 \
             "$package/MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib" \
-            _Build/macos/install/libMoltenVK.dylib
-          install -m 644 "$package/LICENSE" _Build/macos/install/LICENSE.MoltenVK
-          codesign --force --sign - --timestamp=none _Build/macos/install/libMoltenVK.dylib
+            "$moltenvk"
+          install -m 644 "$package/LICENSE" "$install_dir/LICENSE.MoltenVK"
+          codesign --force --sign - --timestamp=none "$moltenvk"
+          install -d -m 755 "$app_dir/Contents/Frameworks" "$app_dir/Contents/Resources"
+          install -m 755 "$moltenvk" "$app_dir/Contents/Frameworks/libMoltenVK.dylib"
+          install -m 644 "$package/LICENSE" "$app_dir/Contents/Resources/LICENSE.MoltenVK"
+          codesign --force --sign - --timestamp=none "$app_dir"
 
       - name: Verify artifacts
         shell: bash
         run: |
-          file _Build/macos/install/launcher
-          file _Build/macos/install/kyty_emulator
-          file _Build/macos/install/libMoltenVK.dylib
-          lipo _Build/macos/install/launcher -verify_arch x86_64
-          lipo _Build/macos/install/kyty_emulator -verify_arch x86_64
-          lipo _Build/macos/install/libMoltenVK.dylib -verify_arch x86_64
-          codesign --verify --strict _Build/macos/install/kyty_emulator
-          codesign --verify --strict _Build/macos/install/libMoltenVK.dylib
+          install_dir="_Build/macos/install"
+          app_dir="$install_dir/KytyPS5.app"
+          test -d "$app_dir"
+          for binary in \
+            "$install_dir/kyty_emulator" \
+            "$install_dir/libMoltenVK.dylib" \
+            "$app_dir/Contents/MacOS/KytyPS5" \
+            "$app_dir/Contents/MacOS/kyty_emulator" \
+            "$app_dir/Contents/Frameworks/libMoltenVK.dylib"; do
+            file "$binary"
+            lipo "$binary" -verify_arch x86_64
+          done
+          for binary in \
+            "$install_dir/kyty_emulator" \
+            "$install_dir/libMoltenVK.dylib" \
+            "$app_dir/Contents/MacOS/KytyPS5" \
+            "$app_dir/Contents/MacOS/kyty_emulator" \
+            "$app_dir/Contents/Frameworks/libMoltenVK.dylib"; do
+            codesign --verify --strict "$binary"
+          done
+          codesign --verify --deep --strict --verbose=2 "$app_dir"
+
+      - name: Package macOS artifacts
+        shell: bash
+        run: |
+          install_dir="_Build/macos/install"
+          archive="$RUNNER_TEMP/KytyPS5-macOS-x86_64.zip"
+          (
+            cd "$install_dir"
+            zip -qry "$archive" .
+          )
+          unzip -t "$archive"
 
       - name: Upload macOS artifacts
         uses: actions/upload-artifact@v4
         with:
           name: KytyPS5-macOS-x86_64
-          path: _Build/macos/install/**
+          path: ${{ runner.temp }}/KytyPS5-macOS-x86_64.zip
           if-no-files-found: error
 
   linux:
@@ -424,16 +456,9 @@ jobs:
         if: needs.macos.result == 'success'
         shell: bash
         run: |
-          macos_dir="artifacts/KytyPS5-macOS-x86_64"
-          test -d "$macos_dir"
-          chmod a+x \
-            "$macos_dir/launcher" \
-            "$macos_dir/kyty_emulator" \
-            "$macos_dir/libMoltenVK.dylib"
-          (
-            cd "$macos_dir"
-            zip -r "$GITHUB_WORKSPACE/$RELEASE_NAME-macOS-x86_64.zip" .
-          )
+          macos_archive="artifacts/KytyPS5-macOS-x86_64/KytyPS5-macOS-x86_64.zip"
+          test -f "$macos_archive"
+          cp "$macos_archive" "$RELEASE_NAME-macOS-x86_64.zip"
 
       - name: Package Linux build
         if: needs.linux.result == 'success'

.gitignore

@@ -5,3 +5,4 @@ build/
 _Build/vscode-clang/
 _Build/
 cmake-build-debug/
+_PipelineCache/

.vscode/launch.json

@@ -10,7 +10,14 @@
             "stopAtEntry": false,
             "cwd": "${workspaceFolder}\\_Build\\vscode-clang\\launcher",
             "environment": [],
-            "externalConsole": false
+            "externalConsole": false,
+            "linux": {
+                "type": "cppdbg",
+                "program": "${workspaceFolder}/_Build/vscode-clang/launcher/launcher",
+                "cwd": "${workspaceFolder}/_Build/vscode-clang/launcher",
+                "MIMode": "gdb",
+                "miDebuggerPath": "/usr/bin/gdb"
+            }
         },
         {
             "name": "Debug kyty_emulator",
@@ -25,13 +32,40 @@
             "stopAtEntry": false,
             "cwd": "${workspaceFolder}\\_Build\\vscode-clang",
             "environment": [],
-            "externalConsole": true
+            "externalConsole": true,
+            "linux": {
+                "type": "cppdbg",
+                "program": "${workspaceFolder}/_Build/vscode-clang/kyty_emulator",
+                "cwd": "${workspaceFolder}/_Build/vscode-clang",
+                "externalConsole": false,
+                "MIMode": "gdb",
+                "miDebuggerPath": "/usr/bin/gdb",
+                "setupCommands": [
+                    {
+                        "description": "Pass Kyty's internal signals to its handlers",
+                        "text": "handle SIG37 SIGSEGV SIGBUS SIGILL nostop noprint pass",
+                        "ignoreFailures": false
+                    }
+                ]
+            }
         },
         {
             "name": "Attach to running kyty_emulator",
             "type": "cppvsdbg",
             "request": "attach",
-            "processId": "${command:pickProcess}"
+            "processId": "${command:pickProcess}",
+            "linux": {
+                "type": "cppdbg",
+                "MIMode": "gdb",
+                "miDebuggerPath": "/usr/bin/gdb",
+                "setupCommands": [
+                    {
+                        "description": "Pass Kyty's internal signals to its handlers",
+                        "text": "handle SIG37 SIGSEGV SIGBUS SIGILL nostop noprint pass",
+                        "ignoreFailures": false
+                    }
+                ]
+            }
         }
     ]
 }

CMakeLists.txt

@@ -254,7 +254,7 @@ foreach(shader_name IN LISTS gpu_tiler_shader_names)
 	add_custom_command(
 		OUTPUT "${shader_header}"
 		COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_tiler_generated_dir}"
-		COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.0 -Os
+		COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.3 -Os
 		        "-I${gpu_tiler_shader_dir}" -o "${shader_spv}" "${shader_source}"
 		COMMAND ${CMAKE_COMMAND} -DINPUT=${shader_spv} -DOUTPUT=${shader_header}
 		        -DSYMBOL=${shader_symbol} -P "${KYTY_SOURCE_DIR}/embed_spirv.cmake"
@@ -263,6 +263,22 @@ foreach(shader_name IN LISTS gpu_tiler_shader_names)
 	)
 	list(APPEND gpu_tiler_shader_headers "${shader_header}")
 endforeach()
+
+set(fault_buffer_shader_source "${gpu_tiler_shader_dir}/fault_buffer_process.comp")
+set(fault_buffer_shader_spv "${gpu_tiler_generated_dir}/fault_buffer_process.spv")
+set(fault_buffer_shader_header "${gpu_tiler_generated_dir}/fault_buffer_process_spv.h")
+add_custom_command(
+	OUTPUT "${fault_buffer_shader_header}"
+	COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_tiler_generated_dir}"
+	COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.3 -Os
+	        -DCACHING_PAGEBITS=14 -DMAX_PAGE_FAULTS=1024
+	        -o "${fault_buffer_shader_spv}" "${fault_buffer_shader_source}"
+	COMMAND ${CMAKE_COMMAND} -DINPUT=${fault_buffer_shader_spv} -DOUTPUT=${fault_buffer_shader_header}
+	        -DSYMBOL=FAULT_BUFFER_PROCESS_SPV -P "${KYTY_SOURCE_DIR}/embed_spirv.cmake"
+	DEPENDS "${fault_buffer_shader_source}"
+	VERBATIM
+)
+list(APPEND kyty_emulator_src "${fault_buffer_shader_header}")
 list(APPEND kyty_emulator_src ${gpu_tiler_shader_headers})
 
 set(gpu_blit_generated_dir "${PROJECT_BINARY_DIR}/gpu_blit_shaders")
@@ -278,7 +294,7 @@ foreach(shader_source IN LISTS gpu_blit_shader_sources)
 	add_custom_command(
 		OUTPUT "${shader_header}"
 		COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_blit_generated_dir}"
-		COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.0 -Os
+		COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.3 -Os
 		        "-I${gpu_tiler_shader_dir}" -o "${shader_spv}" "${shader_source}"
 		COMMAND ${CMAKE_COMMAND} -DINPUT=${shader_spv} -DOUTPUT=${shader_header}
 		        -DSYMBOL=${shader_symbol} -P "${KYTY_SOURCE_DIR}/embed_spirv.cmake"
@@ -430,6 +446,7 @@ add_executable(resource_tracking_tests EXCLUDE_FROM_ALL
 	"${KYTY_TESTS_DIR}/ResourceTrackingTests.cpp"
 	"${KYTY_SOURCE_DIR}/graphics/guest_gpu/gpu_format.cpp"
 	"${KYTY_SOURCE_DIR}/graphics/host_gpu/hostMemory.cpp"
+	"${KYTY_SOURCE_DIR}/graphics/shader/shaderPixelParameter.cpp"
 	"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/SrtWalker.cpp"
 	"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ResourceTracking.cpp"
 	"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ResourceMaterialization.cpp"
@@ -483,7 +500,7 @@ set(gpu_test_ms_depth_header
 add_custom_command(
 	OUTPUT "${gpu_test_ms_depth_header}"
 	COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_test_generated_dir}"
-	COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.0 -Os
+	COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.3 -Os
 	        -o "${gpu_test_ms_depth_spv}" "${gpu_test_ms_depth_source}"
 	COMMAND ${CMAKE_COMMAND} -DINPUT=${gpu_test_ms_depth_spv}
 	        -DOUTPUT=${gpu_test_ms_depth_header} -DSYMBOL=GPU_TEST_MS_DEPTH_SPV
@@ -507,6 +524,16 @@ if(NOT KYTY_CLANG_CL)
 endif()
 
 if(BUILD_TESTING)
+	add_test(NAME emulator_user_name_cli
+	         COMMAND $<TARGET_FILE:kyty_emulator> --help --user-name 1234567890123456)
+	add_test(NAME emulator_user_name_too_long
+	         COMMAND $<TARGET_FILE:kyty_emulator> --help --user-name 12345678901234567)
+	set_tests_properties(emulator_user_name_too_long PROPERTIES WILL_FAIL TRUE)
+	add_test(NAME emulator_present_mode_cli
+	         COMMAND $<TARGET_FILE:kyty_emulator> --help --present-mode Immediate)
+	add_test(NAME emulator_present_mode_invalid
+	         COMMAND $<TARGET_FILE:kyty_emulator> --help --present-mode Invalid)
+	set_tests_properties(emulator_present_mode_invalid PROPERTIES WILL_FAIL TRUE)
 	add_test(NAME ime_dialog COMMAND $<TARGET_FILE:ime_dialog_tests>)
 	add_test(NAME shader_cfg COMMAND $<TARGET_FILE:shader_cfg_tests>)
 	add_test(NAME scalar_provenance COMMAND $<TARGET_FILE:scalar_provenance_tests>)

README.md

@@ -220,17 +220,23 @@ cmake --install _Build/macos --prefix _Build/macos/install
 ```
 
 The build re-signs `kyty_emulator` with the JIT entitlements it needs to execute translated
-guest code; no manual signing step is required.
+guest code; no manual signing step is required. When the launcher is built, the install
+also produces `_Build/macos/install/KytyPS5.app` — double-click to launch the GUI.
+A flat `kyty_emulator` is kept for CLI usage.
 
 Vulkan comes from MoltenVK. Download `MoltenVK-macos.tar` from the
 [MoltenVK releases](https://github.com/KhronosGroup/MoltenVK/releases), then copy
-`MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib` next to `kyty_emulator` and ad-hoc sign it:
+`MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib` next to the flat `kyty_emulator`
+(and, for the bundle, into `KytyPS5.app/Contents/Frameworks/`) and ad-hoc sign it:
 
 ```bash
 codesign --force --sign - _Build/macos/install/libMoltenVK.dylib
+# For the bundle (if present):
+codesign --force --sign - _Build/macos/install/KytyPS5.app/Contents/Frameworks/libMoltenVK.dylib
+codesign --force --sign - _Build/macos/install/KytyPS5.app
 ```
 
-Release archives already include a signed `libMoltenVK.dylib`.
+Release archives already include a signed `libMoltenVK.dylib` (both flat and inside the bundle).
 
 ### Regression tests
 
@@ -274,6 +280,10 @@ To use the graphical launcher:
 ./_Build/linux/install/launcher
 ```
 
+```bash
+open _Build/macos/install/KytyPS5.app  # or double-click in Finder
+```
+
 On first launch, add one or more game folders in the global settings. The launcher searches those
 folders recursively for game directories containing `eboot.bin`. Select a detected game and run it
 from the game list.
@@ -288,12 +298,17 @@ The emulator can also be started directly with a legally obtained game directory
 ./_Build/linux/install/kyty_emulator --game "/games/ExampleGame"
 ```
 
-On macOS, point SDL at the MoltenVK library explicitly; the hardened runtime prevents it from
-being picked up from the executable's directory:
+On macOS, the adjacent flat or app-bundled `libMoltenVK.dylib` is found automatically; no
+environment variable is required:
 
 ```bash
-cd _Build/macos/install
-SDL_VULKAN_LIBRARY="$PWD/libMoltenVK.dylib" ./kyty_emulator --game "/games/ExampleGame"
+./_Build/macos/install/kyty_emulator --game "/games/ExampleGame"
+```
+
+To override the Vulkan loader, set `SDL_VULKAN_LIBRARY`:
+
+```bash
+SDL_VULKAN_LIBRARY=/path/to/libMoltenVK.dylib ./kyty_emulator --game "/games/ExampleGame"
 ```
 
 Run `kyty_emulator --help` to see the available graphics, logging, validation, profiling, and

src/common/emulatorConfig.cpp

@@ -21,6 +21,8 @@ void Shutdown() {
 
 void Load(const ConfigOptions& cfg) {
 	EXIT_IF(g_config == nullptr);
+	EXIT_IF(cfg.user_name.empty() || cfg.user_name.size() > MAX_USER_NAME_LENGTH);
+	EXIT_IF(!IsConfiguredUserIdValid(cfg.user_id));
 
 	*g_config = cfg;
 }
@@ -33,6 +35,18 @@ uint32_t GetScreenHeight() {
 	return g_config->screen_height;
 }
 
+const std::string& GetUserName() {
+	return g_config->user_name;
+}
+
+int32_t GetUserId() {
+	return g_config->user_id;
+}
+
+PresentMode GetPresentMode() {
+	return g_config->present_mode;
+}
+
 bool FullscreenEnabled() {
 	return g_config->fullscreen_enabled;
 }

src/common/emulatorConfig.h

@@ -3,6 +3,7 @@
 
 #include "common/common.h"
 
+#include <cstddef>
 #include <filesystem>
 #include <string>
 #include <vector>
@@ -26,14 +27,27 @@ enum class ProfilerDirection { None, Network };
 
 enum class OutputDirection { Silent, Console, File };
 
+enum class PresentMode { Fifo, Mailbox, Immediate };
+
 using Keymap = std::vector<std::string>;
 
 constexpr uint32_t DEFAULT_CONSOLE_LANGUAGE = 1;
 constexpr uint32_t MAX_CONSOLE_LANGUAGE     = 29;
+constexpr std::size_t MAX_USER_NAME_LENGTH = 16;
+constexpr int32_t DEFAULT_USER_ID           = 1000;
+
+constexpr bool IsConfiguredUserIdValid(int32_t user_id) {
+	constexpr int32_t USER_ID_EVERYONE = 0xfe;
+	constexpr int32_t USER_ID_SYSTEM   = 0xff;
+	return user_id >= 0 && user_id != USER_ID_EVERYONE && user_id != USER_ID_SYSTEM;
+}
 
 struct ConfigOptions {
 	uint32_t               screen_width                = 1280;
 	uint32_t               screen_height               = 720;
+	std::string            user_name                   = "Kyty";
+	int32_t                user_id                     = DEFAULT_USER_ID;
+	PresentMode            present_mode                = PresentMode::Fifo;
 	bool                   fullscreen_enabled          = false;
 	uint32_t               vblank_frequency            = 60;
 	uint32_t               console_language            = DEFAULT_CONSOLE_LANGUAGE;
@@ -45,7 +59,7 @@ struct ConfigOptions {
 	bool                   command_buffer_dump_enabled = false;
 	std::filesystem::path  command_buffer_dump_folder  = "_Buffers";
 	bool                   graphics_debug_dump_enabled = false;
-	OutputDirection        printf_direction            = OutputDirection::Console;
+	OutputDirection        printf_direction            = OutputDirection::Silent;
 	std::filesystem::path  printf_output_file          = "_kyty.txt";
 	ProfilerDirection      profiler_direction          = ProfilerDirection::None;
 	bool                   spirv_debug_printf_enabled  = false;
@@ -63,6 +77,9 @@ void Load(const ConfigOptions& cfg);
 
 uint32_t GetScreenWidth();
 uint32_t GetScreenHeight();
+const std::string& GetUserName();
+int32_t  GetUserId();
+PresentMode GetPresentMode();
 bool     FullscreenEnabled();
 uint32_t GetVblankFrequency();
 uint32_t GetConsoleLanguage();

src/common/file.cpp

@@ -25,9 +25,6 @@
 #ifdef CopyFile
 #undef CopyFile
 #endif
-#ifdef MoveFile
-#undef MoveFile
-#endif
 #endif
 
 namespace Common {
@@ -544,14 +541,14 @@ bool File::CopyFile(const std::filesystem::path& src,
 	return SysFileCopyFile(src, dst);
 }
 
-bool File::MoveFile(const std::filesystem::path& src,
-                    const std::filesystem::path& dst) // @suppress("Member declaration not found")
+bool File::RenameFile(const std::filesystem::path& src,
+                      const std::filesystem::path& dst) // @suppress("Member declaration not found")
 {
 	if (IsFileExisting(dst)) {
 		DeleteFile(dst); // @suppress("Invalid arguments")
 	}
 
-	return SysFileMoveFile(src, dst);
+	return SysFileRenameFile(src, dst);
 }
 
 void File::RemoveReadonly(const std::filesystem::path& name) {

src/common/file.h

@@ -20,9 +20,6 @@
 #ifdef CopyFile
 #undef CopyFile
 #endif
-#ifdef MoveFile
-#undef MoveFile
-#endif
 #endif
 
 namespace Common {
@@ -107,7 +104,7 @@ public:
 	static std::vector<FindInfo> FindFiles(const std::filesystem::path& path);
 	static std::vector<DirEntry> GetDirEntries(const std::filesystem::path& path);
 	static bool CopyFile(const std::filesystem::path& src, const std::filesystem::path& dst);
-	static bool MoveFile(const std::filesystem::path& src, const std::filesystem::path& dst);
+	static bool RenameFile(const std::filesystem::path& src, const std::filesystem::path& dst);
 	static void RemoveReadonly(const std::filesystem::path& name);
 
 	KYTY_CLASS_NO_COPY(File);

src/common/platform/sysFileIO.h

@@ -80,7 +80,7 @@ void SysFileFindFiles(const std::filesystem::path& path, std::vector<sys_file_fi
 // NOLINTNEXTLINE(google-runtime-references)
 void SysFileGetDents(const std::filesystem::path& path, std::vector<sys_dir_entry_t>& out);
 bool SysFileCopyFile(const std::filesystem::path& src, const std::filesystem::path& dst);
-bool SysFileMoveFile(const std::filesystem::path& src, const std::filesystem::path& dst);
+bool SysFileRenameFile(const std::filesystem::path& src, const std::filesystem::path& dst);
 void SysFileRemoveReadonly(const std::filesystem::path& name);
 
 #endif /* KYTY_COMMON_PLATFORM_SYSFILEIO_H_ */

src/common/platform/sysLinuxFileIO.cpp

@@ -693,7 +693,7 @@ bool SysFileCopyFile(const std::filesystem::path& src, const std::filesystem::pa
 	       !error;
 }
 
-bool SysFileMoveFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
+bool SysFileRenameFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
 	auto real_src = get_internal_name(src);
 	auto real_dst = get_internal_name(dst);
 

src/common/platform/sysWindowsFileIO.cpp

@@ -565,7 +565,7 @@ bool SysFileCopyFile(const std::filesystem::path& src, const std::filesystem::pa
 	return CopyFileW(src_wide.c_str(), dst_wide.c_str(), FALSE) != 0;
 }
 
-bool SysFileMoveFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
+bool SysFileRenameFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
 	auto src_wide = src.wstring();
 	auto dst_wide = dst.wstring();
 	return MoveFileW(src_wide.c_str(), dst_wide.c_str()) != 0;

src/emulator.cpp

@@ -15,6 +15,7 @@
 #include "kernel/fileSystem.h"
 #include "kernel/memory.h"
 #include "kernel/pthread.h"
+#include "kytyGitVersion.h"
 #include "libs/agc.h"
 #include "libs/audio.h"
 #include "libs/controller.h"
@@ -26,13 +27,31 @@
 
 #include <cstdlib>
 #include <filesystem>
+#include <thread>
 
 namespace Emulator {
 
 static void PrintSystemInfo() {
-	Common::SystemInfo info = Common::GetSystemInfo();
+	const Common::SystemInfo info = Common::GetSystemInfo();
 
-	LOGF("ProcessorName = %s\n", info.ProcessorName.c_str());
+#if defined(__APPLE__)
+	static constexpr auto platform_name = "macOS";
+#elif KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
+	static constexpr auto platform_name = "Windows";
+#elif KYTY_PLATFORM == KYTY_PLATFORM_LINUX
+	static constexpr auto platform_name = "Linux";
+#else
+	static constexpr auto platform_name = "Unknown";
+#endif
+
+	LOGF("Build\n"
+	     "  version: %s\n\n"
+	     "Host\n"
+	     "  os:      %s\n"
+	     "  cpu:     %s\n"
+	     "  threads: %u\n\n",
+	     KYTY_BUILD_LABEL, platform_name, info.ProcessorName.c_str(),
+	     std::thread::hardware_concurrency());
 }
 
 static void KytyClose() {
@@ -181,6 +200,10 @@ void Run(const RunOptions& options) {
 	int ok = atexit(KytyClose);
 	EXIT_NOT_IMPLEMENTED(ok != 0);
 
+	// Guest threads are still running, so skip KytyClose() and only flush emergency state.
+	ok = at_quick_exit(Common::Subsystems::EmergencyShutdownActive);
+	EXIT_NOT_IMPLEMENTED(ok != 0);
+
 	Libs::LibKernel::FileSystem::Mount(options.app0_dir, "/app0");
 	Libs::LibKernel::FileSystem::Mount(options.app0_dir, "/hostapp");
 

src/generate_version.cmake

@@ -1,5 +1,6 @@
 set(KYTY_GIT_VERSION "unknown")
 set(KYTY_GIT_HASH "unknown")
+set(KYTY_GIT_REVISION "unknown")
 if(GIT_EXECUTABLE)
 	execute_process(
 		COMMAND "${GIT_EXECUTABLE}" describe --tags --always --dirty
@@ -14,16 +15,18 @@ if(GIT_EXECUTABLE)
 	endif()
 
 	execute_process(
-		COMMAND "${GIT_EXECUTABLE}" rev-parse --short=7 HEAD
+		COMMAND "${GIT_EXECUTABLE}" rev-parse HEAD
 		WORKING_DIRECTORY "${GIT_WORKING_DIRECTORY}"
-		OUTPUT_VARIABLE KYTY_GIT_HASH
+		OUTPUT_VARIABLE KYTY_GIT_REVISION
 		OUTPUT_STRIP_TRAILING_WHITESPACE
 		RESULT_VARIABLE GIT_HASH_RESULT
 		ERROR_QUIET
 	)
 	if(NOT GIT_HASH_RESULT EQUAL 0)
 		set(KYTY_GIT_HASH "unknown")
+		set(KYTY_GIT_REVISION "unknown")
 	else()
+		string(SUBSTRING "${KYTY_GIT_REVISION}" 0 7 KYTY_GIT_HASH)
 		execute_process(
 			COMMAND "${GIT_EXECUTABLE}" diff-index --quiet HEAD --
 			WORKING_DIRECTORY "${GIT_WORKING_DIRECTORY}"

src/graphics/guest_gpu/command_processor/commandProcessor.h

@@ -7,6 +7,7 @@
 #include "graphics/host_gpu/renderer/renderContext.h"
 
 #include <cstdint>
+#include <span>
 #include <vector>
 
 namespace Libs::Graphics {
@@ -30,10 +31,9 @@ private:
 	friend class CommandProcessor;
 
 	struct BufferCursor {
-		uint32_t* next_packet         = nullptr;
-		uint32_t  remaining_dw        = 0;
-		uint32_t  total_dw            = 0;
-		uint32_t  deferred_advance_dw = 0;
+		std::span<const uint32_t> commands;
+		uint32_t                  offset_dw           = 0;
+		uint32_t                  deferred_advance_dw = 0;
 	};
 
 	std::vector<BufferCursor> m_buffer_stack;
@@ -50,7 +50,8 @@ public:
 		int64_t flip_arg  = 0;
 	};
 
-	explicit CommandProcessor(RenderContext& renderer): m_renderer(renderer) {}
+	CommandProcessor(RenderContext& renderer, int interrupt_event_id)
+	    : m_renderer(renderer), m_interrupt_event_id(interrupt_event_id) {}
 	~CommandProcessor() = default;
 
 	KYTY_CLASS_NO_COPY(CommandProcessor);
@@ -131,13 +132,14 @@ public:
 	                    const volatile void* address, uint32_t count_in_dwords);
 	[[nodiscard]] bool ShouldSkipPredicatedPackets() const { return m_predicate_skip; }
 
-	Pm4ProcessResult Process(Pm4Execution& execution, uint32_t* buffer, uint32_t size_dw);
-	void             ProcessIndirectBuffer(uint32_t* buffer, uint32_t size_dw);
+	Pm4ProcessResult Process(Pm4Execution& execution, std::span<const uint32_t> commands);
+	void             ProcessIndirectBuffer(std::span<const uint32_t> commands);
 
 	void SetFlip(const FlipInfo& flip) { m_flip = flip; }
 
 	[[nodiscard]] uint64_t GetSubmitId() const { return m_submit_id; }
 	void                   SetSubmitId(uint64_t submit_id) { m_submit_id = submit_id; }
+	[[nodiscard]] bool     IsAsyncComputeQueue() const { return m_interrupt_event_id >= 0x20; }
 
 private:
 	template <typename T>
@@ -151,10 +153,10 @@ private:
 	                          uint32_t render_target_slice_offset, uint32_t first_vertex,
 	                          uint32_t first_instance);
 
-	CommandScheduler&    GetScheduler() const { return m_renderer.GetCommandScheduler(); }
-	RenderCommandBuffer& CurrentBuffer() { return GetScheduler().Current(); }
-	void                 CheckBuffer() const { GetScheduler().CheckActive(); }
-	GpuResourceManager&  GetGpuResources() const { return m_renderer.GetGpuResources(); }
+	CommandScheduler&   GetScheduler() const { return m_renderer.GetCommandScheduler(); }
+	CommandBuffer&      CurrentBuffer() { return GetScheduler().Current(); }
+	void                CheckBuffer() const { GetScheduler().CheckActive(); }
+	GpuResourceManager& GetGpuResources() const { return m_renderer.GetGpuResources(); }
 
 	RenderContext&   m_renderer;
 	HW::Context      m_ctx;
@@ -177,10 +179,11 @@ private:
 
 	uint32_t m_const_ram[0x3000] = {0};
 
-	FlipInfo m_flip;
-	uint64_t m_submit_id                   = 0;
-	uint64_t m_synthetic_occlusion_counter = 0;
-	bool     m_predicate_skip              = false;
+	FlipInfo  m_flip;
+	const int m_interrupt_event_id;
+	uint64_t  m_submit_id                   = 0;
+	uint64_t  m_synthetic_occlusion_counter = 0;
+	bool      m_predicate_skip              = false;
 };
 
 } // namespace Libs::Graphics

src/graphics/guest_gpu/command_processor/pm4Handlers.cpp

@@ -1859,11 +1859,11 @@ KYTY_CP_OP_PARSER(CpOpBranch) {
 	uint64_t reference   = buffer[5] | (static_cast<uint64_t>(buffer[6]) << 32u);
 	uint32_t mode        = buffer[0] & 0x3u;
 	uint32_t function    = (buffer[0] >> 8u) & 0x7u;
-	auto*    then_buffer = reinterpret_cast<uint32_t*>((buffer[7] & 0xfffffffcu) |
-	                                                   (static_cast<uint64_t>(buffer[8]) << 32u));
+	auto*    then_buffer = reinterpret_cast<const uint32_t*>(
+	    (buffer[7] & 0xfffffffcu) | (static_cast<uint64_t>(buffer[8]) << 32u));
 	uint32_t then_num_dw = buffer[9] & 0xfffffu;
-	auto*    else_buffer = reinterpret_cast<uint32_t*>((buffer[10] & 0xfffffffcu) |
-	                                                   (static_cast<uint64_t>(buffer[11]) << 32u));
+	auto*    else_buffer = reinterpret_cast<const uint32_t*>(
+	    (buffer[10] & 0xfffffffcu) | (static_cast<uint64_t>(buffer[11]) << 32u));
 	uint32_t else_num_dw = buffer[12] & 0xfffffu;
 
 	EXIT_NOT_IMPLEMENTED(compare_addr == nullptr);
@@ -1877,10 +1877,10 @@ KYTY_CP_OP_PARSER(CpOpBranch) {
 	     reinterpret_cast<uint64_t>(else_buffer), else_num_dw);
 
 	if (take_then) {
-		cp.ProcessIndirectBuffer(then_buffer, then_num_dw);
+		cp.ProcessIndirectBuffer({then_buffer, then_num_dw});
 	} else if (mode == 2 && else_num_dw != 0) {
 		EXIT_NOT_IMPLEMENTED(else_buffer == nullptr);
-		cp.ProcessIndirectBuffer(else_buffer, else_num_dw);
+		cp.ProcessIndirectBuffer({else_buffer, else_num_dw});
 	}
 
 	return payload_dw;
@@ -2321,7 +2321,7 @@ KYTY_CP_OP_PARSER(CpOpIndirectBuffer) {
 	}
 
 	auto* indirect_buffer =
-	    reinterpret_cast<uint32_t*>(buffer[0] | (static_cast<uint64_t>(buffer[1]) << 32u));
+	    reinterpret_cast<const uint32_t*>(buffer[0] | (static_cast<uint64_t>(buffer[1]) << 32u));
 	uint32_t                     indirect_num_dw = control & 0xfffffu;
 	static std::atomic<uint32_t> indirect_log_count {0};
 	if (indirect_log_count.fetch_add(1) < 128) {
@@ -2341,7 +2341,7 @@ KYTY_CP_OP_PARSER(CpOpIndirectBuffer) {
 
 	GraphicsDbgDumpDcb("ci", indirect_num_dw, indirect_buffer);
 
-	cp.ProcessIndirectBuffer(indirect_buffer, indirect_num_dw);
+	cp.ProcessIndirectBuffer({indirect_buffer, indirect_num_dw});
 
 	return 3;
 }

src/graphics/guest_gpu/graphicsRun.cpp

@@ -66,17 +66,10 @@ static bool GraphicsRunDebugDumpEnabled() {
 	       Config::GetPrintfDirection() != Config::OutputDirection::Silent;
 }
 
-GuestGpu::OwnedCmdBuffer::OwnedCmdBuffer(const uint32_t* data, uint32_t count) {
-	EXIT_IF(data == nullptr && count != 0);
-	if (count != 0) {
-		m_words.assign(data, data + count);
-	}
-}
-
 GuestGpu::GuestGpu(RenderContext& renderer): m_renderer(renderer) {
 	EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
 	GraphicsInitJmpTables();
-	m_gfx_cp = std::make_unique<CommandProcessor>(renderer);
+	m_gfx_cp = std::make_unique<CommandProcessor>(renderer, 0);
 	m_thread = std::jthread(ThreadRun, this);
 }
 
@@ -148,39 +141,33 @@ void GuestGpu::SendCommandSync(Common::UniqueFunction<void>&& command) {
 	done.acquire();
 }
 
-void GuestGpu::Submit(uint32_t* cmd_draw_buffer, uint32_t num_draw_dw, uint32_t* cmd_const_buffer,
-                      uint32_t num_const_dw, bool trigger_agc_interrupt_on_done) {
-	if (num_draw_dw == 0) {
+void GuestGpu::Submit(std::span<const uint32_t> draw_commands,
+                      std::span<const uint32_t> constant_commands) {
+	if (draw_commands.empty()) {
 		return;
 	}
-	EXIT_IF(cmd_draw_buffer == nullptr);
 	GpuMutexLock lock(m_submission_mutex);
 	Submission   submission;
-	submission.type                          = SubmissionType::Graphics;
-	submission.queue_id                      = 0;
-	submission.commands                      = OwnedCmdBuffer(cmd_draw_buffer, num_draw_dw);
-	submission.constant_commands             = OwnedCmdBuffer(cmd_const_buffer, num_const_dw);
-	submission.trigger_agc_interrupt_on_done = trigger_agc_interrupt_on_done;
-	submission.reset_processor               = m_graphics_done;
-	m_graphics_done                          = false;
+	submission.type              = SubmissionType::Graphics;
+	submission.queue_id          = 0;
+	submission.commands          = draw_commands;
+	submission.constant_commands = constant_commands;
+	submission.reset_processor   = m_graphics_done;
+	m_graphics_done              = false;
 	Enqueue(std::move(submission));
 }
 
-void GuestGpu::SubmitCompute(uint32_t queue, uint32_t* cmd_buffer, uint32_t num_dw,
-                             bool trigger_agc_interrupt_on_done) {
-	EXIT_IF(cmd_buffer == nullptr || num_dw == 0);
+void GuestGpu::SubmitCompute(uint32_t queue, std::span<const uint32_t> commands) {
+	EXIT_IF(commands.empty());
 	GpuMutexLock lock(m_submission_mutex);
 
-	constexpr uint32_t compute_queue_base = 0x20u;
-	EXIT_NOT_IMPLEMENTED(queue < compute_queue_base ||
-	                     queue >= compute_queue_base + ComputeQueueCount);
+	EXIT_NOT_IMPLEMENTED(queue < ComputeQueueBase || queue >= ComputeQueueBase + ComputeQueueCount);
 
-	const auto compute_queue = queue - compute_queue_base;
+	const auto compute_queue = queue - ComputeQueueBase;
 	Submission submission;
-	submission.type                          = SubmissionType::Compute;
-	submission.queue_id                      = 1 + compute_queue;
-	submission.commands                      = OwnedCmdBuffer(cmd_buffer, num_dw);
-	submission.trigger_agc_interrupt_on_done = trigger_agc_interrupt_on_done;
+	submission.type     = SubmissionType::Compute;
+	submission.queue_id = 1 + compute_queue;
+	submission.commands = commands;
 	Enqueue(std::move(submission));
 }
 
@@ -215,7 +202,7 @@ CommandProcessor& GuestGpu::GetProcessor(uint32_t queue_id) {
 	}
 	auto& processor = m_compute_cp[queue_id - 1];
 	if (processor == nullptr) {
-		processor = std::make_unique<CommandProcessor>(m_renderer);
+		processor = std::make_unique<CommandProcessor>(m_renderer, ComputeQueueBase + queue_id - 1);
 	}
 	return *processor;
 }
@@ -268,8 +255,7 @@ void CommandProcessor::BufferFlush() {
 }
 
 void CommandProcessor::BufferFlushAndWait() {
-	auto& submitted = GetScheduler().FlushAndGetSubmitted();
-	submitted.WaitForFenceOnly();
+	GetScheduler().FlushAndWait();
 }
 
 void CommandProcessor::BufferWait() {
@@ -583,21 +569,20 @@ bool GuestGpu::Process(Submission& submission) {
 	switch (submission.type) {
 		case SubmissionType::Graphics: {
 			bool progressed = false;
-			submission.constant_complete |= submission.constant_commands.Empty();
+			submission.constant_complete |= submission.constant_commands.empty();
 			for (;;) {
 				bool round_progress = false;
 				if (!submission.constant_complete) {
 					submission.constant_complete =
-					    cp.Process(
-					        submission.constant_execution, submission.constant_commands.Data(),
-					        submission.constant_commands.Size()) == Pm4ProcessResult::Complete;
+					    cp.Process(submission.constant_execution, submission.constant_commands) ==
+					    Pm4ProcessResult::Complete;
 					round_progress |= submission.constant_execution.MadeProgress();
 				}
 				cp.SetCeComplete(submission.constant_complete);
 				if (!submission.command_complete) {
 					submission.command_complete =
-					    cp.Process(submission.command_execution, submission.commands.Data(),
-					               submission.commands.Size()) == Pm4ProcessResult::Complete;
+					    cp.Process(submission.command_execution, submission.commands) ==
+					    Pm4ProcessResult::Complete;
 					round_progress |= submission.command_execution.MadeProgress();
 				}
 				progressed |= round_progress;
@@ -614,14 +599,11 @@ bool GuestGpu::Process(Submission& submission) {
 			} else if (complete) {
 				m_renderer.GetGpuResources().RunGarbageCollector();
 			}
-			if (complete && submission.trigger_agc_interrupt_on_done) {
-				m_renderer.TriggerEopEvent(0);
-			}
 			break;
 		}
 		case SubmissionType::Compute: {
-			const auto      num_dw                  = submission.commands.Size();
-			auto*           buffer                  = submission.commands.Data();
+			const auto      num_dw = static_cast<uint32_t>(submission.commands.size());
+			const auto*     buffer = submission.commands.data();
 			static uint32_t compute_batch_log_count = 0;
 			if (first_slice && num_dw <= 128 && compute_batch_log_count++ < 32) {
 				LOGF("compute direct batch: data=0x%016" PRIx64 ", num_dw=%" PRIu32 "\n",
@@ -633,7 +615,7 @@ bool GuestGpu::Process(Submission& submission) {
 			if (first_slice) {
 				GraphicsDbgDumpDcb("cc", num_dw, buffer);
 			}
-			complete = cp.Process(submission.command_execution, buffer, num_dw) ==
+			complete = cp.Process(submission.command_execution, submission.commands) ==
 			           Pm4ProcessResult::Complete;
 			if (submission.command_execution.MadeProgress()) {
 				if (complete) {
@@ -643,9 +625,6 @@ bool GuestGpu::Process(Submission& submission) {
 			} else if (complete) {
 				m_renderer.GetGpuResources().RunGarbageCollector();
 			}
-			if (complete && submission.trigger_agc_interrupt_on_done) {
-				Sync::TriggerAgcUserInterrupt();
-			}
 			break;
 		}
 		case SubmissionType::FlipPreparation:
@@ -657,12 +636,13 @@ bool GuestGpu::Process(Submission& submission) {
 	return complete;
 }
 
-Pm4ProcessResult CommandProcessor::Process(Pm4Execution& execution, uint32_t* buffer,
-                                           uint32_t size_dw) {
+Pm4ProcessResult CommandProcessor::Process(Pm4Execution&             execution,
+                                           std::span<const uint32_t> commands) {
 	KYTY_PROFILER_BLOCK("CommandProcessor::Process");
 	EXIT_IF(g_current_execution != nullptr);
-	if (execution.m_buffer_stack.empty() && size_dw != 0) {
-		execution.m_buffer_stack.push_back({buffer, size_dw, size_dw, 0});
+	EXIT_IF(commands.size() > UINT32_MAX);
+	if (execution.m_buffer_stack.empty() && !commands.empty()) {
+		execution.m_buffer_stack.push_back({commands});
 	}
 	execution.m_suspended     = false;
 	execution.m_made_progress = false;
@@ -687,14 +667,14 @@ Pm4ProcessResult CommandProcessor::Process(Pm4Execution& execution, uint32_t* bu
 	                                        : Pm4ProcessResult::Blocked;
 }
 
-void CommandProcessor::ProcessIndirectBuffer(uint32_t* buffer, uint32_t size_dw) {
+void CommandProcessor::ProcessIndirectBuffer(std::span<const uint32_t> commands) {
 	EXIT_IF(g_current_execution == nullptr);
-	if (size_dw == 0) {
+	if (commands.empty()) {
 		return;
 	}
 	auto&      execution  = *g_current_execution;
 	const auto stop_depth = execution.m_buffer_stack.size();
-	execution.m_buffer_stack.push_back({buffer, size_dw, size_dw, 0});
+	execution.m_buffer_stack.push_back({commands});
 	ProcessPm4(execution, stop_depth);
 }
 
@@ -710,29 +690,28 @@ void CommandProcessor::ProcessPm4(Pm4Execution& execution, size_t stop_depth) {
 		}
 		const auto buffer_index = execution.m_buffer_stack.size() - 1;
 		auto&      cursor       = execution.m_buffer_stack[buffer_index];
+		EXIT_IF(cursor.offset_dw > cursor.commands.size());
 		if (cursor.deferred_advance_dw != 0) {
-			EXIT_IF(cursor.deferred_advance_dw > cursor.remaining_dw);
-			cursor.next_packet += cursor.deferred_advance_dw;
-			cursor.remaining_dw -= cursor.deferred_advance_dw;
+			EXIT_IF(cursor.deferred_advance_dw > cursor.commands.size() - cursor.offset_dw);
+			cursor.offset_dw += cursor.deferred_advance_dw;
 			cursor.deferred_advance_dw = 0;
 			execution.m_made_progress  = true;
 			continue;
 		}
-		if (cursor.remaining_dw == 0) {
+		if (cursor.offset_dw == cursor.commands.size()) {
 			execution.m_buffer_stack.pop_back();
 			continue;
 		}
 
-		auto* const packet        = cursor.next_packet;
-		const auto  remaining_dw  = cursor.remaining_dw;
-		const auto  total_dw      = cursor.total_dw;
-		const auto  packet_header = packet[0];
-		const auto  opcode        = (packet_header >> 8u) & 0xffu;
+		const auto* const packet        = cursor.commands.data() + cursor.offset_dw;
+		const auto        total_dw      = static_cast<uint32_t>(cursor.commands.size());
+		const auto        remaining_dw  = total_dw - cursor.offset_dw;
+		const auto        packet_header = packet[0];
+		const auto        opcode        = (packet_header >> 8u) & 0xffu;
 		EXIT_NOT_IMPLEMENTED(remaining_dw > total_dw);
 
 		if (packet_header == 0x80000000u) {
-			cursor.next_packet++;
-			cursor.remaining_dw--;
+			cursor.offset_dw++;
 			execution.m_made_progress = true;
 			continue;
 		}
@@ -767,8 +746,7 @@ void CommandProcessor::ProcessPm4(Pm4Execution& execution, size_t stop_depth) {
 					     dst, val, packet[1], packet[2]);
 				}
 			}
-			cursor.next_packet += packet_dw;
-			cursor.remaining_dw -= packet_dw;
+			cursor.offset_dw += packet_dw;
 			execution.m_made_progress = true;
 			continue;
 		}
@@ -801,8 +779,7 @@ void CommandProcessor::ProcessPm4(Pm4Execution& execution, size_t stop_depth) {
 			return;
 		}
 		EXIT_IF(execution.m_buffer_stack.size() != buffer_index + 1);
-		execution.m_buffer_stack[buffer_index].next_packet += packet_dw;
-		execution.m_buffer_stack[buffer_index].remaining_dw -= packet_dw;
+		execution.m_buffer_stack[buffer_index].offset_dw += packet_dw;
 		execution.m_made_progress = true;
 	}
 }
@@ -1245,7 +1222,14 @@ void CommandProcessor::WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_wr
 	switch (interrupt_selector) {
 		case 0x00:
 		case 0x03: with_interrupt = false; break;
-		case 0x01: Sync::TriggerEopEventAtEndOfPipe(CurrentBuffer(), interrupt_context_id); return;
+		case 0x01:
+			if (!IsAsyncComputeQueue()) {
+				Sync::TriggerEopEventAtEndOfPipe(CurrentBuffer(), m_interrupt_event_id,
+				                                 interrupt_context_id);
+				return;
+			}
+			with_interrupt = true;
+			break;
 		case 0x02: with_interrupt = true; break;
 		default: EXIT("unknown interrupt_selector\n");
 	}
@@ -1258,10 +1242,11 @@ void CommandProcessor::WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_wr
 		if (with_interrupt) {
 			if (with_writeback) {
 				Sync::WriteAtEndOfPipeWithInterruptWriteBack32(m_submit_id, CurrentBuffer(), dst,
-				                                               data, interrupt_context_id);
+				                                               data, m_interrupt_event_id,
+				                                               interrupt_context_id);
 			} else {
 				Sync::WriteAtEndOfPipeWithInterrupt32(m_submit_id, CurrentBuffer(), dst, data,
-				                                      interrupt_context_id);
+				                                      m_interrupt_event_id, interrupt_context_id);
 			}
 		} else if (with_writeback) {
 			Sync::WriteAtEndOfPipeWithWriteBack32(m_submit_id, CurrentBuffer(), dst, data);
@@ -1281,7 +1266,7 @@ void CommandProcessor::WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_wr
 					Sync::WriteAtEndOfPipeGds32(m_submit_id, CurrentBuffer(), dst, value & 0xffffu,
 					                            value >> 16u);
 					if (with_interrupt) {
-						m_renderer.TriggerEopEvent(interrupt_context_id);
+						m_renderer.TriggerInterrupt(m_interrupt_event_id, interrupt_context_id);
 					}
 					return;
 				}
@@ -1307,10 +1292,12 @@ void CommandProcessor::WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_wr
 					if (with_interrupt) {
 						if (with_writeback) {
 							Sync::WriteAtEndOfPipeWithInterruptWriteBack64(
-							    m_submit_id, CurrentBuffer(), dst, value, interrupt_context_id);
+							    m_submit_id, CurrentBuffer(), dst, value, m_interrupt_event_id,
+							    interrupt_context_id);
 						} else {
 							Sync::WriteAtEndOfPipeWithInterrupt64(m_submit_id, CurrentBuffer(), dst,
-							                                      value, interrupt_context_id);
+							                                      value, m_interrupt_event_id,
+							                                      interrupt_context_id);
 						}
 					} else if (with_writeback) {
 						Sync::WriteAtEndOfPipeWithWriteBack64(m_submit_id, CurrentBuffer(), dst,
@@ -1389,26 +1376,35 @@ void CommandProcessor::WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_wr
 		case 0x04:
 			if constexpr (sizeof(T) == sizeof(uint64_t)) {
 				const auto clock = Sync::ReadReferenceClock();
-				std::memcpy(dst_gpu_addr, &clock, sizeof(clock));
+				auto*      dst   = static_cast<uint64_t*>(dst_gpu_addr);
+				std::memcpy(dst, &clock, sizeof(clock));
 				switch (cache_action) {
 					case 0x00:
-						if (((eop_event_type == 0x04 && event_index == 0x05) ||
-						     (eop_event_type == 0x28 && event_index == 0x00)) &&
-						    !with_interrupt) {
-							Sync::WriteAtEndOfPipeClockCounter(m_submit_id, CurrentBuffer(),
-							                                   static_cast<uint64_t*>(dst_gpu_addr),
-							                                   clock);
+						if ((eop_event_type == 0x04 && event_index == 0x05) ||
+						    (eop_event_type == 0x28 && event_index == 0x00)) {
+							if (with_interrupt) {
+								Sync::WriteAtEndOfPipeWithInterrupt64(
+								    m_submit_id, CurrentBuffer(), dst, clock, m_interrupt_event_id,
+								    interrupt_context_id);
+							} else {
+								Sync::WriteAtEndOfPipeClockCounter(m_submit_id, CurrentBuffer(),
+								                                   dst, clock);
+							}
 							return;
 						}
 						break;
 					case 0x38:
-						if (((eop_event_type == 0x04 &&
-						      (event_index == 0x00 || event_index == 0x05)) ||
-						     (eop_event_type == 0x28 && event_index == 0x00)) &&
-						    !with_interrupt) {
-							Sync::WriteAtEndOfPipeClockCounterWithWriteBack(
-							    m_submit_id, CurrentBuffer(), static_cast<uint64_t*>(dst_gpu_addr),
-							    clock);
+						if ((eop_event_type == 0x04 &&
+						     (event_index == 0x00 || event_index == 0x05)) ||
+						    (eop_event_type == 0x28 && event_index == 0x00)) {
+							if (with_interrupt) {
+								Sync::WriteAtEndOfPipeWithInterruptWriteBack64(
+								    m_submit_id, CurrentBuffer(), dst, clock, m_interrupt_event_id,
+								    interrupt_context_id);
+							} else {
+								Sync::WriteAtEndOfPipeClockCounterWithWriteBack(
+								    m_submit_id, CurrentBuffer(), dst, clock);
+							}
 							return;
 						}
 						break;
@@ -1465,7 +1461,7 @@ void CommandProcessor::EmitGlobalBarrier() {
 void CommandProcessor::TriggerEopEventAtEndOfPipe(uint32_t interrupt_context_id) {
 	CheckBuffer();
 
-	Sync::TriggerEopEventAtEndOfPipe(CurrentBuffer(), interrupt_context_id);
+	Sync::TriggerEopEventAtEndOfPipe(CurrentBuffer(), m_interrupt_event_id, interrupt_context_id);
 }
 
 void CommandProcessor::TriggerEvent(uint32_t event_type, uint32_t event_index,
@@ -1604,7 +1600,7 @@ void CommandProcessor::FlipWithInterrupt(uint32_t eop_event_type, uint32_t cache
 	                                         m_flip.flip_arg);
 	Sync::WriteAtEndOfPipeWithInterruptWriteBackFlip32(
 	    m_submit_id, command, static_cast<uint32_t*>(dst_gpu_addr), value, m_flip.handle,
-	    m_flip.index, m_flip.flip_mode, m_flip.flip_arg, request);
+	    m_flip.index, m_flip.flip_mode, m_flip.flip_arg, request, m_interrupt_event_id);
 	GetScheduler().Flush();
 }
 
@@ -1625,7 +1621,7 @@ void CommandProcessor::PrepareCpuFlip(uint64_t request_id) {
 }
 
 void CommandProcessor::SynchronizeGpu() {
-	GetScheduler().FinishCurrent();
+	GetScheduler().Finish();
 }
 
 bool GuestGpu::IsGpuThread() noexcept {

src/graphics/guest_gpu/graphicsRun.h

@@ -13,8 +13,8 @@
 #include <deque>
 #include <memory>
 #include <mutex>
+#include <span>
 #include <thread>
-#include <vector>
 
 namespace Libs::Graphics {
 
@@ -31,12 +31,12 @@ public:
 	void               SendCommand(Common::UniqueFunction<void>&& command);
 	void               SendCommandSync(Common::UniqueFunction<void>&& command);
 
-	void Submit(uint32_t* draw_commands, uint32_t draw_size_dw, uint32_t* constant_commands,
-	            uint32_t constant_size_dw, bool trigger_agc_interrupt_on_done = false);
-	void SubmitCompute(uint32_t queue, uint32_t* commands, uint32_t size_dw,
-	                   bool trigger_agc_interrupt_on_done = false);
-	void SubmitFlipPreparation(uint64_t request_id);
-	void Done();
+	// Submitted command memory is borrowed and must remain valid until GPU execution completes.
+	void              Submit(std::span<const uint32_t> draw_commands,
+	                         std::span<const uint32_t> constant_commands);
+	void              SubmitCompute(uint32_t queue, std::span<const uint32_t> commands);
+	void              SubmitFlipPreparation(uint64_t request_id);
+	void              Done();
 	[[nodiscard]] int GetFrameNum() const;
 
 	[[nodiscard]] static bool IsGpuThread() noexcept;
@@ -45,39 +45,24 @@ private:
 	static constexpr uint32_t ComputePipeCount     = 7;
 	static constexpr uint32_t QueuesPerComputePipe = 8;
 	static constexpr uint32_t ComputeQueueCount    = ComputePipeCount * QueuesPerComputePipe;
+	static constexpr uint32_t ComputeQueueBase     = 0x20;
 	static constexpr uint32_t QueueCount           = 1 + ComputeQueueCount;
 
-	class OwnedCmdBuffer {
-	public:
-		OwnedCmdBuffer() = default;
-		explicit OwnedCmdBuffer(const uint32_t* data, uint32_t count);
-
-		[[nodiscard]] bool     Empty() const noexcept { return m_words.empty(); }
-		[[nodiscard]] uint32_t Size() const noexcept {
-			return static_cast<uint32_t>(m_words.size());
-		}
-		[[nodiscard]] uint32_t* Data() noexcept { return m_words.data(); }
-
-	private:
-		std::vector<uint32_t> m_words;
-	};
-
 	enum class SubmissionType { Graphics, Compute, FlipPreparation };
 
 	struct Submission {
-		SubmissionType type     = SubmissionType::Graphics;
-		uint32_t       queue_id = 0;
-		OwnedCmdBuffer commands;
-		OwnedCmdBuffer constant_commands;
-		Pm4Execution   command_execution;
-		Pm4Execution   constant_execution;
-		bool           trigger_agc_interrupt_on_done = false;
-		bool           reset_processor               = false;
-		bool           started                       = false;
-		bool           command_complete              = false;
-		bool           constant_complete             = false;
-		bool           blocked                       = false;
-		uint64_t       flip_request_id               = 0;
+		SubmissionType            type     = SubmissionType::Graphics;
+		uint32_t                  queue_id = 0;
+		std::span<const uint32_t> commands;
+		std::span<const uint32_t> constant_commands;
+		Pm4Execution              command_execution;
+		Pm4Execution              constant_execution;
+		bool                      reset_processor   = false;
+		bool                      started           = false;
+		bool                      command_complete  = false;
+		bool                      constant_complete = false;
+		bool                      blocked           = false;
+		uint64_t                  flip_request_id   = 0;
 	};
 
 	void              Enqueue(Submission submission);

src/graphics/guest_gpu/pm4.cpp

@@ -100,7 +100,7 @@ constexpr auto g_opcode_names   = MakeOpcodeNames();
 
 } // namespace
 
-void DumpPm4PacketStream(Common::File* file, uint32_t* cmd_buffer, uint32_t start_dw,
+void DumpPm4PacketStream(Common::File* file, const uint32_t* cmd_buffer, uint32_t start_dw,
                          uint32_t num_dw) {
 	// db_dump();
 
@@ -143,9 +143,9 @@ void DumpPm4PacketStream(Common::File* file, uint32_t* cmd_buffer, uint32_t star
 				if ((op == IT_SET_CONTEXT_REG_INDIRECT || op == IT_SET_SH_REG_INDIRECT ||
 				     op == IT_SET_UCONFIG_REG_INDIRECT) &&
 				    len == 4) {
-					auto* indirect_buffer =
-					    reinterpret_cast<uint32_t*>((static_cast<uint64_t>(cmd[0]) & 0xfffffffcu) |
-					                                (static_cast<uint64_t>(cmd[1]) << 32u));
+					auto* indirect_buffer = reinterpret_cast<const uint32_t*>(
+					    (static_cast<uint64_t>(cmd[0]) & 0xfffffffcu) |
+					    (static_cast<uint64_t>(cmd[1]) << 32u));
 					const uint32_t indirect_num_regs = cmd[3] & 0x3fffu;
 
 					for (uint32_t i = 0; indirect_buffer != nullptr && i < indirect_num_regs;

src/graphics/guest_gpu/pm4.h

@@ -1088,7 +1088,7 @@ constexpr uint32_t UC_NOP                       = 0x80003FFF;
 
 constexpr uint32_t UC_NUM = 0x3FFF + 1;
 
-void DumpPm4PacketStream(Common::File* file, uint32_t* cmd_buffer, uint32_t start_dw,
+void DumpPm4PacketStream(Common::File* file, const uint32_t* cmd_buffer, uint32_t start_dw,
                          uint32_t num_dw);
 
 } // namespace Libs::Graphics::Pm4

src/graphics/host_gpu/graphicContext.h

@@ -3,20 +3,88 @@
 
 #include "common/abi.h"
 #include "common/common.h"
+#include "common/threads.h"
 #include "graphics/host_gpu/vulkanCommon.h" // IWYU pragma: export
-#include "graphics/host_gpu/vulkanInstance.h"
 
+#include <map>
+#include <mutex>
+#include <tuple>
 #include <vector>
 #include <vk_mem_alloc.h>
 
 namespace Libs::Graphics {
 
-class CommandBuffer;
 struct VulkanBuffer;
 struct VulkanImage;
 struct VulkanMemory;
 
-struct GraphicContext: public VulkanInstance {
+inline constexpr uint32_t VULKAN_TARGET_API_VERSION = VK_API_VERSION_1_3;
+
+struct GraphicContext {
+	vk::Instance                       instance                              = nullptr;
+	vk::DebugUtilsMessengerEXT         debug_messenger                       = nullptr;
+	vk::PhysicalDevice                 physical_device                       = nullptr;
+	vk::PhysicalDeviceProperties       physical_device_properties            = {};
+	vk::PhysicalDeviceMemoryProperties physical_device_memory_properties     = {};
+	vk::Device                         device                                = nullptr;
+	VmaAllocator                       allocator                             = nullptr;
+	bool                               memory_budget_ext_enabled             = false;
+	bool                               rt_extensions_enabled                 = false;
+	bool                               compute_subgroup_size_control_enabled = false;
+	bool                               compute_wave64_supported              = false;
+	bool                               sample_rate_shading_enabled           = false;
+	uint32_t                           subgroup_size                         = 0;
+	uint32_t                           min_subgroup_size                     = 0;
+	uint32_t                           max_subgroup_size                     = 0;
+	uint32_t                           max_push_descriptors                  = 0;
+	vk::ShaderStageFlags               required_subgroup_size_stages         = {};
+	Common::Mutex                      queue_mutex;
+	uint32_t                           queue_family = static_cast<uint32_t>(-1);
+	vk::Queue                          queue        = nullptr;
+
+	[[nodiscard]] const vk::PhysicalDeviceProperties& GetPhysicalDeviceProperties() const {
+		return physical_device_properties;
+	}
+
+	[[nodiscard]] const vk::PhysicalDeviceMemoryProperties&
+	GetPhysicalDeviceMemoryProperties() const {
+		return physical_device_memory_properties;
+	}
+
+	[[nodiscard]] vk::FormatProperties GetFormatProperties(vk::Format format) const {
+		std::scoped_lock lock(m_format_properties_mutex);
+		auto [it, inserted] = m_format_properties.try_emplace(format);
+		if (inserted) {
+			physical_device.getFormatProperties(format, &it->second);
+		}
+		return it->second;
+	}
+
+	[[nodiscard]] vk::Result GetImageFormatProperties(vk::Format format, vk::ImageType type,
+	                                                  vk::ImageTiling            tiling,
+	                                                  vk::ImageUsageFlags        usage,
+	                                                  vk::ImageCreateFlags       flags,
+	                                                  vk::ImageFormatProperties* properties) const {
+		using Key = std::tuple<vk::Format, vk::ImageType, vk::ImageTiling, vk::ImageUsageFlags,
+		                       vk::ImageCreateFlags>;
+		std::scoped_lock lock(m_image_format_properties_mutex);
+		auto [it, inserted] =
+		    m_image_format_properties.try_emplace(Key {format, type, tiling, usage, flags});
+		if (inserted) {
+			it->second.first = physical_device.getImageFormatProperties(format, type, tiling, usage,
+			                                                            flags, &it->second.second);
+		}
+		if (properties != nullptr) {
+			*properties = it->second.second;
+		}
+		return it->second.first;
+	}
+
+	[[nodiscard]] vk::DeviceSize StorageMinAlignment() const {
+		const auto alignment = physical_device_properties.limits.minStorageBufferOffsetAlignment;
+		return alignment != 0 ? alignment : 1;
+	}
+
 	[[nodiscard]] bool CreateAllocator();
 	void               DestroyAllocator();
 	void               LogMemoryBudget() const;
@@ -35,6 +103,15 @@ struct GraphicContext: public VulkanInstance {
 
 	uint32_t screen_width  = 0;
 	uint32_t screen_height = 0;
+
+private:
+	mutable std::mutex                                 m_format_properties_mutex;
+	mutable std::map<vk::Format, vk::FormatProperties> m_format_properties;
+	mutable std::mutex                                 m_image_format_properties_mutex;
+	mutable std::map<std::tuple<vk::Format, vk::ImageType, vk::ImageTiling, vk::ImageUsageFlags,
+	                            vk::ImageCreateFlags>,
+	                 std::pair<vk::Result, vk::ImageFormatProperties>>
+	    m_image_format_properties;
 };
 
 struct VulkanMemory {

src/graphics/host_gpu/rangeSet.h

@@ -53,12 +53,30 @@ public:
 		}
 	}
 
+	void Clear() { m_ranges.clear(); }
+
+	template <typename Func>
+	void ForEach(Func&& func) const {
+		for (const auto& [begin, end]: m_ranges) {
+			func(begin, end);
+		}
+	}
+
 	[[nodiscard]] std::vector<Range> Intersections(uint64_t address, uint64_t size) const {
 		std::vector<Range> result;
 		ForEachIntersection(address, size, [&result](Range range) { result.push_back(range); });
 		return result;
 	}
 
+	[[nodiscard]] bool Intersects(uint64_t address, uint64_t size) const {
+		const auto end = End(address, size);
+		auto       it  = m_ranges.lower_bound(address);
+		if (it != m_ranges.begin() && std::prev(it)->second > address) {
+			return true;
+		}
+		return it != m_ranges.end() && it->first < end;
+	}
+
 	[[nodiscard]] bool Contains(uint64_t address, uint64_t size) const {
 		const auto end = End(address, size);
 		auto       it  = m_ranges.upper_bound(address);
@@ -89,7 +107,7 @@ public:
 
 private:
 	static uint64_t End(uint64_t address, uint64_t size) {
-		if (address == 0 || size == 0 || size > UINT64_MAX - address) {
+		if (size == 0 || size > UINT64_MAX - address) {
 			EXIT("invalid range-set address or size\n");
 		}
 		return address + size;

src/graphics/host_gpu/renderer/cache/bufferCache.cpp

@@ -16,6 +16,7 @@
 #include <array>
 #include <cinttypes>
 #include <cstring>
+#include <memory>
 #include <utility>
 #include <vector>
 
@@ -49,25 +50,57 @@ struct BufferCache::DownloadCopy {
 	uint64_t size          = 0;
 };
 
-struct BufferCache::DownloadRange {
-	uint64_t address = 0;
-	uint64_t size    = 0;
-	uint64_t offset  = 0;
-};
+void BufferCache::Register(BufferId id) {
+	ChangeRegister<true>(id);
+}
 
 void BufferCache::Unregister(BufferId id) {
+	ChangeRegister<false>(id);
+}
+
+template <bool insert>
+void BufferCache::ChangeRegister(BufferId id) {
 	auto& buffer = m_slot_buffers[id];
-	if (buffer.is_deleted) {
-		return;
+	PageTable::PageRange pages {};
+	EXIT_IF(!PageTable::TryGetPageRange(buffer.CpuAddress(), buffer.Size(), pages));
+	for (size_t page = pages.first; page < pages.last_exclusive; ++page) {
+		if constexpr (insert) {
+			m_page_table[page] = id;
+		} else {
+			m_page_table[page] = {};
+		}
 	}
-	const auto found = m_buffers.find(buffer.CpuAddress());
-	EXIT_IF(found == m_buffers.end() || found->second != id);
-	m_buffers.erase(found);
-	if (buffer.Size() > m_total_used_memory) {
-		EXIT("BufferCache: allocation accounting underflow\n");
+	const auto size_pages = pages.last_exclusive - pages.first;
+	if constexpr (insert) {
+		const auto [it, inserted] = m_buffers.emplace(buffer.CpuAddress(), id);
+		(void)it;
+		EXIT_IF(!inserted);
+		m_total_used_memory += buffer.Size();
+		buffer.lru_id = m_lru_cache.Insert(id, m_gc_tick);
+		std::vector<vk::DeviceAddress> addresses;
+		addresses.reserve(size_pages);
+		for (uint64_t i = 0; i < size_pages; ++i) {
+			addresses.push_back(buffer.BufferDeviceAddress() + (i << CACHING_PAGEBITS));
+		}
+		WriteDataBuffer(m_bda_pagetable_buffer, pages.first * sizeof(vk::DeviceAddress),
+		                addresses.data(), addresses.size() * sizeof(vk::DeviceAddress));
+	} else {
+		const auto found = m_buffers.find(buffer.CpuAddress());
+		EXIT_IF(found == m_buffers.end() || found->second != id);
+		m_buffers.erase(found);
+		EXIT_IF(buffer.Size() > m_total_used_memory);
+		m_total_used_memory -= buffer.Size();
+		m_lru_cache.Free(buffer.lru_id);
+		m_bda_pagetable_buffer.Fill(pages.first * sizeof(vk::DeviceAddress),
+		                            size_pages * sizeof(vk::DeviceAddress), 0);
+		buffer.is_deleted = true;
+	}
+}
+
+void BufferCache::TouchBuffer(const Buffer& buffer) {
+	if (!buffer.is_deleted) {
+		m_lru_cache.Touch(buffer.lru_id, m_gc_tick);
 	}
-	m_total_used_memory -= buffer.Size();
-	buffer.is_deleted = true;
 }
 
 void BufferCache::DeleteBuffer(BufferId id) {
@@ -100,93 +133,81 @@ std::pair<uint64_t, uint64_t> BufferCache::DownloadEnvelope(const DownloadCopy&
 	return {begin, end - begin};
 }
 
-std::vector<BufferCache::DownloadRange>
-BufferCache::RecordDownloads(std::span<const DownloadCopy> copies) {
-	uint64_t reservation_size = 0;
+void BufferCache::DownloadBufferMemory(std::span<const DownloadCopy> copies) {
+	std::vector<DownloadCopy> batch;
+	batch.reserve(copies.size());
+	uint64_t                  packed_size = 0;
+	auto&                     download    = m_download_buffer;
+	const auto flush = [&] {
+		const auto [mapped, base_offset] = download.Map(packed_size, DOWNLOAD_ALIGNMENT);
+		EXIT_IF(mapped == nullptr);
+		uint64_t cursor = 0;
+		for (const auto& copy: batch) {
+			const auto [source_begin, envelope_size] = DownloadEnvelope(copy);
+			download.CopyFrom(m_scheduler.Current(), *copy.buffer, source_begin, base_offset + cursor,
+			                  envelope_size, vk::AccessFlagBits::eMemoryWrite, vk::AccessFlags {},
+			                  vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite,
+			                  vk::AccessFlagBits::eHostRead);
+			cursor += AlignDownload(envelope_size);
+		}
+		download.Commit();
+		const auto completion_tick = m_scheduler.CurrentTick();
+		m_scheduler.Finish();
+		m_scheduler.WaitPriorityOperations(completion_tick);
+		cursor = 0;
+		for (const auto& copy: batch) {
+			const auto [source_begin, envelope_size] = DownloadEnvelope(copy);
+			const auto offset = cursor + copy.source_offset - source_begin;
+			download.Invalidate(base_offset + offset, copy.size);
+			Libs::LibKernel::Memory::WriteBacking(copy.address, mapped + offset, copy.size);
+			cursor += AlignDownload(envelope_size);
+		}
+		batch.clear();
+		packed_size = 0;
+	};
+	for (auto copy: copies) {
+		while (copy.size != 0) {
+			const auto available = download.Size() - packed_size;
+			const auto prefix    = copy.source_offset & 3u;
+			const auto bytes     = std::min(copy.size, available - prefix);
+			DownloadCopy part {copy.buffer, copy.source_offset, copy.address, bytes};
+			const auto [source_begin, envelope_size] = DownloadEnvelope(part);
+			(void)source_begin;
+			packed_size += AlignDownload(envelope_size);
+			batch.push_back(part);
+			copy.source_offset += bytes;
+			copy.address += bytes;
+			copy.size -= bytes;
+			if (packed_size == download.Size()) {
+				flush();
+			}
+		}
+	}
+	if (!batch.empty()) {
+		flush();
+	}
 	for (const auto& copy: copies) {
-		const auto [source_begin, envelope_size] = DownloadEnvelope(copy);
-		(void)source_begin;
-		if (envelope_size > UINT64_MAX - (DOWNLOAD_ALIGNMENT - 1)) {
-			EXIT("BufferCache: download batch alignment overflow\n");
-		}
-		const auto aligned_size = AlignDownload(envelope_size);
-		if (aligned_size > UINT64_MAX - reservation_size) {
-			EXIT("BufferCache: download batch overflow\n");
-		}
-		reservation_size += aligned_size;
+		m_gpu_modified_ranges.Subtract(copy.address, copy.size);
 	}
-	if (reservation_size == 0) {
-		return {};
-	}
-
-	auto& download                   = m_download_buffer;
-	const auto [mapped, base_offset] = download.Map(reservation_size, DOWNLOAD_ALIGNMENT);
-	if (mapped == nullptr) {
-		EXIT("BufferCache: download batch could not reserve the shared stream\n");
-	}
-
-	std::vector<DownloadRange> downloads;
-	downloads.reserve(copies.size());
-	uint64_t cursor = 0;
-	for (const auto& copy: copies) {
-		const auto [source_begin, envelope_size] = DownloadEnvelope(copy);
-		const auto prefix                        = copy.source_offset - source_begin;
-		download.CopyFrom(m_scheduler.Current(), *copy.buffer, source_begin, base_offset + cursor,
-		                  envelope_size, vk::AccessFlagBits::eMemoryWrite, vk::AccessFlags {},
-		                  vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite,
-		                  vk::AccessFlagBits::eHostRead);
-		downloads.push_back({copy.address, copy.size, base_offset + cursor + prefix});
-		cursor += AlignDownload(envelope_size);
-	}
-	download.Commit();
-	return downloads;
-}
-
-void BufferCache::PublishDownloads(std::span<const DownloadRange> downloads) {
-	for (const auto& range: downloads) {
-		m_download_buffer.Invalidate(range.offset, range.size);
-		Libs::LibKernel::Memory::WriteBacking(
-		    range.address, m_download_buffer.Mapped().data() + range.offset, range.size);
-	}
-}
-
-void BufferCache::QueueGarbageDownload(std::span<const DownloadCopy> copies, BufferId id,
-                                       uint64_t address, uint64_t size) {
-	EXIT_IF(copies.empty());
-	auto downloads = RecordDownloads(copies);
-	auto retired   = downloads;
-	m_scheduler.DeferPriorityOperation(
-	    [this, downloads = std::move(downloads)]() mutable { PublishDownloads(downloads); });
-	m_scheduler.DeferOperation([this, retired = std::move(retired), id, address, size]() mutable {
-		for (const auto& range: retired) {
-			m_gpu_modified_ranges.Subtract(range.address, range.size);
-		}
-		// ForEachDownloadRange reports full tracker pages, and every exact GPU-owned
-		// interval on those pages was downloaded and removed. Clearing the original
-		// query therefore cannot orphan a dirty sibling on an edge page.
-		m_memory_tracker.UnmarkRegionAsGpuModified(address, size);
-		if (m_memory_tracker.IsRegionGpuModified(address, size) ||
-		    !m_gpu_modified_ranges.Intersections(address, size).empty()) {
-			EXIT("BufferCache: asynchronous garbage collection retained GPU ownership\n");
-		}
-		m_memory_tracker.UntrackMemory(address, size);
-		EXIT_IF(!m_slot_buffers[id].is_deleted);
-		m_slot_buffers.erase(id);
-	});
 }
 
 BufferCache::BufferCache(GraphicContext& graphics, CommandScheduler& scheduler,
                          PageManager& page_manager, TextureCache& texture_cache)
-    : m_graphics(graphics), m_scheduler(scheduler),
-      m_gds_buffer(graphics, scheduler, MemoryUsage::Stream, 0, AllFlags, GdsBufferSize),
-      m_memory_tracker(page_manager),
-      m_staging_buffer(graphics, scheduler, MemoryUsage::Upload, 512 * MiB),
-      m_stream_buffer(graphics, scheduler, MemoryUsage::Stream, 64 * MiB),
-      m_download_buffer(graphics, scheduler, MemoryUsage::Download, 32 * MiB),
-      m_device_buffer(graphics, scheduler, MemoryUsage::DeviceLocal, 128 * MiB),
-      m_texture_cache(texture_cache) {
+	: m_graphics(graphics), m_scheduler(scheduler),
+	  m_fault_manager(graphics, scheduler, *this, CACHING_PAGEBITS, CACHING_NUMPAGES),
+	  m_gds_buffer(graphics, scheduler, MemoryUsage::Stream, 0, AllFlags, GdsBufferSize),
+	  m_bda_pagetable_buffer(graphics, scheduler, MemoryUsage::DeviceLocal, 0, AllFlags,
+	                         BDA_PAGETABLE_SIZE),
+	  m_memory_tracker(page_manager),
+	  m_staging_buffer(graphics, scheduler, MemoryUsage::Upload, 512 * MiB),
+	  m_stream_buffer(graphics, scheduler, MemoryUsage::Stream, 64 * MiB),
+	  m_download_buffer(graphics, scheduler, MemoryUsage::Download, 32 * MiB),
+	  m_device_buffer(graphics, scheduler, MemoryUsage::DeviceLocal, 128 * MiB),
+	  m_texture_cache(texture_cache) {
 	std::memset(m_gds_buffer.Mapped().data(), 0, static_cast<size_t>(m_gds_buffer.Size()));
 	m_gds_buffer.Flush(0, m_gds_buffer.Size());
+	SetVulkanObjectNameF(m_graphics.device, m_bda_pagetable_buffer.Handle(),
+	                     "BDA Page Table Buffer");
 	const auto null_id =
 	    m_slot_buffers.insert(m_graphics, m_scheduler, MemoryUsage::DeviceLocal, 0, AllFlags, 16);
 	EXIT_IF(null_id != NULL_BUFFER_ID);
@@ -251,15 +272,15 @@ void BufferCache::ReadMemoryOnGpu(uint64_t vaddr, uint64_t size, bool is_write)
 		    m_memory_tracker.ValidateGpuDirtyPages(m_gpu_modified_ranges, address, bytes,
 		                                           "memory invalidation");
 	    },
-	    [&](uint64_t address, uint64_t bytes) noexcept {
+		[&](uint64_t address, uint64_t bytes) noexcept {
 		    for (const auto range: m_gpu_modified_ranges.Intersections(address, bytes)) {
 			    for (uint64_t copied = 0; copied < range.size;) {
 				    const auto copy_address = range.address + copied;
-				    auto       owner        = m_buffers.upper_bound(copy_address);
-				    if (owner == m_buffers.begin()) {
+				    const auto* owner = m_page_table.Find(copy_address >> PageTable::kPageBits);
+				    if (owner == nullptr || !*owner) {
 					    EXIT("BufferCache: invalidation readback has no buffer owner\n");
 				    }
-				    auto& buffer = m_slot_buffers[std::prev(owner)->second];
+				    auto& buffer = m_slot_buffers[*owner];
 				    if (!buffer.IsInBounds(copy_address, 1)) {
 					    EXIT("BufferCache: invalidation readback is outside its buffer owner\n");
 				    }
@@ -281,12 +302,7 @@ void BufferCache::ReadMemoryOnGpu(uint64_t vaddr, uint64_t size, bool is_write)
 		m_memory_tracker.MarkRegionAsCpuModified(vaddr, size);
 		return;
 	}
-	auto downloads = RecordDownloads(copies);
-	m_scheduler.FinishCurrent();
-	PublishDownloads(downloads);
-	for (const auto& range: downloads) {
-		m_gpu_modified_ranges.Subtract(range.address, range.size);
-	}
+	DownloadBufferMemory(copies);
 	// The enumeration above covered whole dirty pages and every exact interval on them.
 	m_memory_tracker.UnmarkRegionAsGpuModified(vaddr, size);
 	if (is_write) {
@@ -294,78 +310,6 @@ void BufferCache::ReadMemoryOnGpu(uint64_t vaddr, uint64_t size, bool is_write)
 	}
 }
 
-void BufferCache::UnmapMemory(uint64_t vaddr, uint64_t size) {
-	if (vaddr == 0 || size == 0 || size > UINT64_MAX - vaddr) {
-		EXIT("BufferCache: invalid unmap range\n");
-	}
-	std::vector<DownloadCopy>                  copies;
-	std::vector<std::pair<uint64_t, uint64_t>> modified_buffers;
-	std::vector<std::pair<uint64_t, uint64_t>> retired_buffers;
-	for (const auto& [begin, id]: m_buffers) {
-		const auto& buffer = m_slot_buffers[id];
-		if (vaddr < begin + buffer.Size() && begin < vaddr + size) {
-			retired_buffers.emplace_back(begin, buffer.Size());
-		}
-	}
-	for (const auto& [begin, id]: m_buffers) {
-		const auto& buffer = m_slot_buffers[id];
-		if (vaddr >= begin + buffer.Size() || begin >= vaddr + size ||
-		    !m_memory_tracker.IsRegionGpuModified(begin, buffer.Size())) {
-			continue;
-		}
-		const auto dirty = m_gpu_modified_ranges.Intersections(begin, buffer.Size());
-		if (dirty.empty()) {
-			EXIT("BufferCache: GPU-modified buffer has no dirty ranges\n");
-		}
-		modified_buffers.emplace_back(begin, buffer.Size());
-	}
-	for (const auto& [begin, bytes]: modified_buffers) {
-		auto owner = m_buffers.find(begin);
-		if (owner == m_buffers.end() || m_slot_buffers[owner->second].Size() != bytes) {
-			EXIT("BufferCache: unmap owner changed during collection\n");
-		}
-		auto& buffer = m_slot_buffers[owner->second];
-		m_memory_tracker.ForEachDownloadRange<false>(
-		    begin, buffer.Size(),
-		    [&](uint64_t address, uint64_t bytes) noexcept {
-			    m_memory_tracker.ValidateGpuDirtyPages(m_gpu_modified_ranges, address, bytes,
-			                                           "unmap");
-		    },
-		    [&](uint64_t address, uint64_t bytes) noexcept {
-			    for (const auto& range: m_gpu_modified_ranges.Intersections(address, bytes)) {
-				    copies.push_back({&buffer, range.address - begin, range.address, range.size});
-			    }
-		    });
-	}
-	if (!copies.empty()) {
-		auto downloads = RecordDownloads(copies);
-		m_scheduler.FinishCurrent();
-		PublishDownloads(downloads);
-	} else if (!retired_buffers.empty()) {
-		// Image uploads can reference a clean cached buffer without owning it. Submit the active
-		// command stream before removing such backing.
-		m_scheduler.FinishCurrent();
-	}
-	for (const auto& [begin, bytes]: modified_buffers) {
-		m_gpu_modified_ranges.Subtract(begin, bytes);
-		m_memory_tracker.UnmarkRegionAsGpuModified(begin, bytes);
-	}
-	for (const auto& [begin, bytes]: retired_buffers) {
-		m_memory_tracker.MarkRegionAsCpuModified(begin, bytes);
-	}
-	if (!m_gpu_modified_ranges.Intersections(vaddr, size).empty()) {
-		EXIT("BufferCache: unmap retained dirty byte ranges\n");
-	}
-	m_memory_tracker.UntrackMemory(vaddr, size);
-	for (const auto& [begin, bytes]: retired_buffers) {
-		(void)bytes;
-		const auto found = m_buffers.find(begin);
-		if (found != m_buffers.end()) {
-			DeleteBuffer(found->second);
-		}
-	}
-}
-
 BufferId BufferCache::FindBuffer(uint64_t vaddr, uint64_t size) {
 	if (vaddr == 0) {
 		return NULL_BUFFER_ID;
@@ -373,99 +317,143 @@ BufferId BufferCache::FindBuffer(uint64_t vaddr, uint64_t size) {
 	if (size == 0 || vaddr >= TRACKER_ADDRESS_SIZE || size > TRACKER_ADDRESS_SIZE - vaddr) {
 		EXIT("BufferCache: invalid buffer discovery request\n");
 	}
-	auto& command = m_scheduler.Current();
-	EXIT_IF(command.IsInvalid() || command.IsExecute());
-	auto       begin = vaddr & ~(CACHING_PAGE_SIZE - 1);
-	auto       end   = (vaddr + size + CACHING_PAGE_SIZE - 1) & ~(CACHING_PAGE_SIZE - 1);
-	const auto next  = m_buffers.upper_bound(vaddr);
-	if (next != m_buffers.begin()) {
-		const auto owner  = std::prev(next);
-		auto&      buffer = m_slot_buffers[owner->second];
+	const auto* owner = m_page_table.Find(vaddr >> PageTable::kPageBits);
+	if (owner != nullptr && *owner) {
+		auto& buffer = m_slot_buffers[*owner];
 		if (buffer.IsInBounds(vaddr, size)) {
-			buffer.tick_accessed_last = m_gc_tick;
-			return owner->second;
+			return *owner;
 		}
 	}
+	return CreateBuffer(vaddr, size);
+}
 
-	auto merge = [&](const Buffer& buffer) {
-		const auto buffer_begin = buffer.CpuAddress();
-		const auto buffer_end   = buffer_begin + buffer.Size();
-		if (begin >= buffer_end || buffer_begin >= end) {
-			return false;
-		}
-		begin = std::min(begin, buffer_begin);
-		end   = std::max(end, buffer_end);
-		return true;
-	};
-	std::vector<BufferId> overlaps;
-	auto                  first = m_buffers.lower_bound(begin);
+BufferId BufferCache::CreateBuffer(uint64_t vaddr, uint64_t size) {
+	auto& command = m_scheduler.Current();
+	EXIT_IF(command.IsInvalid());
+	auto       begin = vaddr & ~(CACHING_PAGESIZE - 1);
+	auto       end   = (vaddr + size + CACHING_PAGESIZE - 1) & ~(CACHING_PAGESIZE - 1);
+	auto       first = m_buffers.lower_bound(begin);
 	if (first != m_buffers.begin()) {
 		const auto previous = std::prev(first);
-		if (merge(m_slot_buffers[previous->second])) {
+		if (const auto& buffer = m_slot_buffers[previous->second];
+		    buffer.CpuAddress() + buffer.Size() > begin) {
 			first = previous;
 		}
 	}
-	for (auto candidate = first; candidate != m_buffers.end(); ++candidate) {
-		if (candidate->first >= end) {
-			break;
-		}
-		if (merge(m_slot_buffers[candidate->second])) {
-			overlaps.push_back(candidate->second);
-		}
+	auto last = first;
+	for (; last != m_buffers.end() && last->first < end; ++last) {
+		const auto& buffer = m_slot_buffers[last->second];
+		begin              = std::min(begin, buffer.CpuAddress());
+		end                = std::max(end, buffer.CpuAddress() + buffer.Size());
 	}
 
-	const auto id = m_slot_buffers.insert(m_graphics, m_scheduler, MemoryUsage::DeviceLocal, begin,
-	                                      AllFlags, end - begin);
-	auto&      buffer         = m_slot_buffers[id];
-	buffer.tick_accessed_last = m_gc_tick;
+	const auto id = m_slot_buffers.insert(
+	    m_graphics, m_scheduler, MemoryUsage::DeviceLocal, begin,
+	    AllFlags | vk::BufferUsageFlagBits::eShaderDeviceAddress, end - begin);
+	auto&      buffer = m_slot_buffers[id];
 	SetVulkanObjectNameF(m_graphics.device, buffer.Handle(),
 	                     "Kyty.GameBuffer[guest=0x{:016x} size=0x{:x}]", begin, end - begin);
-	for (const auto overlap: overlaps) {
-		const auto& old = m_slot_buffers[overlap];
+	for (auto overlap = first; overlap != last;) {
+		const auto current = overlap++;
+		const auto old_id  = current->second;
+		const auto& old    = m_slot_buffers[old_id];
 		buffer.CopyFrom(command, old, 0, old.CpuAddress() - begin, old.Size());
+		DeleteBuffer(old_id);
 	}
-	for (const auto overlap: overlaps) {
-		DeleteBuffer(overlap);
-	}
-	m_total_used_memory += buffer.Size();
-	const auto [it, inserted] = m_buffers.emplace(begin, id);
-	(void)it;
-	EXIT_IF(!inserted);
+	Register(id);
 	return id;
 }
 
 bool BufferCache::SynchronizeBuffer(Buffer& buffer, uint64_t vaddr, uint64_t size, bool is_written,
                                     bool is_texel_buffer) {
-	std::vector<std::pair<uint64_t, uint64_t>> uploads;
+	std::vector<vk::BufferCopy> copies;
+	uint64_t                    total_size = 0;
+	vk::Buffer                  source;
 	m_memory_tracker.ForEachUploadRange(
 	    vaddr, size, is_written,
-	    [&](uint64_t address, uint64_t bytes) noexcept { uploads.emplace_back(address, bytes); },
-	    [&]() noexcept {
-		    for (const auto& [address, bytes]: uploads) {
-			    WriteDataBuffer(buffer, address, reinterpret_cast<const void*>(address), bytes);
-		    }
-	    });
+	    [&](uint64_t address, uint64_t bytes) noexcept {
+		    copies.emplace_back(total_size, buffer.Offset(address), bytes);
+		    total_size += bytes;
+	    },
+	    [&]() noexcept { source = UploadCopies(buffer, copies, total_size); });
+	if (source) {
+		auto& command = m_scheduler.Current();
+		command.EndRendering();
+		const auto native = command.Handle();
+		vk::BufferMemoryBarrier before {};
+		before.sType         = vk::StructureType::eBufferMemoryBarrier;
+		before.srcAccessMask = vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite |
+		                       vk::AccessFlagBits::eTransferRead |
+		                       vk::AccessFlagBits::eTransferWrite;
+		before.dstAccessMask       = vk::AccessFlagBits::eTransferWrite;
+		before.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
+		before.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
+		before.buffer              = buffer.Handle();
+		before.offset              = 0;
+		before.size                = buffer.Size();
+		native.pipelineBarrier(vk::PipelineStageFlagBits::eAllCommands,
+		                       vk::PipelineStageFlagBits::eTransfer,
+		                       vk::DependencyFlagBits::eByRegion, 0, nullptr, 1, &before, 0, nullptr);
+		native.copyBuffer(source, buffer.Handle(), static_cast<uint32_t>(copies.size()),
+		                  copies.data());
+		auto after          = before;
+		after.srcAccessMask = vk::AccessFlagBits::eTransferWrite;
+		after.dstAccessMask = vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite;
+		native.pipelineBarrier(vk::PipelineStageFlagBits::eTransfer,
+		                       vk::PipelineStageFlagBits::eAllCommands,
+		                       vk::DependencyFlagBits::eByRegion, 0, nullptr, 1, &after, 0, nullptr);
+	}
 	if (is_texel_buffer && !is_written) {
 		return SynchronizeBufferFromImage(buffer, vaddr, size);
 	}
 	return false;
 }
 
+vk::Buffer BufferCache::UploadCopies(Buffer& buffer, std::span<vk::BufferCopy> copies,
+                                     uint64_t total_size) {
+	if (copies.empty()) {
+		return nullptr;
+	}
+
+	auto [mapped, base_offset] = m_staging_buffer.Map(total_size, 4);
+	if (mapped != nullptr) {
+		for (auto& copy: copies) {
+			const auto address = buffer.CpuAddress() + copy.dstOffset;
+			std::memcpy(mapped + copy.srcOffset, reinterpret_cast<const void*>(address), copy.size);
+			copy.srcOffset += base_offset;
+		}
+		m_staging_buffer.Commit();
+		return m_staging_buffer.Handle();
+	}
+
+	auto temporary = std::make_unique<Buffer>(m_graphics, m_scheduler, MemoryUsage::Upload, 0,
+	                                         vk::BufferUsageFlagBits::eTransferSrc, total_size);
+	for (const auto& copy: copies) {
+		const auto address = buffer.CpuAddress() + copy.dstOffset;
+		std::memcpy(temporary->Mapped().data() + copy.srcOffset,
+		            reinterpret_cast<const void*>(address), copy.size);
+	}
+	temporary->Flush(0, total_size);
+	const auto handle = temporary->Handle();
+	m_scheduler.DeferOperation([owner = std::move(temporary)]() mutable { owner.reset(); });
+	return handle;
+}
+
 std::pair<Buffer*, uint64_t> BufferCache::ObtainBuffer(uint64_t vaddr, uint64_t size,
                                                        bool is_written, bool is_texel_buffer,
                                                        BufferId id) {
 	auto& command = m_scheduler.Current();
-	if (command.IsInvalid() || command.IsExecute() || vaddr == 0 || size == 0 ||
+	if (command.IsInvalid() || vaddr == 0 || size == 0 ||
 	    vaddr >= TRACKER_ADDRESS_SIZE || size > TRACKER_ADDRESS_SIZE - vaddr) {
 		EXIT("BufferCache: buffer request requires a recording command buffer\n");
 	}
 
-	if (!is_written && size <= CACHING_PAGE_SIZE &&
+	if (!is_written && size <= CACHING_PAGESIZE &&
 	    !m_memory_tracker.IsRegionGpuModified(vaddr, size) &&
 	    m_memory_tracker.IsRegionCpuModified(vaddr, size)) {
 		const auto alignment = std::max<uint64_t>(
 		    m_graphics.physical_device_properties.limits.minUniformBufferOffsetAlignment, 1);
-		auto [mapped, offset] = m_stream_buffer.Map(size, alignment);
+		auto [mapped, offset] = m_stream_buffer.Map(size, alignment, false);
 		if (mapped != nullptr && Libs::LibKernel::Memory::TryReadBacking(vaddr, mapped, size)) {
 			m_stream_buffer.Commit();
 			return {&m_stream_buffer, offset};
@@ -477,6 +465,7 @@ std::pair<Buffer*, uint64_t> BufferCache::ObtainBuffer(uint64_t vaddr, uint64_t
 		id     = FindBuffer(vaddr, size);
 		buffer = &m_slot_buffers[id];
 	}
+	TouchBuffer(*buffer);
 	(void)SynchronizeBuffer(*buffer, vaddr, size, is_written, is_texel_buffer);
 	if (is_written) {
 		m_gpu_modified_ranges.Add(vaddr, size);
@@ -490,20 +479,18 @@ std::pair<Buffer*, uint64_t> BufferCache::ObtainBufferForImage(uint64_t vaddr, u
 		EXIT("BufferCache: invalid image source\n");
 	}
 	auto find_owner = [&]() -> Buffer* {
-		auto owner = m_buffers.upper_bound(vaddr);
-		if (owner == m_buffers.begin()) {
+		const auto* owner = m_page_table.Find(vaddr >> PageTable::kPageBits);
+		if (owner == nullptr || !*owner) {
 			return nullptr;
 		}
-		--owner;
-		auto* buffer = m_slot_buffers.try_get(owner->second);
-		return buffer != nullptr && buffer->IsInBounds(vaddr, size) ? buffer : nullptr;
+		auto& buffer = m_slot_buffers[*owner];
+		return buffer.IsInBounds(vaddr, size) ? &buffer : nullptr;
 	};
 
 	{
 		const bool cpu_modified            = m_memory_tracker.IsRegionCpuModified(vaddr, size);
 		const bool gpu_modified            = m_memory_tracker.IsRegionGpuModified(vaddr, size);
-		const auto dirty                   = m_gpu_modified_ranges.Intersections(vaddr, size);
-		const bool has_dirty_buffer_source = !dirty.empty();
+		const bool has_dirty_buffer_source = m_gpu_modified_ranges.Intersects(vaddr, size);
 		m_memory_tracker.ValidateGpuDirtyOwnership(m_gpu_modified_ranges, vaddr, size,
 		                                           "image source");
 
@@ -512,17 +499,10 @@ std::pair<Buffer*, uint64_t> BufferCache::ObtainBufferForImage(uint64_t vaddr, u
 			if (!IsRegionRegistered(vaddr, size)) {
 				EXIT("BufferCache: GPU-dirty image source has no native buffer\n");
 			}
-			const auto id       = FindBuffer(vaddr, size);
-			auto&      buffer   = m_slot_buffers[id];
-			owner               = &buffer;
-			const auto resolved = m_buffers.find(buffer.CpuAddress());
-			if (resolved == m_buffers.end() || resolved->second != id ||
-			    !buffer.IsInBounds(vaddr, size)) {
-				EXIT("BufferCache: merged image source does not contain the requested range\n");
-			}
+			owner = &m_slot_buffers[FindBuffer(vaddr, size)];
 		}
 		if (owner != nullptr && !cpu_modified && (!gpu_modified || has_dirty_buffer_source)) {
-			owner->tick_accessed_last = m_gc_tick;
+			TouchBuffer(*owner);
 			return {owner, owner->Offset(vaddr)};
 		}
 		if (has_dirty_buffer_source && owner == nullptr) {
@@ -537,8 +517,7 @@ std::pair<Buffer*, uint64_t> BufferCache::ObtainBufferForImage(uint64_t vaddr, u
 	}
 	m_staging_buffer.Commit();
 
-	const auto dirty                   = m_gpu_modified_ranges.Intersections(vaddr, size);
-	const bool has_dirty_buffer_source = !dirty.empty();
+	const bool has_dirty_buffer_source = m_gpu_modified_ranges.Intersects(vaddr, size);
 	auto*      owner                   = find_owner();
 	if (has_dirty_buffer_source && owner == nullptr) {
 		EXIT("BufferCache: GPU-dirty image source lost its native owner\n");
@@ -548,7 +527,7 @@ std::pair<Buffer*, uint64_t> BufferCache::ObtainBufferForImage(uint64_t vaddr, u
 		return {&m_staging_buffer, stage_offset};
 	}
 
-	owner->tick_accessed_last = m_gc_tick;
+	TouchBuffer(*owner);
 	std::vector<std::pair<uint64_t, uint64_t>> uploads;
 	m_memory_tracker.ForEachUploadRange(
 	    vaddr, size, false,
@@ -581,7 +560,7 @@ void BufferCache::WriteHostMemory(uint64_t vaddr, std::span<const uint8_t> data)
 			continue;
 		}
 		WriteDataBuffer(buffer, begin, data.data() + begin - vaddr, range_end - begin);
-		buffer.tick_accessed_last = m_gc_tick;
+		TouchBuffer(buffer);
 	}
 }
 
@@ -701,7 +680,7 @@ bool BufferCache::IsRegionGpuModified(uint64_t vaddr, uint64_t size) {
 }
 
 bool BufferCache::HasGpuDirtyBytes(uint64_t vaddr, uint64_t size) {
-	return !m_gpu_modified_ranges.Intersections(vaddr, size).empty();
+	return m_gpu_modified_ranges.Intersects(vaddr, size);
 }
 
 bool BufferCache::IsRegionCpuModified(uint64_t vaddr, uint64_t size) {
@@ -721,30 +700,18 @@ void BufferCache::RunGarbageCollector() {
 	const uint64_t age        = std::min<uint64_t>(aggressive ? 80 : 160, tick);
 	const size_t   limit      = aggressive ? 64 : 32;
 
-	std::vector<uint64_t> candidates;
-	for (const auto& [address, id]: m_buffers) {
-		const auto& buffer = m_slot_buffers[id];
-		if (tick - std::min(tick, buffer.tick_accessed_last) >= age) {
-			candidates.push_back(address);
-		}
-	}
-	std::ranges::sort(candidates, [&](uint64_t left, uint64_t right) {
-		return m_slot_buffers[m_buffers.at(left)].tick_accessed_last <
-		       m_slot_buffers[m_buffers.at(right)].tick_accessed_last;
-	});
-	if (candidates.size() > limit) {
-		candidates.resize(limit);
-	}
-
-	for (const auto address: candidates) {
-		auto found = m_buffers.find(address);
-		EXIT_IF(found == m_buffers.end());
-		const auto id     = found->second;
-		auto&      buffer = m_slot_buffers[id];
+	std::vector<BufferId> dirty_buffers;
+	std::vector<DownloadCopy> copies;
+	size_t                    retire_count = 0;
+	m_lru_cache.ForEachItemBelow(tick - age, [&](BufferId id) {
+		auto& buffer = m_slot_buffers[id];
+		EXIT_IF(buffer.is_deleted);
 		m_memory_tracker.ValidateGpuDirtyOwnership(m_gpu_modified_ranges, buffer.CpuAddress(),
 		                                           buffer.Size(), "garbage collection");
 		const bool dirty = m_memory_tracker.IsRegionGpuModified(buffer.CpuAddress(), buffer.Size());
-		std::vector<DownloadCopy> copies;
+		if (dirty && !aggressive) {
+			return false;
+		}
 		if (dirty) {
 			m_memory_tracker.ForEachDownloadRange<false>(
 			    buffer.CpuAddress(), buffer.Size(),
@@ -753,24 +720,55 @@ void BufferCache::RunGarbageCollector() {
 				                                           dirty_size, "garbage collection");
 			    },
 			    [&](uint64_t dirty_address, uint64_t dirty_size) noexcept {
-				    for (const auto range:
-				         m_gpu_modified_ranges.Intersections(dirty_address, dirty_size)) {
+				    m_gpu_modified_ranges.ForEachIntersection(
+				        dirty_address, dirty_size, [&](RangeSet::Range range) {
 					    copies.push_back({&buffer, range.address - buffer.CpuAddress(),
 					                      range.address, range.size});
-				    }
-			    });
-		}
-
-		const auto retire_address = buffer.CpuAddress();
-		const auto retire_size    = buffer.Size();
-		if (dirty) {
-			EXIT_IF(copies.empty());
-			Unregister(id);
-			QueueGarbageDownload(copies, id, retire_address, retire_size);
+				        });
+				});
+			dirty_buffers.push_back(id);
 		} else {
-			m_memory_tracker.UntrackMemory(retire_address, retire_size);
+			m_memory_tracker.UntrackMemory(buffer.CpuAddress(), buffer.Size());
 			DeleteBuffer(id);
 		}
+		return ++retire_count == limit;
+	});
+	if (dirty_buffers.empty()) {
+		return;
+	}
+
+	EXIT_IF(copies.empty());
+	DownloadBufferMemory(copies);
+	for (const auto id: dirty_buffers) {
+		auto& buffer = m_slot_buffers[id];
+		m_memory_tracker.UnmarkRegionAsGpuModified(buffer.CpuAddress(), buffer.Size());
+		if (m_memory_tracker.IsRegionGpuModified(buffer.CpuAddress(), buffer.Size()) ||
+		    m_gpu_modified_ranges.Intersects(buffer.CpuAddress(), buffer.Size())) {
+			EXIT("BufferCache: garbage collection retained GPU ownership\n");
+		}
+		m_memory_tracker.UntrackMemory(buffer.CpuAddress(), buffer.Size());
+		Unregister(id);
+		m_slot_buffers.erase(id);
+	}
+}
+
+void BufferCache::ProcessFaultBuffer() {
+	m_fault_manager.ProcessFaultBuffer();
+}
+
+void BufferCache::SynchronizeBuffersInRange(uint64_t vaddr, uint64_t size) {
+	const auto end = vaddr + size;
+	auto       it  = m_buffers.upper_bound(vaddr);
+	if (it != m_buffers.begin()) {
+		--it;
+	}
+	for (; it != m_buffers.end() && it->first < end; ++it) {
+		auto&      buffer = m_slot_buffers[it->second];
+		const auto start  = std::max(buffer.CpuAddress(), vaddr);
+		const auto finish = std::min(buffer.CpuAddress() + buffer.Size(), end);
+		if (start < finish) {
+			(void)SynchronizeBuffer(buffer, start, finish - start, false, false);
+		}
 	}
 }
 

src/graphics/host_gpu/renderer/cache/bufferCache.h

@@ -3,9 +3,12 @@
 
 #include "common/abi.h"
 #include "common/common.h"
+#include "common/lruCache.h"
 #include "common/slotVector.h"
 #include "graphics/host_gpu/memoryTracker.h"
 #include "graphics/host_gpu/rangeSet.h"
+#include "graphics/host_gpu/renderer/cache/faultManager.h"
+#include "graphics/host_gpu/renderer/cache/multiLevelPageTable.h"
 #include "graphics/host_gpu/renderer/cache/streamBuffer.h"
 
 #include <map>
@@ -24,7 +27,11 @@ inline constexpr BufferId NULL_BUFFER_ID {0};
 
 class BufferCache {
 public:
-	static constexpr uint64_t CACHING_PAGE_SIZE = 16ull * 1024ull;
+	static constexpr uint32_t CACHING_PAGEBITS  = 14;
+	static constexpr uint64_t CACHING_PAGESIZE  = uint64_t {1} << CACHING_PAGEBITS;
+	static constexpr uint64_t CACHING_NUMPAGES  = uint64_t {1} << (40 - CACHING_PAGEBITS);
+	static constexpr uint64_t BDA_PAGETABLE_SIZE =
+	    CACHING_NUMPAGES * sizeof(vk::DeviceAddress);
 
 	BufferCache(GraphicContext& graphics, CommandScheduler& scheduler, PageManager& page_manager,
 	            TextureCache& texture_cache);
@@ -33,7 +40,6 @@ public:
 
 	void                   InvalidateMemory(uint64_t vaddr, uint64_t size);
 	void                   ReadMemory(uint64_t vaddr, uint64_t size, bool is_write = false);
-	void                   UnmapMemory(uint64_t vaddr, uint64_t size);
 	[[nodiscard]] Buffer&  GetBuffer(BufferId id) { return m_slot_buffers[id]; }
 	[[nodiscard]] BufferId FindBuffer(uint64_t vaddr, uint64_t size);
 	[[nodiscard]] std::pair<Buffer*, uint64_t> ObtainBuffer(uint64_t vaddr, uint64_t size,
@@ -50,6 +56,8 @@ public:
 		EXIT("BufferCache: invalid utility-buffer usage\n");
 	}
 	[[nodiscard]] const Buffer* GetGdsBuffer() const noexcept { return &m_gds_buffer; }
+	[[nodiscard]] Buffer* GetBdaPageTableBuffer() noexcept { return &m_bda_pagetable_buffer; }
+	[[nodiscard]] Buffer* GetFaultBuffer() noexcept { return m_fault_manager.GetFaultBuffer(); }
 	[[nodiscard]] std::pair<Buffer*, uint64_t> ObtainBufferForImage(uint64_t vaddr, uint64_t size);
 	void FillBuffer(uint64_t vaddr, uint64_t size, uint32_t value, bool is_gds);
 	void CopyBuffer(uint64_t dst_vaddr, uint64_t src_vaddr, uint64_t size, bool dst_gds,
@@ -59,47 +67,58 @@ public:
 	[[nodiscard]] bool HasGpuDirtyBytes(uint64_t vaddr, uint64_t size);
 	[[nodiscard]] bool IsRegionCpuModified(uint64_t vaddr, uint64_t size);
 	[[nodiscard]] bool IsRegionGpuModified(uint64_t vaddr, uint64_t size);
+	void               ProcessFaultBuffer();
+	void               SynchronizeBuffersInRange(uint64_t vaddr, uint64_t size);
 	void               RunGarbageCollector();
 
 private:
 	friend struct BufferCacheTestAccess;
 
 	struct DownloadCopy;
-	struct DownloadRange;
+	using PageTable = MultiLevelPageTable<BufferId, CACHING_PAGEBITS, 40, 16>;
+	static_assert(CACHING_PAGESIZE == (uint64_t {1} << PageTable::kPageBits));
 	static constexpr uint64_t               DOWNLOAD_ALIGNMENT = 64;
 	[[nodiscard]] static constexpr uint64_t AlignDownload(uint64_t size) noexcept {
 		return (size + DOWNLOAD_ALIGNMENT - 1) & ~(DOWNLOAD_ALIGNMENT - 1);
 	}
 	[[nodiscard]] static std::pair<uint64_t, uint64_t> DownloadEnvelope(const DownloadCopy& copy);
 	void WriteDataBuffer(Buffer& buffer, uint64_t address, const void* source, uint64_t size);
+	void TouchBuffer(const Buffer& buffer);
+	[[nodiscard]] BufferId CreateBuffer(uint64_t vaddr, uint64_t size);
+	void                   Register(BufferId id);
 	void Unregister(BufferId id);
+	template <bool insert>
+	void ChangeRegister(BufferId id);
 	void DeleteBuffer(BufferId id);
 	[[nodiscard]] bool SynchronizeBuffer(Buffer& buffer, uint64_t vaddr, uint64_t size,
 	                                     bool is_written, bool is_texel_buffer);
+	[[nodiscard]] vk::Buffer UploadCopies(Buffer& buffer, std::span<vk::BufferCopy> copies,
+	                                      uint64_t total_size);
 	[[nodiscard]] bool SynchronizeBufferFromImage(Buffer& buffer, uint64_t vaddr, uint64_t size);
-	[[nodiscard]] std::vector<DownloadRange> RecordDownloads(std::span<const DownloadCopy> copies);
-	void PublishDownloads(std::span<const DownloadRange> downloads);
-	void QueueGarbageDownload(std::span<const DownloadCopy> copies, BufferId id, uint64_t address,
-	                          uint64_t size);
+	void DownloadBufferMemory(std::span<const DownloadCopy> copies);
 	void WriteHostMemory(uint64_t vaddr, std::span<const uint8_t> data);
 	void ReadMemoryOnGpu(uint64_t vaddr, uint64_t size, bool is_write);
 
-	GraphicContext&              m_graphics;
-	CommandScheduler&            m_scheduler;
-	Buffer                       m_gds_buffer;
-	Common::SlotVector<Buffer>   m_slot_buffers;
-	std::map<uint64_t, BufferId> m_buffers;
-	RangeSet                     m_gpu_modified_ranges;
-	MemoryTracker                m_memory_tracker;
-	StreamBuffer                 m_staging_buffer;
-	StreamBuffer                 m_stream_buffer;
-	StreamBuffer                 m_download_buffer;
-	StreamBuffer                 m_device_buffer;
-	TextureCache&                m_texture_cache;
-	uint64_t                     m_total_used_memory  = 0;
-	uint64_t                     m_trigger_gc_memory  = 1ull * 1024 * 1024 * 1024;
-	uint64_t                     m_critical_gc_memory = 2ull * 1024 * 1024 * 1024;
-	uint64_t                     m_gc_tick            = 0;
+	GraphicContext&                                   m_graphics;
+	CommandScheduler&                                 m_scheduler;
+	FaultManager                                      m_fault_manager;
+	Buffer                                            m_gds_buffer;
+	Buffer                                            m_bda_pagetable_buffer;
+	Common::SlotVector<Buffer>                        m_slot_buffers;
+	Common::LeastRecentlyUsedCache<BufferId, uint64_t> m_lru_cache;
+	std::map<uint64_t, BufferId>                      m_buffers;
+	PageTable                                         m_page_table;
+	RangeSet                                          m_gpu_modified_ranges;
+	MemoryTracker                                     m_memory_tracker;
+	StreamBuffer                                      m_staging_buffer;
+	StreamBuffer                                      m_stream_buffer;
+	StreamBuffer                                      m_download_buffer;
+	StreamBuffer                                      m_device_buffer;
+	TextureCache&                                     m_texture_cache;
+	uint64_t                                          m_total_used_memory  = 0;
+	uint64_t m_trigger_gc_memory  = 1ull * 1024 * 1024 * 1024;
+	uint64_t m_critical_gc_memory = 2ull * 1024 * 1024 * 1024;
+	uint64_t m_gc_tick            = 0;
 };
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/cache/faultManager.cpp

@@ -0,0 +1,167 @@
+#include "graphics/host_gpu/renderer/cache/faultManager.h"
+
+#include "common/assert.h"
+#include "common/logging/log.h"
+#include "gpu_tiler_shaders/fault_buffer_process_spv.h"
+#include "graphics/host_gpu/graphicContext.h"
+#include "graphics/host_gpu/renderer/cache/bufferCache.h"
+#include "graphics/host_gpu/renderer/commandScheduler.h"
+#include "graphics/host_gpu/vulkanCommon.h"
+
+#include <bit>
+#include <cinttypes>
+#include <cstring>
+#include <limits>
+
+namespace Libs::Graphics {
+
+namespace {
+
+constexpr size_t MaxPageFaults    = 1024;
+constexpr size_t PageFaultAreaSize = MaxPageFaults * sizeof(uint64_t);
+
+} // namespace
+
+FaultManager::FaultManager(GraphicContext& graphics, CommandScheduler& scheduler,
+                           BufferCache& buffer_cache, uint32_t caching_pagebits,
+                           uint64_t caching_num_pages)
+    : m_graphics(graphics), m_scheduler(scheduler), m_buffer_cache(buffer_cache),
+      m_caching_pagesize(uint64_t {1} << caching_pagebits),
+      m_caching_num_pages(caching_num_pages), m_fault_buffer_size(caching_num_pages / 8),
+      m_fault_buffer(graphics, scheduler, MemoryUsage::DeviceLocal, 0, AllFlags,
+                     m_fault_buffer_size),
+      m_download_buffer(graphics, scheduler, MemoryUsage::Download, 0, AllFlags,
+                        MaxPendingFaults * PageFaultAreaSize) {
+	SetVulkanObjectNameF(m_graphics.device, m_fault_buffer.Handle(), "Fault Buffer");
+
+	const vk::DescriptorSetLayoutBinding bindings[] {
+	    {0, vk::DescriptorType::eStorageBuffer, 1, vk::ShaderStageFlagBits::eCompute, nullptr},
+	    {1, vk::DescriptorType::eStorageBuffer, 1, vk::ShaderStageFlagBits::eCompute, nullptr},
+	};
+	vk::DescriptorSetLayoutCreateInfo layout_info {};
+	layout_info.sType        = vk::StructureType::eDescriptorSetLayoutCreateInfo;
+	layout_info.flags        = vk::DescriptorSetLayoutCreateFlagBits::ePushDescriptorKHR;
+	layout_info.bindingCount = std::size(bindings);
+	layout_info.pBindings    = bindings;
+	RequireVulkanSuccess(
+	    m_graphics.device.createDescriptorSetLayout(&layout_info, nullptr,
+	                                                &m_fault_process_desc_layout),
+	    "create fault-buffer descriptor layout");
+
+	vk::ShaderModuleCreateInfo module_info {};
+	module_info.sType    = vk::StructureType::eShaderModuleCreateInfo;
+	module_info.codeSize = std::size(FAULT_BUFFER_PROCESS_SPV) * sizeof(uint32_t);
+	module_info.pCode    = FAULT_BUFFER_PROCESS_SPV;
+	vk::ShaderModule module = nullptr;
+	RequireVulkanSuccess(m_graphics.device.createShaderModule(&module_info, nullptr, &module),
+	                     "create fault-buffer shader module");
+
+	vk::PipelineLayoutCreateInfo pipeline_layout_info {};
+	pipeline_layout_info.sType          = vk::StructureType::ePipelineLayoutCreateInfo;
+	pipeline_layout_info.setLayoutCount = 1;
+	pipeline_layout_info.pSetLayouts    = &m_fault_process_desc_layout;
+	RequireVulkanSuccess(
+	    m_graphics.device.createPipelineLayout(&pipeline_layout_info, nullptr,
+	                                           &m_fault_process_pipeline_layout),
+	    "create fault-buffer pipeline layout");
+
+	vk::PipelineShaderStageCreateInfo stage {};
+	stage.sType  = vk::StructureType::ePipelineShaderStageCreateInfo;
+	stage.stage  = vk::ShaderStageFlagBits::eCompute;
+	stage.module = module;
+	stage.pName  = "main";
+	vk::ComputePipelineCreateInfo pipeline_info {};
+	pipeline_info.sType  = vk::StructureType::eComputePipelineCreateInfo;
+	pipeline_info.stage  = stage;
+	pipeline_info.layout = m_fault_process_pipeline_layout;
+	const auto result = m_graphics.device.createComputePipelines(
+	    nullptr, 1, &pipeline_info, nullptr, &m_fault_process_pipeline);
+	m_graphics.device.destroyShaderModule(module, nullptr);
+	RequireVulkanSuccess(result, "create fault-buffer pipeline");
+	SetVulkanObjectNameF(m_graphics.device, m_fault_process_pipeline, "Fault Buffer Parser");
+}
+
+FaultManager::~FaultManager() {
+	m_graphics.device.destroyPipeline(m_fault_process_pipeline, nullptr);
+	m_graphics.device.destroyPipelineLayout(m_fault_process_pipeline_layout, nullptr);
+	m_graphics.device.destroyDescriptorSetLayout(m_fault_process_desc_layout, nullptr);
+}
+
+void FaultManager::ProcessFaultBuffer() {
+	if (const auto wait_tick = m_fault_areas[m_current_area]; wait_tick != 0) {
+		m_scheduler.Wait(wait_tick);
+		m_scheduler.PopPendingOperations();
+	}
+
+	const auto offset = m_current_area * PageFaultAreaSize;
+	auto*      mapped = m_download_buffer.Mapped().data() + offset;
+	std::memset(mapped, 0, PageFaultAreaSize);
+	m_download_buffer.Flush(offset, PageFaultAreaSize);
+
+	vk::BufferMemoryBarrier2 pre_barrier {};
+	pre_barrier.sType         = vk::StructureType::eBufferMemoryBarrier2;
+	pre_barrier.srcStageMask  = vk::PipelineStageFlagBits2::eAllCommands;
+	pre_barrier.srcAccessMask = vk::AccessFlagBits2::eShaderWrite;
+	pre_barrier.dstStageMask  = vk::PipelineStageFlagBits2::eComputeShader;
+	pre_barrier.dstAccessMask = vk::AccessFlagBits2::eShaderRead;
+	pre_barrier.buffer        = m_fault_buffer.Handle();
+	pre_barrier.offset        = 0;
+	pre_barrier.size          = m_fault_buffer_size;
+	auto post_barrier         = pre_barrier;
+	post_barrier.srcStageMask  = vk::PipelineStageFlagBits2::eComputeShader;
+	post_barrier.srcAccessMask = vk::AccessFlagBits2::eShaderWrite;
+	post_barrier.dstStageMask  = vk::PipelineStageFlagBits2::eAllCommands;
+	post_barrier.dstAccessMask = vk::AccessFlagBits2::eShaderWrite;
+
+	const vk::DescriptorBufferInfo infos[] {
+	    {m_fault_buffer.Handle(), 0, m_fault_buffer_size},
+	    {m_download_buffer.Handle(), offset, PageFaultAreaSize},
+	};
+	std::array<vk::WriteDescriptorSet, 2> writes {};
+	for (uint32_t index = 0; index < writes.size(); ++index) {
+		writes[index].sType           = vk::StructureType::eWriteDescriptorSet;
+		writes[index].dstBinding      = index;
+		writes[index].descriptorCount = 1;
+		writes[index].descriptorType  = vk::DescriptorType::eStorageBuffer;
+		writes[index].pBufferInfo     = &infos[index];
+	}
+
+	m_scheduler.EndRendering();
+	auto command = m_scheduler.Current().Handle();
+	vk::DependencyInfo dependency {};
+	dependency.sType                    = vk::StructureType::eDependencyInfo;
+	dependency.dependencyFlags          = vk::DependencyFlagBits::eByRegion;
+	dependency.bufferMemoryBarrierCount = 1;
+	dependency.pBufferMemoryBarriers    = &pre_barrier;
+	command.pipelineBarrier2(dependency);
+	command.bindPipeline(vk::PipelineBindPoint::eCompute, m_fault_process_pipeline);
+	command.pushDescriptorSetKHR(vk::PipelineBindPoint::eCompute,
+	                             m_fault_process_pipeline_layout, 0, writes);
+	const auto num_threads    = m_caching_num_pages / 32;
+	const auto num_workgroups = (num_threads + 63) / 64;
+	command.dispatch(static_cast<uint32_t>(num_workgroups), 1, 1);
+	dependency.pBufferMemoryBarriers = &post_barrier;
+	command.pipelineBarrier2(dependency);
+
+	const auto area = m_current_area;
+	m_scheduler.DeferOperation([this, mapped, offset, area] {
+		m_download_buffer.Invalidate(offset, PageFaultAreaSize);
+		m_fault_ranges.Clear();
+		const auto* faults = std::bit_cast<const uint64_t*>(mapped);
+		const auto  count  = static_cast<uint32_t>(faults[0]);
+		for (uint32_t index = 1; index <= count; ++index) {
+			m_fault_ranges.Add(faults[index], m_caching_pagesize);
+			LOGF("Accessed non-GPU cached memory at 0x%016" PRIx64 "\n", faults[index]);
+		}
+		m_fault_ranges.ForEach([this](uint64_t start, uint64_t end) {
+			EXIT_IF(end - start > std::numeric_limits<uint32_t>::max());
+			(void)m_buffer_cache.FindBuffer(start, end - start);
+		});
+		m_fault_areas[area] = 0;
+	});
+
+	m_fault_areas[m_current_area++] = m_scheduler.CurrentTick();
+	m_current_area %= MaxPendingFaults;
+}
+
+} // namespace Libs::Graphics

src/graphics/host_gpu/renderer/cache/faultManager.h

@@ -0,0 +1,46 @@
+#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_CACHE_FAULTMANAGER_H_
+#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_CACHE_FAULTMANAGER_H_
+
+#include "common/abi.h"
+#include "graphics/host_gpu/rangeSet.h"
+#include "graphics/host_gpu/renderer/cache/streamBuffer.h"
+
+#include <array>
+#include <cstdint>
+
+namespace Libs::Graphics {
+
+class BufferCache;
+
+class FaultManager {
+	static constexpr size_t MaxPendingFaults = 8;
+
+public:
+	FaultManager(GraphicContext& graphics, CommandScheduler& scheduler, BufferCache& buffer_cache,
+	             uint32_t caching_pagebits, uint64_t caching_num_pages);
+	~FaultManager();
+	KYTY_CLASS_NO_COPY(FaultManager);
+
+	[[nodiscard]] Buffer* GetFaultBuffer() noexcept { return &m_fault_buffer; }
+	void                  ProcessFaultBuffer();
+
+private:
+	GraphicContext&                            m_graphics;
+	CommandScheduler&                          m_scheduler;
+	BufferCache&                               m_buffer_cache;
+	RangeSet                                   m_fault_ranges;
+	uint64_t                                   m_caching_pagesize;
+	uint64_t                                   m_caching_num_pages;
+	uint64_t                                   m_fault_buffer_size;
+	Buffer                                     m_fault_buffer;
+	Buffer                                     m_download_buffer;
+	std::array<uint64_t, MaxPendingFaults>      m_fault_areas {};
+	uint32_t                                   m_current_area = 0;
+	vk::DescriptorSetLayout                    m_fault_process_desc_layout = nullptr;
+	vk::Pipeline                               m_fault_process_pipeline = nullptr;
+	vk::PipelineLayout                         m_fault_process_pipeline_layout = nullptr;
+};
+
+} // namespace Libs::Graphics
+
+#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_CACHE_FAULTMANAGER_H_

src/graphics/host_gpu/renderer/cache/gpuResourceManager.cpp

@@ -60,10 +60,10 @@ void GpuResourceManager::UnmapMemory(uint64_t vaddr, uint64_t size) {
 	const auto unmap = [this, vaddr, size] {
 		if (m_scheduler.Active()) {
 			const auto tick = m_scheduler.CurrentTick();
-			m_scheduler.FinishCurrent();
+			m_scheduler.Finish();
 			m_scheduler.WaitPriorityOperations(tick);
 		}
-		m_buffer_cache.UnmapMemory(vaddr, size);
+		m_buffer_cache.InvalidateMemory(vaddr, size);
 		m_texture_cache.UnmapMemory(vaddr, size);
 		m_page_manager.OnGpuUnmap(vaddr, size);
 		std::lock_guard lock(m_mapped_ranges_mutex);
@@ -76,7 +76,19 @@ void GpuResourceManager::UnmapMemory(uint64_t vaddr, uint64_t size) {
 	m_gpu->SendCommandSync(unmap);
 }
 
+void GpuResourceManager::PrepareBda() {
+	std::shared_lock lock(m_mapped_ranges_mutex);
+	m_mapped_ranges.ForEach([this](uint64_t start, uint64_t end) {
+		m_buffer_cache.SynchronizeBuffersInRange(start, end - start);
+	});
+	m_fault_process_pending = true;
+}
+
 void GpuResourceManager::RunGarbageCollector() {
+	if (m_fault_process_pending) {
+		m_fault_process_pending = false;
+		m_buffer_cache.ProcessFaultBuffer();
+	}
 	m_texture_cache.ProcessDownloadImages();
 	m_texture_cache.RunGarbageCollector();
 	m_buffer_cache.RunGarbageCollector();

src/graphics/host_gpu/renderer/cache/gpuResourceManager.h

@@ -30,6 +30,7 @@ public:
 	[[nodiscard]] bool IsMapped(uint64_t vaddr, uint64_t size) const noexcept;
 	void               MapMemory(uint64_t vaddr, uint64_t size);
 	void               UnmapMemory(uint64_t vaddr, uint64_t size);
+	void               PrepareBda();
 	void               RunGarbageCollector();
 
 private:
@@ -40,6 +41,7 @@ private:
 	mutable std::shared_mutex m_mapped_ranges_mutex;
 	RangeSet                  m_mapped_ranges;
 	GuestGpu*                 m_gpu = nullptr;
+	bool                      m_fault_process_pending = false;
 };
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/cache/streamBuffer.cpp

@@ -73,8 +73,12 @@ Buffer::Buffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsag
 	buffer_info.usage       = flags;
 	buffer_info.sharingMode = vk::SharingMode::eExclusive;
 
+	const bool with_bda = bool(flags & vk::BufferUsageFlagBits::eShaderDeviceAddress);
+	const VmaAllocationCreateFlags bda_flag =
+	    with_bda ? VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT : 0;
 	VmaAllocationCreateInfo allocation_info {};
-	allocation_info.flags = VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT | AllocationFlags(usage);
+	allocation_info.flags =
+	    VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT | bda_flag | AllocationFlags(usage);
 	allocation_info.usage = AllocationUsage(usage);
 	allocation_info.preferredFlags = usage == MemoryUsage::DeviceLocal
 	                                     ? VkMemoryPropertyFlags {}
@@ -99,6 +103,13 @@ Buffer::Buffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsag
 	m_buffer->memory.type            = allocation_result.memoryType;
 	m_buffer->memory.unique_id       = VulkanNextMemoryUniqueId();
 	graphics.device.getBufferMemoryRequirements(m_buffer->buffer, &m_buffer->memory.requirements);
+	if (static_cast<bool>(flags & vk::BufferUsageFlagBits::eShaderDeviceAddress)) {
+		vk::BufferDeviceAddressInfo address_info {};
+		address_info.sType  = vk::StructureType::eBufferDeviceAddressInfo;
+		address_info.buffer = m_buffer->buffer;
+		m_device_address    = graphics.device.getBufferAddress(address_info);
+		EXIT_IF(m_device_address == 0);
+	}
 
 	VkMemoryPropertyFlags properties = 0;
 	vmaGetAllocationMemoryProperties(graphics.allocator, m_buffer->memory.allocation, &properties);
@@ -122,6 +133,11 @@ vk::Buffer Buffer::Handle() const noexcept {
 	return m_buffer->buffer;
 }
 
+vk::DeviceAddress Buffer::BufferDeviceAddress() const noexcept {
+	EXIT_IF(m_device_address == 0);
+	return m_device_address;
+}
+
 bool Buffer::IsInBounds(uint64_t address, uint64_t size) const noexcept {
 	return address >= m_cpu_address && size <= m_size && address - m_cpu_address <= m_size - size;
 }
@@ -141,6 +157,15 @@ void Buffer::Flush(uint64_t offset, uint64_t size) {
 	}
 }
 
+void Buffer::Invalidate(uint64_t offset, uint64_t size) {
+	EXIT_IF(m_usage != MemoryUsage::Download || offset > m_size || size > m_size - offset);
+	if (!m_is_coherent && size != 0) {
+		const auto result =
+		    vmaInvalidateAllocation(m_graphics->allocator, m_buffer->memory.allocation, offset, size);
+		EXIT_NOT_IMPLEMENTED(static_cast<vk::Result>(result) != vk::Result::eSuccess);
+	}
+}
+
 vk::BufferMemoryBarrier Buffer::Barrier(uint64_t offset, uint64_t size, vk::AccessFlags source,
                                         vk::AccessFlags destination) const {
 	if (Handle() == nullptr || size == 0 || offset > m_size || size > m_size - offset) {
@@ -310,16 +335,6 @@ void StreamBuffer::Commit() {
 	watch.tick        = tick;
 }
 
-void StreamBuffer::Invalidate(uint64_t offset, uint64_t size) {
-	EXIT_IF(Usage() != MemoryUsage::Download || offset > Size() || size > Size() - offset);
-	if (IsCoherent() || size == 0) {
-		return;
-	}
-	const auto result = vmaInvalidateAllocation(Graphics().allocator,
-	                                            NativeBuffer().memory.allocation, offset, size);
-	EXIT_NOT_IMPLEMENTED(static_cast<vk::Result>(result) != vk::Result::eSuccess);
-}
-
 uint64_t StreamBuffer::Copy(const void* source, uint64_t size, uint64_t alignment) {
 	EXIT_IF(source == nullptr);
 	const auto [data, offset] = Map(size, alignment);

src/graphics/host_gpu/renderer/cache/streamBuffer.h

@@ -48,12 +48,14 @@ public:
 	[[nodiscard]] bool               IsCoherent() const noexcept { return m_is_coherent; }
 	[[nodiscard]] MemoryUsage        Usage() const noexcept { return m_usage; }
 	[[nodiscard]] uint64_t           CpuAddress() const noexcept { return m_cpu_address; }
+	[[nodiscard]] vk::DeviceAddress BufferDeviceAddress() const noexcept;
 	[[nodiscard]] uint64_t           Offset(uint64_t address) const noexcept {
 		return address - m_cpu_address;
 	}
 	[[nodiscard]] bool IsInBounds(uint64_t address, uint64_t size) const noexcept;
 	void               Write(uint64_t offset, const void* source, uint64_t size);
 	void               Flush(uint64_t offset, uint64_t size);
+	void               Invalidate(uint64_t offset, uint64_t size);
 	void CopyFrom(CommandBuffer& command, const Buffer& source, uint64_t source_offset,
 	              uint64_t destination_offset, uint64_t size,
 	              vk::AccessFlags source_before      = vk::AccessFlagBits::eMemoryWrite,
@@ -66,8 +68,8 @@ public:
 	void Fill(uint64_t offset, uint64_t size, uint32_t value);
 
 	// BufferCache state lives directly on the resource.
-	bool     is_deleted         = false;
-	uint64_t tick_accessed_last = 0;
+	bool   is_deleted = false;
+	size_t lru_id     = 0;
 
 protected:
 	[[nodiscard]] GraphicContext&   Graphics() const noexcept { return *m_graphics; }
@@ -84,6 +86,7 @@ private:
 	MemoryUsage                   m_usage       = MemoryUsage::DeviceLocal;
 	uint64_t                      m_cpu_address = 0;
 	uint64_t                      m_size        = 0;
+	vk::DeviceAddress             m_device_address = 0;
 	std::unique_ptr<VulkanBuffer> m_buffer;
 	std::span<uint8_t>            m_mapped;
 	bool                          m_is_coherent = false;
@@ -97,9 +100,6 @@ public:
 	[[nodiscard]] std::pair<uint8_t*, uint64_t> Map(uint64_t size, uint64_t alignment = 0,
 	                                                bool allow_wait = true);
 	void                                        Commit();
-	// Download mappings become visible to the CPU only after their GPU completion tick is free.
-	// Call this from the scheduler's deferred completion operation before reading Mapped().
-	void                   Invalidate(uint64_t offset, uint64_t size);
 	[[nodiscard]] uint64_t Copy(const void* source, uint64_t size, uint64_t alignment = 0);
 
 private:

src/graphics/host_gpu/renderer/cache/textureCache.cpp

@@ -1209,7 +1209,7 @@ ImageId TextureCache::FindImageFromRange(uint64_t address, uint64_t size, bool e
 		if (ensure_valid && owner->depth_id) {
 			owner = m_slot_images.try_get(owner->depth_id);
 		}
-		if (owner == nullptr || (ensure_valid && !owner->SafeToDownload())) {
+		if (owner == nullptr || (ensure_valid && !SafeToDownload(*owner))) {
 			continue;
 		}
 		matches.push_back(id);
@@ -1580,7 +1580,7 @@ bool BufferCache::SynchronizeBufferFromImage(Buffer& buffer, uint64_t vaddr, uin
 		if (owner->depth_id) {
 			owner = m_texture_cache.m_slot_images.try_get(owner->depth_id);
 		}
-		if (owner != nullptr && owner->SafeToDownload()) {
+		if (owner != nullptr && m_texture_cache.SafeToDownload(*owner)) {
 			matches.push_back(id);
 		}
 	}
@@ -1871,9 +1871,6 @@ void TextureCache::UnmapMemory(uint64_t address, uint64_t size) {
 		metadata = m_surface_metas.erase(metadata);
 	}
 	auto images = FindImagesInRegion(address, size, false);
-	if (!images.empty()) {
-		m_scheduler.Finish();
-	}
 	for (const auto id: images) {
 		auto owner = m_slot_images.try_get(id);
 		if (owner == nullptr) {

src/graphics/host_gpu/renderer/colorRenderTarget.cpp

@@ -14,6 +14,7 @@
 #include "graphics/host_gpu/vulkanCommon.h"
 
 #include <algorithm>
+#include <array>
 #include <atomic>
 
 namespace Libs::Graphics {
@@ -43,7 +44,7 @@ static void ResolveDccClearInfo(RenderColorInfo& info, vk::Format format, bool h
 }
 
 // NOLINTNEXTLINE(readability-function-cognitive-complexity)
-void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
+void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, CommandBuffer& buffer,
                                               RenderColorInfo& r,
                                               uint32_t         render_target_slice_offset,
                                               uint32_t render_target_slot, bool ignore_target_mask,
@@ -143,12 +144,23 @@ void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandB
 	uint32_t   pitch  = 0;
 	uint64_t   size   = 0;
 	bool       tile   = false;
-	const bool volume = rt.attrib3.dimension == 2;
-	if (rt.attrib3.dimension != 1 && !volume) {
+	static constexpr std::array image_types {Prospero::ImageType::kColor1D,
+	                                         Prospero::ImageType::kColor2D,
+	                                         Prospero::ImageType::kColor3D};
+	if (rt.attrib3.dimension >= image_types.size()) {
 		EXIT("unsupported render-target dimension: %u\n", rt.attrib3.dimension);
 	}
+	const auto image_type = image_types[rt.attrib3.dimension];
+	const bool is_1d      = image_type == Prospero::ImageType::kColor1D;
+	const bool volume     = image_type == Prospero::ImageType::kColor3D;
+	if (is_1d && rt.attrib2.height != 0) {
+		EXIT("1D render target has nonzero height: %u\n", rt.attrib2.height);
+	}
 	if (!volume && rt.attrib3.depth != 0) {
-		EXIT("2D render target has nonzero depth: %u\n", rt.attrib3.depth);
+		EXIT("non-3D render target has nonzero depth: %u\n", rt.attrib3.depth);
+	}
+	if (is_1d && samples != 1) {
+		EXIT("multisampled 1D render targets are unsupported\n");
 	}
 	if (volume && samples != 1) {
 		EXIT("multisampled 3D render targets are unsupported\n");
@@ -207,8 +219,9 @@ void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandB
 		     rt.attrib.num_fragments, bytes_per_element, rt.cmask.addr, rt.fmask.addr,
 		     rt.dcc_addr.addr);
 	}
-	if ((standard64 || depth_tile) && (rt.attrib3.dimension != 1 || rt.attrib3.depth != 0 ||
-	                                   view.base_layer != 0 || view.image_layers != 1)) {
+	if ((standard64 || depth_tile) &&
+	    (rt.attrib3.dimension != 1 || rt.attrib3.depth != 0 ||
+	     (depth_tile && (view.base_layer != 0 || view.image_layers != 1)))) {
 		EXIT("unsupported 64KB texture-tiled render-target view: dimension=%u depth=%u"
 		     " layer=%u/%u\n",
 		     rt.attrib3.dimension, rt.attrib3.depth, view.base_layer, view.image_layers);
@@ -325,7 +338,7 @@ void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandB
 	desc.info.data         = {rt.base.addr, backing_size};
 	desc.info.pixel_format = target_format.format;
 	desc.info.guest_format = transfer_format;
-	desc.info.type         = volume ? Prospero::ImageType::kColor3D : Prospero::ImageType::kColor2D;
+	desc.info.type         = image_type;
 	desc.info.extent       = {width, height, depth};
 	desc.info.resources    = {levels, volume ? 1u : view.image_layers};
 	desc.info.pitch        = pitch;
@@ -364,8 +377,13 @@ void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandB
 		};
 	}
 	desc.view_info.format = target_format.format;
-	desc.view_info.type =
-	    view.layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray;
+	if (is_1d) {
+		desc.view_info.type =
+		    view.layer_count == 1 ? vk::ImageViewType::e1D : vk::ImageViewType::e1DArray;
+	} else {
+		desc.view_info.type =
+		    view.layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray;
+	}
 	desc.view_info.aspect      = vk::ImageAspectFlagBits::eColor;
 	desc.view_info.base_level  = rt.view.current_mip_level;
 	desc.view_info.level_count = 1;

src/graphics/host_gpu/renderer/colorRenderTarget.h

@@ -10,8 +10,6 @@
 
 namespace Libs::Graphics {
 
-class RenderCommandBuffer;
-
 enum class RenderColorType {
 	NoColorOutput,
 	RenderTexture,

src/graphics/host_gpu/renderer/commandScheduler.cpp

@@ -1,96 +1,97 @@
 #include "graphics/host_gpu/renderer/commandScheduler.h"
 
 #include "common/assert.h"
+#include "common/logging/log.h"
 #include "graphics/host_gpu/graphicContext.h"
 
 #include <algorithm>
+#include <cstdio>
+#include <optional>
 
 namespace Libs::Graphics {
 
 static thread_local CommandScheduler* g_deferred_callback_scheduler = nullptr;
 
-void CommandSlot::Reset() {
-	EXIT_IF(buffer == nullptr);
-	const auto result = buffer.reset(vk::CommandBufferResetFlagBits::eReleaseResources);
-	if (result != vk::Result::eSuccess) {
-		EXIT("failed to reset Vulkan command buffer: %s (%d)\n", VulkanToString(result).c_str(),
-		     static_cast<int>(result));
-	}
+namespace {
+
+void ReportVulkanFatal(const char* what, vk::Result result, uint64_t tick, uint32_t debug_op,
+                       uint64_t debug_submit, uint32_t arg0, uint32_t arg1, uint32_t arg2,
+                       uint32_t arg3, uint64_t arg4) {
+	LOGF("%s failed: %s (%d), tick=%" PRIu64 " debug_op=%u debug_submit=%" PRIu64
+	     " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
+	     what, VulkanToString(result).c_str(), static_cast<int>(result), tick, debug_op,
+	     debug_submit, arg0, arg1, arg2, arg3, arg4);
+	std::printf("%s failed: %s (%d), tick=%" PRIu64 " debug_op=%u debug_submit=%" PRIu64
+	            " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
+	            what, VulkanToString(result).c_str(), static_cast<int>(result), tick, debug_op,
+	            debug_submit, arg0, arg1, arg2, arg3, arg4);
+	std::fflush(stdout);
 }
 
-CommandScheduler::CommandPool::~CommandPool() {
-	Destroy();
-}
-
-void CommandScheduler::CommandPool::Create(GraphicContext& graphics) {
-	EXIT_IF(m_pool != nullptr || m_graphics != nullptr ||
-	        graphics.queue_family == static_cast<uint32_t>(-1));
-	m_graphics = &graphics;
+} // namespace
 
+CommandScheduler::CommandPool::CommandPool(GraphicContext& graphics, MasterSemaphore& master)
+    : m_graphics(graphics), m_master(master) {
+	EXIT_IF(graphics.queue_family == static_cast<uint32_t>(-1));
 	vk::CommandPoolCreateInfo create {};
 	create.sType            = vk::StructureType::eCommandPoolCreateInfo;
 	create.queueFamilyIndex = graphics.queue_family;
-	create.flags            = vk::CommandPoolCreateFlagBits::eResetCommandBuffer;
+	create.flags            = vk::CommandPoolCreateFlagBits::eTransient |
+	                          vk::CommandPoolCreateFlagBits::eResetCommandBuffer;
 	const auto result       = graphics.device.createCommandPool(&create, nullptr, &m_pool);
 	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess || m_pool == nullptr);
 }
 
-CommandSlot* CommandScheduler::CommandPool::CreateSlot() {
-	EXIT_IF(m_graphics == nullptr);
-	auto& graphics = *m_graphics;
+CommandScheduler::CommandPool::~CommandPool() {
+	m_graphics.device.destroyCommandPool(m_pool, nullptr);
+}
+
+size_t CommandScheduler::CommandPool::Grow() {
+	const auto first = m_ticks.size();
+	m_ticks.resize(first + GrowStep);
+	m_buffers.resize(first + GrowStep);
 
 	vk::CommandBufferAllocateInfo allocate {};
 	allocate.sType              = vk::StructureType::eCommandBufferAllocateInfo;
 	allocate.commandPool        = m_pool;
 	allocate.level              = vk::CommandBufferLevel::ePrimary;
-	allocate.commandBufferCount = 1;
-	vk::CommandBuffer buffer    = nullptr;
-	EXIT_IF(graphics.device.allocateCommandBuffers(&allocate, &buffer) != vk::Result::eSuccess);
-
-	vk::FenceCreateInfo fence_create {};
-	fence_create.sType = vk::StructureType::eFenceCreateInfo;
-	fence_create.flags = vk::FenceCreateFlagBits::eSignaled;
-	vk::Fence fence    = nullptr;
-	if (graphics.device.createFence(&fence_create, nullptr, &fence) != vk::Result::eSuccess) {
-		graphics.device.freeCommandBuffers(m_pool, 1, &buffer);
-		EXIT("failed to create command-buffer fence\n");
-	}
-
-	auto& slot      = m_slots.emplace_back();
-	slot.pool_mutex = &m_mutex;
-	slot.id         = static_cast<uint32_t>(m_slots.size() - 1);
-	slot.buffer     = buffer;
-	slot.fence      = fence;
-	return &slot;
+	allocate.commandBufferCount = static_cast<uint32_t>(GrowStep);
+	EXIT_IF(m_graphics.device.allocateCommandBuffers(&allocate, m_buffers.data() + first) !=
+	        vk::Result::eSuccess);
+	return first;
 }
 
-CommandSlot* CommandScheduler::CommandPool::Allocate(GraphicContext& graphics) {
-	Common::LockGuard lock(m_mutex);
-	if (m_pool == nullptr) {
-		Create(graphics);
-	}
-	EXIT_IF(m_graphics != &graphics);
-	auto  found = std::ranges::find_if(m_slots, [](const auto& slot) { return !slot.busy; });
-	auto* slot  = found != m_slots.end() ? &*found : CreateSlot();
-	slot->busy  = true;
-	slot->Reset();
-	return slot;
-}
+vk::CommandBuffer CommandScheduler::CommandPool::Commit() {
+	auto       gpu_tick = m_master.KnownGpuTick();
+	const auto search   = [this, &gpu_tick](size_t begin, size_t end) -> std::optional<size_t> {
+		for (size_t index = begin; index < end; ++index) {
+			if (gpu_tick >= m_ticks[index]) {
+				m_ticks[index] = m_master.CurrentTick();
+				return index;
+			}
+		}
+		return std::nullopt;
+	};
 
-void CommandScheduler::CommandPool::Destroy() {
-	Common::LockGuard lock(m_mutex);
-	if (m_pool == nullptr) {
-		return;
+	auto found = search(m_hint, m_ticks.size());
+	if (!found) {
+		m_master.Refresh();
+		gpu_tick = m_master.KnownGpuTick();
+		found    = search(m_hint, m_ticks.size());
 	}
-	EXIT_IF(std::ranges::any_of(m_slots, [](const auto& slot) { return slot.busy; }));
-	EXIT_IF(m_graphics == nullptr);
-	for (const auto& slot: m_slots) {
-		m_graphics->device.destroyFence(slot.fence, nullptr);
+	if (!found) {
+		found = search(0, m_hint);
 	}
-	m_graphics->device.destroyCommandPool(m_pool, nullptr);
-	m_slots.clear();
-	m_pool     = nullptr;
-	m_graphics = nullptr;
+	if (!found) {
+		found           = Grow();
+		m_ticks[*found] = m_master.CurrentTick();
+	}
+
+	m_hint = *found + 1;
+	if (m_hint == m_ticks.size()) {
+		m_hint = 0;
+	}
+	return m_buffers[*found];
 }
 
 bool CommandScheduler::InDeferredOperation() noexcept {
@@ -99,10 +100,8 @@ bool CommandScheduler::InDeferredOperation() noexcept {
 
 CommandScheduler::CommandScheduler(RenderContext& context, GraphicContext& graphics)
     : m_master(graphics), m_context(context), m_graphics(graphics),
-      m_priority_thread([this](std::stop_token stop) { PriorityOperationsThread(stop); }) {
-	m_buffers.reserve(CommandBufferGrowStep);
-	m_buffer_ticks.reserve(CommandBufferGrowStep);
-}
+      m_command_pool(graphics, m_master), m_command(*this),
+      m_priority_thread([this](std::stop_token stop) { PriorityOperationsThread(stop); }) {}
 
 CommandScheduler::~CommandScheduler() {
 	Shutdown();
@@ -128,9 +127,11 @@ void CommandScheduler::Shutdown() {
 		}
 		m_operation_state = OperationState::Draining;
 	}
-	if (Active() && m_recording) {
-		Finish();
+	if (!m_command.IsInvalid()) {
+		Submit();
 	}
+	m_master.Wait(CurrentTick() - 1);
+	PopPendingOperations();
 	DrainPriorityOperations();
 	m_priority_thread.request_stop();
 	m_operation_available.notify_all();
@@ -156,14 +157,10 @@ void CommandScheduler::Begin(HW::Context& registers, HW::UserConfig& user_config
 	m_user_config = &user_config;
 	m_shaders     = &shaders;
 
-	if (!Active()) {
-		m_current = static_cast<int>(GrowCommandBuffers());
-	}
-
-	BindCurrent();
-	if (!m_recording) {
-		Current().Begin();
-		m_recording = true;
+	if (m_command.IsInvalid()) {
+		BeginNext();
+	} else {
+		BindCurrent();
 	}
 }
 
@@ -172,7 +169,7 @@ void CommandScheduler::BeginRendering(const RenderState& state) {
 }
 
 void CommandScheduler::EndRendering() {
-	if (Active() && m_recording) {
+	if (Active() && !m_command.IsInvalid()) {
 		Current().EndRendering();
 	}
 }
@@ -183,60 +180,40 @@ void CommandScheduler::Flush() {
 }
 
 void CommandScheduler::Flush(SubmitInfo& submit) {
-	SubmitCurrent(submit);
+	Submit(submit);
 	BeginNext();
 }
 
-CommandBuffer& CommandScheduler::FlushAndGetSubmitted() {
-	SubmitInfo submit;
-	auto&      submitted = SubmitCurrent(submit);
+void CommandScheduler::FlushAndWait() {
+	const auto tick = Submit();
+	m_master.Wait(tick);
 	BeginNext();
-	return submitted;
 }
 
 void CommandScheduler::Finish() {
 	CheckActive();
-	const auto tick = CurrentTick();
-	if (m_recording) {
-		SubmitInfo submit;
-		SubmitCurrent(submit);
+	if (!m_command.IsInvalid()) {
+		Submit();
 	}
-	for (auto& buffer: m_buffers) {
-		buffer->WaitForFenceAndReset();
-	}
-	m_master.Wait(tick);
+	m_master.Wait(CurrentTick() - 1);
+	BeginNext();
 	PopPendingOperations();
-	BindCurrent();
-	Current().Begin();
-	m_recording = true;
-}
-
-void CommandScheduler::FinishCurrent() {
-	SubmitInfo submit;
-	auto&      submitted = SubmitCurrent(submit);
-	submitted.WaitForFenceAndReset();
-	m_master.Refresh();
-	PopPendingOperations();
-	submitted.Begin();
-	m_recording = true;
 }
 
 void CommandScheduler::Wait(uint64_t tick) {
-	CheckActive();
 	EXIT_IF(tick > CurrentTick());
-	if (tick >= CurrentTick()) {
+	if (tick == CurrentTick()) {
+		CheckActive();
 		// A stream-buffer wrap can wait while a draw is being prepared through a reference to
-		// Current(). Recycle the same command object so that reference remains valid. Deferred
+		// Current(). The wrapper stays stable while its pooled Vulkan buffer is retired. Deferred
 		// resources are released only at the next GPU operation boundary.
-		SubmitInfo submit;
-		auto&      submitted = SubmitCurrent(submit);
-		submitted.WaitForFenceAndReset();
-		m_master.Refresh();
-		submitted.Begin();
-		m_recording = true;
-		return;
+		const auto submitted_tick = Submit();
+		EXIT_IF(submitted_tick != tick);
+		m_master.Wait(tick);
+		BeginNext();
+	} else {
+		m_master.Wait(tick);
 	}
-	m_master.Wait(tick);
 }
 
 void CommandScheduler::PopPendingOperations() {
@@ -365,83 +342,81 @@ bool CommandScheduler::IsFree(uint64_t tick) {
 	return m_master.IsFree(tick);
 }
 
-CommandSlot* CommandScheduler::AllocateCommandBuffer() {
-	return m_command_pool.Allocate(m_graphics);
-}
-
-uint64_t CommandScheduler::NextSubmitSequence() noexcept {
-	return m_submit_sequence.fetch_add(1, std::memory_order_relaxed) + 1;
-}
-
 void CommandScheduler::CheckActive() const {
-	EXIT_IF(!Active() || static_cast<size_t>(m_current) >= m_buffers.size());
+	EXIT_IF(!Active());
 }
 
-RenderCommandBuffer& CommandScheduler::Current() const {
+CommandBuffer& CommandScheduler::Current() {
 	CheckActive();
-	EXIT_IF(m_buffers[m_current] == nullptr);
-	return *m_buffers[m_current];
+	return m_command;
 }
 
-void CommandScheduler::BindCurrent() const {
+void CommandScheduler::BindCurrent() {
 	EXIT_IF(m_registers == nullptr || m_user_config == nullptr || m_shaders == nullptr);
-	Current().Bind(*m_registers, *m_user_config, *m_shaders);
+	m_command.Bind(*m_registers, *m_user_config, *m_shaders);
 }
 
-CommandBuffer& CommandScheduler::SubmitCurrent(SubmitInfo& submit) {
-	CheckActive();
-	EXIT_IF(!m_recording);
-	auto& submitted = Current();
-	submitted.End();
-	const auto signal_tick = m_master.NextTick();
-	submit.AddSignal(m_master.Handle(), signal_tick);
-	submitted.Execute(submit);
-	m_buffer_ticks[static_cast<size_t>(m_current)] = signal_tick;
-	m_recording                                    = false;
-	return submitted;
+CommandBuffer& CommandScheduler::BeginCommand() {
+	EXIT_IF(!m_command.IsInvalid());
+	m_command.m_buffer = m_command_pool.Commit();
+	m_command.Begin();
+	return m_command;
 }
 
-int CommandScheduler::FindReusableBuffer(uint64_t gpu_tick) const {
-	EXIT_IF(m_buffer_ticks.size() != m_buffers.size());
-	const auto buffer_count = m_buffers.size();
-	for (size_t offset = 1; offset <= buffer_count; ++offset) {
-		const auto candidate = (static_cast<size_t>(m_current) + offset) % buffer_count;
-		if (gpu_tick >= m_buffer_ticks[candidate]) {
-			return static_cast<int>(candidate);
-		}
+uint64_t CommandScheduler::Submit(SubmitInfo submit) {
+	EXIT_IF(m_command.IsInvalid());
+	EXIT_IF(submit.num_wait_semaphores > SubmitInfo::MaxSemaphores ||
+	        submit.num_signal_semaphores >= SubmitInfo::MaxSemaphores);
+
+	m_command.End();
+	const auto buffer   = m_command.m_buffer;
+	auto&      graphics = m_graphics;
+	EXIT_IF(graphics.queue == nullptr);
+
+	vk::Result result;
+	uint64_t   tick;
+	{
+		Common::LockGuard lock(graphics.queue_mutex);
+		tick = m_master.NextTick();
+		submit.AddSignal(m_master.Handle(), tick);
+
+		vk::TimelineSemaphoreSubmitInfo timeline_info {};
+		timeline_info.sType                     = vk::StructureType::eTimelineSemaphoreSubmitInfo;
+		timeline_info.waitSemaphoreValueCount   = submit.num_wait_semaphores;
+		timeline_info.pWaitSemaphoreValues      = submit.wait_ticks.data();
+		timeline_info.signalSemaphoreValueCount = submit.num_signal_semaphores;
+		timeline_info.pSignalSemaphoreValues    = submit.signal_ticks.data();
+
+		vk::SubmitInfo submit_info {};
+		submit_info.sType                = vk::StructureType::eSubmitInfo;
+		submit_info.pNext                = &timeline_info;
+		submit_info.waitSemaphoreCount   = submit.num_wait_semaphores;
+		submit_info.pWaitSemaphores      = submit.wait_semaphores.data();
+		submit_info.pWaitDstStageMask    = submit.wait_stages.data();
+		submit_info.commandBufferCount   = 1;
+		submit_info.pCommandBuffers      = &buffer;
+		submit_info.signalSemaphoreCount = submit.num_signal_semaphores;
+		submit_info.pSignalSemaphores    = submit.signal_semaphores.data();
+
+		result = graphics.queue.submit(1, &submit_info, nullptr);
 	}
-	return -1;
-}
 
-size_t CommandScheduler::GrowCommandBuffers() {
-	const auto first = m_buffers.size();
-	const auto end   = first + CommandBufferGrowStep;
-	m_buffers.reserve(end);
-	m_buffer_ticks.reserve(end);
-	for (size_t i = first; i < end; ++i) {
-		m_buffers.emplace_back(std::make_unique<RenderCommandBuffer>(*this));
-		m_buffer_ticks.push_back(0);
+	if (result != vk::Result::eSuccess) {
+		ReportVulkanFatal("vkQueueSubmit", result, tick, m_command.m_debug_op,
+		                  m_command.m_debug_submit_id, m_command.m_debug_arg0,
+		                  m_command.m_debug_arg1, m_command.m_debug_arg2, m_command.m_debug_arg3,
+		                  m_command.m_debug_arg4);
 	}
-	return first;
+	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
+
+	m_command.m_buffer = nullptr;
+	return tick;
 }
 
 void CommandScheduler::BeginNext() {
-	EXIT_IF(m_recording);
-
-	auto candidate = FindReusableBuffer(m_master.KnownGpuTick());
-	if (candidate < 0) {
-		m_master.Refresh();
-		candidate = FindReusableBuffer(m_master.KnownGpuTick());
-	}
-	if (candidate < 0) {
-		candidate = static_cast<int>(GrowCommandBuffers());
-	}
-
-	m_current = candidate;
-	Current().WaitForFenceAndReset();
+	EXIT_IF(!m_command.IsInvalid());
 	BindCurrent();
-	Current().Begin();
-	m_recording = true;
+	BeginCommand();
 }
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/commandScheduler.h

@@ -6,11 +6,7 @@
 #include "graphics/host_gpu/renderer/masterSemaphore.h"
 #include "graphics/host_gpu/renderer/render.h"
 
-#include <array>
-#include <atomic>
 #include <condition_variable>
-#include <deque>
-#include <memory>
 #include <mutex>
 
 #include <queue>
@@ -20,16 +16,6 @@
 
 namespace Libs::Graphics {
 
-struct CommandSlot {
-	Common::Mutex*    pool_mutex = nullptr;
-	uint32_t          id         = 0;
-	vk::CommandBuffer buffer     = nullptr;
-	vk::Fence         fence      = nullptr;
-	bool              busy       = false;
-
-	void Reset();
-};
-
 class CommandScheduler {
 public:
 	CommandScheduler(RenderContext& context, GraphicContext& graphics);
@@ -41,9 +27,10 @@ public:
 	void           EndRendering();
 	void           Flush();
 	void           Flush(SubmitInfo& submit);
-	CommandBuffer& FlushAndGetSubmitted();
+	void           FlushAndWait();
 	void           Finish();
-	void           FinishCurrent();
+	CommandBuffer& BeginCommand();
+	uint64_t       Submit(SubmitInfo submit = {});
 	// Deferred callbacks can observe an externally owned drain, but cannot initiate shutdown:
 	// the priority runner cannot join itself.
 	void                      Shutdown();
@@ -55,9 +42,9 @@ public:
 	void                      DeferPriorityOperation(Common::UniqueFunction<void>&& operation);
 	[[nodiscard]] static bool InDeferredOperation() noexcept;
 
-	[[nodiscard]] bool             Active() const noexcept { return m_current >= 0; }
+	[[nodiscard]] bool             Active() const noexcept { return m_registers != nullptr; }
 	void                           CheckActive() const;
-	RenderCommandBuffer&           Current() const;
+	CommandBuffer&                 Current();
 	[[nodiscard]] uint64_t         CurrentTick() const noexcept { return m_master.CurrentTick(); }
 	[[nodiscard]] bool             IsFree(uint64_t tick);
 	[[nodiscard]] MasterSemaphore& GetMasterSemaphore() noexcept { return m_master; }
@@ -65,25 +52,25 @@ public:
 	[[nodiscard]] GraphicContext&  Graphics() const noexcept { return m_graphics; }
 
 private:
-	static constexpr size_t CommandBufferGrowStep = 4;
-
 	class CommandPool {
 	public:
-		CommandPool() = default;
+		CommandPool(GraphicContext& graphics, MasterSemaphore& master);
 		~CommandPool();
 		KYTY_CLASS_NO_COPY(CommandPool);
 
-		CommandSlot* Allocate(GraphicContext& graphics);
+		vk::CommandBuffer Commit();
 
 	private:
-		void         Create(GraphicContext& graphics);
-		CommandSlot* CreateSlot();
-		void         Destroy();
+		static constexpr size_t GrowStep = 4;
 
-		GraphicContext*         m_graphics = nullptr;
-		Common::Mutex           m_mutex;
-		vk::CommandPool         m_pool = nullptr;
-		std::deque<CommandSlot> m_slots;
+		size_t Grow();
+
+		GraphicContext&                m_graphics;
+		MasterSemaphore&               m_master;
+		vk::CommandPool                m_pool = nullptr;
+		std::vector<vk::CommandBuffer> m_buffers;
+		std::vector<uint64_t>          m_ticks;
+		size_t                         m_hint = 0;
 	};
 
 	enum class OperationState { Open, Draining, Closed };
@@ -93,39 +80,28 @@ private:
 		uint64_t                     tick = 0;
 	};
 
-	void                       BindCurrent() const;
-	CommandBuffer&             SubmitCurrent(SubmitInfo& submit);
-	void                       BeginNext();
-	void                       PopPendingOperations(bool refresh_gpu_tick);
-	[[nodiscard]] int          FindReusableBuffer(uint64_t gpu_tick) const;
-	[[nodiscard]] size_t       GrowCommandBuffers();
-	void                       PriorityOperationsThread(std::stop_token stop);
-	void                       RunOperation(Common::UniqueFunction<void>&& operation);
-	[[nodiscard]] CommandSlot* AllocateCommandBuffer();
-	[[nodiscard]] uint64_t     NextSubmitSequence() noexcept;
+	void BindCurrent();
+	void BeginNext();
+	void PopPendingOperations(bool refresh_gpu_tick);
+	void PriorityOperationsThread(std::stop_token stop);
+	void RunOperation(Common::UniqueFunction<void>&& operation);
 
-	MasterSemaphore                                   m_master;
-	RenderContext&                                    m_context;
-	GraphicContext&                                   m_graphics;
-	CommandPool                                       m_command_pool;
-	std::vector<std::unique_ptr<RenderCommandBuffer>> m_buffers;
-	std::vector<uint64_t>                             m_buffer_ticks;
-	std::queue<PendingOperation>                      m_pending_operations;
-	std::queue<PendingOperation>                      m_priority_operations;
-	std::mutex                                        m_operation_mutex;
-	std::condition_variable                           m_operation_available;
-	std::jthread                                      m_priority_thread;
-	bool                                              m_priority_active      = false;
-	uint64_t                                          m_priority_active_tick = 0;
-	OperationState                                    m_operation_state      = OperationState::Open;
-	int                                               m_current              = -1;
-	bool                                              m_recording            = false;
-	HW::Context*                                      m_registers            = nullptr;
-	HW::UserConfig*                                   m_user_config          = nullptr;
-	HW::Shader*                                       m_shaders              = nullptr;
-	std::atomic<uint64_t>                             m_submit_sequence      = 0;
-
-	friend class CommandBuffer;
+	MasterSemaphore              m_master;
+	RenderContext&               m_context;
+	GraphicContext&              m_graphics;
+	CommandPool                  m_command_pool;
+	CommandBuffer                m_command;
+	std::queue<PendingOperation> m_pending_operations;
+	std::queue<PendingOperation> m_priority_operations;
+	std::mutex                   m_operation_mutex;
+	std::condition_variable      m_operation_available;
+	std::jthread                 m_priority_thread;
+	bool                         m_priority_active      = false;
+	uint64_t                     m_priority_active_tick = 0;
+	OperationState               m_operation_state      = OperationState::Open;
+	HW::Context*                 m_registers            = nullptr;
+	HW::UserConfig*              m_user_config          = nullptr;
+	HW::Shader*                  m_shaders              = nullptr;
 };
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/context.cpp

@@ -1,7 +1,5 @@
 #include "common/assert.h"
 #include "common/common.h"
-#include "common/emulatorConfig.h"
-#include "common/logging/log.h"
 #include "common/profiler.h"
 #include "common/threads.h"
 #include "graphics/host_gpu/graphicContext.h"
@@ -16,70 +14,29 @@
 
 #include <algorithm>
 #include <bit>
-#include <cstdio>
 #include <cstring>
 namespace Libs::Graphics {
 
-namespace {
-
-void ReportVulkanFatal(const char* what, vk::Result result, uint32_t slot, uint64_t submit_seq,
-                       uint32_t debug_op, uint64_t debug_submit, uint32_t arg0, uint32_t arg1,
-                       uint32_t arg2, uint32_t arg3, uint64_t arg4) {
-	LOGF("%s failed: %s (%d), slot=%u submit_seq=%" PRIu64 " debug_op=%u debug_submit=%" PRIu64
-	     " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
-	     what, VulkanToString(result).c_str(), static_cast<int>(result), slot, submit_seq, debug_op,
-	     debug_submit, arg0, arg1, arg2, arg3, arg4);
-	std::printf("%s failed: %s (%d), slot=%u submit_seq=%" PRIu64
-	            " debug_op=%u debug_submit=%" PRIu64 " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
-	            what, VulkanToString(result).c_str(), static_cast<int>(result), slot, submit_seq,
-	            debug_op, debug_submit, arg0, arg1, arg2, arg3, arg4);
-	std::fflush(stdout);
-}
-
-} // namespace
-
 CommandBuffer::CommandBuffer(CommandScheduler& scheduler)
-    : m_context(scheduler.Context()), m_scheduler(scheduler), m_graphics(scheduler.Graphics()),
-      m_slot(scheduler.AllocateCommandBuffer()) {}
-
-CommandBuffer::~CommandBuffer() {
-	Release();
-}
+    : m_context(scheduler.Context()), m_graphics(scheduler.Graphics()) {}
 
 bool CommandBuffer::IsInvalid() const {
-	return m_slot == nullptr;
+	return m_buffer == nullptr;
 }
 
 vk::CommandBuffer CommandBuffer::Handle() const {
 	EXIT_IF(IsInvalid());
-
-	const auto handle = m_slot->buffer;
-	EXIT_IF(handle == nullptr);
-	return handle;
+	return m_buffer;
 }
 
-void CommandBuffer::Release() {
-	EXIT_IF(IsInvalid());
-
-	Common::LockGuard lock(*m_slot->pool_mutex);
-
-	WaitForFence();
-
-	m_slot->busy = false;
-	m_slot->Reset();
-	m_slot = nullptr;
-
-	EXIT_NOT_IMPLEMENTED(!IsInvalid());
-}
-
-void CommandBuffer::Begin() const {
-	EXIT_IF(m_rendering);
+void CommandBuffer::Begin() {
+	EXIT_IF(m_rendering || IsInvalid());
 	auto buffer = Handle();
 
 	vk::CommandBufferBeginInfo begin_info {};
 	begin_info.sType            = vk::StructureType::eCommandBufferBeginInfo;
 	begin_info.pNext            = nullptr;
-	begin_info.flags            = {};
+	begin_info.flags            = vk::CommandBufferUsageFlagBits::eOneTimeSubmit;
 	begin_info.pInheritanceInfo = nullptr;
 
 	auto result = buffer.begin(&begin_info);
@@ -107,106 +64,6 @@ void CommandBuffer::SetDebugInfo(uint32_t op, uint64_t submit_id, uint32_t arg0,
 	m_debug_arg4      = arg4;
 }
 
-void CommandBuffer::Execute(const SubmitInfo& submit) {
-	EXIT_IF(IsInvalid());
-	EXIT_IF(m_execute);
-	EXIT_IF(submit.num_wait_semaphores > SubmitInfo::MaxSemaphores ||
-	        submit.num_signal_semaphores > SubmitInfo::MaxSemaphores);
-
-	auto buffer = Handle();
-	auto fence  = m_slot->fence;
-
-	vk::TimelineSemaphoreSubmitInfo timeline_info {};
-	timeline_info.sType                     = vk::StructureType::eTimelineSemaphoreSubmitInfo;
-	timeline_info.waitSemaphoreValueCount   = submit.num_wait_semaphores;
-	timeline_info.pWaitSemaphoreValues      = submit.wait_ticks.data();
-	timeline_info.signalSemaphoreValueCount = submit.num_signal_semaphores;
-	timeline_info.pSignalSemaphoreValues    = submit.signal_ticks.data();
-
-	vk::SubmitInfo submit_info {};
-	submit_info.sType                = vk::StructureType::eSubmitInfo;
-	submit_info.pNext                = &timeline_info;
-	submit_info.waitSemaphoreCount   = submit.num_wait_semaphores;
-	submit_info.pWaitSemaphores      = submit.wait_semaphores.data();
-	submit_info.pWaitDstStageMask    = submit.wait_stages.data();
-	submit_info.commandBufferCount   = 1;
-	submit_info.pCommandBuffers      = &buffer;
-	submit_info.signalSemaphoreCount = submit.num_signal_semaphores;
-	submit_info.pSignalSemaphores    = submit.signal_semaphores.data();
-
-	auto& graphics = m_graphics;
-	EXIT_IF(graphics.queue == nullptr);
-
-	auto result = graphics.device.resetFences(1, &fence);
-	if (result != vk::Result::eSuccess) {
-		ReportVulkanFatal("vkResetFences (before submit)", result, m_slot->id, m_submit_seq,
-		                  m_debug_op, m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2,
-		                  m_debug_arg3, m_debug_arg4);
-	}
-	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-
-	if (Config::GraphicsDebugDumpEnabled()) {
-		LOGF("vkQueueSubmit begin: slot=%u waits=%u signals=%u debug_op=%u debug_submit=%" PRIu64
-		     " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
-		     m_slot->id, submit.num_wait_semaphores, submit.num_signal_semaphores, m_debug_op,
-		     m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
-		     m_debug_arg4);
-	}
-
-	{
-		Common::LockGuard lock(graphics.queue_mutex);
-		m_submit_seq = m_scheduler.NextSubmitSequence();
-		result       = graphics.queue.submit(1, &submit_info, fence);
-	}
-
-	m_execute      = true;
-	m_fence_waited = false;
-
-	if (result != vk::Result::eSuccess) {
-		ReportVulkanFatal("vkQueueSubmit", result, m_slot->id, m_submit_seq, m_debug_op,
-		                  m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
-		                  m_debug_arg4);
-	}
-	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-}
-
-void CommandBuffer::WaitForFence() {
-	FinalizeFence(false);
-}
-
-void CommandBuffer::WaitForFenceOnly() {
-	EXIT_IF(IsInvalid());
-	if (!m_execute || m_fence_waited) {
-		return;
-	}
-	auto device = m_graphics.device;
-	auto result = device.waitForFences(1, &m_slot->fence, VK_TRUE, UINT64_MAX);
-	if (result != vk::Result::eSuccess) {
-		ReportVulkanFatal("vkWaitForFences", result, m_slot->id, m_submit_seq, m_debug_op,
-		                  m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
-		                  m_debug_arg4);
-	}
-	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-	m_fence_waited = true;
-}
-
-void CommandBuffer::WaitForFenceAndReset() {
-	FinalizeFence(true);
-}
-
-void CommandBuffer::FinalizeFence(bool reset_recording) {
-	const bool was_executed = m_execute;
-	WaitForFenceOnly();
-	if (was_executed) {
-		m_execute      = false;
-		m_fence_waited = false;
-		if (reset_recording) {
-			Common::LockGuard lock(*m_slot->pool_mutex);
-			m_slot->Reset();
-		}
-	}
-}
-
 void CommandBuffer::BeginRendering(const RenderState& state) const {
 	EXIT_IF(state.width == 0 || state.height == 0 || state.num_layers == 0 ||
 	        state.num_color_attachments > RENDER_COLOR_ATTACHMENTS_MAX);

src/graphics/host_gpu/renderer/debug.cpp

@@ -27,7 +27,7 @@ static bool RenderTargetMaskHasMrt(uint32_t mask) {
 	return (mask & ~0x0fu) != 0;
 }
 
-static bool RenderTargetMaskHasBoundMrt(const RenderCommandBuffer& buffer) {
+static bool RenderTargetMaskHasBoundMrt(const CommandBuffer& buffer) {
 	const auto& hw   = buffer.GetRegisters();
 	const auto  mask = hw.GetRenderTargetMask();
 
@@ -45,7 +45,7 @@ static bool RenderTargetMaskHasBoundMrt(const RenderCommandBuffer& buffer) {
 	return bound_targets > 1;
 }
 
-uint32_t render_target_first_bound_slot(const RenderCommandBuffer& buffer) {
+uint32_t render_target_first_bound_slot(const CommandBuffer& buffer) {
 	const auto& hw   = buffer.GetRegisters();
 	const auto  mask = hw.GetRenderTargetMask();
 	for (uint32_t i = 0; i < 8; i++) {
@@ -84,17 +84,8 @@ void uc_print(const char* func, const HW::UserConfig& uc) {
 }
 
 void uc_check(const HW::UserConfig& uc) {
-	const auto& ge_cntl = uc.GetGeControl();
 	const auto& user_en = uc.GetGeUserVgprEn();
 
-	if (ge_cntl.primitive_group_size > 0x0040) {
-		LOGF("\t warning: unsupported GE_CNTL primitive_group_size = 0x%04" PRIx16 "\n",
-		     ge_cntl.primitive_group_size);
-	}
-	if (ge_cntl.vertex_group_size > 0x0040) {
-		LOGF("\t warning: unsupported GE_CNTL vertex_group_size = 0x%04" PRIx16 "\n",
-		     ge_cntl.vertex_group_size);
-	}
 	EXIT_NOT_IMPLEMENTED(user_en.vgpr1 != false);
 	EXIT_NOT_IMPLEMENTED(user_en.vgpr2 != false);
 	EXIT_NOT_IMPLEMENTED(user_en.vgpr3 != false);
@@ -266,11 +257,10 @@ static void RtCheck(const HW::RenderTarget& rt) {
 				logged = true;
 			}
 		}
-		const bool native_2x_one_frag = rt.info.fmask_compression_enable &&
-		                                !rt.info.fmask_data_compression_disable &&
-		                                rt.info.cmask_fast_clear_enable && rt.fmask.addr != 0 &&
-		                                rt.cmask.addr != 0 && rt.attrib.num_samples == 1 &&
-		                                rt.attrib.num_fragments == 1;
+		const bool native_2x_one_frag =
+		    rt.info.fmask_compression_enable && !rt.info.fmask_data_compression_disable &&
+		    rt.info.cmask_fast_clear_enable && rt.fmask.addr != 0 && rt.cmask.addr != 0 &&
+		    rt.attrib.num_samples == 1 && rt.attrib.num_fragments == 1;
 		EXIT_NOT_IMPLEMENTED(rt.info.fmask_one_frag_mode && !native_2x_one_frag);
 		if (rt.info.fmask_compression_enable) {
 			EXIT_NOT_IMPLEMENTED(rt.attrib.num_samples == 0 && rt.attrib.num_fragments == 0);
@@ -889,12 +879,10 @@ void LogDrawPhase(const char* draw_name, const char* phase) {
 	}
 }
 
-void LogPipelineTrace(const char* phase, uint32_t vs_hash0, uint32_t vs_crc32, uint32_t ps_hash0,
-                      uint32_t ps_crc32) {
+void LogPipelineTrace(const char* phase, uint64_t vertex_program_id, uint64_t pixel_program_id) {
 	if (graphics_debug_dump_enabled()) {
-		LOGF("PipelineTrace: %s VS=0x%08" PRIx32 "/0x%08" PRIx32 " PS=0x%08" PRIx32 "/0x%08" PRIx32
-		     "\n",
-		     phase, vs_hash0, vs_crc32, ps_hash0, ps_crc32);
+		LOGF("PipelineTrace: %s VS=%" PRIu64 " PS=%" PRIu64 "\n", phase, vertex_program_id,
+		     pixel_program_id);
 	}
 }
 
@@ -1158,7 +1146,7 @@ ScissorRect calc_final_scissor(const HW::ScreenViewport& vp, const HW::ScanModeC
 	return ScissorRectClamp(final, extent.width, extent.height);
 }
 
-void hw_check(const RenderCommandBuffer& buffer) {
+void hw_check(const CommandBuffer& buffer) {
 	const auto& hw      = buffer.GetRegisters();
 	const auto  rt_slot = render_target_first_bound_slot(buffer);
 	const auto& rt      = hw.GetRenderTarget(rt_slot);
@@ -1231,7 +1219,7 @@ void hw_check(const RenderCommandBuffer& buffer) {
 	// EXIT_NOT_IMPLEMENTED(hw.GetStencilClearValue() != 0);
 }
 
-void hw_print(const RenderCommandBuffer& buffer) {
+void hw_print(const CommandBuffer& buffer) {
 	const auto& hw      = buffer.GetRegisters();
 	const auto  rt_slot = render_target_first_bound_slot(buffer);
 	const auto& rt      = hw.GetRenderTarget(rt_slot);

src/graphics/host_gpu/renderer/debug.h

@@ -9,7 +9,7 @@
 
 namespace Libs::Graphics {
 
-class RenderCommandBuffer;
+class CommandBuffer;
 
 namespace HW {
 class Context;
@@ -28,15 +28,15 @@ struct ScissorRect {
 };
 
 uint32_t                 render_target_mask_slot(uint32_t mask, uint32_t slot);
-uint32_t                 render_target_first_bound_slot(const RenderCommandBuffer& buffer);
+uint32_t                 render_target_first_bound_slot(const CommandBuffer& buffer);
 bool                     graphics_debug_dump_enabled();
 void                     uc_print(const char* func, const HW::UserConfig& uc);
 void                     uc_check(const HW::UserConfig& uc);
 void                     sh_print(const char* func, const HW::Shader& uc);
 std::vector<std::string> rt_print(const char* func, const HW::RenderTarget& rt);
 bool                     RenderIsColorTileModeLinear(Prospero::TileMode tile_mode);
-void                     hw_print(const RenderCommandBuffer& buffer);
-void                     hw_check(const RenderCommandBuffer& buffer);
+void                     hw_print(const CommandBuffer& buffer);
+void                     hw_check(const CommandBuffer& buffer);
 void                     LogDrawPhase(const char* draw_name, const char* phase);
 ScissorRect calc_final_scissor(const HW::ScreenViewport& vp, const HW::ScanModeControl& smc,
                                vk::Extent2D extent);

src/graphics/host_gpu/renderer/depthRenderTarget.cpp

@@ -70,8 +70,8 @@ static bool UsesStencilOpValue(uint8_t fail, uint8_t pass, uint8_t depth_fail) {
 	return fail == replace_op || pass == replace_op || depth_fail == replace_op;
 }
 
-[[nodiscard]] static vk::Format ResolveHostDepthAttachmentFormat(const RenderCommandBuffer& buffer,
-                                                                 const DepthFormatPolicy&   policy,
+[[nodiscard]] static vk::Format ResolveHostDepthAttachmentFormat(const CommandBuffer&     buffer,
+                                                                 const DepthFormatPolicy& policy,
                                                                  bool     has_stencil,
                                                                  uint32_t samples) {
 	auto&      graphics         = buffer.GetGraphics();
@@ -97,7 +97,7 @@ static bool UsesStencilOpValue(uint8_t fail, uint8_t pass, uint8_t depth_fail) {
 }
 
 // NOLINTNEXTLINE(readability-function-cognitive-complexity)
-void RenderExecutor::ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
+void RenderExecutor::ResolveRenderDepthTarget(uint64_t submit_id, CommandBuffer& buffer,
                                               RenderDepthInfo& r) {
 	KYTY_PROFILER_FUNCTION();
 	(void)submit_id;

src/graphics/host_gpu/renderer/depthRenderTarget.h

@@ -11,8 +11,6 @@
 
 namespace Libs::Graphics {
 
-class RenderCommandBuffer;
-
 inline constexpr bool depth_htile_stencil_acceleration_compatible(bool has_stencil, bool has_htile,
                                                                   bool htile_stencil_disabled) {
 	return htile_stencil_disabled || (has_stencil && has_htile);

src/graphics/host_gpu/renderer/image/image.cpp

@@ -736,15 +736,10 @@ Image::~Image() {
 	if (m_graphics == nullptr) {
 		return;
 	}
-	{
-		std::lock_guard lock(views.mutex);
-		for (auto& cached: views.views) {
-			if (cached.view != nullptr) {
-				m_graphics->device.destroyImageView(cached.view, nullptr);
-				cached.view = nullptr;
-			}
+	for (const auto& cached: views) {
+		if (cached.view != nullptr) {
+			m_graphics->device.destroyImageView(cached.view, nullptr);
 		}
-		views.views.clear();
 	}
 	if (backing.image != nullptr) {
 		m_graphics->DeleteImage(backing);

src/graphics/host_gpu/renderer/image/image.h

@@ -8,7 +8,6 @@
 
 #include <compare>
 #include <limits>
-#include <mutex>
 #include <optional>
 #include <span>
 #include <utility>
@@ -27,14 +26,6 @@ struct CachedImageView {
 	vk::ImageView view = nullptr;
 };
 
-struct ImageViewCache {
-	ImageViewCache() = default;
-	KYTY_CLASS_NO_COPY(ImageViewCache);
-
-	std::mutex                   mutex;
-	std::vector<CachedImageView> views;
-};
-
 struct ImageUsage {
 	bool texture       = false;
 	bool storage       = false;
@@ -150,7 +141,7 @@ public:
 
 	ImageInfo        info;
 	VulkanImage      backing;
-	ImageViewCache   views;
+	std::vector<CachedImageView> views;
 	ImageUsage       usage;
 	ImageBinding     binding;
 	bool             registered     = false;

src/graphics/host_gpu/renderer/image/imageView.cpp

@@ -4,8 +4,6 @@
 #include "graphics/host_gpu/graphicContext.h"
 #include "graphics/host_gpu/renderer/image/image.h"
 
-#include <mutex>
-
 namespace Libs::Graphics {
 
 namespace {
@@ -357,8 +355,7 @@ vk::ImageView Image::FindView(const ImageViewInfo& view_info) {
 		     image.layers);
 	}
 
-	std::lock_guard lock(views.mutex);
-	for (const auto& cached: views.views) {
+	for (const auto& cached: views) {
 		if (cached.info == normalized) {
 			return cached.view;
 		}
@@ -394,7 +391,7 @@ vk::ImageView Image::FindView(const ImageViewInfo& view_info) {
 		     view_info.level_count, view_info.base_layer, view_info.layer_count,
 		     static_cast<vk::ImageUsageFlags::MaskType>(view_info.usage));
 	}
-	views.views.push_back({normalized, view});
+	views.push_back({normalized, view});
 	return view;
 }
 

src/graphics/host_gpu/renderer/pipeline/descriptors.cpp

@@ -216,48 +216,6 @@ static BufferView NativeStorageBuffer(RenderContext&
 	return result;
 }
 
-static BufferView
-NativeAddressBuffer(RenderContext& context, const ShaderRecompiler::IR::AddressResource& resource,
-                    const ShaderRecompiler::IR::ResourceSnapshot::Address& address,
-                    ShaderType stage, uint32_t slot, uint32_t& address_offset, BufferId id) {
-	BufferView result;
-	address_offset = 0;
-	if (address.binding_base == 0) {
-		BindNullStorageBuffer(context, result);
-		return result;
-	}
-	if (resource.written) {
-		EXIT("writable address resources are unsupported\n");
-	}
-	const auto limit =
-	    resource.kind == ShaderRecompiler::IR::ResourceKind::Flat
-	        ? ShaderRecompiler::IR::FlatAddressWindowSize
-	        : static_cast<uint64_t>(
-	              context.GetGraphics().GetPhysicalDeviceProperties().limits.maxStorageBufferRange);
-	uint64_t   size   = 0;
-	const auto access = HostMemoryAccess::Mapped;
-	if (!HostMemoryQueryRange(address.binding_base, limit, access, size)) {
-		EXIT("address resource is not host-accessible: base=0x%016" PRIx64 "\n",
-		     address.binding_base);
-	}
-	const auto& graphics  = context.GetGraphics();
-	const auto  alignment = graphics.StorageMinAlignment();
-	auto [buffer, offset] =
-	    context.GetBufferCache().ObtainBuffer(address.binding_base, size, false, false, id);
-	const auto aligned_offset = offset - offset % alignment;
-	const auto adjustment     = offset - aligned_offset;
-	const auto max_range      = graphics.GetPhysicalDeviceProperties().limits.maxStorageBufferRange;
-	const auto visible_size   = std::min(size, static_cast<uint64_t>(max_range - adjustment));
-	address_offset            = static_cast<uint32_t>(adjustment);
-	result.buffer             = buffer->Handle();
-	result.offset             = aligned_offset;
-	result.range              = static_cast<vk::DeviceSize>(visible_size + adjustment);
-	SetVulkanObjectNameF(graphics.device, result.buffer,
-	                     "Kyty.{}.AddressBuffer[slot={} guest=0x{:016x} size=0x{:x}]",
-	                     ShaderStageResourceName(stage), slot, address.binding_base, visible_size);
-	return result;
-}
-
 static bool IsSupportedSampledColorResource(const ShaderRecompiler::IR::ImageResource& resource) {
 	bool supported_dimension = false;
 	switch (resource.dimension) {
@@ -891,7 +849,7 @@ static vk::Sampler NativeSampler(RenderContext&                       context,
 static BufferView NativeUpload(RenderContext& context, std::span<const uint32_t> data) {
 	EXIT_IF(data.empty());
 	auto& command_buffer = context.GetCommandScheduler().Current();
-	EXIT_IF(command_buffer.IsInvalid() || command_buffer.IsExecute());
+	EXIT_IF(command_buffer.IsInvalid());
 	auto&      buffer = context.GetBufferCache().GetUtilityBuffer(MemoryUsage::Stream);
 	const auto offset = buffer.Copy(data.data(), data.size_bytes(), 256);
 	return {.buffer = buffer.Handle(), .offset = offset, .range = data.size_bytes()};
@@ -1000,25 +958,6 @@ void RenderExecutor::FindBuffers(PreparedBindings& prepared) {
 		prepared.buffer_ids.push_back(cache.FindBuffer(address, size));
 	}
 
-	prepared.address_ids.clear();
-	prepared.address_ids.reserve(program.info.addresses.size());
-	for (uint32_t i = 0; i < program.info.addresses.size(); i++) {
-		const auto& resource = program.info.addresses[i];
-		const auto& address  = snapshot.addresses[i];
-		if (address.binding_base == 0) {
-			prepared.address_ids.emplace_back();
-			continue;
-		}
-		EXIT_IF(resource.written);
-		const auto limit = resource.kind == ShaderRecompiler::IR::ResourceKind::Flat
-		                       ? ShaderRecompiler::IR::FlatAddressWindowSize
-		                       : static_cast<uint64_t>(m_context.GetGraphics()
-		                                                   .GetPhysicalDeviceProperties()
-		                                                   .limits.maxStorageBufferRange);
-		uint64_t   size  = 0;
-		EXIT_IF(!HostMemoryQueryRange(address.binding_base, limit, HostMemoryAccess::Mapped, size));
-		prepared.address_ids.push_back(cache.FindBuffer(address.binding_base, size));
-	}
 }
 
 void RenderExecutor::RebindBuffers(PreparedBindings& prepared) {
@@ -1028,8 +967,7 @@ void RenderExecutor::RebindBuffers(PreparedBindings& prepared) {
 	const auto& snapshot  = *prepared.snapshot;
 	auto&       resources = prepared.resources;
 	const auto& layout    = program.bindings;
-	EXIT_IF(prepared.buffer_ids.size() != program.info.buffers.size() ||
-	        prepared.address_ids.size() != program.info.addresses.size());
+	EXIT_IF(prepared.buffer_ids.size() != program.info.buffers.size());
 
 	resources.buffers.clear();
 	resources.buffers.reserve(program.info.buffers.size());
@@ -1049,15 +987,6 @@ void RenderExecutor::RebindBuffers(PreparedBindings& prepared) {
 		                                                buffer_offset, prepared.buffer_ids[i]));
 		pack_memory_offset(i, buffer_offset);
 	}
-	resources.addresses.clear();
-	resources.addresses.reserve(program.info.addresses.size());
-	for (uint32_t i = 0; i < program.info.addresses.size(); i++) {
-		uint32_t address_offset = 0;
-		resources.addresses.push_back(NativeAddressBuffer(m_context, program.info.addresses[i],
-		                                                  snapshot.addresses[i], program.stage, i,
-		                                                  address_offset, prepared.address_ids[i]));
-		pack_memory_offset(static_cast<uint32_t>(program.info.buffers.size()) + i, address_offset);
-	}
 	if (!prepared.flattened_srt.empty()) {
 		resources.flattened_srt = NativeUpload(m_context, prepared.flattened_srt);
 	}
@@ -1129,6 +1058,10 @@ RenderExecutor::PrepareGraphicsBindings(const ShaderStageRuntime& vertex,
 	if (bindings.pixel) {
 		FindBuffers(*bindings.pixel);
 	}
+	if (bindings.vertex.program->info.uses_dma ||
+	    (bindings.pixel && bindings.pixel->program->info.uses_dma)) {
+		m_context.GetGpuResources().PrepareBda();
+	}
 	RebindBuffers(bindings.vertex);
 	if (bindings.pixel) {
 		RebindBuffers(*bindings.pixel);
@@ -1241,13 +1174,15 @@ void RenderExecutor::CommitBindings(CommandBuffer&                     buffer,
 						m_descriptor_buffers.emplace_back(view.buffer, view.offset, view.range);
 					}
 					break;
-				case BindingKind::AddressMemory:
-					for (const auto resource: binding.resources) {
-						const auto& view = descriptors.addresses.at(resource);
-						EXIT_IF(view.buffer == nullptr);
-						m_descriptor_buffers.emplace_back(view.buffer, view.offset, view.range);
-					}
+				case BindingKind::BdaPagetable:
+				case BindingKind::FaultBuffer: {
+					auto& cache = m_context.GetBufferCache();
+					const auto* bda_buffer = binding.kind == BindingKind::BdaPagetable
+					                             ? cache.GetBdaPageTableBuffer()
+					                             : cache.GetFaultBuffer();
+					m_descriptor_buffers.emplace_back(bda_buffer->Handle(), 0, bda_buffer->Size());
 					break;
+				}
 				case BindingKind::FlattenedSrt:
 				case BindingKind::UserData:
 				case BindingKind::Gds: {

src/graphics/host_gpu/renderer/pipeline/descriptors.h

@@ -38,7 +38,6 @@ struct NativeDescriptors {
 	std::vector<BufferView>     buffers;
 	std::vector<TextureBinding> images;
 	std::vector<vk::Sampler>    samplers;
-	std::vector<BufferView>     addresses;
 	BufferView                  gds;
 	BufferView                  flattened_srt;
 	BufferView                  user_data;
@@ -49,7 +48,6 @@ struct PreparedBindings {
 	const ShaderRecompiler::IR::ResourceSnapshot* snapshot = nullptr;
 	NativeDescriptors                             resources;
 	std::vector<BufferId>                         buffer_ids;
-	std::vector<BufferId>                         address_ids;
 	std::vector<uint32_t>                         flattened_srt;
 	std::vector<uint32_t>                         user_data;
 	bool                                          committed = false;

src/graphics/host_gpu/renderer/pipeline/pipelineCache.cpp

@@ -1,6 +1,7 @@
 #include "graphics/host_gpu/renderer/pipeline/pipelineCache.h"
 
 #include "common/assert.h"
+#include "common/file.h"
 #include "common/logging/log.h"
 #include "common/profiler.h"
 #include "graphics/guest_gpu/hardwareContext.h"
@@ -10,16 +11,65 @@
 #include "graphics/host_gpu/renderer/image/imageView.h"
 #include "graphics/host_gpu/renderer/render.h"
 #include "graphics/host_gpu/renderer/renderContext.h"
+#include "graphics/shader/shaderCompiler.h"
+#include "kytyGitVersion.h"
+#include "loader/systemContent.h"
 
+#include <algorithm>
 #include <atomic>
+#include <cctype>
+#include <cstdio>
 #include <cstring>
+#include <limits>
 #include <span>
+#include <string_view>
 #include <utility>
+#include <vector>
+
+#include <fmt/format.h>
+#include <xxhash.h>
 
 namespace Libs::Graphics {
 
 namespace {
 
+std::string DriverCacheSignature(const vk::PhysicalDeviceProperties& properties) {
+	constexpr char hex[] = "0123456789abcdef";
+	std::string    uuid(VK_UUID_SIZE * 2, '0');
+	for (size_t i = 0; i < VK_UUID_SIZE; i++) {
+		uuid[i * 2]     = hex[properties.pipelineCacheUUID[i] >> 4u];
+		uuid[i * 2 + 1] = hex[properties.pipelineCacheUUID[i] & 0xfu];
+	}
+	return fmt::format("KytyPC1:{}:{:08x}:{:08x}:{:08x}:{}\n", KYTY_GIT_REVISION,
+	                   properties.vendorID, properties.deviceID, properties.driverVersion, uuid);
+}
+
+std::string PipelineCacheTitleId() {
+	std::string title_id;
+	if ((!Loader::SystemContentParamSfoGetString("TITLE_ID", &title_id) || title_id.empty()) &&
+	    (!Loader::SystemContentParamSfoGetString("CONTENT_ID", &title_id) || title_id.empty())) {
+		return {};
+	}
+	if (!std::ranges::all_of(title_id, [](unsigned char c) {
+		    return std::isalnum(c) != 0 || c == '-' || c == '_';
+	    })) {
+		return {};
+	}
+	return title_id;
+}
+
+template <typename... Args>
+void PipelineCacheLog(fmt::format_string<Args...> format, Args&&... args) {
+	auto message = fmt::format(format, std::forward<Args>(args)...);
+	message += '\n';
+	if (Log::GetDirection() != Log::Direction::Console) {
+		std::fwrite(message.data(), 1, message.size(), stdout);
+		std::fflush(stdout);
+	}
+	Log::Write(message);
+	Log::Flush();
+}
+
 void NormalizeStaticParamsForDynamicState(PipelineStaticParameters& static_params) {
 	static_params.viewport_scale[0]  = 0.5f;
 	static_params.viewport_scale[1]  = 0.5f;
@@ -36,7 +86,91 @@ void NormalizeStaticParamsForDynamicState(PipelineStaticParameters& static_param
 
 } // namespace
 
+struct PipelineCache::ProgramCache {
+	struct SourceKey {
+		ShaderType stage = ShaderType::Unknown;
+		uint64_t   hash  = 0;
+
+		bool operator==(const SourceKey&) const = default;
+	};
+
+	struct Permutation {
+		std::vector<uint32_t>                               static_key;
+		std::shared_ptr<const ShaderRecompiler::IR::Program> program;
+		ShaderProgram                                       handle;
+	};
+
+	struct SourceKeyHash {
+		std::size_t operator()(const SourceKey& key) const {
+			std::size_t hash = static_cast<std::size_t>(key.stage);
+			PipelineKeyHash::Mix(hash, static_cast<std::size_t>(key.hash));
+			if constexpr (sizeof(std::size_t) < sizeof(uint64_t)) {
+				PipelineKeyHash::Mix(hash, static_cast<std::size_t>(key.hash >> 32u));
+			}
+			return hash;
+		}
+	};
+
+	static uint64_t MixId(uint64_t hash, uint64_t value) {
+		return hash ^ (value + 0x9e3779b97f4a7c15ull + (hash << 6u) + (hash >> 2u));
+	}
+
+	static constexpr std::size_t MaxStaticKeyWords =
+	    5 + ShaderVertexInputInfo::RES_MAX * 17;
+
+	template <typename InputInfo>
+	ShaderProgram Get(const ShaderParams& params, InputInfo& input_info) {
+		constexpr ShaderType stage = [] {
+			if constexpr (std::is_same_v<InputInfo, ShaderVertexInputInfo>) {
+				return ShaderType::Vertex;
+			} else if constexpr (std::is_same_v<InputInfo, ShaderPixelInputInfo>) {
+				return ShaderType::Pixel;
+			} else {
+				static_assert(std::is_same_v<InputInfo, ShaderComputeInputInfo>);
+				return ShaderType::Compute;
+			}
+		}();
+
+		BuildStageStaticKey(input_info, key_scratch);
+		auto& permutations = programs[{stage, params.hash}];
+		for (const auto& permutation: permutations) {
+			if (permutation.static_key == key_scratch &&
+			    MaterializeProgram(permutation.program, params, input_info)) {
+				return permutation.handle;
+			}
+		}
+
+		const auto module = CompileProgram(device, params, input_info);
+		EXIT_IF(module == nullptr || !input_info.stage);
+		uint64_t id = MixId(MixId(params.hash, static_cast<uint64_t>(stage)), permutations.size());
+		if (id == 0) {
+			id = 1;
+		}
+		const ShaderProgram handle {.id = id, .module = module};
+		permutations.push_back({key_scratch, input_info.stage.program, handle});
+
+		std::printf("Num compiled %u shaders\n", ++num_compiled);
+		return handle;
+	}
+
+	explicit ProgramCache(vk::Device device): device(device) {
+		key_scratch.reserve(MaxStaticKeyWords);
+	}
+
+	std::unordered_map<SourceKey, std::vector<Permutation>, SourceKeyHash> programs;
+	std::vector<uint32_t> key_scratch;
+	vk::Device device;
+	uint32_t   num_compiled = 0;
+};
+
+PipelineCache::PipelineCache(GraphicContext& graphics)
+    : m_graphics(graphics), m_program_cache(std::make_unique<ProgramCache>(graphics.device)) {
+	EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
+	InitializeDriverCache();
+}
+
 PipelineCache::~PipelineCache() {
+	Save();
 	auto destroy = [this](const auto& pipelines) {
 		for (const auto& [key, pipeline]: pipelines) {
 			(void)key;
@@ -47,6 +181,188 @@ PipelineCache::~PipelineCache() {
 	};
 	destroy(m_graphics_pipelines);
 	destroy(m_compute_pipelines);
+	for (const auto& [key, permutations]: m_program_cache->programs) {
+		(void)key;
+		for (const auto& permutation: permutations) {
+			m_graphics.device.destroyShaderModule(permutation.handle.module, nullptr);
+		}
+	}
+	if (m_driver_cache != nullptr) {
+		m_graphics.device.destroyPipelineCache(m_driver_cache, nullptr);
+	}
+}
+
+void PipelineCache::InitializeDriverCache() {
+	const auto title_id = PipelineCacheTitleId();
+	if (title_id.empty()) {
+		return;
+	}
+	if (KYTY_BUILD != KYTY_BUILD_RELEASE) {
+		PipelineCacheLog("Vulkan pipeline cache: disabled (non-Release build)");
+		return;
+	}
+	const std::string_view git_hash     = KYTY_GIT_HASH;
+	const std::string_view git_revision = KYTY_GIT_REVISION;
+	if (git_hash == "unknown" || git_revision == "unknown") {
+		PipelineCacheLog("Vulkan pipeline cache: disabled (unknown git revision)");
+		return;
+	}
+	if (git_hash.ends_with("-dirty")) {
+		PipelineCacheLog("Vulkan pipeline cache: disabled (dirty build)");
+		return;
+	}
+
+	m_driver_cache_path = std::filesystem::path("_PipelineCache") / (title_id + ".bin");
+	const auto path         = Common::PathToString(m_driver_cache_path);
+	const bool cache_exists = Common::File::IsFileExisting(m_driver_cache_path);
+	if (cache_exists) {
+		PipelineCacheLog("Vulkan pipeline cache: loading {}", path);
+	} else {
+		PipelineCacheLog("Vulkan pipeline cache: initializing {}", path);
+	}
+	std::vector<uint8_t> initial_data;
+	if (cache_exists) {
+		Common::File file(m_driver_cache_path, Common::File::Mode::Read);
+		const auto   file_size = file.IsInvalid() ? 0 : file.Size();
+		const auto signature = DriverCacheSignature(m_graphics.GetPhysicalDeviceProperties());
+		if (file_size >= signature.size() + sizeof(uint64_t) &&
+		    file_size <= std::numeric_limits<uint32_t>::max()) {
+			std::string cached_signature(signature.size(), '\0');
+			uint64_t    payload_hash = 0;
+			initial_data.resize(file_size - signature.size() - sizeof(payload_hash));
+			uint32_t signature_read = 0;
+			uint32_t hash_read      = 0;
+			uint32_t payload_read   = 0;
+			file.Read(cached_signature.data(), static_cast<uint32_t>(cached_signature.size()),
+			          &signature_read);
+			file.Read(&payload_hash, sizeof(payload_hash), &hash_read);
+			file.Read(initial_data.data(), static_cast<uint32_t>(initial_data.size()), &payload_read);
+			file.Close();
+			if (signature_read != cached_signature.size() || hash_read != sizeof(payload_hash) ||
+			    payload_read != initial_data.size() || cached_signature != signature ||
+			    XXH3_64bits(initial_data.data(), initial_data.size()) != payload_hash) {
+				initial_data.clear();
+				PipelineCacheLog(
+				    "Vulkan pipeline cache: invalidating {} (driver, emulator, or data mismatch)",
+				    path);
+			}
+		} else {
+			file.Close();
+			PipelineCacheLog("Vulkan pipeline cache: invalidating {} (invalid file size)",
+			                 path);
+		}
+	}
+
+	vk::PipelineCacheCreateInfo create {};
+	create.sType           = vk::StructureType::ePipelineCacheCreateInfo;
+	create.initialDataSize = initial_data.size();
+	create.pInitialData    = initial_data.empty() ? nullptr : initial_data.data();
+	auto result = m_graphics.device.createPipelineCache(&create, nullptr, &m_driver_cache);
+	if (result != vk::Result::eSuccess && !initial_data.empty()) {
+		PipelineCacheLog("Vulkan pipeline cache: driver rejected {} ({}); starting empty", path,
+		                 VulkanToString(result));
+		initial_data.clear();
+		create.initialDataSize = 0;
+		create.pInitialData    = nullptr;
+		result = m_graphics.device.createPipelineCache(&create, nullptr, &m_driver_cache);
+	}
+	if (result != vk::Result::eSuccess) {
+		PipelineCacheLog("Vulkan pipeline cache: disabled ({})", VulkanToString(result));
+		m_driver_cache = nullptr;
+		return;
+	}
+	if (!initial_data.empty()) {
+		PipelineCacheLog("Vulkan pipeline cache: loaded {} bytes from {}", initial_data.size(),
+		                 path);
+	} else {
+		PipelineCacheLog("Vulkan pipeline cache: initialized empty");
+	}
+}
+
+void PipelineCache::Save() {
+	Common::LockGuard lock(m_mutex);
+	if (m_driver_cache == nullptr) {
+		return;
+	}
+
+	size_t               size = 0;
+	vk::Result           result;
+	std::vector<uint8_t> payload;
+	for (uint32_t attempt = 0; attempt < 3; attempt++) {
+		size   = 0;
+		result = m_graphics.device.getPipelineCacheData(m_driver_cache, &size, nullptr);
+		if (result != vk::Result::eSuccess || size == 0 ||
+		    size > std::numeric_limits<uint32_t>::max()) {
+			break;
+		}
+		payload.resize(size);
+		result = m_graphics.device.getPipelineCacheData(m_driver_cache, &size, payload.data());
+		if (result != vk::Result::eIncomplete) {
+			break;
+		}
+	}
+	if (result != vk::Result::eSuccess || size == 0 ||
+	    size > std::numeric_limits<uint32_t>::max()) {
+		PipelineCacheLog("Vulkan pipeline cache: save failed ({}, {} bytes)",
+		                 VulkanToString(result), size);
+		return;
+	}
+	payload.resize(size);
+	auto prefix = DriverCacheSignature(m_graphics.GetPhysicalDeviceProperties());
+	const auto payload_hash = XXH3_64bits(payload.data(), payload.size());
+	prefix.append(reinterpret_cast<const char*>(&payload_hash), sizeof(payload_hash));
+	if (!Common::File::CreateDirectories(m_driver_cache_path.parent_path())) {
+		PipelineCacheLog("Vulkan pipeline cache: failed to create cache directory");
+		return;
+	}
+	auto temp_path = m_driver_cache_path;
+	temp_path += ".tmp";
+	Common::File file;
+	uint32_t     prefix_written  = 0;
+	uint32_t     payload_written = 0;
+	if (file.Create(temp_path)) {
+		file.Write(prefix.data(), static_cast<uint32_t>(prefix.size()), &prefix_written);
+		file.Write(payload.data(), static_cast<uint32_t>(payload.size()), &payload_written);
+	}
+	const bool flushed = !file.IsInvalid() && file.Flush();
+	file.Close();
+	if (prefix_written != prefix.size() || payload_written != payload.size() || !flushed ||
+	    !Common::File::RenameFile(temp_path, m_driver_cache_path)) {
+		PipelineCacheLog("Vulkan pipeline cache: failed to write {}",
+		                 Common::PathToString(m_driver_cache_path));
+		return;
+	}
+	PipelineCacheLog("Vulkan pipeline cache: saved {} bytes to {}", payload.size(),
+	                 Common::PathToString(m_driver_cache_path));
+	m_graphics.device.destroyPipelineCache(m_driver_cache, nullptr);
+	m_driver_cache = nullptr;
+}
+
+ShaderProgram PipelineCache::GetVertexProgram(const HW::VertexShaderInfo& regs,
+                                              const HW::ShaderRegisters&  sh,
+                                              ShaderVertexInputInfo&      input_info) {
+	const auto params = PrepareProgram(regs, sh, input_info);
+	Common::LockGuard lock(m_mutex);
+	return m_program_cache->Get(params, input_info);
+}
+
+ShaderProgram PipelineCache::GetPixelProgram(
+    const HW::PixelShaderInfo& regs, const HW::ShaderRegisters& sh,
+    const ShaderVertexInputInfo&                        vertex_info,
+    std::span<const Prospero::ColorComponentMapping, 8> target_export_mapping,
+    ShaderPixelInputInfo&                               input_info) {
+	const auto params = PrepareProgram(regs, sh, vertex_info, target_export_mapping, input_info);
+	Common::LockGuard lock(m_mutex);
+	return m_program_cache->Get(params, input_info);
+}
+
+ShaderProgram PipelineCache::GetComputeProgram(const HW::ComputeShaderInfo& regs,
+                                               const HW::ShaderRegisters&   sh,
+                                               ShaderComputeInputInfo&      input_info) {
+	input_info.needs_lds_barriers = !m_graphics.compute_wave64_supported;
+	const auto params             = PrepareProgram(regs, sh, input_info);
+	Common::LockGuard lock(m_mutex);
+	return m_program_cache->Get(params, input_info);
 }
 
 bool PipelineStaticParameters::operator==(const PipelineStaticParameters& other) const noexcept {
@@ -54,24 +370,22 @@ bool PipelineStaticParameters::operator==(const PipelineStaticParameters& other)
 }
 
 PipelineCache::GraphicsPipeline& PipelineCache::CreateGraphicsPipeline(
-    RenderColorInfo* colors, uint32_t color_count, RenderDepthInfo& depth,
-    ShaderVertexInputInfo& vs_input_info, RenderCommandBuffer& command,
-    ShaderPixelInputInfo* ps_input_info, vk::PrimitiveTopology topology,
-    bool primitive_restart_enable, bool ps_active, std::span<const uint32_t> vs_spirv,
-    std::span<const uint32_t> ps_spirv) {
+    std::span<const RenderColorInfo> colors, const RenderDepthInfo& depth,
+    const ShaderVertexInputInfo& vs_input_info, CommandBuffer& command,
+    const ShaderPixelInputInfo* ps_input_info, vk::PrimitiveTopology topology,
+    bool primitive_restart_enable, const ShaderProgram& vertex_program,
+    const ShaderProgram& pixel_program) {
 	KYTY_PROFILER_BLOCK("PipelineCache::CreatePipeline(Gfx)", profiler::colors::DeepOrangeA200);
 
-	EXIT_IF(colors == nullptr);
-	EXIT_IF(color_count > RENDER_COLOR_ATTACHMENTS_MAX);
-	EXIT_IF(vs_spirv.empty());
-	EXIT_IF(ps_active && ps_spirv.empty());
+	EXIT_IF(colors.size() > RENDER_COLOR_ATTACHMENTS_MAX);
+	EXIT_IF(!vertex_program);
+	const bool ps_active = ps_input_info != nullptr;
+	EXIT_IF(ps_active && !pixel_program);
+	const auto color_count = static_cast<uint32_t>(colors.size());
 
 	Common::LockGuard lock(m_mutex);
-	auto&             ctx    = command.GetRegisters();
-	auto&             sh_ctx = command.GetShaders();
+	auto&             ctx = command.GetRegisters();
 
-	const auto&           vertex_info                              = sh_ctx.GetVs();
-	const auto&           ps_regs                                  = sh_ctx.GetPs();
 	const HW::BlendColor& bclr                                     = ctx.GetBlendColor();
 	uint32_t              color_mask[RENDER_COLOR_ATTACHMENTS_MAX] = {};
 	for (uint32_t i = 0; i < color_count; i++) {
@@ -82,11 +396,8 @@ PipelineCache::GraphicsPipeline& PipelineCache::CreateGraphicsPipeline(
 	}
 	const HW::ModeControl& mc = ctx.GetModeControl();
 
-	auto     vs_id = ShaderGetIdVS(vertex_info, vs_input_info, true);
-	ShaderId ps_id {};
-	if (ps_active) {
-		ps_id = ShaderGetIdPS(ps_regs, *ps_input_info, true);
-	}
+	const auto vs_id = vertex_program.id;
+	const auto ps_id = ps_active ? pixel_program.id : 0;
 
 	PipelineStaticParameters static_params {};
 	GraphicsPipeline         p {};
@@ -202,20 +513,17 @@ PipelineCache::GraphicsPipeline& PipelineCache::CreateGraphicsPipeline(
 		if (ps_active) {
 			ShaderDbgDumpInputInfo(*ps_input_info);
 		}
-		LOGF("PipelineTrace: shader binaries VS=0x%08" PRIx32 "/0x%08" PRIx32 " words=%" PRIu64
-		     " PS=0x%08" PRIx32 "/0x%08" PRIx32 " words=%" PRIu64 "\n",
-		     vs_id.hash0, vs_id.crc32, static_cast<uint64_t>(vs_spirv.size()), ps_id.hash0,
-		     ps_id.crc32, static_cast<uint64_t>(ps_spirv.size()));
+		LOGF("PipelineTrace: shader modules VS=%" PRIu64 " module=%p PS=%" PRIu64
+		     " module=%p\n",
+		     vs_id, static_cast<void*>(vertex_program.module), ps_id,
+		     static_cast<void*>(pixel_program.module));
 	}
 
 	auto cached = std::make_unique<GraphicsPipeline>(p);
-	LogPipelineTrace("CreatePipelineInternal begin", vs_id.hash0, vs_id.crc32, ps_id.hash0,
-	                 ps_id.crc32);
-	CreatePipelineInternal(m_graphics, *cached, rendering, vs_input_info, vs_spirv, ps_input_info,
-	                       ps_spirv, static_params, vs_id.hash0, vs_id.crc32, ps_id.hash0,
-	                       ps_id.crc32, ps_active);
-	LogPipelineTrace("CreatePipelineInternal done", vs_id.hash0, vs_id.crc32, ps_id.hash0,
-	                 ps_id.crc32);
+	LogPipelineTrace("CreatePipelineInternal begin", vs_id, ps_id);
+	CreatePipelineInternal(m_graphics, *cached, rendering, vs_input_info, vertex_program.module,
+	                       ps_input_info, pixel_program.module, static_params, m_driver_cache);
+	LogPipelineTrace("CreatePipelineInternal done", vs_id, ps_id);
 
 	EXIT_NOT_IMPLEMENTED(cached->pipeline == nullptr);
 	EXIT_NOT_IMPLEMENTED(cached->pipeline_layout == nullptr);
@@ -227,19 +535,16 @@ PipelineCache::GraphicsPipeline& PipelineCache::CreateGraphicsPipeline(
 }
 
 PipelineCache::ComputePipeline&
-PipelineCache::CreateComputePipeline(ShaderComputeInputInfo&      input_info,
-                                     const HW::ComputeShaderInfo& cs_regs,
-                                     std::span<const uint32_t>    cs_spirv) {
+PipelineCache::CreateComputePipeline(ShaderComputeInputInfo& input_info,
+                                     const ShaderProgram&    compute_program) {
 	KYTY_PROFILER_BLOCK("PipelineCache::CreatePipeline(Compute)", profiler::colors::RedA100);
 
-	EXIT_IF(cs_spirv.empty());
+	EXIT_IF(!compute_program);
 
 	Common::LockGuard lock(m_mutex);
 
-	auto cs_id = ShaderGetIdCS(cs_regs, input_info, true);
-
 	ComputePipeline p {};
-	p.cs_shader_id = cs_id;
+	p.cs_shader_id = compute_program.id;
 
 	ComputePipelineKey key {};
 	key.cs_shader_id = p.cs_shader_id;
@@ -253,7 +558,7 @@ PipelineCache::CreateComputePipeline(ShaderComputeInputInfo&      input_info,
 	}
 
 	auto cached = std::make_unique<ComputePipeline>(p);
-	CreatePipelineInternal(m_graphics, *cached, input_info, cs_spirv);
+	CreatePipelineInternal(m_graphics, *cached, input_info, compute_program.module, m_driver_cache);
 
 	EXIT_NOT_IMPLEMENTED(cached->pipeline == nullptr);
 	EXIT_NOT_IMPLEMENTED(cached->pipeline_layout == nullptr);

src/graphics/host_gpu/renderer/pipeline/pipelineCache.h

@@ -10,6 +10,7 @@
 #include "graphics/shader/shader.h"
 
 #include <cstddef>
+#include <filesystem>
 #include <memory>
 #include <span>
 #include <type_traits>
@@ -20,7 +21,7 @@ namespace Libs::Graphics {
 struct GraphicContext;
 struct RenderColorInfo;
 struct RenderDepthInfo;
-class RenderCommandBuffer;
+class CommandBuffer;
 
 namespace HW {
 class Context;
@@ -95,13 +96,21 @@ struct PipelineRenderingState {
 	bool operator==(const PipelineRenderingState&) const = default;
 };
 
+struct ShaderProgram {
+	uint64_t         id     = 0;
+	vk::ShaderModule module = nullptr;
+
+	explicit operator bool() const {
+		return id != 0 && module != nullptr;
+	}
+};
+
 class PipelineCache {
 public:
-	explicit PipelineCache(GraphicContext& graphics): m_graphics(graphics) {
-		EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
-	}
+	explicit PipelineCache(GraphicContext& graphics);
 	~PipelineCache();
 	KYTY_CLASS_NO_COPY(PipelineCache);
+	void Save();
 
 	struct Pipeline {
 		vk::PipelineLayout      pipeline_layout       = nullptr;
@@ -111,29 +120,42 @@ public:
 	};
 
 	struct GraphicsPipeline: Pipeline {
-		ShaderId vs_shader_id;
-		ShaderId ps_shader_id;
+		uint64_t vs_shader_id = 0;
+		uint64_t ps_shader_id = 0;
 	};
 
 	struct ComputePipeline: Pipeline {
-		ShaderId cs_shader_id;
+		uint64_t cs_shader_id = 0;
 	};
 
-	GraphicsPipeline&
-	CreateGraphicsPipeline(RenderColorInfo* colors, uint32_t color_count, RenderDepthInfo& depth,
-	                       ShaderVertexInputInfo& vs_input_info, RenderCommandBuffer& command,
-	                       ShaderPixelInputInfo* ps_input_info, vk::PrimitiveTopology topology,
-	                       bool primitive_restart_enable, bool ps_active,
-	                       std::span<const uint32_t> vs_spirv, std::span<const uint32_t> ps_spirv);
-	ComputePipeline& CreateComputePipeline(ShaderComputeInputInfo&      input_info,
-	                                       const HW::ComputeShaderInfo& cs_regs,
-	                                       std::span<const uint32_t>    cs_spirv);
+	ShaderProgram GetVertexProgram(const HW::VertexShaderInfo& regs,
+	                               const HW::ShaderRegisters&  sh,
+	                               ShaderVertexInputInfo&      input_info);
+	ShaderProgram
+	GetPixelProgram(const HW::PixelShaderInfo& regs, const HW::ShaderRegisters& sh,
+	                const ShaderVertexInputInfo&                        vertex_info,
+	                std::span<const Prospero::ColorComponentMapping, 8> target_export_mapping,
+	                ShaderPixelInputInfo&                               input_info);
+	ShaderProgram GetComputeProgram(const HW::ComputeShaderInfo& regs,
+	                                const HW::ShaderRegisters& sh,
+	                                ShaderComputeInputInfo& input_info);
+
+	GraphicsPipeline& CreateGraphicsPipeline(
+	    std::span<const RenderColorInfo> colors, const RenderDepthInfo& depth,
+	    const ShaderVertexInputInfo& vs_input_info, CommandBuffer& command,
+	    const ShaderPixelInputInfo* ps_input_info, vk::PrimitiveTopology topology,
+	    bool primitive_restart_enable, const ShaderProgram& vertex_program,
+	    const ShaderProgram& pixel_program);
+	ComputePipeline& CreateComputePipeline(ShaderComputeInputInfo& input_info,
+	                                       const ShaderProgram&    compute_program);
 
 private:
+	struct ProgramCache;
+
 	struct GraphicsPipelineKey {
 		PipelineRenderingState   rendering;
-		ShaderId                 vs_shader_id;
-		ShaderId                 ps_shader_id;
+		uint64_t                 vs_shader_id = 0;
+		uint64_t                 ps_shader_id = 0;
 		PipelineStaticParameters static_params;
 
 		bool operator==(const GraphicsPipelineKey& other) const {
@@ -143,7 +165,7 @@ private:
 	};
 
 	struct ComputePipelineKey {
-		ShaderId cs_shader_id;
+		uint64_t cs_shader_id = 0;
 
 		bool operator==(const ComputePipelineKey& other) const {
 			return cs_shader_id == other.cs_shader_id;
@@ -156,15 +178,6 @@ private:
 			        (hash >> 2u);
 		}
 
-		static void MixShaderId(std::size_t& hash, const ShaderId& id) {
-			Mix(hash, id.hash0);
-			Mix(hash, id.crc32);
-			Mix(hash, id.ids.size());
-			for (auto value: id.ids) {
-				Mix(hash, value);
-			}
-		}
-
 		static void MixStaticParams(std::size_t& hash, const PipelineStaticParameters& params) {
 			const auto* bytes = reinterpret_cast<const uint8_t*>(&params);
 			for (std::size_t i = 0; i < sizeof(params); i++) {
@@ -186,8 +199,8 @@ private:
 		std::size_t operator()(const GraphicsPipelineKey& key) const {
 			std::size_t hash = 0;
 			PipelineKeyHash::MixRendering(hash, key.rendering);
-			PipelineKeyHash::MixShaderId(hash, key.vs_shader_id);
-			PipelineKeyHash::MixShaderId(hash, key.ps_shader_id);
+			PipelineKeyHash::Mix(hash, key.vs_shader_id);
+			PipelineKeyHash::Mix(hash, key.ps_shader_id);
 			PipelineKeyHash::MixStaticParams(hash, key.static_params);
 			return hash;
 		}
@@ -196,31 +209,35 @@ private:
 	struct ComputePipelineKeyHash {
 		std::size_t operator()(const ComputePipelineKey& key) const {
 			std::size_t hash = 0;
-			PipelineKeyHash::MixShaderId(hash, key.cs_shader_id);
+			PipelineKeyHash::Mix(hash, key.cs_shader_id);
 			return hash;
 		}
 	};
 
-	GraphicContext& m_graphics;
+	GraphicContext&                m_graphics;
+	std::unique_ptr<ProgramCache> m_program_cache;
+	vk::PipelineCache             m_driver_cache = nullptr;
+	std::filesystem::path         m_driver_cache_path;
 	std::unordered_map<GraphicsPipelineKey, std::unique_ptr<GraphicsPipeline>,
 	                   GraphicsPipelineKeyHash>
 	    m_graphics_pipelines;
 	std::unordered_map<ComputePipelineKey, std::unique_ptr<ComputePipeline>, ComputePipelineKeyHash>
 	              m_compute_pipelines;
 	Common::Mutex m_mutex;
+
+	void InitializeDriverCache();
 };
 
-void LogPipelineTrace(const char* phase, uint32_t vs_hash0, uint32_t vs_crc32, uint32_t ps_hash0,
-                      uint32_t ps_crc32);
+void LogPipelineTrace(const char* phase, uint64_t vertex_program_id, uint64_t pixel_program_id);
 void CreatePipelineInternal(
     GraphicContext& graphics, PipelineCache::GraphicsPipeline& pipeline,
     const PipelineRenderingState& rendering, const ShaderVertexInputInfo& vs_input_info,
-    std::span<const uint32_t> vs_shader, const ShaderPixelInputInfo* ps_input_info,
-    std::span<const uint32_t> ps_shader, const PipelineStaticParameters& static_params,
-    uint32_t vs_hash0, uint32_t vs_crc32, uint32_t ps_hash0, uint32_t ps_crc32, bool ps_active);
+    vk::ShaderModule vertex_module, const ShaderPixelInputInfo* ps_input_info,
+	vk::ShaderModule pixel_module, const PipelineStaticParameters& static_params,
+	vk::PipelineCache driver_cache);
 void CreatePipelineInternal(GraphicContext& graphics, PipelineCache::ComputePipeline& pipeline,
                             const ShaderComputeInputInfo& input_info,
-                            std::span<const uint32_t>     cs_shader);
+	vk::ShaderModule compute_module, vk::PipelineCache driver_cache);
 
 } // namespace Libs::Graphics
 

src/graphics/host_gpu/renderer/pipeline/shaders.cpp

@@ -422,43 +422,20 @@ static void CreateDescriptorLayout(GraphicContext& graphics, PipelineCache::Pipe
 void CreatePipelineInternal(
     GraphicContext& graphics, PipelineCache::GraphicsPipeline& pipeline,
     const PipelineRenderingState& rendering, const ShaderVertexInputInfo& vs_input_info,
-    std::span<const uint32_t> vs_shader, const ShaderPixelInputInfo* ps_input_info,
-    std::span<const uint32_t> ps_shader, const PipelineStaticParameters& static_params,
-    uint32_t vs_hash0, uint32_t vs_crc32, uint32_t ps_hash0, uint32_t ps_crc32, bool ps_active) {
-	EXIT_IF(ps_active && ps_input_info == nullptr);
+    vk::ShaderModule vertex_module, const ShaderPixelInputInfo* ps_input_info,
+	vk::ShaderModule pixel_module, const PipelineStaticParameters& static_params,
+	vk::PipelineCache driver_cache) {
+	const bool ps_active = ps_input_info != nullptr;
+	EXIT_IF(vertex_module == nullptr || (ps_active && pixel_module == nullptr));
 
 	const bool rect_list = static_params.topology == vk::PrimitiveTopology::ePatchList;
 
-	vk::ShaderModule vert_shader_module         = nullptr;
 	vk::ShaderModule tess_control_shader_module = nullptr;
 	vk::ShaderModule tess_eval_shader_module    = nullptr;
-	vk::ShaderModule frag_shader_module         = nullptr;
 
 	vk::ShaderModuleCreateInfo create_info {};
-
 	create_info.sType = vk::StructureType::eShaderModuleCreateInfo;
-	create_info.pNext = nullptr;
-	create_info.flags = {};
-
-	create_info.codeSize = vs_shader.size() * 4;
-	create_info.pCode    = vs_shader.data();
-	auto result = graphics.device.createShaderModule(&create_info, nullptr, &vert_shader_module);
-	if (graphics_debug_dump_enabled()) {
-		LOGF("PipelineTrace: vkCreateShaderModule VS done result=%s module=%p\n",
-		     VulkanToString(result).c_str(), static_cast<void*>(vert_shader_module));
-	}
-	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-
-	if (ps_active) {
-		create_info.codeSize = ps_shader.size() * 4;
-		create_info.pCode    = ps_shader.data();
-		result = graphics.device.createShaderModule(&create_info, nullptr, &frag_shader_module);
-		if (graphics_debug_dump_enabled()) {
-			LOGF("PipelineTrace: vkCreateShaderModule PS done result=%s module=%p\n",
-			     VulkanToString(result).c_str(), static_cast<void*>(frag_shader_module));
-		}
-		EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-	}
+	vk::Result result {};
 	if (rect_list) {
 		const auto shaders =
 		    BuildRectListShaders(vs_input_info, ps_active ? ps_input_info : nullptr);
@@ -483,10 +460,8 @@ void CreatePipelineInternal(
 		EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
 	}
 
-	EXIT_NOT_IMPLEMENTED(vert_shader_module == nullptr);
 	EXIT_NOT_IMPLEMENTED(
 	    rect_list && (tess_control_shader_module == nullptr || tess_eval_shader_module == nullptr));
-	EXIT_NOT_IMPLEMENTED(ps_active && frag_shader_module == nullptr);
 
 	vk::PipelineShaderStageCreateInfo vert_shader_stage_info {};
 
@@ -494,7 +469,7 @@ void CreatePipelineInternal(
 	vert_shader_stage_info.pNext               = nullptr;
 	vert_shader_stage_info.flags               = {};
 	vert_shader_stage_info.stage               = vk::ShaderStageFlagBits::eVertex;
-	vert_shader_stage_info.module              = vert_shader_module;
+	vert_shader_stage_info.module              = vertex_module;
 	vert_shader_stage_info.pName               = "main";
 	vert_shader_stage_info.pSpecializationInfo = nullptr;
 
@@ -503,7 +478,7 @@ void CreatePipelineInternal(
 	frag_shader_stage_info.pNext               = nullptr;
 	frag_shader_stage_info.flags               = {};
 	frag_shader_stage_info.stage               = vk::ShaderStageFlagBits::eFragment;
-	frag_shader_stage_info.module              = frag_shader_module;
+	frag_shader_stage_info.module              = pixel_module;
 	frag_shader_stage_info.pName               = "main";
 	frag_shader_stage_info.pSpecializationInfo = nullptr;
 
@@ -548,9 +523,10 @@ void CreatePipelineInternal(
 
 			uint32_t attr_size       = 4;
 			auto     registers_num   = vs_input_info.resources_dst[index].registers_num;
-			auto     used_components = (vs_input_info.resource_fetch_components[index] > 0
-			                                ? vs_input_info.resource_fetch_components[index]
-			                                : registers_num);
+			const auto compiled_components =
+			    vs_input_info.stage.program->info.vertex_fetch_components[index];
+			auto used_components =
+			    compiled_components > 0 ? static_cast<int>(compiled_components) : registers_num;
 			GetInputFormat(vs_input_info.resources[index], input_attr[index].format, attr_size,
 			               static_cast<uint32_t>(used_components));
 
@@ -853,9 +829,9 @@ void CreatePipelineInternal(
 	EXIT_IF(pipeline.pipeline_layout != nullptr);
 
 	if (graphics_debug_dump_enabled()) {
-		LOGF("PipelineTrace: vkCreatePipelineLayout begin VS=0x%08" PRIx32 "/0x%08" PRIx32
-		     " PS=0x%08" PRIx32 "/0x%08" PRIx32 " set_layouts=1 push_constants=%" PRIu32 "\n",
-		     vs_hash0, vs_crc32, ps_hash0, ps_crc32, 1u);
+		LOGF("PipelineTrace: vkCreatePipelineLayout begin VS=%" PRIu64 " PS=%" PRIu64
+		     " set_layouts=1 push_constants=%" PRIu32 "\n",
+		     pipeline.vs_shader_id, pipeline.ps_shader_id, 1u);
 	}
 	result = graphics.device.createPipelineLayout(&pipeline_layout_info, nullptr,
 	                                              &pipeline.pipeline_layout);
@@ -954,17 +930,18 @@ void CreatePipelineInternal(
 	EXIT_IF(pipeline.pipeline != nullptr);
 
 	if (graphics_debug_dump_enabled()) {
-		LOGF("PipelineTrace: vkCreateGraphicsPipelines begin VS=0x%08" PRIx32 "/0x%08" PRIx32
-		     " PS=0x%08" PRIx32 "/0x%08" PRIx32 " topology=%" PRIu32 " color_mask=0x%08" PRIx32
+		LOGF("PipelineTrace: vkCreateGraphicsPipelines begin VS=%" PRIu64 " PS=%" PRIu64
+		     " topology=%" PRIu32 " color_mask=0x%08" PRIx32
 		     " depth=%s blend=%s dyn_states=%" PRIu32 " viewport=%f,%f %fx%f"
 		     " scissor=%d,%d %ux%u\n",
-		     vs_hash0, vs_crc32, ps_hash0, ps_crc32, static_cast<uint32_t>(static_params.topology),
+		     pipeline.vs_shader_id, pipeline.ps_shader_id,
+		     static_cast<uint32_t>(static_params.topology),
 		     static_params.color_mask[0], (static_params.with_depth ? "true" : "false"),
 		     (static_params.blend_enable[0] ? "true" : "false"), dynamic_states_count, viewport.x,
 		     viewport.y, viewport.width, viewport.height, scissor.offset.x, scissor.offset.y,
 		     scissor.extent.width, scissor.extent.height);
 	}
-	result = graphics.device.createGraphicsPipelines(nullptr, 1, &pipeline_info, nullptr,
+	result = graphics.device.createGraphicsPipelines(driver_cache, 1, &pipeline_info, nullptr,
 	                                                 &pipeline.pipeline);
 	if (graphics_debug_dump_enabled()) {
 		LOGF("PipelineTrace: vkCreateGraphicsPipelines done result=%s pipeline=%p\n",
@@ -974,39 +951,19 @@ void CreatePipelineInternal(
 
 	EXIT_NOT_IMPLEMENTED(pipeline.pipeline == nullptr);
 
-	if (frag_shader_module != nullptr) {
-		graphics.device.destroyShaderModule(frag_shader_module, nullptr);
-	}
 	if (tess_control_shader_module != nullptr) {
 		graphics.device.destroyShaderModule(tess_control_shader_module, nullptr);
 	}
 	if (tess_eval_shader_module != nullptr) {
 		graphics.device.destroyShaderModule(tess_eval_shader_module, nullptr);
 	}
-	graphics.device.destroyShaderModule(vert_shader_module, nullptr);
 }
 
 // NOLINTNEXTLINE(readability-function-cognitive-complexity)
 void CreatePipelineInternal(GraphicContext& graphics, PipelineCache::ComputePipeline& pipeline,
                             const ShaderComputeInputInfo& input_info,
-                            std::span<const uint32_t>     cs_shader) {
-	vk::ShaderModule comp_shader_module = nullptr;
-
-	vk::ShaderModuleCreateInfo create_info {};
-
-	create_info.sType    = vk::StructureType::eShaderModuleCreateInfo;
-	create_info.pNext    = nullptr;
-	create_info.flags    = {};
-	create_info.codeSize = cs_shader.size() * 4;
-	create_info.pCode    = cs_shader.data();
-	LOGF("PipelineTrace: vkCreateShaderModule CS begin words=%" PRIu64 "\n",
-	     static_cast<uint64_t>(cs_shader.size()));
-	auto result = graphics.device.createShaderModule(&create_info, nullptr, &comp_shader_module);
-	LOGF("PipelineTrace: vkCreateShaderModule CS done result=%s module=%p\n",
-	     VulkanToString(result).c_str(), static_cast<void*>(comp_shader_module));
-	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
-
-	EXIT_NOT_IMPLEMENTED(comp_shader_module == nullptr);
+	vk::ShaderModule compute_module, vk::PipelineCache driver_cache) {
+	EXIT_IF(compute_module == nullptr);
 
 	vk::PipelineShaderStageCreateInfo                     comp_shader_stage_info {};
 	vk::PipelineShaderStageRequiredSubgroupSizeCreateInfo comp_subgroup_size {};
@@ -1014,13 +971,12 @@ void CreatePipelineInternal(GraphicContext& graphics, PipelineCache::ComputePipe
 	comp_shader_stage_info.pNext               = nullptr;
 	comp_shader_stage_info.flags               = {};
 	comp_shader_stage_info.stage               = vk::ShaderStageFlagBits::eCompute;
-	comp_shader_stage_info.module              = comp_shader_module;
+	comp_shader_stage_info.module              = compute_module;
 	comp_shader_stage_info.pName               = "main";
 	comp_shader_stage_info.pSpecializationInfo = nullptr;
 	EXIT_IF(!input_info.stage);
 	const auto wave_size = input_info.stage.program->wave_size;
-	if (graphics.subgroup_size_control_enabled &&
-	    (graphics.required_subgroup_size_stages & vk::ShaderStageFlagBits::eCompute) &&
+	if (graphics.compute_subgroup_size_control_enabled &&
 	    wave_size >= graphics.min_subgroup_size && wave_size <= graphics.max_subgroup_size) {
 		comp_subgroup_size.sType =
 		    vk::StructureType::ePipelineShaderStageRequiredSubgroupSizeCreateInfo;
@@ -1048,8 +1004,8 @@ void CreatePipelineInternal(GraphicContext& graphics, PipelineCache::ComputePipe
 
 	LOGF("PipelineTrace: vkCreatePipelineLayout CS begin set_layouts=1 push_constants=%u\n",
 	     1u);
-	result = graphics.device.createPipelineLayout(&pipeline_layout_info, nullptr,
-	                                              &pipeline.pipeline_layout);
+	auto result = graphics.device.createPipelineLayout(&pipeline_layout_info, nullptr,
+	                                                  &pipeline.pipeline_layout);
 	LOGF("PipelineTrace: vkCreatePipelineLayout CS done result=%s layout=%p\n",
 	     VulkanToString(result).c_str(), static_cast<void*>(pipeline.pipeline_layout));
 	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
@@ -1069,14 +1025,13 @@ void CreatePipelineInternal(GraphicContext& graphics, PipelineCache::ComputePipe
 
 	LOGF("PipelineTrace: vkCreateComputePipelines begin layout=%p\n",
 	     static_cast<void*>(pipeline.pipeline_layout));
-	result = graphics.device.createComputePipelines(nullptr, 1, &info, nullptr, &pipeline.pipeline);
+	result = graphics.device.createComputePipelines(driver_cache, 1, &info, nullptr,
+	                                                &pipeline.pipeline);
 	LOGF("PipelineTrace: vkCreateComputePipelines done result=%s pipeline=%p\n",
 	     VulkanToString(result).c_str(), static_cast<void*>(pipeline.pipeline));
 	EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
 
 	EXIT_NOT_IMPLEMENTED(pipeline.pipeline == nullptr);
-
-	graphics.device.destroyShaderModule(comp_shader_module, nullptr);
 }
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/render.h

@@ -25,7 +25,6 @@ class Shader;
 struct GraphicContext;
 struct ShaderBufferResource;
 struct ShaderComputeInputInfo;
-struct CommandSlot;
 struct RenderDepthInfo;
 struct RenderColorInfo;
 struct DrawCallInfo;
@@ -77,40 +76,38 @@ struct SubmitInfo {
 
 class CommandBuffer {
 public:
-	explicit CommandBuffer(CommandScheduler& scheduler);
-	~CommandBuffer();
+	~CommandBuffer() = default;
 
 	KYTY_CLASS_NO_COPY(CommandBuffer);
 
 	[[nodiscard]] bool IsInvalid() const;
 
-	void Begin() const;
-	void End() const;
-	void Execute(const SubmitInfo& submit = {});
 	void SetDebugInfo(uint32_t op, uint64_t submit_id, uint32_t arg0 = 0, uint32_t arg1 = 0,
 	                  uint32_t arg2 = 0, uint32_t arg3 = 0, uint64_t arg4 = 0);
 	void BeginRendering(const RenderState& state) const;
 	void EndRendering() const;
-	void WaitForFenceOnly();
-	void WaitForFence();
-	void WaitForFenceAndReset();
 
 	[[nodiscard]] vk::CommandBuffer Handle() const;
 	[[nodiscard]] GraphicContext&   GetGraphics() const noexcept { return m_graphics; }
 	[[nodiscard]] RenderContext&    GetContext() const noexcept { return m_context; }
-	[[nodiscard]] bool              IsExecute() const { return m_execute; }
+	[[nodiscard]] HW::Context&      GetRegisters() const noexcept { return *m_registers; }
+	[[nodiscard]] HW::UserConfig&   GetUserConfig() const noexcept { return *m_user_config; }
+	[[nodiscard]] HW::Shader&       GetShaders() const noexcept { return *m_shaders; }
 
 private:
-	void Release();
-	void FinalizeFence(bool reset_recording);
+	explicit CommandBuffer(CommandScheduler& scheduler);
+	void Bind(HW::Context& registers, HW::UserConfig& user_config, HW::Shader& shaders) noexcept {
+		m_registers   = &registers;
+		m_user_config = &user_config;
+		m_shaders     = &shaders;
+	}
+
+	void Begin();
+	void End() const;
 
 	RenderContext&      m_context;
-	CommandScheduler&   m_scheduler;
 	GraphicContext&     m_graphics;
-	CommandSlot*        m_slot            = nullptr;
-	bool                m_execute         = false;
-	bool                m_fence_waited    = false;
-	uint64_t            m_submit_seq      = 0;
+	vk::CommandBuffer   m_buffer          = nullptr;
 	uint32_t            m_debug_op        = 0;
 	uint64_t            m_debug_submit_id = 0;
 	uint32_t            m_debug_arg0      = 0;
@@ -119,27 +116,12 @@ private:
 	uint32_t            m_debug_arg3      = 0;
 	uint64_t            m_debug_arg4      = 0;
 	mutable RenderState m_render_state;
-	mutable bool        m_rendering = false;
-};
+	mutable bool        m_rendering   = false;
+	HW::Context*        m_registers   = nullptr;
+	HW::UserConfig*     m_user_config = nullptr;
+	HW::Shader*         m_shaders     = nullptr;
 
-class RenderCommandBuffer final: public CommandBuffer {
-public:
-	explicit RenderCommandBuffer(CommandScheduler& scheduler): CommandBuffer(scheduler) {}
-
-	void Bind(HW::Context& registers, HW::UserConfig& user_config, HW::Shader& shaders) noexcept {
-		m_registers   = &registers;
-		m_user_config = &user_config;
-		m_shaders     = &shaders;
-	}
-
-	[[nodiscard]] HW::Context&    GetRegisters() const noexcept { return *m_registers; }
-	[[nodiscard]] HW::UserConfig& GetUserConfig() const noexcept { return *m_user_config; }
-	[[nodiscard]] HW::Shader&     GetShaders() const noexcept { return *m_shaders; }
-
-private:
-	HW::Context*    m_registers   = nullptr;
-	HW::UserConfig* m_user_config = nullptr;
-	HW::Shader*     m_shaders     = nullptr;
+	friend class CommandScheduler;
 };
 
 class RenderExecutor {
@@ -147,15 +129,14 @@ public:
 	explicit RenderExecutor(RenderContext& context): m_context(context) {}
 	KYTY_CLASS_NO_COPY(RenderExecutor);
 
-	void DrawIndex(uint64_t submit_id, RenderCommandBuffer& buffer, uint32_t index_type_and_size,
+	void DrawIndex(uint64_t submit_id, CommandBuffer& buffer, uint32_t index_type_and_size,
 	               uint32_t index_count, const void* index_addr, uint32_t flags, uint32_t type,
 	               uint32_t instance_count = 1, uint32_t render_target_slice_offset = 0,
 	               int32_t vertex_offset_add = 0, uint32_t first_instance = 0);
-	void DrawAuto(uint64_t submit_id, RenderCommandBuffer& buffer, uint32_t index_count,
-	              uint32_t flags, uint32_t render_target_slice_offset = 0,
-	              uint32_t instance_count = 1, uint32_t first_vertex = 0,
-	              uint32_t first_instance = 0);
-	void DispatchDirect(uint64_t submit_id, RenderCommandBuffer& buffer, uint32_t thread_group_x,
+	void DrawAuto(uint64_t submit_id, CommandBuffer& buffer, uint32_t index_count, uint32_t flags,
+	              uint32_t render_target_slice_offset = 0, uint32_t instance_count = 1,
+	              uint32_t first_vertex = 0, uint32_t first_instance = 0);
+	void DispatchDirect(uint64_t submit_id, CommandBuffer& buffer, uint32_t thread_group_x,
 	                    uint32_t thread_group_y, uint32_t thread_group_z, uint32_t mode);
 
 	[[nodiscard]] PreparedBindings PrepareBindings(const ShaderStageRuntime& runtime);
@@ -177,32 +158,31 @@ private:
 	[[nodiscard]] GraphicsBindings PrepareGraphicsBindings(const ShaderStageRuntime& vertex,
 	                                                       const ShaderStageRuntime& pixel,
 	                                                       bool                      pixel_active);
-	void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
+	void ResolveRenderColorTarget(uint64_t submit_id, CommandBuffer& buffer,
 	                              RenderColorInfo& target, uint32_t render_target_slice_offset = 0,
 	                              uint32_t render_target_slot = UINT32_MAX,
 	                              bool ignore_target_mask = false, bool exact_format = false);
-	void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
+	void ResolveRenderDepthTarget(uint64_t submit_id, CommandBuffer& buffer,
 	                              RenderDepthInfo& target);
-	[[nodiscard]] bool PrepareDrawRenderState(uint64_t submit_id, RenderCommandBuffer& buffer,
+	[[nodiscard]] bool PrepareDrawRenderState(uint64_t submit_id, CommandBuffer& buffer,
 	                                          const DrawCallInfo& draw,
 	                                          uint32_t            render_target_slice_offset,
 	                                          bool log_setup_phases, DrawRenderState& state);
-	void               ExecutePreparedDraw(uint64_t submit_id, RenderCommandBuffer& buffer,
-	                                       const DrawCallInfo& draw, DrawRenderState& state,
-	                                       vk::PrimitiveTopology topology, const DrawEmitInfo& emit,
-	                                       const DrawIndexBufferSource& index_source,
-	                                       bool primitive_restart_enable, bool log_pipeline_phase,
-	                                       bool set_bind_debug, bool set_auto_debug);
+	void ExecutePreparedDraw(uint64_t submit_id, CommandBuffer& buffer, const DrawCallInfo& draw,
+	                         DrawRenderState& state, vk::PrimitiveTopology topology,
+	                         const DrawEmitInfo& emit, const DrawIndexBufferSource& index_source,
+	                         bool primitive_restart_enable, bool log_pipeline_phase,
+	                         bool set_bind_debug, bool set_auto_debug);
 	[[nodiscard]] RenderState AcquireRenderTargets(CommandBuffer& buffer, RenderColorInfo* colors,
 	                                               uint32_t color_count, RenderDepthInfo& depth);
-	[[nodiscard]] bool        ResolveColorTargets(uint64_t submit_id, RenderCommandBuffer& buffer,
+	[[nodiscard]] bool        ResolveColorTargets(uint64_t submit_id, CommandBuffer& buffer,
 	                                              uint32_t render_target_slice_offset);
 	void                      BindImage(ImageId id, bool storage);
 	void                      BindRenderTarget(ImageId id);
 	void                      TrackImageBinding(ImageId id);
 	void                      ResetBindings();
 	[[nodiscard]] bool        TryConsumeComputeMetaClear(const ShaderComputeInputInfo& input,
-	                                                     const RenderCommandBuffer&    buffer);
+	                                                     const CommandBuffer&          buffer);
 
 	RenderContext&                        m_context;
 	std::vector<ImageId>                  m_bound_images;

src/graphics/host_gpu/renderer/renderCompute.cpp

@@ -44,7 +44,7 @@ static uint64_t BufferDescriptorSize(const ShaderBufferResource& descriptor) {
 }
 
 bool RenderExecutor::TryConsumeComputeMetaClear(const ShaderComputeInputInfo& input,
-                                                const RenderCommandBuffer&    buffer) {
+                                                const CommandBuffer&          buffer) {
 	const auto& program   = *input.stage.program;
 	const auto& resources = *input.stage.resources;
 	if (resources.buffers.size() != program.info.buffers.size()) {
@@ -81,9 +81,8 @@ bool ResolveComputeImageClear(const ShaderComputeInputInfo& input, uint32_t grou
 	const auto& program   = *input.stage.program;
 	const auto& resources = *input.stage.resources;
 	if (program.info.buffers.size() != 1 || resources.buffers.size() != 1 ||
-	    !program.info.images.empty() || !program.info.samplers.empty() ||
-	    !program.info.addresses.empty() || !resources.images.empty() ||
-	    !resources.samplers.empty() || !resources.addresses.empty()) {
+	    !program.info.images.empty() || !program.info.samplers.empty() || program.info.uses_dma ||
+	    !resources.images.empty() || !resources.samplers.empty()) {
 		return false;
 	}
 	const auto& resource   = program.info.buffers.front();
@@ -163,7 +162,7 @@ static bool TryConsumeComputeImageClear(const ShaderComputeInputInfo& input, Com
 	return true;
 }
 
-void RenderExecutor::DispatchDirect(uint64_t submit_id, RenderCommandBuffer& buffer,
+void RenderExecutor::DispatchDirect(uint64_t submit_id, CommandBuffer& buffer,
                                     uint32_t thread_group_x, uint32_t thread_group_y,
                                     uint32_t thread_group_z, uint32_t mode) {
 	EXIT_IF(buffer.IsInvalid());
@@ -208,16 +207,12 @@ void RenderExecutor::DispatchDirect(uint64_t submit_id, RenderCommandBuffer& buf
 	const auto& cs_regs = sh_ctx.GetCs();
 	const auto& sh_regs = ctx.GetShaderRegisters();
 
-	ShaderComputeInputInfo    input_info {};
-	std::span<const uint32_t> cs_shader;
-	if (!ShaderCompileInfoCS(cs_regs, sh_regs, input_info, cs_shader)) {
-		EXIT("ShaderCompileInfoCS failed for dispatch with CS shader 0x%016" PRIx64 "\n",
-		     cs_regs.cs_regs.data_addr);
-	}
-
+	ShaderComputeInputInfo input_info {};
 	const bool use_thread_dimensions = (mode & DISPATCH_INITIATOR_USE_THREAD_DIMENSIONS) != 0;
+	input_info.dispatch_thread_dimensions = use_thread_dimensions;
+	const auto compute_program =
+	    m_context.GetPipelineCache().GetComputeProgram(cs_regs, sh_regs, input_info);
 	if (use_thread_dimensions) {
-		input_info.dispatch_thread_dimensions = true;
 		input_info.dispatch_threads_num[0]    = thread_group_x;
 		input_info.dispatch_threads_num[1]    = thread_group_y;
 		input_info.dispatch_threads_num[2]    = thread_group_z;
@@ -332,9 +327,12 @@ void RenderExecutor::DispatchDirect(uint64_t submit_id, RenderCommandBuffer& buf
 
 	buffer.EndRendering();
 	auto& pipeline =
-	    m_context.GetPipelineCache().CreateComputePipeline(input_info, sh_ctx.GetCs(), cs_shader);
+	    m_context.GetPipelineCache().CreateComputePipeline(input_info, compute_program);
 	auto bindings = PrepareBindings(input_info.stage);
 	FindBuffers(bindings);
+	if (program.info.uses_dma) {
+		m_context.GetGpuResources().PrepareBda();
+	}
 	RebindBuffers(bindings);
 	RebindImages(bindings);
 

src/graphics/host_gpu/renderer/renderContext.cpp

@@ -4,7 +4,6 @@
 #include "common/logging/log.h"
 #include "graphics/guest_gpu/graphicsRun.h"
 #include "graphics/presentation/videoOut.h"
-#include "kernel/pthread.h"
 #include "libs/errno.h"
 
 #include <algorithm>
@@ -39,7 +38,7 @@ void RenderContext::ShutdownGpu() {
 	}
 	if (m_video_out != nullptr) {
 		if (m_command_scheduler.Active()) {
-			m_command_scheduler.FinishCurrent();
+			m_command_scheduler.Finish();
 		}
 		m_command_scheduler.DrainPriorityOperations();
 		m_video_out = nullptr;
@@ -56,60 +55,54 @@ VideoOut::VideoOutDriver& RenderContext::GetVideoOut() const {
 	return *m_video_out;
 }
 
-void RenderContext::AddEopEq(LibKernel::EventQueue::KernelEqueue eq, int id) {
-	auto queue = LibKernel::EventQueue::KernelPinEqueue(eq);
-	if (!queue) {
+void RenderContext::AddInterruptEq(LibKernel::EventQueue::KernelEqueue eq, int event_id) {
+	Common::LockGuard lock(m_interrupt_mutex);
+
+	auto it = std::find_if(
+	    m_interrupt_eqs.begin(), m_interrupt_eqs.end(),
+	    [eq, event_id](const auto& entry) { return entry.eq == eq && entry.event_id == event_id; });
+	if (it != m_interrupt_eqs.end()) {
 		return;
 	}
 
-	Common::LockGuard lock(m_eop_mutex);
-
-	auto it = std::find_if(m_eop_eqs.begin(), m_eop_eqs.end(), [eq, id](const auto& entry) {
-		return entry.eq == eq && entry.id == id;
-	});
-	if (it != m_eop_eqs.end()) {
-		return;
-	}
-
-	m_eop_eqs.push_back({eq, std::move(queue), id});
+	m_interrupt_eqs.push_back({eq, event_id});
 }
 
-void RenderContext::DeleteEopEq(LibKernel::EventQueue::KernelEqueue eq, int id) {
-	Common::LockGuard lock(m_eop_mutex);
+void RenderContext::DeleteInterruptEq(LibKernel::EventQueue::KernelEqueue eq, int event_id) {
+	Common::LockGuard lock(m_interrupt_mutex);
 
-	auto it = std::find_if(m_eop_eqs.begin(), m_eop_eqs.end(), [eq, id](const auto& entry) {
-		return entry.eq == eq && entry.id == id;
-	});
-	if (it == m_eop_eqs.end()) {
+	auto it = std::find_if(
+	    m_interrupt_eqs.begin(), m_interrupt_eqs.end(),
+	    [eq, event_id](const auto& entry) { return entry.eq == eq && entry.event_id == event_id; });
+	if (it == m_interrupt_eqs.end()) {
 		return;
 	}
 
-	m_eop_eqs.erase(it);
+	m_interrupt_eqs.erase(it);
 }
 
-void RenderContext::TriggerEopEvent(uint32_t context_id) {
-	std::vector<EopEqRegistration> registrations;
+void RenderContext::TriggerInterrupt(int event_id, uint32_t context_id) {
+	std::vector<InterruptEqRegistration> registrations;
 	{
-		Common::LockGuard lock(m_eop_mutex);
-		registrations = m_eop_eqs;
+		Common::LockGuard lock(m_interrupt_mutex);
+		for (const auto& registration: m_interrupt_eqs) {
+			if (registration.event_id == event_id) {
+				registrations.push_back(registration);
+			}
+		}
 	}
 
 	for (const auto& registration: registrations) {
 		const auto result = LibKernel::EventQueue::KernelTriggerEvent(
-		    registration.eq, static_cast<uintptr_t>(registration.id),
+		    registration.eq, static_cast<uintptr_t>(registration.event_id),
 		    LibKernel::EventQueue::KERNEL_EVFILT_GRAPHICS,
 		    reinterpret_cast<void*>(static_cast<uintptr_t>(context_id)));
 		if (result == LibKernel::KERNEL_ERROR_EBADF || result == LibKernel::KERNEL_ERROR_ENOENT) {
-			DeleteEopEq(registration.eq, registration.id);
+			DeleteInterruptEq(registration.eq, registration.event_id);
 			continue;
 		}
 		EXIT_NOT_IMPLEMENTED(result != OK);
 	}
-
-	auto tsc    = LibKernel::KernelReadTsc();
-	auto result = LibKernel::EventQueue::KernelTriggerUserEventForAll(AGC_USER_INTERRUPT_EVENT,
-	                                                                  reinterpret_cast<void*>(tsc));
-	EXIT_NOT_IMPLEMENTED(result != OK && result != LibKernel::KERNEL_ERROR_ENOENT);
 }
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/renderContext.h

@@ -23,7 +23,6 @@ class VideoOutDriver;
 
 namespace Libs::Graphics {
 
-constexpr int AGC_USER_INTERRUPT_EVENT = 0x1800;
 class GuestGpu;
 
 class RenderContext {
@@ -48,15 +47,14 @@ public:
 	TextureCache&       GetTextureCache() { return m_gpu_resources.GetTextureCache(); }
 	RenderExecutor&     GetRenderExecutor() { return m_render_executor; }
 
-	void AddEopEq(LibKernel::EventQueue::KernelEqueue eq, int id);
-	void DeleteEopEq(LibKernel::EventQueue::KernelEqueue eq, int id);
-	void TriggerEopEvent(uint32_t context_id);
+	void AddInterruptEq(LibKernel::EventQueue::KernelEqueue eq, int event_id);
+	void DeleteInterruptEq(LibKernel::EventQueue::KernelEqueue eq, int event_id);
+	void TriggerInterrupt(int event_id, uint32_t context_id);
 
 private:
-	struct EopEqRegistration {
-		LibKernel::EventQueue::KernelEqueue    eq = LibKernel::EventQueue::KERNEL_EQUEUE_INVALID;
-		LibKernel::EventQueue::KernelEqueueRef queue;
-		int                                    id = 0;
+	struct InterruptEqRegistration {
+		LibKernel::EventQueue::KernelEqueue eq       = LibKernel::EventQueue::KERNEL_EQUEUE_INVALID;
+		int                                 event_id = 0;
 	};
 
 	GraphicContext&           m_graphics;
@@ -70,8 +68,8 @@ private:
 	std::unique_ptr<GuestGpu> m_gpu;
 	VideoOut::VideoOutDriver* m_video_out = nullptr;
 
-	Common::Mutex                  m_eop_mutex;
-	std::vector<EopEqRegistration> m_eop_eqs;
+	Common::Mutex                        m_interrupt_mutex;
+	std::vector<InterruptEqRegistration> m_interrupt_eqs;
 };
 
 } // namespace Libs::Graphics

src/graphics/host_gpu/renderer/renderDraw.cpp

@@ -103,7 +103,7 @@ static bool IsDualSourceBlendFactor(uint32_t factor) {
 }
 
 static void LogFramebufferSkip(const char* draw_name, const RenderColorInfo& color,
-                               const RenderDepthInfo& depth, const RenderCommandBuffer& buffer,
+                               const RenderDepthInfo& depth, const CommandBuffer& buffer,
                                uint32_t index_count, uint32_t flags) {
 	const auto& ctx  = buffer.GetRegisters();
 	const auto& ucfg = buffer.GetUserConfig();
@@ -126,7 +126,7 @@ static void LogFramebufferSkip(const char* draw_name, const RenderColorInfo& col
 	    static_cast<uint32_t>(ucfg.GetPrimType()), index_count, flags);
 }
 
-static void LogMrtState(const char* draw_name, const RenderCommandBuffer& buffer,
+static void LogMrtState(const char* draw_name, const CommandBuffer& buffer,
                         const ShaderPixelInputInfo& ps_input_info) {
 	const auto& ctx            = buffer.GetRegisters();
 	const auto& sh_regs        = ctx.GetShaderRegisters();
@@ -177,7 +177,7 @@ static void LogMrtState(const char* draw_name, const RenderCommandBuffer& buffer
 }
 
 static void LogDrawTargetState(const char* draw_name, const RenderColorInfo& color,
-                               const RenderDepthInfo& depth, const RenderCommandBuffer& buffer,
+                               const RenderDepthInfo& depth, const CommandBuffer& buffer,
                                const ShaderPixelInputInfo& ps_input_info, uint32_t index_count,
                                uint32_t flags) {
 	const auto& ctx  = buffer.GetRegisters();
@@ -231,7 +231,7 @@ static void LogDrawTargetState(const char* draw_name, const RenderColorInfo& col
 	LogMrtState(draw_name, buffer, ps_input_info);
 }
 
-static void LogDrawInputState(const RenderCommandBuffer& buffer, const RenderColorInfo& color,
+static void LogDrawInputState(const CommandBuffer& buffer, const RenderColorInfo& color,
                               const ShaderVertexInputInfo& vs_input_info,
                               uint32_t index_type_and_size, uint32_t index_count,
                               const void* index_addr) {
@@ -309,17 +309,14 @@ static void LogDrawInputState(const RenderCommandBuffer& buffer, const RenderCol
 	}
 }
 
-static PipelineDynamicParameters BuildGraphicsDynamicParams(const RenderCommandBuffer& buffer,
-                                                            const RenderColorInfo*     colors,
-                                                            uint32_t                   color_count,
-                                                            const RenderDepthInfo&     depth) {
+static void SetGraphicsDynamicParams(const CommandBuffer& buffer, vk::CommandBuffer vk_buffer,
+                                     const RenderColorInfo* colors, uint32_t color_count,
+                                     const RenderDepthInfo& depth) {
+	KYTY_PROFILER_FUNCTION();
+
 	EXIT_IF(colors == nullptr);
 	const auto& ctx = buffer.GetRegisters();
 
-	PipelineDynamicParameters ret {};
-	ret.color_write_count   = color_count;
-	ret.stencil_test_enable = depth.stencil_test_enable;
-
 	const auto&  vp = ctx.GetScreenViewport();
 	vk::Extent2D framebuffer_extent {};
 	if (color_count > 0 && colors[0].image_id) {
@@ -333,68 +330,22 @@ static PipelineDynamicParameters BuildGraphicsDynamicParams(const RenderCommandB
 
 	const auto final_scissor = calc_final_scissor(vp, ctx.GetScanModeControl(), framebuffer_extent);
 
-	ret.viewport_scale[0]  = vp.viewports[0].xscale;
-	ret.viewport_scale[1]  = vp.viewports[0].yscale;
-	ret.viewport_scale[2]  = vp.viewports[0].zscale;
-	ret.viewport_offset[0] = vp.viewports[0].xoffset;
-	ret.viewport_offset[1] = vp.viewports[0].yoffset;
-	ret.viewport_offset[2] = vp.viewports[0].zoffset;
-	ret.scissor_ltrb[0]    = final_scissor.left;
-	ret.scissor_ltrb[1]    = final_scissor.top;
-	ret.scissor_ltrb[2]    = final_scissor.right;
-	ret.scissor_ltrb[3]    = final_scissor.bottom;
-	ret.line_width         = ctx.GetLineWidth();
-	ret.stencil_front      = depth.stencil_dynamic_front;
-	ret.stencil_back       = depth.stencil_dynamic_back;
-
-	const auto& mode        = ctx.GetModeControl();
-	const auto& poly_offset = ctx.GetPolyOffset();
-	const bool  use_front   = mode.poly_offset_front_enable && !mode.cull_front;
-	const bool  use_back    = mode.poly_offset_back_enable && !mode.cull_back;
-	ret.depth_bias_enable   = use_front || use_back;
-	if (ret.depth_bias_enable) {
-		// Vulkan has one bias for both faces. Prefer a visible front face when both are enabled.
-		const float guest_constant_factor =
-		    use_front ? poly_offset.front_offset : poly_offset.back_offset;
-		ret.depth_bias_constant_factor =
-		    ConvertPolygonOffsetConstantFactor(guest_constant_factor, poly_offset, depth.format);
-		ret.depth_bias_clamp = poly_offset.clamp;
-		ret.depth_bias_slope_factor =
-		    (use_front ? poly_offset.front_scale : poly_offset.back_scale) / 16.0f;
-	}
-
-	// Color-control operation selects special color-buffer paths, not the normal component write
-	// mask. Attachment availability therefore follows the target write mask.
-	for (uint32_t i = 0; i < RENDER_COLOR_ATTACHMENTS_MAX; i++) {
-		ret.color_write_enable[i] =
-		    (i < color_count &&
-		     render_target_mask_slot(ctx.GetRenderTargetMask(), colors[i].target_slot) != 0);
-	}
-
-	return ret;
-}
-
-static void SetDynamicParams(const RenderCommandBuffer& buffer, vk::CommandBuffer vk_buffer,
-                             const PipelineDynamicParameters& dynamic_params) {
-	KYTY_PROFILER_FUNCTION();
-
 	vk::Viewport viewport {};
-	viewport.x        = dynamic_params.viewport_offset[0] - dynamic_params.viewport_scale[0];
-	viewport.y        = dynamic_params.viewport_offset[1] - dynamic_params.viewport_scale[1];
-	viewport.width    = dynamic_params.viewport_scale[0] * 2.0f;
-	viewport.height   = dynamic_params.viewport_scale[1] * 2.0f;
-	viewport.minDepth = dynamic_params.viewport_offset[2];
-	viewport.maxDepth = dynamic_params.viewport_scale[2] + dynamic_params.viewport_offset[2];
+	viewport.x        = vp.viewports[0].xoffset - vp.viewports[0].xscale;
+	viewport.y        = vp.viewports[0].yoffset - vp.viewports[0].yscale;
+	viewport.width    = vp.viewports[0].xscale * 2.0f;
+	viewport.height   = vp.viewports[0].yscale * 2.0f;
+	viewport.minDepth = vp.viewports[0].zoffset;
+	viewport.maxDepth = vp.viewports[0].zscale + vp.viewports[0].zoffset;
 	vk_buffer.setViewport(0, 1, &viewport);
 
 	vk::Rect2D scissor {};
-	scissor.offset = {dynamic_params.scissor_ltrb[0], dynamic_params.scissor_ltrb[1]};
-	scissor.extent = {
-	    static_cast<uint32_t>(dynamic_params.scissor_ltrb[2] - dynamic_params.scissor_ltrb[0]),
-	    static_cast<uint32_t>(dynamic_params.scissor_ltrb[3] - dynamic_params.scissor_ltrb[1])};
+	scissor.offset = {final_scissor.left, final_scissor.top};
+	scissor.extent = {static_cast<uint32_t>(final_scissor.right - final_scissor.left),
+	                  static_cast<uint32_t>(final_scissor.bottom - final_scissor.top)};
 	vk_buffer.setScissor(0, 1, &scissor);
 
-	float line_width = dynamic_params.line_width;
+	float line_width = ctx.GetLineWidth();
 	if (line_width != 1.0f) {
 		static bool logged = false;
 		if (!logged) {
@@ -407,26 +358,36 @@ static void SetDynamicParams(const RenderCommandBuffer& buffer, vk::CommandBuffe
 	}
 	vk_buffer.setLineWidth(line_width);
 
-	vk_buffer.setDepthBiasEnable(dynamic_params.depth_bias_enable ? VK_TRUE : VK_FALSE);
-	if (dynamic_params.depth_bias_enable) {
-		vk_buffer.setDepthBias(dynamic_params.depth_bias_constant_factor,
-		                       dynamic_params.depth_bias_clamp,
-		                       dynamic_params.depth_bias_slope_factor);
+	const auto& mode              = ctx.GetModeControl();
+	const auto& poly_offset       = ctx.GetPolyOffset();
+	const bool  use_front         = mode.poly_offset_front_enable && !mode.cull_front;
+	const bool  use_back          = mode.poly_offset_back_enable && !mode.cull_back;
+	const bool  depth_bias_enable = use_front || use_back;
+	vk_buffer.setDepthBiasEnable(depth_bias_enable ? VK_TRUE : VK_FALSE);
+	if (depth_bias_enable) {
+		// Vulkan has one bias for both faces. Prefer a visible front face when both are enabled.
+		const float guest_constant_factor =
+		    use_front ? poly_offset.front_offset : poly_offset.back_offset;
+		const float constant_factor =
+		    ConvertPolygonOffsetConstantFactor(guest_constant_factor, poly_offset, depth.format);
+		const float slope_factor =
+		    (use_front ? poly_offset.front_scale : poly_offset.back_scale) / 16.0f;
+		vk_buffer.setDepthBias(constant_factor, poly_offset.clamp, slope_factor);
 	}
 
-	if (dynamic_params.stencil_test_enable) {
+	if (depth.stencil_test_enable) {
 		vk_buffer.setStencilCompareMask(vk::StencilFaceFlagBits::eFront,
-		                                dynamic_params.stencil_front.compareMask);
+		                                depth.stencil_dynamic_front.compareMask);
 		vk_buffer.setStencilCompareMask(vk::StencilFaceFlagBits::eBack,
-		                                dynamic_params.stencil_back.compareMask);
+		                                depth.stencil_dynamic_back.compareMask);
 		vk_buffer.setStencilWriteMask(vk::StencilFaceFlagBits::eFront,
-		                              dynamic_params.stencil_front.writeMask);
+		                              depth.stencil_dynamic_front.writeMask);
 		vk_buffer.setStencilWriteMask(vk::StencilFaceFlagBits::eBack,
-		                              dynamic_params.stencil_back.writeMask);
+		                              depth.stencil_dynamic_back.writeMask);
 		vk_buffer.setStencilReference(vk::StencilFaceFlagBits::eFront,
-		                              dynamic_params.stencil_front.reference);
+		                              depth.stencil_dynamic_front.reference);
 		vk_buffer.setStencilReference(vk::StencilFaceFlagBits::eBack,
-		                              dynamic_params.stencil_back.reference);
+		                              depth.stencil_dynamic_back.reference);
 	}
 
 #if defined(__APPLE__)
@@ -434,11 +395,15 @@ static void SetDynamicParams(const RenderCommandBuffer& buffer, vk::CommandBuffe
 	// eColorWriteEnableEXT dynamic state and relies on the static colorWriteMask instead.
 #else
 	vk::Bool32 enable[RENDER_COLOR_ATTACHMENTS_MAX] = {};
-	for (uint32_t i = 0; i < dynamic_params.color_write_count; i++) {
-		enable[i] = (dynamic_params.color_write_enable[i] ? VK_TRUE : VK_FALSE);
+	// Color-control operation selects special color-buffer paths, not the normal component write
+	// mask. Attachment availability therefore follows the target write mask.
+	for (uint32_t i = 0; i < color_count; i++) {
+		enable[i] = render_target_mask_slot(ctx.GetRenderTargetMask(), colors[i].target_slot) != 0
+		                ? VK_TRUE
+		                : VK_FALSE;
 	}
-	if (dynamic_params.color_write_count != 0) {
-		vk_buffer.setColorWriteEnableEXT(dynamic_params.color_write_count, enable);
+	if (color_count != 0) {
+		vk_buffer.setColorWriteEnableEXT(color_count, enable);
 	}
 #endif
 }
@@ -456,7 +421,7 @@ static bool PixelShaderHasDepthOrCoverageSideEffects(const HW::ShaderRegisters&
 	       db.shader_execute_on_noop;
 }
 
-static bool ShouldSkipGeShader(const RenderCommandBuffer& buffer) {
+static bool ShouldSkipGeShader(const CommandBuffer& buffer) {
 	const auto& ctx         = buffer.GetRegisters();
 	const auto& ucfg        = buffer.GetUserConfig();
 	const auto& sh_ctx      = buffer.GetShaders();
@@ -485,15 +450,14 @@ static bool ShouldSkipGeShader(const RenderCommandBuffer& buffer) {
 	const bool unsupported_stage_mask = (stages != 0 && stages != 0x02002000);
 	const bool unsupported_gs_stage = (vertex_info.es_regs.data_addr != 0 &&
 	                                   vertex_info.gs_regs.data_addr != 0 && !ps5_ngg_vertex_path);
-	const bool ge_group_size =
-	    ge_cntl.primitive_group_size > 0x0040 || ge_cntl.vertex_group_size > 0x0040;
+	// GE_CNTL group sizes control guest scheduling and do not constrain the host vertex path.
 	const bool ge_shader_regs =
 	    (sh_regs.m_geNggSubgrpCntl != 0x00000000 && sh_regs.m_geNggSubgrpCntl != 0x00000001) ||
 	    sh_regs.m_vgtGsMaxVertOut != 0x00000000 ||
 	    !is_known_gs_out_prim_type(sh_regs.m_vgtGsOutPrimType) ||
 	    sh_regs.m_geMaxOutputPerSubgroup > 0x00000040;
 
-	if (unsupported_stage_mask || unsupported_gs_stage || ge_group_size || ge_shader_regs) {
+	if (unsupported_stage_mask || unsupported_gs_stage || ge_shader_regs) {
 		static std::once_flag warning_once;
 		std::call_once(warning_once, [] {
 			std::printf("Warning: game uses unsupported graphics pipelines; some draw calls were "
@@ -518,15 +482,15 @@ static bool ShouldSkipGeShader(const RenderCommandBuffer& buffer) {
 }
 
 struct DrawRenderState {
-	RenderDepthInfo           depth_info;
-	RenderColorInfo           color_info[RENDER_COLOR_ATTACHMENTS_MAX] = {};
-	uint32_t                  color_count                              = 0;
-	bool                      ps_active                                = true;
-	RenderState               rendering;
-	ShaderVertexInputInfo     vs_input_info;
-	ShaderPixelInputInfo      ps_input_info;
-	std::span<const uint32_t> vs_shader;
-	std::span<const uint32_t> ps_shader;
+	RenderDepthInfo       depth_info;
+	RenderColorInfo       color_info[RENDER_COLOR_ATTACHMENTS_MAX] = {};
+	uint32_t              color_count                              = 0;
+	bool                  ps_active                                = true;
+	RenderState           rendering;
+	ShaderVertexInputInfo vs_input_info;
+	ShaderPixelInputInfo  ps_input_info;
+	ShaderProgram         vertex_program;
+	ShaderProgram         pixel_program;
 };
 
 struct DrawCallInfo {
@@ -741,11 +705,12 @@ RenderState RenderExecutor::AcquireRenderTargets(CommandBuffer& buffer, RenderCo
 	return state;
 }
 
-static bool DrawHasActivePixelShader(const RenderCommandBuffer& buffer) {
+static bool DrawHasActivePixelShader(const CommandBuffer& buffer) {
 	const auto& ctx              = buffer.GetRegisters();
 	const auto& sh_regs          = ctx.GetShaderRegisters();
 	const bool  has_color_output = (ctx.GetRenderTargetMask() & sh_regs.m_cbShaderMask) != 0;
-	return has_color_output || PixelShaderHasDepthOrCoverageSideEffects(sh_regs);
+	return ShaderAddressValid(buffer.GetShaders().GetPs().ps_regs.data_addr) &&
+	       (has_color_output || PixelShaderHasDepthOrCoverageSideEffects(sh_regs));
 }
 
 enum class CbColorMode : uint8_t {
@@ -757,7 +722,7 @@ enum class CbColorMode : uint8_t {
 	DccDecompress      = 6,
 };
 
-static bool ConsumeMetadataColorOperation(const RenderCommandBuffer& buffer) {
+static bool ConsumeMetadataColorOperation(const CommandBuffer& buffer) {
 	const auto& ctx  = buffer.GetRegisters();
 	const auto  mode = ctx.GetColorControl().mode;
 	// These special modes run color-buffer metadata or decompression operations. The shader is a
@@ -813,7 +778,7 @@ struct PreparedVertexBuffers {
 	uint32_t                               count = 0;
 };
 
-static PreparedVertexBuffers AcquireVertexBuffers(RenderCommandBuffer&         buffer,
+static PreparedVertexBuffers AcquireVertexBuffers(CommandBuffer&               buffer,
                                                   const ShaderVertexInputInfo& vs_input_info) {
 	EXIT_IF(vs_input_info.buffers_num < 0 ||
 	        vs_input_info.buffers_num > ShaderVertexInputInfo::RES_MAX);
@@ -903,8 +868,8 @@ static PreparedVertexBuffers AcquireVertexBuffers(RenderCommandBuffer&         b
 	return prepared;
 }
 
-static void SetDrawDebugPhase(RenderCommandBuffer& buffer, uint64_t submit_id,
-                              const DrawCallInfo& draw, uint32_t phase) {
+static void SetDrawDebugPhase(CommandBuffer& buffer, uint64_t submit_id, const DrawCallInfo& draw,
+                              uint32_t phase) {
 	EXIT_IF(draw.name == nullptr);
 
 	buffer.SetDebugInfo(static_cast<uint32_t>(draw.debug_op), submit_id, phase, draw.index_count,
@@ -952,8 +917,8 @@ static bool GetDrawTopology(const HW::UserConfig& ucfg, bool auto_draw,
 	return true;
 }
 
-static bool ResolvePrimitiveRestart(const RenderCommandBuffer& buffer,
-                                    vk::PrimitiveTopology topology, uint32_t index_type_and_size) {
+static bool ResolvePrimitiveRestart(const CommandBuffer& buffer, vk::PrimitiveTopology topology,
+                                    uint32_t index_type_and_size) {
 	const auto control = buffer.GetUserConfig().GetPrimitiveResetControl();
 	EXIT_NOT_IMPLEMENTED((control & ~0x3u) != 0);
 	if ((control & 0x1u) == 0) {
@@ -987,7 +952,7 @@ static bool ResolvePrimitiveRestart(const RenderCommandBuffer& buffer,
 	return true;
 }
 
-bool RenderExecutor::PrepareDrawRenderState(uint64_t submit_id, RenderCommandBuffer& buffer,
+bool RenderExecutor::PrepareDrawRenderState(uint64_t submit_id, CommandBuffer& buffer,
                                             const DrawCallInfo& draw,
                                             uint32_t            render_target_slice_offset,
                                             bool log_setup_phases, DrawRenderState& state) {
@@ -1027,7 +992,7 @@ bool RenderExecutor::PrepareDrawRenderState(uint64_t submit_id, RenderCommandBuf
 	return true;
 }
 
-static void RefreshShaders(RenderCommandBuffer& buffer, const DrawCallInfo& draw, bool log_phases,
+static void RefreshShaders(CommandBuffer& buffer, const DrawCallInfo& draw, bool log_phases,
                            DrawRenderState& state) {
 	EXIT_IF(draw.name == nullptr);
 	auto& ctx    = buffer.GetRegisters();
@@ -1037,36 +1002,33 @@ static void RefreshShaders(RenderCommandBuffer& buffer, const DrawCallInfo& draw
 	const auto& pixel_shader_info  = sh_ctx.GetPs();
 	const auto& shader_regs        = ctx.GetShaderRegisters();
 
-	state.vs_shader     = {};
-	state.ps_shader     = {};
-	state.ps_input_info = {};
+	state.vertex_program = {};
+	state.pixel_program  = {};
+	state.ps_input_info  = {};
 	std::array<Prospero::ColorComponentMapping, RENDER_COLOR_ATTACHMENTS_MAX>
 	    target_export_mapping {};
 	for (uint32_t i = 0; i < state.color_count; i++) {
 		target_export_mapping[state.color_info[i].target_slot] = state.color_info[i].export_mapping;
 	}
 	if (log_phases) {
-		LogDrawPhase(draw.name, "ShaderCompileInfoVS");
-	}
-	if (!ShaderCompileInfoVS(vertex_shader_info, shader_regs, state.vs_input_info,
-	                         state.vs_shader)) {
-		EXIT("ShaderCompileInfoVS failed for draw %s\n", draw.name);
+		LogDrawPhase(draw.name, "GetVertexProgram");
 	}
+	auto& pipeline_cache = buffer.GetContext().GetPipelineCache();
+	state.vertex_program =
+	    pipeline_cache.GetVertexProgram(vertex_shader_info, shader_regs, state.vs_input_info);
 
 	if (!state.ps_active) {
 		return;
 	}
 	if (log_phases) {
-		LogDrawPhase(draw.name, "ShaderCompileInfoPS");
-	}
-	if (!ShaderCompileInfoPS(pixel_shader_info, shader_regs, state.vs_input_info,
-	                         target_export_mapping,
-	                         state.ps_input_info, state.ps_shader)) {
-		EXIT("ShaderCompileInfoPS failed for draw %s\n", draw.name);
+		LogDrawPhase(draw.name, "GetPixelProgram");
 	}
+	state.pixel_program =
+	    pipeline_cache.GetPixelProgram(pixel_shader_info, shader_regs, state.vs_input_info,
+	                                   target_export_mapping, state.ps_input_info);
 }
 
-static PreparedVertexBuffers PrepareVertexBuffers(uint64_t submit_id, RenderCommandBuffer& buffer,
+static PreparedVertexBuffers PrepareVertexBuffers(uint64_t submit_id, CommandBuffer& buffer,
                                                   const DrawCallInfo&          draw,
                                                   const ShaderVertexInputInfo& vs_input_info) {
 	EXIT_IF(draw.name == nullptr);
@@ -1076,7 +1038,7 @@ static PreparedVertexBuffers PrepareVertexBuffers(uint64_t submit_id, RenderComm
 	return AcquireVertexBuffers(buffer, vs_input_info);
 }
 
-static PreparedIndexBuffer PrepareIndexBuffer(RenderCommandBuffer&         buffer,
+static PreparedIndexBuffer PrepareIndexBuffer(CommandBuffer&               buffer,
                                               const DrawIndexBufferSource& source) {
 	PreparedIndexBuffer prepared;
 	if (!source.enabled) {
@@ -1126,7 +1088,7 @@ static void CommitIndexBuffer(vk::CommandBuffer vk_buffer, const PreparedIndexBu
 	vk_buffer.bindIndexBuffer(prepared.buffer, prepared.offset, prepared.type);
 }
 
-static void LogDrawStateIfNeeded(const RenderCommandBuffer& buffer, const DrawCallInfo& draw,
+static void LogDrawStateIfNeeded(const CommandBuffer& buffer, const DrawCallInfo& draw,
                                  const DrawRenderState& state, bool always_log,
                                  bool force_legacy_rect_log, uint32_t index_type_and_size,
                                  const void* index_addr) {
@@ -1194,7 +1156,7 @@ static void EmitDrawPrimitives(const HW::UserConfig& ucfg, vk::CommandBuffer vk_
 	}
 }
 
-void RenderExecutor::ExecutePreparedDraw(uint64_t submit_id, RenderCommandBuffer& buffer,
+void RenderExecutor::ExecutePreparedDraw(uint64_t submit_id, CommandBuffer& buffer,
                                          const DrawCallInfo& draw, DrawRenderState& state,
                                          vk::PrimitiveTopology topology, const DrawEmitInfo& emit,
                                          const DrawIndexBufferSource& index_source,
@@ -1215,9 +1177,9 @@ void RenderExecutor::ExecutePreparedDraw(uint64_t submit_id, RenderCommandBuffer
 		LogDrawPhase(draw.name, "CreatePipeline");
 	}
 	auto& pipeline = m_context.GetPipelineCache().CreateGraphicsPipeline(
-	    state.color_info, state.color_count, state.depth_info, state.vs_input_info, buffer,
-	    &state.ps_input_info, topology, primitive_restart_enable, state.ps_active, state.vs_shader,
-	    state.ps_shader);
+	    std::span {state.color_info, state.color_count}, state.depth_info, state.vs_input_info, buffer,
+	    state.ps_active ? &state.ps_input_info : nullptr, topology, primitive_restart_enable,
+	    state.vertex_program, state.pixel_program);
 
 	// Resource preparation above may synchronously finish and restart the scheduler. From this
 	// point onward, every operation targets the current command buffer and cannot touch guest
@@ -1244,9 +1206,8 @@ void RenderExecutor::ExecutePreparedDraw(uint64_t submit_id, RenderCommandBuffer
 	               std::span {descriptor_stages.data(), descriptor_stage_count});
 	CommitIndexBuffer(vk_buffer, index_binding);
 
-	const auto dynamic_params =
-	    BuildGraphicsDynamicParams(buffer, state.color_info, state.color_count, state.depth_info);
-	SetDynamicParams(buffer, vk_buffer, dynamic_params);
+	SetGraphicsDynamicParams(buffer, vk_buffer, state.color_info, state.color_count,
+	                         state.depth_info);
 
 	LogDrawPhase(draw.name, "BeginRendering");
 	if (set_auto_debug) {
@@ -1279,7 +1240,7 @@ void RenderExecutor::ExecutePreparedDraw(uint64_t submit_id, RenderCommandBuffer
 	}
 }
 
-void RenderExecutor::DrawIndex(uint64_t submit_id, RenderCommandBuffer& buffer,
+void RenderExecutor::DrawIndex(uint64_t submit_id, CommandBuffer& buffer,
                                uint32_t index_type_and_size, uint32_t index_count,
                                const void* index_addr, uint32_t flags, uint32_t type,
                                uint32_t instance_count, uint32_t render_target_slice_offset,
@@ -1412,7 +1373,7 @@ void RenderExecutor::DrawIndex(uint64_t submit_id, RenderCommandBuffer& buffer,
 }
 
 // NOLINTNEXTLINE(readability-function-cognitive-complexity)
-void RenderExecutor::DrawAuto(uint64_t submit_id, RenderCommandBuffer& buffer, uint32_t index_count,
+void RenderExecutor::DrawAuto(uint64_t submit_id, CommandBuffer& buffer, uint32_t index_count,
                               uint32_t flags, uint32_t render_target_slice_offset,
                               uint32_t instance_count, uint32_t first_vertex,
                               uint32_t first_instance) {
@@ -1511,7 +1472,7 @@ void RenderExecutor::DrawAuto(uint64_t submit_id, RenderCommandBuffer& buffer, u
 	ResetBindings();
 }
 
-bool RenderExecutor::ResolveColorTargets(uint64_t submit_id, RenderCommandBuffer& buffer,
+bool RenderExecutor::ResolveColorTargets(uint64_t submit_id, CommandBuffer& buffer,
                                          uint32_t render_target_slice_offset) {
 	const auto& hw = buffer.GetRegisters();
 	if (hw.GetColorControl().mode != 3) {

src/graphics/host_gpu/renderer/renderDraw.h

@@ -1,43 +1,12 @@
 #ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_RENDERDRAW_H_
 #define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_RENDERDRAW_H_
 
-#include "graphics/host_gpu/renderer/renderTarget.h"
-
 #include <cstdint>
-#include <type_traits>
 
 namespace Libs::Graphics {
 
-struct RenderColorInfo;
 struct ShaderVertexInputInfo;
 
-#pragma pack(push, 1)
-
-struct PipelineDynamicParameters {
-	bool stencil_test_enable = false;
-	bool depth_bias_enable   = false;
-
-	float viewport_scale[3]  = {};
-	float viewport_offset[3] = {};
-	int   scissor_ltrb[4]    = {0};
-
-	float    line_width                                       = 1.0f;
-	float    depth_bias_constant_factor                       = 0.0f;
-	float    depth_bias_clamp                                 = 0.0f;
-	float    depth_bias_slope_factor                          = 0.0f;
-	uint32_t color_write_count                                = 1;
-	bool     color_write_enable[RENDER_COLOR_ATTACHMENTS_MAX] = {true, true};
-
-	PipelineStencilDynamicState stencil_front;
-	PipelineStencilDynamicState stencil_back;
-};
-
-#pragma pack(pop)
-
-static_assert(std::is_trivially_copyable_v<PipelineDynamicParameters>);
-static_assert(std::is_standard_layout_v<PipelineDynamicParameters>);
-static_assert(alignof(PipelineDynamicParameters) == 1);
-
 [[nodiscard]] int32_t ResolveVertexOffset(uint32_t                     index_offset,
                                           const ShaderVertexInputInfo& vs_input_info);
 

src/graphics/host_gpu/renderer/sync.cpp

@@ -20,9 +20,7 @@
 
 namespace Libs::Graphics::Sync {
 
-constexpr int      GRAPHICS_EVENT_QUEUED_GRAPHICS_INTERRUPT = 0x00;
-constexpr int      GRAPHICS_EVENT_EOP                       = 0x40;
-constexpr uint64_t GRAPHICS_REFERENCE_CLOCK_FREQUENCY       = 100000000;
+constexpr uint64_t GRAPHICS_REFERENCE_CLOCK_FREQUENCY = 100000000;
 
 bool ScaleReferenceClock(uint64_t host_ticks, uint64_t host_frequency, uint64_t& value) {
 	if (host_frequency == 0) {
@@ -66,8 +64,9 @@ struct EndOfPipeSignal {
 	std::array<uint32_t, 4> debug_args      = {};
 	uint64_t                debug_data      = 0;
 	std::optional<uint64_t> destination;
-	EndOfPipeCompletion     completion      = EndOfPipeCompletion::None;
-	uint64_t                completion_data = 0;
+	EndOfPipeCompletion     completion         = EndOfPipeCompletion::None;
+	uint64_t                completion_data    = 0;
+	int                     interrupt_event_id = 0;
 };
 
 enum class EndOfPipeWriteSize : uint32_t { Dword = 4, Qword = 8 };
@@ -96,8 +95,10 @@ static void RecordEndOfPipeSignal(const EndOfPipeSignal& signal) {
 		case EndOfPipeCompletion::None: return;
 		case EndOfPipeCompletion::Interrupt: {
 			const auto context_id = static_cast<uint32_t>(signal.completion_data);
-			scheduler.DeferPriorityOperation(
-			    [&renderer, context_id] { renderer.TriggerEopEvent(context_id); });
+			const auto event_id   = signal.interrupt_event_id;
+			scheduler.DeferPriorityOperation([&renderer, event_id, context_id] {
+				renderer.TriggerInterrupt(event_id, context_id);
+			});
 			return;
 		}
 		case EndOfPipeCompletion::Flip: {
@@ -108,9 +109,10 @@ static void RecordEndOfPipeSignal(const EndOfPipeSignal& signal) {
 		}
 		case EndOfPipeCompletion::FlipAndInterrupt: {
 			const auto request_id = signal.completion_data;
-			scheduler.DeferPriorityOperation([&renderer, request_id] {
+			const auto event_id   = signal.interrupt_event_id;
+			scheduler.DeferPriorityOperation([&renderer, event_id, request_id] {
 				renderer.GetVideoOut().CompleteFlip(request_id);
-				renderer.TriggerEopEvent(0);
+				renderer.TriggerInterrupt(event_id, 0);
 			});
 			return;
 		}
@@ -135,7 +137,8 @@ static bool TriggersInterrupt(EndOfPipeWriteAction action) {
 
 static void RecordEndOfPipeWrite(uint64_t submit_id, CommandBuffer& buffer, uint64_t destination,
                                  uint64_t value, EndOfPipeWriteSize size,
-                                 EndOfPipeWriteAction action, uint32_t context_id = 0) {
+                                 EndOfPipeWriteAction action, int interrupt_event_id = 0,
+                                 uint32_t context_id = 0) {
 	const auto width      = static_cast<uint32_t>(size);
 	const auto value_low  = static_cast<uint32_t>(value);
 	const auto value_high = static_cast<uint32_t>(value >> 32u);
@@ -150,18 +153,12 @@ static void RecordEndOfPipeWrite(uint64_t submit_id, CommandBuffer& buffer, uint
 	    .debug_data      = destination,
 	    .destination     = destination,
 	    .completion      = interrupt ? EndOfPipeCompletion::Interrupt : EndOfPipeCompletion::None,
-	    .completion_data = context_id != 0 ? context_id : value,
+	    .completion_data = interrupt ? context_id : value,
+	    .interrupt_event_id = interrupt_event_id,
 	};
 	RecordEndOfPipeSignal(signal);
 }
 
-void TriggerAgcUserInterrupt() {
-	auto tsc    = LibKernel::KernelReadTsc();
-	auto result = LibKernel::EventQueue::KernelTriggerUserEventForAll(AGC_USER_INTERRUPT_EVENT,
-	                                                                  reinterpret_cast<void*>(tsc));
-	EXIT_NOT_IMPLEMENTED(result != OK && result != LibKernel::KERNEL_ERROR_ENOENT);
-}
-
 void WriteAtEndOfPipe32(uint64_t submit_id, CommandBuffer& buffer, uint32_t* dst_gpu_addr,
                         uint32_t value) {
 	RecordEndOfPipeWrite(submit_id, buffer, reinterpret_cast<uint64_t>(dst_gpu_addr), value,
@@ -220,31 +217,35 @@ void WriteAtEndOfPipeWithWriteBack32(uint64_t submit_id, CommandBuffer& buffer,
 }
 
 void WriteAtEndOfPipeWithInterruptWriteBack64(uint64_t submit_id, CommandBuffer& buffer,
-                                              uint64_t* dst_gpu_addr, uint64_t value,
+                                              uint64_t* dst_gpu_addr, uint64_t value, int event_id,
                                               uint32_t context_id) {
 	RecordEndOfPipeWrite(submit_id, buffer, reinterpret_cast<uint64_t>(dst_gpu_addr), value,
 	                     EndOfPipeWriteSize::Qword, EndOfPipeWriteAction::InterruptWriteBack,
-	                     context_id);
+	                     event_id, context_id);
 }
 
 void WriteAtEndOfPipeWithInterruptWriteBack32(uint64_t submit_id, CommandBuffer& buffer,
-                                              uint32_t* dst_gpu_addr, uint32_t value,
+                                              uint32_t* dst_gpu_addr, uint32_t value, int event_id,
                                               uint32_t context_id) {
 	RecordEndOfPipeWrite(submit_id, buffer, reinterpret_cast<uint64_t>(dst_gpu_addr), value,
 	                     EndOfPipeWriteSize::Dword, EndOfPipeWriteAction::InterruptWriteBack,
-	                     context_id);
+	                     event_id, context_id);
 }
 
 void WriteAtEndOfPipeWithInterrupt64(uint64_t submit_id, CommandBuffer& buffer,
-                                     uint64_t* dst_gpu_addr, uint64_t value, uint32_t context_id) {
+                                     uint64_t* dst_gpu_addr, uint64_t value, int event_id,
+                                     uint32_t context_id) {
 	RecordEndOfPipeWrite(submit_id, buffer, reinterpret_cast<uint64_t>(dst_gpu_addr), value,
-	                     EndOfPipeWriteSize::Qword, EndOfPipeWriteAction::Interrupt, context_id);
+	                     EndOfPipeWriteSize::Qword, EndOfPipeWriteAction::Interrupt, event_id,
+	                     context_id);
 }
 
 void WriteAtEndOfPipeWithInterrupt32(uint64_t submit_id, CommandBuffer& buffer,
-                                     uint32_t* dst_gpu_addr, uint32_t value, uint32_t context_id) {
+                                     uint32_t* dst_gpu_addr, uint32_t value, int event_id,
+                                     uint32_t context_id) {
 	RecordEndOfPipeWrite(submit_id, buffer, reinterpret_cast<uint64_t>(dst_gpu_addr), value,
-	                     EndOfPipeWriteSize::Dword, EndOfPipeWriteAction::Interrupt, context_id);
+	                     EndOfPipeWriteSize::Dword, EndOfPipeWriteAction::Interrupt, event_id,
+	                     context_id);
 }
 
 uint64_t PrepareVideoOutFlip(CommandBuffer& buffer, int handle, int index, int flip_mode,
@@ -270,18 +271,20 @@ uint64_t PrepareVideoOutFlip(CommandBuffer& buffer, int handle, int index, int f
 void WriteAtEndOfPipeWithInterruptWriteBackFlip32(uint64_t submit_id, CommandBuffer& buffer,
                                                   uint32_t* dst_gpu_addr, uint32_t value,
                                                   int handle, int index, int flip_mode,
-                                                  int64_t flip_arg, uint64_t request_id) {
+                                                  int64_t flip_arg, uint64_t request_id,
+                                                  int event_id) {
 	const auto destination = reinterpret_cast<uint64_t>(dst_gpu_addr);
 	RecordEndOfPipeSignal({
-	    .buffer          = &buffer,
-	    .submit_id       = submit_id,
-	    .debug_operation = CommandBufferDebugOp::EopWriteBackFlip,
-	    .debug_args      = {static_cast<uint32_t>(handle), static_cast<uint32_t>(index),
-	                        static_cast<uint32_t>(flip_mode), value},
-	    .debug_data      = static_cast<uint64_t>(flip_arg),
-	    .destination     = destination,
-	    .completion      = EndOfPipeCompletion::FlipAndInterrupt,
-	    .completion_data = request_id,
+	    .buffer             = &buffer,
+	    .submit_id          = submit_id,
+	    .debug_operation    = CommandBufferDebugOp::EopWriteBackFlip,
+	    .debug_args         = {static_cast<uint32_t>(handle), static_cast<uint32_t>(index),
+	                           static_cast<uint32_t>(flip_mode), value},
+	    .debug_data         = static_cast<uint64_t>(flip_arg),
+	    .destination        = destination,
+	    .completion         = EndOfPipeCompletion::FlipAndInterrupt,
+	    .completion_data    = request_id,
+	    .interrupt_event_id = event_id,
 	});
 }
 
@@ -316,36 +319,24 @@ void WriteAtEndOfPipeOnlyFlip(uint64_t submit_id, CommandBuffer& buffer, int han
 	});
 }
 
-void TriggerEopEventAtEndOfPipe(CommandBuffer& buffer, uint32_t context_id) {
+void TriggerEopEventAtEndOfPipe(CommandBuffer& buffer, int event_id, uint32_t context_id) {
 	ValidateEndOfPipeSignal({.buffer = &buffer});
 	auto& renderer  = buffer.GetContext();
 	auto& scheduler = renderer.GetCommandScheduler();
 	EXIT_IF(!scheduler.Active() || &buffer != &scheduler.Current());
 	scheduler.DeferPriorityOperation(
-	    [&renderer, context_id] { renderer.TriggerEopEvent(context_id); });
+	    [&renderer, event_id, context_id] { renderer.TriggerInterrupt(event_id, context_id); });
 }
 
-static void EopEventResetFunc(LibKernel::EventQueue::KernelEqueueEvent* event) {
+static void InterruptEventResetFunc(LibKernel::EventQueue::KernelEqueueEvent* event) {
 	EXIT_IF(event == nullptr);
 	event->triggered    = false;
 	event->event.fflags = 0;
 	event->event.data   = 0;
 }
 
-static void EopEventDeleteFunc(LibKernel::EventQueue::KernelEqueue       eq,
-                               LibKernel::EventQueue::KernelEqueueEvent* event) {
-	EXIT_IF(event == nullptr);
-	EXIT_NOT_IMPLEMENTED(event->event.filter != LibKernel::EventQueue::KERNEL_EVFILT_GRAPHICS);
-	if (event->event.ident == GRAPHICS_EVENT_QUEUED_GRAPHICS_INTERRUPT ||
-	    event->event.ident == GRAPHICS_EVENT_EOP) {
-		auto* renderer = static_cast<RenderContext*>(event->filter.data);
-		EXIT_IF(renderer == nullptr);
-		renderer->DeleteEopEq(eq, static_cast<int>(event->event.ident));
-	}
-}
-
-static void EopEventTriggerFunc(LibKernel::EventQueue::KernelEqueueEvent* event,
-                                void*                                     trigger_data) {
+static void InterruptEventTriggerFunc(LibKernel::EventQueue::KernelEqueueEvent* event,
+                                      void*                                     trigger_data) {
 	EXIT_IF(event == nullptr);
 
 	auto triggered_event = event->event;
@@ -362,30 +353,30 @@ static void EopEventTriggerFunc(LibKernel::EventQueue::KernelEqueueEvent* event,
 int AddEqEvent(RenderContext& renderer, LibKernel::EventQueue::KernelEqueue eq, int id,
                void* udata) {
 	LibKernel::EventQueue::KernelEqueueEvent event;
-	event.triggered                = false;
-	event.event.ident              = static_cast<uintptr_t>(id);
-	event.event.filter             = LibKernel::EventQueue::KERNEL_EVFILT_GRAPHICS;
-	event.event.udata              = udata;
-	event.event.fflags             = 0;
-	event.event.data               = id;
-	event.filter.delete_event_func = EopEventDeleteFunc;
-	event.filter.reset_func        = EopEventResetFunc;
-	event.filter.trigger_func      = EopEventTriggerFunc;
-	event.filter.data              = &renderer;
+	event.triggered           = false;
+	event.event.ident         = static_cast<uintptr_t>(id);
+	event.event.filter        = LibKernel::EventQueue::KERNEL_EVFILT_GRAPHICS;
+	event.event.udata         = udata;
+	event.event.fflags        = 0;
+	event.event.data          = id;
+	event.filter.reset_func   = InterruptEventResetFunc;
+	event.filter.trigger_func = InterruptEventTriggerFunc;
 
 	int result = LibKernel::EventQueue::KernelAddEvent(eq, event);
 
-	if (result == 0 &&
-	    (id == GRAPHICS_EVENT_QUEUED_GRAPHICS_INTERRUPT || id == GRAPHICS_EVENT_EOP)) {
-		renderer.AddEopEq(eq, id);
+	if (result == 0) {
+		renderer.AddInterruptEq(eq, id);
 	}
 
 	return result;
 }
 
-int DeleteEqEvent(LibKernel::EventQueue::KernelEqueue eq, int id) {
+int DeleteEqEvent(RenderContext& renderer, LibKernel::EventQueue::KernelEqueue eq, int id) {
 	int result = LibKernel::EventQueue::KernelDeleteEvent(
 	    eq, static_cast<uintptr_t>(id), LibKernel::EventQueue::KERNEL_EVFILT_GRAPHICS);
+	if (result == OK || result == LibKernel::KERNEL_ERROR_ENOENT) {
+		renderer.DeleteInterruptEq(eq, id);
+	}
 
 	return result;
 }

src/graphics/host_gpu/renderer/sync.h

@@ -17,8 +17,7 @@ namespace Sync {
                                            uint64_t& value);
 [[nodiscard]] uint64_t ReadReferenceClock();
 
-void TriggerAgcUserInterrupt();
-void TriggerEopEventAtEndOfPipe(CommandBuffer& buffer, uint32_t context_id);
+void TriggerEopEventAtEndOfPipe(CommandBuffer& buffer, int event_id, uint32_t context_id);
 
 void WriteAtEndOfPipe32(uint64_t submit_id, CommandBuffer& buffer, uint32_t* dst_gpu_addr,
                         uint32_t value);
@@ -35,16 +34,16 @@ void WriteAtEndOfPipeWithWriteBack32(uint64_t submit_id, CommandBuffer& buffer,
 void WriteAtEndOfPipeWithWriteBack64(uint64_t submit_id, CommandBuffer& buffer,
                                      uint64_t* dst_gpu_addr, uint64_t value);
 void WriteAtEndOfPipeWithInterrupt32(uint64_t submit_id, CommandBuffer& buffer,
-                                     uint32_t* dst_gpu_addr, uint32_t value,
+                                     uint32_t* dst_gpu_addr, uint32_t value, int event_id,
                                      uint32_t context_id = 0);
 void WriteAtEndOfPipeWithInterrupt64(uint64_t submit_id, CommandBuffer& buffer,
-                                     uint64_t* dst_gpu_addr, uint64_t value,
+                                     uint64_t* dst_gpu_addr, uint64_t value, int event_id,
                                      uint32_t context_id = 0);
 void WriteAtEndOfPipeWithInterruptWriteBack32(uint64_t submit_id, CommandBuffer& buffer,
-                                              uint32_t* dst_gpu_addr, uint32_t value,
+                                              uint32_t* dst_gpu_addr, uint32_t value, int event_id,
                                               uint32_t context_id = 0);
 void WriteAtEndOfPipeWithInterruptWriteBack64(uint64_t submit_id, CommandBuffer& buffer,
-                                              uint64_t* dst_gpu_addr, uint64_t value,
+                                              uint64_t* dst_gpu_addr, uint64_t value, int event_id,
                                               uint32_t context_id = 0);
 
 [[nodiscard]] uint64_t PrepareVideoOutFlip(CommandBuffer& buffer, int handle, int index,
@@ -57,11 +56,12 @@ void WriteAtEndOfPipeWithFlip32(uint64_t submit_id, CommandBuffer& buffer, uint3
 void WriteAtEndOfPipeWithInterruptWriteBackFlip32(uint64_t submit_id, CommandBuffer& buffer,
                                                   uint32_t* dst_gpu_addr, uint32_t value,
                                                   int handle, int index, int flip_mode,
-                                                  int64_t flip_arg, uint64_t request_id);
+                                                  int64_t flip_arg, uint64_t request_id,
+                                                  int event_id);
 
 int  AddEqEvent(RenderContext& renderer, LibKernel::EventQueue::KernelEqueue eq, int id,
                 void* udata);
-int  DeleteEqEvent(LibKernel::EventQueue::KernelEqueue eq, int id);
+int  DeleteEqEvent(RenderContext& renderer, LibKernel::EventQueue::KernelEqueue eq, int id);
 void ReadGds(const Buffer& gds, uint32_t* dst, uint32_t dw_offset, uint32_t dw_size);
 
 } // namespace Sync

src/graphics/host_gpu/shaders/fault_buffer_process.comp

@@ -0,0 +1,35 @@
+// SPDX-FileCopyrightText: Copyright 2024 shadPS4 Emulator Project
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#version 450
+#extension GL_ARB_gpu_shader_int64 : enable
+
+layout(local_size_x = 64, local_size_y = 1, local_size_z = 1) in;
+
+layout(std430, binding = 0) buffer input_buf {
+    uint fault_buffer[];
+};
+
+layout(std430, binding = 1) buffer output_buf {
+    uint64_t download_buffer[];
+};
+layout(std430, binding = 1) buffer output_buf32 {
+    uint download_buffer32[];
+};
+
+void main() {
+    const uint id = gl_GlobalInvocationID.x;
+    uint word = fault_buffer[id];
+    fault_buffer[id] = 0u;
+    const uint base_bit = id * 32u;
+    while (word != 0u) {
+        const uint store_index = atomicAdd(download_buffer32[0], 1u) + 1u;
+        if (store_index >= MAX_PAGE_FAULTS) {
+            return;
+        }
+        const uint bit = findLSB(word);
+        word &= word - 1;
+        const uint page = base_bit + bit;
+        download_buffer[store_index] = uint64_t(page) << CACHING_PAGEBITS;
+    }
+}

src/graphics/host_gpu/vma.cpp

@@ -72,7 +72,10 @@ bool GraphicContext::CreateAllocator() {
 	info.device           = device;
 	info.pVulkanFunctions = &functions;
 	info.vulkanApiVersion = VULKAN_TARGET_API_VERSION;
-	info.flags = memory_budget_ext_enabled ? VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT : 0;
+	info.flags = VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT;
+	if (memory_budget_ext_enabled) {
+		info.flags |= VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT;
+	}
 
 	const auto result = static_cast<vk::Result>(vmaCreateAllocator(&info, &allocator));
 	if (result != vk::Result::eSuccess) {

src/graphics/host_gpu/vulkanInstance.h

@@ -1,94 +0,0 @@
-#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_VULKANINSTANCE_H_
-#define EMULATOR_SRC_GRAPHICS_HOST_GPU_VULKANINSTANCE_H_
-
-#include "common/abi.h"
-#include "common/common.h"
-#include "common/threads.h"
-#include "graphics/host_gpu/vulkanCommon.h"
-
-#include <map>
-#include <mutex>
-#include <tuple>
-#include <vk_mem_alloc.h>
-
-namespace Libs::Graphics {
-
-inline constexpr uint32_t VULKAN_TARGET_API_VERSION = VK_API_VERSION_1_3;
-
-struct VulkanInstance {
-	vk::Instance                       instance                          = nullptr;
-	vk::DebugUtilsMessengerEXT         debug_messenger                   = nullptr;
-	vk::PhysicalDevice                 physical_device                   = nullptr;
-	vk::PhysicalDeviceProperties       physical_device_properties        = {};
-	vk::PhysicalDeviceMemoryProperties physical_device_memory_properties = {};
-	vk::Device                         device                            = nullptr;
-	VmaAllocator                       allocator                         = nullptr;
-	bool                               memory_budget_ext_enabled         = false;
-	bool                               rt_extensions_enabled             = false;
-	bool                               subgroup_size_control_enabled     = false;
-	bool                               sample_rate_shading_enabled       = false;
-	uint32_t                           subgroup_size                     = 0;
-	uint32_t                           min_subgroup_size                 = 0;
-	uint32_t                           max_subgroup_size                 = 0;
-	uint32_t                           max_push_descriptors              = 0;
-	vk::ShaderStageFlags               required_subgroup_size_stages     = {};
-	Common::Mutex                      queue_mutex;
-	uint32_t                           queue_family = static_cast<uint32_t>(-1);
-	vk::Queue                          queue        = nullptr;
-
-	[[nodiscard]] const vk::PhysicalDeviceProperties& GetPhysicalDeviceProperties() const {
-		return physical_device_properties;
-	}
-
-	[[nodiscard]] const vk::PhysicalDeviceMemoryProperties&
-	GetPhysicalDeviceMemoryProperties() const {
-		return physical_device_memory_properties;
-	}
-
-	[[nodiscard]] vk::FormatProperties GetFormatProperties(vk::Format format) const {
-		std::scoped_lock lock(m_format_properties_mutex);
-		auto [it, inserted] = m_format_properties.try_emplace(format);
-		if (inserted) {
-			physical_device.getFormatProperties(format, &it->second);
-		}
-		return it->second;
-	}
-
-	[[nodiscard]] vk::Result GetImageFormatProperties(vk::Format format, vk::ImageType type,
-	                                                  vk::ImageTiling            tiling,
-	                                                  vk::ImageUsageFlags        usage,
-	                                                  vk::ImageCreateFlags       flags,
-	                                                  vk::ImageFormatProperties* properties) const {
-		using Key = std::tuple<vk::Format, vk::ImageType, vk::ImageTiling, vk::ImageUsageFlags,
-		                       vk::ImageCreateFlags>;
-		std::scoped_lock lock(m_image_format_properties_mutex);
-		auto [it, inserted] =
-		    m_image_format_properties.try_emplace(Key {format, type, tiling, usage, flags});
-		if (inserted) {
-			it->second.first = physical_device.getImageFormatProperties(format, type, tiling, usage,
-			                                                            flags, &it->second.second);
-		}
-		if (properties != nullptr) {
-			*properties = it->second.second;
-		}
-		return it->second.first;
-	}
-
-	[[nodiscard]] vk::DeviceSize StorageMinAlignment() const {
-		const auto alignment = physical_device_properties.limits.minStorageBufferOffsetAlignment;
-		return alignment != 0 ? alignment : 1;
-	}
-
-private:
-	mutable std::mutex                                 m_format_properties_mutex;
-	mutable std::map<vk::Format, vk::FormatProperties> m_format_properties;
-	mutable std::mutex                                 m_image_format_properties_mutex;
-	mutable std::map<std::tuple<vk::Format, vk::ImageType, vk::ImageTiling, vk::ImageUsageFlags,
-	                            vk::ImageCreateFlags>,
-	                 std::pair<vk::Result, vk::ImageFormatProperties>>
-	    m_image_format_properties;
-};
-
-} // namespace Libs::Graphics
-
-#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_VULKANINSTANCE_H_

src/graphics/presentation/window/swapchain.cpp

@@ -1,30 +1,9 @@
-#include "SDL.h"
-#include "SDL_error.h"
-#include "SDL_events.h"
-#include "SDL_gamecontroller.h"
-#include "SDL_hints.h"
-#include "SDL_joystick.h"
-#include "SDL_keyboard.h"
-#include "SDL_keycode.h"
-#include "SDL_mouse.h"
-#include "SDL_pixels.h"
-#include "SDL_rwops.h"
-#include "SDL_stdinc.h"
-#include "SDL_surface.h"
-#include "SDL_thread.h"
-#include "SDL_touch.h"
-#include "SDL_video.h"
-#include "SDL_vulkan.h"
 #include "common/assert.h"
 #include "common/common.h"
 #include "common/emulatorConfig.h"
-#include "common/file.h"
 #include "common/logging/log.h"
 #include "common/profiler.h"
-#include "common/stringUtils.h"
-#include "common/systemInfo.h"
 #include "common/threads.h"
-#include "common/timer.h"
 #include "graphics/host_gpu/graphicContext.h"
 #include "graphics/host_gpu/renderer/render.h"
 #include "graphics/host_gpu/renderer/renderContext.h"
@@ -34,19 +13,11 @@
 #include "graphics/presentation/presenter.h"
 #include "graphics/presentation/videoOut.h"
 #include "graphics/presentation/window/windowInternal.h"
-#include "libs/controller.h"
-#include "loader/systemContent.h"
 
 #include <algorithm>
-#include <array>
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
 #include <deque>
-#include <fmt/format.h>
 #include <limits>
 #include <memory>
-#include <string>
 #include <vector>
 #include <vulkan/vk_platform.h>
 
@@ -58,10 +29,10 @@
 namespace Libs::Graphics {
 
 struct Presenter::Frame {
-	VulkanImage                    image;
-	std::unique_ptr<CommandBuffer> present_commands;
-	bool                           busy         = false;
-	bool                           reusing_last = false;
+	VulkanImage image;
+	uint64_t    present_tick = 0;
+	bool        busy         = false;
+	bool        reusing_last = false;
 
 	void Configure(GraphicContext& graphics, vk::Extent2D extent, vk::Format format);
 	void Transit(vk::CommandBuffer command, vk::ImageLayout layout, vk::AccessFlags2 access);
@@ -71,12 +42,11 @@ struct Presenter::Frame {
 
 class FramePool final {
 public:
-	explicit FramePool(WindowContext& window): m_window(window) {}
+	FramePool(WindowContext& window, CommandScheduler& scheduler)
+	    : m_window(window), m_scheduler(scheduler) {}
 	~FramePool() {
+		m_scheduler.Wait(m_scheduler.CurrentTick() - 1);
 		for (auto& frame: m_frames) {
-			if (frame->present_commands != nullptr) {
-				frame->present_commands->WaitForFenceOnly();
-			}
 			if (frame->image.image != nullptr) {
 				m_window.graphic_ctx.DeleteImage(frame->image);
 			}
@@ -188,15 +158,10 @@ public:
 	}
 
 private:
-	static void WaitForFrame(Presenter::Frame& frame) {
-		// The producer only waits here. Reset stays on the presentation thread that owns the
-		// allocating Vulkan command pool.
-		if (frame.present_commands != nullptr) {
-			frame.present_commands->WaitForFenceOnly();
-		}
-	}
+	void WaitForFrame(Presenter::Frame& frame) { m_scheduler.Wait(frame.present_tick); }
 
 	WindowContext&                                 m_window;
+	CommandScheduler&                              m_scheduler;
 	Common::Mutex                                  m_mutex;
 	Common::CondVar                                m_available;
 	std::vector<std::unique_ptr<Presenter::Frame>> m_frames;
@@ -330,7 +295,7 @@ public:
 	[[nodiscard]] Status AcquireNextImage();
 	[[nodiscard]] bool   PrepareImeOverlay();
 	void RecordPresentCommands(CommandBuffer& command, VulkanImage& source, bool draw_ime_overlay);
-	void Submit(CommandBuffer& command);
+	uint64_t             Submit(CommandScheduler& scheduler);
 	[[nodiscard]] Status Present();
 
 	[[nodiscard]] uint32_t ImageCount() const noexcept {
@@ -340,7 +305,6 @@ public:
 
 private:
 	void Destroy();
-	void RefreshSurfaceSize();
 
 	WindowContext&              m_window;
 	vk::SwapchainKHR            m_handle = nullptr;
@@ -358,7 +322,7 @@ private:
 struct Presenter::Impl {
 	explicit Impl(WindowContext& owner)
 	    : renderer(*owner.render_context), window(owner), swapchain(owner),
-	      present_scheduler(renderer, owner.graphic_ctx), frames(owner) {
+	      present_scheduler(renderer, owner.graphic_ctx), frames(owner, present_scheduler) {
 		EXIT_IF(owner.render_context == nullptr);
 		swapchain.Create();
 		frames.Initialize(swapchain.ImageCount(), swapchain.Format());
@@ -402,12 +366,12 @@ struct Presenter::Impl {
 
 void Swapchain::Create() {
 	auto& graphics = m_window.graphic_ctx;
-	EXIT_IF(graphics.screen_width == 0);
-	EXIT_IF(graphics.screen_height == 0);
 	EXIT_IF(graphics.device == nullptr);
 	EXIT_IF(m_window.surface == nullptr);
 
 	Common::LockGuard lock(m_window.mutex);
+	EXIT_IF(graphics.screen_width == 0);
+	EXIT_IF(graphics.screen_height == 0);
 	const auto&       surface = m_window.surface_capabilities;
 	EXIT_NOT_IMPLEMENTED(surface.formats.empty());
 
@@ -465,7 +429,21 @@ void Swapchain::Create() {
 	create_info.imageSharingMode = vk::SharingMode::eExclusive;
 	create_info.preTransform     = transform;
 	create_info.compositeAlpha   = composite;
-	create_info.presentMode      = vk::PresentModeKHR::eFifo;
+	switch (Config::GetPresentMode()) {
+		case Config::PresentMode::Mailbox:
+			create_info.presentMode = vk::PresentModeKHR::eMailbox;
+			break;
+		case Config::PresentMode::Immediate:
+			create_info.presentMode = vk::PresentModeKHR::eImmediate;
+			break;
+		case Config::PresentMode::Fifo:
+		default: create_info.presentMode = vk::PresentModeKHR::eFifo; break;
+	}
+	if (std::find(surface.present_modes.begin(), surface.present_modes.end(),
+	              create_info.presentMode) == surface.present_modes.end()) {
+		LOGF("warning: requested present mode is unavailable; falling back to Fifo\n");
+		create_info.presentMode = vk::PresentModeKHR::eFifo;
+	}
 	create_info.clipped          = VK_TRUE;
 	RequireVulkanSuccess(graphics.device.createSwapchainKHR(&create_info, nullptr, &m_handle),
 	                     "vkCreateSwapchainKHR");
@@ -560,30 +538,18 @@ void Swapchain::Destroy() {
 	m_render_complete.clear();
 }
 
-void Swapchain::RefreshSurfaceSize() {
-	int width  = 0;
-	int height = 0;
-	SDL_Vulkan_GetDrawableSize(m_window.window, &width, &height);
-	if (width > 0 && height > 0) {
-		m_window.graphic_ctx.screen_width  = static_cast<uint32_t>(width);
-		m_window.graphic_ctx.screen_height = static_cast<uint32_t>(height);
-	}
-
-	m_window.RefreshSurfaceCapabilities();
-}
-
 void Swapchain::Recreate(bool surface_lost) {
 	Destroy();
 	if (surface_lost) {
 #if defined(__APPLE__)
 		// Surface recreation goes through SDL_Vulkan_CreateSurface, which touches the
 		// window's view/layer and must run on the main thread on macOS.
-		m_window.RunOnMainThread([this] { m_window.RecreateSurface(); }, true);
+		m_window.RunOnMainThread([this] { m_window.RecreateSurface(); });
 #else
 		m_window.RecreateSurface();
 #endif
 	}
-	RefreshSurfaceSize();
+	m_window.RefreshSurfaceCapabilities();
 	Create();
 }
 
@@ -628,7 +594,6 @@ void Swapchain::RecordPresentCommands(CommandBuffer& command, VulkanImage& sourc
 	}
 	EXIT_IF(m_image_index >= m_images.size());
 	auto vk_command = command.Handle();
-	command.Begin();
 
 	vk::ImageMemoryBarrier to_transfer {};
 	to_transfer.sType                           = vk::StructureType::eImageMemoryBarrier;
@@ -699,15 +664,14 @@ void Swapchain::RecordPresentCommands(CommandBuffer& command, VulkanImage& sourc
 		                           vk::DependencyFlagBits::eByRegion, 0, nullptr, 0, nullptr, 1,
 		                           &to_present);
 	}
-	command.End();
 }
 
-void Swapchain::Submit(CommandBuffer& command) {
+uint64_t Swapchain::Submit(CommandScheduler& scheduler) {
 	EXIT_IF(m_frame_index >= m_image_acquired.size() || m_image_index >= m_render_complete.size());
 	SubmitInfo submit;
 	submit.AddWait(m_image_acquired[m_frame_index], 1, vk::PipelineStageFlagBits::eTransfer);
 	submit.AddSignal(m_render_complete[m_image_index]);
-	command.Execute(submit);
+	return scheduler.Submit(submit);
 }
 
 Swapchain::Status Swapchain::Present() {
@@ -782,15 +746,9 @@ Presenter::Frame& Presenter::PrepareBlankFrame(uint32_t width, uint32_t height,
 		EXIT_IF(producer->IsInvalid());
 		frame->Clear(*producer, clear);
 	} else {
-		if (frame->present_commands == nullptr) {
-			frame->present_commands = std::make_unique<CommandBuffer>(m_impl->present_scheduler);
-		}
-		auto& command = *frame->present_commands;
-		command.WaitForFenceAndReset();
-		command.Begin();
+		auto& command = m_impl->present_scheduler.BeginCommand();
 		frame->Clear(command, clear);
-		command.End();
-		command.Execute();
+		frame->present_tick = m_impl->present_scheduler.Submit();
 	}
 	return *frame;
 }
@@ -817,29 +775,6 @@ void Presenter::Present(Frame& frame, bool reuse) {
 	KYTY_PROFILER_FUNCTION();
 	m_impl->frames.ValidateForPresent(&frame, reuse);
 
-	auto& window = m_impl->window;
-	if (window.window_hidden) {
-#if defined(__APPLE__)
-		// AppKit traps if a window is shown off the main thread; marshal and wait so the
-		// swapchain below is recreated against a visible window.
-		window.RunOnMainThread(
-		    [&window] {
-			    window.UpdateIcon();
-			    SDL_ShowWindow(window.window);
-			    SDL_RaiseWindow(window.window);
-		    },
-		    true);
-#else
-		window.UpdateIcon();
-
-		SDL_ShowWindow(window.window);
-		SDL_RaiseWindow(window.window);
-#endif
-
-		window.window_hidden = false;
-		m_impl->RecoverSwapchain(Swapchain::Status::Recreate);
-	}
-
 	const auto ime_visual = GetImeVisualState();
 	auto&      swapchain  = m_impl->swapchain;
 	for (uint32_t attempt = 0; attempt < 2; attempt++) {
@@ -848,16 +783,12 @@ void Presenter::Present(Frame& frame, bool reuse) {
 			m_impl->RecoverSwapchain(status);
 			continue;
 		}
-		if (frame.present_commands == nullptr) {
-			frame.present_commands = std::make_unique<CommandBuffer>(m_impl->present_scheduler);
-		}
 		{
 			Common::LockGuard render_lock(m_impl->renderer.GetMutex());
-			frame.present_commands->WaitForFenceAndReset();
-			auto&      command          = *frame.present_commands;
-			const bool draw_ime_overlay = ime_visual.active && swapchain.PrepareImeOverlay();
+			auto&             command          = m_impl->present_scheduler.BeginCommand();
+			const bool        draw_ime_overlay = ime_visual.active && swapchain.PrepareImeOverlay();
 			swapchain.RecordPresentCommands(command, frame.image, draw_ime_overlay);
-			swapchain.Submit(command);
+			frame.present_tick = swapchain.Submit(m_impl->present_scheduler);
 		}
 		status = swapchain.Present();
 		if (status != Swapchain::Status::Success) {
@@ -866,7 +797,7 @@ void Presenter::Present(Frame& frame, bool reuse) {
 		}
 
 		m_impl->presented_ime_revision.store(ime_visual.revision, std::memory_order_release);
-		window.UpdateTitle();
+		m_impl->window.UpdateTitle();
 		m_impl->frames.Release(&frame, true);
 		return;
 	}

src/graphics/presentation/window/vulkanWindow.cpp

@@ -22,7 +22,6 @@
 #include "common/logging/log.h"
 #include "common/profiler.h"
 #include "common/stringUtils.h"
-#include "common/systemInfo.h"
 #include "common/threads.h"
 #include "common/timer.h"
 #include "graphics/host_gpu/graphicContext.h"
@@ -493,14 +492,19 @@ static void VulkanInitSubgroupSizeControl(vk::PhysicalDevice physical_device,
 	graphics.min_subgroup_size             = subgroup_size_control.minSubgroupSize;
 	graphics.max_subgroup_size             = subgroup_size_control.maxSubgroupSize;
 	graphics.required_subgroup_size_stages = subgroup_size_control.requiredSubgroupSizeStages;
-	graphics.subgroup_size_control_enabled = features13.subgroupSizeControl == VK_TRUE &&
-	                                         subgroup_size_control.minSubgroupSize <= 64 &&
-	                                         subgroup_size_control.maxSubgroupSize >= 64;
+	graphics.compute_subgroup_size_control_enabled =
+	    features13.subgroupSizeControl == VK_TRUE &&
+	    (graphics.required_subgroup_size_stages & vk::ShaderStageFlagBits::eCompute) &&
+	    subgroup_size_control.minSubgroupSize <= 64 &&
+	    subgroup_size_control.maxSubgroupSize >= 64;
+	graphics.compute_wave64_supported =
+	    graphics.subgroup_size == 64u || graphics.compute_subgroup_size_control_enabled;
 
-	LOGF("Vulkan subgroup: default=%u min=%u max=%u stages=0x%08x size_control=%s\n",
+	LOGF("Vulkan subgroup: default=%u min=%u max=%u stages=0x%08x size_control=%s wave64=%s\n",
 	     graphics.subgroup_size, graphics.min_subgroup_size, graphics.max_subgroup_size,
 	     static_cast<vk::ShaderStageFlags::MaskType>(graphics.required_subgroup_size_stages),
-	     graphics.subgroup_size_control_enabled ? "true" : "false");
+	     graphics.compute_subgroup_size_control_enabled ? "true" : "false",
+	     graphics.compute_wave64_supported ? "true" : "false");
 }
 
 static vk::Device VulkanCreateDevice(vk::PhysicalDevice physical_device, const VulkanExtensions& r,
@@ -589,20 +593,16 @@ static vk::Device VulkanCreateDevice(vk::PhysicalDevice physical_device, const V
 	EXIT_NOT_IMPLEMENTED(supported_features2.features.shaderClipDistance != VK_TRUE);
 	EXIT_NOT_IMPLEMENTED(supported_features2.features.shaderCullDistance != VK_TRUE);
 	EXIT_NOT_IMPLEMENTED(supported_features2.features.largePoints != VK_TRUE);
+	EXIT_NOT_IMPLEMENTED(supported_features2.features.shaderInt64 != VK_TRUE);
+	EXIT_NOT_IMPLEMENTED(supported_features2.features.vertexPipelineStoresAndAtomics != VK_TRUE);
 	EXIT_NOT_IMPLEMENTED(supported_features12.shaderOutputLayer != VK_TRUE);
+	EXIT_NOT_IMPLEMENTED(supported_features12.bufferDeviceAddress != VK_TRUE);
 #if !defined(__APPLE__)
 	EXIT_NOT_IMPLEMENTED(supported_fragment_barycentric.fragmentShaderBarycentric != VK_TRUE);
 #endif
 	features12.timelineSemaphore = VK_TRUE;
 	features12.shaderOutputLayer = VK_TRUE;
-
-	if (Config::GpuAssistedValidationEnabled()) {
-		if (supported_features12.bufferDeviceAddress != VK_TRUE) {
-			EXIT("--gpu-assisted-validation requires bufferDeviceAddress, unsupported by this "
-			     "device\n");
-		}
-		features12.bufferDeviceAddress = VK_TRUE;
-	}
+	features12.bufferDeviceAddress = VK_TRUE;
 
 	vk::PhysicalDeviceFeatures device_features {};
 	device_features.fragmentStoresAndAtomics = VK_TRUE;
@@ -620,12 +620,9 @@ static vk::Device VulkanCreateDevice(vk::PhysicalDevice physical_device, const V
 	device_features.shaderClipDistance                   = VK_TRUE;
 	device_features.shaderCullDistance                   = VK_TRUE;
 	device_features.largePoints                          = VK_TRUE;
+	device_features.vertexPipelineStoresAndAtomics       = VK_TRUE;
 	graphics.sample_rate_shading_enabled                 = true;
-	device_features.vertexPipelineStoresAndAtomics =
-	    supported_features2.features.vertexPipelineStoresAndAtomics;
-	if (Config::GpuAssistedValidationEnabled()) {
-		device_features.shaderInt64 = supported_features2.features.shaderInt64;
-	}
+	device_features.shaderInt64 = VK_TRUE;
 
 	vk::PhysicalDeviceRobustness2FeaturesEXT robustness2 {};
 	robustness2.sType = vk::StructureType::ePhysicalDeviceRobustness2FeaturesEXT;
@@ -645,8 +642,9 @@ static vk::Device VulkanCreateDevice(vk::PhysicalDevice physical_device, const V
 		robustness2.nullDescriptor      = supported_robustness2.nullDescriptor;
 	}
 
-	const bool subgroup_size_control_enabled = graphics.subgroup_size_control_enabled &&
-	                                           supported_features13.subgroupSizeControl == VK_TRUE;
+	const bool subgroup_size_control_enabled =
+	    graphics.compute_subgroup_size_control_enabled &&
+	    supported_features13.subgroupSizeControl == VK_TRUE;
 
 	auto features13 = required_features13;
 #if defined(__APPLE__)
@@ -944,6 +942,18 @@ void WindowContext::CreateVulkan() {
 	inst_info.sType                   = vk::StructureType::eInstanceCreateInfo;
 	inst_info.pNext                   = (r.enable_validation_layers ? &dbg_create_info : nullptr);
 	inst_info.flags                   = {};
+#if defined(__APPLE__)
+	// MoltenVK requires VK_KHR_portability_enumeration + flag to surface
+	// portability devices. Without this, enumeratePhysicalDevices hides the
+	// MoltenVK adapter on some driver versions.
+	if (HasExtension(r.available_extensions, VK_KHR_PORTABILITY_ENUMERATION_EXTENSION_NAME)) {
+		if (!HasExtension(r.required_extensions, VK_KHR_PORTABILITY_ENUMERATION_EXTENSION_NAME)) {
+			r.required_extensions.push_back(VK_KHR_PORTABILITY_ENUMERATION_EXTENSION_NAME);
+		}
+		inst_info.flags |= vk::InstanceCreateFlagBits::eEnumeratePortabilityKHR;
+		LOGF("Vulkan instance: enabled %s\n", VK_KHR_PORTABILITY_ENUMERATION_EXTENSION_NAME);
+	}
+#endif
 	inst_info.pApplicationInfo        = &app_info;
 	inst_info.enabledExtensionCount   = static_cast<uint32_t>(r.required_extensions.size());
 	inst_info.ppEnabledExtensionNames = r.required_extensions.data();
@@ -1038,10 +1048,6 @@ void WindowContext::CreateVulkan() {
 		}
 	}
 
-	memcpy(device_name, device_properties.deviceName, sizeof(device_name));
-	std::snprintf(processor_name, sizeof(processor_name), "%s",
-	              Common::GetSystemInfo().ProcessorName.c_str());
-
 	VulkanInitSubgroupSizeControl(graphic_ctx.physical_device, graphic_ctx);
 
 	graphic_ctx.device = VulkanCreateDevice(graphic_ctx.physical_device, r, queue_family,
@@ -1080,7 +1086,6 @@ void WindowContext::RecreateSurface() {
 		EXIT("Could not recreate the Vulkan surface: %s\n", SDL_GetError());
 	}
 	surface = native_surface;
-	RefreshSurfaceCapabilities();
 }
 
 WindowContext::~WindowContext() {

src/graphics/presentation/window/window.cpp

@@ -3,6 +3,7 @@
 #include "SDL.h"
 #include "SDL_error.h"
 #include "SDL_events.h"
+#include "SDL_filesystem.h"
 #include "SDL_gamecontroller.h"
 #include "SDL_hints.h"
 #include "SDL_joystick.h"
@@ -23,12 +24,12 @@
 #include "common/file.h"
 #include "common/logging/log.h"
 #include "common/profiler.h"
-#include "common/stringUtils.h"
 #include "common/systemInfo.h"
 #include "common/threads.h"
 #include "common/timer.h"
 #include "graphics/host_gpu/graphicContext.h"
 #include "graphics/host_gpu/renderer/render.h"
+#include "graphics/host_gpu/renderer/renderContext.h"
 #include "graphics/host_gpu/vma.h"
 #include "graphics/host_gpu/vulkanCommon.h"
 #include "graphics/presentation/imeOverlay.h"
@@ -39,10 +40,8 @@
 #include "libs/controller.h"
 #include "loader/systemContent.h"
 
-#include <algorithm>
-#include <cstdio>
 #include <cstdlib>
-#include <cstring>
+#include <fmt/format.h>
 #include <memory>
 #include <string>
 #include <vector>
@@ -52,8 +51,6 @@
 #define STBI_NO_SIMD
 #include "stb_image.h"
 
-#include <fmt/format.h>
-
 // IWYU pragma: no_include <intrin.h>
 
 #define KYTY_ENABLE_DEBUG_PRINTF
@@ -197,8 +194,6 @@ std::unique_ptr<WindowContext> g_window;
 } // namespace
 
 constexpr const char* KYTY_SDL_WINDOW_CAPTION = "Game";
-constexpr uint32_t    KYTY_SDL_WINDOW_FLAGS =
-    (static_cast<uint32_t>(SDL_WINDOW_HIDDEN) | static_cast<uint32_t>(SDL_WINDOW_VULKAN));
 constexpr int KYTY_SDL_WINDOWPOS_CENTERED = SDL_WINDOWPOS_CENTERED; /*NOLINT(hicpp-signed-bitwise)*/
 
 static void SetPause(WindowLoopState& game, bool flag) {
@@ -407,6 +402,7 @@ static void GameEventDidEnterForeground(WindowLoopState& game) {
 
 void WindowContext::Resize(uint32_t new_width, uint32_t new_height) {
 	EXIT_IF(new_width == 0 || new_height == 0);
+	Common::LockGuard lock(mutex);
 	graphic_ctx.screen_width  = new_width;
 	graphic_ctx.screen_height = new_height;
 }
@@ -728,8 +724,7 @@ void WindowContext::ProcessEvent(double time_s) {
 	}
 }
 
-#if defined(__APPLE__)
-void WindowContext::RunOnMainThread(std::function<void()> task, bool wait) {
+void WindowContext::RunOnMainThread(std::function<void()> task) {
 	if (Common::Thread::IsMainThread()) {
 		task();
 		return;
@@ -747,9 +742,6 @@ void WindowContext::RunOnMainThread(std::function<void()> task, bool wait) {
 	event.type = SDL_USEREVENT;
 	SDL_PushEvent(&event);
 
-	if (!wait) {
-		return;
-	}
 	Common::LockGuard lock(main_task_mutex);
 	while (main_tasks_run < ticket) {
 		main_task_done.Wait(&main_task_mutex);
@@ -772,7 +764,6 @@ void WindowContext::DrainMainThreadTasks() {
 	main_tasks_run += tasks.size();
 	main_task_done.SignalAll();
 }
-#endif
 
 void WindowContext::Run() {
 	Common::Timer timer;
@@ -783,9 +774,7 @@ void WindowContext::Run() {
 	loop.paused.store(false, std::memory_order_release);
 
 	while (!loop.need_exit) {
-#if defined(__APPLE__)
 		DrainMainThreadTasks();
-#endif
 		if (loop.paused.load(std::memory_order_acquire)) {
 			if (!timer.IsPaused()) {
 				timer.Pause();
@@ -821,11 +810,26 @@ static void WindowCreate(WindowContext& context) {
 
 	LOGF("WindowCreate(): width = %d, height = %d\n", width, height);
 
-	uint32_t window_flags = KYTY_SDL_WINDOW_FLAGS;
-	if (Config::FullscreenEnabled()) {
-		window_flags |= static_cast<uint32_t>(SDL_WINDOW_FULLSCREEN_DESKTOP);
-	}
+	uint32_t window_flags = WindowContext::InitialWindowFlags(Config::FullscreenEnabled());
 #if defined(__APPLE__)
+	// SDL loads Vulkan while creating a Vulkan window, so select the bundled
+	// MoltenVK loader before calling SDL_CreateWindow. Keep an explicit user
+	// override, and fall back to SDL's normal loader search when no bundle is present.
+	if (std::getenv("SDL_VULKAN_LIBRARY") == nullptr) {
+		if (char* base_path = SDL_GetBasePath(); base_path != nullptr) {
+			const std::string base_path_str = base_path;
+			SDL_free(base_path);
+			std::string moltenvk_path = base_path_str + "libMoltenVK.dylib";
+			if (!Common::File::IsFileExisting(moltenvk_path)) {
+				moltenvk_path = base_path_str + "../Frameworks/libMoltenVK.dylib";
+			}
+			if (Common::File::IsFileExisting(moltenvk_path) &&
+			    SDL_setenv("SDL_VULKAN_LIBRARY", moltenvk_path.c_str(), 0) == 0) {
+				LOGF("Vulkan loader: %s\n", moltenvk_path.c_str());
+			}
+		}
+	}
+
 	// macOS 26 window chrome (CoreUI asset decode, SwiftUI titlebar) has been observed
 	// throwing NSExceptions under Rosetta during the first CATransaction commit. A
 	// borderless window skips that machinery entirely.
@@ -836,13 +840,20 @@ static void WindowCreate(WindowContext& context) {
 	context.window = SDL_CreateWindow(KYTY_SDL_WINDOW_CAPTION, KYTY_SDL_WINDOWPOS_CENTERED,
 	                                  KYTY_SDL_WINDOWPOS_CENTERED, width, height, window_flags);
 
-	context.window_hidden = true;
-
 	if (context.window == nullptr) {
 		EXIT("%s\n", SDL_GetError());
 	}
 
 	SDL_SetWindowResizable(context.window, SDL_FALSE);
+	context.UpdateIcon();
+}
+
+uint32_t WindowContext::InitialWindowFlags(bool fullscreen) noexcept {
+	auto flags = static_cast<uint32_t>(SDL_WINDOW_VULKAN);
+	if (fullscreen) {
+		flags |= static_cast<uint32_t>(SDL_WINDOW_FULLSCREEN_DESKTOP);
+	}
+	return flags;
 }
 
 Presenter& WindowInit(uint32_t width, uint32_t height) {
@@ -866,6 +877,7 @@ void WindowRun() {
 	EXIT_IF(g_window == nullptr);
 
 	g_window->Run();
+	g_window->render_context->GetPipelineCache().Save();
 }
 
 void WindowShutdown() {
@@ -959,6 +971,7 @@ void WindowContext::UpdateTitle() {
 	    Loader::SystemContentParamSfoGetString("TITLE_ID", title_id, sizeof(title_id));
 	static bool has_app_ver =
 	    Loader::SystemContentParamSfoGetString("APP_VER", app_ver, sizeof(app_ver));
+	static const std::string processor_name = Common::GetSystemInfo().ProcessorName;
 	static uint64_t fps_start   = Common::Timer::QueryPerformanceCounter();
 	static uint64_t frame_num   = 0;
 	static uint64_t fps_frames  = 0;
@@ -983,19 +996,14 @@ void WindowContext::UpdateTitle() {
 		fps_frames  = 0;
 	}
 
-	auto fps = fmt::format(
+	const auto* device_name = graphic_ctx.GetPhysicalDeviceProperties().deviceName.data();
+	auto text = fmt::format(
 	    "[{} | {}] {}{}{}{}{}{}[{}] [{}], frame: {}, fps: {:f}", KYTY_BUILD_LABEL, build_type,
 	    (has_title ? title : ""), (has_title ? ", " : ""), (has_title_id ? title_id : ""),
 	    (has_title_id ? ", " : ""), (has_app_ver ? app_ver : ""), (has_app_ver ? " " : ""),
 	    device_name, processor_name, frame_num, current_fps);
 
-#if defined(__APPLE__)
-	// AppKit traps on title changes off the main thread; fire-and-forget keeps present pacing.
-	RunOnMainThread([this, fps = std::move(fps)] { SDL_SetWindowTitle(window, fps.c_str()); },
-	                false);
-#else
-	SDL_SetWindowTitle(window, fps.c_str());
-#endif
+	RunOnMainThread([this, text = std::move(text)] { SDL_SetWindowTitle(window, text.c_str()); });
 }
 
 } // namespace Libs::Graphics

src/graphics/presentation/window/windowInternal.h

@@ -9,13 +9,10 @@
 
 #include <atomic>
 #include <cstdint>
+#include <functional>
 #include <memory>
 #include <vector>
 
-#if defined(__APPLE__)
-#include <functional>
-#endif
-
 namespace Libs::Graphics {
 
 class Presenter;
@@ -39,6 +36,7 @@ struct WindowContext {
 	KYTY_CLASS_NO_COPY(WindowContext);
 
 	[[nodiscard]] static vk::PhysicalDeviceVulkan13Features RequiredVulkan13Features() noexcept;
+	[[nodiscard]] static uint32_t InitialWindowFlags(bool fullscreen) noexcept;
 	void                                                    CreateVulkan();
 	void                                                    RecreateSurface();
 	void                                                    RefreshSurfaceCapabilities();
@@ -49,36 +47,25 @@ struct WindowContext {
 	void ProcessDisplayEvent(const SDL_DisplayEvent& event);
 	void ProcessEvent(double time_seconds);
 	void Run();
-
-#if defined(__APPLE__)
-	// AppKit only allows window operations (show, icon, title, view/layer changes) on the
-	// main thread; the present thread marshals them through the SDL main loop with these.
-	// wait=true blocks until the task has run on the main thread.
-	void RunOnMainThread(std::function<void()> task, bool wait);
+	// SDL window operations must complete on the main thread.
+	void RunOnMainThread(std::function<void()> task);
 	void DrainMainThreadTasks();
-#endif
 
 	GraphicContext                 graphic_ctx;
 	SDL_Window*                    window        = nullptr;
-	bool                           window_hidden = true;
 	vk::SurfaceKHR                 surface       = nullptr;
 	SurfaceCapabilities            surface_capabilities;
 	std::unique_ptr<RenderContext> render_context;
 	std::unique_ptr<Presenter>     presenter;
 	WindowLoopState                loop;
 
-	char device_name[VK_MAX_PHYSICAL_DEVICE_NAME_SIZE] = {0};
-	char processor_name[64]                            = {0};
-
 	Common::Mutex mutex;
 
-#if defined(__APPLE__)
 	Common::Mutex                      main_task_mutex;
 	Common::CondVar                    main_task_done;
 	std::vector<std::function<void()>> main_tasks;            // guarded by main_task_mutex
 	uint64_t                           main_tasks_queued = 0; // guarded by main_task_mutex
 	uint64_t                           main_tasks_run    = 0; // guarded by main_task_mutex
-#endif
 };
 
 } // namespace Libs::Graphics

src/graphics/shader/recompiler/ShaderRecompiler.cpp

@@ -7,7 +7,6 @@
 #include "graphics/shader/recompiler/frontend/decode/ShaderDecoder.h"
 #include "graphics/shader/recompiler/frontend/translate/Translate.h"
 #include "graphics/shader/recompiler/ir/ShaderIR.h"
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
 #include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
 #include "graphics/shader/recompiler/ir/passes/ConstantPropagation.h"
 #include "graphics/shader/recompiler/ir/passes/DeadCodeElimination.h"
@@ -15,6 +14,7 @@
 #include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
 #include "graphics/shader/recompiler/ir/passes/ResourceTracking.h"
 #include "graphics/shader/recompiler/ir/passes/ShaderInfoCollection.h"
+#include "graphics/shader/recompiler/ir/passes/SharedMemoryBarrier.h"
 #include "graphics/shader/recompiler/ir/passes/SrtWalker.h"
 #include "graphics/shader/recompiler/ir/passes/SsaRewrite.h"
 
@@ -48,7 +48,7 @@ std::string MakeIrDump(const CFG::Graph& cfg, const IR::Program& ir) {
 	dump += "\nIR:\n";
 	dump += fmt::format("mode={} scratch_dwords={}\n",
 	                    ir.dispatcher_fallback ? "dispatcher" : "structured", ir.scratch_dwords);
-	dump += ir.values != nullptr ? IR::ValueProgramToString(*ir.values) : IR::ProgramToString(ir);
+	dump += IR::ProgramToString(ir);
 	return dump;
 }
 
@@ -116,17 +116,8 @@ void ClearEmbeddedFetchVectorLanes(EmbeddedFetchVectorLanes* lanes, uint32_t reg
 	lanes->erase(first, last);
 }
 
-struct EmbeddedFetchLoad {
-	uint32_t              pc         = 0;
-	int                   attrib_id  = -1;
-	uint32_t              components = 0;
-	std::vector<uint32_t> prolog_loads;
-};
-
-struct EmbeddedFetchData {
-	std::vector<EmbeddedFetchLoad> loads;
-	int32_t                        vertex_offset_sgpr = -1;
-};
+using EmbeddedFetchLoad = Frontend::EmbeddedFetchLoad;
+using EmbeddedFetchData = Frontend::EmbeddedFetchPlan;
 
 bool IsDecodedSgpr(const Decoder::Operand& op) {
 	return op.kind == Decoder::OperandKind::Sgpr || op.kind == Decoder::OperandKind::VccLo ||
@@ -245,15 +236,6 @@ int BufferTableAttribFromOffset(uint32_t raw_offset, int dword) {
 	return static_cast<int>((raw_offset + static_cast<uint32_t>(dword) * 4u) / 16u);
 }
 
-void AppendUniquePcs(std::vector<uint32_t>& dst, const std::vector<uint32_t>& src) {
-	dst.reserve(dst.size() + src.size());
-	for (auto pc: src) {
-		if (std::find(dst.begin(), dst.end(), pc) == dst.end()) {
-			dst.push_back(pc);
-		}
-	}
-}
-
 EmbeddedFetchData DetectEmbeddedVertexFetch(const Decoder::Program&      decoded,
                                             const ShaderVertexInputInfo* input_info,
                                             uint32_t user_data_base, uint32_t user_data_count,
@@ -488,103 +470,6 @@ EmbeddedFetchData DetectEmbeddedVertexFetch(const Decoder::Program&      decoded
 	return data;
 }
 
-const EmbeddedFetchLoad* FindEmbeddedFetchLoad(const std::vector<EmbeddedFetchLoad>& loads,
-                                               uint32_t                              pc) {
-	for (const auto& load: loads) {
-		if (load.pc == pc) {
-			return &load;
-		}
-	}
-	return nullptr;
-}
-
-bool EmbeddedFetchPcInList(const std::vector<uint32_t>& pcs, uint32_t pc) {
-	return std::find(pcs.begin(), pcs.end(), pc) != pcs.end();
-}
-
-bool IsIrFetchPrologLoad(const IR::Instruction& inst) {
-	return inst.op == IR::Opcode::SLoadDword || inst.op == IR::Opcode::SBufferLoadDword;
-}
-
-int ResolveEmbeddedFetchResource(const ShaderVertexInputInfo* input_info,
-                                 const EmbeddedFetchLoad&     load) {
-	if (load.attrib_id >= 0 && load.attrib_id < input_info->resources_num &&
-	    input_info->resources_dst[load.attrib_id].attr_id == load.attrib_id) {
-		return load.attrib_id;
-	}
-	for (int i = 0; i < input_info->resources_num; i++) {
-		const auto& dst = input_info->resources_dst[i];
-		if (dst.attr_id == load.attrib_id &&
-		    load.components <= static_cast<uint32_t>(std::max(dst.registers_num, 1))) {
-			return i;
-		}
-	}
-	for (int i = 0; i < input_info->resources_num; i++) {
-		if (input_info->resources_dst[i].attr_id == load.attrib_id) {
-			return i;
-		}
-	}
-	return -1;
-}
-
-uint32_t RewriteEmbeddedVertexFetches(IR::Program& ir, const ShaderVertexInputInfo* input_info,
-                                      const std::vector<EmbeddedFetchLoad>& loads) {
-	if (loads.empty()) {
-		return 0;
-	}
-
-	std::vector<uint32_t> prolog_pcs;
-	prolog_pcs.reserve(loads.size());
-	for (const auto& load: loads) {
-		AppendUniquePcs(prolog_pcs, load.prolog_loads);
-	}
-
-	auto*    mutable_input_info = const_cast<ShaderVertexInputInfo*>(input_info);
-	uint32_t rewritten          = 0;
-	for (auto& block: ir.blocks) {
-		for (auto& inst: block.instructions) {
-			if (IsIrFetchPrologLoad(inst) && EmbeddedFetchPcInList(prolog_pcs, inst.pc)) {
-				auto pc = inst.pc;
-				inst    = {};
-				inst.pc = pc;
-				inst.op = IR::Opcode::ControlNop;
-				continue;
-			}
-
-			if (inst.op != IR::Opcode::BufferLoadDword) {
-				continue;
-			}
-			const auto* load = FindEmbeddedFetchLoad(loads, inst.pc);
-			if (load == nullptr || inst.memory.data_dwords != load->components ||
-			    inst.dst.kind != IR::OperandKind::Register ||
-			    inst.dst.reg.file != IR::RegisterFile::Vector) {
-				continue;
-			}
-
-			const auto resource_id = ResolveEmbeddedFetchResource(input_info, *load);
-			if (resource_id < 0 || resource_id >= input_info->resources_num) {
-				LOGF("ShaderRecompiler VS embedded fetch remap failed: pc=0x%08" PRIx32
-				     " attrib=%d resources=%d\n",
-				     inst.pc, load->attrib_id, input_info->resources_num);
-				continue;
-			}
-
-			inst.op                         = IR::Opcode::LoadInputF32;
-			inst.input_info.attr            = static_cast<uint32_t>(resource_id);
-			inst.input_info.chan            = 0;
-			inst.input_info.component_count = load->components;
-			inst.memory                     = {};
-			inst.src_count                  = 0;
-			mutable_input_info->resource_fetch_components[resource_id] =
-			    std::max(mutable_input_info->resource_fetch_components[resource_id],
-			             static_cast<int>(load->components));
-			rewritten += load->components;
-		}
-	}
-
-	return rewritten;
-}
-
 } // namespace
 
 bool TryRecompile(std::span<const uint32_t> code, const CompileOptions& options,
@@ -676,27 +561,6 @@ bool TryRecompile(std::span<const uint32_t> code, const CompileOptions& options,
 	}
 
 	IR::Program ir;
-	LOGF("%s phase begin: stage=%s hash=0x%016" PRIx64 " IR LowerProgram\n", GetDumpLabel(options),
-	     StageName(options.stage), options.shader_hash);
-	const auto lower_program = [&](IR::Program& output) {
-		output = {};
-		if (!IR::LowerProgram(decoded, cfg, options.stage, options.wave_size, output, error)) {
-			return false;
-		}
-		output.shader_hash     = options.shader_hash;
-		output.user_data_base  = options.user_data_base;
-		output.user_data_count = options.user_data_count;
-		output.scratch_dwords  = options.scratch_dwords;
-		return true;
-	};
-	if (!lower_program(ir)) {
-		return false;
-	}
-
-	LOGF("%s phase end: stage=%s hash=0x%016" PRIx64 " IR LowerProgram blocks=%" PRIu64
-	     " elapsed_ms=%" PRIu64 "\n",
-	     GetDumpLabel(options), StageName(options.stage), options.shader_hash,
-	     static_cast<uint64_t>(ir.blocks.size()), phase_ms());
 	const ShaderVertexInputInfo*  vertex  = nullptr;
 	const ShaderPixelInputInfo*   pixel   = nullptr;
 	const ShaderComputeInputInfo* compute = nullptr;
@@ -708,38 +572,59 @@ bool TryRecompile(std::span<const uint32_t> code, const CompileOptions& options,
 	}
 	EmbeddedFetchData embedded_fetch;
 	if (options.stage == ShaderType::Vertex && vertex->fetch_embedded) {
-		embedded_fetch = DetectEmbeddedVertexFetch(decoded, vertex, ir.user_data_base,
-		                                           ir.user_data_count, options.wave_size);
-		auto rewritten = RewriteEmbeddedVertexFetches(ir, vertex, embedded_fetch.loads);
-		if (rewritten > 0 || !embedded_fetch.loads.empty()) {
-			LOGF("%s embedded vertex fetch rewrite: detected=%" PRIu64 " rewritten=%" PRIu32 "\n",
-			     GetDumpLabel(options), static_cast<uint64_t>(embedded_fetch.loads.size()),
-			     rewritten);
+		embedded_fetch = DetectEmbeddedVertexFetch(decoded, vertex, options.user_data_base,
+		                                           options.user_data_count, options.wave_size);
+		if (!embedded_fetch.loads.empty()) {
+			LOGF("%s embedded vertex fetch plan: detected=%" PRIu64 "\n", GetDumpLabel(options),
+			     static_cast<uint64_t>(embedded_fetch.loads.size()));
 		}
 	}
-	ir.dispatcher_fallback = dispatcher_fallback;
-	if (!dispatcher_reason.empty()) {
-		ir.fallback_reason = dispatcher_reason;
-	}
-
-	ir.values = std::make_shared<IR::ValueProgram>();
-	if (!Frontend::TranslateProgram(ir, *ir.values, vertex, pixel, compute, error)) {
+	Frontend::TranslateOptions translate_options {
+	    .stage               = options.stage,
+	    .wave_size           = options.wave_size,
+	    .shader_hash         = options.shader_hash,
+	    .user_data_base      = options.user_data_base,
+	    .user_data_count     = options.user_data_count,
+	    .scratch_dwords      = options.scratch_dwords,
+	    .dispatcher_fallback = dispatcher_fallback,
+	    .cfg_failure_kind    = cfg.failure_kind,
+	    .fallback_reason     = dispatcher_reason.empty() ? cfg.unsupported_reason
+	                                                    : dispatcher_reason,
+	    .vertex              = vertex,
+	    .pixel               = pixel,
+	    .compute             = compute,
+	    .embedded_fetch      = embedded_fetch.loads.empty() ? nullptr : &embedded_fetch,
+	};
+	LOGF("%s phase begin: stage=%s hash=0x%016" PRIx64 " IR TranslateProgram\n",
+	     GetDumpLabel(options), StageName(options.stage), options.shader_hash);
+	if (!Frontend::TranslateProgram(decoded, cfg, translate_options, ir, error)) {
 		return false;
 	}
-	ir.blocks.clear();
-	IR::RewriteToSsa(ir.values->blocks);
-	IR::ConstantPropagationPass(ir.values->blocks);
-	IR::ResolveControlFlowIdentities(*ir.values);
-	IR::RemoveIdentities(ir.values->blocks);
-	IR::EliminateDeadCode(ir.values->blocks);
-	const auto read_lane_stats = IR::EliminateReadLane(*ir.values, ir.wave_size);
+	LOGF("%s phase end: stage=%s hash=0x%016" PRIx64 " IR TranslateProgram blocks=%" PRIu64
+	     " elapsed_ms=%" PRIu64 "\n",
+	     GetDumpLabel(options), StageName(options.stage), options.shader_hash,
+	     static_cast<uint64_t>(ir.blocks.size()), phase_ms());
+	IR::RewriteToSsa(ir.blocks);
+	IR::ConstantPropagationPass(ir.blocks);
+	IR::ResolveControlFlowIdentities(ir);
+	IR::RemoveIdentities(ir.blocks);
+	IR::EliminateDeadCode(ir.blocks);
+	const auto read_lane_stats = IR::EliminateReadLane(ir, ir.wave_size);
 	if (read_lane_stats.rewritten_reads != 0) {
 		LOGF("%s read-lane elimination: reads=%" PRIu32 "\n", GetDumpLabel(options),
 		     read_lane_stats.rewritten_reads);
-		IR::ConstantPropagationPass(ir.values->blocks);
-		IR::ResolveControlFlowIdentities(*ir.values);
-		IR::RemoveIdentities(ir.values->blocks);
-		IR::EliminateDeadCode(ir.values->blocks);
+		IR::ConstantPropagationPass(ir.blocks);
+		IR::ResolveControlFlowIdentities(ir);
+		IR::RemoveIdentities(ir.blocks);
+		IR::EliminateDeadCode(ir.blocks);
+	}
+	if (options.stage == ShaderType::Compute) {
+		const auto lds_barriers =
+		    IR::InsertSharedMemoryBarriers(ir, ir.wave_size, *compute);
+		if (lds_barriers.inserted_barriers != 0) {
+			LOGF("%s wave64 LDS synchronization: barriers=%" PRIu32 "\n", GetDumpLabel(options),
+			     lds_barriers.inserted_barriers);
+		}
 	}
 	std::string srt_error;
 	if (!IR::BuildSrtPlan(ir, &srt_error)) {
@@ -749,11 +634,11 @@ bool TryRecompile(std::span<const uint32_t> code, const CompileOptions& options,
 		}
 		return false;
 	}
-	IR::EliminateDeadCode(ir.values->blocks);
+	IR::EliminateDeadCode(ir.blocks);
 	if (!IR::TrackResources(ir, error)) {
 		return false;
 	}
-	IR::EliminateDeadCode(ir.values->blocks);
+	IR::EliminateDeadCode(ir.blocks);
 	if (options.stage == ShaderType::Vertex) {
 		ir.info.vertex_offset_sgpr = embedded_fetch.vertex_offset_sgpr;
 	}
@@ -780,7 +665,6 @@ bool TryRecompile(std::span<const uint32_t> code, const CompileOptions& options,
 			runtime.read_memory = ReadZeroMemory;
 		}
 		runtime.userdata         = options.read_memory_data;
-		runtime.flat_memory_base = options.flat_memory_base;
 		if (!IR::MaterializeResources(ir, runtime, resources, error)) {
 			return false;
 		}

src/graphics/shader/recompiler/ShaderRecompiler.h

@@ -6,7 +6,6 @@
 #include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
 #include "graphics/shader/shader.h"
 
-#include <optional>
 #include <span>
 #include <vector>
 
@@ -20,7 +19,6 @@ struct CompileOptions {
 	uint32_t                    scratch_dwords  = 0;
 	uint64_t                    shader_hash     = 0;
 	uint64_t                    shader_base     = 0;
-	std::optional<uint64_t>     flat_memory_base;
 	uint32_t                    push_constant_offset      = 0;
 	bool                        dump_ir                    = true;
 	bool                        early_dump                 = false;

src/graphics/shader/recompiler/backend/spirv/SpirvEmitter.cpp

@@ -1,7 +1,7 @@
 #include "graphics/shader/recompiler/backend/spirv/SpirvEmitter.h"
 
 #include "graphics/shader/recompiler/backend/spirv/spirvEmitterInternal.h"
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
+#include "graphics/shader/recompiler/ir/ShaderIR.h"
 #include "graphics/shader/shader.h"
 
 #include <algorithm>
@@ -51,36 +51,34 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 		Expect(Kind::Samplers, Dense(program.info.samplers.size()));
 	}
 	bool uses_gds = false;
-	if (program.values != nullptr) {
-		for (const auto* block: program.values->blocks) {
-			for (const auto& inst: *block) {
-				if (IR::SharedAccessOf(inst.GetOpcode()) == IR::SharedAccess::None) {
-					continue;
-				}
-				const auto index = inst.Flags<IR::MemoryFlags>().index;
-				if (index >= program.values->memory_info.size()) {
-					return Fail(error, "shared operation has invalid memory metadata");
-				}
-				const auto kind = program.values->memory_info[index].kind;
-				if (kind != IR::ResourceKind::Lds && kind != IR::ResourceKind::Gds) {
-					return Fail(error, "shared operation has invalid resource kind");
-				}
-				uses_gds |= kind == IR::ResourceKind::Gds;
+	for (const auto* block: program.blocks) {
+		for (const auto& inst: *block) {
+			if (IR::SharedAccessOf(inst.GetOpcode()) == IR::SharedAccess::None) {
+				continue;
 			}
+			const auto index = inst.Flags<IR::MemoryFlags>().index;
+			if (index >= program.memory_info.size()) {
+				return Fail(error, "shared operation has invalid memory metadata");
+			}
+			const auto kind = program.memory_info[index].kind;
+			if (kind != IR::ResourceKind::Lds && kind != IR::ResourceKind::Gds) {
+				return Fail(error, "shared operation has invalid resource kind");
+			}
+			uses_gds |= kind == IR::ResourceKind::Gds;
 		}
 	}
 	if (uses_gds) {
 		Expect(Kind::Gds);
 	}
-	if (!program.info.addresses.empty()) {
-		Expect(Kind::AddressMemory, Dense(program.info.addresses.size()));
+	if (program.info.uses_dma) {
+		Expect(Kind::BdaPagetable);
+		Expect(Kind::FaultBuffer);
 	}
 	const bool uses_flattened_runtime =
-	    program.values != nullptr &&
-	    (!program.values->srt_reads.empty() ||
+	    !program.srt_reads.empty() ||
 	     std::ranges::any_of(program.info.images, [](const IR::ImageResource& image) {
 		     return image.indirect_mapping_capacity != 0u;
-	     }));
+	     });
 	if (uses_flattened_runtime) {
 		Expect(Kind::FlattenedSrt);
 	}
@@ -107,8 +105,7 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 	    program.bindings.push_constant_offset + program.bindings.push_constant_size >
 	        IR::NativePushConstantSize ||
 	    program.bindings.memory_offset_dword != program.bindings.user_data_registers.size() ||
-	    program.bindings.memory_offset_count !=
-	        program.info.buffers.size() + program.info.addresses.size() ||
+	    program.bindings.memory_offset_count != program.info.buffers.size() ||
 	    (!has_user_storage &&
 	     program.bindings.push_constant_size != program.bindings.ShaderDataDwords() * 4u) ||
 	    (has_user_storage && program.bindings.push_constant_size != 0) ||
@@ -120,9 +117,6 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 		return Fail(error, "native user-data layout is inconsistent");
 	}
 
-	if (program.values == nullptr) {
-		return Fail(error, "native shader plan has no typed SSA");
-	}
 	const auto planning_only_handle = [&](const IR::Inst& handle) {
 		return !handle.Uses().empty() &&
 		       std::ranges::all_of(handle.Uses(), [&](const IR::Use& use) {
@@ -132,11 +126,11 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 				       return false;
 			       }
 			       const auto index = use.user->Flags<IR::MemoryFlags>().index;
-			       return index < program.values->memory_info.size() &&
-			              program.values->memory_info[index].planning_only;
+			       return index < program.memory_info.size() &&
+			              program.memory_info[index].planning_only;
 		       });
 	};
-	for (const auto* block: program.values->blocks) {
+	for (const auto* block: program.blocks) {
 		for (const auto& inst: *block) {
 			const auto dense = inst.Flags<uint32_t>();
 			switch (inst.GetOpcode()) {
@@ -152,8 +146,8 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 					if (planning_only_handle(inst)) {
 						break;
 					}
-					if (dense >= program.info.addresses.size()) {
-						return Fail(error, "typed address handle has an invalid dense resource");
+					if (inst.NumArgs() != 2 || !program.info.uses_dma) {
+						return Fail(error, "typed address handle has invalid DMA metadata");
 					}
 					break;
 				case IR::ValueOpcode::GetScratchResource:
@@ -174,7 +168,7 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 				case IR::ValueOpcode::ReadConst: {
 					const auto slot = inst.Arg(1).Resolve();
 					if (!slot.IsImmediate() || slot.GetType() != IR::Type::U32 ||
-					    slot.U32() >= program.values->srt_reads.size()) {
+					    slot.U32() >= program.srt_reads.size()) {
 						return Fail(error, "flattened SRT read has an invalid dense slot");
 					}
 					break;
@@ -191,31 +185,30 @@ bool ValidateNativeProgram(const IR::Program& program, std::string* error) {
 bool AnalyzeProgramRequirements(IR::Program& program, std::string* error) {
 	program.spirv_requirements.reset();
 	IR::SpirvRequirements requirements {};
-	if (program.values == nullptr) {
-		program.spirv_requirements.emplace(requirements);
-		return true;
-	}
 	const auto MarkBallot = [&] { requirements.subgroup_ballot = true; };
-	for (const auto* block: program.values->blocks) {
+	for (const auto* block: program.blocks) {
 		for (const auto& inst: *block) {
-			if (IR::AddressOpcodeInfoOf(inst.GetOpcode()).access != IR::AddressAccess::None) {
+			const auto address_access = IR::AddressOpcodeInfoOf(inst.GetOpcode()).access;
+			if (address_access != IR::AddressAccess::None) {
 				const auto memory_index = inst.Flags<IR::MemoryFlags>().index;
-				if (memory_index >= program.values->memory_info.size()) {
+				if (memory_index >= program.memory_info.size()) {
 					return Fail(error, "address operation has invalid memory metadata");
 				}
-				if (program.values->memory_info[memory_index].kind == IR::ResourceKind::Scratch) {
+				if (program.memory_info[memory_index].kind == IR::ResourceKind::Scratch) {
 					if (program.scratch_dwords == 0) {
 						return Fail(error, "scratch operation has no per-thread storage");
 					}
 					requirements.function_scratch = true;
+				} else if (address_access == IR::AddressAccess::Write) {
+					return Fail(error, "writable FLAT/GLOBAL addresses require GPU ownership tracking");
 				}
 			}
 			if (IR::BufferAccessOf(inst.GetOpcode()) != IR::BufferAccess::None) {
 				const auto memory_index = inst.Flags<IR::MemoryFlags>().index;
-				if (memory_index >= program.values->memory_info.size()) {
+				if (memory_index >= program.memory_info.size()) {
 					return Fail(error, "buffer operation has invalid memory metadata");
 				}
-				const auto& memory = program.values->memory_info[memory_index];
+				const auto& memory = program.memory_info[memory_index];
 				if (memory.kind == IR::ResourceKind::Buffer) {
 					if (memory.resource >= program.info.buffers.size()) {
 						return Fail(error, "buffer operation has invalid resource metadata");
@@ -231,10 +224,10 @@ bool AnalyzeProgramRequirements(IR::Program& program, std::string* error) {
 			const auto shared_access = IR::SharedAccessOf(inst.GetOpcode());
 			if (shared_access != IR::SharedAccess::None) {
 				const auto index = inst.Flags<IR::MemoryFlags>().index;
-				if (index >= program.values->memory_info.size()) {
+				if (index >= program.memory_info.size()) {
 					return Fail(error, "shared operation has invalid memory metadata");
 				}
-				const auto kind = program.values->memory_info[index].kind;
+				const auto kind = program.memory_info[index].kind;
 				if (kind != IR::ResourceKind::Lds && kind != IR::ResourceKind::Gds) {
 					return Fail(error, "shared operation has invalid resource kind");
 				}
@@ -288,11 +281,11 @@ bool AnalyzeProgramRequirements(IR::Program& program, std::string* error) {
 					break;
 				case IR::ValueOpcode::SetAttribute: {
 					const auto index = inst.Flags<IR::ExportFlags>().index;
-					if (index >= program.values->export_info.size()) {
+					if (index >= program.export_info.size()) {
 						return Fail(error, "attribute export has invalid metadata");
 					}
 					if (program.stage == ShaderType::Pixel &&
-					    program.values->export_info[index].vm) {
+					    program.export_info[index].vm) {
 						requirements.pixel_valid_mask = true;
 					}
 					break;
@@ -330,12 +323,7 @@ bool EmitProgram(const IR::Program& program, const IR::ResourceSnapshot& resourc
 	if (!ValidateNativeProgram(program, error)) {
 		return false;
 	}
-	if (program.values == nullptr) {
-		SetError(error, "SPIR-V emitter requires planned typed SSA");
-		return false;
-	}
-	const auto& value_program = *program.values;
-	if (!IR::ValidateValueProgram(value_program, true, error)) {
+	if (!IR::ValidateProgram(program, true, error)) {
 		return false;
 	}
 	EmitterState state(program, resources, input_info);
@@ -348,7 +336,7 @@ bool EmitProgram(const IR::Program& program, const IR::ResourceSnapshot& resourc
 	AllocateInputVariables(state);
 	AllocateOutputVariables(state);
 	DefineModule(state);
-	if (!EmitValueProgram(state, value_program, error)) {
+	if (!EmitProgram(state, program, error)) {
 		return false;
 	}
 

src/graphics/shader/recompiler/backend/spirv/spirvEmitterAnalysis.cpp

@@ -2,7 +2,7 @@
 #include "graphics/guest_gpu/gpu_defs.h"
 #include "graphics/shader/recompiler/backend/spirv/SpirvEmitter.h"
 #include "graphics/shader/recompiler/backend/spirv/spirvEmitterInternal.h"
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
+#include "graphics/shader/recompiler/ir/ShaderIR.h"
 
 namespace Libs::Graphics::ShaderRecompiler::Spirv::Emitter {
 

src/graphics/shader/recompiler/backend/spirv/spirvEmitterFlow.cpp

@@ -5,13 +5,11 @@
 namespace Libs::Graphics::ShaderRecompiler::Spirv::Emitter {
 namespace {
 
-bool UserDataDwordIndex(const EmitterState& state, IR::Register reg, uint32_t& dword_index) {
-	if (reg.file != IR::RegisterFile::Scalar) {
-		return false;
-	}
+bool UserDataDwordIndex(const EmitterState& state, IR::ScalarReg reg, uint32_t& dword_index) {
+	const auto register_index = IR::RegIndex(reg);
 	const auto& registers = state.program.bindings.user_data_registers;
-	const auto  found     = std::lower_bound(registers.begin(), registers.end(), reg.index);
-	if (found == registers.end() || *found != reg.index) {
+	const auto  found     = std::lower_bound(registers.begin(), registers.end(), register_index);
+	if (found == registers.end() || *found != register_index) {
 		return false;
 	}
 	dword_index = static_cast<uint32_t>(found - registers.begin());
@@ -504,7 +502,7 @@ bool EmitValueFlow(ValueEmitContext& ctx, const IR::Inst& inst) {
 		case IR::ValueOpcode::GetUserData: {
 			const auto reg   = inst.Arg(0).ScalarRegister();
 			uint32_t   dword = 0;
-			if (!UserDataDwordIndex(state, {IR::RegisterFile::Scalar, IR::RegIndex(reg)}, dword)) {
+			if (!UserDataDwordIndex(state, reg, dword)) {
 				ctx.Define(inst, ConstantU32(state, 0));
 			} else {
 				ctx.Define(inst, EmitShaderDataDwordLoad(state, dword));

src/graphics/shader/recompiler/backend/spirv/spirvEmitterImage.cpp

@@ -439,6 +439,49 @@ uint32_t UnpackImageGather(ValueEmitContext& ctx, const IR::MemoryInfo& mem, uin
 	return result;
 }
 
+uint32_t EmitOneDimensionalGatherLz(ValueEmitContext& ctx, const IR::MemoryInfo& mem, uint32_t pc,
+	                                 uint32_t coord, bool integer) {
+	auto& state = ctx.state;
+	state.builder.RequireCapability(CapabilityImageQuery);
+	const auto image = LoadSampledImageDescriptor(state, mem, pc, ImageViewKind::Dim1D);
+	const auto width = state.builder.AllocateId();
+	state.builder.AddFunction(
+	    {OpImageQuerySizeLod, TypeU32(state), width, image, ConstantU32(state, 0)});
+	const auto width_f32 = state.builder.AllocateId();
+	state.builder.AddFunction({OpConvertUToF, TypeF32(state), width_f32, width});
+	const auto left = state.builder.AllocateId();
+	state.builder.AddFunction(
+	    {OpExtInst, TypeF32(state), left, GlslStd450(state), GlslFloor,
+	     Binary(state, OpFSub, TypeF32(state),
+	            Binary(state, OpFMul, TypeF32(state), coord, width_f32),
+	            ConstantF32(state, 0x3f000000u))});
+
+	const auto sampled = MakeSampledImage(state, mem, pc, ImageViewKind::Dim1D);
+	const auto vector_type = integer ? TypeU32Vector(state, 4) : TypeF32Vector(state, 4);
+	const auto scalar_type = integer ? TypeU32(state) : TypeF32(state);
+	const auto component = ImageConversionFormat(state, mem).format == Prospero::BufferFormat::kInvalid
+	                           ? ImageGatherComponent(mem.dmask)
+	                           : 0u;
+	uint32_t values[2] {};
+	for (uint32_t index = 0; index < 2u; index++) {
+		const auto sample_coord = Binary(
+		    state, OpFDiv, TypeF32(state),
+		    Binary(state, OpFAdd, TypeF32(state), left,
+		           ConstantF32(state, index == 0u ? 0x3f000000u : 0x3fc00000u)),
+		    width_f32);
+		const auto texel = state.builder.AllocateId();
+		state.builder.AddFunction({OpImageSampleExplicitLod, vector_type, texel, sampled, sample_coord,
+		                           ImageOperandsLodMask, ZeroF32(state)});
+		values[index] = state.builder.AllocateId();
+		state.builder.AddFunction(
+		    {OpCompositeExtract, scalar_type, values[index], texel, component});
+	}
+	const auto result = state.builder.AllocateId();
+	state.builder.AddFunction({OpCompositeConstruct, vector_type, result, values[0], values[1],
+	                           values[1], values[0]});
+	return result;
+}
+
 uint32_t PackImageTexel(ValueEmitContext& ctx, const IR::MemoryInfo& mem, uint32_t texel) {
 	const auto info = ImageConversionFormat(ctx.state, mem);
 	if (info.format == Prospero::BufferFormat::kInvalid) return texel;
@@ -613,6 +656,22 @@ bool EmitValueImage(ValueEmitContext& ctx, const IR::Inst& inst) {
 		const auto coord =
 		    CoordF32(ctx, mem, *address, layout.coord, ImageViewCoordinateComponents(view));
 		if (op == IR::ValueOpcode::ImageGatherRaw) {
+			if (view == ImageViewKind::Dim1D) {
+				if (dref || !HasFlag(mem, Decoder::ImageSampleFlagLevelZero) ||
+				    HasFlag(mem, Decoder::ImageSampleFlagOffset) ||
+				    HasFlag(mem, Decoder::ImageSampleFlagGatherHorizontal)) {
+					ctx.Fail(inst, "has an unsupported 1D gather variant");
+					return true;
+				}
+				const auto sample = EmitOneDimensionalGatherLz(ctx, mem, pc, coord, integer);
+				ctx.Define(inst, ResultVector(ctx, UnpackImageGather(ctx, mem, sample), integer,
+				                              false, mem, true));
+				return true;
+			}
+			if (view == ImageViewKind::Dim1DArray) {
+				ctx.Fail(inst, "has an unsupported 1D-array gather");
+				return true;
+			}
 			const auto            sampled = MakeSampledImage(state, mem, pc, view);
 			const auto            sample  = state.builder.AllocateId();
 			std::vector<uint32_t> words =

src/graphics/shader/recompiler/backend/spirv/spirvEmitterInternal.h

@@ -6,7 +6,7 @@
 #include "graphics/shader/recompiler/BufferFormat.h"
 #include "graphics/shader/recompiler/backend/spirv/SpirvBuilder.h"
 #include "graphics/shader/recompiler/ir/ShaderIR.h"
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
+#include "graphics/shader/recompiler/ir/ShaderIR.h"
 #include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
 #include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
 
@@ -36,8 +36,10 @@ enum : uint32_t {
 	ExecutionModeSignedZeroInfNanPreserve    = 4461,
 	ExecutionModeDerivativeGroupQuadsKHR     = 5289,
 	AddressingModelLogical                   = 0,
+	AddressingModelPhysicalStorageBuffer64   = 5348,
 	MemoryModelGLSL450                       = 1,
 	CapabilityShader                         = 1,
+	CapabilityInt64                          = 11,
 	CapabilityImageGatherExtended            = 25,
 	CapabilityClipDistance                   = 32,
 	CapabilityCullDistance                   = 33,
@@ -52,6 +54,7 @@ enum : uint32_t {
 	CapabilityShaderViewportIndexLayerEXT    = 5254,
 	CapabilityFragmentBarycentricKHR         = 5284,
 	CapabilityComputeDerivativeGroupQuadsKHR = 5288,
+	CapabilityPhysicalStorageBufferAddresses = 5347,
 	StorageClassUniformConstant              = 0,
 	StorageClassInput                        = 1,
 	StorageClassOutput                       = 3,
@@ -60,6 +63,7 @@ enum : uint32_t {
 	StorageClassPushConstant                 = 9,
 	StorageClassImage                        = 11,
 	StorageClassStorageBuffer                = 12,
+	StorageClassPhysicalStorageBuffer        = 5349,
 	FunctionControlNone                      = 0,
 	SelectionControlNone                     = 0,
 	LoopControlNone                          = 0,
@@ -148,7 +152,9 @@ enum : uint32_t {
 	OpConstantComposite            = 44,
 	OpUndef                        = 1,
 	OpFunction                     = 54,
+	OpFunctionParameter            = 55,
 	OpFunctionEnd                  = 56,
+	OpFunctionCall                 = 57,
 	OpVariable                     = 59,
 	OpImageTexelPointer            = 60,
 	OpLoad                         = 61,
@@ -178,6 +184,8 @@ enum : uint32_t {
 	OpConvertFToS                  = 110,
 	OpConvertSToF                  = 111,
 	OpConvertUToF                  = 112,
+	OpUConvert                     = 113,
+	OpConvertUToPtr                = 120,
 	OpBitcast                      = 124,
 	OpSNegate                      = 126,
 	OpFNegate                      = 127,
@@ -256,11 +264,16 @@ enum : uint32_t {
 	OpSwitch                       = 251,
 	OpKill                         = 252,
 	OpReturn                       = 253,
+	OpReturnValue                  = 254,
 	OpGroupNonUniformBallot        = 339,
 	OpGroupNonUniformBallotFindLSB = 343,
 	OpGroupNonUniformShuffle       = 345,
 };
 
+enum : uint32_t {
+	MemoryAccessAlignedMask = 0x2,
+};
+
 enum : uint32_t {
 	GlslRoundEven       = 2,
 	GlslTrunc           = 3,
@@ -345,9 +358,10 @@ struct EmitterState {
 	ShaderType                   stage                   = ShaderType::Unknown;
 	uint32_t                     wave_size               = 64;
 	uint32_t                     storage_buffer_variable = 0;
-	std::array<uint32_t, IR::ShaderInfo::MaxBuffers + IR::ShaderInfo::MaxAddresses>
-	                                                     memory_byte_offsets {};
-	uint32_t                                             address_memory_variable = 0;
+	std::array<uint32_t, IR::ShaderInfo::MaxBuffers>      memory_byte_offsets {};
+	uint32_t                                             bda_pagetable_variable = 0;
+	uint32_t                                             fault_buffer_variable  = 0;
+	uint32_t                                             bda_pointer_function   = 0;
 	uint32_t                                             gds_variable            = 0;
 	uint32_t                                             gds_length              = 0;
 	uint32_t                                             push_constant_variable  = 0;
@@ -384,6 +398,7 @@ uint32_t TypeBool(EmitterState& state);
 uint32_t TypeBoolVector(EmitterState& state, uint32_t components);
 uint32_t TypeU32(EmitterState& state);
 uint32_t TypeU64(EmitterState& state);
+uint32_t TypeDeviceAddress(EmitterState& state);
 uint32_t TypeU32Pair(EmitterState& state);
 uint32_t TypeI32(EmitterState& state);
 uint32_t TypeI32Pair(EmitterState& state);
@@ -397,6 +412,8 @@ uint32_t TypePointer(EmitterState& state, uint32_t storage_class, uint32_t point
 uint32_t TypeFunction(EmitterState& state);
 uint32_t TypeStorageBufferPointer(EmitterState& state);
 uint32_t TypeStorageBufferElementPointer(EmitterState& state);
+uint32_t TypeDeviceAddressStoragePointer(EmitterState& state);
+uint32_t TypePhysicalU32Pointer(EmitterState& state);
 uint32_t TypePushConstantElementPointer(EmitterState& state);
 uint32_t TypeU32ArrayPointer(EmitterState& state, uint32_t storage_class, uint32_t dwords);
 uint32_t TypeU32ElementPointer(EmitterState& state, uint32_t storage_class);
@@ -407,7 +424,7 @@ inline void EmitLabel(EmitterState& state, uint32_t label) {
 }
 
 struct ValueEmitContext {
-	ValueEmitContext(EmitterState& state_, const IR::ValueProgram& program_)
+	ValueEmitContext(EmitterState& state_, const IR::Program& program_)
 	    : state(state_), program(program_) {}
 
 	uint32_t              Def(IR::Value value);
@@ -425,7 +442,7 @@ struct ValueEmitContext {
 	void                  Fail(const IR::Inst& inst, const char* reason);
 
 	EmitterState&                                                      state;
-	const IR::ValueProgram&                                            program;
+	const IR::Program&                                            program;
 	std::unordered_map<const IR::Inst*, uint32_t>                      definitions;
 	std::unordered_map<const IR::Block*, uint32_t>                     labels;
 	const std::unordered_map<const IR::Inst*, uint32_t>*               dispatcher_spills = nullptr;
@@ -847,7 +864,9 @@ bool EmitValueMemory(ValueEmitContext& ctx, const IR::Inst& inst);
 
 bool EmitValueImage(ValueEmitContext& ctx, const IR::Inst& inst);
 
-bool EmitValueProgram(EmitterState& state, const IR::ValueProgram& program, std::string* error);
+bool EmitProgram(EmitterState& state, const IR::Program& program, std::string* error);
+
+void DefineGetBdaPointer(EmitterState& state);
 
 // These templates accept local lambdas from several emitter translation units.
 template <typename Fn>

src/graphics/shader/recompiler/backend/spirv/spirvEmitterMemory.cpp

@@ -1,5 +1,7 @@
 #include "graphics/shader/recompiler/backend/spirv/spirvEmitterInternal.h"
 
+#include "graphics/host_gpu/renderer/cache/bufferCache.h"
+
 #include <algorithm>
 
 namespace Libs::Graphics::ShaderRecompiler::Spirv::Emitter {
@@ -95,60 +97,146 @@ uint32_t AddU64Low(EmitterState& state, uint32_t low, uint32_t high, uint32_t ad
 	return result;
 }
 
-uint32_t AddressByteAddress(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem,
-                            uint32_t low, uint32_t high) {
+uint32_t ScratchByteAddress(ValueEmitContext& ctx, const IR::MemoryInfo& mem, uint32_t low,
+                            uint32_t high) {
 	auto& state     = ctx.state;
 	auto  immediate = static_cast<int32_t>(mem.offset);
-	if (mem.kind == IR::ResourceKind::ScalarAddress) {
-		immediate = static_cast<int32_t>(static_cast<uint32_t>(immediate) & ~3u);
-		low       = Binary(state, OpBitwiseAnd, TypeU32(state), low, ConstantU32(state, ~3u));
-	}
 	const auto immediate_low  = ConstantU32(state, static_cast<uint32_t>(immediate));
 	const auto immediate_high = ConstantU32(state, immediate < 0 ? UINT32_MAX : 0u);
-	if (mem.kind == IR::ResourceKind::Scratch) {
-		low              = AddU64Low(state, low, high, immediate_low, immediate_high, high);
-		const auto valid = Binary(state, OpIEqual, TypeBool(state), high, ConstantU32(state, 0));
-		return Select(state, TypeU32(state), valid, low, ConstantU32(state, UINT32_MAX));
-	}
-	const auto base = state.program.info.addresses[mem.resource].specialized_base;
-	if (mem.kind == IR::ResourceKind::Flat ||
-	    (mem.kind == IR::ResourceKind::Global &&
-	     state.program.info.addresses[mem.resource].unbased)) {
-		low                      = AddU64Low(state, low, high, immediate_low, immediate_high, high);
-		const auto base_low      = ConstantU32(state, static_cast<uint32_t>(base));
-		const auto base_high     = ConstantU32(state, static_cast<uint32_t>(base >> 32u));
-		const auto relative      = Binary(state, OpISub, TypeU32(state), low, base_low);
-		const auto borrow        = Binary(state, OpULessThan, TypeBool(state), low, base_low);
-		const auto relative_high = Binary(
-		    state, OpISub, TypeU32(state), Binary(state, OpISub, TypeU32(state), high, base_high),
-		    Select(state, TypeU32(state), borrow, ConstantU32(state, 1), ConstantU32(state, 0)));
-		const auto valid =
-		    Binary(state, OpIEqual, TypeBool(state), relative_high, ConstantU32(state, 0));
-		return Select(state, TypeU32(state), valid, relative, ConstantU32(state, UINT32_MAX));
-	}
-	const auto initial       = base + static_cast<uint64_t>(static_cast<int64_t>(immediate));
-	auto       relative_high = ConstantU32(state, static_cast<uint32_t>(initial >> 32u));
-	const auto relative      = AddU64Low(state, ConstantU32(state, static_cast<uint32_t>(initial)),
-	                                     relative_high, low, ConstantU32(state, 0), relative_high);
+	low                      = AddU64Low(state, low, high, immediate_low, immediate_high, high);
 	const auto valid =
-	    Binary(state, OpIEqual, TypeBool(state), relative_high, ConstantU32(state, 0));
-	return Select(state, TypeU32(state), valid, relative, ConstantU32(state, UINT32_MAX));
+	    Binary(state, OpIEqual, TypeBool(state), high, ConstantU32(state, 0));
+	return Select(state, TypeU32(state), valid, low, ConstantU32(state, UINT32_MAX));
 }
 
-uint32_t AddAddressMemoryOffset(ValueEmitContext& ctx, const IR::MemoryInfo& mem,
-                                uint32_t address) {
-	auto&      state = ctx.state;
-	const auto binding =
-	    ResourceForDescriptor(state, IR::DescriptorBindingKind::AddressMemory, mem.resource);
-	const auto offset =
-	    state.memory_byte_offsets[state.program.info.buffers.size() + binding.array_index];
-	const auto corrected = Binary(state, OpIAdd, TypeU32(state), address, offset);
-	const auto not_invalid =
-	    Binary(state, OpINotEqual, TypeBool(state), address, ConstantU32(state, UINT32_MAX));
-	const auto no_overflow =
-	    Binary(state, OpUGreaterThanEqual, TypeBool(state), corrected, address);
-	return Select(state, TypeU32(state), AndCondition(state, not_invalid, no_overflow), corrected,
-	              ConstantU32(state, UINT32_MAX));
+uint32_t ConstantDeviceAddress(EmitterState& state, uint64_t value) {
+	return state.builder.Constant(OpConstant, TypeDeviceAddress(state),
+	                              {static_cast<uint32_t>(value),
+	                               static_cast<uint32_t>(value >> 32u)});
+}
+
+uint32_t DeviceAddressFromWords(EmitterState& state, uint32_t low, uint32_t high) {
+	const auto low64  = Unary(state, OpUConvert, TypeDeviceAddress(state), low);
+	const auto high64 = Binary(state, OpShiftLeftLogical, TypeDeviceAddress(state),
+	                           Unary(state, OpUConvert, TypeDeviceAddress(state), high),
+	                           ConstantDeviceAddress(state, 32));
+	return Binary(state, OpBitwiseOr, TypeDeviceAddress(state), low64, high64);
+}
+
+uint32_t GuestAddress(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem) {
+	auto& state = ctx.state;
+	auto  low   = ctx.Arg(inst, 1);
+	if (mem.kind == IR::ResourceKind::ScalarAddress) {
+		low = Binary(state, OpBitwiseAnd, TypeU32(state), low, ConstantU32(state, ~3u));
+	}
+	uint32_t address = 0;
+	if (mem.address_is_full) {
+		address = DeviceAddressFromWords(state, low, ctx.Arg(inst, 2));
+	} else {
+		const auto* handle = inst.Arg(0).Resolve().TryInstruction();
+		if (handle == nullptr || handle->GetOpcode() != IR::ValueOpcode::GetAddressResource ||
+		    handle->NumArgs() != 2) {
+			ctx.Fail(inst, "has no address base pair");
+			return ConstantDeviceAddress(state, 0);
+		}
+		const auto base = DeviceAddressFromWords(state, ctx.Arg(*handle, 0), ctx.Arg(*handle, 1));
+		address = Binary(state, OpIAdd, TypeDeviceAddress(state), base,
+		                 Unary(state, OpUConvert, TypeDeviceAddress(state), low));
+	}
+	auto immediate = static_cast<int32_t>(mem.offset);
+	if (mem.kind == IR::ResourceKind::ScalarAddress) {
+		immediate = static_cast<int32_t>(static_cast<uint32_t>(immediate) & ~3u);
+	}
+	return immediate == 0
+	           ? address
+	           : Binary(state, OpIAdd, TypeDeviceAddress(state), address,
+	                    ConstantDeviceAddress(state,
+	                                          static_cast<uint64_t>(static_cast<int64_t>(immediate))));
+}
+
+uint32_t FaultElementPointer(EmitterState& state, uint32_t index) {
+	const auto pointer = state.builder.AllocateId();
+	state.builder.AddFunction({OpAccessChain, TypeStorageBufferElementPointer(state), pointer,
+	                           state.fault_buffer_variable, ConstantU32(state, 0), index});
+	return pointer;
+}
+
+void RecordBdaFault(EmitterState& state, uint32_t page) {
+	const auto word = Binary(state, OpShiftRightLogical, TypeU32(state), page,
+	                         ConstantU32(state, 5));
+	const auto bit = Binary(
+	    state, OpShiftLeftLogical, TypeU32(state), ConstantU32(state, 1),
+	    Binary(state, OpBitwiseAnd, TypeU32(state), page, ConstantU32(state, 31)));
+	const auto pointer = FaultElementPointer(state, word);
+	const auto value   = state.builder.AllocateId();
+	state.builder.AddFunction({OpLoad, TypeU32(state), value, pointer});
+	state.builder.AddFunction(
+	    {OpStore, pointer, Binary(state, OpBitwiseOr, TypeU32(state), value, bit)});
+}
+
+uint32_t GetBdaPointer(ValueEmitContext& ctx, uint32_t address) {
+	auto&      state  = ctx.state;
+	const auto result = state.builder.AllocateId();
+	state.builder.AddFunction(
+	    {OpFunctionCall, TypeDeviceAddress(state), result, state.bda_pointer_function, address});
+	return result;
+}
+
+uint32_t LoadBdaDword(ValueEmitContext& ctx, uint32_t address) {
+	auto&      state   = ctx.state;
+	const auto bda     = GetBdaPointer(ctx, address);
+	const auto present = Binary(state, OpINotEqual, TypeBool(state), bda,
+	                            ConstantDeviceAddress(state, 0));
+	return EmitValueOrZeroIfCondition(state, present, [&]() {
+		const auto pointer = state.builder.AllocateId();
+		state.builder.AddFunction(
+		    {OpConvertUToPtr, TypePhysicalU32Pointer(state), pointer, bda});
+		const auto value = state.builder.AllocateId();
+		state.builder.AddFunction(
+		    {OpLoad, TypeU32(state), value, pointer, MemoryAccessAlignedMask, sizeof(uint32_t)});
+		return value;
+	});
+}
+
+uint32_t LoadBda(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem,
+	             uint32_t bits) {
+	auto&      state   = ctx.state;
+	const auto address = GuestAddress(ctx, inst, mem);
+	const auto active  = ctx.Arg(inst, inst.NumArgs() - 1);
+	return EmitValueOrZeroIfCondition(state, active, [&]() {
+		const auto aligned = Binary(state, OpBitwiseAnd, TypeDeviceAddress(state), address,
+		                            ConstantDeviceAddress(state, ~uint64_t {3}));
+		const auto first   = LoadBdaDword(ctx, aligned);
+		const auto byte = Binary(state, OpBitwiseAnd, TypeU32(state),
+		                         Unary(state, OpUConvert, TypeU32(state), address),
+		                         ConstantU32(state, 3));
+		const auto crosses = bits == 8u
+		                          ? ConstantBool(state, false)
+		                          : Binary(state, bits == 16u ? OpUGreaterThan : OpINotEqual,
+		                                   TypeBool(state), byte,
+		                                   ConstantU32(state, bits == 16u ? 2u : 0u));
+		const auto second = EmitValueOrZeroIfCondition(state, crosses, [&]() {
+			return LoadBdaDword(
+			    ctx, Binary(state, OpIAdd, TypeDeviceAddress(state), aligned,
+			                ConstantDeviceAddress(state, sizeof(uint32_t))));
+		});
+		const auto shift = Binary(state, OpShiftLeftLogical, TypeU32(state), byte,
+		                          ConstantU32(state, 3));
+		const auto upper_shift = Binary(
+		    state, OpShiftLeftLogical, TypeU32(state),
+		    Binary(state, OpBitwiseAnd, TypeU32(state),
+		           Binary(state, OpISub, TypeU32(state), ConstantU32(state, 4), byte),
+		           ConstantU32(state, 3)),
+		    ConstantU32(state, 3));
+		const auto merged = Binary(
+		    state, OpBitwiseOr, TypeU32(state),
+		    Binary(state, OpShiftRightLogical, TypeU32(state), first, shift),
+		    Binary(state, OpShiftLeftLogical, TypeU32(state), second, upper_shift));
+		return bits == 32u
+		           ? merged
+		           : Binary(state, OpBitwiseAnd, TypeU32(state), merged,
+		                    ConstantU32(state, bits == 8u ? 0xffu : 0xffffu));
+	});
 }
 
 uint32_t ByteAddress(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem) {
@@ -163,11 +251,10 @@ uint32_t ByteAddress(ValueEmitContext& ctx, const IR::Inst& inst, const IR::Memo
 		return Binary(ctx.state, OpIAdd, TypeU32(ctx.state), ctx.Arg(inst, 0),
 		              ConstantU32(ctx.state, mem.offset));
 	}
-	const auto address = AddressByteAddress(ctx, inst, mem, ctx.Arg(inst, 1), ctx.Arg(inst, 2));
-	if (mem.kind == IR::ResourceKind::Scratch) {
-		return address;
+	if (mem.kind != IR::ResourceKind::Scratch) {
+		EXIT("physical address memory must use the BDA emitter\n");
 	}
-	return AddAddressMemoryOffset(ctx, mem, address);
+	return ScratchByteAddress(ctx, mem, ctx.Arg(inst, 1), ctx.Arg(inst, 2));
 }
 
 uint32_t DwordIndex(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem) {
@@ -436,12 +523,12 @@ void StoreSubwordInBounds(ValueEmitContext& ctx, const IR::MemoryInfo& mem,
 		                     Unary(ctx.state, OpNot, TypeU32(ctx.state), mask)),
 		              value);
 	};
-	if (mem.kind == IR::ResourceKind::Lds || mem.kind == IR::ResourceKind::Gds) {
-		AtomicUpdate(ctx.state, pointer, mem.kind, merge);
-	} else {
+	if (mem.kind == IR::ResourceKind::Scratch) {
 		const auto old = ctx.state.builder.AllocateId();
 		ctx.state.builder.AddFunction({OpLoad, TypeU32(ctx.state), old, pointer});
 		ctx.state.builder.AddFunction({OpStore, pointer, merge(old)});
+	} else {
+		AtomicUpdate(ctx.state, pointer, mem.kind, merge);
 	}
 }
 
@@ -923,6 +1010,58 @@ void StoreWideShared(ValueEmitContext& ctx, const IR::Inst& inst, uint32_t compo
 
 } // namespace
 
+void DefineGetBdaPointer(EmitterState& state) {
+	if (!state.program.info.uses_dma) {
+		return;
+	}
+	const auto type            = TypeDeviceAddress(state);
+	const auto function_type   = state.builder.Type(OpTypeFunction, {type, type});
+	state.bda_pointer_function = state.builder.AllocateId();
+	const auto address         = state.builder.AllocateId();
+	const auto entry_label     = state.builder.AllocateId();
+	state.builder.AddName(state.bda_pointer_function, "get_bda_pointer");
+	state.builder.AddFunction(
+	    {OpFunction, type, state.bda_pointer_function, FunctionControlNone, function_type});
+	state.builder.AddFunction({OpFunctionParameter, type, address});
+	EmitLabel(state, entry_label);
+
+	const auto page64 = Binary(
+	    state, OpShiftRightLogical, type, address,
+	    ConstantDeviceAddress(state, BufferCache::CACHING_PAGEBITS));
+	const auto page = Unary(state, OpUConvert, TypeU32(state), page64);
+	const auto entry_pointer = state.builder.AllocateId();
+	state.builder.AddFunction({OpAccessChain, TypeDeviceAddressStoragePointer(state), entry_pointer,
+	                           state.bda_pagetable_variable, ConstantU32(state, 0), page});
+	const auto base = state.builder.AllocateId();
+	state.builder.AddFunction({OpLoad, type, base, entry_pointer});
+	const auto missing =
+	    Binary(state, OpIEqual, TypeBool(state), base, ConstantDeviceAddress(state, 0));
+	const auto fault_label     = state.builder.AllocateId();
+	const auto available_label = state.builder.AllocateId();
+	const auto merge_label     = state.builder.AllocateId();
+	state.builder.AddFunction({OpSelectionMerge, merge_label, SelectionControlNone});
+	state.builder.AddFunction(
+	    {OpBranchConditional, missing, fault_label, available_label});
+
+	EmitLabel(state, fault_label);
+	RecordBdaFault(state, page);
+	state.builder.AddFunction({OpBranch, merge_label});
+
+	EmitLabel(state, available_label);
+	const auto offset = Binary(
+	    state, OpBitwiseAnd, type, address,
+	    ConstantDeviceAddress(state, BufferCache::CACHING_PAGESIZE - 1));
+	const auto available = Binary(state, OpIAdd, type, base, offset);
+	state.builder.AddFunction({OpBranch, merge_label});
+
+	EmitLabel(state, merge_label);
+	const auto result = state.builder.AllocateId();
+	state.builder.AddFunction({OpPhi, type, result, ConstantDeviceAddress(state, 0), fault_label,
+	                           available, available_label});
+	state.builder.AddFunction({OpReturnValue, result});
+	state.builder.AddFunction({OpFunctionEnd});
+}
+
 bool EmitValueMemory(ValueEmitContext& ctx, const IR::Inst& inst) {
 	auto&      state             = ctx.state;
 	const auto op                = inst.GetOpcode();
@@ -981,6 +1120,10 @@ bool EmitValueMemory(ValueEmitContext& ctx, const IR::Inst& inst) {
 	}
 	const auto address_info = IR::AddressOpcodeInfoOf(op);
 	const bool load_address = address_info.access == IR::AddressAccess::Read;
+	if (load_address && ctx.Memory(inst).kind != IR::ResourceKind::Scratch) {
+		ctx.Define(inst, LoadBda(ctx, inst, ctx.Memory(inst), address_info.data_bits));
+		return true;
+	}
 	const bool load_buffer  = op == IR::ValueOpcode::LoadBufferU8 ||
 	                          op == IR::ValueOpcode::LoadBufferU16 ||
 	                          op == IR::ValueOpcode::LoadBufferU32;

src/graphics/shader/recompiler/backend/spirv/spirvEmitterMemoryHelpers.cpp

@@ -108,20 +108,8 @@ MemoryResourceAccess PrepareMemoryResourceAccess(EmitterState& state, const IR::
 			return access;
 		case IR::ResourceKind::ScalarAddress:
 		case IR::ResourceKind::Flat:
-		case IR::ResourceKind::Global: {
-			if (state.address_memory_variable == 0) {
-				ExitDescriptorBindingFailure(state, IR::DescriptorBindingKind::AddressMemory,
-				                             mem.resource,
-				                             "address memory binding was not emitted");
-			}
-			const auto binding = ResourceForDescriptor(
-			    state, IR::DescriptorBindingKind::AddressMemory, mem.resource);
-			access.object_pointer = state.builder.AllocateId();
-			state.builder.AddFunction({OpAccessChain, TypeStorageBufferPointer(state),
-			                           access.object_pointer, state.address_memory_variable,
-			                           ConstantU32(state, binding.array_index)});
-			break;
-		}
+		case IR::ResourceKind::Global:
+			EXIT("physical address memory must use the BDA emitter\n");
 		case IR::ResourceKind::ScalarBuffer:
 		case IR::ResourceKind::Buffer: {
 			if (state.storage_buffer_variable == 0) {

src/graphics/shader/recompiler/backend/spirv/spirvEmitterModule.cpp

@@ -28,6 +28,10 @@ uint32_t TypeU64(EmitterState& state) {
 	return TypeU32Vector(state, 2);
 }
 
+uint32_t TypeDeviceAddress(EmitterState& state) {
+	return state.builder.Type(OpTypeInt, {64, 0});
+}
+
 uint32_t TypeU32Pair(EmitterState& state) {
 	const auto element = TypeU32(state);
 	return state.builder.Type(OpTypeStruct, {element, element});
@@ -90,6 +94,14 @@ uint32_t TypeStorageBufferElementPointer(EmitterState& state) {
 	return TypePointer(state, StorageClassStorageBuffer, TypeU32(state));
 }
 
+uint32_t TypeDeviceAddressStoragePointer(EmitterState& state) {
+	return TypePointer(state, StorageClassStorageBuffer, TypeDeviceAddress(state));
+}
+
+uint32_t TypePhysicalU32Pointer(EmitterState& state) {
+	return TypePointer(state, StorageClassPhysicalStorageBuffer, TypeU32(state));
+}
+
 uint32_t TypePushConstantElementPointer(EmitterState& state) {
 	return TypePointer(state, StorageClassPushConstant, TypeU32(state));
 }
@@ -130,6 +142,15 @@ uint32_t SampleMaskArrayType(EmitterState& state) {
 	return state.builder.Type(OpTypeArray, {TypeI32(state), ConstantU32(state, 1)});
 }
 
+uint32_t BdaPagetableType(EmitterState& state) {
+	const auto array = state.builder.DecoratedType(
+	    OpTypeRuntimeArray, {TypeDeviceAddress(state)},
+	    {{OpDecorate, {DecorationArrayStride, sizeof(uint64_t)}}});
+	return state.builder.DecoratedType(
+	    OpTypeStruct, {array},
+	    {{OpMemberDecorate, {0, DecorationOffset, 0}}, {OpDecorate, {DecorationBlock}}});
+}
+
 uint32_t F32ArrayType(EmitterState& state, uint32_t count) {
 	return state.builder.Type(OpTypeArray, {TypeF32(state), ConstantU32(state, count)});
 }
@@ -143,13 +164,14 @@ void DefineDescriptorVariables(EmitterState& state) {
 		state.storage_buffer_variable =
 		    state.builder.DefineGlobalVariable(pointer_type, StorageClassStorageBuffer);
 	}
-	if (DescriptorBinding(state, IR::DescriptorBindingKind::AddressMemory) != nullptr) {
-		const auto count =
-		    ConstantU32(state, DescriptorCount(state, IR::DescriptorBindingKind::AddressMemory));
-		const auto array_type = state.builder.Type(OpTypeArray, {StorageBufferType(state), count});
-		const auto pointer_type = TypePointer(state, StorageClassStorageBuffer, array_type);
-		state.address_memory_variable =
-		    state.builder.DefineGlobalVariable(pointer_type, StorageClassStorageBuffer);
+	if (DescriptorBinding(state, IR::DescriptorBindingKind::BdaPagetable) != nullptr) {
+		state.bda_pagetable_variable = state.builder.DefineGlobalVariable(
+		    TypePointer(state, StorageClassStorageBuffer, BdaPagetableType(state)),
+		    StorageClassStorageBuffer);
+	}
+	if (DescriptorBinding(state, IR::DescriptorBindingKind::FaultBuffer) != nullptr) {
+		state.fault_buffer_variable = state.builder.DefineGlobalVariable(
+		    TypeStorageBufferPointer(state), StorageClassStorageBuffer);
 	}
 	if (state.program.bindings.push_constant_size != 0) {
 		const auto pointer_type =
@@ -564,9 +586,13 @@ void AddDescriptorAnnotationsAndNames(EmitterState& state) {
 	if (state.storage_buffer_variable != 0) {
 		Decorate(state.storage_buffer_variable, "buffers", IR::DescriptorBindingKind::Buffers);
 	}
-	if (state.address_memory_variable != 0) {
-		Decorate(state.address_memory_variable, "address_memory",
-		         IR::DescriptorBindingKind::AddressMemory);
+	if (state.bda_pagetable_variable != 0) {
+		Decorate(state.bda_pagetable_variable, "bda_pagetable",
+		         IR::DescriptorBindingKind::BdaPagetable);
+	}
+	if (state.fault_buffer_variable != 0) {
+		Decorate(state.fault_buffer_variable, "fault_buffer",
+		         IR::DescriptorBindingKind::FaultBuffer);
 	}
 	constexpr const char* SampledNames[] = {"sampled_1d",
 	                                        "sampled_1d_array",
@@ -644,6 +670,11 @@ void DefineModule(EmitterState& state) {
 
 	state.builder.RequireCapability(CapabilityShader);
 	state.builder.RequireCapability(CapabilitySignedZeroInfNanPreserve);
+	if (state.program.info.uses_dma) {
+		state.builder.RequireCapability(CapabilityInt64);
+		state.builder.RequireCapability(CapabilityPhysicalStorageBufferAddresses);
+		state.builder.RequireExtension("SPV_KHR_physical_storage_buffer");
+	}
 	if (state.clip_distance_variable != 0) {
 		state.builder.RequireCapability(CapabilityClipDistance);
 	}
@@ -682,7 +713,10 @@ void DefineModule(EmitterState& state) {
 		state.builder.RequireExtension("SPV_KHR_fragment_shader_barycentric");
 	}
 	state.builder.RequireExtension("SPV_KHR_float_controls");
-	state.builder.AddMemoryModel({AddressingModelLogical, MemoryModelGLSL450});
+	state.builder.AddMemoryModel(
+	    {state.program.info.uses_dma ? AddressingModelPhysicalStorageBuffer64
+	                                 : AddressingModelLogical,
+	     MemoryModelGLSL450});
 	state.builder.AddEntryPoint(ExecutionModelForStage(state.stage), state.main_func, "main",
 	                            state.interface_variables);
 	// GCN/RDNA arithmetic preserves 32-bit signed zero, infinity, and NaN. Declaring that

src/graphics/shader/recompiler/backend/spirv/spirvEmitterProgram.cpp

@@ -79,9 +79,9 @@ void StoreDispatcherPhiEdge(ValueEmitContext& ctx, const DispatcherFunctionState
 	}
 }
 
-const IR::Block* TargetBlock(const IR::ValueProgram& program, uint32_t id) {
+const IR::Block* TargetBlock(const IR::Program& program, uint32_t id) {
 	const auto found = std::ranges::find_if(
-	    program.block_info, [&](const IR::ValueBlockInfo& info) { return info.id == id; });
+	    program.block_info, [&](const IR::BlockInfo& info) { return info.id == id; });
 	if (found == program.block_info.end()) {
 		return nullptr;
 	}
@@ -94,7 +94,7 @@ void EmitReturn(ValueEmitContext& ctx) {
 }
 
 void EmitStructuredTerminator(ValueEmitContext& ctx, const IR::Block* block,
-                              const IR::ValueBlockInfo& info) {
+                              const IR::BlockInfo& info) {
 	const auto& term       = info.terminator;
 	const auto  emit_merge = [&]() {
 		if (term.loop_header) {
@@ -150,7 +150,7 @@ void EmitDispatcherTarget(ValueEmitContext& ctx, const DispatcherFunctionState&
 }
 
 uint32_t EmitDispatcherNextPc(ValueEmitContext& ctx, const DispatcherFunctionState& dispatcher,
-                              const IR::Block* block, const IR::ValueBlockInfo& info) {
+                              const IR::Block* block, const IR::BlockInfo& info) {
 	const auto& term = info.terminator;
 	switch (term.kind) {
 		case CFG::TerminatorKind::Branch:
@@ -503,7 +503,7 @@ void ValueEmitContext::Fail(const IR::Inst& inst, const char* reason) {
 	error  = fmt::format("typed opcode {} {}", IR::ValueOpcodeName(inst.GetOpcode()), reason);
 }
 
-bool EmitValueProgram(EmitterState& state, const IR::ValueProgram& program, std::string* error) {
+bool EmitProgram(EmitterState& state, const IR::Program& program, std::string* error) {
 	ValueEmitContext                       ctx(state, program);
 	std::optional<DispatcherFunctionState> dispatcher;
 	if (state.stage == ShaderType::Pixel && state.requirements.pixel_valid_mask) {
@@ -566,6 +566,7 @@ bool EmitValueProgram(EmitterState& state, const IR::ValueProgram& program, std:
 		SetError(error, ctx.error.c_str());
 		return false;
 	}
+	DefineGetBdaPointer(state);
 	for (const auto* block: program.blocks) {
 		if (std::ranges::any_of(*block, [](const IR::Inst& inst) {
 			    return inst.GetOpcode() == IR::ValueOpcode::SwizzleU32 ||

src/graphics/shader/recompiler/frontend/cfg/ShaderCFG.cpp

@@ -1342,7 +1342,7 @@ bool IsolateSemanticLoopHeaders(Graph& graph, std::string* error) {
 		// SPIR-V requires OpLoopMerge and its branch to remain in the loop header's
 		// physical block. Keep that header as a dedicated control node, exactly like
 		// the separate Loop node in shadPS4's structured AST. Guest instructions live
-		// in the body so later lowering may introduce bounds/EXEC control flow without
+		// in the body so later translation may introduce bounds/EXEC control flow without
 		// displacing OpLoopMerge into a helper-created block.
 		if (!IsolateSemanticLoopHeader(graph, loop->header)) {
 			SetFailure(graph, FailureKind::StructuredControlFlow, loop->header,

src/graphics/shader/recompiler/frontend/decode/MemoryOps.cpp

@@ -103,6 +103,9 @@ constexpr MemoryOpcodeInfo DS_OPCODE_LIST[] = {
     {0x4du, Opcode::DS_WRITE_B64, 2, 32},        {0x4eu, Opcode::DS_WRITE2_B64, 4, 32},
     {0x4fu, Opcode::DS_WRITE2ST64_B64, 4, 32},   {0x76u, Opcode::DS_READ_B64, 2, 32},
     {0x77u, Opcode::DS_READ2_B64, 4, 32},        {0x78u, Opcode::DS_READ2ST64_B64, 4, 32},
+    {0xa1u, Opcode::DS_WRITE_B16_D16_HI, 1, 16},
+    {0xa6u, Opcode::DS_READ_U16_D16, 1, 16},
+    {0xa7u, Opcode::DS_READ_U16_D16_HI, 1, 16},
     {0xb0u, Opcode::DS_WRITE_ADDTID_B32, 1, 32}, {0xb1u, Opcode::DS_READ_ADDTID_B32, 1, 32},
     {0xdeu, Opcode::DS_WRITE_B96, 3, 32},        {0xdfu, Opcode::DS_WRITE_B128, 4, 32},
     {0xfeu, Opcode::DS_READ_B96, 3, 32},         {0xffu, Opcode::DS_READ_B128, 4, 32},
@@ -146,6 +149,7 @@ bool IsDsWriteOpcode(Opcode opcode) {
 	switch (opcode) {
 		case Opcode::DS_WRITE_B8:
 		case Opcode::DS_WRITE_B16:
+		case Opcode::DS_WRITE_B16_D16_HI:
 		case Opcode::DS_WRITE2_B32:
 		case Opcode::DS_WRITE2ST64_B32:
 		case Opcode::DS_WRITE2_B64:
@@ -420,8 +424,17 @@ void DecodeDs(uint32_t pc, std::span<const uint32_t> code, uint32_t word_index,
 	}
 
 	DecodeVectorGpr(vdst, inst.dst);
+	if (inst.opcode == Opcode::DS_READ_U16_D16 ||
+	    inst.opcode == Opcode::DS_READ_U16_D16_HI) {
+		// Native D16 reads update only the selected destination half. Reuse the partial
+		// destination representation so typed IR translation preserves the other half.
+		inst.dst.sdwa_sel = inst.opcode == Opcode::DS_READ_U16_D16 ? 4u : 5u;
+	}
 	DecodeVectorGpr(addr, inst.src0);
 	DecodeVectorGpr(data0, inst.src1);
+	if (inst.opcode == Opcode::DS_WRITE_B16_D16_HI) {
+		inst.src1.sdwa_sel = 5u;
+	}
 	DecodeVectorGpr(data1, inst.src2);
 	inst.src_count = DsSourceCount(inst.opcode);
 }

src/graphics/shader/recompiler/frontend/decode/ShaderDecoder.cpp

@@ -675,6 +675,7 @@ std::string InstructionToString(const Instruction& inst) {
 		case Opcode::DS_READ_B128:
 		case Opcode::DS_WRITE_B8:
 		case Opcode::DS_WRITE_B16:
+		case Opcode::DS_WRITE_B16_D16_HI:
 		case Opcode::DS_WRITE2_B32:
 		case Opcode::DS_WRITE2ST64_B32:
 		case Opcode::DS_WRITE2_B64:

src/graphics/shader/recompiler/frontend/decode/ShaderDecoder.h

@@ -185,6 +185,7 @@ enum class Opcode {
 	V_BFREV_B32,
 	V_FFBH_U32,
 	V_FFBL_B32,
+	V_FFBH_I32,
 	V_ADD_F32,
 	V_SUB_F32,
 	V_SUBREV_F32,
@@ -299,6 +300,8 @@ enum class Opcode {
 	V_LSHRREV_B32,
 	V_ASHR_I32,
 	V_ASHRREV_I32,
+	V_LSHLREV_B64,
+	V_LSHRREV_B64,
 	V_LSHLREV_B16,
 	V_LSHRREV_B16,
 	V_ASHRREV_I16,
@@ -364,6 +367,7 @@ enum class Opcode {
 	V_CMPX_LE_F16,
 	V_CMPX_GT_F16,
 	V_CMPX_GE_F16,
+	V_CMPX_NGT_F16,
 	V_CMPX_NEQ_F16,
 	V_CMPX_NLT_F16,
 	V_CMPX_LT_I32,
@@ -376,6 +380,7 @@ enum class Opcode {
 	V_CMP_EQ_U16,
 	V_CMP_LE_U16,
 	V_CMP_GT_U16,
+	V_CMPX_GT_U16,
 	V_CMP_NE_U16,
 	V_CMP_GE_U16,
 	V_CMP_F_U32,
@@ -387,8 +392,10 @@ enum class Opcode {
 	V_CMP_GE_U32,
 	V_CMP_T_U32,
 	V_CMP_EQ_I64,
+	V_CMP_EQ_U64,
 	V_CMP_GT_U64,
 	V_CMP_NE_U64,
+	V_CMPX_NE_I64,
 	V_CMPX_NE_U64,
 	V_CMPX_LT_U32,
 	V_CMPX_EQ_U32,
@@ -491,6 +498,8 @@ enum class Opcode {
 	DS_READ_U8,
 	DS_READ_I16,
 	DS_READ_U16,
+	DS_READ_U16_D16,
+	DS_READ_U16_D16_HI,
 	DS_READ2_B32,
 	DS_READ2ST64_B32,
 	DS_READ_B32,
@@ -501,6 +510,7 @@ enum class Opcode {
 	DS_READ_B128,
 	DS_WRITE_B8,
 	DS_WRITE_B16,
+	DS_WRITE_B16_D16_HI,
 	DS_WRITE2_B32,
 	DS_WRITE2ST64_B32,
 	DS_WRITE2_B64,

src/graphics/shader/recompiler/frontend/decode/VectorAluOps.cpp

@@ -7,7 +7,7 @@ namespace {
 
 using Detail::OpcodeMap;
 
-// These profiles describe the selectors that the current lowering implements. They are not
+// These profiles describe the selectors that the current translator implements. They are not
 // architectural SDWA legality classes.
 enum class Vop2SdwaProfile {
 	None,
@@ -59,7 +59,7 @@ constexpr Vop2OpcodeInfo VOP2_OPCODE_LIST[] = {
     {0x23u, Opcode::V_MBCNT_LO_U32_B32},
     {0x24u, Opcode::V_MBCNT_HI_U32_B32},
     {0x25u, Opcode::V_ADD_NC_U32, Vop2SdwaProfile::IntegerPartialDestination},
-    {0x28u, Opcode::V_ADDC_U32},
+    {0x28u, Opcode::V_ADDC_U32, Vop2SdwaProfile::IntegerFullDestination},
     {0x2au, Opcode::V_SUBREV_CO_CI_U32},
     {0x26u, Opcode::V_SUB_NC_U32, Vop2SdwaProfile::IntegerPartialDestination},
     {0x27u, Opcode::V_SUBREV_NC_U32, Vop2SdwaProfile::IntegerFullDestination},
@@ -119,6 +119,7 @@ constexpr OpcodeMap VOP1_OPCODE_LIST[] = {
     {0x38u, Opcode::V_BFREV_B32},
     {0x39u, Opcode::V_FFBH_U32},
     {0x3au, Opcode::V_FFBL_B32},
+    {0x3bu, Opcode::V_FFBH_I32},
     {0x3fu, Opcode::V_FREXP_EXP_I32_F32},
     {0x40u, Opcode::V_FREXP_MANT_F32},
     {0x42u, Opcode::V_MOVRELD_B32},
@@ -169,6 +170,7 @@ constexpr OpcodeMap VOP3_ENCODED_VOP1_OPCODE_LIST[] = {
     {0x38u, Opcode::V_BFREV_B32},
     {0x39u, Opcode::V_FFBH_U32},
     {0x3au, Opcode::V_FFBL_B32},
+    {0x3bu, Opcode::V_FFBH_I32},
     {0x3fu, Opcode::V_FREXP_EXP_I32_F32},
     {0x40u, Opcode::V_FREXP_MANT_F32},
     {0x42u, Opcode::V_MOVRELD_B32},
@@ -224,15 +226,18 @@ constexpr VopcOpcodeInfo VOPC_OPCODE_LIST[] = {
     {0x96u, Opcode::V_CMPX_GE_I32},       {0xa9u, Opcode::V_CMP_LT_U16},
     {0xaau, Opcode::V_CMP_EQ_U16},        {0xabu, Opcode::V_CMP_LE_U16},
     {0xacu, Opcode::V_CMP_GT_U16},        {0xadu, Opcode::V_CMP_NE_U16},
-    {0xaeu, Opcode::V_CMP_GE_U16},        {0xc0u, Opcode::V_CMP_F_U32},
+    {0xaeu, Opcode::V_CMP_GE_U16},        {0xbcu, Opcode::V_CMPX_GT_U16},
+    {0xc0u, Opcode::V_CMP_F_U32},
     {0xc1u, Opcode::V_CMP_LT_U32},        {0xc2u, Opcode::V_CMP_EQ_U32},
     {0xc3u, Opcode::V_CMP_LE_U32},        {0xc4u, Opcode::V_CMP_GT_U32},
     {0xc5u, Opcode::V_CMP_NE_U32},        {0xc6u, Opcode::V_CMP_GE_U32},
     {0xc7u, Opcode::V_CMP_T_U32},         {0xa2u, Opcode::V_CMP_EQ_I64, false},
+    {0xb5u, Opcode::V_CMPX_NE_I64, false},
     {0xd1u, Opcode::V_CMPX_LT_U32},       {0xd2u, Opcode::V_CMPX_EQ_U32},
     {0xd3u, Opcode::V_CMPX_LE_U32},       {0xd4u, Opcode::V_CMPX_GT_U32},
     {0xd5u, Opcode::V_CMPX_NE_U32},       {0xd6u, Opcode::V_CMPX_GE_U32},
-    {0xe4u, Opcode::V_CMP_GT_U64, false}, {0xe5u, Opcode::V_CMP_NE_U64, false},
+    {0xe2u, Opcode::V_CMP_EQ_U64, false}, {0xe4u, Opcode::V_CMP_GT_U64, false},
+    {0xe5u, Opcode::V_CMP_NE_U64, false},
     {0xf5u, Opcode::V_CMPX_NE_U64, false},
     {0xc9u, Opcode::V_CMP_LT_F16},        {0xcau, Opcode::V_CMP_EQ_F16},
     {0xcbu, Opcode::V_CMP_LE_F16},        {0xccu, Opcode::V_CMP_GT_F16},
@@ -240,7 +245,8 @@ constexpr VopcOpcodeInfo VOPC_OPCODE_LIST[] = {
     {0xedu, Opcode::V_CMP_NEQ_F16},       {0xd9u, Opcode::V_CMPX_LT_F16},
     {0xdau, Opcode::V_CMPX_EQ_F16},       {0xdbu, Opcode::V_CMPX_LE_F16},
     {0xdcu, Opcode::V_CMPX_GT_F16},       {0xdeu, Opcode::V_CMPX_GE_F16},
-    {0xfdu, Opcode::V_CMPX_NEQ_F16},      {0xfeu, Opcode::V_CMPX_NLT_F16},
+    {0xfbu, Opcode::V_CMPX_NGT_F16},      {0xfdu, Opcode::V_CMPX_NEQ_F16},
+    {0xfeu, Opcode::V_CMPX_NLT_F16},
 };
 
 constexpr auto VOPC_OPS = Detail::MakeOpcodeTable<0x100>(VOPC_OPCODE_LIST);
@@ -280,6 +286,8 @@ constexpr OpcodeMap VOP3_OPCODE_LIST[] = {
     {0x16au, Opcode::V_MUL_HI_U32},
     {0x16bu, Opcode::V_MUL_LO_I32},
     {0x16cu, Opcode::V_MUL_HI_I32},
+    {0x2ffu, Opcode::V_LSHLREV_B64},
+    {0x300u, Opcode::V_LSHRREV_B64},
     {0x303u, Opcode::V_ADD_NC_U16},
     {0x304u, Opcode::V_SUB_NC_U16},
     {0x307u, Opcode::V_LSHRREV_B16},
@@ -510,6 +518,8 @@ constexpr Vop1SdwaRule VOP1_SDWA_RULES[] = {
     {Opcode::V_CVT_F32_U32, SdwaSelBytes() | SdwaSelWords() | SdwaSelFull(), 0, 0, false},
     {Opcode::V_CVT_F32_I32, SdwaSelBytes() | SdwaSelWords() | SdwaSelFull(), 0, 0, false},
     {Opcode::V_CVT_F32_UBYTE0, SdwaSelBytes() | SdwaSelWords() | SdwaSelFull(), 0, 0, false},
+    {Opcode::V_NOT_B32, SdwaSelBytes() | SdwaSelWords() | SdwaSelFull(), 0, 0, false},
+    {Opcode::V_FFBL_B32, SdwaSelBytes() | SdwaSelWords() | SdwaSelFull(), 0, 0, false},
     {Opcode::V_CVT_F32_F16, SdwaSelWords() | SdwaSelFull(), 0, 0, true},
     {Opcode::V_CVT_F16_F32, SdwaSelFull(), SdwaSelWords(), SdwaSelFull(), false},
     {Opcode::V_CVT_F16_U16, SdwaSelWords() | SdwaSelFull(), SdwaSelWords(),
@@ -870,6 +880,7 @@ bool IsVopcFloatCompareOpcode(Opcode opcode) {
 		case Opcode::V_CMPX_LE_F16:
 		case Opcode::V_CMPX_GT_F16:
 		case Opcode::V_CMPX_GE_F16:
+		case Opcode::V_CMPX_NGT_F16:
 		case Opcode::V_CMPX_NEQ_F16:
 		case Opcode::V_CMPX_NLT_F16:
 		case Opcode::V_CMP_CLASS_F32: return true;
@@ -934,8 +945,8 @@ constexpr Vop2SdwaRule VOP2_SDWA_RULES[] = {
      true, true},
     {SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), false, false},
     {SdwaSelAll(), SdwaSelAll(), SdwaSelAll(), true, false},
-    {SdwaSelFull(), SdwaSelFull(), SdwaSelWords() | SdwaSelFull(), false, false},
-    {SdwaSelFull(), SdwaSelFull(), SdwaSelAll(), false, false},
+    {SdwaSelFull(), SdwaSelAll(), SdwaSelWords() | SdwaSelFull(), false, false},
+    {SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), false, false},
     {SdwaSelWords() | SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), true, false},
 };
 static_assert(sizeof(VOP2_SDWA_RULES) / sizeof(VOP2_SDWA_RULES[0]) ==
@@ -1246,6 +1257,8 @@ uint32_t NativeVop3SourceCount(Opcode opcode) {
 		case Opcode::V_MIN_I16:
 		case Opcode::V_ADD_NC_I16:
 		case Opcode::V_SUB_NC_I16:
+		case Opcode::V_LSHLREV_B64:
+		case Opcode::V_LSHRREV_B64:
 		case Opcode::V_LSHLREV_B16:
 		case Opcode::V_LSHRREV_B16:
 		case Opcode::V_ASHRREV_I16:
@@ -1499,12 +1512,15 @@ bool IsVopcCompareExec(Opcode opcode) {
 		case Opcode::V_CMPX_GT_U32:
 		case Opcode::V_CMPX_NE_U32:
 		case Opcode::V_CMPX_GE_U32:
+		case Opcode::V_CMPX_NE_I64:
 		case Opcode::V_CMPX_NE_U64:
+		case Opcode::V_CMPX_GT_U16:
 		case Opcode::V_CMPX_LT_F16:
 		case Opcode::V_CMPX_EQ_F16:
 		case Opcode::V_CMPX_LE_F16:
 		case Opcode::V_CMPX_GT_F16:
 		case Opcode::V_CMPX_GE_F16:
+		case Opcode::V_CMPX_NGT_F16:
 		case Opcode::V_CMPX_NEQ_F16:
 		case Opcode::V_CMPX_NLT_F16: return true;
 		default: return false;

src/graphics/shader/recompiler/frontend/translate/Attribute.cpp

@@ -1,42 +1,108 @@
 #include "graphics/shader/recompiler/frontend/translate/Translator.h"
+#include "graphics/shader/shader.h"
 
 #include <algorithm>
 #include <array>
+#include <fmt/format.h>
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
-IR::ExportFlags Translator::AddExportInfo(const IR::Instruction& inst) {
-	const auto index = static_cast<uint32_t>(value_program.export_info.size());
-	value_program.export_info.push_back(inst.export_info);
+namespace {
+
+IR::ExportTargetKind ExportTargetKindFromTarget(uint32_t target, uint32_t& index) {
+	index = 0;
+	switch (target) {
+		case 0x08u: return IR::ExportTargetKind::MrtZ;
+		case 0x09u: return IR::ExportTargetKind::Null;
+		case 0x14u: return IR::ExportTargetKind::Primitive;
+		default: break;
+	}
+	if (target <= 0x07u) {
+		index = target;
+		return IR::ExportTargetKind::Mrt;
+	}
+	if (target >= 0x0cu && target <= 0x0fu) {
+		index = target - 0x0cu;
+		return IR::ExportTargetKind::Position;
+	}
+	if (target >= 0x20u && target <= 0x3fu) {
+		index = target - 0x20u;
+		return IR::ExportTargetKind::Parameter;
+	}
+	return IR::ExportTargetKind::Unknown;
+}
+
+} // namespace
+
+IR::ExportFlags Translator::AddExportInfo(const Decoder::Instruction& inst) {
+	IR::ExportInfo info;
+	info.kind        = ExportTargetKindFromTarget(inst.exp.target, info.index);
+	info.target      = inst.exp.target;
+	info.en          = inst.exp.en;
+	info.done        = inst.exp.done;
+	info.compr       = inst.exp.compr;
+	info.vm          = inst.exp.vm;
+	const auto index = static_cast<uint32_t>(program.export_info.size());
+	program.export_info.push_back(info);
 	return {.index = index, .pc = inst.pc};
 }
 
-bool Translator::TranslateAttributeOperation(const IR::Instruction& inst) {
-	if (inst.op == IR::Opcode::LoadInputF32) {
-		for (uint32_t component = 0; component < inst.input_info.component_count; component++) {
-			const auto chan  = inst.input_info.chan + component;
-			const auto value = inst.input_info.interpolation_mode < 3u
-			                       ? ir.Emit(IR::ValueOpcode::GetInterpolationParameter,
-			                                 {IR::Value(inst.input_info.attr), IR::Value(chan),
-			                                  IR::Value(inst.input_info.interpolation_mode)})
-			                       : ir.Emit(IR::ValueOpcode::GetAttribute,
-			                                 {IR::Value(inst.input_info.attr), IR::Value(chan)});
-			WriteOperand(OffsetOperand(inst.dst, component), value);
-		}
-		return true;
+bool Translator::TranslateEmbeddedFetch(const Decoder::Instruction& inst, uint32_t attribute,
+                                        uint32_t component_count) {
+	for (uint32_t component = 0; component < component_count; component++) {
+		const auto value =
+		    ir.Emit(IR::ValueOpcode::GetAttribute, {IR::Value(attribute), IR::Value(component)});
+		WriteOperand(OffsetOperand(inst.dst, component), value);
 	}
-	if (inst.op == IR::Opcode::Export) {
-		std::array<IR::Value, 4> components {IR::Value(0u), IR::Value(0u), IR::Value(0u),
-		                                     IR::Value(0u)};
-		for (uint32_t index = 0; index < std::min(inst.src_count, 4u); index++) {
-			components[index] = ReadRawU32(PlainOperand(inst.src[index]));
-		}
-		const auto data = ir.Emit(IR::ValueOpcode::CompositeConstructU32x4,
-		                          {components[0], components[1], components[2], components[3]});
-		ir.Emit(IR::ValueOpcode::SetAttribute, {data, ir.GetExec()}, AddExportInfo(inst));
-		return true;
-	}
-	return false;
+	return true;
 }
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+void Translator::V_INTERP_P1_F32() {}
+
+void Translator::V_INTERP_P2_F32(const Decoder::Instruction& inst) {
+	const auto value = ir.Emit(IR::ValueOpcode::GetAttribute,
+	                           {IR::Value(inst.src1.value), IR::Value(inst.src2.value)});
+	WriteOperand(inst.dst, value);
+}
+
+bool Translator::V_INTERP_MOV_F32(const Decoder::Instruction& inst, std::string* error) {
+	if (inst.src0.value >= 3u) {
+		return ShaderError::Fail(error,
+		                         fmt::format("v_interp_mov_f32 mode {} is reserved at pc 0x{:08x}",
+		                                     inst.src0.value, inst.pc));
+	}
+	const auto value = ir.Emit(
+	    IR::ValueOpcode::GetInterpolationParameter,
+	    {IR::Value(inst.src1.value), IR::Value(inst.src2.value), IR::Value(inst.src0.value)});
+	WriteOperand(inst.dst, value);
+	return true;
+}
+
+bool Translator::EXP(const Decoder::Instruction& inst, std::string* error) {
+	uint32_t index = 0;
+	if (ExportTargetKindFromTarget(inst.exp.target, index) == IR::ExportTargetKind::Unknown) {
+		return ShaderError::Fail(error,
+		                         fmt::format("unsupported EXP target 0x{:02x} at pc 0x{:08x}",
+		                                     inst.exp.target, inst.pc));
+	}
+	std::array<IR::Value, 4> components {IR::Value(0u), IR::Value(0u), IR::Value(0u),
+	                                     IR::Value(0u)};
+	for (uint32_t source = 0; source < std::min(inst.src_count, 4u); source++) {
+		components[source] = ReadRawU32(PlainOperand(SourceAt(inst, source)));
+	}
+	const auto data = ir.Emit(IR::ValueOpcode::CompositeConstructU32x4,
+	                          {components[0], components[1], components[2], components[3]});
+	ir.Emit(IR::ValueOpcode::SetAttribute, {data, ir.GetExec()}, AddExportInfo(inst));
+	return true;
+}
+
+bool Translator::EmitInterpolation(const Decoder::Instruction& inst, std::string* error) {
+	switch (inst.opcode) {
+		case Decoder::Opcode::V_INTERP_P1_F32: V_INTERP_P1_F32(); return true;
+		case Decoder::Opcode::V_INTERP_P2_F32: V_INTERP_P2_F32(inst); return true;
+		case Decoder::Opcode::V_INTERP_MOV_F32: return V_INTERP_MOV_F32(inst, error);
+		default: return false;
+	}
+}
+
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Compare.cpp

@@ -1,199 +1,65 @@
 #include "graphics/shader/recompiler/frontend/translate/Translator.h"
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
-bool Translator::TranslateIntegerCompare(const IR::Instruction& inst) {
-	IR::ValueOpcode opcode {};
-	IR::Type        type = IR::Type::U32;
-	switch (inst.op) {
-		case IR::Opcode::CompareFalse:
-			WriteCompareResult(inst.dst, IR::U1(IR::Value(false)));
-			return true;
-		case IR::Opcode::CompareTrue:
-			WriteCompareResult(inst.dst, IR::U1(IR::Value(true)));
-			return true;
-		case IR::Opcode::CompareEqU32:
-		case IR::Opcode::CompareMaskEqU32:
-		case IR::Opcode::CompareEqI32:
-		case IR::Opcode::CompareMaskEqI32: opcode = IR::ValueOpcode::IEqual32; break;
-		case IR::Opcode::CompareNeU32:
-		case IR::Opcode::CompareMaskNeU32:
-		case IR::Opcode::CompareNeI32:
-		case IR::Opcode::CompareMaskNeI32: opcode = IR::ValueOpcode::INotEqual32; break;
-		case IR::Opcode::CompareGtU32:
-		case IR::Opcode::CompareMaskGtU32: opcode = IR::ValueOpcode::UGreaterThan32; break;
-		case IR::Opcode::CompareGeU32:
-		case IR::Opcode::CompareMaskGeU32: opcode = IR::ValueOpcode::UGreaterThanEqual32; break;
-		case IR::Opcode::CompareLtU32:
-		case IR::Opcode::CompareMaskLtU32: opcode = IR::ValueOpcode::ULessThan32; break;
-		case IR::Opcode::CompareLeU32:
-		case IR::Opcode::CompareMaskLeU32: opcode = IR::ValueOpcode::ULessThanEqual32; break;
-		case IR::Opcode::CompareGtI32:
-		case IR::Opcode::CompareMaskGtI32: opcode = IR::ValueOpcode::SGreaterThan32; break;
-		case IR::Opcode::CompareGeI32:
-		case IR::Opcode::CompareMaskGeI32: opcode = IR::ValueOpcode::SGreaterThanEqual32; break;
-		case IR::Opcode::CompareLtI32:
-		case IR::Opcode::CompareMaskLtI32: opcode = IR::ValueOpcode::SLessThan32; break;
-		case IR::Opcode::CompareLeI32:
-		case IR::Opcode::CompareMaskLeI32: opcode = IR::ValueOpcode::SLessThanEqual32; break;
-		case IR::Opcode::CompareEqU64:
-			opcode = IR::ValueOpcode::IEqual64;
-			type   = IR::Type::U64;
-			break;
-		case IR::Opcode::CompareGtU64:
-			opcode = IR::ValueOpcode::UGreaterThan64;
-			type   = IR::Type::U64;
-			break;
-		case IR::Opcode::CompareNeU64:
-			opcode = IR::ValueOpcode::INotEqual64;
-			type   = IR::Type::U64;
-			break;
-		default: return false;
+void Translator::EmitCompareResult(const Decoder::Instruction& inst, IR::U1 value, bool scalar,
+                                   bool cmpx) {
+	if (scalar) {
+		ir.SetScc(value);
+		return;
 	}
-	const auto lhs = ReadOperand(inst.src[0], type);
-	const auto rhs = ReadOperand(inst.src[1], type);
-	WriteCompareResult(inst.dst, IR::U1(ir.Emit(opcode, {lhs, rhs})));
-	return true;
+	const auto masked = ir.LogicalAnd(ir.GetExec(), value);
+	if (cmpx) {
+		const auto mask = BallotMask(masked);
+		ir.SetExec(masked);
+		ir.SetExecLo(mask[0]);
+		ir.SetExecHi(mask[1]);
+		return;
+	}
+	WriteMask(inst.dst, masked);
 }
 
-bool Translator::TranslateInteger16Compare(const IR::Instruction& inst) {
-	IR::ValueOpcode opcode {};
-	bool            signed_value = false;
-	switch (inst.op) {
-		case IR::Opcode::CompareEqU16:
-		case IR::Opcode::CompareEqI16: opcode = IR::ValueOpcode::IEqual32; break;
-		case IR::Opcode::CompareNeU16:
-		case IR::Opcode::CompareNeI16: opcode = IR::ValueOpcode::INotEqual32; break;
-		case IR::Opcode::CompareGtU16: opcode = IR::ValueOpcode::UGreaterThan32; break;
-		case IR::Opcode::CompareGeU16: opcode = IR::ValueOpcode::UGreaterThanEqual32; break;
-		case IR::Opcode::CompareLtU16: opcode = IR::ValueOpcode::ULessThan32; break;
-		case IR::Opcode::CompareLeU16: opcode = IR::ValueOpcode::ULessThanEqual32; break;
-		case IR::Opcode::CompareGtI16:
-			opcode       = IR::ValueOpcode::SGreaterThan32;
-			signed_value = true;
-			break;
-		case IR::Opcode::CompareGeI16:
-			opcode       = IR::ValueOpcode::SGreaterThanEqual32;
-			signed_value = true;
-			break;
-		case IR::Opcode::CompareLtI16:
-			opcode       = IR::ValueOpcode::SLessThan32;
-			signed_value = true;
-			break;
-		case IR::Opcode::CompareLeI16:
-			opcode       = IR::ValueOpcode::SLessThanEqual32;
-			signed_value = true;
-			break;
-		default: return false;
-	}
-	if (inst.op == IR::Opcode::CompareEqI16 || inst.op == IR::Opcode::CompareNeI16) {
-		signed_value = true;
-	}
-	WriteCompareResult(inst.dst,
-	                   IR::U1(ir.Emit(opcode, {ReadU16AsU32(inst.src[0], signed_value),
-	                                           ReadU16AsU32(inst.src[1], signed_value)})));
-	return true;
+void Translator::EmitCompareConstant(const Decoder::Instruction& inst, bool value, bool scalar,
+                                     bool cmpx) {
+	EmitCompareResult(inst, IR::U1(IR::Value(value)), scalar, cmpx);
 }
 
-bool Translator::TranslateFloatCompare(const IR::Instruction& inst) {
-	IR::ValueOpcode opcode {};
-	bool            half = false;
-	switch (inst.op) {
-		case IR::Opcode::CompareEqF32:
-		case IR::Opcode::CompareMaskEqF32: opcode = IR::ValueOpcode::FPOrdEqual32; break;
-		case IR::Opcode::CompareNeF32:
-		case IR::Opcode::CompareMaskNeF32: opcode = IR::ValueOpcode::FPOrdNotEqual32; break;
-		case IR::Opcode::CompareGtF32:
-		case IR::Opcode::CompareMaskGtF32: opcode = IR::ValueOpcode::FPOrdGreaterThan32; break;
-		case IR::Opcode::CompareGeF32:
-		case IR::Opcode::CompareMaskGeF32: opcode = IR::ValueOpcode::FPOrdGreaterThanEqual32; break;
-		case IR::Opcode::CompareLtF32:
-		case IR::Opcode::CompareMaskLtF32: opcode = IR::ValueOpcode::FPOrdLessThan32; break;
-		case IR::Opcode::CompareLeF32:
-		case IR::Opcode::CompareMaskLeF32: opcode = IR::ValueOpcode::FPOrdLessThanEqual32; break;
-		case IR::Opcode::CompareUnordEqF32:
-		case IR::Opcode::CompareMaskUnordEqF32: opcode = IR::ValueOpcode::FPUnordEqual32; break;
-		case IR::Opcode::CompareUnordNeF32:
-		case IR::Opcode::CompareMaskUnordNeF32: opcode = IR::ValueOpcode::FPUnordNotEqual32; break;
-		case IR::Opcode::CompareUnordGtF32:
-		case IR::Opcode::CompareMaskUnordGtF32:
-			opcode = IR::ValueOpcode::FPUnordGreaterThan32;
-			break;
-		case IR::Opcode::CompareUnordGeF32:
-		case IR::Opcode::CompareMaskUnordGeF32:
-			opcode = IR::ValueOpcode::FPUnordGreaterThanEqual32;
-			break;
-		case IR::Opcode::CompareUnordLtF32:
-		case IR::Opcode::CompareMaskUnordLtF32: opcode = IR::ValueOpcode::FPUnordLessThan32; break;
-		case IR::Opcode::CompareUnordLeF32:
-		case IR::Opcode::CompareMaskUnordLeF32:
-			opcode = IR::ValueOpcode::FPUnordLessThanEqual32;
-			break;
-		case IR::Opcode::CompareEqF16:
-		case IR::Opcode::CompareMaskEqF16:
-			opcode = IR::ValueOpcode::FPOrdEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareNeF16:
-		case IR::Opcode::CompareMaskNeF16:
-			opcode = IR::ValueOpcode::FPOrdNotEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareGtF16:
-		case IR::Opcode::CompareMaskGtF16:
-			opcode = IR::ValueOpcode::FPOrdGreaterThan32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareGeF16:
-		case IR::Opcode::CompareMaskGeF16:
-			opcode = IR::ValueOpcode::FPOrdGreaterThanEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareLtF16:
-		case IR::Opcode::CompareMaskLtF16:
-			opcode = IR::ValueOpcode::FPOrdLessThan32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareLeF16:
-		case IR::Opcode::CompareMaskLeF16:
-			opcode = IR::ValueOpcode::FPOrdLessThanEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareUnordNeF16:
-		case IR::Opcode::CompareMaskUnordNeF16:
-			opcode = IR::ValueOpcode::FPUnordNotEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareMaskUnordGeF16:
-			opcode = IR::ValueOpcode::FPUnordGreaterThanEqual32;
-			half   = true;
-			break;
-		case IR::Opcode::CompareOrderedF32:
-		case IR::Opcode::CompareUnorderedF32: {
-			const auto lhs       = IR::F32(ReadOperand(inst.src[0], IR::Type::F32));
-			const auto rhs       = IR::F32(ReadOperand(inst.src[1], IR::Type::F32));
-			const auto unordered = ir.LogicalOr(IR::U1(ir.Emit(IR::ValueOpcode::FPIsNan32, {lhs})),
-			                                    IR::U1(ir.Emit(IR::ValueOpcode::FPIsNan32, {rhs})));
-			const auto result =
-			    inst.op == IR::Opcode::CompareUnorderedF32 ? unordered : ir.LogicalNot(unordered);
-			WriteCompareResult(inst.dst, result);
-			return true;
-		}
-		case IR::Opcode::CompareClassF32: {
-			const auto value = ReadOperand(inst.src[0], IR::Type::F32);
-			const auto mask  = ReadOperand(inst.src[1], IR::Type::U32);
-			WriteCompareResult(inst.dst,
-			                   IR::U1(ir.Emit(IR::ValueOpcode::FPCmpClass32, {value, mask})));
-			return true;
-		}
-		default: return false;
-	}
+void Translator::EmitIntegerCompare(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                                    IR::Type type, bool scalar, bool cmpx) {
+	const auto lhs = ReadOperand(inst.src0, type);
+	const auto rhs = ReadOperand(inst.src1, type);
+	EmitCompareResult(inst, IR::U1(ir.Emit(opcode, {lhs, rhs})), scalar, cmpx);
+}
+
+void Translator::EmitInteger16Compare(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                                      bool signed_value, bool cmpx) {
+	const auto lhs = ReadU16AsU32(inst.src0, signed_value);
+	const auto rhs = ReadU16AsU32(inst.src1, signed_value);
+	EmitCompareResult(inst, IR::U1(ir.Emit(opcode, {lhs, rhs})), false, cmpx);
+}
+
+void Translator::EmitFloatCompare(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                                  bool half, bool cmpx) {
 	const auto lhs =
-	    half ? IR::Value(ReadF16AsF32(inst.src[0])) : ReadOperand(inst.src[0], IR::Type::F32);
+	    half ? IR::Value(ReadF16AsF32(inst.src0)) : ReadOperand(inst.src0, IR::Type::F32);
 	const auto rhs =
-	    half ? IR::Value(ReadF16AsF32(inst.src[1])) : ReadOperand(inst.src[1], IR::Type::F32);
-	WriteCompareResult(inst.dst, IR::U1(ir.Emit(opcode, {lhs, rhs})));
-	return true;
+	    half ? IR::Value(ReadF16AsF32(inst.src1)) : ReadOperand(inst.src1, IR::Type::F32);
+	EmitCompareResult(inst, IR::U1(ir.Emit(opcode, {lhs, rhs})), false, cmpx);
 }
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+void Translator::EmitFloatOrderedCompare(const Decoder::Instruction& inst, bool ordered) {
+	const auto lhs       = IR::F32(ReadOperand(inst.src0, IR::Type::F32));
+	const auto rhs       = IR::F32(ReadOperand(inst.src1, IR::Type::F32));
+	const auto unordered = ir.LogicalOr(IR::U1(ir.Emit(IR::ValueOpcode::FPIsNan32, {lhs})),
+	                                    IR::U1(ir.Emit(IR::ValueOpcode::FPIsNan32, {rhs})));
+	EmitCompareResult(inst, ordered ? ir.LogicalNot(unordered) : unordered, false, false);
+}
+
+void Translator::V_CMP_CLASS_F32(const Decoder::Instruction& inst) {
+	const auto value = ReadOperand(inst.src0, IR::Type::F32);
+	const auto mask  = ReadOperand(inst.src1, IR::Type::U32);
+	EmitCompareResult(inst, IR::U1(ir.Emit(IR::ValueOpcode::FPCmpClass32, {value, mask})), false,
+	                  false);
+}
+
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Control.cpp

@@ -1,290 +1,370 @@
 #include "graphics/shader/recompiler/frontend/translate/Translator.h"
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
+namespace {
 
-bool Translator::TranslateStateOperation(const IR::Instruction& inst) {
-	const auto zero = IR::U32(IR::Value(0u));
-	switch (inst.op) {
-		case IR::Opcode::SaveexecB32:
-		case IR::Opcode::SaveexecB64: {
-			const auto old = ir.GetExec();
-			const auto src = ReadMask(inst.src[0]);
-			IR::U1     result;
-			switch (inst.saveexec_mode) {
-				case IR::SaveexecMode::And: result = ir.LogicalAnd(old, src); break;
-				case IR::SaveexecMode::Andn1:
-					result = ir.LogicalAnd(old, ir.LogicalNot(src));
-					break;
-				case IR::SaveexecMode::Orn2: result = ir.LogicalOr(ir.LogicalNot(old), src); break;
-			}
-			if (inst.op == IR::Opcode::SaveexecB64) {
-				WriteMask64(inst.dst, old);
-			} else {
-				WriteMask(inst.dst, old);
-			}
-			const auto mask = BallotMask(result);
-			ir.SetExec(result);
-			ir.SetExecLo(mask[0]);
-			ir.SetExecHi(mask[1]);
-			ir.SetScc(result);
-			return true;
-		}
-		case IR::Opcode::ScalarAddCarryU32:
-		case IR::Opcode::IAddCarryU32: {
-			const auto lhs      = ReadU32(inst.src[0]);
-			const auto rhs      = ReadU32(inst.src[1]);
-			const auto carry_in = ConditionBit(inst.src[2]);
-			const auto add0     = ir.Emit(IR::ValueOpcode::IAddCarry32, {lhs, rhs});
-			const auto partial  = ir.CompositeExtract(add0, 0);
-			const auto carry0   = ir.CompositeExtract(add0, 1);
-			const auto add1     = ir.Emit(IR::ValueOpcode::IAddCarry32, {partial, carry_in});
-			const auto result   = ir.CompositeExtract(add1, 0);
-			const auto carry1   = ir.CompositeExtract(add1, 1);
-			const auto carry = ir.INotEqual(ir.BitwiseOr(carry0, carry1), IR::U32(IR::Value(0u)));
-			WriteOperand(inst.dst, result);
-			if (inst.op == IR::Opcode::ScalarAddCarryU32) {
-				ir.SetScc(carry);
-			} else {
-				WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), carry));
-			}
-			return true;
-		}
-		case IR::Opcode::ScalarSubBorrowU32:
-		case IR::Opcode::ISubBorrowU32: {
-			const auto lhs    = ReadU32(inst.src[0]);
-			const auto rhs    = ReadU32(inst.src[1]);
-			const auto result = ir.ISub(lhs, rhs);
-			const auto borrow = ir.UGreaterThan(rhs, lhs);
-			WriteOperand(inst.dst, result);
-			if (inst.op == IR::Opcode::ScalarSubBorrowU32) {
-				ir.SetScc(borrow);
-			} else {
-				WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), borrow));
-			}
-			return true;
-		}
-		case IR::Opcode::ScalarSubBorrowCarryU32:
-		case IR::Opcode::ISubBorrowCarryU32: {
-			const auto lhs       = ReadU32(inst.src[0]);
-			const auto rhs       = ReadU32(inst.src[1]);
-			const auto borrow_in = ConditionBit(inst.src[2]);
-			const auto partial   = ir.ISub(lhs, rhs);
-			const auto result    = ir.ISub(partial, borrow_in);
-			const auto borrow0   = ir.UGreaterThan(rhs, lhs);
-			const auto borrow1   = ir.UGreaterThan(borrow_in, partial);
-			WriteOperand(inst.dst, result);
-			const auto borrow = ir.LogicalOr(borrow0, borrow1);
-			if (inst.op == IR::Opcode::ScalarSubBorrowCarryU32) {
-				ir.SetScc(borrow);
-			} else {
-				WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), borrow));
-			}
-			return true;
-		}
-		case IR::Opcode::ScalarSignedAddOverflowI32:
-		case IR::Opcode::ScalarSignedSubOverflowI32: {
-			const auto lhs      = ReadU32(inst.src[0]);
-			const auto rhs      = ReadU32(inst.src[1]);
-			const bool subtract = inst.op == IR::Opcode::ScalarSignedSubOverflowI32;
-			const auto result   = subtract ? ir.ISub(lhs, rhs) : ir.IAdd(lhs, rhs);
-			const auto shift    = IR::U32(IR::Value(31u));
-			const auto lhs_sign = ir.ShiftRightLogical(lhs, shift);
-			const auto rhs_sign = ir.ShiftRightLogical(rhs, shift);
-			const auto out_sign = ir.ShiftRightLogical(result, shift);
-			const auto inputs =
-			    subtract ? ir.INotEqual(lhs_sign, rhs_sign) : ir.IEqual(lhs_sign, rhs_sign);
-			const auto changed = ir.INotEqual(lhs_sign, out_sign);
-			WriteOperand(inst.dst, result);
-			ir.SetScc(ir.LogicalAnd(inputs, changed));
-			return true;
-		}
-		case IR::Opcode::ScalarShiftLeftAddCarryU32: {
-			const auto lhs           = ReadU32(inst.src[0]);
-			const auto shift         = ReadU32(inst.src[1]);
-			const auto rhs           = ReadU32(inst.src[2]);
-			const auto shifted       = ir.ShiftLeftLogical(lhs, shift);
-			const auto result        = ir.IAdd(shifted, rhs);
-			const auto add_carry     = ir.ULessThan(result, shifted);
-			const auto inverse_shift = ir.ISub(IR::U32(IR::Value(32u)), shift);
-			const auto shifted_out   = ir.ShiftRightLogical(lhs, inverse_shift);
-			const auto shift_carry   = ir.INotEqual(shifted_out, zero);
-			WriteOperand(inst.dst, result);
-			ir.SetScc(ir.LogicalOr(add_carry, shift_carry));
-			return true;
-		}
+bool IsExecOrVcc(const Decoder::Operand& operand) {
+	switch (operand.kind) {
+		case Decoder::OperandKind::ExecLo:
+		case Decoder::OperandKind::ExecHi:
+		case Decoder::OperandKind::VccLo:
+		case Decoder::OperandKind::VccHi: return true;
 		default: return false;
 	}
 }
 
-bool Translator::TranslateControlOperation(const IR::Instruction& inst) {
-	switch (inst.op) {
-		case IR::Opcode::ControlNop: ir.Emit(IR::ValueOpcode::ControlNop); return true;
-		case IR::Opcode::Waitcnt: ir.Emit(IR::ValueOpcode::Waitcnt); return true;
-		case IR::Opcode::Barrier: ir.Emit(IR::ValueOpcode::Barrier); return true;
-		case IR::Opcode::Sendmsg: ir.Emit(IR::ValueOpcode::Sendmsg); return true;
-		case IR::Opcode::TtraceData: ir.Emit(IR::ValueOpcode::TtraceData); return true;
-		case IR::Opcode::InstPrefetch: ir.Emit(IR::ValueOpcode::InstPrefetch); return true;
-		default: return false;
+Decoder::Operand ConditionOperand(Decoder::OperandKind kind) {
+	Decoder::Operand operand;
+	operand.kind = kind;
+	return operand;
+}
+
+} // namespace
+
+void Translator::S_SAVEEXEC(const Decoder::Instruction& inst, IR::ValueOpcode operation,
+                            bool negate_exec, bool negate_source, bool write_64) {
+	const auto old    = ir.GetExec();
+	const auto src    = ReadMask(inst.src0);
+	const auto lhs    = negate_exec ? ir.LogicalNot(old) : old;
+	const auto rhs    = negate_source ? ir.LogicalNot(src) : src;
+	const auto result = IR::U1(ir.Emit(operation, {lhs, rhs}));
+	if (write_64) {
+		WriteMask64(inst.dst, old);
+	} else {
+		WriteMask(inst.dst, old);
+	}
+	const auto mask = BallotMask(result);
+	ir.SetExec(result);
+	ir.SetExecLo(mask[0]);
+	ir.SetExecHi(mask[1]);
+	ir.SetScc(result);
+}
+
+void Translator::ADD_U32(const Decoder::Instruction& inst, bool vector, bool use_carry_in) {
+	const auto zero     = IR::U32(IR::Value(0u));
+	const auto lhs      = ReadU32(inst.src0);
+	const auto rhs      = ReadU32(inst.src1);
+	const auto carry_in = !use_carry_in ? zero
+	                      : vector      ? ConditionBit(inst.src2)
+	                                    : ConditionBit(ConditionOperand(Decoder::OperandKind::Scc));
+	const auto add0     = ir.Emit(IR::ValueOpcode::IAddCarry32, {lhs, rhs});
+	const auto partial  = ir.CompositeExtract(add0, 0);
+	const auto carry0   = ir.CompositeExtract(add0, 1);
+	const auto add1     = ir.Emit(IR::ValueOpcode::IAddCarry32, {partial, carry_in});
+	const auto result   = ir.CompositeExtract(add1, 0);
+	const auto carry1   = ir.CompositeExtract(add1, 1);
+	const auto carry    = ir.INotEqual(ir.BitwiseOr(carry0, carry1), zero);
+	WriteOperand(DestinationOperand(inst), result);
+	if (vector) {
+		WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), carry));
+	} else {
+		ir.SetScc(carry);
 	}
 }
 
-bool Translator::TranslateMove(const IR::Instruction& inst) {
-	switch (inst.op) {
-		case IR::Opcode::MoveU32:
-			if (inst.src[0].kind == IR::OperandKind::Register &&
-			    (inst.src[0].reg.file == IR::RegisterFile::Exec ||
-			     inst.src[0].reg.file == IR::RegisterFile::Vcc) &&
-			    inst.dst.kind == IR::OperandKind::Register &&
-			    (inst.dst.reg.file == IR::RegisterFile::Exec ||
-			     inst.dst.reg.file == IR::RegisterFile::Vcc)) {
-				WriteMask(inst.dst, ReadMask(inst.src[0]));
+void Translator::SUB_U32(const Decoder::Instruction& inst, bool vector, bool reverse) {
+	const auto lhs    = ReadU32(reverse ? inst.src1 : inst.src0);
+	const auto rhs    = ReadU32(reverse ? inst.src0 : inst.src1);
+	const auto result = ir.ISub(lhs, rhs);
+	const auto borrow = ir.UGreaterThan(rhs, lhs);
+	WriteOperand(DestinationOperand(inst), result);
+	if (vector) {
+		WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), borrow));
+	} else {
+		ir.SetScc(borrow);
+	}
+}
+
+void Translator::SUBB_U32(const Decoder::Instruction& inst, bool vector) {
+	const auto lhs       = ReadU32(vector ? inst.src1 : inst.src0);
+	const auto rhs       = ReadU32(vector ? inst.src0 : inst.src1);
+	const auto borrow_in = vector ? ConditionBit(inst.src2)
+	                              : ConditionBit(ConditionOperand(Decoder::OperandKind::Scc));
+	const auto partial   = ir.ISub(lhs, rhs);
+	const auto result    = ir.ISub(partial, borrow_in);
+	const auto borrow0   = ir.UGreaterThan(rhs, lhs);
+	const auto borrow1   = ir.UGreaterThan(borrow_in, partial);
+	const auto borrow    = ir.LogicalOr(borrow0, borrow1);
+	WriteOperand(DestinationOperand(inst), result);
+	if (vector) {
+		WriteMask(inst.dst2, ir.LogicalAnd(ir.GetExec(), borrow));
+	} else {
+		ir.SetScc(borrow);
+	}
+}
+
+void Translator::S_ADD_SUB_I32(const Decoder::Instruction& inst, bool subtract) {
+	const auto lhs      = ReadU32(inst.src0);
+	const auto rhs      = ReadU32(inst.src1);
+	const auto result   = subtract ? ir.ISub(lhs, rhs) : ir.IAdd(lhs, rhs);
+	const auto shift    = IR::U32(IR::Value(31u));
+	const auto lhs_sign = ir.ShiftRightLogical(lhs, shift);
+	const auto rhs_sign = ir.ShiftRightLogical(rhs, shift);
+	const auto out_sign = ir.ShiftRightLogical(result, shift);
+	const auto inputs = subtract ? ir.INotEqual(lhs_sign, rhs_sign) : ir.IEqual(lhs_sign, rhs_sign);
+	const auto changed = ir.INotEqual(lhs_sign, out_sign);
+	WriteOperand(DestinationOperand(inst), result);
+	ir.SetScc(ir.LogicalAnd(inputs, changed));
+}
+
+void Translator::S_LSHL_ADD_U32(const Decoder::Instruction& inst, uint32_t shift_amount) {
+	const auto lhs           = ReadU32(inst.src0);
+	const auto shift         = IR::U32(IR::Value(shift_amount));
+	const auto rhs           = ReadU32(inst.src1);
+	const auto shifted       = ir.ShiftLeftLogical(lhs, shift);
+	const auto result        = ir.IAdd(shifted, rhs);
+	const auto add_carry     = ir.ULessThan(result, shifted);
+	const auto inverse_shift = ir.ISub(IR::U32(IR::Value(32u)), shift);
+	const auto shifted_out   = ir.ShiftRightLogical(lhs, inverse_shift);
+	const auto shift_carry   = ir.INotEqual(shifted_out, IR::U32(IR::Value(0u)));
+	WriteOperand(DestinationOperand(inst), result);
+	ir.SetScc(ir.LogicalOr(add_carry, shift_carry));
+}
+
+void Translator::ScalarMinMax32(const Decoder::Instruction& inst, IR::ValueOpcode value_opcode,
+                                IR::ValueOpcode compare_opcode) {
+	const auto lhs = ReadU32(inst.src0);
+	const auto rhs = ReadU32(inst.src1);
+	WriteOperand(DestinationOperand(inst), ir.Emit(value_opcode, {lhs, rhs}));
+	ir.SetScc(IR::U1(ir.Emit(compare_opcode, {lhs, rhs})));
+}
+
+void Translator::EmitControlNop() {
+	ir.Emit(IR::ValueOpcode::ControlNop);
+}
+
+void Translator::EmitWaitcnt() {
+	ir.Emit(IR::ValueOpcode::Waitcnt);
+}
+
+void Translator::S_BARRIER() {
+	ir.Emit(IR::ValueOpcode::Barrier);
+}
+
+void Translator::S_SENDMSG() {
+	ir.Emit(IR::ValueOpcode::Sendmsg);
+}
+
+void Translator::S_TTRACEDATA() {
+	ir.Emit(IR::ValueOpcode::TtraceData);
+}
+
+void Translator::S_INST_PREFETCH() {
+	ir.Emit(IR::ValueOpcode::InstPrefetch);
+}
+
+bool Translator::S_GETPC_B64(const Decoder::Instruction& inst, std::string* error) {
+	const auto base    = IR::U64(ir.Emit(IR::ValueOpcode::GetShaderBase));
+	const auto address = IR::U64(
+	    ir.Emit(IR::ValueOpcode::IAdd64, {base, IR::Value(static_cast<uint64_t>(inst.pc) + 4u)}));
+	if (inst.dst.kind == Decoder::OperandKind::Null) {
+		return true;
+	}
+	auto high = PlainOperand(inst.dst);
+	switch (inst.dst.kind) {
+		case Decoder::OperandKind::Sgpr:
+			if (inst.dst.reg < 105u) {
+				high.reg++;
+			} else if (inst.dst.reg == 105u) {
+				high.kind = Decoder::OperandKind::VccLo;
+				high.reg  = 0u;
 			} else {
-				WriteOperand(inst.dst, ReadOperand(inst.src[0], IR::Type::U32));
-			}
-			return true;
-		case IR::Opcode::MoveF32Bits:
-			WriteOperand(inst.dst, ReadOperand(inst.src[0], IR::Type::F32));
-			return true;
-		case IR::Opcode::MoveU64: {
-			const auto is_exec_or_vcc = [](const IR::Operand& operand) {
-				return operand.kind == IR::OperandKind::Register &&
-				       (operand.reg.file == IR::RegisterFile::Exec ||
-				        operand.reg.file == IR::RegisterFile::Vcc);
-			};
-			if (is_exec_or_vcc(inst.dst) || is_exec_or_vcc(inst.src[0])) {
-				WriteMask64(inst.dst, ReadMask(inst.src[0]));
-				return true;
-			}
-			const bool scalar_copy = inst.dst.kind == IR::OperandKind::Register &&
-			                         inst.src[0].kind == IR::OperandKind::Register &&
-			                         inst.dst.reg.file == IR::RegisterFile::Scalar &&
-			                         inst.src[0].reg.file == IR::RegisterFile::Scalar;
-			IR::U1     source_mask;
-			IR::U1     source_mask_valid;
-			if (scalar_copy) {
-				const auto src    = static_cast<IR::ScalarReg>(inst.src[0].reg.index);
-				source_mask       = ir.GetThreadBitScalarReg(src);
-				source_mask_valid = ir.GetScalarMaskTag(src);
-			}
-			WriteU32Pair(inst.dst, ReadU32Pair(inst.src[0]));
-			if (scalar_copy) {
-				ir.SetThreadBitScalarReg(static_cast<IR::ScalarReg>(inst.dst.reg.index),
-				                         source_mask);
-				ir.SetScalarMaskTag(static_cast<IR::ScalarReg>(inst.dst.reg.index),
-				                    source_mask_valid);
-			}
-			return true;
-		}
-		case IR::Opcode::WqmB64: {
-			const auto is_exec_or_vcc = [](const IR::Operand& operand) {
-				return operand.kind == IR::OperandKind::Register &&
-				       (operand.reg.file == IR::RegisterFile::Exec ||
-				        operand.reg.file == IR::RegisterFile::Vcc);
-			};
-			if (!is_exec_or_vcc(inst.dst) && !is_exec_or_vcc(inst.src[0])) {
-				const auto mask_valid = ReadMaskValid(inst.src[0]);
-				const auto invocation_result =
-				    IR::U1(ir.Emit(IR::ValueOpcode::WqmMask, {ReadMask(inst.src[0])}));
-				const auto result = IR::U64(
-				    ir.Emit(IR::ValueOpcode::WqmU64, {ReadOperand(inst.src[0], IR::Type::U64)}));
-				WriteOperand(inst.dst, result);
-				if (inst.dst.kind == IR::OperandKind::Register &&
-				    inst.dst.reg.file == IR::RegisterFile::Scalar) {
-					const auto dst = static_cast<IR::ScalarReg>(inst.dst.reg.index);
-					ir.SetThreadBitScalarReg(dst, invocation_result);
-					ir.SetScalarMaskTag(dst, mask_valid);
-					const auto raw_nonzero = IR::U1(
-					    ir.Emit(IR::ValueOpcode::INotEqual64, {result, IR::Value(uint64_t {0})}));
-					ir.SetScc(IR::U1(ir.Emit(
-					    IR::ValueOpcode::SelectU1, {mask_valid, invocation_result, raw_nonzero})));
-				} else {
-					ir.SetScc(IR::U1(
-					    ir.Emit(IR::ValueOpcode::INotEqual64, {result, IR::Value(uint64_t {0})})));
+				if (error != nullptr) {
+					*error = "S_GETPC_B64 destination does not name a valid scalar pair";
 				}
-				return true;
+				return false;
 			}
-			const auto result = IR::U1(ir.Emit(IR::ValueOpcode::WqmMask, {ReadMask(inst.src[0])}));
-			WriteMask64(inst.dst, result);
-			ir.SetScc(result);
-			return true;
-		}
-		default: return false;
+			break;
+		case Decoder::OperandKind::VccLo: high.kind = Decoder::OperandKind::VccHi; break;
+		case Decoder::OperandKind::M0: high.kind = Decoder::OperandKind::Null; break;
+		case Decoder::OperandKind::ExecLo: high.kind = Decoder::OperandKind::ExecHi; break;
+		default:
+			if (error != nullptr) {
+				*error = "S_GETPC_B64 destination does not name a valid scalar pair";
+			}
+			return false;
+	}
+	const auto words = ExtractU64(address);
+	WriteOperand(inst.dst, words[0]);
+	WriteOperand(high, words[1]);
+	return true;
+}
+
+void Translator::S_CSELECT_B32(const Decoder::Instruction& inst) {
+	const auto result = ir.Select(ir.GetScc(), ReadU32(inst.src0), ReadU32(inst.src1));
+	WriteOperand(DestinationOperand(inst), result);
+}
+
+void Translator::S_CSELECT_B64(const Decoder::Instruction& inst) {
+	const auto condition     = ir.GetScc();
+	const auto lhs           = ReadU32Pair(inst.src0);
+	const auto rhs           = ReadU32Pair(inst.src1);
+	const auto selected_mask = IR::U1(
+	    ir.Emit(IR::ValueOpcode::SelectU1, {condition, ReadMask(inst.src0), ReadMask(inst.src1)}));
+	const auto selected_mask_valid =
+	    IR::U1(ir.Emit(IR::ValueOpcode::SelectU1,
+	                   {condition, ReadMaskValid(inst.src0), ReadMaskValid(inst.src1)}));
+	if (IsExecOrVcc(inst.dst)) {
+		WriteMask64(inst.dst, selected_mask);
+		return;
+	}
+	WriteU32Pair(inst.dst,
+	             {ir.Select(condition, lhs[0], rhs[0]), ir.Select(condition, lhs[1], rhs[1])});
+	if (inst.dst.kind == Decoder::OperandKind::Sgpr) {
+		const auto dst = static_cast<IR::ScalarReg>(inst.dst.reg);
+		ir.SetThreadBitScalarReg(dst, selected_mask);
+		ir.SetScalarMaskTag(dst, selected_mask_valid);
 	}
 }
 
-bool Translator::TranslateLaneOperation(const IR::Instruction& inst) {
+void Translator::MOV_B32(const Decoder::Instruction& inst, bool apply_float_modifiers) {
+	if (IsExecOrVcc(inst.src0) && IsExecOrVcc(inst.dst)) {
+		WriteMask(inst.dst, ReadMask(inst.src0));
+	} else if (apply_float_modifiers && (inst.src0.negate || inst.src0.absolute)) {
+		WriteOperand(DestinationOperand(inst), ReadOperand(inst.src0, IR::Type::F32));
+	} else {
+		WriteOperand(DestinationOperand(inst), ReadOperand(inst.src0, IR::Type::U32));
+	}
+}
+
+void Translator::S_MOV_B64(const Decoder::Instruction& inst) {
+	if (IsExecOrVcc(inst.dst) || IsExecOrVcc(inst.src0)) {
+		WriteMask64(inst.dst, ReadMask(inst.src0));
+		return;
+	}
+	const bool scalar_copy =
+	    inst.dst.kind == Decoder::OperandKind::Sgpr && inst.src0.kind == Decoder::OperandKind::Sgpr;
+	IR::U1 source_mask;
+	IR::U1 source_mask_valid;
+	if (scalar_copy) {
+		source_mask       = ir.GetThreadBitScalarReg(static_cast<IR::ScalarReg>(inst.src0.reg));
+		source_mask_valid = ir.GetScalarMaskTag(static_cast<IR::ScalarReg>(inst.src0.reg));
+	}
+	WriteU32Pair(inst.dst, ReadU32Pair(inst.src0));
+	if (scalar_copy) {
+		ir.SetThreadBitScalarReg(static_cast<IR::ScalarReg>(inst.dst.reg), source_mask);
+		ir.SetScalarMaskTag(static_cast<IR::ScalarReg>(inst.dst.reg), source_mask_valid);
+	}
+}
+
+void Translator::S_WQM_B64(const Decoder::Instruction& inst) {
+	if (!IsExecOrVcc(inst.dst) && !IsExecOrVcc(inst.src0)) {
+		const auto mask_valid = ReadMaskValid(inst.src0);
+		const auto invocation_result =
+		    IR::U1(ir.Emit(IR::ValueOpcode::WqmMask, {ReadMask(inst.src0)}));
+		const auto result =
+		    IR::U64(ir.Emit(IR::ValueOpcode::WqmU64, {ReadOperand(inst.src0, IR::Type::U64)}));
+		WriteOperand(DestinationOperand(inst), result);
+		if (inst.dst.kind == Decoder::OperandKind::Sgpr) {
+			const auto dst = static_cast<IR::ScalarReg>(inst.dst.reg);
+			ir.SetThreadBitScalarReg(dst, invocation_result);
+			ir.SetScalarMaskTag(dst, mask_valid);
+			const auto raw_nonzero =
+			    IR::U1(ir.Emit(IR::ValueOpcode::INotEqual64, {result, IR::Value(uint64_t {0})}));
+			ir.SetScc(IR::U1(
+			    ir.Emit(IR::ValueOpcode::SelectU1, {mask_valid, invocation_result, raw_nonzero})));
+		} else {
+			ir.SetScc(
+			    IR::U1(ir.Emit(IR::ValueOpcode::INotEqual64, {result, IR::Value(uint64_t {0})})));
+		}
+		return;
+	}
+	const auto result = IR::U1(ir.Emit(IR::ValueOpcode::WqmMask, {ReadMask(inst.src0)}));
+	WriteMask64(inst.dst, result);
+	ir.SetScc(result);
+}
+
+bool Translator::V_MOVRELS_B32(const Decoder::Instruction& inst, std::string* error) {
+	if (inst.dst.kind != Decoder::OperandKind::Vgpr ||
+	    inst.src0.kind != Decoder::OperandKind::Vgpr) {
+		if (error != nullptr) {
+			*error = "V_MOVRELS_B32 requires VGPR source and destination";
+		}
+		return false;
+	}
+	if (inst.dst.sdwa_sel != 6u || inst.dst.omod != 0u || inst.dst.clamp ||
+	    inst.src0.sdwa_sel != 6u || inst.src0.sdwa_sext || inst.src0.negate || inst.src0.absolute ||
+	    inst.src0.dpp) {
+		if (error != nullptr) {
+			*error = "V_MOVRELS_B32 modifiers are not implemented";
+		}
+		return false;
+	}
+	const auto base     = inst.src0.reg;
+	const auto m0       = ir.BitwiseAnd(ReadU32(ConditionOperand(Decoder::OperandKind::M0)),
+	                                    IR::U32(IR::Value(0xffu)));
+	auto       selected = ir.GetVectorReg(static_cast<IR::VectorReg>(base));
+	for (uint32_t index = base + 1u; index < current_vector_limit; index++) {
+		const auto match = ir.IEqual(m0, IR::U32(IR::Value(index - base)));
+		selected = ir.Select(match, ir.GetVectorReg(static_cast<IR::VectorReg>(index)), selected);
+	}
+	WriteOperand(DestinationOperand(inst), selected);
+	return true;
+}
+
+bool Translator::V_MOVRELD_B32(const Decoder::Instruction& inst, std::string* error) {
+	if (inst.dst.kind != Decoder::OperandKind::Vgpr) {
+		if (error != nullptr) {
+			*error = "V_MOVRELD_B32 requires VGPR destination";
+		}
+		return false;
+	}
+	if (inst.dst.sdwa_sel != 6u || inst.dst.omod != 0u || inst.dst.clamp ||
+	    inst.src0.sdwa_sel != 6u || inst.src0.sdwa_sext || inst.src0.negate || inst.src0.absolute ||
+	    inst.src0.dpp) {
+		if (error != nullptr) {
+			*error = "V_MOVRELD_B32 modifiers are not implemented";
+		}
+		return false;
+	}
+	const auto base  = inst.dst.reg;
+	const auto value = ReadU32(inst.src0);
+	const auto m0    = ir.BitwiseAnd(ReadU32(ConditionOperand(Decoder::OperandKind::M0)),
+	                                 IR::U32(IR::Value(0xffu)));
+	for (uint32_t index = base; index < current_vector_limit; index++) {
+		const auto reg   = static_cast<IR::VectorReg>(index);
+		const auto match = ir.IEqual(m0, IR::U32(IR::Value(index - base)));
+		const auto write = ir.LogicalAnd(ir.GetExec(), match);
+		ir.SetVectorReg(reg, ir.Select(write, value, ir.GetVectorReg(reg)));
+	}
+	return true;
+}
+
+void Translator::V_READFIRSTLANE_B32(const Decoder::Instruction& inst) {
+	const auto result = ir.Emit(IR::ValueOpcode::ReadFirstLane, {ReadU32(inst.src0), ir.GetExec()});
+	WriteOperand(DestinationOperand(inst), result);
+}
+
+void Translator::V_READLANE_B32(const Decoder::Instruction& inst) {
 	const auto lane_mask = IR::U32(IR::Value(current_wave_size == 32u ? 31u : 63u));
-	switch (inst.op) {
-		case IR::Opcode::MoveRelSourceU32: {
-			EXIT_IF(inst.src[0].kind != IR::OperandKind::Register ||
-			        inst.src[0].reg.file != IR::RegisterFile::Vector);
-			const auto base     = inst.src[0].reg.index;
-			const auto m0       = ir.BitwiseAnd(ReadU32(inst.src[1]), IR::U32(IR::Value(0xffu)));
-			auto       selected = ir.GetVectorReg(static_cast<IR::VectorReg>(base));
-			for (uint32_t index = base + 1u; index < current_vector_limit; index++) {
-				const auto match = ir.IEqual(m0, IR::U32(IR::Value(index - base)));
-				selected =
-				    ir.Select(match, ir.GetVectorReg(static_cast<IR::VectorReg>(index)), selected);
-			}
-			WriteOperand(inst.dst, selected);
-			return true;
-		}
-		case IR::Opcode::MoveRelDestU32: {
-			EXIT_IF(inst.dst.kind != IR::OperandKind::Register ||
-			        inst.dst.reg.file != IR::RegisterFile::Vector);
-			const auto base  = inst.dst.reg.index;
-			const auto value = ReadU32(inst.src[0]);
-			const auto m0    = ir.BitwiseAnd(ReadU32(inst.src[1]), IR::U32(IR::Value(0xffu)));
-			for (uint32_t index = base; index < current_vector_limit; index++) {
-				const auto reg   = static_cast<IR::VectorReg>(index);
-				const auto match = ir.IEqual(m0, IR::U32(IR::Value(index - base)));
-				const auto write = ir.LogicalAnd(ir.GetExec(), match);
-				ir.SetVectorReg(reg, ir.Select(write, value, ir.GetVectorReg(reg)));
-			}
-			return true;
-		}
-		case IR::Opcode::ReadFirstLaneU32: {
-			const auto result =
-			    ir.Emit(IR::ValueOpcode::ReadFirstLane, {ReadU32(inst.src[0]), ir.GetExec()});
-			WriteOperand(inst.dst, result);
-			return true;
-		}
-		case IR::Opcode::ReadLaneU32: {
-			const auto lane = ir.BitwiseAnd(ReadU32(inst.src[1]), lane_mask);
-			WriteOperand(inst.dst,
-			             ir.Emit(IR::ValueOpcode::ReadLane, {ReadU32(inst.src[0]), lane}));
-			return true;
-		}
-		case IR::Opcode::WriteLaneU32: {
-			EXIT_IF(inst.dst.kind != IR::OperandKind::Register ||
-			        inst.dst.reg.file != IR::RegisterFile::Vector);
-			const auto reg    = static_cast<IR::VectorReg>(inst.dst.reg.index);
-			const auto lane   = ir.BitwiseAnd(ReadU32(inst.src[1]), lane_mask);
-			const auto result = IR::U32(ir.Emit(
-			    IR::ValueOpcode::WriteLane, {ir.GetVectorReg(reg), ReadU32(inst.src[0]), lane}));
-			// V_WRITELANE_B32 ignores EXEC; bypass the ordinary VGPR destination path.
-			ir.SetVectorReg(reg, result);
-			return true;
-		}
-		case IR::Opcode::Permlane16B32:
-		case IR::Opcode::Permlanex16B32: {
-			const IR::PermlaneFlags flags {
-			    .x16            = inst.op == IR::Opcode::Permlanex16B32,
-			    .fetch_inactive = inst.dst.op_sel,
-			    .bound_control  = inst.dst.op_sel_hi,
-			};
-			const auto result = ir.Emit(
-			    IR::ValueOpcode::Permlane16U32,
-			    {ReadU32(inst.src[0]), ReadU32(inst.src[1]), ReadU32(inst.src[2]), ir.GetExec()},
-			    flags);
-			auto dst      = inst.dst;
-			dst.op_sel    = false;
-			dst.op_sel_hi = false;
-			WriteOperand(dst, result);
-			return true;
-		}
-		default: return false;
-	}
+	const auto lane      = ir.BitwiseAnd(ReadU32(inst.src1), lane_mask);
+	WriteOperand(DestinationOperand(inst),
+	             ir.Emit(IR::ValueOpcode::ReadLane, {ReadU32(inst.src0), lane}));
 }
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+void Translator::V_WRITELANE_B32(const Decoder::Instruction& inst) {
+	EXIT_IF(inst.dst.kind != Decoder::OperandKind::Vgpr);
+	const auto lane_mask = IR::U32(IR::Value(current_wave_size == 32u ? 31u : 63u));
+	const auto reg       = static_cast<IR::VectorReg>(inst.dst.reg);
+	const auto lane      = ir.BitwiseAnd(ReadU32(inst.src1), lane_mask);
+	const auto result    = IR::U32(
+	    ir.Emit(IR::ValueOpcode::WriteLane, {ir.GetVectorReg(reg), ReadU32(inst.src0), lane}));
+	ir.SetVectorReg(reg, result);
+}
+
+void Translator::V_PERMLANE16_B32(const Decoder::Instruction& inst, bool x16) {
+	const IR::PermlaneFlags flags {
+	    .x16            = x16,
+	    .fetch_inactive = inst.dst.op_sel,
+	    .bound_control  = inst.dst.op_sel_hi,
+	};
+	const auto result =
+	    ir.Emit(IR::ValueOpcode::Permlane16U32,
+	            {ReadU32(inst.src0), ReadU32(inst.src1), ReadU32(inst.src2), ir.GetExec()}, flags);
+	auto dst      = DestinationOperand(inst);
+	dst.op_sel    = false;
+	dst.op_sel_hi = false;
+	WriteOperand(dst, result);
+}
+
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Convert.cpp

@@ -1,6 +1,6 @@
 #include "graphics/shader/recompiler/frontend/translate/Translator.h"
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
 IR::F32 Translator::SelectF32(IR::U1 condition, IR::F32 true_value, IR::F32 false_value) {
 	return IR::F32(ir.Emit(IR::ValueOpcode::SelectF32, {condition, true_value, false_value}));
@@ -38,132 +38,136 @@ IR::U32 Translator::ConvertF32ToI32Saturated(IR::F32 value, float lower_bound, f
 	return ir.Select(nan, IR::U32(IR::Value(0u)), clamped);
 }
 
-bool Translator::TranslateConversion(const IR::Instruction& inst) {
-	const auto f32 = [&](uint32_t index) {
-		return IR::F32(ReadOperand(inst.src[index], IR::Type::F32));
-	};
-	const auto u32         = [&](uint32_t index) { return ReadU32(inst.src[index]); };
-	const auto convert_u32 = [&](IR::F32 value) {
-		return ConvertF32ToU32Saturated(value, 4294967296.0f, 4294967040.0f, 0xffffffffu);
-	};
-	const auto convert_i32 = [&](IR::F32 value) {
-		return ConvertF32ToI32Saturated(value, -2147483648.0f, 2147483648.0f, 2147483520.0f,
-		                                0x80000000u, 0x7fffffffu);
-	};
+void Translator::V_CVT_F32_UBYTE(const Decoder::Instruction& inst, uint32_t byte_index) {
+	const auto source = ReadU32(SourceAt(inst, 0));
+	const auto byte =
+	    ir.BitwiseAnd(ir.ShiftRightLogical(source, IR::U32(IR::Value(byte_index * 8u))),
+	                  IR::U32(IR::Value(0xffu)));
+	WriteOperand(DestinationOperand(inst), ir.Emit(IR::ValueOpcode::ConvertF32U32, {byte}));
+}
 
-	switch (inst.op) {
-		case IR::Opcode::ConvertByteU32ToF32: {
-			const auto index =
-			    inst.src[1].kind == IR::OperandKind::ImmediateU32 ? inst.src[1].imm & 3u : 0u;
-			const auto byte =
-			    ir.BitwiseAnd(ir.ShiftRightLogical(u32(0), IR::U32(IR::Value(index * 8u))),
-			                  IR::U32(IR::Value(0xffu)));
-			WriteOperand(inst.dst, ir.Emit(IR::ValueOpcode::ConvertF32U32, {byte}));
-			return true;
-		}
-		case IR::Opcode::ConvertU32ToF32:
-			WriteOperand(inst.dst, ir.Emit(IR::ValueOpcode::ConvertF32U32, {u32(0)}));
-			return true;
-		case IR::Opcode::ConvertI32ToF32:
-			WriteOperand(inst.dst, ir.Emit(IR::ValueOpcode::ConvertF32S32, {u32(0)}));
-			return true;
-		case IR::Opcode::ConvertF32ToU32: WriteOperand(inst.dst, convert_u32(f32(0))); return true;
-		case IR::Opcode::ConvertF32ToI32: WriteOperand(inst.dst, convert_i32(f32(0))); return true;
-		case IR::Opcode::ConvertF32ToF16: WriteF16(inst.dst, f32(0)); return true;
-		case IR::Opcode::ConvertF16ToF32:
-			WriteOperand(inst.dst, ReadF16AsF32(inst.src[0]));
-			return true;
-		case IR::Opcode::ConvertU16ToF16: {
-			const auto value = IR::F32(
-			    ir.Emit(IR::ValueOpcode::ConvertF32U32, {ReadU16AsU32(inst.src[0], false)}));
-			WriteF16(inst.dst, value);
-			return true;
-		}
-		case IR::Opcode::ConvertI16ToF16: {
-			const auto value =
-			    IR::F32(ir.Emit(IR::ValueOpcode::ConvertF32S32, {ReadU16AsU32(inst.src[0], true)}));
-			WriteF16(inst.dst, value);
-			return true;
-		}
-		case IR::Opcode::ConvertF16ToU16: {
-			const auto converted =
-			    ConvertF32ToU32Saturated(ReadF16AsF32(inst.src[0]), 65536.0f, 65535.0f, 0xffffu);
-			WriteU16(inst.dst, converted);
-			return true;
-		}
-		case IR::Opcode::ConvertF16ToI16: {
-			const auto converted = ConvertF32ToI32Saturated(
-			    ReadF16AsF32(inst.src[0]), -32768.0f, 32768.0f, 32767.0f, 0xffff8000u, 0x7fffu);
-			WriteU16(inst.dst, ir.BitwiseAnd(converted, IR::U32(IR::Value(0xffffu))));
-			return true;
-		}
-		case IR::Opcode::ConvertRoundPlusInfF32ToI32: {
-			const auto biased =
-			    IR::F32(ir.Emit(IR::ValueOpcode::FPAdd32, {f32(0), IR::Value::F32(0.5f)}));
-			const auto rounded = IR::F32(ir.Emit(IR::ValueOpcode::FPFloor32, {biased}));
-			WriteOperand(inst.dst, convert_i32(rounded));
-			return true;
-		}
-		case IR::Opcode::ConvertFloorF32ToI32: {
-			const auto rounded = IR::F32(ir.Emit(IR::ValueOpcode::FPFloor32, {f32(0)}));
-			WriteOperand(inst.dst, convert_i32(rounded));
-			return true;
-		}
-		case IR::Opcode::FrexpExpI32F32: {
-			const auto bits     = ir.BitCastU32(f32(0));
-			const auto exponent = IR::U32(
-			    ir.Emit(IR::ValueOpcode::BitFieldUExtract, {bits, IR::Value(23u), IR::Value(8u)}));
-			const auto mantissa  = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x007fffffu)));
-			const auto normal    = ir.ISub(exponent, IR::U32(IR::Value(126u)));
-			const auto msb       = IR::U32(ir.Emit(IR::ValueOpcode::FindUMsb32, {mantissa}));
-			const auto subnormal = ir.ISub(msb, IR::U32(IR::Value(148u)));
-			const auto denormal  = ir.Select(ir.INotEqual(mantissa, IR::U32(IR::Value(0u))),
-			                                 subnormal, IR::U32(IR::Value(0u)));
-			const auto finite    = ir.Select(
-			    ir.INotEqual(exponent, IR::U32(IR::Value(0xffu))),
-			    ir.Select(ir.INotEqual(exponent, IR::U32(IR::Value(0u))), normal, denormal),
-			    IR::U32(IR::Value(0u)));
-			WriteOperand(inst.dst, finite);
-			return true;
-		}
-		case IR::Opcode::ConvertI4ToOffsetF32: {
-			const auto nibble = IR::U32(
-			    ir.Emit(IR::ValueOpcode::BitFieldSExtract, {u32(0), IR::Value(0u), IR::Value(4u)}));
-			const auto value = IR::F32(ir.Emit(IR::ValueOpcode::ConvertF32S32, {nibble}));
-			WriteOperand(inst.dst,
-			             ir.Emit(IR::ValueOpcode::FPMul32, {value, IR::Value::F32(1.0f / 16.0f)}));
-			return true;
-		}
-		case IR::Opcode::PackF32ToF16Rtz:
-			WriteOperand(inst.dst, ir.Emit(IR::ValueOpcode::PackFloat2x16Rtz, {f32(0), f32(1)}));
-			return true;
-		case IR::Opcode::PackSnorm2x16F32:
-		case IR::Opcode::PackUnorm2x16F32: {
-			const auto pair   = ir.Emit(IR::ValueOpcode::CompositeConstructF32x2, {f32(0), f32(1)});
-			const auto opcode = inst.op == IR::Opcode::PackSnorm2x16F32
-			                        ? IR::ValueOpcode::PackSnorm2x16
-			                        : IR::ValueOpcode::PackUnorm2x16;
-			WriteOperand(inst.dst, ir.Emit(opcode, {pair}));
-			return true;
-		}
-		case IR::Opcode::PackU8F32: {
-			const auto byte  = ConvertF32ToU32Saturated(f32(0), 255.0f, 255.0f, 255u);
-			const auto index = ir.BitwiseAnd(u32(1), IR::U32(IR::Value(3u)));
-			const auto shift = ir.ShiftLeftLogical(index, IR::U32(IR::Value(3u)));
-			const auto mask  = ir.ShiftLeftLogical(IR::U32(IR::Value(0xffu)), shift);
-			const auto base  = ir.BitwiseAnd(u32(2), ir.BitwiseNot(mask));
-			WriteOperand(inst.dst, ir.BitwiseOr(base, ir.ShiftLeftLogical(byte, shift)));
-			return true;
-		}
-		case IR::Opcode::PackB32F16: {
-			const auto low = ReadF16LaneBits(inst.src[0], false);
-			const auto high =
-			    ir.ShiftLeftLogical(ReadF16LaneBits(inst.src[1], false), IR::U32(IR::Value(16u)));
-			WriteOperand(inst.dst, ir.BitwiseOr(low, high));
-			return true;
-		}
-		default: return false;
+void Translator::V_CVT_F32_U32(const Decoder::Instruction& inst) {
+	WriteOperand(DestinationOperand(inst),
+	             ir.Emit(IR::ValueOpcode::ConvertF32U32, {ReadU32(SourceAt(inst, 0))}));
+}
+
+void Translator::V_CVT_F32_I32(const Decoder::Instruction& inst) {
+	WriteOperand(DestinationOperand(inst),
+	             ir.Emit(IR::ValueOpcode::ConvertF32S32, {ReadU32(SourceAt(inst, 0))}));
+}
+
+void Translator::V_CVT_U32_F32(const Decoder::Instruction& inst) {
+	const auto value = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	WriteOperand(DestinationOperand(inst),
+	             ConvertF32ToU32Saturated(value, 4294967296.0f, 4294967040.0f, 0xffffffffu));
+}
+
+void Translator::V_CVT_I32_F32(const Decoder::Instruction& inst) {
+	const auto value = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	WriteOperand(DestinationOperand(inst),
+	             ConvertF32ToI32Saturated(value, -2147483648.0f, 2147483648.0f, 2147483520.0f,
+	                                      0x80000000u, 0x7fffffffu));
+}
+
+void Translator::V_CVT_F16_F32(const Decoder::Instruction& inst) {
+	WriteF16(DestinationOperand(inst), IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32)));
+}
+
+void Translator::V_CVT_F32_F16(const Decoder::Instruction& inst) {
+	WriteOperand(DestinationOperand(inst), ReadF16AsF32(inst.src0));
+}
+
+void Translator::V_CVT_F16_16(const Decoder::Instruction& inst, bool signed_value) {
+	const auto value = IR::F32(
+	    ir.Emit(signed_value ? IR::ValueOpcode::ConvertF32S32 : IR::ValueOpcode::ConvertF32U32,
+	            {ReadU16AsU32(inst.src0, signed_value)}));
+	WriteF16(DestinationOperand(inst), value);
+}
+
+void Translator::V_CVT_16_F16(const Decoder::Instruction& inst, bool signed_value) {
+	const auto value = ReadF16AsF32(inst.src0);
+	if (signed_value) {
+		const auto converted =
+		    ConvertF32ToI32Saturated(value, -32768.0f, 32768.0f, 32767.0f, 0xffff8000u, 0x7fffu);
+		WriteU16(DestinationOperand(inst), ir.BitwiseAnd(converted, IR::U32(IR::Value(0xffffu))));
+		return;
 	}
+	WriteU16(DestinationOperand(inst),
+	         ConvertF32ToU32Saturated(value, 65536.0f, 65535.0f, 0xffffu));
+}
+
+void Translator::V_CVT_RPI_I32_F32(const Decoder::Instruction& inst) {
+	const auto source  = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	const auto biased  = IR::F32(ir.Emit(IR::ValueOpcode::FPAdd32, {source, IR::Value::F32(0.5f)}));
+	const auto rounded = IR::F32(ir.Emit(IR::ValueOpcode::FPFloor32, {biased}));
+	WriteOperand(DestinationOperand(inst),
+	             ConvertF32ToI32Saturated(rounded, -2147483648.0f, 2147483648.0f, 2147483520.0f,
+	                                      0x80000000u, 0x7fffffffu));
+}
+
+void Translator::V_CVT_FLR_I32_F32(const Decoder::Instruction& inst) {
+	const auto source  = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	const auto rounded = IR::F32(ir.Emit(IR::ValueOpcode::FPFloor32, {source}));
+	WriteOperand(DestinationOperand(inst),
+	             ConvertF32ToI32Saturated(rounded, -2147483648.0f, 2147483648.0f, 2147483520.0f,
+	                                      0x80000000u, 0x7fffffffu));
+}
+
+void Translator::V_FREXP_EXP_I32_F32(const Decoder::Instruction& inst) {
+	const auto source = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	const auto bits   = ir.BitCastU32(source);
+	const auto exponent =
+	    IR::U32(ir.Emit(IR::ValueOpcode::BitFieldUExtract, {bits, IR::Value(23u), IR::Value(8u)}));
+	const auto mantissa  = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x007fffffu)));
+	const auto normal    = ir.ISub(exponent, IR::U32(IR::Value(126u)));
+	const auto msb       = IR::U32(ir.Emit(IR::ValueOpcode::FindUMsb32, {mantissa}));
+	const auto subnormal = ir.ISub(msb, IR::U32(IR::Value(148u)));
+	const auto denormal  = ir.Select(ir.INotEqual(mantissa, IR::U32(IR::Value(0u))), subnormal,
+	                                 IR::U32(IR::Value(0u)));
+	const auto finite =
+	    ir.Select(ir.INotEqual(exponent, IR::U32(IR::Value(0xffu))),
+	              ir.Select(ir.INotEqual(exponent, IR::U32(IR::Value(0u))), normal, denormal),
+	              IR::U32(IR::Value(0u)));
+	WriteOperand(DestinationOperand(inst), finite);
+}
+
+void Translator::V_CVT_OFF_F32_I4(const Decoder::Instruction& inst) {
+	const auto nibble =
+	    IR::U32(ir.Emit(IR::ValueOpcode::BitFieldSExtract,
+	                    {ReadU32(SourceAt(inst, 0)), IR::Value(0u), IR::Value(4u)}));
+	const auto value = IR::F32(ir.Emit(IR::ValueOpcode::ConvertF32S32, {nibble}));
+	WriteOperand(DestinationOperand(inst),
+	             ir.Emit(IR::ValueOpcode::FPMul32, {value, IR::Value::F32(1.0f / 16.0f)}));
+}
+
+void Translator::V_CVT_PKRTZ_F16_F32(const Decoder::Instruction& inst) {
+	const auto lhs = ReadOperand(SourceAt(inst, 0), IR::Type::F32);
+	const auto rhs = ReadOperand(SourceAt(inst, 1), IR::Type::F32);
+	WriteOperand(DestinationOperand(inst), ir.Emit(IR::ValueOpcode::PackFloat2x16Rtz, {lhs, rhs}));
+}
+
+void Translator::V_CVT_PKNORM_F32(const Decoder::Instruction& inst, IR::ValueOpcode opcode) {
+	const auto lhs  = ReadOperand(SourceAt(inst, 0), IR::Type::F32);
+	const auto rhs  = ReadOperand(SourceAt(inst, 1), IR::Type::F32);
+	const auto pair = ir.Emit(IR::ValueOpcode::CompositeConstructF32x2, {lhs, rhs});
+	WriteOperand(DestinationOperand(inst), ir.Emit(opcode, {pair}));
+}
+
+void Translator::V_CVT_PK_U8_F32(const Decoder::Instruction& inst) {
+	const auto source = IR::F32(ReadOperand(SourceAt(inst, 0), IR::Type::F32));
+	const auto byte   = ConvertF32ToU32Saturated(source, 255.0f, 255.0f, 255u);
+	const auto index  = ir.BitwiseAnd(ReadU32(SourceAt(inst, 1)), IR::U32(IR::Value(3u)));
+	const auto shift  = ir.ShiftLeftLogical(index, IR::U32(IR::Value(3u)));
+	const auto mask   = ir.ShiftLeftLogical(IR::U32(IR::Value(0xffu)), shift);
+	const auto base   = ir.BitwiseAnd(ReadU32(SourceAt(inst, 2)), ir.BitwiseNot(mask));
+	WriteOperand(DestinationOperand(inst), ir.BitwiseOr(base, ir.ShiftLeftLogical(byte, shift)));
+}
+
+void Translator::V_PACK_B32_F16(const Decoder::Instruction& inst) {
+	const auto low = ReadF16LaneBits(inst.src0, false);
+	const auto high =
+	    ir.ShiftLeftLogical(ReadF16LaneBits(inst.src1, false), IR::U32(IR::Value(16u)));
+	WriteOperand(DestinationOperand(inst), ir.BitwiseOr(low, high));
 }
 
 IR::U32 Translator::PackU16Lanes(IR::U32 low, IR::U32 high) {
@@ -172,4 +176,4 @@ IR::U32 Translator::PackU16Lanes(IR::U32 low, IR::U32 high) {
 	                    ir.ShiftLeftLogical(ir.BitwiseAnd(high, mask), IR::U32(IR::Value(16u))));
 }
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Dispatch.cpp

@@ -1,103 +1,53 @@
 #include "graphics/shader/recompiler/frontend/translate/Translator.h"
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
-bool Translator::TranslateU64MaskOperation(const IR::Instruction& inst) {
-	const auto invocation_result = EvaluateU64Mask(inst);
-	const auto is_exec_or_vcc    = [](const IR::Operand& operand) {
-		return operand.kind == IR::OperandKind::Register &&
-		       (operand.reg.file == IR::RegisterFile::Exec ||
-		        operand.reg.file == IR::RegisterFile::Vcc);
-	};
-	if (is_exec_or_vcc(inst.dst) || is_exec_or_vcc(inst.src[0]) ||
-	    (inst.src_count > 1u && is_exec_or_vcc(inst.src[1]))) {
-		WriteMask64(inst.dst, invocation_result);
-		ir.SetScc(invocation_result);
-		return true;
-	}
+bool Translator::TranslateInstruction(const Decoder::Instruction& inst, std::string* error) {
+	current_opcode = inst.opcode;
+	current_pc     = inst.pc;
 
-	const auto lhs        = ReadU32Pair(inst.src[0]);
-	auto       mask_valid = ReadMaskValid(inst.src[0]);
-	if (inst.src_count > 1u) {
-		mask_valid = ir.LogicalAnd(mask_valid, ReadMaskValid(inst.src[1]));
-	}
-	std::array<IR::U32, 2> result;
-	if (inst.op == IR::Opcode::BitwiseNotU64) {
-		result = {ir.BitwiseNot(lhs[0]), ir.BitwiseNot(lhs[1])};
-	} else {
-		const auto rhs = ReadU32Pair(inst.src[1]);
-		for (uint32_t component = 0; component < 2u; component++) {
-			switch (inst.op) {
-				case IR::Opcode::BitwiseAndU64:
-					result[component] = ir.BitwiseAnd(lhs[component], rhs[component]);
-					break;
-				case IR::Opcode::BitwiseAndNotU64:
-					result[component] =
-					    ir.BitwiseAnd(lhs[component], ir.BitwiseNot(rhs[component]));
-					break;
-				case IR::Opcode::BitwiseOrU64:
-					result[component] = ir.BitwiseOr(lhs[component], rhs[component]);
-					break;
-				case IR::Opcode::BitwiseOrNotU64:
-					result[component] = ir.BitwiseOr(lhs[component], ir.BitwiseNot(rhs[component]));
-					break;
-				case IR::Opcode::BitwiseXorU64:
-					result[component] = ir.BitwiseXor(lhs[component], rhs[component]);
-					break;
-				case IR::Opcode::BitwiseNandU64:
-					result[component] =
-					    ir.BitwiseNot(ir.BitwiseAnd(lhs[component], rhs[component]));
-					break;
-				case IR::Opcode::BitwiseNorU64:
-					result[component] = ir.BitwiseNot(ir.BitwiseOr(lhs[component], rhs[component]));
-					break;
-				case IR::Opcode::BitwiseXnorU64:
-					result[component] =
-					    ir.BitwiseNot(ir.BitwiseXor(lhs[component], rhs[component]));
-					break;
-				default: return false;
+	switch (inst.opcode) {
+		case Decoder::Opcode::UNKNOWN:
+		case Decoder::Opcode::COUNT:
+			if (error != nullptr) {
+				*error = "decoded opcode has no IR translation";
 			}
-		}
+			return false;
+		case Decoder::Opcode::UNSUPPORTED:
+			if (error != nullptr) {
+				*error = "unsupported decoded instruction: " + Decoder::InstructionToString(inst);
+			}
+			return false;
+		default: break;
 	}
-	WriteU32Pair(inst.dst, result);
-	if (inst.dst.kind == IR::OperandKind::Register &&
-	    inst.dst.reg.file == IR::RegisterFile::Scalar) {
-		const auto dst = static_cast<IR::ScalarReg>(inst.dst.reg.index);
-		ir.SetThreadBitScalarReg(dst, invocation_result);
-		ir.SetScalarMaskTag(dst, mask_valid);
-		const auto raw_nonzero =
-		    ir.INotEqual(ir.BitwiseOr(result[0], result[1]), IR::U32(IR::Value(0u)));
-		ir.SetScc(IR::U1(
-		    ir.Emit(IR::ValueOpcode::SelectU1, {mask_valid, invocation_result, raw_nonzero})));
-	} else {
-		ir.SetScc(ir.INotEqual(ir.BitwiseOr(result[0], result[1]), IR::U32(IR::Value(0u))));
+
+	bool translated = false;
+	switch (inst.family) {
+		case Decoder::Family::SOP1:
+		case Decoder::Family::SOP2:
+		case Decoder::Family::SOPK:
+		case Decoder::Family::SOPC:
+		case Decoder::Family::SOPP: translated = EmitScalar(inst, error); break;
+		case Decoder::Family::VOP1:
+		case Decoder::Family::VOP2:
+		case Decoder::Family::VOP3:
+		case Decoder::Family::VOP3P:
+		case Decoder::Family::VOPC: translated = EmitVector(inst, error); break;
+		case Decoder::Family::SMEM:
+		case Decoder::Family::MUBUF:
+		case Decoder::Family::MTBUF:
+		case Decoder::Family::FLAT:
+		case Decoder::Family::DS:
+		case Decoder::Family::MIMG: translated = EmitMemory(inst, error); break;
+		case Decoder::Family::VINTRP: translated = EmitInterpolation(inst, error); break;
+		case Decoder::Family::EXP: translated = EXP(inst, error); break;
+		default: break;
 	}
-	return true;
+
+	if (!translated && error != nullptr && error->empty()) {
+		*error = "decoded opcode has no IR translation";
+	}
+	return translated;
 }
 
-bool Translator::TranslateInstruction(const IR::Instruction& inst) {
-	switch (IR::GetOpcodeInfo(inst.op).lowering_class) {
-		case IR::LoweringClass::Control: return TranslateControlOperation(inst);
-		case IR::LoweringClass::Move: return TranslateMove(inst);
-		case IR::LoweringClass::Lane: return TranslateLaneOperation(inst);
-		case IR::LoweringClass::State: return TranslateStateOperation(inst);
-		case IR::LoweringClass::Memory: return TranslateMemoryOperation(inst);
-		case IR::LoweringClass::Attribute: return TranslateAttributeOperation(inst);
-		case IR::LoweringClass::IntegerCompare: return TranslateIntegerCompare(inst);
-		case IR::LoweringClass::Integer16Compare: return TranslateInteger16Compare(inst);
-		case IR::LoweringClass::FloatCompare: return TranslateFloatCompare(inst);
-		case IR::LoweringClass::Conversion: return TranslateConversion(inst);
-		case IR::LoweringClass::Integer16: return TranslateInteger16Operation(inst);
-		case IR::LoweringClass::PackedInteger16: return TranslatePackedInteger16(inst);
-		case IR::LoweringClass::PackedFloat16: return TranslatePackedFloat16(inst);
-		case IR::LoweringClass::Float16: return TranslateFloat16Operation(inst);
-		case IR::LoweringClass::Float: return TranslateFloatOperation(inst);
-		case IR::LoweringClass::U64Mask: return TranslateU64MaskOperation(inst);
-		case IR::LoweringClass::SimpleInteger: return TranslateSimpleInteger(inst);
-		case IR::LoweringClass::ComposedInteger: return TranslateComposedInteger(inst);
-		case IR::LoweringClass::ExtendedInteger: return TranslateExtendedInteger(inst);
-	}
-	return false;
-}
-
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Float.cpp

@@ -3,36 +3,29 @@
 #include <array>
 #include <utility>
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
-bool Translator::TranslatePackedFloat16(const IR::Instruction& inst) {
-	IR::ValueOpcode opcode;
-	switch (inst.op) {
-		case IR::Opcode::PackedAddF16: opcode = IR::ValueOpcode::FPAdd32; break;
-		case IR::Opcode::PackedMulF16: opcode = IR::ValueOpcode::FPMul32; break;
-		case IR::Opcode::PackedMinF16: opcode = IR::ValueOpcode::FPMin32; break;
-		case IR::Opcode::PackedMaxF16: opcode = IR::ValueOpcode::FPMax32; break;
-		case IR::Opcode::PackedFmaF16: opcode = IR::ValueOpcode::FPFma32; break;
-		default: return false;
-	}
+bool Translator::PackedFloat16(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                               bool accumulator, bool quiet_snan) {
 	const auto translate_lane = [&](bool high) {
-		const auto lhs = ReadF16LaneAsF32(inst.src[0], high, true);
-		const auto rhs = ReadF16LaneAsF32(inst.src[1], high, true);
+		const auto lhs = ReadF16LaneAsF32(inst.src0, high, true);
+		const auto rhs = ReadF16LaneAsF32(inst.src1, high, true);
 		IR::F32    result;
-		if (inst.src_count == 3u) {
-			result = IR::F32(
-			    ir.Emit(opcode, {lhs, rhs, ReadF16LaneAsF32(inst.src[2], high, true)}));
+		if (accumulator) {
+			result = IR::F32(ir.Emit(opcode, {lhs, rhs, ReadF16LaneAsF32(inst.dst, high, true)}));
+		} else if (inst.src_count == 3u) {
+			result = IR::F32(ir.Emit(opcode, {lhs, rhs, ReadF16LaneAsF32(inst.src2, high, true)}));
 		} else {
 			result = IR::F32(ir.Emit(opcode, {lhs, rhs}));
 		}
 		return ApplyF32ResultModifiers(inst.dst, result);
 	};
-	auto raw    = inst.dst;
+	auto raw    = DestinationOperand(inst);
 	raw.omod    = 0u;
 	raw.clamp   = false;
 	auto result = PackHalf2x16(translate_lane(false), translate_lane(true));
-	if (inst.op == IR::Opcode::PackedMinF16 || inst.op == IR::Opcode::PackedMaxF16) {
-		const auto quiet_snan = [&](const IR::Operand& operand, bool high) {
+	if (quiet_snan) {
+		const auto quiet_snan_lane = [&](const Decoder::Operand& operand, bool high) {
 			const auto bits     = ReadU16LaneAsU32(operand, high, false);
 			const auto exponent = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x7c00u)));
 			const auto payload  = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x01ffu)));
@@ -41,8 +34,8 @@ bool Translator::TranslatePackedFloat16(const IR::Instruction& inst) {
 			return std::pair {snan, ir.BitwiseOr(bits, IR::U32(IR::Value(0x0200u)))};
 		};
 		const auto override_lane = [&](bool high) {
-			const auto [lhs_snan, lhs_quiet] = quiet_snan(inst.src[0], high);
-			const auto [rhs_snan, rhs_quiet] = quiet_snan(inst.src[1], high);
+			const auto [lhs_snan, lhs_quiet] = quiet_snan_lane(inst.src0, high);
+			const auto [rhs_snan, rhs_quiet] = quiet_snan_lane(inst.src1, high);
 			const auto normal = high ? ir.ShiftRightLogical(result, IR::U32(IR::Value(16u)))
 			                         : ir.BitwiseAnd(result, IR::U32(IR::Value(0xffffu)));
 			return ir.Select(lhs_snan, lhs_quiet, ir.Select(rhs_snan, rhs_quiet, normal));
@@ -53,152 +46,133 @@ bool Translator::TranslatePackedFloat16(const IR::Instruction& inst) {
 	return true;
 }
 
-bool Translator::TranslateFloat16Operation(const IR::Instruction& inst) {
-	IR::ValueOpcode opcode;
-	switch (inst.op) {
-		case IR::Opcode::AddF16: opcode = IR::ValueOpcode::FPAdd32; break;
-		case IR::Opcode::SubF16: opcode = IR::ValueOpcode::FPSub32; break;
-		case IR::Opcode::MulF16: opcode = IR::ValueOpcode::FPMul32; break;
-		case IR::Opcode::MinF16: opcode = IR::ValueOpcode::FPMin32; break;
-		case IR::Opcode::MaxF16: opcode = IR::ValueOpcode::FPMax32; break;
-		case IR::Opcode::FmaF16:
-		case IR::Opcode::MadMixF16: opcode = IR::ValueOpcode::FPFma32; break;
-		case IR::Opcode::RcpF16: opcode = IR::ValueOpcode::FPRecip32; break;
-		case IR::Opcode::SqrtF16: opcode = IR::ValueOpcode::FPSqrt; break;
-		case IR::Opcode::InverseSqrtF16: opcode = IR::ValueOpcode::FPRecipSqrt32; break;
-		case IR::Opcode::Log2F16: opcode = IR::ValueOpcode::FPLog2; break;
-		case IR::Opcode::Exp2F16: opcode = IR::ValueOpcode::FPExp2; break;
-		case IR::Opcode::FloorF16: opcode = IR::ValueOpcode::FPFloor32; break;
-		case IR::Opcode::CeilF16: opcode = IR::ValueOpcode::FPCeil32; break;
-		case IR::Opcode::TruncF16: opcode = IR::ValueOpcode::FPTrunc32; break;
-		case IR::Opcode::RoundEvenF16: opcode = IR::ValueOpcode::FPRoundEven32; break;
-		case IR::Opcode::Min3F16: opcode = IR::ValueOpcode::FPMinTri32; break;
-		case IR::Opcode::Max3F16: opcode = IR::ValueOpcode::FPMaxTri32; break;
-		case IR::Opcode::Med3F16: opcode = IR::ValueOpcode::FPMedTri32; break;
-		default: return false;
-	}
-	std::array<IR::Value, 3> args;
-	for (uint32_t index = 0; index < inst.src_count; index++) {
-		args[index] = inst.op == IR::Opcode::MadMixF16 ? IR::Value(ReadMixF32(inst.src[index]))
-		                                               : IR::Value(ReadF16AsF32(inst.src[index]));
-	}
-	IR::F32 result;
-	switch (inst.src_count) {
-		case 1: result = IR::F32(ir.Emit(opcode, {args[0]})); break;
-		case 2: result = IR::F32(ir.Emit(opcode, {args[0], args[1]})); break;
-		case 3: result = IR::F32(ir.Emit(opcode, {args[0], args[1], args[2]})); break;
-		default: EXIT("invalid half-float source count: %u", inst.src_count);
-	}
-	if (inst.op == IR::Opcode::SqrtF16 || inst.op == IR::Opcode::InverseSqrtF16 ||
-	    inst.op == IR::Opcode::Log2F16) {
-		const auto negative = IR::U1(
-		    ir.Emit(IR::ValueOpcode::FPOrdLessThan32, {args[0], IR::Value::F32(0.0f)}));
-		result = ApplyF32ResultModifiers(inst.dst, result);
-		const auto bits = PackHalf2x16(result, IR::F32(IR::Value::F32(0.0f)));
-		const auto invalid = IR::U32(IR::Value(inst.dst.clamp ? 0u : 0xfe00u));
-		WriteU16(inst.dst, ir.Select(negative, invalid, bits));
+bool Translator::Float16Unary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                              bool invalid_negative) {
+	const auto argument = ReadF16AsF32(inst.src0);
+	auto       result   = IR::F32(ir.Emit(opcode, {argument}));
+	if (!invalid_negative) {
+		WriteF16(DestinationOperand(inst), result);
 		return true;
 	}
-	WriteF16(inst.dst, result);
+	const auto negative =
+	    IR::U1(ir.Emit(IR::ValueOpcode::FPOrdLessThan32, {argument, IR::Value::F32(0.0f)}));
+	result             = ApplyF32ResultModifiers(inst.dst, result);
+	const auto bits    = PackHalf2x16(result, IR::F32(IR::Value::F32(0.0f)));
+	const auto invalid = IR::U32(IR::Value(inst.dst.clamp ? 0u : 0xfe00u));
+	WriteU16(DestinationOperand(inst), ir.Select(negative, invalid, bits));
 	return true;
 }
 
-bool Translator::TranslateFloatOperation(const IR::Instruction& inst) {
-	if (inst.op == IR::Opcode::FrexpMantF32) {
-		const auto bits     = ReadU32(inst.src[0]);
-		const auto exponent = IR::U32(
-		    ir.Emit(IR::ValueOpcode::BitFieldUExtract, {bits, IR::Value(23u), IR::Value(8u)}));
-		const auto mantissa = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x007fffffu)));
-		const auto sign     = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x80000000u)));
-		const auto base     = ir.BitwiseOr(sign, IR::U32(IR::Value(0x3f000000u)));
-		const auto normal   = ir.BitwiseOr(base, mantissa);
-		const auto msb      = IR::U32(ir.Emit(IR::ValueOpcode::FindUMsb32, {mantissa}));
-		const auto shift    = ir.ISub(IR::U32(IR::Value(23u)), msb);
-		const auto fraction =
-		    ir.BitwiseAnd(ir.ShiftLeftLogical(mantissa, shift), IR::U32(IR::Value(0x007fffffu)));
-		const auto subnormal = ir.BitwiseOr(base, fraction);
-		const auto zero      = ir.IEqual(mantissa, IR::U32(IR::Value(0u)));
-		const auto finite    = ir.Select(ir.INotEqual(exponent, IR::U32(IR::Value(0u))), normal,
-		                                 ir.Select(zero, bits, subnormal));
-		const auto result = ir.Select(ir.IEqual(exponent, IR::U32(IR::Value(0xffu))), bits, finite);
-		WriteOperand(inst.dst, ir.BitCastF32(result));
-		return true;
-	}
-	IR::ValueOpcode opcode {};
-	switch (inst.op) {
-		case IR::Opcode::RcpF32: opcode = IR::ValueOpcode::FPRecip32; break;
-		case IR::Opcode::RcpIflagF32: opcode = IR::ValueOpcode::FPRecipIFlag32; break;
-		case IR::Opcode::FractF32: opcode = IR::ValueOpcode::FPFract32; break;
-		case IR::Opcode::TruncF32: opcode = IR::ValueOpcode::FPTrunc32; break;
-		case IR::Opcode::CeilF32: opcode = IR::ValueOpcode::FPCeil32; break;
-		case IR::Opcode::RoundEvenF32: opcode = IR::ValueOpcode::FPRoundEven32; break;
-		case IR::Opcode::FloorF32: opcode = IR::ValueOpcode::FPFloor32; break;
-		case IR::Opcode::Exp2F32: opcode = IR::ValueOpcode::FPExp2; break;
-		case IR::Opcode::Log2F32: opcode = IR::ValueOpcode::FPLog2; break;
-		case IR::Opcode::InverseSqrtF32: opcode = IR::ValueOpcode::FPRecipSqrt32; break;
-		case IR::Opcode::SqrtF32: opcode = IR::ValueOpcode::FPSqrt; break;
-		case IR::Opcode::SinF32: opcode = IR::ValueOpcode::FPSin; break;
-		case IR::Opcode::CosF32: opcode = IR::ValueOpcode::FPCos; break;
-		case IR::Opcode::FAddF32: opcode = IR::ValueOpcode::FPAdd32; break;
-		case IR::Opcode::FSubF32: opcode = IR::ValueOpcode::FPSub32; break;
-		case IR::Opcode::FMulF32: opcode = IR::ValueOpcode::FPMul32; break;
-		case IR::Opcode::FMinF32: opcode = IR::ValueOpcode::FPMin32; break;
-		case IR::Opcode::FMaxF32: opcode = IR::ValueOpcode::FPMax32; break;
-		case IR::Opcode::FMadF32: opcode = IR::ValueOpcode::FPFma32; break;
-		case IR::Opcode::FMin3F32: opcode = IR::ValueOpcode::FPMinTri32; break;
-		case IR::Opcode::FMax3F32: opcode = IR::ValueOpcode::FPMaxTri32; break;
-		case IR::Opcode::FMed3F32: opcode = IR::ValueOpcode::FPMedTri32; break;
-		case IR::Opcode::LdexpF32: opcode = IR::ValueOpcode::FPLdexp; break;
-		default: break;
-	}
-	if (opcode != IR::ValueOpcode {}) {
-		std::array<IR::Value, 3> args;
-		for (uint32_t index = 0; index < inst.src_count; index++) {
-			const auto arg_type = IR::ArgTypeOf(opcode, index);
-			args[index] =
-			    arg_type == IR::Type::F32
-			        ? (inst.op == IR::Opcode::FMadF32 ? IR::Value(ReadMixF32(inst.src[index]))
-			                                          : ReadOperand(inst.src[index], IR::Type::F32))
-			        : ReadOperand(inst.src[index], arg_type);
-		}
-		IR::Value result;
-		switch (inst.src_count) {
-			case 1: result = ir.Emit(opcode, {args[0]}); break;
-			case 2: result = ir.Emit(opcode, {args[0], args[1]}); break;
-			case 3: result = ir.Emit(opcode, {args[0], args[1], args[2]}); break;
-			default: EXIT("invalid floating-point source count: %u", inst.src_count);
-		}
-		WriteOperand(inst.dst, result);
-		return true;
-	}
+bool Translator::Float16Binary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                               bool reverse) {
+	const auto lhs = ReadF16AsF32(reverse ? inst.src1 : inst.src0);
+	const auto rhs = ReadF16AsF32(reverse ? inst.src0 : inst.src1);
+	WriteF16(DestinationOperand(inst), IR::F32(ir.Emit(opcode, {lhs, rhs})));
+	return true;
+}
 
+bool Translator::Float16Ternary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                                bool accumulator, bool mix) {
+	std::array<IR::Value, 3> args;
+	for (uint32_t index = 0; index < args.size(); index++) {
+		const auto operand = accumulator && index == 2u ? inst.dst : SourceAt(inst, index);
+		args[index] = mix ? IR::Value(ReadMixF32(operand)) : IR::Value(ReadF16AsF32(operand));
+	}
+	WriteF16(DestinationOperand(inst), IR::F32(ir.Emit(opcode, {args[0], args[1], args[2]})));
+	return true;
+}
+
+bool Translator::FloatUnary(const Decoder::Instruction& inst, IR::ValueOpcode opcode) {
+	const auto type = IR::ArgTypeOf(opcode, 0);
+	WriteOperand(DestinationOperand(inst), ir.Emit(opcode, {ReadOperand(inst.src0, type)}));
+	return true;
+}
+
+bool Translator::FloatBinary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                             bool reverse) {
+	std::array<IR::Value, 2> args;
+	for (uint32_t index = 0; index < args.size(); index++) {
+		const auto operand = SourceAt(inst, reverse ? 1u - index : index);
+		args[index]        = ReadOperand(operand, IR::ArgTypeOf(opcode, index));
+	}
+	WriteOperand(DestinationOperand(inst), ir.Emit(opcode, {args[0], args[1]}));
+	return true;
+}
+
+bool Translator::FloatTernary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+                              bool accumulator, bool mix) {
+	std::array<IR::Value, 3> args;
+	for (uint32_t index = 0; index < args.size(); index++) {
+		const auto operand = accumulator && index == 2u ? inst.dst : SourceAt(inst, index);
+		const auto type    = IR::ArgTypeOf(opcode, index);
+		args[index]        = type == IR::Type::F32 && mix ? IR::Value(ReadMixF32(operand))
+		                                                  : ReadOperand(operand, type);
+	}
+	WriteOperand(DestinationOperand(inst), ir.Emit(opcode, {args[0], args[1], args[2]}));
+	return true;
+}
+
+bool Translator::V_FREXP_MANT_F32(const Decoder::Instruction& inst) {
+	const auto bits = ReadU32(inst.src0);
+	const auto exponent =
+	    IR::U32(ir.Emit(IR::ValueOpcode::BitFieldUExtract, {bits, IR::Value(23u), IR::Value(8u)}));
+	const auto mantissa = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x007fffffu)));
+	const auto sign     = ir.BitwiseAnd(bits, IR::U32(IR::Value(0x80000000u)));
+	const auto base     = ir.BitwiseOr(sign, IR::U32(IR::Value(0x3f000000u)));
+	const auto normal   = ir.BitwiseOr(base, mantissa);
+	const auto msb      = IR::U32(ir.Emit(IR::ValueOpcode::FindUMsb32, {mantissa}));
+	const auto shift    = ir.ISub(IR::U32(IR::Value(23u)), msb);
+	const auto fraction =
+	    ir.BitwiseAnd(ir.ShiftLeftLogical(mantissa, shift), IR::U32(IR::Value(0x007fffffu)));
+	const auto subnormal = ir.BitwiseOr(base, fraction);
+	const auto zero      = ir.IEqual(mantissa, IR::U32(IR::Value(0u)));
+	const auto finite    = ir.Select(ir.INotEqual(exponent, IR::U32(IR::Value(0u))), normal,
+	                                 ir.Select(zero, bits, subnormal));
+	const auto result    = ir.Select(ir.IEqual(exponent, IR::U32(IR::Value(0xffu))), bits, finite);
+	WriteOperand(DestinationOperand(inst), ir.BitCastF32(result));
+	return true;
+}
+
+bool Translator::V_DOT2C_F32_F16(const Decoder::Instruction& inst) {
+	auto a          = inst.src0;
+	a.op_sel        = false;
+	a.op_sel_hi     = true;
+	auto b          = inst.src1;
+	b.op_sel        = false;
+	b.op_sel_hi     = true;
+	const auto a_lo = ReadF16LaneAsF32(a, false);
+	const auto a_hi = ReadF16LaneAsF32(a, true);
+	const auto b_lo = ReadF16LaneAsF32(b, false);
+	const auto b_hi = ReadF16LaneAsF32(b, true);
+	const auto acc  = ReadMixF32(inst.dst);
+	const auto lo   = IR::F32(ir.Emit(IR::ValueOpcode::FPFma32, {a_lo, b_lo, acc}));
+	WriteOperand(DestinationOperand(inst), ir.Emit(IR::ValueOpcode::FPFma32, {a_hi, b_hi, lo}));
+	return true;
+}
+
+bool Translator::V_CUBEID_F32(const Decoder::Instruction& inst) {
+	return FloatCube(inst, 0u);
+}
+
+bool Translator::V_CUBESC_F32(const Decoder::Instruction& inst) {
+	return FloatCube(inst, 1u);
+}
+
+bool Translator::V_CUBETC_F32(const Decoder::Instruction& inst) {
+	return FloatCube(inst, 2u);
+}
+
+bool Translator::V_CUBEMA_F32(const Decoder::Instruction& inst) {
+	return FloatCube(inst, 3u);
+}
+
+bool Translator::FloatCube(const Decoder::Instruction& inst, uint32_t result_kind) {
 	const auto select_f32 = [&](IR::U1 condition, IR::F32 true_value, IR::F32 false_value) {
 		return IR::F32(ir.Emit(IR::ValueOpcode::SelectF32, {condition, true_value, false_value}));
 	};
-	if (inst.op == IR::Opcode::Dot2AccF32F16) {
-		auto a          = inst.src[0];
-		a.op_sel        = false;
-		a.op_sel_hi     = true;
-		auto b          = inst.src[1];
-		b.op_sel        = false;
-		b.op_sel_hi     = true;
-		const auto a_lo = ReadF16LaneAsF32(a, false);
-		const auto a_hi = ReadF16LaneAsF32(a, true);
-		const auto b_lo = ReadF16LaneAsF32(b, false);
-		const auto b_hi = ReadF16LaneAsF32(b, true);
-		const auto acc  = ReadMixF32(inst.src[2]);
-		const auto lo   = IR::F32(ir.Emit(IR::ValueOpcode::FPFma32, {a_lo, b_lo, acc}));
-		WriteOperand(inst.dst, ir.Emit(IR::ValueOpcode::FPFma32, {a_hi, b_hi, lo}));
-		return true;
-	}
-	if (inst.op != IR::Opcode::CubeIdF32 && inst.op != IR::Opcode::CubeScF32 &&
-	    inst.op != IR::Opcode::CubeTcF32 && inst.op != IR::Opcode::CubeMaF32) {
-		return false;
-	}
-
-	const auto x  = ReadMixF32(inst.src[0]);
-	const auto y  = ReadMixF32(inst.src[1]);
-	const auto z  = ReadMixF32(inst.src[2]);
+	const auto x  = ReadMixF32(inst.src0);
+	const auto y  = ReadMixF32(inst.src1);
+	const auto z  = ReadMixF32(inst.src2);
 	const auto nx = IR::F32(ir.Emit(IR::ValueOpcode::FPNeg32, {x}));
 	const auto ny = IR::F32(ir.Emit(IR::ValueOpcode::FPNeg32, {y}));
 	const auto nz = IR::F32(ir.Emit(IR::ValueOpcode::FPNeg32, {z}));
@@ -216,28 +190,26 @@ bool Translator::TranslateFloatOperation(const IR::Instruction& inst) {
 		return select_f32(z_face, z_value, select_f32(y_face, y_value, x_value));
 	};
 	IR::F32 result;
-	switch (inst.op) {
-		case IR::Opcode::CubeIdF32:
+	switch (result_kind) {
+		case 0u:
 			result = select_face(
 			    select_f32(x_neg, IR::F32(IR::Value::F32(1.0f)), IR::F32(IR::Value::F32(0.0f))),
 			    select_f32(y_neg, IR::F32(IR::Value::F32(3.0f)), IR::F32(IR::Value::F32(2.0f))),
 			    select_f32(z_neg, IR::F32(IR::Value::F32(5.0f)), IR::F32(IR::Value::F32(4.0f))));
 			break;
-		case IR::Opcode::CubeScF32:
-			result = select_face(select_f32(x_neg, z, nz), x, select_f32(z_neg, nx, x));
-			break;
-		case IR::Opcode::CubeTcF32: result = select_face(ny, select_f32(y_neg, nz, z), ny); break;
-		case IR::Opcode::CubeMaF32: {
+		case 1u: result = select_face(select_f32(x_neg, z, nz), x, select_f32(z_neg, nx, x)); break;
+		case 2u: result = select_face(ny, select_f32(y_neg, nz, z), ny); break;
+		case 3u: {
 			const auto two = IR::F32(IR::Value::F32(2.0f));
 			result         = select_face(IR::F32(ir.Emit(IR::ValueOpcode::FPMul32, {x, two})),
 			                             IR::F32(ir.Emit(IR::ValueOpcode::FPMul32, {y, two})),
 			                             IR::F32(ir.Emit(IR::ValueOpcode::FPMul32, {z, two})));
 			break;
 		}
-		default: return false;
+		default: EXIT("invalid cube result kind");
 	}
-	WriteOperand(inst.dst, result);
+	WriteOperand(DestinationOperand(inst), result);
 	return true;
 }
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Scalar.cpp

@@ -0,0 +1,221 @@
+#include "graphics/shader/recompiler/frontend/translate/Translator.h"
+
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
+
+bool Translator::EmitScalar(const Decoder::Instruction& inst, std::string* error) {
+	using O = Decoder::Opcode;
+	switch (inst.opcode) {
+		case O::S_MOV_B32:
+		case O::S_MOVK_I32: MOV_B32(inst, false); return true;
+		case O::S_MOV_B64: S_MOV_B64(inst); return true;
+		case O::S_WQM_B64: S_WQM_B64(inst); return true;
+		case O::S_GETPC_B64: return S_GETPC_B64(inst, error);
+		case O::S_SETPC_B64: return true;
+		case O::S_CSELECT_B32: S_CSELECT_B32(inst); return true;
+		case O::S_CSELECT_B64: S_CSELECT_B64(inst); return true;
+		case O::S_SETREG_B32: EmitControlNop(); return true;
+		case O::S_WAITCNT: EmitWaitcnt(); return true;
+
+		case O::S_AND_SAVEEXEC_B32:
+			S_SAVEEXEC(inst, IR::ValueOpcode::LogicalAnd, false, false, false);
+			return true;
+		case O::S_ANDN1_SAVEEXEC_B32:
+			S_SAVEEXEC(inst, IR::ValueOpcode::LogicalAnd, false, true, false);
+			return true;
+		case O::S_AND_SAVEEXEC_B64:
+			S_SAVEEXEC(inst, IR::ValueOpcode::LogicalAnd, false, false, true);
+			return true;
+		case O::S_ANDN1_SAVEEXEC_B64:
+			S_SAVEEXEC(inst, IR::ValueOpcode::LogicalAnd, false, true, true);
+			return true;
+		case O::S_ORN2_SAVEEXEC_B64:
+			S_SAVEEXEC(inst, IR::ValueOpcode::LogicalOr, true, false, true);
+			return true;
+		case O::S_ADD_U32: ADD_U32(inst, false, false); return true;
+		case O::S_ADDC_U32: ADD_U32(inst, false, true); return true;
+		case O::S_SUB_U32: SUB_U32(inst, false, false); return true;
+		case O::S_SUBB_U32: SUBB_U32(inst, false); return true;
+		case O::S_ADD_I32: S_ADD_SUB_I32(inst, false); return true;
+		case O::S_SUB_I32: S_ADD_SUB_I32(inst, true); return true;
+		case O::S_LSHL1_ADD_U32: S_LSHL_ADD_U32(inst, 1u); return true;
+		case O::S_LSHL2_ADD_U32: S_LSHL_ADD_U32(inst, 2u); return true;
+		case O::S_LSHL3_ADD_U32: S_LSHL_ADD_U32(inst, 3u); return true;
+		case O::S_LSHL4_ADD_U32: S_LSHL_ADD_U32(inst, 4u); return true;
+		case O::S_MIN_I32:
+			ScalarMinMax32(inst, IR::ValueOpcode::SMin32, IR::ValueOpcode::SLessThan32);
+			return true;
+		case O::S_MAX_I32:
+			ScalarMinMax32(inst, IR::ValueOpcode::SMax32, IR::ValueOpcode::SGreaterThan32);
+			return true;
+		case O::S_MIN_U32:
+			ScalarMinMax32(inst, IR::ValueOpcode::UMin32, IR::ValueOpcode::ULessThan32);
+			return true;
+		case O::S_MAX_U32:
+			ScalarMinMax32(inst, IR::ValueOpcode::UMax32, IR::ValueOpcode::UGreaterThan32);
+			return true;
+
+		case O::S_CMP_EQ_U32:
+		case O::S_CMP_EQ_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::IEqual32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_LG_U32:
+		case O::S_CMP_LG_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_GT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThan32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_GE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThanEqual32, IR::Type::U32, true,
+			                   false);
+			return true;
+		case O::S_CMP_LT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThan32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_LE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThanEqual32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_GT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThan32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_GE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThanEqual32, IR::Type::U32, true,
+			                   false);
+			return true;
+		case O::S_CMP_LT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThan32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_LE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThanEqual32, IR::Type::U32, true, false);
+			return true;
+		case O::S_CMP_EQ_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::IEqual64, IR::Type::U64, true, false);
+			return true;
+		case O::S_CMP_LG_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual64, IR::Type::U64, true, false);
+			return true;
+
+		case O::S_AND_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalAnd, IR::ValueOpcode::BitwiseAnd32,
+			                  false, false, false);
+		case O::S_ANDN2_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalAnd, IR::ValueOpcode::BitwiseAnd32,
+			                  true, false, false);
+		case O::S_OR_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalOr, IR::ValueOpcode::BitwiseOr32, false,
+			                  false, false);
+		case O::S_ORN2_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalOr, IR::ValueOpcode::BitwiseOr32, true,
+			                  false, false);
+		case O::S_XOR_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalXor, IR::ValueOpcode::BitwiseXor32,
+			                  false, false, false);
+		case O::S_NAND_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalAnd, IR::ValueOpcode::BitwiseAnd32,
+			                  false, true, false);
+		case O::S_NOR_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalOr, IR::ValueOpcode::BitwiseOr32, false,
+			                  true, false);
+		case O::S_XNOR_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalXor, IR::ValueOpcode::BitwiseXor32,
+			                  false, true, false);
+		case O::S_NOT_B64:
+			return S_U64_MASK(inst, IR::ValueOpcode::LogicalAnd, IR::ValueOpcode::BitwiseAnd32,
+			                  false, false, true);
+
+		case O::S_ABS_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::IAbs32, IR::Type::U32, false, false, true);
+		case O::S_MUL_I32:
+		case O::S_MULK_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::IMul32, IR::Type::U32, false, false, false);
+		case O::S_MUL_HI_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMulHi, IR::Type::U32, false, false, false);
+		case O::S_AND_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseAnd32, IR::Type::U32, false, false,
+			                     true);
+		case O::S_OR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseOr32, IR::Type::U32, false, false,
+			                     true);
+		case O::S_XOR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseXor32, IR::Type::U32, false, false,
+			                     true);
+		case O::S_NOT_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseNot32, IR::Type::U32, false, false,
+			                     true);
+		case O::S_BREV_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitReverse32, IR::Type::U32, false, false,
+			                     false);
+		case O::S_BCNT1_I32_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitCount32, IR::Type::U32, false, false,
+			                     true);
+		case O::S_BCNT1_I32_B64:
+			return SimpleInteger(inst, IR::ValueOpcode::BitCount64, IR::Type::U64, false, false,
+			                     true);
+		case O::S_FF1_I32_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::FindILsb32, IR::Type::U32, false, false,
+			                     false);
+		case O::S_LSHL_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftLeftLogical32, IR::Type::U32, false,
+			                     true, true);
+		case O::S_LSHR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightLogical32, IR::Type::U32, false,
+			                     true, true);
+		case O::S_ASHR_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightArithmetic32, IR::Type::U32,
+			                     false, true, true);
+		case O::S_LSHL_B64:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftLeftLogical64, IR::Type::U64, false,
+			                     false, true);
+		case O::S_LSHR_B64:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightLogical64, IR::Type::U64, false,
+			                     false, true);
+
+		case O::S_ANDN2_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseAnd32, true, false, true);
+		case O::S_ORN2_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseOr32, true, false, true);
+		case O::S_NAND_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseAnd32, false, true, true);
+		case O::S_NOR_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseOr32, false, true, true);
+		case O::S_XNOR_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseXor32, false, true, true);
+		case O::S_FF1_I32_B64: return S_FF1_I32_B64(inst);
+		case O::S_FLBIT_I32_B64: return S_FLBIT_I32_B64(inst);
+
+		case O::S_BITSET0_B32: return S_BITSET_B32(inst, false);
+		case O::S_BITSET1_B32: return S_BITSET_B32(inst, true);
+		case O::S_BITREPLICATE_B64_B32: return S_BITREPLICATE_B64_B32(inst);
+		case O::S_QUADMASK_B64: return S_QUADMASK_B64(inst);
+		case O::S_BFM_B32: return BFM_B32(inst);
+		case O::S_BFM_B64: return S_BFM_B64(inst);
+		case O::S_BFE_U32: return S_BFE_U32(inst, false);
+		case O::S_BFE_I32: return S_BFE_U32(inst, true);
+		case O::S_BFE_U64: return S_BFE_U64(inst);
+		case O::S_BITCMP0_B32: return S_BITCMP_B32(inst, false);
+		case O::S_BITCMP1_B32: return S_BITCMP_B32(inst, true);
+		case O::S_PACK_LL_B32_B16: return PackB16(inst, false, false);
+		case O::S_PACK_LH_B32_B16: return PackB16(inst, false, true);
+		case O::S_PACK_HH_B32_B16: return PackB16(inst, true, true);
+
+		case O::S_NOP:
+		case O::S_SLEEP:
+		case O::S_TRAP: EmitControlNop(); return true;
+		case O::S_WAITCNT_DEPCTR: EmitWaitcnt(); return true;
+		case O::S_BARRIER: S_BARRIER(); return true;
+		case O::S_SENDMSG: S_SENDMSG(); return true;
+		case O::S_TTRACEDATA: S_TTRACEDATA(); return true;
+		case O::S_INST_PREFETCH: S_INST_PREFETCH(); return true;
+		case O::S_BRANCH:
+		case O::S_CBRANCH_SCC0:
+		case O::S_CBRANCH_SCC1:
+		case O::S_CBRANCH_VCCZ:
+		case O::S_CBRANCH_VCCNZ:
+		case O::S_CBRANCH_EXECZ:
+		case O::S_CBRANCH_EXECNZ:
+		case O::S_ENDPGM: return true;
+		default: return false;
+	}
+}
+
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Translate.h

@@ -1,12 +1,44 @@
 #pragma once
 
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
+#include "graphics/shader/recompiler/frontend/cfg/ShaderCFG.h"
+#include "graphics/shader/recompiler/frontend/decode/ShaderDecoder.h"
+#include "graphics/shader/recompiler/ir/ShaderIR.h"
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
 
 namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
-bool TranslateProgram(const IR::Program& source, IR::ValueProgram& result,
-                      const ShaderVertexInputInfo*  vertex_input_info,
-                      const ShaderPixelInputInfo*   pixel_input_info,
-                      const ShaderComputeInputInfo* compute_input_info, std::string* error);
+struct EmbeddedFetchLoad {
+	uint32_t              pc         = 0;
+	int                   attrib_id  = -1;
+	uint32_t              components = 0;
+	std::vector<uint32_t> prolog_loads;
+};
+
+struct EmbeddedFetchPlan {
+	std::vector<EmbeddedFetchLoad> loads;
+	int32_t                        vertex_offset_sgpr = -1;
+};
+
+struct TranslateOptions {
+	ShaderType                    stage               = ShaderType::Unknown;
+	uint32_t                      wave_size           = 64;
+	uint64_t                      shader_hash         = 0;
+	uint32_t                      user_data_base      = 0;
+	uint32_t                      user_data_count     = 64;
+	uint32_t                      scratch_dwords      = 0;
+	bool                          dispatcher_fallback = false;
+	CFG::FailureKind              cfg_failure_kind    = CFG::FailureKind::None;
+	std::string_view              fallback_reason;
+	const ShaderVertexInputInfo*  vertex         = nullptr;
+	const ShaderPixelInputInfo*   pixel          = nullptr;
+	const ShaderComputeInputInfo* compute        = nullptr;
+	const EmbeddedFetchPlan*      embedded_fetch = nullptr;
+};
+
+bool TranslateProgram(const Decoder::Program& decoded, const CFG::Graph& cfg,
+                      const TranslateOptions& options, IR::Program& result, std::string* error);
 
 } // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Translator.h

@@ -5,58 +5,59 @@
 
 #include <array>
 
-namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail {
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
 
 class Translator {
 public:
-	Translator(IR::ValueProgram& program, IR::Block* block, uint32_t vector_limit,
-	           uint32_t wave_size)
-	    : value_program(program), ir(block), current_vector_limit(vector_limit),
+	Translator(IR::Program& program, IR::Block* block, uint32_t vector_limit, uint32_t wave_size)
+	    : program(program), ir(block), current_vector_limit(vector_limit),
 	      current_wave_size(wave_size) {}
 
-	bool TranslateBlock(const IR::BasicBlock& source, std::string* error);
-	bool AddBranchCondition(const IR::BasicBlock& source, IR::ValueBlockInfo& info,
-	                        std::string* error);
+	bool TranslateInstruction(const Decoder::Instruction& inst, std::string* error);
+	bool TranslateEmbeddedFetch(const Decoder::Instruction& inst, uint32_t attribute,
+	                            uint32_t component_count);
+	bool AddBranchCondition(const CFG::BasicBlock& source, IR::BlockInfo& info, std::string* error);
 
 private:
-	IR::Operand            OffsetOperand(const IR::Operand& operand, uint32_t offset);
-	IR::Operand            ScalarDestinationOperand(const IR::Operand& operand, uint32_t offset);
-	IR::Operand            PlainOperand(const IR::Operand& operand);
-	IR::U32                ReadPcRelativeU32(const IR::Operand& operand);
+	Decoder::Operand SourceAt(const Decoder::Instruction& inst, uint32_t index);
+	Decoder::Operand DestinationOperand(const Decoder::Instruction& inst);
+	Decoder::Operand OffsetOperand(const Decoder::Operand& operand, uint32_t offset);
+	Decoder::Operand ScalarDestinationOperand(const Decoder::Operand& operand, uint32_t offset);
+	Decoder::Operand PlainOperand(const Decoder::Operand& operand);
 	std::array<IR::U32, 2> BallotMask(IR::U1 value);
-	IR::U32                ReadRawU32(const IR::Operand& operand);
+	IR::U32                ReadRawU32(const Decoder::Operand& operand);
 	IR::U32                ReadScalarCode(uint32_t code);
-	IR::U32                ApplyBitSourceModifiers(const IR::Operand& operand, IR::U32 value);
-	IR::Value              ReadOperand(const IR::Operand& operand, IR::Type type);
+	IR::U32                ApplyBitSourceModifiers(const Decoder::Operand& operand, IR::U32 value);
+	IR::Value              ReadOperand(const Decoder::Operand& operand, IR::Type type);
 	IR::U1                 ThreadBit(IR::U32 low);
-	void                   WriteRawU32(const IR::Operand& operand, IR::U32 value);
-	IR::F32                ApplyF32ResultModifiers(const IR::Operand& operand, IR::F32 value);
-	void                   WriteOperand(const IR::Operand& operand, IR::Value value);
+	void                   WriteRawU32(const Decoder::Operand& operand, IR::U32 value);
+	IR::F32                ApplyF32ResultModifiers(const Decoder::Operand& operand, IR::F32 value);
+	void                   WriteOperand(const Decoder::Operand& operand, IR::Value value);
 	IR::U32                PackHalf2x16(IR::F32 low, IR::F32 high);
-	void                   WriteF16(const IR::Operand& operand, IR::F32 value);
-	void                   WriteU16(const IR::Operand& operand, IR::U32 value);
-	IR::U32                ReadU32(const IR::Operand& operand);
-	std::array<IR::U32, 2> ReadU32Pair(const IR::Operand& operand);
-	IR::U64                ReadU64(const IR::Operand& operand);
-	IR::F32 ReadF16LaneAsF32(const IR::Operand& operand, bool high_lane, bool packed = false);
-	IR::F32 ReadF16AsF32(const IR::Operand& operand);
-	IR::F32 ReadMixF32(const IR::Operand& operand);
-	IR::U32 ReadU16LaneRaw(const IR::Operand& operand, bool high_lane);
-	IR::U32 ReadU16LaneAsU32(const IR::Operand& operand, bool high_lane, bool sign_extend);
-	IR::U32 ReadU16AsU32(const IR::Operand& operand, bool sign_extend);
-	IR::U32 ReadF16LaneBits(const IR::Operand& operand, bool high_lane);
+	void                   WriteF16(const Decoder::Operand& operand, IR::F32 value);
+	void                   WriteU16(const Decoder::Operand& operand, IR::U32 value);
+	IR::U32                ReadU32(const Decoder::Operand& operand);
+	std::array<IR::U32, 2> ReadU32Pair(const Decoder::Operand& operand);
+	IR::U64                ReadU64(const Decoder::Operand& operand);
+	IR::F32 ReadF16LaneAsF32(const Decoder::Operand& operand, bool high_lane, bool packed = false);
+	IR::F32 ReadF16AsF32(const Decoder::Operand& operand);
+	IR::F32 ReadMixF32(const Decoder::Operand& operand);
+	IR::U32 ReadU16LaneRaw(const Decoder::Operand& operand, bool high_lane);
+	IR::U32 ReadU16LaneAsU32(const Decoder::Operand& operand, bool high_lane, bool sign_extend);
+	IR::U32 ReadU16AsU32(const Decoder::Operand& operand, bool sign_extend);
+	IR::U32 ReadF16LaneBits(const Decoder::Operand& operand, bool high_lane);
 	std::array<IR::U32, 2> ExtractU64(IR::U64 value);
-	void    WriteU32Pair(const IR::Operand& operand, const std::array<IR::U32, 2>& value);
-	IR::U1  ReadCondition(const IR::Operand& operand);
-	IR::U32 ConditionBit(const IR::Operand& operand);
-	IR::U1  ReadMask(const IR::Operand& operand);
-	IR::U1  ReadMaskValid(const IR::Operand& operand);
-	void    WriteMask(const IR::Operand& operand, IR::U1 value);
-	void    WriteMask64(const IR::Operand& operand, IR::U1 value);
-	void    WriteCompareResult(const IR::Operand& operand, IR::U1 value);
+	void    WriteU32Pair(const Decoder::Operand& operand, const std::array<IR::U32, 2>& value);
+	IR::U1  ReadCondition(const Decoder::Operand& operand);
+	IR::U32 ConditionBit(const Decoder::Operand& operand);
+	IR::U1  ReadMask(const Decoder::Operand& operand);
+	IR::U1  ReadMaskValid(const Decoder::Operand& operand);
+	void    WriteMask(const Decoder::Operand& operand, IR::U1 value);
+	void    WriteMask64(const Decoder::Operand& operand, IR::U1 value);
+	void    WriteCompareResult(const Decoder::Operand& operand, IR::U1 value);
 
-	IR::MemoryFlags AddMemoryInfo(const IR::Instruction& inst);
-	IR::ExportFlags AddExportInfo(const IR::Instruction& inst);
+	IR::MemoryFlags AddMemoryInfo(const IR::MemoryInfo& memory, uint32_t pc);
+	IR::ExportFlags AddExportInfo(const Decoder::Instruction& inst);
 	struct AddressOperands {
 		IR::Value resource;
 		IR::Value low;
@@ -67,31 +68,47 @@ private:
 		IR::U32 offset;
 		IR::U32 soffset;
 	};
-	AddressOperands ReadAddressOperands(const IR::Instruction& inst, uint32_t first_source);
+	AddressOperands ReadAddressOperands(const Decoder::Instruction& inst, uint32_t first_source);
 	IR::U32         GetResourceDword(uint32_t index, uint32_t dword);
 	IR::Value       GetBufferResource(const IR::MemoryInfo& memory);
 	IR::Value       GetAddressResource(IR::Value low, IR::Value high);
 	IR::Value       GetScalarAddressResource(uint32_t base);
 	IR::Value       GetImageResource(const IR::MemoryInfo& memory);
 	IR::Value       GetSamplerResource(const IR::MemoryInfo& memory);
-	IR::Value       MakeImageAddress(const IR::Instruction& inst, const IR::Operand& base);
-	IR::Value       ConstructU32x4(const IR::Operand& base, uint32_t count);
-	void WriteImageComponents(const IR::Operand& dst, IR::Value value, const IR::MemoryInfo& memory,
-	                          uint32_t component_limit);
-	IR::ValueOpcode ImageAtomicOpcode(IR::Opcode opcode);
-	BufferAddress   ReadBufferAddress(const IR::Instruction& inst, uint32_t source_offset);
-	IR::U32         WidenSubdword(IR::Value value, uint32_t bits, bool sign);
-	IR::Value       NarrowSubdword(IR::U32 value, uint32_t bits);
-	IR::ValueOpcode BufferAtomicOpcode(IR::Opcode opcode);
-	IR::ValueOpcode SharedAtomicOpcode(IR::Opcode opcode);
-	bool            TranslateScalarMemory(const IR::Instruction& inst);
-	bool            TranslateBufferLoad(const IR::Instruction& inst);
-	bool            TranslateBufferStore(const IR::Instruction& inst);
-	bool            TranslateAtomicMemory(const IR::Instruction& inst);
-	bool            TranslateFlatLoad(const IR::Instruction& inst);
-	bool            TranslateFlatStore(const IR::Instruction& inst);
-	bool            TranslateImageMemory(const IR::Instruction& inst);
-	bool            TranslateSharedMemory(const IR::Instruction& inst);
+	IR::Value     MakeImageAddress(const Decoder::Instruction& inst, const Decoder::Operand& base);
+	IR::Value     ConstructU32x4(const Decoder::Operand& base, uint32_t count);
+	void          WriteImageComponents(const Decoder::Operand& dst, IR::Value value,
+	                                   const IR::MemoryInfo& memory, uint32_t component_limit);
+	BufferAddress ReadBufferAddress(const Decoder::Instruction& inst, uint32_t source_offset);
+	IR::U32       WidenSubdword(IR::Value value, uint32_t bits, bool sign);
+	IR::Value     NarrowSubdword(IR::U32 value, uint32_t bits);
+	bool          S_LOAD(const Decoder::Instruction& inst, bool raw);
+	bool          BUFFER_LOAD(const Decoder::Instruction& inst);
+	bool          BUFFER_STORE(const Decoder::Instruction& inst);
+	bool          BUFFER_ATOMIC(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool          IMAGE_ATOMIC(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool DS_ATOMIC(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool returns_value);
+	bool FLAT_LOAD(const Decoder::Instruction& inst);
+	bool FLAT_STORE(const Decoder::Instruction& inst);
+	bool IMAGE_GET_RESINFO(const Decoder::Instruction& inst);
+	bool IMAGE_GET_LOD(const Decoder::Instruction& inst);
+	bool IMAGE_LOAD(const Decoder::Instruction& inst);
+	bool IMAGE_STORE(const Decoder::Instruction& inst);
+	bool IMAGE_SAMPLE(const Decoder::Instruction& inst);
+	bool IMAGE_GATHER(const Decoder::Instruction& inst);
+	IR::Value LoadSharedU32(uint32_t width, IR::U32 address, const IR::MemoryInfo& memory,
+	                        uint32_t pc);
+	IR::Value ExtractSharedU32(IR::Value value, uint32_t width, uint32_t index);
+	void WriteSharedU32(uint32_t width, IR::U32 address, const std::array<IR::Value, 4>& values,
+	                    const IR::MemoryInfo& memory, uint32_t pc);
+	bool DS_READ(const Decoder::Instruction& inst);
+	bool DS_READ2(const Decoder::Instruction& inst);
+	bool DS_WRITE(const Decoder::Instruction& inst);
+	bool DS_WRITE2(const Decoder::Instruction& inst);
+	bool DS_MINMAX_F32(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool DS_APPEND_CONSUME(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool DS_ADDTID(const Decoder::Instruction& inst, bool write);
+	bool DS_SWIZZLE_B32(const Decoder::Instruction& inst);
 
 	IR::F32 SelectF32(IR::U1 condition, IR::F32 true_value, IR::F32 false_value);
 	IR::U32 ConvertF32ToU32Saturated(IR::F32 value, float upper_bound, float safe_upper,
@@ -100,35 +117,143 @@ private:
 	                                 float safe_upper, uint32_t lower_result,
 	                                 uint32_t upper_result);
 	IR::U32 PackU16Lanes(IR::U32 low, IR::U32 high);
+	void EmitCompareResult(const Decoder::Instruction& inst, IR::U1 value, bool scalar, bool cmpx);
+	void EmitCompareConstant(const Decoder::Instruction& inst, bool value, bool scalar, bool cmpx);
+	void EmitIntegerCompare(const Decoder::Instruction& inst, IR::ValueOpcode opcode, IR::Type type,
+	                        bool scalar, bool cmpx);
+	void EmitInteger16Compare(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+	                          bool signed_value, bool cmpx);
+	void EmitFloatCompare(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool half,
+	                      bool cmpx);
+	void EmitFloatOrderedCompare(const Decoder::Instruction& inst, bool ordered);
+	void V_CMP_CLASS_F32(const Decoder::Instruction& inst);
+	void V_CVT_F32_UBYTE(const Decoder::Instruction& inst, uint32_t byte_index);
+	void V_CVT_F32_U32(const Decoder::Instruction& inst);
+	void V_CVT_F32_I32(const Decoder::Instruction& inst);
+	void V_CVT_U32_F32(const Decoder::Instruction& inst);
+	void V_CVT_I32_F32(const Decoder::Instruction& inst);
+	void V_CVT_F16_F32(const Decoder::Instruction& inst);
+	void V_CVT_F32_F16(const Decoder::Instruction& inst);
+	void V_CVT_F16_16(const Decoder::Instruction& inst, bool signed_value);
+	void V_CVT_16_F16(const Decoder::Instruction& inst, bool signed_value);
+	void V_CVT_RPI_I32_F32(const Decoder::Instruction& inst);
+	void V_CVT_FLR_I32_F32(const Decoder::Instruction& inst);
+	void V_FREXP_EXP_I32_F32(const Decoder::Instruction& inst);
+	void V_CVT_OFF_F32_I4(const Decoder::Instruction& inst);
+	void V_CVT_PKRTZ_F16_F32(const Decoder::Instruction& inst);
+	void V_CVT_PKNORM_F32(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	void V_CVT_PK_U8_F32(const Decoder::Instruction& inst);
+	void V_PACK_B32_F16(const Decoder::Instruction& inst);
+	bool PackedFloat16(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool accumulator,
+	                   bool quiet_snan);
+	bool Float16Unary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+	                  bool invalid_negative);
+	bool Float16Binary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool reverse);
+	bool Float16Ternary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool accumulator,
+	                    bool mix);
+	bool FloatUnary(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool FloatBinary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool reverse);
+	bool FloatTernary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool accumulator,
+	                  bool mix);
+	bool V_FREXP_MANT_F32(const Decoder::Instruction& inst);
+	bool V_DOT2C_F32_F16(const Decoder::Instruction& inst);
+	bool V_CUBEID_F32(const Decoder::Instruction& inst);
+	bool V_CUBESC_F32(const Decoder::Instruction& inst);
+	bool V_CUBETC_F32(const Decoder::Instruction& inst);
+	bool V_CUBEMA_F32(const Decoder::Instruction& inst);
+	bool FloatCube(const Decoder::Instruction& inst, uint32_t result_kind);
+	bool Integer16Shift(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool arithmetic);
+	bool Integer16Binary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool sign);
+	bool V_MED3_I16(const Decoder::Instruction& inst);
+	bool PackedInteger16Shift(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+	                          bool arithmetic);
+	bool PackedInteger16Binary(const Decoder::Instruction& inst, IR::ValueOpcode opcode);
+	bool PackedInteger16Mad(const Decoder::Instruction& inst, bool sign);
+	bool PackedInteger16MinMax(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool sign);
+	bool S_U64_MASK(const Decoder::Instruction& inst, IR::ValueOpcode logical_opcode,
+	                IR::ValueOpcode bit_opcode, bool negate_rhs, bool negate_result, bool unary);
+	IR::U1  U64MaskBinary(const Decoder::Instruction& inst, IR::ValueOpcode opcode, bool negate_rhs,
+	                      bool negate_result);
+	bool    SimpleInteger(const Decoder::Instruction& inst, IR::ValueOpcode opcode, IR::Type type,
+	                      bool reverse, bool mask_shift_count, bool update_scc);
+	bool    ComposedIntegerBinary(const Decoder::Instruction& inst, IR::ValueOpcode opcode,
+	                              bool negate_rhs, bool negate_result, bool update_scc);
+	bool    V_AND_OR_B32(const Decoder::Instruction& inst);
+	bool    V_OR3_B32(const Decoder::Instruction& inst);
+	bool    V_XOR3_B32(const Decoder::Instruction& inst);
+	bool    S_FF1_I32_B64(const Decoder::Instruction& inst);
+	bool    V_FFBH_32(const Decoder::Instruction& inst, bool sign);
+	bool    S_FLBIT_I32_B64(const Decoder::Instruction& inst);
+	bool    Integer24(const Decoder::Instruction& inst, bool sign, bool addend);
+	bool    V_MAD_U64_U32(const Decoder::Instruction& inst);
+	bool    V_SAD_U32(const Decoder::Instruction& inst);
+	bool    V_ADD3_U32(const Decoder::Instruction& inst);
+	bool    S_BITSET_B32(const Decoder::Instruction& inst, bool set);
+	bool    V_BCNT_U32_B32(const Decoder::Instruction& inst);
+	bool    V_MBCNT_U32_B32(const Decoder::Instruction& inst, bool low);
+	bool    S_BITREPLICATE_B64_B32(const Decoder::Instruction& inst);
+	bool    S_QUADMASK_B64(const Decoder::Instruction& inst);
+	bool    BFM_B32(const Decoder::Instruction& inst);
+	IR::U32 RightMask32(IR::U32 count);
+	IR::U64 RightMask64(IR::U32 count);
+	bool    S_BFM_B64(const Decoder::Instruction& inst);
+	bool    S_BFE_U32(const Decoder::Instruction& inst, bool sign);
+	bool    S_BFE_U64(const Decoder::Instruction& inst);
+	bool    V_BFE_U32(const Decoder::Instruction& inst, bool sign);
+	bool    V_BFI_B32(const Decoder::Instruction& inst);
+	bool    S_BITCMP_B32(const Decoder::Instruction& inst, bool expected);
+	bool    V_ALIGNBIT_B32(const Decoder::Instruction& inst);
+	bool    V_ALIGNBYTE_B32(const Decoder::Instruction& inst);
+	bool    V_LSHL_ADD_U32(const Decoder::Instruction& inst);
+	bool    V_ADD_LSHL_U32(const Decoder::Instruction& inst);
+	bool    V_XAD_U32(const Decoder::Instruction& inst);
+	bool    V_LSHL_OR_B32(const Decoder::Instruction& inst);
+	bool    V_CNDMASK_B32(const Decoder::Instruction& inst);
+	bool    PackB16(const Decoder::Instruction& inst, bool high0, bool high1);
 
-	bool   TranslateInstruction(const IR::Instruction& inst);
-	bool   TranslateStateOperation(const IR::Instruction& inst);
-	bool   TranslateControlOperation(const IR::Instruction& inst);
-	bool   TranslateMove(const IR::Instruction& inst);
-	bool   TranslateLaneOperation(const IR::Instruction& inst);
-	bool   TranslateAttributeOperation(const IR::Instruction& inst);
-	bool   TranslateMemoryOperation(const IR::Instruction& inst);
-	bool   TranslateIntegerCompare(const IR::Instruction& inst);
-	bool   TranslateInteger16Compare(const IR::Instruction& inst);
-	bool   TranslateFloatCompare(const IR::Instruction& inst);
-	bool   TranslateConversion(const IR::Instruction& inst);
-	bool   TranslateInteger16Operation(const IR::Instruction& inst);
-	bool   TranslatePackedInteger16(const IR::Instruction& inst);
-	bool   TranslatePackedFloat16(const IR::Instruction& inst);
-	bool   TranslateFloat16Operation(const IR::Instruction& inst);
-	bool   TranslateFloatOperation(const IR::Instruction& inst);
-	IR::U1 EvaluateU64Mask(const IR::Instruction& inst);
-	bool   TranslateU64MaskOperation(const IR::Instruction& inst);
-	bool   TranslateSimpleInteger(const IR::Instruction& inst);
-	bool   TranslateComposedInteger(const IR::Instruction& inst);
-	bool   TranslateExtendedInteger(const IR::Instruction& inst);
+	void S_SAVEEXEC(const Decoder::Instruction& inst, IR::ValueOpcode operation, bool negate_exec,
+	                bool negate_source, bool write_64);
+	void ADD_U32(const Decoder::Instruction& inst, bool vector, bool use_carry_in);
+	void SUB_U32(const Decoder::Instruction& inst, bool vector, bool reverse);
+	void SUBB_U32(const Decoder::Instruction& inst, bool vector);
+	void S_ADD_SUB_I32(const Decoder::Instruction& inst, bool subtract);
+	void S_LSHL_ADD_U32(const Decoder::Instruction& inst, uint32_t shift_amount);
+	void ScalarMinMax32(const Decoder::Instruction& inst, IR::ValueOpcode value_opcode,
+	                    IR::ValueOpcode compare_opcode);
+	void EmitControlNop();
+	void EmitWaitcnt();
+	void S_BARRIER();
+	void S_SENDMSG();
+	void S_TTRACEDATA();
+	void S_INST_PREFETCH();
+	bool S_GETPC_B64(const Decoder::Instruction& inst, std::string* error);
+	void S_CSELECT_B32(const Decoder::Instruction& inst);
+	void S_CSELECT_B64(const Decoder::Instruction& inst);
+	void MOV_B32(const Decoder::Instruction& inst, bool apply_float_modifiers);
+	void S_MOV_B64(const Decoder::Instruction& inst);
+	void S_WQM_B64(const Decoder::Instruction& inst);
+	bool V_MOVRELS_B32(const Decoder::Instruction& inst, std::string* error);
+	bool V_MOVRELD_B32(const Decoder::Instruction& inst, std::string* error);
+	void V_READFIRSTLANE_B32(const Decoder::Instruction& inst);
+	void V_READLANE_B32(const Decoder::Instruction& inst);
+	void V_WRITELANE_B32(const Decoder::Instruction& inst);
+	void V_PERMLANE16_B32(const Decoder::Instruction& inst, bool x16);
+	void V_INTERP_P1_F32();
+	void V_INTERP_P2_F32(const Decoder::Instruction& inst);
+	bool V_INTERP_MOV_F32(const Decoder::Instruction& inst, std::string* error);
+	bool EXP(const Decoder::Instruction& inst, std::string* error);
 
-	IR::ValueProgram& value_program;
-	IR::IREmitter     ir;
-	IR::Opcode        current_opcode       = IR::Opcode::ControlNop;
-	uint32_t          current_pc           = 0;
-	uint32_t          current_vector_limit = 1;
-	uint32_t          current_wave_size    = 64;
+	bool EmitScalar(const Decoder::Instruction& inst, std::string* error);
+	bool EmitVector(const Decoder::Instruction& inst, std::string* error);
+	bool EmitInterpolation(const Decoder::Instruction& inst, std::string* error);
+	bool EmitMemory(const Decoder::Instruction& inst, std::string* error);
+
+	IR::Program&    program;
+	IR::IREmitter   ir;
+	Decoder::Opcode current_opcode       = Decoder::Opcode::UNKNOWN;
+	uint32_t        current_pc           = 0;
+	uint32_t        current_vector_limit = 1;
+	uint32_t        current_wave_size    = 64;
 };
 
-} // namespace Libs::Graphics::ShaderRecompiler::Frontend::Detail
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/frontend/translate/Vector.cpp

@@ -0,0 +1,522 @@
+#include "graphics/shader/recompiler/frontend/translate/Translator.h"
+
+namespace Libs::Graphics::ShaderRecompiler::Frontend {
+
+bool Translator::EmitVector(const Decoder::Instruction& inst, std::string* error) {
+	using O = Decoder::Opcode;
+	switch (inst.opcode) {
+		case O::V_NOP: return true;
+		case O::V_ADD_I32: ADD_U32(inst, true, false); return true;
+		case O::V_ADDC_U32: ADD_U32(inst, true, true); return true;
+		case O::V_SUB_I32: SUB_U32(inst, true, false); return true;
+		case O::V_SUBREV_I32: SUB_U32(inst, true, true); return true;
+		case O::V_SUBREV_CO_CI_U32: SUBB_U32(inst, true); return true;
+
+		case O::V_MOV_B32: MOV_B32(inst, true); return true;
+		case O::V_MOVRELS_B32: return V_MOVRELS_B32(inst, error);
+		case O::V_MOVRELD_B32: return V_MOVRELD_B32(inst, error);
+		case O::V_READFIRSTLANE_B32: V_READFIRSTLANE_B32(inst); return true;
+		case O::V_READLANE_B32: V_READLANE_B32(inst); return true;
+		case O::V_WRITELANE_B32: V_WRITELANE_B32(inst); return true;
+		case O::V_PERMLANE16_B32: V_PERMLANE16_B32(inst, false); return true;
+		case O::V_PERMLANEX16_B32: V_PERMLANE16_B32(inst, true); return true;
+
+		case O::V_CMP_F_I32:
+		case O::V_CMP_F_U32: EmitCompareConstant(inst, false, false, false); return true;
+		case O::V_CMP_T_I32:
+		case O::V_CMP_T_U32: EmitCompareConstant(inst, true, false, false); return true;
+		case O::V_CMP_EQ_U32:
+		case O::V_CMP_EQ_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::IEqual32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_EQ_U32:
+		case O::V_CMPX_EQ_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::IEqual32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_NE_U32:
+		case O::V_CMP_NE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_NE_U32:
+		case O::V_CMPX_NE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_GT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThan32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_GT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThan32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_GE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThanEqual32, IR::Type::U32, false,
+			                   false);
+			return true;
+		case O::V_CMPX_GE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThanEqual32, IR::Type::U32, false,
+			                   true);
+			return true;
+		case O::V_CMP_LT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThan32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_LT_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThan32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_LE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThanEqual32, IR::Type::U32, false,
+			                   false);
+			return true;
+		case O::V_CMPX_LE_U32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::ULessThanEqual32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_GT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThan32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_GT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThan32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_GE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThanEqual32, IR::Type::U32, false,
+			                   false);
+			return true;
+		case O::V_CMPX_GE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SGreaterThanEqual32, IR::Type::U32, false,
+			                   true);
+			return true;
+		case O::V_CMP_LT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThan32, IR::Type::U32, false, false);
+			return true;
+		case O::V_CMPX_LT_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThan32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_LE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThanEqual32, IR::Type::U32, false,
+			                   false);
+			return true;
+		case O::V_CMPX_LE_I32:
+			EmitIntegerCompare(inst, IR::ValueOpcode::SLessThanEqual32, IR::Type::U32, false, true);
+			return true;
+		case O::V_CMP_EQ_I64:
+		case O::V_CMP_EQ_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::IEqual64, IR::Type::U64, false, false);
+			return true;
+		case O::V_CMP_GT_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::UGreaterThan64, IR::Type::U64, false, false);
+			return true;
+		case O::V_CMP_NE_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual64, IR::Type::U64, false, false);
+			return true;
+		case O::V_CMPX_NE_I64:
+		case O::V_CMPX_NE_U64:
+			EmitIntegerCompare(inst, IR::ValueOpcode::INotEqual64, IR::Type::U64, false, true);
+			return true;
+
+		case O::V_CMP_EQ_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::IEqual32, false, false);
+			return true;
+		case O::V_CMP_EQ_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::IEqual32, true, false);
+			return true;
+		case O::V_CMP_NE_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::INotEqual32, false, false);
+			return true;
+		case O::V_CMP_NE_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::INotEqual32, true, false);
+			return true;
+		case O::V_CMP_GT_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::UGreaterThan32, false, false);
+			return true;
+		case O::V_CMPX_GT_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::UGreaterThan32, false, true);
+			return true;
+		case O::V_CMP_GE_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::UGreaterThanEqual32, false, false);
+			return true;
+		case O::V_CMP_LT_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::ULessThan32, false, false);
+			return true;
+		case O::V_CMP_LE_U16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::ULessThanEqual32, false, false);
+			return true;
+		case O::V_CMP_GT_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::SGreaterThan32, true, false);
+			return true;
+		case O::V_CMP_GE_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::SGreaterThanEqual32, true, false);
+			return true;
+		case O::V_CMP_LT_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::SLessThan32, true, false);
+			return true;
+		case O::V_CMP_LE_I16:
+			EmitInteger16Compare(inst, IR::ValueOpcode::SLessThanEqual32, true, false);
+			return true;
+
+		case O::V_CMP_F_F32: EmitCompareConstant(inst, false, false, false); return true;
+		case O::V_CMP_TRU_F32: EmitCompareConstant(inst, true, false, false); return true;
+		case O::V_CMP_EQ_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdEqual32, false, false);
+			return true;
+		case O::V_CMPX_EQ_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdEqual32, false, true);
+			return true;
+		case O::V_CMP_LG_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdNotEqual32, false, false);
+			return true;
+		case O::V_CMPX_LG_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdNotEqual32, false, true);
+			return true;
+		case O::V_CMP_GT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThan32, false, false);
+			return true;
+		case O::V_CMPX_GT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThan32, false, true);
+			return true;
+		case O::V_CMP_GE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThanEqual32, false, false);
+			return true;
+		case O::V_CMPX_GE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThanEqual32, false, true);
+			return true;
+		case O::V_CMP_LT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThan32, false, false);
+			return true;
+		case O::V_CMPX_LT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThan32, false, true);
+			return true;
+		case O::V_CMP_LE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThanEqual32, false, false);
+			return true;
+		case O::V_CMPX_LE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThanEqual32, false, true);
+			return true;
+		case O::V_CMP_NLG_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordEqual32, false, false);
+			return true;
+		case O::V_CMPX_NLG_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordEqual32, false, true);
+			return true;
+		case O::V_CMP_NEQ_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordNotEqual32, false, false);
+			return true;
+		case O::V_CMPX_NEQ_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordNotEqual32, false, true);
+			return true;
+		case O::V_CMP_NLE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordGreaterThan32, false, false);
+			return true;
+		case O::V_CMPX_NLE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordGreaterThan32, false, true);
+			return true;
+		case O::V_CMP_NLT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordGreaterThanEqual32, false, false);
+			return true;
+		case O::V_CMPX_NLT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordGreaterThanEqual32, false, true);
+			return true;
+		case O::V_CMP_NGE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordLessThan32, false, false);
+			return true;
+		case O::V_CMPX_NGE_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordLessThan32, false, true);
+			return true;
+		case O::V_CMP_NGT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordLessThanEqual32, false, false);
+			return true;
+		case O::V_CMPX_NGT_F32:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordLessThanEqual32, false, true);
+			return true;
+		case O::V_CMP_EQ_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdEqual32, true, false);
+			return true;
+		case O::V_CMPX_EQ_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdEqual32, true, true);
+			return true;
+		case O::V_CMP_LG_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdNotEqual32, true, false);
+			return true;
+		case O::V_CMP_GT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThan32, true, false);
+			return true;
+		case O::V_CMPX_GT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThan32, true, true);
+			return true;
+		case O::V_CMP_GE_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThanEqual32, true, false);
+			return true;
+		case O::V_CMPX_GE_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdGreaterThanEqual32, true, true);
+			return true;
+		case O::V_CMP_LT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThan32, true, false);
+			return true;
+		case O::V_CMPX_LT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThan32, true, true);
+			return true;
+		case O::V_CMP_LE_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThanEqual32, true, false);
+			return true;
+		case O::V_CMPX_LE_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPOrdLessThanEqual32, true, true);
+			return true;
+		case O::V_CMPX_NGT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordLessThanEqual32, true, true);
+			return true;
+		case O::V_CMP_NEQ_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordNotEqual32, true, false);
+			return true;
+		case O::V_CMPX_NEQ_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordNotEqual32, true, true);
+			return true;
+		case O::V_CMPX_NLT_F16:
+			EmitFloatCompare(inst, IR::ValueOpcode::FPUnordGreaterThanEqual32, true, true);
+			return true;
+		case O::V_CMP_O_F32: EmitFloatOrderedCompare(inst, true); return true;
+		case O::V_CMP_U_F32: EmitFloatOrderedCompare(inst, false); return true;
+		case O::V_CMP_CLASS_F32: V_CMP_CLASS_F32(inst); return true;
+
+		case O::V_CVT_F32_UBYTE0: V_CVT_F32_UBYTE(inst, 0); return true;
+		case O::V_CVT_F32_UBYTE1: V_CVT_F32_UBYTE(inst, 1); return true;
+		case O::V_CVT_F32_UBYTE2: V_CVT_F32_UBYTE(inst, 2); return true;
+		case O::V_CVT_F32_UBYTE3: V_CVT_F32_UBYTE(inst, 3); return true;
+		case O::V_CVT_F32_U32: V_CVT_F32_U32(inst); return true;
+		case O::V_CVT_F32_I32: V_CVT_F32_I32(inst); return true;
+		case O::V_CVT_U32_F32: V_CVT_U32_F32(inst); return true;
+		case O::V_CVT_I32_F32: V_CVT_I32_F32(inst); return true;
+		case O::V_CVT_F16_F32: V_CVT_F16_F32(inst); return true;
+		case O::V_CVT_F32_F16: V_CVT_F32_F16(inst); return true;
+		case O::V_CVT_F16_U16: V_CVT_F16_16(inst, false); return true;
+		case O::V_CVT_F16_I16: V_CVT_F16_16(inst, true); return true;
+		case O::V_CVT_U16_F16: V_CVT_16_F16(inst, false); return true;
+		case O::V_CVT_I16_F16: V_CVT_16_F16(inst, true); return true;
+		case O::V_CVT_RPI_I32_F32: V_CVT_RPI_I32_F32(inst); return true;
+		case O::V_CVT_FLR_I32_F32: V_CVT_FLR_I32_F32(inst); return true;
+		case O::V_FREXP_EXP_I32_F32: V_FREXP_EXP_I32_F32(inst); return true;
+		case O::V_CVT_OFF_F32_I4: V_CVT_OFF_F32_I4(inst); return true;
+		case O::V_CVT_PKRTZ_F16_F32: V_CVT_PKRTZ_F16_F32(inst); return true;
+		case O::V_CVT_PKNORM_I16_F32:
+			V_CVT_PKNORM_F32(inst, IR::ValueOpcode::PackSnorm2x16);
+			return true;
+		case O::V_CVT_PKNORM_U16_F32:
+			V_CVT_PKNORM_F32(inst, IR::ValueOpcode::PackUnorm2x16);
+			return true;
+		case O::V_CVT_PK_U8_F32: V_CVT_PK_U8_F32(inst); return true;
+		case O::V_PACK_B32_F16: V_PACK_B32_F16(inst); return true;
+		case O::V_CVT_PK_U16_U32:
+		case O::V_CVT_PK_I16_I32: return PackB16(inst, false, false);
+
+		case O::V_LSHLREV_B16:
+			return Integer16Shift(inst, IR::ValueOpcode::ShiftLeftLogical32, false);
+		case O::V_LSHRREV_B16:
+			return Integer16Shift(inst, IR::ValueOpcode::ShiftRightLogical32, false);
+		case O::V_ASHRREV_I16:
+			return Integer16Shift(inst, IR::ValueOpcode::ShiftRightArithmetic32, true);
+		case O::V_ADD_NC_U16:
+		case O::V_ADD_NC_I16: return Integer16Binary(inst, IR::ValueOpcode::IAdd32, false);
+		case O::V_SUB_NC_U16:
+		case O::V_SUB_NC_I16: return Integer16Binary(inst, IR::ValueOpcode::ISub32, false);
+		case O::V_MED3_I16: return V_MED3_I16(inst);
+		case O::V_MIN_I16: return Integer16Binary(inst, IR::ValueOpcode::SMin32, true);
+		case O::V_MAX_I16: return Integer16Binary(inst, IR::ValueOpcode::SMax32, true);
+		case O::V_MIN_U16: return Integer16Binary(inst, IR::ValueOpcode::UMin32, false);
+		case O::V_MAX_U16: return Integer16Binary(inst, IR::ValueOpcode::UMax32, false);
+
+		case O::V_PK_LSHLREV_B16:
+			return PackedInteger16Shift(inst, IR::ValueOpcode::ShiftLeftLogical32, false);
+		case O::V_PK_LSHRREV_B16:
+			return PackedInteger16Shift(inst, IR::ValueOpcode::ShiftRightLogical32, false);
+		case O::V_PK_ASHRREV_I16:
+			return PackedInteger16Shift(inst, IR::ValueOpcode::ShiftRightArithmetic32, true);
+		case O::V_PK_MAD_I16: return PackedInteger16Mad(inst, true);
+		case O::V_PK_MAD_U16: return PackedInteger16Mad(inst, false);
+		case O::V_PK_MUL_LO_U16: return PackedInteger16Binary(inst, IR::ValueOpcode::IMul32);
+		case O::V_PK_ADD_I16:
+		case O::V_PK_ADD_U16: return PackedInteger16Binary(inst, IR::ValueOpcode::IAdd32);
+		case O::V_PK_SUB_I16:
+		case O::V_PK_SUB_U16: return PackedInteger16Binary(inst, IR::ValueOpcode::ISub32);
+		case O::V_PK_MAX_I16: return PackedInteger16MinMax(inst, IR::ValueOpcode::SMax32, true);
+		case O::V_PK_MIN_I16: return PackedInteger16MinMax(inst, IR::ValueOpcode::SMin32, true);
+		case O::V_PK_MAX_U16: return PackedInteger16MinMax(inst, IR::ValueOpcode::UMax32, false);
+		case O::V_PK_MIN_U16: return PackedInteger16MinMax(inst, IR::ValueOpcode::UMin32, false);
+
+		case O::V_PK_ADD_F16: return PackedFloat16(inst, IR::ValueOpcode::FPAdd32, false, false);
+		case O::V_PK_MUL_F16: return PackedFloat16(inst, IR::ValueOpcode::FPMul32, false, false);
+		case O::V_PK_MIN_F16: return PackedFloat16(inst, IR::ValueOpcode::FPMin32, false, true);
+		case O::V_PK_MAX_F16: return PackedFloat16(inst, IR::ValueOpcode::FPMax32, false, true);
+		case O::V_PK_FMA_F16: return PackedFloat16(inst, IR::ValueOpcode::FPFma32, false, false);
+		case O::V_PK_FMAC_F16: return PackedFloat16(inst, IR::ValueOpcode::FPFma32, true, false);
+
+		case O::V_ADD_F16: return Float16Binary(inst, IR::ValueOpcode::FPAdd32, false);
+		case O::V_SUB_F16: return Float16Binary(inst, IR::ValueOpcode::FPSub32, false);
+		case O::V_SUBREV_F16: return Float16Binary(inst, IR::ValueOpcode::FPSub32, true);
+		case O::V_MUL_F16: return Float16Binary(inst, IR::ValueOpcode::FPMul32, false);
+		case O::V_MIN_F16: return Float16Binary(inst, IR::ValueOpcode::FPMin32, false);
+		case O::V_MAX_F16: return Float16Binary(inst, IR::ValueOpcode::FPMax32, false);
+		case O::V_FMAC_F16: return Float16Ternary(inst, IR::ValueOpcode::FPFma32, true, false);
+		case O::V_FMAMK_F16:
+		case O::V_FMAAK_F16:
+		case O::V_FMA_F16: return Float16Ternary(inst, IR::ValueOpcode::FPFma32, false, false);
+		case O::V_MAD_MIXLO_F16:
+		case O::V_MAD_MIXHI_F16: return Float16Ternary(inst, IR::ValueOpcode::FPFma32, false, true);
+		case O::V_RCP_F16: return Float16Unary(inst, IR::ValueOpcode::FPRecip32, false);
+		case O::V_SQRT_F16: return Float16Unary(inst, IR::ValueOpcode::FPSqrt, true);
+		case O::V_RSQ_F16: return Float16Unary(inst, IR::ValueOpcode::FPRecipSqrt32, true);
+		case O::V_LOG_F16: return Float16Unary(inst, IR::ValueOpcode::FPLog2, true);
+		case O::V_EXP_F16: return Float16Unary(inst, IR::ValueOpcode::FPExp2, false);
+		case O::V_FLOOR_F16: return Float16Unary(inst, IR::ValueOpcode::FPFloor32, false);
+		case O::V_CEIL_F16: return Float16Unary(inst, IR::ValueOpcode::FPCeil32, false);
+		case O::V_TRUNC_F16: return Float16Unary(inst, IR::ValueOpcode::FPTrunc32, false);
+		case O::V_RNDNE_F16: return Float16Unary(inst, IR::ValueOpcode::FPRoundEven32, false);
+		case O::V_MIN3_F16: return Float16Ternary(inst, IR::ValueOpcode::FPMinTri32, false, false);
+		case O::V_MAX3_F16: return Float16Ternary(inst, IR::ValueOpcode::FPMaxTri32, false, false);
+		case O::V_MED3_F16: return Float16Ternary(inst, IR::ValueOpcode::FPMedTri32, false, false);
+
+		case O::V_FREXP_MANT_F32: return V_FREXP_MANT_F32(inst);
+		case O::V_RCP_F32: return FloatUnary(inst, IR::ValueOpcode::FPRecip32);
+		case O::V_RCP_IFLAG_F32: return FloatUnary(inst, IR::ValueOpcode::FPRecipIFlag32);
+		case O::V_FRACT_F32: return FloatUnary(inst, IR::ValueOpcode::FPFract32);
+		case O::V_TRUNC_F32: return FloatUnary(inst, IR::ValueOpcode::FPTrunc32);
+		case O::V_CEIL_F32: return FloatUnary(inst, IR::ValueOpcode::FPCeil32);
+		case O::V_RNDNE_F32: return FloatUnary(inst, IR::ValueOpcode::FPRoundEven32);
+		case O::V_FLOOR_F32: return FloatUnary(inst, IR::ValueOpcode::FPFloor32);
+		case O::V_EXP_F32: return FloatUnary(inst, IR::ValueOpcode::FPExp2);
+		case O::V_LOG_F32: return FloatUnary(inst, IR::ValueOpcode::FPLog2);
+		case O::V_RSQ_F32: return FloatUnary(inst, IR::ValueOpcode::FPRecipSqrt32);
+		case O::V_SQRT_F32: return FloatUnary(inst, IR::ValueOpcode::FPSqrt);
+		case O::V_SIN_F32: return FloatUnary(inst, IR::ValueOpcode::FPSin);
+		case O::V_COS_F32: return FloatUnary(inst, IR::ValueOpcode::FPCos);
+		case O::V_ADD_F32: return FloatBinary(inst, IR::ValueOpcode::FPAdd32, false);
+		case O::V_SUB_F32: return FloatBinary(inst, IR::ValueOpcode::FPSub32, false);
+		case O::V_SUBREV_F32: return FloatBinary(inst, IR::ValueOpcode::FPSub32, true);
+		case O::V_MUL_F32: return FloatBinary(inst, IR::ValueOpcode::FPMul32, false);
+		case O::V_MIN_F32: return FloatBinary(inst, IR::ValueOpcode::FPMin32, false);
+		case O::V_MAX_F32: return FloatBinary(inst, IR::ValueOpcode::FPMax32, false);
+		case O::V_LDEXP_F32: return FloatBinary(inst, IR::ValueOpcode::FPLdexp, false);
+		case O::V_MAC_F32: return FloatTernary(inst, IR::ValueOpcode::FPFma32, true, true);
+		case O::V_MADMK_F32:
+		case O::V_MADAK_F32:
+		case O::V_MAD_F32:
+		case O::V_FMA_F32: return FloatTernary(inst, IR::ValueOpcode::FPFma32, false, true);
+		case O::V_MIN3_F32: return FloatTernary(inst, IR::ValueOpcode::FPMinTri32, false, false);
+		case O::V_MAX3_F32: return FloatTernary(inst, IR::ValueOpcode::FPMaxTri32, false, false);
+		case O::V_MED3_F32: return FloatTernary(inst, IR::ValueOpcode::FPMedTri32, false, false);
+		case O::V_DOT2C_F32_F16: return V_DOT2C_F32_F16(inst);
+		case O::V_CUBEID_F32: return V_CUBEID_F32(inst);
+		case O::V_CUBESC_F32: return V_CUBESC_F32(inst);
+		case O::V_CUBETC_F32: return V_CUBETC_F32(inst);
+		case O::V_CUBEMA_F32: return V_CUBEMA_F32(inst);
+
+		case O::V_MUL_LO_U32:
+		case O::V_MUL_LO_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::IMul32, IR::Type::U32, false, false, false);
+		case O::V_MUL_HI_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMulHi, IR::Type::U32, false, false, false);
+		case O::V_MUL_HI_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMulHi, IR::Type::U32, false, false, false);
+		case O::V_ADD_NC_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::IAdd32, IR::Type::U32, false, false, false);
+		case O::V_SUB_NC_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::ISub32, IR::Type::U32, false, false, false);
+		case O::V_SUBREV_NC_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::ISub32, IR::Type::U32, true, false, false);
+		case O::V_MIN_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMin32, IR::Type::U32, false, false, false);
+		case O::V_MAX_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMax32, IR::Type::U32, false, false, false);
+		case O::V_MIN_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMin32, IR::Type::U32, false, false, false);
+		case O::V_MAX_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMax32, IR::Type::U32, false, false, false);
+		case O::V_MIN3_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMinTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_MAX3_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMaxTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_MED3_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::SMedTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_MIN3_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMinTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_MAX3_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMaxTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_MED3_U32:
+			return SimpleInteger(inst, IR::ValueOpcode::UMedTri32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_AND_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseAnd32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_OR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseOr32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_XOR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseXor32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_NOT_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitwiseNot32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_BFREV_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::BitReverse32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_FFBL_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::FindILsb32, IR::Type::U32, false, false,
+			                     false);
+		case O::V_LSHL_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftLeftLogical32, IR::Type::U32, false,
+			                     true, false);
+		case O::V_LSHLREV_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftLeftLogical32, IR::Type::U32, true,
+			                     true, false);
+		case O::V_LSHR_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightLogical32, IR::Type::U32, false,
+			                     true, false);
+		case O::V_LSHRREV_B32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightLogical32, IR::Type::U32, true,
+			                     true, false);
+		case O::V_ASHR_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightArithmetic32, IR::Type::U32,
+			                     false, true, false);
+		case O::V_ASHRREV_I32:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightArithmetic32, IR::Type::U32, true,
+			                     true, false);
+		case O::V_LSHLREV_B64:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftLeftLogical64, IR::Type::U64, true,
+			                     false, false);
+		case O::V_LSHRREV_B64:
+			return SimpleInteger(inst, IR::ValueOpcode::ShiftRightLogical64, IR::Type::U64, true,
+			                     false, false);
+
+		case O::V_XNOR_B32:
+			return ComposedIntegerBinary(inst, IR::ValueOpcode::BitwiseXor32, false, true, false);
+		case O::V_AND_OR_B32: return V_AND_OR_B32(inst);
+		case O::V_OR3_B32: return V_OR3_B32(inst);
+		case O::V_XOR3_B32: return V_XOR3_B32(inst);
+		case O::V_FFBH_U32: return V_FFBH_32(inst, false);
+		case O::V_FFBH_I32: return V_FFBH_32(inst, true);
+
+		case O::V_MAD_I32_I24: return Integer24(inst, true, true);
+		case O::V_MAD_U32_U24: return Integer24(inst, false, true);
+		case O::V_MUL_I32_I24: return Integer24(inst, true, false);
+		case O::V_MUL_U32_U24: return Integer24(inst, false, false);
+		case O::V_MAD_U64_U32: return V_MAD_U64_U32(inst);
+		case O::V_SAD_U32: return V_SAD_U32(inst);
+		case O::V_ADD3_U32: return V_ADD3_U32(inst);
+		case O::V_BCNT_U32_B32: return V_BCNT_U32_B32(inst);
+		case O::V_MBCNT_LO_U32_B32: return V_MBCNT_U32_B32(inst, true);
+		case O::V_MBCNT_HI_U32_B32: return V_MBCNT_U32_B32(inst, false);
+		case O::V_BFM_B32: return BFM_B32(inst);
+		case O::V_BFE_U32: return V_BFE_U32(inst, false);
+		case O::V_BFE_I32: return V_BFE_U32(inst, true);
+		case O::V_BFI_B32: return V_BFI_B32(inst);
+		case O::V_ALIGNBIT_B32: return V_ALIGNBIT_B32(inst);
+		case O::V_ALIGNBYTE_B32: return V_ALIGNBYTE_B32(inst);
+		case O::V_LSHL_ADD_U32: return V_LSHL_ADD_U32(inst);
+		case O::V_ADD_LSHL_U32: return V_ADD_LSHL_U32(inst);
+		case O::V_XAD_U32: return V_XAD_U32(inst);
+		case O::V_LSHL_OR_B32: return V_LSHL_OR_B32(inst);
+		case O::V_CNDMASK_B32: return V_CNDMASK_B32(inst);
+		default: return false;
+	}
+}
+
+} // namespace Libs::Graphics::ShaderRecompiler::Frontend

src/graphics/shader/recompiler/ir/Program.cpp

@@ -1,11 +1,13 @@
-#include "graphics/shader/recompiler/ir/ValueProgram.h"
+#include "graphics/shader/recompiler/ir/ShaderIR.h"
 
 #include "graphics/shader/shader.h"
 
 #include <fmt/format.h>
 #include <map>
+#include <new>
 #include <unordered_map>
 #include <unordered_set>
+#include <utility>
 
 namespace Libs::Graphics::ShaderRecompiler::IR {
 
@@ -50,7 +52,7 @@ bool IsRuntimeRead(ValueOpcode opcode) {
 	       AddressOpcodeInfoOf(opcode).access == AddressAccess::Read;
 }
 
-bool EquivalentValue(const ValueProgram& program, Value left, Value right,
+bool EquivalentValue(const Program& program, Value left, Value right,
                      std::vector<std::pair<const Inst*, const Inst*>>& visited) {
 	left  = left.Resolve();
 	right = right.Resolve();
@@ -93,7 +95,7 @@ bool EquivalentValue(const ValueProgram& program, Value left, Value right,
 
 } // namespace
 
-ValueProgram::~ValueProgram() {
+Program::~Program() {
 	// Values may cross block boundaries. Detach all arguments before any block starts destroying
 	// its instruction storage so reverse-use links always point to live definitions.
 	for (auto* block: blocks) {
@@ -103,12 +105,20 @@ ValueProgram::~ValueProgram() {
 	}
 }
 
-bool EquivalentValue(const ValueProgram& program, Value left, Value right) {
+Program& Program::operator=(Program&& other) noexcept {
+	if (this != &other) {
+		this->~Program();
+		new (this) Program(std::move(other));
+	}
+	return *this;
+}
+
+bool EquivalentValue(const Program& program, Value left, Value right) {
 	std::vector<std::pair<const Inst*, const Inst*>> visited;
 	return EquivalentValue(program, left, right, visited);
 }
 
-Value ResolveInvariantPhi(const ValueProgram& program, Value value) {
+Value ResolveInvariantPhi(const Program& program, Value value) {
 	value            = value.Resolve();
 	const auto* root = value.TryInstruction();
 	if (root == nullptr || root->GetOpcode() != ValueOpcode::Phi) {
@@ -139,7 +149,7 @@ Value ResolveInvariantPhi(const ValueProgram& program, Value value) {
 	return invariant;
 }
 
-bool ValidateValueProgram(const ValueProgram& program, bool require_ssa, std::string* error) {
+bool ValidateProgram(const Program& program, bool require_ssa, std::string* error) {
 	if (program.blocks.size() != program.block_info.size() ||
 	    program.blocks.size() != program.block_storage.size()) {
 		return Fail(error, "value IR block storage is inconsistent");
@@ -591,14 +601,14 @@ bool ValidateValueProgram(const ValueProgram& program, bool require_ssa, std::st
 	return true;
 }
 
-void ResolveControlFlowIdentities(ValueProgram& program) {
+void ResolveControlFlowIdentities(Program& program) {
 	for (auto& info: program.block_info) {
 		info.condition       = info.condition.Resolve();
 		info.indirect_target = info.indirect_target.Resolve();
 	}
 }
 
-std::string ValueProgramToString(const ValueProgram& program) {
+std::string ProgramToString(const Program& program) {
 	std::map<const Inst*, size_t> ids;
 	size_t                        next_id = 1;
 	for (const auto* block: program.blocks) {

src/graphics/shader/recompiler/ir/ShaderIR.h

@@ -6,7 +6,7 @@
 #include "graphics/guest_gpu/gpu_defs.h"
 #include "graphics/shader/recompiler/frontend/cfg/ShaderCFG.h"
 #include "graphics/shader/recompiler/frontend/decode/ShaderDecoder.h"
-#include "graphics/shader/recompiler/ir/opcodes/Opcodes.h"
+#include "graphics/shader/recompiler/ir/Block.h"
 #include "graphics/shader/recompiler/ir/opcodes/ValueOpcodes.h"
 #include "graphics/shader/shader.h"
 
@@ -19,47 +19,6 @@
 
 namespace Libs::Graphics::ShaderRecompiler::IR {
 
-struct ValueProgram;
-
-enum class RegisterFile { Scalar, Vector, Vcc, Exec, Scc, M0 };
-
-struct Register {
-	RegisterFile file  = RegisterFile::Scalar;
-	uint32_t     index = 0;
-
-	bool operator==(const Register& other) const {
-		return file == other.file && index == other.index;
-	}
-};
-
-enum class OperandKind { Register, ImmediateU32, PcRelativeU32, PcRelativeHighU32, Null };
-
-struct Operand {
-	OperandKind kind = OperandKind::Null;
-	Register    reg;
-	uint32_t    imm                = 0;
-	bool        float_inline       = false;
-	bool        sext_64            = false;
-	uint32_t    sdwa_sel           = 6;
-	uint32_t    sdwa_dst_unused    = 2;
-	uint32_t    omod               = 0;
-	bool        sdwa_sext          = false;
-	bool        op_sel             = false;
-	bool        op_sel_hi          = false;
-	bool        negate             = false;
-	bool        negate_hi          = false;
-	bool        absolute           = false;
-	bool        clamp              = false;
-	uint32_t    dpp_ctrl           = 0;
-	uint32_t    dpp_row_mask       = 0xf;
-	uint32_t    dpp_bank_mask      = 0xf;
-	bool        dpp_fetch_inactive = false;
-	bool        dpp_bound_ctrl     = false;
-	bool        dpp                = false;
-
-	bool operator==(const Operand& other) const = default;
-};
-
 enum class ResourceKind {
 	None,
 	ScalarBuffer,
@@ -144,44 +103,6 @@ struct ExportInfo {
 	bool operator==(const ExportInfo& other) const = default;
 };
 
-struct InputInfo {
-	uint32_t attr               = 0;
-	uint32_t chan               = 0;
-	uint32_t component_count    = 1;
-	uint32_t interpolation_mode = 3;
-
-	bool operator==(const InputInfo& other) const = default;
-};
-
-enum class SaveexecMode { And, Orn2, Andn1 };
-
-struct Instruction {
-	uint32_t     pc = 0;
-	Opcode       op = Opcode::MoveU32;
-	Operand      dst;
-	Operand      dst2;
-	Operand      src[4];
-	uint32_t     src_count = 0;
-	MemoryInfo   memory;
-	ExportInfo   export_info;
-	InputInfo    input_info;
-	SaveexecMode saveexec_mode = SaveexecMode::And;
-
-	bool operator==(const Instruction& other) const = default;
-};
-
-struct BasicBlock {
-	uint32_t                 id         = 0;
-	uint32_t                 start_pc   = 0;
-	uint32_t                 end_pc     = 0;
-	uint32_t                 inst_begin = 0;
-	uint32_t                 inst_end   = 0;
-	std::vector<uint32_t>    predecessors;
-	std::vector<uint32_t>    successors;
-	CFG::Terminator          terminator;
-	std::vector<Instruction> instructions;
-};
-
 struct DescriptorValue {
 	std::array<uint32_t, 8> dwords      = {};
 	uint32_t                dword_count = 0;
@@ -255,20 +176,6 @@ struct SampledResourcePair {
 	bool operator==(const SampledResourcePair& other) const = default;
 };
 
-struct AddressResource {
-	uint32_t     source           = UINT32_MAX;
-	uint32_t     first_use_pc     = 0;
-	ResourceKind kind             = ResourceKind::Flat;
-	int32_t      min_offset       = 0;
-	uint64_t     specialized_base = 0;
-	bool         unbased          = true;
-	bool         read             = false;
-	bool         written          = false;
-	bool         atomic           = false;
-
-	bool operator==(const AddressResource& other) const = default;
-};
-
 enum class StageInputKind {
 	VertexIndex,
 	InstanceIndex,
@@ -400,7 +307,8 @@ enum class DescriptorBindingKind {
 	StorageAtomic3D,
 	Samplers,
 	Gds,
-	AddressMemory,
+	BdaPagetable,
+	FaultBuffer,
 	FlattenedSrt,
 	UserData,
 	Count,
@@ -510,20 +418,20 @@ struct BindingLayout {
 
 struct ShaderInfo {
 	static constexpr uint32_t MaxBuffers      = 32;
-	static constexpr uint32_t MaxAddresses    = 32;
 	static constexpr uint32_t MaxImages       = 32;
 	static constexpr uint32_t MaxSamplers     = 32;
 	static constexpr uint32_t MaxSampledPairs = 64;
 
 	std::vector<BufferResource>      buffers;
-	std::vector<AddressResource>     addresses;
 	std::vector<ImageResource>       images;
 	std::vector<SamplerResource>     samplers;
 	std::vector<SampledResourcePair> sampled_pairs;
 	std::vector<StageInput>          inputs;
 	std::vector<StageOutput>         outputs;
+	std::array<uint8_t, 32>          vertex_fetch_components {};
 	int32_t                          vertex_offset_sgpr = -1;
 	bool                             has_bitwise_xor    = false;
+	bool                             uses_dma           = false;
 
 	bool operator==(const ShaderInfo& other) const = default;
 };
@@ -539,7 +447,50 @@ struct SpirvRequirements {
 	bool pixel_valid_mask             = false;
 };
 
+struct BlockInfo {
+	uint32_t        id       = 0;
+	uint32_t        start_pc = 0;
+	uint32_t        end_pc   = 0;
+	CFG::Terminator terminator;
+	Value           condition;
+	Value           indirect_target;
+};
+
+struct DescriptorSource {
+	struct IndirectImage {
+		uint32_t material_source = 0;
+		uint32_t heap_source     = 0;
+		uint32_t selector_stride = 0;
+		uint32_t selector_offset = 0;
+		uint32_t key_arg         = 0;
+
+		bool operator==(const IndirectImage& other) const = default;
+	};
+
+	std::array<Value, 8>         dwords {};
+	uint32_t                     dword_count = 0;
+	std::optional<IndirectImage> indirect_image;
+
+	bool operator==(const DescriptorSource& other) const = default;
+};
+
+struct SrtRead {
+	Value    value;
+	uint32_t flat_offset = 0;
+	uint32_t use_pc      = 0;
+
+	bool operator==(const SrtRead& other) const = default;
+};
+
 struct Program {
+	Program() = default;
+	~Program();
+
+	Program(const Program&)            = delete;
+	Program& operator=(const Program&) = delete;
+	Program(Program&&) noexcept         = default;
+	Program& operator=(Program&& other) noexcept;
+
 	ShaderType                    stage               = ShaderType::Unknown;
 	uint64_t                      shader_hash         = 0;
 	uint32_t                      wave_size           = 64;
@@ -549,8 +500,17 @@ struct Program {
 	bool                          dispatcher_fallback = false;
 	CFG::FailureKind              cfg_failure_kind    = CFG::FailureKind::None;
 	std::string                   fallback_reason;
-	std::vector<BasicBlock>       blocks;
-	std::shared_ptr<ValueProgram> values;
+	std::vector<std::unique_ptr<Block>> block_storage;
+	BlockList                           blocks;
+	std::vector<BlockInfo>              block_info;
+	// Decoded MIMG/VMEM metadata carries details such as RDNA2 NSA address registers and
+	// storage-image swizzles. Typed memory instructions carry a dense index into these shader-local
+	// tables until those fields are consumed by emission.
+	std::vector<MemoryInfo>       memory_info;
+	std::vector<ExportInfo>       export_info;
+	std::vector<DescriptorSource> descriptor_sources;
+	std::vector<SrtRead>          srt_reads;
+	std::vector<Value>            dynamic_reads;
 	bool                          srt_plan_complete = false;
 	ShaderInfo                    info;
 	bool                          resource_tracking_complete = false;
@@ -561,15 +521,13 @@ struct Program {
 	std::optional<SpirvRequirements> spirv_requirements;
 };
 
-bool LowerProgram(const Decoder::Program& decoded, const CFG::Graph& cfg, ShaderType stage,
-                  uint32_t wave_size, Program& program, std::string* error);
-
-std::string RegisterToString(Register reg);
-std::string OperandToString(const Operand& operand);
-std::string ExportTargetKindToString(ExportTargetKind kind);
-std::string InstructionToString(const Instruction& inst);
 std::string ProgramToString(const Program& program);
 
+bool  ValidateProgram(const Program& program, bool require_ssa, std::string* error);
+void  ResolveControlFlowIdentities(Program& program);
+bool  EquivalentValue(const Program& program, Value left, Value right);
+Value ResolveInvariantPhi(const Program& program, Value value);
+
 } // namespace Libs::Graphics::ShaderRecompiler::IR
 
 #endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_SHADER_RECOMPILER_SHADERIR_H_ */

src/graphics/shader/recompiler/ir/ShaderIRInternal.h

@@ -1,51 +0,0 @@
-#ifndef EMULATOR_SRC_GRAPHICS_SHADER_RECOMPILER_SHADERIR_SHADERIRINTERNAL_H_
-#define EMULATOR_SRC_GRAPHICS_SHADER_RECOMPILER_SHADERIR_SHADERIRINTERNAL_H_
-
-#include "graphics/shader/recompiler/ir/ShaderIR.h"
-
-#include <optional>
-
-namespace Libs::Graphics::ShaderRecompiler::IR {
-
-void SetError(std::string* error, const char* message);
-
-std::optional<Opcode> LookupIrOpcode(Decoder::Opcode opcode);
-bool                  IsReversedBinary(Decoder::Opcode opcode);
-bool                  IsVectorCarryOutOpcode(Decoder::Opcode opcode);
-bool                  ScalarResultWritesSccNonZero(Decoder::Opcode opcode);
-bool                  ScalarResultIs64Bit(Decoder::Opcode opcode);
-
-Operand  MakeSccOperand();
-Operand  MakeImmediateU32(uint32_t value);
-Operand  MakePcRelativeU32(uint32_t value);
-Operand  MakePcRelativeHighU32(uint32_t value);
-bool     AppendScalarResultSccNonZero(const Decoder::Instruction& decoded, BasicBlock& block,
-                                      std::string* error);
-uint32_t VectorByteConvertIndex(Decoder::Opcode opcode);
-void     CopyOperandModifiers(const Decoder::Operand& decoded, Operand& operand);
-bool LowerRegisterOperand(const Decoder::Operand& decoded, Operand& operand, std::string* error);
-bool LowerSourceOperand(const Decoder::Operand& decoded, Operand& operand, std::string* error);
-void ApplyDppDestinationMask(const Decoder::Instruction& decoded, Operand& dst);
-uint32_t         ResourceIndexFromOperand(const Decoder::Operand& operand);
-MemoryInfo       MemoryInfoFromDecoded(const Decoder::Instruction& decoded, ResourceKind kind);
-uint32_t         RawScalarLoadBase(const Decoder::Operand& operand);
-void             ClearRegisterOffsetModifiers(Decoder::Operand& operand);
-bool             TryGetEncodedScalarCode(const Decoder::Operand& operand, uint32_t& code);
-bool             TryOffsetScalarDestination(const Decoder::Operand& operand, uint32_t index,
-                                            Decoder::Operand& result);
-Decoder::Operand OffsetDecodedRegister(const Decoder::Operand& operand, uint32_t index);
-MemoryInfo       OffsetMemoryInfo(const Decoder::Instruction& decoded, ResourceKind kind,
-                                  uint32_t dword_index);
-MemoryInfo       ByteOffsetMemoryInfo(const Decoder::Instruction& decoded, ResourceKind kind,
-                                      uint32_t byte_offset);
-ResourceKind     FlatSegmentResourceKind(uint32_t segment);
-const Decoder::Operand& DecodedSourceAt(const Decoder::Instruction& decoded, uint32_t index);
-Decoder::Operand        M0Operand();
-
-bool IsMemoryOpcode(Decoder::Opcode opcode);
-bool LowerMemoryInstruction(const Decoder::Instruction& decoded, BasicBlock& block,
-                            std::string* error);
-
-} // namespace Libs::Graphics::ShaderRecompiler::IR
-
-#endif // EMULATOR_SRC_GRAPHICS_SHADER_RECOMPILER_SHADERIR_SHADERIRINTERNAL_H_

src/graphics/shader/recompiler/ir/ShaderIRLog.cpp

@@ -1,214 +0,0 @@
-#include "common/assert.h"
-#include "graphics/shader/recompiler/BufferFormat.h"
-#include "graphics/shader/recompiler/ir/ShaderIR.h"
-#include "graphics/shader/recompiler/ir/ShaderIRInternal.h"
-
-#include <algorithm>
-#include <fmt/format.h>
-#include <utility>
-
-namespace Libs::Graphics::ShaderRecompiler::IR {
-
-std::string RegisterToString(Register reg) {
-	switch (reg.file) {
-		case RegisterFile::Scalar: return fmt::format("s{}", reg.index);
-		case RegisterFile::Vector: return fmt::format("v{}", reg.index);
-		case RegisterFile::Vcc:
-			return reg.index == 0 ? std::string("vcc_lo") : std::string("vcc_hi");
-		case RegisterFile::Exec:
-			return reg.index == 0 ? std::string("exec_lo") : std::string("exec_hi");
-		case RegisterFile::Scc:
-			if (reg.index == static_cast<uint32_t>(Decoder::OperandKind::Scc)) {
-				return "scc";
-			}
-			if (reg.index == static_cast<uint32_t>(Decoder::OperandKind::VccZ)) {
-				return "vccz";
-			}
-			if (reg.index == static_cast<uint32_t>(Decoder::OperandKind::ExecZ)) {
-				return "execz";
-			}
-			return fmt::format("scc_like_{}", reg.index);
-		case RegisterFile::M0: return "m0";
-		default: return "unknown_reg";
-	}
-}
-
-std::string OperandToString(const Operand& operand) {
-	std::string text;
-	switch (operand.kind) {
-		case OperandKind::Register: text = RegisterToString(operand.reg); break;
-		case OperandKind::ImmediateU32: text = fmt::format("0x{:08x}", operand.imm); break;
-		case OperandKind::PcRelativeU32: text = fmt::format("pc_rel(0x{:08x})", operand.imm); break;
-		case OperandKind::PcRelativeHighU32:
-			text = fmt::format("pc_rel_hi(0x{:08x})", operand.imm);
-			break;
-		case OperandKind::Null: text = "null"; break;
-		default: text = "unknown"; break;
-	}
-	if (operand.sdwa_sel != 6 || operand.sdwa_sext) {
-		text += fmt::format(".sdwa(sel={},sext={})", operand.sdwa_sel, operand.sdwa_sext ? 1u : 0u);
-	}
-	if (operand.absolute) {
-		text += ".abs";
-	}
-	if (operand.negate) {
-		text += ".neg";
-	}
-	if (operand.op_sel || operand.op_sel_hi || operand.negate_hi) {
-		text += fmt::format(".opsel(lo={},hi={},neghi={})", operand.op_sel ? 1u : 0u,
-		                    operand.op_sel_hi ? 1u : 0u, operand.negate_hi ? 1u : 0u);
-	}
-	if (operand.omod != 0) {
-		text += fmt::format(".omod({})", operand.omod);
-	}
-	if (operand.clamp) {
-		text += ".clamp";
-	}
-	if (operand.dpp) {
-		text += fmt::format(".dpp(ctrl=0x{:x},fi={},bc={})", operand.dpp_ctrl,
-		                    operand.dpp_fetch_inactive ? 1u : 0u, operand.dpp_bound_ctrl ? 1u : 0u);
-	}
-	return text;
-}
-
-std::string ResourceKindToString(ResourceKind kind) {
-	switch (kind) {
-		case ResourceKind::ScalarBuffer: return "scalar_buffer";
-		case ResourceKind::ScalarAddress: return "scalar_address";
-		case ResourceKind::Buffer: return "buffer";
-		case ResourceKind::Flat: return "flat";
-		case ResourceKind::Global: return "global";
-		case ResourceKind::Scratch: return "scratch";
-		case ResourceKind::Lds: return "lds";
-		case ResourceKind::Gds: return "gds";
-		case ResourceKind::Image: return "image";
-		case ResourceKind::ImageUint: return "image_uint";
-		case ResourceKind::StorageImage: return "storage_image";
-		case ResourceKind::StorageImageUint: return "storage_image_uint";
-		case ResourceKind::Sampler: return "sampler";
-		default: return "none";
-	}
-}
-
-std::string MemoryInfoToString(const MemoryInfo& mem) {
-	if (mem.kind == ResourceKind::None) {
-		return "";
-	}
-	std::string text = fmt::format(
-	    " ; {} resource={} sampler={} offset={} offset2={} dmask=0x{:x} data_dwords={} "
-	    "data_bits={} component={}/{} dfmt={} nfmt={} signed={} typed={} formatted={} segment={} "
-	    "glc={} slc={} idxen={} offen={} image_flags=0x{:x} image_dim={} image_addr={} "
-	    "image_mip={} image_r128={}",
-	    ResourceKindToString(mem.kind).c_str(), mem.resource, mem.sampler, mem.offset,
-	    mem.secondary_offset, mem.dmask, mem.data_dwords, mem.data_bits, mem.component_index,
-	    mem.component_count, mem.data_format, mem.number_format, mem.data_signed ? 1u : 0u,
-	    mem.typed ? 1u : 0u, mem.formatted ? 1u : 0u, mem.memory_segment, mem.glc ? 1u : 0u,
-	    mem.slc ? 1u : 0u, mem.idxen ? 1u : 0u, mem.offen ? 1u : 0u, mem.image_sample_flags,
-	    Decoder::ImageDimensionToString(mem.image_dimension), mem.image_address_components,
-	    mem.image_has_mip ? 1u : 0u, mem.image_r128 ? 1u : 0u);
-	if (mem.image_nsa_dwords != 0) {
-		text += fmt::format(" image_nsa_dwords={} image_nsa_addr=", mem.image_nsa_dwords);
-		const auto count =
-		    std::min<uint32_t>(mem.image_nsa_dwords * 4u, Decoder::MaxImageNsaAddressComponents);
-		for (uint32_t i = 0; i < count; i++) {
-			if (i != 0) {
-				text += ",";
-			}
-			text += fmt::format("{}", mem.image_nsa_addr[i]);
-		}
-	}
-	return text;
-}
-
-std::string ExportInfoToString(const ExportInfo& exp) {
-	return fmt::format(" ; target={} index={} raw=0x{:02x} en=0x{:x} done={} compr={} vm={}",
-	                   ExportTargetKindToString(exp.kind).c_str(), exp.index, exp.target, exp.en,
-	                   exp.done ? 1u : 0u, exp.compr ? 1u : 0u, exp.vm ? 1u : 0u);
-}
-
-std::string InputInfoToString(Opcode op, const InputInfo& input) {
-	if (op != Opcode::LoadInputF32) {
-		return "";
-	}
-	return fmt::format(" ; input_attr={} input_chan={} input_components={}", input.attr, input.chan,
-	                   input.component_count);
-}
-
-std::string TerminatorToString(const CFG::Terminator& term) {
-	return fmt::format(
-	    "term={} cond={} true={} false={} merge={} continue={} loop={} indirect_sgpr={} "
-	    "indirect_selector={} indirect_targets={} selector_values={}",
-	    static_cast<uint32_t>(term.kind), CFG::BranchConditionToString(term.condition).c_str(),
-	    term.true_block, term.false_block, term.merge_block, term.continue_block,
-	    term.loop_header ? 1u : 0u, term.indirect_pc_sgpr, term.indirect_selector_code,
-	    static_cast<uint32_t>(term.indirect_targets.size()),
-	    static_cast<uint32_t>(term.indirect_selector_values.size()));
-}
-
-std::string ExportSourcesToString(const Instruction& inst) {
-	std::string text;
-	for (uint32_t i = 0; i < inst.src_count && i < 4u; i++) {
-		text += i == 0 ? " " : ", ";
-		text += OperandToString(inst.src[i]);
-	}
-	return text;
-}
-
-std::string InstructionToString(const Instruction& inst) {
-	std::string text = fmt::format("0x{:08x}: ", inst.pc);
-	if (inst.op == Opcode::Export) {
-		text += "Export";
-		text += ExportSourcesToString(inst);
-		text += ExportInfoToString(inst.export_info);
-		return text;
-	}
-	text += OpcodeName(inst.op);
-	text += " ";
-	text += OperandToString(inst.dst);
-	if (inst.dst2.kind != OperandKind::Null) {
-		text += ", ";
-		text += OperandToString(inst.dst2);
-	}
-	for (uint32_t i = 0; i < inst.src_count; i++) {
-		text += i == 0 ? ", " : ", ";
-		text += OperandToString(inst.src[i]);
-	}
-	text += MemoryInfoToString(inst.memory);
-	text += InputInfoToString(inst.op, inst.input_info);
-	return text;
-}
-
-std::string ExportTargetKindToString(ExportTargetKind kind) {
-	switch (kind) {
-		case ExportTargetKind::Null: return "null";
-		case ExportTargetKind::Position: return "position";
-		case ExportTargetKind::Primitive: return "primitive";
-		case ExportTargetKind::Parameter: return "parameter";
-		case ExportTargetKind::Mrt: return "mrt";
-		case ExportTargetKind::MrtZ: return "mrtz";
-		default: return "unknown";
-	}
-}
-
-std::string ProgramToString(const Program& program) {
-	std::string text;
-	text += fmt::format("mode={} fallback={} reason={}\n",
-	                    program.dispatcher_fallback ? "dispatcher" : "structured",
-	                    CFG::FailureKindToString(program.cfg_failure_kind).c_str(),
-	                    program.fallback_reason.c_str());
-	for (const auto& block: program.blocks) {
-		text += fmt::format("block_{} pc=0x{:08x} end=0x{:08x} preds={} succs={} {}\n", block.id,
-		                    block.start_pc, block.end_pc,
-		                    static_cast<uint32_t>(block.predecessors.size()),
-		                    static_cast<uint32_t>(block.successors.size()),
-		                    TerminatorToString(block.terminator).c_str());
-		for (const auto& inst: block.instructions) {
-			text += "  ";
-			text += InstructionToString(inst);
-			text += "\n";
-		}
-	}
-	return text;
-}
-
-} // namespace Libs::Graphics::ShaderRecompiler::IR

src/graphics/shader/recompiler/ir/ShaderIRMemory.cpp

@@ -1,810 +0,0 @@
-#include "common/assert.h"
-#include "graphics/shader/recompiler/BufferFormat.h"
-#include "graphics/shader/recompiler/ir/ShaderIR.h"
-#include "graphics/shader/recompiler/ir/ShaderIRInternal.h"
-
-#include <algorithm>
-#include <fmt/format.h>
-#include <utility>
-
-namespace Libs::Graphics::ShaderRecompiler::IR {
-namespace {
-
-Opcode BufferLoadOpcode(const Decoder::Instruction& decoded) {
-	switch (decoded.data_bits) {
-		case 8u: return decoded.data_signed ? Opcode::BufferLoadSbyte : Opcode::BufferLoadUbyte;
-		case 16u: return decoded.data_signed ? Opcode::BufferLoadSshort : Opcode::BufferLoadUshort;
-		default: return Opcode::BufferLoadDword;
-	}
-}
-
-Opcode BufferStoreOpcode(uint32_t data_bits) {
-	switch (data_bits) {
-		case 8u: return Opcode::BufferStoreByte;
-		case 16u: return Opcode::BufferStoreShort;
-		default: return Opcode::BufferStoreDword;
-	}
-}
-
-Opcode DsReadOpcode(const Decoder::Instruction& decoded) {
-	switch (decoded.data_bits) {
-		case 8u: return decoded.data_signed ? Opcode::DsReadSbyte : Opcode::DsReadUbyte;
-		case 16u: return decoded.data_signed ? Opcode::DsReadSshort : Opcode::DsReadUshort;
-		default: return Opcode::DsReadB32;
-	}
-}
-
-Opcode FlatLoadOpcode(const Decoder::Instruction& decoded) {
-	switch (decoded.data_bits) {
-		case 8u: return decoded.data_signed ? Opcode::FlatLoadSbyte : Opcode::FlatLoadUbyte;
-		case 16u: return decoded.data_signed ? Opcode::FlatLoadSshort : Opcode::FlatLoadUshort;
-		default: return Opcode::FlatLoadDword;
-	}
-}
-
-Opcode FlatStoreOpcode(uint32_t data_bits) {
-	switch (data_bits) {
-		case 8u: return Opcode::FlatStoreByte;
-		case 16u: return Opcode::FlatStoreShort;
-		default: return Opcode::FlatStoreDword;
-	}
-}
-
-Opcode DsWriteOpcode(uint32_t data_bits) {
-	switch (data_bits) {
-		case 8u: return Opcode::DsWriteByte;
-		case 16u: return Opcode::DsWriteShort;
-		default: return Opcode::DsWriteB32;
-	}
-}
-
-Opcode LoweredImageOpcode(Decoder::Opcode opcode) {
-	switch (opcode) {
-		case Decoder::Opcode::IMAGE_GET_RESINFO: return Opcode::ImageGetResinfo;
-		case Decoder::Opcode::IMAGE_GET_LOD: return Opcode::ImageGetLod;
-		case Decoder::Opcode::IMAGE_LOAD:
-		case Decoder::Opcode::IMAGE_LOAD_MIP: return Opcode::ImageLoad;
-		case Decoder::Opcode::IMAGE_GATHER4_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_C:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4H: return Opcode::ImageGather4;
-		default: return Opcode::ImageSample;
-	}
-}
-
-bool LowerScalarMemoryLoadDword(const Decoder::Instruction& decoded, BasicBlock& block, Opcode op,
-                                std::string* error) {
-	uint32_t destination_code = 0;
-	const auto alignment = decoded.data_dwords == 1u ? 1u
-	                       : decoded.data_dwords == 2u ? 2u
-	                                                    : 4u;
-	if (!TryGetEncodedScalarCode(decoded.dst, destination_code) ||
-	    destination_code % alignment != 0u ||
-	    destination_code + decoded.data_dwords > 108u) {
-		SetError(error, "scalar-memory destination has an invalid aligned SGPR span");
-		return false;
-	}
-	uint32_t base_code = 0;
-	const auto base_alignment = op == Opcode::SLoadDword ? 2u : 4u;
-	const auto base_width     = op == Opcode::SLoadDword ? 2u : 4u;
-	if (!TryGetEncodedScalarCode(decoded.src0, base_code) ||
-	    base_code % base_alignment != 0u || base_code + base_width > 108u) {
-		SetError(error, "scalar-memory base has an invalid aligned SGPR span");
-		return false;
-	}
-	Instruction inst;
-	inst.pc        = decoded.pc;
-	inst.op        = op;
-	inst.src_count = 1;
-	const auto kind =
-	    op == Opcode::SLoadDword ? ResourceKind::ScalarAddress : ResourceKind::ScalarBuffer;
-	inst.memory = MemoryInfoFromDecoded(decoded, kind);
-	// Raw s_load uses a two-SGPR pointer base; s_buffer_load uses a four-SGPR
-	// descriptor resource index.
-	inst.memory.resource = op == Opcode::SLoadDword ? RawScalarLoadBase(decoded.src0)
-	                                                : ResourceIndexFromOperand(decoded.src0);
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(decoded.src1, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerBufferAddressSources(const Decoder::Instruction& decoded, Instruction& inst,
-                               uint32_t first_src, std::string* error) {
-	uint32_t src = first_src;
-	if (decoded.idxen) {
-		if (!LowerSourceOperand(decoded.src0, inst.src[src++], error)) {
-			return false;
-		}
-	}
-	if (decoded.offen) {
-		const auto offset_source =
-		    decoded.idxen ? OffsetDecodedRegister(decoded.src0, 1) : decoded.src0;
-		if (!LowerSourceOperand(offset_source, inst.src[src++], error)) {
-			return false;
-		}
-	}
-	if (!LowerSourceOperand(decoded.src2, inst.src[src++], error)) {
-		return false;
-	}
-	inst.src_count = src;
-	return true;
-}
-
-bool LowerBufferLoad(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	Instruction inst;
-	inst.pc     = decoded.pc;
-	inst.op     = BufferLoadOpcode(decoded);
-	inst.memory = MemoryInfoFromDecoded(decoded, ResourceKind::Buffer);
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerBufferAddressSources(decoded, inst, 0, error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerBufferStore(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	Instruction inst;
-	inst.pc       = decoded.pc;
-	inst.op       = BufferStoreOpcode(decoded.data_bits);
-	inst.memory   = MemoryInfoFromDecoded(decoded, ResourceKind::Buffer);
-	inst.dst.kind = OperandKind::Null;
-	if (!LowerSourceOperand(decoded.dst, inst.src[0], error) ||
-	    !LowerBufferAddressSources(decoded, inst, 1, error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerBufferAtomicDword(const Decoder::Instruction& decoded, BasicBlock& block, Opcode op,
-                            std::string* error) {
-	Instruction inst;
-	inst.pc       = decoded.pc;
-	inst.op       = op;
-	inst.memory   = MemoryInfoFromDecoded(decoded, ResourceKind::Buffer);
-	inst.dst.kind = OperandKind::Null;
-	if ((decoded.glc && !LowerRegisterOperand(decoded.dst, inst.dst, error)) ||
-	    !LowerSourceOperand(decoded.dst, inst.src[0], error) ||
-	    !LowerBufferAddressSources(decoded, inst, 1, error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-ResourceKind DsMemoryKind(const Decoder::Instruction& decoded) {
-	return decoded.gds ? ResourceKind::Gds : ResourceKind::Lds;
-}
-
-bool LowerDsRead(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	const auto  ir_op = DsReadOpcode(decoded);
-	Instruction inst;
-	inst.pc        = decoded.pc;
-	inst.op        = ir_op;
-	inst.src_count = 1;
-	inst.memory    = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsRead2(const Decoder::Instruction& decoded, BasicBlock& block, uint32_t dwords_per_read,
-                  std::string* error) {
-	Instruction inst;
-	inst.pc                 = decoded.pc;
-	inst.op                 = Opcode::DsRead2B32;
-	inst.src_count          = 1;
-	inst.memory             = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.memory.data_dwords = dwords_per_read * 2u;
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsAtomicU32(const Decoder::Instruction& decoded, BasicBlock& block, Opcode op,
-                      bool return_old, std::string* error) {
-	Instruction inst;
-	inst.pc        = decoded.pc;
-	inst.op        = op;
-	inst.src_count = 2;
-	inst.memory    = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.dst.kind  = return_old ? OperandKind::Register : OperandKind::Null;
-	if ((return_old && !LowerRegisterOperand(decoded.dst, inst.dst, error)) ||
-	    !LowerSourceOperand(decoded.src1, inst.src[0], error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[1], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsSwizzleB32(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	Instruction inst;
-	inst.pc          = decoded.pc;
-	inst.op          = Opcode::DsSwizzleB32;
-	inst.src_count   = 2;
-	inst.src[1].kind = OperandKind::ImmediateU32;
-	inst.src[1].imm  = decoded.offset & 0xffffu;
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsFloatMinMaxF32(const Decoder::Instruction& decoded, BasicBlock& block, Opcode op,
-                           std::string* error) {
-	Instruction inst;
-	inst.pc              = decoded.pc;
-	inst.op              = op;
-	inst.src_count       = 2;
-	inst.memory          = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.memory.resource = 0;
-	inst.dst.kind        = OperandKind::Null;
-	if (!LowerSourceOperand(decoded.src1, inst.src[0], error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[1], error) ||
-	    !LowerSourceOperand(decoded.src2, inst.src[2], error)) {
-		return false;
-	}
-	inst.src_count = 3;
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsWriteAddtidB32(const Decoder::Instruction& decoded, BasicBlock& block,
-                           std::string* error) {
-	Instruction inst;
-	inst.pc              = decoded.pc;
-	inst.op              = Opcode::DsWriteAddtidB32;
-	inst.src_count       = 2;
-	inst.memory          = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.memory.resource = 0;
-	inst.dst.kind        = OperandKind::Null;
-	const auto m0        = M0Operand();
-	if (!LowerSourceOperand(decoded.src1, inst.src[0], error) ||
-	    !LowerSourceOperand(m0, inst.src[1], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsReadAddtidB32(const Decoder::Instruction& decoded, BasicBlock& block,
-                          std::string* error) {
-	Instruction inst;
-	inst.pc              = decoded.pc;
-	inst.op              = Opcode::DsReadAddtidB32;
-	inst.src_count       = 1;
-	inst.memory          = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.memory.resource = 0;
-	const auto m0        = M0Operand();
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(m0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsAppendConsume(const Decoder::Instruction& decoded, BasicBlock& block, Opcode op,
-                          std::string* error) {
-	Instruction inst;
-	inst.pc        = decoded.pc;
-	inst.op        = op;
-	inst.src_count = 1;
-	inst.memory    = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	const auto m0  = M0Operand();
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(m0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-Opcode ImageAtomicIrOpcode(Decoder::Opcode opcode) {
-	switch (opcode) {
-		case Decoder::Opcode::IMAGE_ATOMIC_ADD: return Opcode::AtomicAddU32;
-		case Decoder::Opcode::IMAGE_ATOMIC_UMIN: return Opcode::AtomicUMinU32;
-		case Decoder::Opcode::IMAGE_ATOMIC_UMAX: return Opcode::AtomicUMaxU32;
-		case Decoder::Opcode::IMAGE_ATOMIC_AND: return Opcode::AtomicAndU32;
-		case Decoder::Opcode::IMAGE_ATOMIC_OR: return Opcode::AtomicOrU32;
-		case Decoder::Opcode::IMAGE_ATOMIC_XOR: return Opcode::AtomicXorU32;
-		default: return Opcode::AtomicAddU32;
-	}
-}
-
-bool LowerImageAtomicU32(const Decoder::Instruction& decoded, BasicBlock& block,
-                         std::string* error) {
-	Instruction inst;
-	inst.pc             = decoded.pc;
-	inst.op             = ImageAtomicIrOpcode(decoded.opcode);
-	inst.src_count      = 2;
-	inst.memory         = MemoryInfoFromDecoded(decoded, ResourceKind::StorageImageUint);
-	inst.memory.sampler = 0;
-	inst.dst.kind       = OperandKind::Null;
-	if ((decoded.glc && !LowerRegisterOperand(decoded.dst, inst.dst, error)) ||
-	    !LowerSourceOperand(decoded.dst, inst.src[0], error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[1], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerFlatLoad(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	const auto kind  = FlatSegmentResourceKind(decoded.memory_segment);
-	const auto ir_op = FlatLoadOpcode(decoded);
-	const auto count = decoded.data_bits == 32u ? decoded.data_dwords : 1u;
-	if (decoded.data_bits == 32u && count > 1u) {
-		Instruction inst;
-		inst.pc              = decoded.pc;
-		inst.op              = ir_op;
-		inst.src_count       = decoded.src_count;
-		inst.memory          = MemoryInfoFromDecoded(decoded, kind);
-		inst.memory.resource = 0;
-		if (!LowerRegisterOperand(decoded.dst, inst.dst, error)) {
-			return false;
-		}
-		for (uint32_t i = 0; i < decoded.src_count && i < 3u; i++) {
-			if (!LowerSourceOperand(DecodedSourceAt(decoded, i), inst.src[i], error)) {
-				return false;
-			}
-		}
-		block.instructions.push_back(inst);
-		return true;
-	}
-
-	for (uint32_t dword = 0; dword < count; dword++) {
-		Instruction inst;
-		inst.pc              = decoded.pc;
-		inst.op              = ir_op;
-		inst.src_count       = decoded.src_count;
-		inst.memory          = OffsetMemoryInfo(decoded, kind, dword);
-		inst.memory.resource = 0;
-		if (!LowerRegisterOperand(OffsetDecodedRegister(decoded.dst, dword), inst.dst, error)) {
-			return false;
-		}
-		for (uint32_t i = 0; i < decoded.src_count && i < 3u; i++) {
-			if (!LowerSourceOperand(DecodedSourceAt(decoded, i), inst.src[i], error)) {
-				return false;
-			}
-		}
-		block.instructions.push_back(inst);
-	}
-	return true;
-}
-
-bool LowerFlatStore(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	const auto kind  = FlatSegmentResourceKind(decoded.memory_segment);
-	const auto ir_op = FlatStoreOpcode(decoded.data_bits);
-	const auto count = decoded.data_bits == 32u ? decoded.data_dwords : 1u;
-	for (uint32_t dword = 0; dword < count; dword++) {
-		Instruction inst;
-		inst.pc              = decoded.pc;
-		inst.op              = ir_op;
-		inst.src_count       = decoded.src_count + 1u;
-		inst.memory          = OffsetMemoryInfo(decoded, kind, dword);
-		inst.memory.resource = 0;
-		inst.dst.kind        = OperandKind::Null;
-		if (!LowerSourceOperand(OffsetDecodedRegister(decoded.dst, dword), inst.src[0], error)) {
-			return false;
-		}
-		for (uint32_t i = 0; i < decoded.src_count && i + 1u < 3u; i++) {
-			if (!LowerSourceOperand(DecodedSourceAt(decoded, i), inst.src[i + 1u], error)) {
-				return false;
-			}
-		}
-		block.instructions.push_back(inst);
-	}
-	return true;
-}
-
-bool LowerDsWrite(const Decoder::Instruction& decoded, BasicBlock& block, std::string* error) {
-	const auto  ir_op = DsWriteOpcode(decoded.data_bits);
-	Instruction inst;
-	inst.pc        = decoded.pc;
-	inst.op        = ir_op;
-	inst.src_count = 2;
-	inst.memory    = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.dst.kind  = OperandKind::Null;
-	if (!LowerSourceOperand(decoded.src1, inst.src[0], error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[1], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerDsWrite2(const Decoder::Instruction& decoded, BasicBlock& block,
-                   uint32_t dwords_per_write, std::string* error) {
-	Instruction inst;
-	inst.pc                 = decoded.pc;
-	inst.op                 = Opcode::DsWrite2B32;
-	inst.src_count          = 3;
-	inst.memory             = MemoryInfoFromDecoded(decoded, DsMemoryKind(decoded));
-	inst.memory.data_dwords = dwords_per_write * 2u;
-	inst.dst.kind           = OperandKind::Null;
-	if (!LowerSourceOperand(decoded.src1, inst.src[0], error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[1], error) ||
-	    !LowerSourceOperand(decoded.src2, inst.src[2], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-bool LowerImageOperation(const Decoder::Instruction& decoded, BasicBlock& block,
-                         std::string* error) {
-	Instruction inst;
-	inst.pc = decoded.pc;
-	if (decoded.opcode == Decoder::Opcode::IMAGE_STORE ||
-	    decoded.opcode == Decoder::Opcode::IMAGE_STORE_MIP) {
-		inst.op             = Opcode::ImageStore;
-		inst.src_count      = 2;
-		inst.memory         = MemoryInfoFromDecoded(decoded, ResourceKind::StorageImage);
-		inst.memory.sampler = 0;
-		inst.dst.kind       = OperandKind::Null;
-		if (!LowerSourceOperand(decoded.dst, inst.src[0], error) ||
-		    !LowerSourceOperand(decoded.src0, inst.src[1], error)) {
-			return false;
-		}
-		block.instructions.push_back(inst);
-		return true;
-	}
-
-	inst.op        = LoweredImageOpcode(decoded.opcode);
-	inst.src_count = 1;
-	inst.memory    = MemoryInfoFromDecoded(decoded, ResourceKind::Image);
-	if (!LowerRegisterOperand(decoded.dst, inst.dst, error) ||
-	    !LowerSourceOperand(decoded.src0, inst.src[0], error)) {
-		return false;
-	}
-	block.instructions.push_back(inst);
-	return true;
-}
-
-} // namespace
-
-bool LowerMemoryInstruction(const Decoder::Instruction& decoded, BasicBlock& block,
-                            std::string* error) {
-	switch (decoded.opcode) {
-		case Decoder::Opcode::S_LOAD_DWORD:
-		case Decoder::Opcode::S_LOAD_DWORDX2:
-		case Decoder::Opcode::S_LOAD_DWORDX4:
-		case Decoder::Opcode::S_LOAD_DWORDX8:
-		case Decoder::Opcode::S_LOAD_DWORDX16:
-			return LowerScalarMemoryLoadDword(decoded, block, Opcode::SLoadDword, error);
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORD:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX2:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX4:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX8:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX16:
-			return LowerScalarMemoryLoadDword(decoded, block, Opcode::SBufferLoadDword, error);
-		case Decoder::Opcode::BUFFER_LOAD_UBYTE:
-		case Decoder::Opcode::BUFFER_LOAD_SBYTE:
-		case Decoder::Opcode::BUFFER_LOAD_USHORT:
-		case Decoder::Opcode::BUFFER_LOAD_SSHORT:
-		case Decoder::Opcode::BUFFER_LOAD_DWORD:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX2:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX3:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX4:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_X:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XY:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XYZ:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XYZW:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_X:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XY:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XYZ:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XYZW:
-			return LowerBufferLoad(decoded, block, error);
-		case Decoder::Opcode::BUFFER_STORE_DWORD:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX2:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX3:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX4:
-		case Decoder::Opcode::BUFFER_STORE_BYTE:
-		case Decoder::Opcode::BUFFER_STORE_SHORT:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_X:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XY:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XYZ:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XYZW:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_X:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XY:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XYZ:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XYZW:
-			return LowerBufferStore(decoded, block, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_SWAP:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicSwapU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_ADD:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicAddU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_SUB:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicSubU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_SMIN:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicSMinI32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_UMIN:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicUMinU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_SMAX:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicSMaxI32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_UMAX:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicUMaxU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_AND:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicAndU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_OR:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicOrU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_XOR:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicXorU32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_FMIN:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicFMinF32, error);
-		case Decoder::Opcode::BUFFER_ATOMIC_FMAX:
-			return LowerBufferAtomicDword(decoded, block, Opcode::AtomicFMaxF32, error);
-		case Decoder::Opcode::FLAT_LOAD_UBYTE:
-		case Decoder::Opcode::FLAT_LOAD_SBYTE:
-		case Decoder::Opcode::FLAT_LOAD_USHORT:
-		case Decoder::Opcode::FLAT_LOAD_SSHORT:
-		case Decoder::Opcode::FLAT_LOAD_DWORD:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX2:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX3:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX4: return LowerFlatLoad(decoded, block, error);
-		case Decoder::Opcode::FLAT_STORE_BYTE:
-		case Decoder::Opcode::FLAT_STORE_SHORT:
-		case Decoder::Opcode::FLAT_STORE_DWORD:
-		case Decoder::Opcode::FLAT_STORE_DWORDX2:
-		case Decoder::Opcode::FLAT_STORE_DWORDX3:
-		case Decoder::Opcode::FLAT_STORE_DWORDX4: return LowerFlatStore(decoded, block, error);
-		case Decoder::Opcode::DS_ADD_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicAddU32, false, error);
-		case Decoder::Opcode::DS_ADD_RTN_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicAddU32, true, error);
-		case Decoder::Opcode::DS_SUB_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSubU32, false, error);
-		case Decoder::Opcode::DS_SUB_RTN_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSubU32, true, error);
-		case Decoder::Opcode::DS_MIN_I32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSMinI32, false, error);
-		case Decoder::Opcode::DS_MIN_RTN_I32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSMinI32, true, error);
-		case Decoder::Opcode::DS_MAX_I32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSMaxI32, false, error);
-		case Decoder::Opcode::DS_MAX_RTN_I32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSMaxI32, true, error);
-		case Decoder::Opcode::DS_MIN_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicUMinU32, false, error);
-		case Decoder::Opcode::DS_MIN_RTN_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicUMinU32, true, error);
-		case Decoder::Opcode::DS_MAX_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicUMaxU32, false, error);
-		case Decoder::Opcode::DS_MAX_RTN_U32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicUMaxU32, true, error);
-		case Decoder::Opcode::DS_AND_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicAndU32, false, error);
-		case Decoder::Opcode::DS_AND_RTN_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicAndU32, true, error);
-		case Decoder::Opcode::DS_OR_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicOrU32, false, error);
-		case Decoder::Opcode::DS_OR_RTN_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicOrU32, true, error);
-		case Decoder::Opcode::DS_XOR_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicXorU32, false, error);
-		case Decoder::Opcode::DS_XOR_RTN_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicXorU32, true, error);
-		case Decoder::Opcode::DS_WRXCHG_RTN_B32:
-			return LowerDsAtomicU32(decoded, block, Opcode::AtomicSwapU32, true, error);
-		case Decoder::Opcode::DS_MIN_F32:
-			return LowerDsFloatMinMaxF32(decoded, block, Opcode::DsMinF32, error);
-		case Decoder::Opcode::DS_MAX_F32:
-			return LowerDsFloatMinMaxF32(decoded, block, Opcode::DsMaxF32, error);
-		case Decoder::Opcode::DS_SWIZZLE_B32: return LowerDsSwizzleB32(decoded, block, error);
-		case Decoder::Opcode::DS_CONSUME:
-			return LowerDsAppendConsume(decoded, block, Opcode::DsConsume, error);
-		case Decoder::Opcode::DS_APPEND:
-			return LowerDsAppendConsume(decoded, block, Opcode::DsAppend, error);
-		case Decoder::Opcode::DS_WRITE_ADDTID_B32:
-			return LowerDsWriteAddtidB32(decoded, block, error);
-		case Decoder::Opcode::DS_READ_ADDTID_B32:
-			return LowerDsReadAddtidB32(decoded, block, error);
-		case Decoder::Opcode::DS_READ2_B32:
-		case Decoder::Opcode::DS_READ2ST64_B32: return LowerDsRead2(decoded, block, 1, error);
-		case Decoder::Opcode::DS_READ2_B64:
-		case Decoder::Opcode::DS_READ2ST64_B64: return LowerDsRead2(decoded, block, 2, error);
-		case Decoder::Opcode::DS_READ_I8:
-		case Decoder::Opcode::DS_READ_U8:
-		case Decoder::Opcode::DS_READ_I16:
-		case Decoder::Opcode::DS_READ_U16:
-		case Decoder::Opcode::DS_READ_B32:
-		case Decoder::Opcode::DS_READ_B64:
-		case Decoder::Opcode::DS_READ_B96:
-		case Decoder::Opcode::DS_READ_B128: return LowerDsRead(decoded, block, error);
-		case Decoder::Opcode::DS_WRITE2_B32:
-		case Decoder::Opcode::DS_WRITE2ST64_B32: return LowerDsWrite2(decoded, block, 1, error);
-		case Decoder::Opcode::DS_WRITE2_B64:
-		case Decoder::Opcode::DS_WRITE2ST64_B64: return LowerDsWrite2(decoded, block, 2, error);
-		case Decoder::Opcode::DS_WRITE_B8:
-		case Decoder::Opcode::DS_WRITE_B16:
-		case Decoder::Opcode::DS_WRITE_B32:
-		case Decoder::Opcode::DS_WRITE_B64:
-		case Decoder::Opcode::DS_WRITE_B96:
-		case Decoder::Opcode::DS_WRITE_B128: return LowerDsWrite(decoded, block, error);
-		case Decoder::Opcode::IMAGE_GET_RESINFO:
-		case Decoder::Opcode::IMAGE_GET_LOD:
-		case Decoder::Opcode::IMAGE_LOAD:
-		case Decoder::Opcode::IMAGE_LOAD_MIP:
-		case Decoder::Opcode::IMAGE_STORE:
-		case Decoder::Opcode::IMAGE_STORE_MIP:
-		case Decoder::Opcode::IMAGE_GATHER4_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_C:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4H:
-		case Decoder::Opcode::IMAGE_SAMPLE: return LowerImageOperation(decoded, block, error);
-		case Decoder::Opcode::IMAGE_ATOMIC_ADD:
-		case Decoder::Opcode::IMAGE_ATOMIC_UMIN:
-		case Decoder::Opcode::IMAGE_ATOMIC_UMAX:
-		case Decoder::Opcode::IMAGE_ATOMIC_AND:
-		case Decoder::Opcode::IMAGE_ATOMIC_OR:
-		case Decoder::Opcode::IMAGE_ATOMIC_XOR: return LowerImageAtomicU32(decoded, block, error);
-		default:
-			if (error != nullptr) {
-				*error = fmt::format("memory opcode has no specialized IR lowering: {}",
-				                     magic_enum::enum_name(decoded.opcode));
-			}
-			return false;
-	}
-}
-
-bool IsMemoryOpcode(Decoder::Opcode opcode) {
-	switch (opcode) {
-		case Decoder::Opcode::S_LOAD_DWORD:
-		case Decoder::Opcode::S_LOAD_DWORDX2:
-		case Decoder::Opcode::S_LOAD_DWORDX4:
-		case Decoder::Opcode::S_LOAD_DWORDX8:
-		case Decoder::Opcode::S_LOAD_DWORDX16:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORD:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX2:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX4:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX8:
-		case Decoder::Opcode::S_BUFFER_LOAD_DWORDX16:
-		case Decoder::Opcode::BUFFER_LOAD_UBYTE:
-		case Decoder::Opcode::BUFFER_LOAD_SBYTE:
-		case Decoder::Opcode::BUFFER_LOAD_USHORT:
-		case Decoder::Opcode::BUFFER_LOAD_SSHORT:
-		case Decoder::Opcode::BUFFER_LOAD_DWORD:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX2:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX3:
-		case Decoder::Opcode::BUFFER_LOAD_DWORDX4:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_X:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XY:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XYZ:
-		case Decoder::Opcode::BUFFER_LOAD_FORMAT_XYZW:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_X:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XY:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XYZ:
-		case Decoder::Opcode::TBUFFER_LOAD_FORMAT_XYZW:
-		case Decoder::Opcode::BUFFER_STORE_DWORD:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX2:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX3:
-		case Decoder::Opcode::BUFFER_STORE_DWORDX4:
-		case Decoder::Opcode::BUFFER_STORE_BYTE:
-		case Decoder::Opcode::BUFFER_STORE_SHORT:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_X:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XY:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XYZ:
-		case Decoder::Opcode::BUFFER_STORE_FORMAT_XYZW:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_X:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XY:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XYZ:
-		case Decoder::Opcode::TBUFFER_STORE_FORMAT_XYZW:
-		case Decoder::Opcode::BUFFER_ATOMIC_SWAP:
-		case Decoder::Opcode::BUFFER_ATOMIC_ADD:
-		case Decoder::Opcode::BUFFER_ATOMIC_SUB:
-		case Decoder::Opcode::BUFFER_ATOMIC_SMIN:
-		case Decoder::Opcode::BUFFER_ATOMIC_UMIN:
-		case Decoder::Opcode::BUFFER_ATOMIC_SMAX:
-		case Decoder::Opcode::BUFFER_ATOMIC_UMAX:
-		case Decoder::Opcode::BUFFER_ATOMIC_AND:
-		case Decoder::Opcode::BUFFER_ATOMIC_OR:
-		case Decoder::Opcode::BUFFER_ATOMIC_XOR:
-		case Decoder::Opcode::BUFFER_ATOMIC_FMIN:
-		case Decoder::Opcode::BUFFER_ATOMIC_FMAX:
-		case Decoder::Opcode::FLAT_LOAD_UBYTE:
-		case Decoder::Opcode::FLAT_LOAD_SBYTE:
-		case Decoder::Opcode::FLAT_LOAD_USHORT:
-		case Decoder::Opcode::FLAT_LOAD_SSHORT:
-		case Decoder::Opcode::FLAT_LOAD_DWORD:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX2:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX3:
-		case Decoder::Opcode::FLAT_LOAD_DWORDX4:
-		case Decoder::Opcode::FLAT_STORE_BYTE:
-		case Decoder::Opcode::FLAT_STORE_SHORT:
-		case Decoder::Opcode::FLAT_STORE_DWORD:
-		case Decoder::Opcode::FLAT_STORE_DWORDX2:
-		case Decoder::Opcode::FLAT_STORE_DWORDX3:
-		case Decoder::Opcode::FLAT_STORE_DWORDX4:
-		case Decoder::Opcode::DS_ADD_U32:
-		case Decoder::Opcode::DS_ADD_RTN_U32:
-		case Decoder::Opcode::DS_SUB_U32:
-		case Decoder::Opcode::DS_SUB_RTN_U32:
-		case Decoder::Opcode::DS_MIN_I32:
-		case Decoder::Opcode::DS_MIN_RTN_I32:
-		case Decoder::Opcode::DS_MAX_I32:
-		case Decoder::Opcode::DS_MAX_RTN_I32:
-		case Decoder::Opcode::DS_MIN_U32:
-		case Decoder::Opcode::DS_MIN_RTN_U32:
-		case Decoder::Opcode::DS_MAX_U32:
-		case Decoder::Opcode::DS_MAX_RTN_U32:
-		case Decoder::Opcode::DS_AND_B32:
-		case Decoder::Opcode::DS_AND_RTN_B32:
-		case Decoder::Opcode::DS_OR_B32:
-		case Decoder::Opcode::DS_OR_RTN_B32:
-		case Decoder::Opcode::DS_XOR_B32:
-		case Decoder::Opcode::DS_XOR_RTN_B32:
-		case Decoder::Opcode::DS_WRXCHG_RTN_B32:
-		case Decoder::Opcode::DS_MIN_F32:
-		case Decoder::Opcode::DS_MAX_F32:
-		case Decoder::Opcode::DS_SWIZZLE_B32:
-		case Decoder::Opcode::DS_CONSUME:
-		case Decoder::Opcode::DS_APPEND:
-		case Decoder::Opcode::DS_READ_I8:
-		case Decoder::Opcode::DS_READ_U8:
-		case Decoder::Opcode::DS_READ_I16:
-		case Decoder::Opcode::DS_READ_U16:
-		case Decoder::Opcode::DS_READ2_B32:
-		case Decoder::Opcode::DS_READ2ST64_B32:
-		case Decoder::Opcode::DS_READ_B32:
-		case Decoder::Opcode::DS_READ_B64:
-		case Decoder::Opcode::DS_READ2_B64:
-		case Decoder::Opcode::DS_READ2ST64_B64:
-		case Decoder::Opcode::DS_READ_B96:
-		case Decoder::Opcode::DS_READ_B128:
-		case Decoder::Opcode::DS_WRITE_B8:
-		case Decoder::Opcode::DS_WRITE_B16:
-		case Decoder::Opcode::DS_WRITE2_B32:
-		case Decoder::Opcode::DS_WRITE2ST64_B32:
-		case Decoder::Opcode::DS_WRITE2_B64:
-		case Decoder::Opcode::DS_WRITE2ST64_B64:
-		case Decoder::Opcode::DS_WRITE_B32:
-		case Decoder::Opcode::DS_WRITE_B64:
-		case Decoder::Opcode::DS_WRITE_B96:
-		case Decoder::Opcode::DS_WRITE_B128:
-		case Decoder::Opcode::DS_WRITE_ADDTID_B32:
-		case Decoder::Opcode::DS_READ_ADDTID_B32:
-		case Decoder::Opcode::IMAGE_GET_RESINFO:
-		case Decoder::Opcode::IMAGE_GET_LOD:
-		case Decoder::Opcode::IMAGE_LOAD:
-		case Decoder::Opcode::IMAGE_LOAD_MIP:
-		case Decoder::Opcode::IMAGE_STORE:
-		case Decoder::Opcode::IMAGE_STORE_MIP:
-		case Decoder::Opcode::IMAGE_ATOMIC_ADD:
-		case Decoder::Opcode::IMAGE_ATOMIC_UMIN:
-		case Decoder::Opcode::IMAGE_ATOMIC_UMAX:
-		case Decoder::Opcode::IMAGE_ATOMIC_AND:
-		case Decoder::Opcode::IMAGE_ATOMIC_OR:
-		case Decoder::Opcode::IMAGE_ATOMIC_XOR:
-		case Decoder::Opcode::IMAGE_GATHER4_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_C:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ:
-		case Decoder::Opcode::IMAGE_GATHER4_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_O:
-		case Decoder::Opcode::IMAGE_GATHER4_C_LZ_O:
-		case Decoder::Opcode::IMAGE_GATHER4H:
-		case Decoder::Opcode::IMAGE_SAMPLE: return true;
-		default: return false;
-	}
-}
-
-} // namespace Libs::Graphics::ShaderRecompiler::IR

src/graphics/shader/recompiler/ir/ValueProgram.h

@@ -1,76 +0,0 @@
-#pragma once
-
-#include "graphics/shader/recompiler/ir/Block.h"
-#include "graphics/shader/recompiler/ir/ShaderIR.h"
-
-#include <memory>
-#include <optional>
-#include <string>
-#include <vector>
-
-namespace Libs::Graphics::ShaderRecompiler::IR {
-
-struct ValueBlockInfo {
-	uint32_t        id       = 0;
-	uint32_t        start_pc = 0;
-	uint32_t        end_pc   = 0;
-	CFG::Terminator terminator;
-	Value           condition;
-	Value           indirect_target;
-};
-
-struct DescriptorSource {
-	struct IndirectImage {
-		uint32_t material_source = 0;
-		uint32_t heap_source     = 0;
-		uint32_t selector_stride = 0;
-		uint32_t selector_offset = 0;
-		uint32_t key_arg         = 0;
-
-		bool operator==(const IndirectImage& other) const = default;
-	};
-
-	std::array<Value, 8>         dwords {};
-	uint32_t                     dword_count = 0;
-	std::optional<IndirectImage> indirect_image;
-
-	bool operator==(const DescriptorSource& other) const = default;
-};
-
-struct SrtRead {
-	Value    value;
-	uint32_t flat_offset = 0;
-	uint32_t use_pc      = 0;
-
-	bool operator==(const SrtRead& other) const = default;
-};
-
-struct ValueProgram {
-	ValueProgram() = default;
-	~ValueProgram();
-
-	ValueProgram(const ValueProgram&)            = delete;
-	ValueProgram& operator=(const ValueProgram&) = delete;
-	ValueProgram(ValueProgram&&)                 = default;
-	ValueProgram& operator=(ValueProgram&&)      = default;
-
-	std::vector<std::unique_ptr<Block>> block_storage;
-	BlockList                           blocks;
-	std::vector<ValueBlockInfo>         block_info;
-	// Decoded MIMG/VMEM metadata carries details such as RDNA2 NSA address registers and
-	// storage-image swizzles. Typed memory instructions carry a dense index into these shader-local
-	// tables until those fields are consumed by emission.
-	std::vector<MemoryInfo>       memory_info;
-	std::vector<ExportInfo>       export_info;
-	std::vector<DescriptorSource> descriptor_sources;
-	std::vector<SrtRead>          srt_reads;
-	std::vector<Value>            dynamic_reads;
-};
-
-bool        ValidateValueProgram(const ValueProgram& program, bool require_ssa, std::string* error);
-void        ResolveControlFlowIdentities(ValueProgram& program);
-bool        EquivalentValue(const ValueProgram& program, Value left, Value right);
-Value       ResolveInvariantPhi(const ValueProgram& program, Value value);
-std::string ValueProgramToString(const ValueProgram& program);
-
-} // namespace Libs::Graphics::ShaderRecompiler::IR

src/graphics/shader/recompiler/ir/opcodes/Opcodes.h

@@ -1,71 +0,0 @@
-#pragma once
-
-#include "graphics/shader/recompiler/ir/Type.h"
-
-#include <array>
-#include <cstddef>
-#include <string_view>
-
-namespace Libs::Graphics::ShaderRecompiler::IR {
-
-enum class Opcode {
-#define IR_OPCODE(name, category, lowering, ...) name,
-#include "graphics/shader/recompiler/ir/opcodes/ShaderIROpcodes.inc"
-#undef IR_OPCODE
-	Count,
-};
-
-enum class OpcodeClass { General, Compare, Compare64 };
-enum class LoweringClass {
-	Control,
-	Move,
-	Lane,
-	State,
-	Memory,
-	Attribute,
-	IntegerCompare,
-	Integer16Compare,
-	FloatCompare,
-	Conversion,
-	Integer16,
-	PackedInteger16,
-	PackedFloat16,
-	Float16,
-	Float,
-	U64Mask,
-	SimpleInteger,
-	ComposedInteger,
-	ExtendedInteger,
-};
-
-struct OpcodeInfo {
-	std::string_view name;
-	OpcodeClass      opcode_class   = OpcodeClass::General;
-	LoweringClass    lowering_class = LoweringClass::Control;
-};
-
-inline constexpr OpcodeInfo OPCODE_INFO[] = {
-#define IR_OPCODE(name, category, lowering, ...)                                                   \
-	{#name, OpcodeClass::category, LoweringClass::lowering},
-#include "graphics/shader/recompiler/ir/opcodes/ShaderIROpcodes.inc"
-#undef IR_OPCODE
-};
-static_assert(std::size(OPCODE_INFO) == static_cast<size_t>(Opcode::Count));
-
-constexpr const OpcodeInfo& GetOpcodeInfo(Opcode opcode) {
-	return OPCODE_INFO[static_cast<size_t>(opcode)];
-}
-
-constexpr std::string_view OpcodeName(Opcode opcode) {
-	return GetOpcodeInfo(opcode).name;
-}
-
-constexpr bool IsCompareOpcode(Opcode opcode) {
-	return GetOpcodeInfo(opcode).opcode_class != OpcodeClass::General;
-}
-
-constexpr bool IsCompare64Opcode(Opcode opcode) {
-	return GetOpcodeInfo(opcode).opcode_class == OpcodeClass::Compare64;
-}
-
-} // namespace Libs::Graphics::ShaderRecompiler::IR