Compare commits

...
Author SHA1 Message Date
nmzik 7b9997baea shader: fold exact float class masks into native NaN predicates 2026-10-09 02:54:21 +02:00
nmzik 393257eaae shader: classify float NaNs with one magnitude comparison 2026-10-09 02:54:21 +02:00
nmzik 13e3ca314c shader: unpack buffer offsets from one load per packed word 2026-10-09 02:54:21 +02:00
nmzik b4853a0e08 shader: preserve export vectors instead of rebuilding scalar components 2026-10-09 02:54:21 +02:00
nmzik ee4dd90649 shader: emit native integer operations and remove signedness casts 2026-10-09 02:54:21 +02:00
nmzik 3df456675d shader: own control flow on IR blocks with direct targets 2026-10-09 01:59:41 +02:00
nmzik 8abf99c56e shader: emit bounded SPIR-V operands without temporary vectors 2026-10-09 01:58:39 +02:00
nmzik 7c6121cba5 shader: build CFG labels and targets from ordered storage 2026-10-09 01:55:50 +02:00
nmzik 46997dfc7a shader: retain resource type IDs in SPIR-V emission state 2026-10-09 01:55:50 +02:00
nmzik 97fda7b10f renderer: bind prepared buffer descriptors without copying 2026-10-09 01:49:18 +02:00
nmzik dbe5353f3a shader: eliminate retained SRT lookup aliases 2026-10-09 01:44:34 +02:00
nmzik e1ac893a0c shader: collect usage once and emit descriptor slots directly 2026-10-09 02:17:38 +04:00
nmzik b35cdc39b7 shader: append deferred Phi operands directly to SPIR-V storage 2026-10-09 01:40:52 +04:00
nmzik 52dbc724f8 shader: store SPIR-V definitions directly in IR nodes 2026-10-09 01:40:41 +04:00
Mac 2f4b8a335d shader: allow SDWA byte destinations for bitwise VOP2 ops 2026-10-08 22:12:05 +04:00
nmzik b38b7454a6 Preserve vertex export locations for mixed pixel interpolation 2026-10-08 22:05:53 +04:00
nmzik b65dc12208 Track pad port ownership and reject duplicate opens 2026-10-08 21:58:53 +04:00
Devran Cosmo Uenal 58978d154a font: rasterize glyphs larger than 128 pixels (#1214)
Glyph images were rendered into a fixed 128x128 buffer and their width and
height were clamped to 128. Since glyphs are rasterized at the requested
scale (f96a14a0), large text such as title cards lost every row and column
past 128 pixels, so letters were drawn cut into bands.

The glyph buffers now grow to the glyph's bitmap box and width_byte is the
glyph's width. The VGA fallback font keeps its 128-pixel limit; scalable
glyphs are limited to 4096 pixels only to reject absurd scales.
2026-10-08 20:03:13 +04:00
1f3adf5109 cmake: accept glslang's current tool name and document Python 3 (#1161)
* cmake: accept glslang's current tool name and document Python 3

Official glslang release archives for Windows ship the compiler only as
glslang.exe, so configuring with one of them on PATH could not find
glslangValidator. Fall back to glslang; glslangValidator is still
preferred where both exist.

The bundled SPIRV-Tools needs Python 3 at configure time, which the
Windows build requirements did not mention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: list Python 3 in the Linux and macOS prerequisites

The bundled SPIRV-Tools calls find_host_package(Python3 REQUIRED) on every platform, so the apt and Homebrew package lists need it too, not only the Windows requirements.

* cmake: remove glslang naming comment

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: nmzik <Nmzik@mail.ru>
2026-10-08 19:46:09 +04:00
NeySlimandnmzik 107d5f750a libNet: finish NpCommerce dialogs at open (#1166)
* libNet: finish NpCommerce dialogs at open

* libNet: complete NpCommerce dialog cancellation

---------

Co-authored-by: nmzik <Nmzik@mail.ru>
2026-10-08 19:31:47 +04:00
f40755173f ampr: keep resolved host paths in native form (#1175)
* ampr: keep resolved host paths in native form

ResolveOnePath stored each resolved file as a UTF-8 std::string
(Common::PathToString) and the read, stat and size paths turned that
string back into a std::filesystem::path. On POSIX this round trip is a
no-op, but on Windows path(std::string) decodes the bytes with the ANSI
code page, not UTF-8 (/utf-8 only sets the source and execution
charsets). Any non-ASCII character in the game folder therefore produced
a different path: the resolve step found the file through the native
path, then every APR read of it failed with ENOENT.

Tokyo Xtreme Racer (#1156) installed under "D:/Инсталляторы/[ps5]/..."
hits this on its first pak read:

  APR read missing host file: .../tokyoxtremeracer/content/paks/global.utoc
  Apr read failure 80020002 at CB offset 0   (LowLevelFatalError, abort)

even though KernelStat and the directory listing see the same file.

Store std::filesystem::path in ResolvedPathInfo and the file-id table
and pass it straight to Common::File. Convert to UTF-8 only for the log
lines.

* tests: cover APR stat and size under a non-ASCII host folder

Resolve a file inside a Cyrillic directory through APR and check that
the stat and size exports find it by ID, matching the Tokyo Xtreme
Racer install layout from #1156. On POSIX the old string round trip
also passed, so this guards the Windows path.

* tests: read the non-ASCII APR fixture through an APR command

Review on #1175 noted that the Unicode fixture only checked the stat
and size exports. A submitted APR read opens the stored host path in
ExecuteCommand/ReadHostFileToGuest, which the fixture did not reach.

Add AprReadMatches, which submits one APR read command for a file ID
and compares the bytes, and use it on the file under the Cyrillic
folder. As with the stat and size checks, POSIX hosts run this path but
only Windows would show an ANSI code page round trip.

* Remove APR path tests from this change

* Remove unrelated formatting changes

* Remove native path explanatory comments

---------

Co-authored-by: overlordxrz-source <overlordxrz-source@users.noreply.github.com>
Co-authored-by: nmzik <Nmzik@mail.ru>
2026-10-08 19:27:10 +04:00
57 changed files with 2799 additions and 2505 deletions

No files matched your search

+3 -2
View File
@@ -244,7 +244,7 @@ file(GLOB kyty_emulator_src CONFIGURE_DEPENDS
"${KYTY_SOURCE_DIR}/loader/*.h"
)
find_program(KYTY_GLSLANG_VALIDATOR glslangValidator REQUIRED)
find_program(KYTY_GLSLANG_VALIDATOR NAMES glslangValidator glslang REQUIRED)
set(gpu_tiler_shader_dir "${KYTY_SOURCE_DIR}/graphics/host_gpu/shaders")
set(gpu_tiler_generated_dir "${PROJECT_BINARY_DIR}/gpu_tiler_shaders")
set(gpu_tiler_shader_names
@@ -393,8 +393,10 @@ add_kyty_full_emulator_test(shader_cfg_tests "${KYTY_TESTS_DIR}/shaderCfgTests.c
add_executable(scalar_provenance_tests EXCLUDE_FROM_ALL
"${KYTY_TESTS_DIR}/ScalarProvenanceTests.cpp"
"${KYTY_SOURCE_DIR}/graphics/guest_gpu/gpu_format.cpp"
"${KYTY_SOURCE_DIR}/graphics/host_gpu/hostMemory.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ReadLaneElimination.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ResourceMaterialization.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/SrtWalker.cpp"
)
target_link_libraries(scalar_provenance_tests fmt::fmt)
@@ -477,7 +479,6 @@ add_executable(resource_tracking_tests EXCLUDE_FROM_ALL
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ResourceTracking.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ResourceMaterialization.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/ShaderInfoCollection.cpp"
"${KYTY_SOURCE_DIR}/graphics/shader/recompiler/ir/passes/BindingLayout.cpp"
)
target_link_libraries(resource_tracking_tests fmt::fmt)
target_include_directories(resource_tracking_tests PRIVATE ${inc_headers})
+4 -3
View File
@@ -141,7 +141,8 @@ the Vulkan/SPIR-V validation rules.
- Visual Studio 2022 or Build Tools 2022 with the **Desktop development with C++** workload and
**C++ Clang tools for Windows** component
- Qt 6 for MSVC 2022 64-bit, including Concurrent, Network, and Widgets
- [glslang](https://github.com/KhronosGroup/glslang/releases) (`glslangValidator`) on `PATH`
- [glslang](https://github.com/KhronosGroup/glslang/releases) (`glslang` or `glslangValidator`) on `PATH`
- Python 3 on `PATH`; the bundled SPIRV-Tools runs it at configure time
The Microsoft C++ compiler (`cl.exe`) is not supported; use `clang-cl`.
@@ -176,7 +177,7 @@ build has no working sound and no gamepad hotplug:
```bash
sudo apt-get install --no-install-recommends \
clang lld ninja-build cmake git glslang-tools pkg-config \
clang lld ninja-build cmake git glslang-tools python3 pkg-config \
libgl1-mesa-dev libx11-dev libxcursor-dev libxext-dev libxfixes-dev \
libxi-dev libxrandr-dev libxss-dev libxtst-dev libxkbcommon-dev \
libasound2-dev libpulse-dev libudev-dev libdbus-1-dev libwayland-dev wayland-protocols
@@ -251,7 +252,7 @@ Requirements:
- An Apple Silicon Mac with Rosetta 2 installed (`softwareupdate --install-rosetta`)
- Xcode (or the Command Line Tools)
- Homebrew packages: `brew install cmake ninja glslang`
- Homebrew packages: `brew install cmake ninja glslang python`
- Qt 6 (Concurrent, Network, Widgets) with x86-64 support. The official Qt installation is
universal and works; Homebrew's Qt is arm64-only and will not link
@@ -26,7 +26,6 @@
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
#include "graphics/shader/shader.h"
#include "kernel/memory.h"
@@ -76,10 +75,6 @@ vk::DescriptorType NativeDescriptorType(BindingKind kind) {
EXIT("invalid native descriptor binding kind");
}
uint32_t NativeDescriptorCount(const ShaderRecompiler::IR::DescriptorBinding& binding) {
return binding.resources.empty() ? 1u : static_cast<uint32_t>(binding.resources.size());
}
vk::DescriptorImageInfo MakeImageInfo(const TextureBinding& texture, uint32_t element) {
vk::ImageView view = nullptr;
if (texture.mip_views.empty()) {
@@ -799,12 +794,11 @@ void RenderExecutor::PrepareBindings(const ShaderStageRuntime& runtime,
m_context, program, i, snapshot.samplers[program.info.samplers[i].snapshot_index]));
}
prepared.shader_data.reserve(program.bindings.ShaderDataDwords());
for (const auto reg: program.bindings.user_data_registers) {
for (const auto reg: program.info.user_data_registers) {
prepared.shader_data.push_back(snapshot.user_data[reg - program.user_data_base]);
}
prepared.shader_data.resize(program.bindings.ShaderDataDwords());
if (ShaderRecompiler::IR::FindBinding(
program.bindings, ShaderRecompiler::IR::DescriptorBindingKind::Gds) != nullptr) {
if (program.bindings.descriptor_counts[static_cast<size_t>(BindingKind::Gds)] != 0) {
prepared.gds.buffer = m_context.GetBufferCache().GetGdsBuffer()->Handle();
}
}
@@ -817,19 +811,20 @@ void RenderExecutor::FindBuffers(std::span<PreparedBindings* const> stages) {
EXIT_IF(prepared.runtime == nullptr || !*prepared.runtime);
const auto& program = *prepared.runtime->program;
const auto& snapshot = *prepared.runtime->resources;
prepared.buffer_sources.clear();
const auto& layout = program.bindings;
if (layout.memory_offset_count == 0) {
prepared.buffer_sources.resize(layout.descriptor_counts[static_cast<size_t>(BindingKind::Buffers)]);
if (layout.descriptor_counts[static_cast<size_t>(BindingKind::Buffers)] == 0) {
continue;
}
const auto& resources = layout.descriptors.front().resources;
prepared.buffer_sources.reserve(resources.size());
for (const auto resource: resources) {
for (uint32_t resource = 0; resource < program.info.buffers.size(); ++resource) {
const auto& metadata = program.info.buffers[resource];
if (metadata.descriptor_index == UINT32_MAX) continue;
auto& source = prepared.buffer_sources[metadata.descriptor_index];
const auto descriptor = DecodeNativeDescriptor<ShaderBufferResource>(snapshot.buffers[resource]);
const auto address = descriptor.Base48();
auto size = descriptor.GetSize();
if (address == 0 || size == 0) {
prepared.buffer_sources.push_back({});
source = {};
continue;
}
if (descriptor.NumRecords() == UINT32_MAX) {
@@ -853,7 +848,7 @@ void RenderExecutor::FindBuffers(std::span<PreparedBindings* const> stages) {
}
}
size = Libs::LibKernel::Memory::ClampRangeSize(address, size);
prepared.buffer_sources.push_back({address, size, cache.FindBuffer(address, size)});
source = {address, size, cache.FindBuffer(address, size)};
}
}
}
@@ -864,10 +859,9 @@ void RenderExecutor::RebindBuffers(PreparedBindings& prepared) {
const auto& program = *prepared.runtime->program;
const auto& snapshot = *prepared.runtime->resources;
const auto& layout = program.bindings;
EXIT_IF(prepared.buffer_sources.size() != layout.memory_offset_count);
EXIT_IF(prepared.buffer_sources.size() != layout.descriptor_counts[static_cast<size_t>(BindingKind::Buffers)]);
prepared.buffers.clear();
prepared.buffers.reserve(layout.memory_offset_count);
prepared.buffers.resize(layout.descriptor_counts[static_cast<size_t>(BindingKind::Buffers)]);
EXIT_IF(prepared.shader_data.size() != layout.ShaderDataDwords());
std::fill(prepared.shader_data.begin() + layout.memory_offset_dword,
prepared.shader_data.end(), 0);
@@ -876,20 +870,18 @@ void RenderExecutor::RebindBuffers(PreparedBindings& prepared) {
const auto shift = (index % 4u) * 8u;
prepared.shader_data[dword] |= offset << shift;
};
for (uint32_t i = 0; i < layout.memory_offset_count; i++) {
const auto resource = layout.descriptors.front().resources[i];
for (const auto& resource: program.info.buffers) {
const auto index = resource.descriptor_index;
if (index == UINT32_MAX) continue;
uint32_t buffer_offset = 0;
prepared.buffers.push_back(NativeStorageBuffer(m_context, prepared.buffer_sources[i],
program.info.buffers[resource],
buffer_offset));
pack_memory_offset(i, buffer_offset);
prepared.buffers[index] = NativeStorageBuffer(m_context, prepared.buffer_sources[index],
resource, buffer_offset);
pack_memory_offset(index, buffer_offset);
}
if (ShaderRecompiler::IR::FindBinding(
layout, ShaderRecompiler::IR::DescriptorBindingKind::FlattenedSrt) != nullptr) {
if (layout.descriptor_counts[static_cast<size_t>(BindingKind::FlattenedSrt)] != 0) {
prepared.flattened_srt = NativeUpload(m_context, snapshot.flattened_srt);
}
if (ShaderRecompiler::IR::FindBinding(
program.bindings, ShaderRecompiler::IR::DescriptorBindingKind::ShaderData) != nullptr) {
if (layout.descriptor_counts[static_cast<size_t>(BindingKind::ShaderData)] != 0) {
prepared.shader_data_buffer = NativeUpload(m_context, prepared.shader_data);
}
}
@@ -981,7 +973,7 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
std::span<PreparedBindings* const> prepared_bindings) {
KYTY_PROFILER_FUNCTION();
auto vk_buffer = buffer.Handle();
size_t descriptor_count = 0;
size_t image_descriptor_count = 0;
size_t write_count = 0;
ShaderRecompiler::IR::PushData push_data;
bool has_push_data = false;
@@ -995,9 +987,14 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
for (const auto* prepared: prepared_bindings) {
EXIT_IF(prepared == nullptr || prepared->runtime == nullptr || !*prepared->runtime);
const auto& program = *prepared->runtime->program;
write_count += program.bindings.descriptors.size();
for (const auto& binding: program.bindings.descriptors) {
descriptor_count += NativeDescriptorCount(binding);
write_count += std::popcount(program.bindings.descriptor_mask);
for (auto mask = program.bindings.descriptor_mask; mask != 0; mask &= mask - 1) {
const auto index = std::countr_zero(mask);
const auto count = program.bindings.descriptor_counts[index];
if (index >= ShaderRecompiler::IR::FirstImageBinding &&
index <= static_cast<size_t>(BindingKind::Samplers)) {
image_descriptor_count += count;
}
}
const auto shader_stage = NativeShaderStage(program.stage);
push_stages |= shader_stage;
@@ -1029,10 +1026,9 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
}
for (const auto* writer: prepared_bindings) {
const auto& program = *writer->runtime->program;
for (uint32_t i = 0; i < writer->buffer_sources.size(); ++i) {
const auto resource = program.bindings.descriptors.front().resources[i];
if (!program.info.buffers[resource].written) continue;
const auto& written = writer->buffer_sources[i];
for (const auto& resource: program.info.buffers) {
if (!resource.written || resource.descriptor_index == UINT32_MAX) continue;
const auto& written = writer->buffer_sources[resource.descriptor_index];
if (written.size != 0 && ImageRangeOverlaps(address, size,
written.address, written.size)) {
EXIT("scalar resource reads overlap a shader buffer write\n");
@@ -1041,12 +1037,11 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
}
}
}
m_descriptor_buffers.clear();
m_descriptor_images.clear();
m_descriptor_images.resize(image_descriptor_count);
m_descriptor_writes.clear();
m_descriptor_buffers.reserve(descriptor_count);
m_descriptor_images.reserve(descriptor_count);
m_descriptor_writes.reserve(write_count);
size_t image_cursor = 0;
std::array<vk::DescriptorBufferInfo, 2> address_buffers;
for (auto* prepared: prepared_bindings) {
const auto& program = *prepared->runtime->program;
@@ -1111,37 +1106,37 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
binding.layout = image.backing.state.layout;
}
m_image_occurrences.assign(descriptors.images.size(), 0);
for (const auto& binding: program.bindings.descriptors) {
std::array<size_t, ShaderRecompiler::IR::ImageBindingCount> image_starts {};
for (auto mask = program.bindings.descriptor_mask; mask != 0; mask &= mask - 1) {
const auto index = std::countr_zero(mask);
const auto kind = static_cast<BindingKind>(index);
vk::WriteDescriptorSet write {};
write.dstBinding = ShaderRecompiler::IR::NativeBinding(program.stage, binding.kind);
write.descriptorType = NativeDescriptorType(binding.kind);
write.descriptorCount = NativeDescriptorCount(binding);
const auto buffer_start = m_descriptor_buffers.size();
const auto image_start = m_descriptor_images.size();
if (ShaderRecompiler::IR::ImageBindingResourceClass(binding.kind) !=
write.dstBinding = ShaderRecompiler::IR::NativeBinding(program.stage, kind);
write.descriptorType = NativeDescriptorType(kind);
write.descriptorCount = program.bindings.descriptor_counts[index];
const auto image_start = image_cursor;
if (ShaderRecompiler::IR::ImageBindingResourceClass(kind) !=
ShaderRecompiler::IR::ImageResourceClass::None) {
for (const auto resource: binding.resources) {
m_descriptor_images.push_back(MakeImageInfo(
descriptors.images.at(resource), m_image_occurrences.at(resource)++));
}
image_starts[ShaderRecompiler::IR::ImageBindingIndex(kind)] = image_start;
image_cursor += write.descriptorCount;
} else {
switch (binding.kind) {
switch (kind) {
case BindingKind::Buffers:
EXIT_IF(descriptors.buffers.size() != binding.resources.size());
EXIT_IF(descriptors.buffers.size() != write.descriptorCount);
for (const auto& view: descriptors.buffers) {
EXIT_IF(view.buffer == nullptr);
m_descriptor_buffers.push_back(view);
}
write.pBufferInfo = descriptors.buffers.data();
break;
case BindingKind::BdaPagetable:
case BindingKind::FaultBuffer: {
auto& cache = m_context.GetBufferCache();
const auto* bda_buffer = binding.kind == BindingKind::BdaPagetable
auto& cache = m_context.GetBufferCache();
const auto* bda_buffer = kind == BindingKind::BdaPagetable
? cache.GetBdaPageTableBuffer()
: cache.GetFaultBuffer();
m_descriptor_buffers.emplace_back(bda_buffer->Handle(), 0,
bda_buffer->Size());
auto& view = address_buffers[kind == BindingKind::FaultBuffer];
view = {bda_buffer->Handle(), 0, bda_buffer->Size()};
write.pBufferInfo = &view;
break;
}
case BindingKind::FlattenedSrt:
@@ -1149,42 +1144,41 @@ void RenderExecutor::CommitBindings(CommandBuffer& buffer,
case BindingKind::SharedMemory:
case BindingKind::Gds: {
const vk::DescriptorBufferInfo* view = &descriptors.gds;
if (binding.kind == BindingKind::FlattenedSrt) {
if (kind == BindingKind::FlattenedSrt) {
view = &descriptors.flattened_srt;
} else if (binding.kind == BindingKind::ShaderData) {
} else if (kind == BindingKind::ShaderData) {
view = &descriptors.shader_data_buffer;
} else if (binding.kind == BindingKind::SharedMemory) {
} else if (kind == BindingKind::SharedMemory) {
view = &descriptors.shared_memory;
}
EXIT_IF(view->buffer == nullptr);
m_descriptor_buffers.push_back(*view);
write.pBufferInfo = view;
break;
}
case BindingKind::Samplers:
for (const auto resource: binding.resources) {
const auto sampler = descriptors.samplers.at(resource);
for (const auto sampler: descriptors.samplers) {
EXIT_IF(sampler == nullptr);
m_descriptor_images.emplace_back(sampler, nullptr,
vk::ImageLayout::eUndefined);
m_descriptor_images[image_cursor++] = {sampler, nullptr, vk::ImageLayout::eUndefined};
}
break;
case BindingKind::Count: EXIT("invalid descriptor binding kind");
}
}
if (m_descriptor_buffers.size() != buffer_start) {
write.pBufferInfo = m_descriptor_buffers.data() + buffer_start;
}
if (m_descriptor_images.size() != image_start) {
if (image_cursor != image_start) {
write.pImageInfo = m_descriptor_images.data() + image_start;
}
m_descriptor_writes.push_back(write);
}
for (uint32_t i = 0; i < descriptors.images.size(); i++) {
const auto expected =
descriptors.images[i].mip_views.empty()
? 1u
: static_cast<uint32_t>(descriptors.images[i].mip_views.size());
EXIT_IF(m_image_occurrences[i] != expected);
for (uint32_t i = 0; i < program.info.images.size(); ++i) {
const auto& resource = program.info.images[i];
const auto& texture = descriptors.images[i];
EXIT_IF(resource.mip_count != (texture.mip_views.empty() ? 1u : texture.mip_views.size()));
const auto kind = *ShaderRecompiler::IR::DescriptorBindingForImage(resource);
const auto start = image_starts[ShaderRecompiler::IR::ImageBindingIndex(kind)] +
resource.descriptor_index;
for (uint32_t mip = 0; mip < resource.mip_count; ++mip) {
m_descriptor_images[start + mip] = MakeImageInfo(texture, mip);
}
}
const auto shader_data_dwords = program.bindings.ShaderDataDwords();
@@ -50,8 +50,6 @@ struct PreparedBindings {
[[nodiscard]] vk::DescriptorType
NativeDescriptorType(ShaderRecompiler::IR::DescriptorBindingKind kind);
[[nodiscard]] uint32_t
NativeDescriptorCount(const ShaderRecompiler::IR::DescriptorBinding& binding);
[[nodiscard]] vk::DescriptorImageInfo MakeImageInfo(const TextureBinding& texture,
uint32_t element = 0);
@@ -617,6 +617,13 @@ PipelineCache::GraphicsPrograms PipelineCache::GetGraphicsPrograms(
ShaderParams pixel_params;
if (pixel_active) {
pixel_params = PrepareProgram(pixel_regs, sh, target_export_mapping, pixel_info);
if (!mesh_active && !tess_active && Prospero::IsRectList(user_config.GetPrimType())) {
pixel_info.parameter_mode = ShaderPixelParameterMode::Rectangle;
} else if (context.GetModeControl().provoking_vtx_last) {
pixel_info.parameter_mode = ShaderPixelParameterMode::LastVertex;
} else {
pixel_info.parameter_mode = ShaderPixelParameterMode::FirstVertex;
}
const auto& blend = context.GetBlendControl(0);
pixel_info.dual_source_blending =
blend.enable && !context.GetRenderTarget(0).info.blend_bypass &&
@@ -19,6 +19,7 @@
#include "graphics/shader/shader.h"
#include <algorithm>
#include <bit>
#include <limits>
#include <span>
#include <vector>
@@ -180,10 +181,12 @@ static vk::BlendOp GetBlendOp(uint32_t op) {
static void AddLayoutBindings(std::vector<vk::DescriptorSetLayoutBinding>& descriptor_bindings,
const ShaderRecompiler::IR::CompiledShaderInfo& program,
vk::ShaderStageFlagBits stage) {
for (const auto& binding: program.bindings.descriptors) {
for (auto mask = program.bindings.descriptor_mask; mask != 0; mask &= mask - 1) {
const auto index = std::countr_zero(mask);
const auto kind = static_cast<ShaderRecompiler::IR::DescriptorBindingKind>(index);
descriptor_bindings.push_back(
{ShaderRecompiler::IR::NativeBinding(program.stage, binding.kind),
NativeDescriptorType(binding.kind), NativeDescriptorCount(binding), stage, nullptr});
{ShaderRecompiler::IR::NativeBinding(program.stage, kind), NativeDescriptorType(kind),
program.bindings.descriptor_counts[index], stage, nullptr});
}
}
-2
View File
@@ -213,10 +213,8 @@ private:
GraphicsBindings m_graphics_bindings;
PreparedBindings m_compute_bindings;
std::vector<ImageId> m_bound_images;
std::vector<vk::DescriptorBufferInfo> m_descriptor_buffers;
std::vector<vk::DescriptorImageInfo> m_descriptor_images;
std::vector<vk::WriteDescriptorSet> m_descriptor_writes;
std::vector<uint32_t> m_image_occurrences;
friend class CommandProcessor;
friend struct RenderExecutorTestAccess;
@@ -20,7 +20,6 @@
#include "graphics/host_gpu/vulkanCommon.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "graphics/shader/shader.h"
#include "kernel/eventQueue.h"
#include "kernel/pthread.h"
@@ -198,8 +197,8 @@ bool RenderExecutor::TryConsumeComputeImageClear(const ShaderComputeInputInfo& i
static void BindSharedMemory(RenderContext& context, ShaderComputeInputInfo& input,
PreparedBindings& bindings, uint64_t indirect_args = 0) {
if (ShaderRecompiler::IR::FindBinding(input.stage.program->bindings,
ShaderRecompiler::IR::DescriptorBindingKind::SharedMemory) == nullptr) {
if (input.stage.program->bindings.descriptor_counts[
static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::SharedMemory)] == 0) {
return;
}
auto& cache = context.GetBufferCache();
@@ -8,7 +8,6 @@
#include "graphics/shader/recompiler/frontend/decode/ShaderDecoder.h"
#include "graphics/shader/recompiler/frontend/translate/Translate.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "graphics/shader/recompiler/ir/passes/ConstantPropagation.h"
#include "graphics/shader/recompiler/ir/passes/DeadCodeElimination.h"
#include "graphics/shader/recompiler/ir/passes/ReadLaneElimination.h"
@@ -670,20 +669,17 @@ CompileResult CompileProgram(TranslateResult translated, const CompileOptions& o
IR::EliminateDeadCode(ir.blocks);
IR::CollectShaderInfo(ir, options.input_info);
IR::AllocateBindings(ir, push_data_start_dword,
ir.stage == ShaderType::Compute && options.input_info.compute != nullptr &&
options.input_info.compute->lds_storage);
std::string ir_dump;
if (options.dump_ir) {
ir_dump = MakeIrDump(translated.cfg_dump, ir);
if (options.early_dump) {
LOGF("%s native IR and bindings (early):\n%s", GetDumpLabel(options), ir_dump.c_str());
LOGF("%s native IR (early):\n%s", GetDumpLabel(options), ir_dump.c_str());
}
}
LOGF("%s phase begin: stage=%s hash=0x%016" PRIx64 " SPIR-V EmitProgram\n",
GetDumpLabel(options), StageName(ir.stage), ir.shader_hash);
auto spirv = Spirv::EmitProgram(ir, options.input_info);
auto spirv = Spirv::EmitProgram(ir, options.input_info, push_data_start_dword);
LOGF("%s phase end: stage=%s hash=0x%016" PRIx64 " SPIR-V EmitProgram words=%" PRIu64
" elapsed_ms=%" PRIu64 "\n",
GetDumpLabel(options), StageName(ir.stage), ir.shader_hash,
@@ -171,10 +171,12 @@ void Builder::AddFunction(std::span<const uint32_t> words) {
}
DeferredPhi Builder::AddDeferredPhi(uint32_t type, uint32_t result, size_t incoming_count) {
std::vector<uint32_t> words {spv::OpPhi, type, result};
words.resize(words.size() + incoming_count * 2u);
const DeferredPhi phi {m_functions.size()};
AddFunction(words);
const auto word_count = static_cast<uint32_t>(3u + incoming_count * 2u);
m_functions.push_back((word_count << spv::WordCountShift) | spv::OpPhi);
m_functions.push_back(type);
m_functions.push_back(result);
m_functions.resize(m_functions.size() + incoming_count * 2u);
m_unpatched_phi_incomings += incoming_count;
return phi;
}
@@ -3,7 +3,6 @@
#include "common/assert.h"
#include "graphics/shader/recompiler/backend/spirv/spirvEmitterInternal.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include <algorithm>
#include <array>
@@ -19,92 +18,14 @@ namespace {
}
void ValidateNativeProgram(const IR::Program& program, bool lds_storage) {
using Kind = IR::DescriptorBindingKind;
constexpr auto KindCount = static_cast<size_t>(Kind::Count);
std::array<std::vector<uint32_t>, KindCount> expected;
std::array<bool, KindCount> present {};
const auto Dense = [](size_t size) {
std::vector<uint32_t> values(size);
for (uint32_t i = 0; i < values.size(); i++) {
values[i] = i;
}
return values;
uint64_t live_buffers = 0;
bool uses_lds = false;
bool uses_gds = false;
const auto uses_mapping = [](const auto& resource) {
return resource.indirect_root != UINT32_MAX;
};
auto Expect = [&](Kind kind, std::vector<uint32_t> resources = {}) {
const auto index = static_cast<size_t>(kind);
present[index] = true;
expected[index] = std::move(resources);
};
for (uint32_t i = 0; i < program.info.images.size(); i++) {
const auto kind = IR::DescriptorBindingForImage(program.info.images[i]);
if (!kind.has_value()) {
Fail(program, "native shader plan has an invalid image class");
}
present[static_cast<size_t>(*kind)] = true;
const auto dynamic = program.info.images[i].mip_mode == IR::ImageMipMode::Dynamic;
const auto count = dynamic ? program.info.images[i].mip_count : 1u;
if (count == 0u || (!dynamic && program.info.images[i].mip_count != 1u)) {
Fail(program, "native shader plan has an invalid image mip descriptor count");
}
expected[static_cast<size_t>(*kind)].insert(expected[static_cast<size_t>(*kind)].end(),
count, i);
}
if (!program.info.samplers.empty()) {
Expect(Kind::Samplers, Dense(program.info.samplers.size()));
}
auto& buffers = expected[static_cast<size_t>(Kind::Buffers)];
const auto shared = IR::CollectMemoryResources(program, buffers);
present[static_cast<size_t>(Kind::Buffers)] = !buffers.empty();
if (shared.gds) {
Expect(Kind::Gds);
}
if (shared.lds && lds_storage) {
Expect(Kind::SharedMemory);
}
if (program.info.uses_dma) {
Expect(Kind::BdaPagetable);
Expect(Kind::FaultBuffer);
}
if (IR::UsesFlattenedSrt(program)) {
Expect(Kind::FlattenedSrt);
}
if (program.bindings.ShaderDataDwords() != 0 && !program.bindings.UsesPushData()) {
Expect(Kind::ShaderData);
}
std::array<bool, KindCount> seen {};
for (const auto& binding: program.bindings.descriptors) {
const auto kind = static_cast<size_t>(binding.kind);
if (kind >= KindCount || seen[kind] || !present[kind] ||
binding.resources != expected[kind]) {
Fail(program, "native descriptor groups do not match shader topology");
}
seen[kind] = true;
}
for (size_t i = 0; i < KindCount; i++) {
if (present[i] != seen[i]) {
Fail(program, "native shader plan is missing a required descriptor group");
}
}
const auto has_shader_data_storage = present[static_cast<size_t>(Kind::ShaderData)];
const auto shader_data_dwords = program.bindings.ShaderDataDwords();
const auto user_data_dwords = program.bindings.user_data_registers.size();
const bool has_dispatch_threads = program.bindings.dispatch_thread_dword != IR::PushData::NoStart;
if ((program.bindings.UsesPushData() &&
!IR::PushData::CanFit(program.bindings.push_data_start_dword, shader_data_dwords)) ||
(has_dispatch_threads && (program.stage != ShaderType::Compute ||
program.bindings.dispatch_thread_dword != user_data_dwords)) ||
program.bindings.memory_offset_dword != user_data_dwords + (has_dispatch_threads ? 3u : 0u) ||
program.bindings.memory_offset_count != buffers.size() ||
has_shader_data_storage != (shader_data_dwords != 0 && !program.bindings.UsesPushData()) ||
!std::is_sorted(program.bindings.user_data_registers.begin(),
program.bindings.user_data_registers.end()) ||
std::adjacent_find(program.bindings.user_data_registers.begin(),
program.bindings.user_data_registers.end()) !=
program.bindings.user_data_registers.end()) {
Fail(program, "native shader-data layout is inconsistent");
}
bool uses_flattened_srt = std::ranges::any_of(program.info.buffers, uses_mapping) ||
std::ranges::any_of(program.info.images, uses_mapping);
const auto planning_only_handle = [&](const IR::Inst& handle) {
return !handle.Uses().empty() &&
std::ranges::all_of(handle.Uses(), [&](const IR::Use& use) {
@@ -141,6 +62,39 @@ void ValidateNativeProgram(const IR::Program& program, bool lds_storage) {
};
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
const auto op = inst.GetOpcode();
uses_flattened_srt |= op == IR::ValueOpcode::ReadConst;
if (IR::BufferAccessOf(op) != IR::BufferAccess::None ||
IR::SharedAccessOf(op) != IR::SharedAccess::None ||
IR::AddressOpcodeInfoOf(op).access != IR::AddressAccess::None) {
const auto index = inst.Flags<IR::MemoryFlags>().index;
if (index >= program.memory_info.size()) {
Fail(program, "typed shader contains invalid memory metadata");
}
const auto& memory = program.memory_info[index];
if (!memory.planning_only) {
uses_lds |= memory.kind == IR::ResourceKind::FlatLocal;
if (IR::SharedAccessOf(op) != IR::SharedAccess::None) {
if (memory.kind != IR::ResourceKind::Lds && memory.kind != IR::ResourceKind::Gds) {
Fail(program, "typed shader contains invalid shared-memory metadata");
}
uses_gds |= memory.kind == IR::ResourceKind::Gds;
uses_lds |= memory.kind == IR::ResourceKind::Lds;
} else if (memory.kind == IR::ResourceKind::Buffer ||
memory.kind == IR::ResourceKind::ScalarBuffer) {
if (memory.resource >= program.info.buffers.size()) {
Fail(program, "typed shader contains an invalid buffer resource");
}
live_buffers |= uint64_t{1} << memory.resource;
for (const auto child: program.info.buffers[memory.resource].indirect_resources) {
if (child >= program.info.buffers.size()) {
Fail(program, "typed shader contains an invalid indirect buffer resource");
}
live_buffers |= uint64_t{1} << child;
}
}
}
}
const auto dense = inst.Flags<uint32_t>();
switch (inst.GetOpcode()) {
case IR::ValueOpcode::GetBufferResource:
@@ -186,163 +140,64 @@ void ValidateNativeProgram(const IR::Program& program, bool lds_storage) {
}
}
}
using Kind = IR::DescriptorBindingKind;
std::array<uint32_t, static_cast<size_t>(Kind::Count)> expected {};
auto& buffer_count = expected[static_cast<size_t>(Kind::Buffers)];
for (uint32_t index = 0; index < program.info.buffers.size(); ++index) {
const auto slot = (live_buffers & (uint64_t{1} << index)) != 0 ? buffer_count++ : UINT32_MAX;
if (program.info.buffers[index].descriptor_index != slot) {
Fail(program, "native buffer descriptor index does not match shader topology");
}
}
for (const auto& image: program.info.images) {
const auto kind = IR::DescriptorBindingForImage(image);
if (!kind.has_value()) {
Fail(program, "native shader plan has an invalid image class");
}
if (image.mip_count == 0u ||
(image.mip_mode != IR::ImageMipMode::Dynamic && image.mip_count != 1u)) {
Fail(program, "native shader plan has an invalid image mip descriptor count");
}
auto& count = expected[static_cast<size_t>(*kind)];
if (image.descriptor_index != count) {
Fail(program, "native image descriptor index does not match shader topology");
}
count += image.mip_count;
}
expected[static_cast<size_t>(Kind::Samplers)] = static_cast<uint32_t>(program.info.samplers.size());
expected[static_cast<size_t>(Kind::Gds)] = uses_gds;
expected[static_cast<size_t>(Kind::SharedMemory)] = uses_lds && lds_storage;
expected[static_cast<size_t>(Kind::BdaPagetable)] = program.info.uses_dma;
expected[static_cast<size_t>(Kind::FaultBuffer)] = program.info.uses_dma;
expected[static_cast<size_t>(Kind::FlattenedSrt)] = uses_flattened_srt;
expected[static_cast<size_t>(Kind::ShaderData)] =
program.bindings.ShaderDataDwords() != 0 && !program.bindings.UsesPushData();
uint64_t descriptor_mask = 0;
for (uint32_t index = 0; index < expected.size(); ++index) {
if (expected[index] != 0) descriptor_mask |= uint64_t{1} << index;
}
if (program.bindings.descriptor_counts != expected ||
program.bindings.descriptor_mask != descriptor_mask) {
Fail(program, "native descriptor counts do not match shader topology");
}
const auto shader_data_dwords = program.bindings.ShaderDataDwords();
const auto& registers = program.info.user_data_registers;
const bool has_dispatch_threads = program.bindings.dispatch_thread_dword != IR::PushData::NoStart;
if ((program.bindings.UsesPushData() &&
!IR::PushData::CanFit(program.bindings.push_data_start_dword, shader_data_dwords)) ||
(has_dispatch_threads && (program.stage != ShaderType::Compute ||
program.bindings.dispatch_thread_dword != registers.size())) ||
program.bindings.memory_offset_dword != registers.size() + (has_dispatch_threads ? 3u : 0u) ||
!std::is_sorted(registers.begin(), registers.end()) ||
std::adjacent_find(registers.begin(), registers.end()) != registers.end()) {
Fail(program, "native shader-data layout is inconsistent");
}
}
} // namespace
Emitter::SpirvRequirements Emitter::AnalyzeProgramRequirements(const IR::Program& program) {
SpirvRequirements requirements {};
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
requirements.float64 |= inst.GetType() == IR::Type::F64;
if (IR::BufferAccessOf(inst.GetOpcode()) == IR::BufferAccess::Atomic &&
inst.GetType() == IR::Type::U64) {
requirements.buffer_int64_atomics = true;
}
const auto address_access = IR::AddressOpcodeInfoOf(inst.GetOpcode()).access;
if (address_access != IR::AddressAccess::None) {
const auto memory_index = inst.Flags<IR::MemoryFlags>().index;
if (memory_index >= program.memory_info.size()) {
Fail(program, "address operation has invalid memory metadata");
}
const auto& memory = program.memory_info[memory_index];
const auto kind = memory.kind;
if (kind == IR::ResourceKind::Scratch || kind == IR::ResourceKind::FlatLocal) {
if (program.scratch_dwords == 0) {
Fail(program, "scratch operation has no per-thread storage");
}
requirements.function_scratch = true;
if (kind == IR::ResourceKind::FlatLocal && program.stage != ShaderType::Compute &&
program.stage != ShaderType::Mesh) {
requirements.function_lds = true;
}
} else if (address_access == IR::AddressAccess::Write) {
Fail(program, "writable FLAT/GLOBAL addresses require GPU ownership tracking");
} else {
// Descriptor stores can alias coherent physical-address loads.
requirements.coherent_buffers |= memory.coherent;
}
}
if (IR::BufferAccessOf(inst.GetOpcode()) != IR::BufferAccess::None) {
const auto memory_index = inst.Flags<IR::MemoryFlags>().index;
if (memory_index >= program.memory_info.size()) {
Fail(program, "buffer operation has invalid memory metadata");
}
const auto& memory = program.memory_info[memory_index];
if (memory.kind == IR::ResourceKind::IndirectBuffer &&
inst.GetOpcode() != IR::ValueOpcode::ReadConstBuffer) {
requirements.subgroup_local_invocation_id = true;
}
if (memory.kind == IR::ResourceKind::Buffer) {
requirements.coherent_buffers |= memory.coherent;
if (memory.resource >= program.info.buffers.size()) {
Fail(program, "buffer operation has invalid resource metadata");
}
const auto bits = StorageBufferElementBits(program, memory);
requirements.buffer_u8 |= bits == 8u;
requirements.buffer_u16 |= bits == 16u;
if ((program.info.buffers[memory.resource].packed_stride & (1u << 20u)) != 0u) {
if (program.stage != ShaderType::Compute) {
Fail(program, "buffer ADD_TID is only valid for compute shaders");
}
requirements.subgroup_local_invocation_id = true;
}
}
}
const auto shared_access = IR::SharedAccessOf(inst.GetOpcode());
if (shared_access != IR::SharedAccess::None) {
const auto index = inst.Flags<IR::MemoryFlags>().index;
if (index >= program.memory_info.size()) {
Fail(program, "shared operation has invalid memory metadata");
}
const auto kind = program.memory_info[index].kind;
if (kind != IR::ResourceKind::Lds && kind != IR::ResourceKind::Gds) {
Fail(program, "shared operation has invalid resource kind");
}
if (shared_access == IR::SharedAccess::Atomic &&
IR::SharedComponentCount(inst.GetOpcode()) == 2u) {
if (kind != IR::ResourceKind::Lds || program.stage != ShaderType::Compute) {
Fail(program, "64-bit shared atomics require compute LDS");
}
requirements.shared_int64_atomics = true;
}
if (program.stage != ShaderType::Compute && program.stage != ShaderType::Mesh &&
kind == IR::ResourceKind::Lds) {
requirements.function_lds = true;
}
if (shared_access == IR::SharedAccess::Append ||
shared_access == IR::SharedAccess::Consume) {
requirements.subgroup_ballot = true;
requirements.subgroup_shuffle = true;
requirements.subgroup_local_invocation_id = true;
}
}
switch (inst.GetOpcode()) {
case IR::ValueOpcode::StoreCompletion: requirements.subgroup_barrier = true; break;
case IR::ValueOpcode::BvhIntersect: requirements.bvh = true; break;
case IR::ValueOpcode::ConditionRef:
requirements.subgroup_ballot |=
inst.Flags<CFG::BranchCondition>() != CFG::BranchCondition::ScalarInstruction;
break;
case IR::ValueOpcode::Ballot: requirements.subgroup_ballot = true; break;
case IR::ValueOpcode::DppMoveU32:
case IR::ValueOpcode::ReadFirstLane:
case IR::ValueOpcode::ReadLane: {
requirements.subgroup_ballot = true;
requirements.subgroup_shuffle = true;
if (inst.GetOpcode() == IR::ValueOpcode::DppMoveU32) {
requirements.subgroup_local_invocation_id = true;
}
break;
}
case IR::ValueOpcode::DppUpdateU32:
case IR::ValueOpcode::WriteLane: {
requirements.subgroup_ballot = true;
requirements.subgroup_local_invocation_id = true;
break;
}
case IR::ValueOpcode::Permlane16U32: {
requirements.subgroup_ballot = true;
requirements.subgroup_shuffle = true;
requirements.subgroup_local_invocation_id = true;
break;
}
case IR::ValueOpcode::SwizzleU32:
case IR::ValueOpcode::PermuteU32:
case IR::ValueOpcode::BpermuteU32: {
requirements.subgroup_ballot = true;
requirements.subgroup_shuffle = true;
requirements.subgroup_local_invocation_id = true;
break;
}
case IR::ValueOpcode::LaneId:
requirements.subgroup_local_invocation_id |=
program.stage != ShaderType::TessellationControl;
break;
case IR::ValueOpcode::ImageQueryLod: requirements.compute_derivatives = true; break;
case IR::ValueOpcode::ImageGatherRaw:
requirements.image_gather_extended = true;
break;
case IR::ValueOpcode::SetAttribute: {
const auto index = inst.Flags<IR::ExportFlags>().index;
if (index >= program.export_info.size()) {
Fail(program, "attribute export has invalid metadata");
}
if (program.stage == ShaderType::Pixel &&
program.export_info[index].vm) {
requirements.pixel_valid_mask = true;
}
break;
}
default: break;
}
}
}
return requirements;
}
std::vector<uint32_t> EmitProgram(const IR::Program& program,
ShaderStageInputInfo input_info) {
std::vector<uint32_t> EmitProgram(IR::Program& program, ShaderStageInputInfo input_info,
uint32_t push_data_start_dword) {
using namespace Emitter;
if (program.stage != ShaderType::Compute && program.stage != ShaderType::Vertex &&
@@ -352,12 +207,14 @@ std::vector<uint32_t> EmitProgram(const IR::Program& program,
Fail(program, "binary SPIR-V emitter received an unsupported shader stage");
}
if (!program.srt_plan_complete || !program.resource_tracking_complete ||
!program.shader_info_complete || !program.binding_layout_complete) {
!program.shader_info_complete) {
Fail(program, "SPIR-V emitter requires a fully planned native shader program");
}
ValidateNativeProgram(program, program.stage == ShaderType::Compute &&
input_info.compute != nullptr && input_info.compute->lds_storage);
if (program.info.buffers.size() > IR::ShaderInfo::MaxBuffers) {
Fail(program, "native shader exceeds the buffer resource limit");
}
IR::ValidateProgram(program, true);
program.bindings = {.push_data_start_dword = push_data_start_dword};
EmitterState state(program, input_info);
const auto* workgroup = ShaderWorkgroupInput(program.stage, input_info);
state.lane_count =
@@ -365,6 +222,8 @@ std::vector<uint32_t> EmitProgram(const IR::Program& program,
? 2u
: 1u;
DefineModule(state);
ValidateNativeProgram(program, program.stage == ShaderType::Compute &&
input_info.compute != nullptr && input_info.compute->lds_storage);
EmitProgram(state);
state.builder.AddEntryPoint(ExecutionModelForStage(state.program.stage), state.main_func,
"main", state.interface_variables);
@@ -9,8 +9,9 @@
namespace Libs::Graphics::ShaderRecompiler::Spirv {
std::vector<uint32_t> EmitProgram(const IR::Program& program,
ShaderStageInputInfo input_info);
// Resets and writes backend IDs and descriptor indices; do not emit the same program concurrently.
std::vector<uint32_t> EmitProgram(IR::Program& program, ShaderStageInputInfo input_info,
uint32_t push_data_start_dword = 0);
} // namespace Libs::Graphics::ShaderRecompiler::Spirv
@@ -56,33 +56,14 @@ uint32_t EmitMulHigh(EmitterState& state, uint32_t lhs, uint32_t rhs, bool signe
return signed_value ? Unary(state, spv::OpBitcast, TypeU32(state), high) : high;
}
uint32_t EmitMinMax3(EmitterState& state, uint32_t a, uint32_t b, uint32_t c, bool signed_value,
bool max_value) {
const auto ab = signed_value ? EmitMinMaxI32Value(state, a, b, max_value)
: EmitMinMaxU32Value(state, a, b, max_value);
return signed_value ? EmitMinMaxI32Value(state, ab, c, max_value)
: EmitMinMaxU32Value(state, ab, c, max_value);
}
uint32_t EmitMed3(EmitterState& state, uint32_t a, uint32_t b, uint32_t c, bool signed_value) {
const auto minimum = EmitMinMax3(state, a, b, c, signed_value, false);
const auto maximum = EmitMinMax3(state, a, b, c, signed_value, true);
const auto ab = Binary(state, spv::OpIAdd, TypeU32(state), a, b);
const auto abc = Binary(state, spv::OpIAdd, TypeU32(state), ab, c);
return Binary(state, spv::OpISub, TypeU32(state),
Binary(state, spv::OpISub, TypeU32(state), abc, minimum), maximum);
}
uint32_t EmitFMinMax3(EmitterState& state, uint32_t a, uint32_t b, uint32_t c, bool max_value) {
return EmitMinMaxF32Value(state, EmitMinMaxF32Value(state, a, b, max_value), c, max_value);
}
uint32_t EmitExt(EmitterState& state, uint32_t type, uint32_t opcode,
std::initializer_list<uint32_t> args) {
const auto result = state.builder.AllocateId();
std::vector<uint32_t> words {spv::OpExtInst, type, result, GlslStd450(state), opcode};
words.insert(words.end(), args.begin(), args.end());
state.builder.AddFunction(words);
const auto result = state.builder.AllocateId();
state.builder.AddFunction(spv::OpExtInst, type, result, GlslStd450(state), opcode, args);
return result;
}
@@ -248,14 +229,6 @@ uint32_t EmitUMulHi(EmitterState& state, uint32_t arg0, uint32_t arg1) {
return EmitMulHigh(state, arg0, arg1, false);
}
uint32_t EmitIAbs32(EmitterState& state, uint32_t arg0) {
const auto value = arg0;
const auto neg = Unary(state, spv::OpSNegate, TypeU32(state), value);
const auto negative =
Binary(state, spv::OpSLessThan, TypeBool(state), value, ConstantU32(state, 0));
return Select(state, TypeU32(state), negative, neg, value);
}
uint32_t EmitBitCount64(EmitterState& state, uint32_t arg0) {
// Vulkan bit-count operands remain 32-bit, including when counting a U64.
const auto unpacked = Unary(state, spv::OpBitcast, TypeU32Vector(state, 2), arg0);
@@ -264,58 +237,30 @@ uint32_t EmitBitCount64(EmitterState& state, uint32_t arg0) {
return Binary(state, spv::OpIAdd, TypeU32(state), pair.low, pair.high);
}
uint32_t EmitFindILsb32(EmitterState& state, uint32_t arg0) {
const auto value = EmitExt(state, TypeI32(state), GLSLstd450FindILsb, {arg0});
return Unary(state, spv::OpBitcast, TypeU32(state), value);
}
uint32_t EmitFindUMsb32(EmitterState& state, uint32_t arg0) {
const auto value = EmitExt(state, TypeI32(state), GLSLstd450FindUMsb, {arg0});
return Unary(state, spv::OpBitcast, TypeU32(state), value);
}
uint32_t EmitSMin32(EmitterState& state, uint32_t arg0, uint32_t arg1) {
return EmitMinMaxI32Value(state, arg0, arg1, false);
}
uint32_t EmitSMax32(EmitterState& state, uint32_t arg0, uint32_t arg1) {
return EmitMinMaxI32Value(state, arg0, arg1, true);
}
uint32_t EmitUMin32(EmitterState& state, uint32_t arg0, uint32_t arg1) {
return EmitMinMaxU32Value(state, arg0, arg1, false);
}
uint32_t EmitUMax32(EmitterState& state, uint32_t arg0, uint32_t arg1) {
return EmitMinMaxU32Value(state, arg0, arg1, true);
}
uint32_t EmitSMinTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMinMax3(state, arg0, arg1, arg2, true, false);
return EmitSMin32(state, arg0, EmitSMin32(state, arg1, arg2));
}
uint32_t EmitSMaxTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMinMax3(state, arg0, arg1, arg2, true, true);
return EmitSMax32(state, arg0, EmitSMax32(state, arg1, arg2));
}
uint32_t EmitUMinTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMinMax3(state, arg0, arg1, arg2, false, false);
return EmitUMin32(state, arg0, EmitUMin32(state, arg1, arg2));
}
uint32_t EmitUMaxTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMinMax3(state, arg0, arg1, arg2, false, true);
return EmitUMax32(state, arg0, EmitUMax32(state, arg1, arg2));
}
uint32_t EmitSMedTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMed3(state, arg0, arg1, arg2, true);
const auto high_min = EmitSMin32(state, EmitSMax32(state, arg0, arg1), arg2);
return EmitSMax32(state, EmitSMin32(state, arg0, arg1), high_min);
}
uint32_t EmitUMedTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2) {
return EmitMed3(state, arg0, arg1, arg2, false);
}
uint32_t EmitFPIsNan32(EmitterState& state, uint32_t arg0) {
return EmitNative<spv::OpFUnordNotEqual, IR::Type::U1>(state, arg0, arg0);
const auto high_min = EmitUMin32(state, EmitUMax32(state, arg0, arg1), arg2);
return EmitUMax32(state, EmitUMin32(state, arg0, arg1), high_min);
}
uint32_t EmitFPMin32(EmitterState& state, uint32_t arg0, uint32_t arg1) {
@@ -82,35 +82,12 @@ uint32_t EmitF32ToF16RtzBits(EmitterState& state, uint32_t f32) {
return EmitAndConstant(state, EmitSelectValueU32(state, exp_eq_255, special, finite2), 0xffffu);
}
uint32_t EmitMinMaxU32Value(EmitterState& state, uint32_t lhs, uint32_t rhs, bool max_value) {
const auto cond = state.builder.AllocateId();
const auto ret = state.builder.AllocateId();
state.builder.AddFunction(max_value ? spv::OpUGreaterThan : spv::OpULessThan, TypeBool(state),
cond, lhs, rhs);
state.builder.AddFunction(spv::OpSelect, TypeU32(state), ret, cond, lhs, rhs);
return ret;
}
uint32_t EmitMinMaxI32Value(EmitterState& state, uint32_t lhs, uint32_t rhs, bool max_value) {
const auto cond = state.builder.AllocateId();
const auto ret = state.builder.AllocateId();
state.builder.AddFunction(max_value ? spv::OpSGreaterThan : spv::OpSLessThan, TypeBool(state),
cond, lhs, rhs);
state.builder.AddFunction(spv::OpSelect, TypeU32(state), ret, cond, lhs, rhs);
return ret;
}
F32Class EmitClassifyF32Bits(EmitterState& state, uint32_t bits) {
F32Class cls;
cls.bits = bits;
const auto abs_bits = EmitAndConstant(state, cls.bits, 0x7fffffffu);
const auto exponent_bits = EmitAndConstant(state, abs_bits, 0x7f800000u);
const auto mantissa_bits = EmitAndConstant(state, abs_bits, 0x007fffffu);
const auto exponent_max =
EmitCompareU32Constant(state, spv::OpIEqual, exponent_bits, 0x7f800000u);
const auto mantissa_nonzero = EmitCompareU32Constant(state, spv::OpINotEqual, mantissa_bits, 0);
cls.nan = EmitLogicalAndBool(state, exponent_max, mantissa_nonzero);
cls.zero = EmitCompareU32Constant(state, spv::OpIEqual, abs_bits, 0);
cls.bits = bits;
const auto abs_bits = EmitAndConstant(state, cls.bits, 0x7fffffffu);
cls.nan = EmitCompareU32Constant(state, spv::OpUGreaterThan, abs_bits, 0x7f800000u);
cls.zero = EmitCompareU32Constant(state, spv::OpIEqual, abs_bits, 0);
return cls;
}
@@ -59,21 +59,6 @@ uint32_t ConstantU32(EmitterState& state, uint32_t value);
std::abort();
}
uint32_t ResourceForDescriptor(const EmitterState& state, IR::DescriptorBindingKind kind,
uint32_t resource) {
const auto* descriptor = IR::FindBinding(state.program.bindings, kind);
if (descriptor == nullptr) {
ExitDescriptorBindingFailure(state, kind, resource, "descriptor group was not allocated");
}
const auto found =
std::find(descriptor->resources.begin(), descriptor->resources.end(), resource);
if (found == descriptor->resources.end()) {
ExitDescriptorBindingFailure(state, kind, resource,
"resource is absent from descriptor group");
}
return static_cast<uint32_t>(found - descriptor->resources.begin());
}
uint32_t DescriptorElementPointer(EmitterState& state, uint32_t result_ptr_type,
uint32_t variable_id, uint32_t array_index,
IR::DescriptorBindingKind kind, uint32_t resource,
@@ -156,22 +141,22 @@ uint32_t LoadImageDescriptor(EmitterState& state, uint32_t resource, uint32_t mi
uint32_t array_index) {
const auto pointer = ImageDescriptorPointer(state, resource, mip, array_index);
const auto image = state.builder.AllocateId();
state.builder.AddFunction(spv::OpLoad, ImageType(state, state.program.info.images.at(resource)),
const auto kind = *IR::DescriptorBindingForImage(state.program.info.images[resource]);
state.builder.AddFunction(spv::OpLoad, state.images[IR::ImageBindingIndex(kind)].type,
image, pointer);
return image;
}
uint32_t LoadSamplerDescriptor(EmitterState& state, uint32_t sampler) {
const auto array_index =
ResourceForDescriptor(state, IR::DescriptorBindingKind::Samplers, sampler);
const auto sampler_type = state.builder.Type(spv::OpTypeSampler);
const auto pointer_type =
state.builder.Type(spv::OpTypePointer, spv::StorageClassUniformConstant, sampler_type);
if (sampler >= state.program.info.samplers.size()) {
ExitDescriptorBindingFailure(state, IR::DescriptorBindingKind::Samplers, sampler,
"sampler resource index is out of range");
}
const auto pointer = DescriptorElementPointer(
state, pointer_type, state.sampler_variable, ConstantU32(state, array_index),
state, state.sampler_pointer_type, state.sampler_variable, ConstantU32(state, sampler),
IR::DescriptorBindingKind::Samplers, sampler, "sampler descriptor array was not emitted");
const auto sampler_id = state.builder.AllocateId();
state.builder.AddFunction(spv::OpLoad, sampler_type, sampler_id, pointer);
state.builder.AddFunction(spv::OpLoad, state.sampler_type, sampler_id, pointer);
return sampler_id;
}
@@ -181,8 +166,12 @@ uint32_t MakeSampledImage(EmitterState& state, uint32_t resource, uint32_t sampl
EXIT_IF(image_resource.resource_class != IR::ImageResourceClass::Sampled);
const auto image = LoadImageDescriptor(state, resource, mip, array_index);
const auto sampled_image = state.builder.AllocateId();
const auto sampled_type =
state.builder.Type(spv::OpTypeSampledImage, ImageType(state, image_resource));
const auto kind = *IR::DescriptorBindingForImage(image_resource);
auto& sampled_type = state.images[IR::ImageBindingIndex(kind)].sampled_type;
if (sampled_type == 0) {
sampled_type = state.builder.Type(
spv::OpTypeSampledImage, state.images[IR::ImageBindingIndex(kind)].type);
}
state.builder.AddFunction(spv::OpSampledImage, sampled_type, sampled_image, image, sampler_id);
if (array_index != 0u) {
state.builder.RequireExtension("SPV_EXT_descriptor_indexing");
@@ -200,12 +189,11 @@ uint32_t ImageDescriptorPointer(EmitterState& state, uint32_t resource, uint32_t
const auto kind = IR::DescriptorBindingForImage(image);
EXIT_IF(!kind.has_value());
if (array_index == 0u) {
array_index = ConstantU32(state, ResourceForDescriptor(state, *kind, resource) + mip);
array_index = ConstantU32(state, image.descriptor_index + mip);
}
const auto pointer_type = state.builder.Type(
spv::OpTypePointer, spv::StorageClassUniformConstant, ImageType(state, image));
const auto variable = state.image_variables[IR::ImageBindingIndex(*kind)];
return DescriptorElementPointer(state, pointer_type, variable, array_index, *kind, resource,
const auto& definition = state.images[IR::ImageBindingIndex(*kind)];
return DescriptorElementPointer(state, definition.pointer_type, definition.variable,
array_index, *kind, resource,
"image descriptor array was not emitted");
}
@@ -223,7 +223,7 @@ uint32_t Boxes(EmitterState& s, uint32_t first, uint32_t second, uint32_t half,
} // namespace
void DefineBvhIntersect(EmitterState& s) {
if (!s.requirements.bvh) return;
if (!s.program.info.bvh) return;
const auto u = TypeU32(s), f = TypeF32(s), b = TypeBool(s), wide = TypeU64(s);
const auto vec4 = TypeU32Vector(s, 4), vec3 = TypeF32Vector(s, 3);
const auto signature = s.builder.Type(spv::OpTypeFunction, vec4, vec4, wide, f, vec3, vec3, vec3);
@@ -10,7 +10,7 @@ namespace {
bool UserDataDwordIndex(const EmitterState& state, IR::ScalarReg reg, uint32_t& dword_index) {
const auto register_index = IR::RegIndex(reg);
const auto& registers = state.program.bindings.user_data_registers;
const auto& registers = state.program.info.user_data_registers;
const auto found = std::lower_bound(registers.begin(), registers.end(), register_index);
if (found == registers.end() || *found != register_index) {
return false;
@@ -177,6 +177,13 @@ uint32_t EmitAttribute(EmitterState& state, uint32_t attr, uint32_t chan) {
return value;
};
if (input->per_vertex) {
if (PixelParameterIsFlat(state, attr)) {
const auto value = load_per_vertex(
state.input_info.pixel->parameter_mode == ShaderPixelParameterMode::LastVertex ? 2u : 0u);
const auto bits = state.builder.AllocateId();
state.builder.AddFunction(spv::OpBitcast, TypeU32(state), bits, value);
return bits;
}
const auto barycentric_kind = state.input_info.pixel->ps_no_perspective
? IR::StageInputKind::BaryCoordNoPerspective
: IR::StageInputKind::BaryCoordSmooth;
@@ -231,8 +238,12 @@ uint32_t EmitInterpolationParameter(ValueEmitContext& ctx, uint32_t attr, uint32
return value;
};
const auto selected_vertex = (mode + 1u) % 3u;
uint32_t value = load_vertex(selected_vertex);
uint32_t selected_vertex = (mode + 1u) % 3u;
if (mode == 2u && PixelParameterIsFlat(state, attr)) {
selected_vertex =
state.input_info.pixel->parameter_mode == ShaderPixelParameterMode::LastVertex ? 2u : 0u;
}
uint32_t value = load_vertex(selected_vertex);
if (!PixelParameterIsCustom(state, attr) && mode < 2u) {
const auto delta = state.builder.AllocateId();
state.builder.AddFunction(spv::OpFSub, TypeF32(state), delta, value, load_vertex(0));
@@ -293,50 +304,46 @@ uint32_t ExportVector(ValueEmitContext& ctx, uint32_t data, const IR::ExportInfo
f32[0], f32[1], f32[2], f32[3]);
return vector;
}
uint32_t raw[4] = {
ConstantU32(state, 0),
ConstantU32(state, 0),
ConstantU32(state, 0),
ConstantU32(state, uint_output ? 1u : 0x3f800000u),
};
if (exp.compr) {
for (uint32_t pair = 0; pair < 2u; pair++) {
if ((exp.en & (3u << (pair * 2u))) == 0u) {
continue;
}
const auto packed = ExportRawComponent(ctx, data, pair);
for (uint32_t lane = 0; lane < 2u; lane++) {
const auto component = pair * 2u + lane;
if (((exp.en >> component) & 1u) == 0u) {
if (exp.compr || exp.en != 0xfu) {
uint32_t raw[4] = {
ConstantU32(state, 0),
ConstantU32(state, 0),
ConstantU32(state, 0),
ConstantU32(state, uint_output ? 1u : 0x3f800000u),
};
if (exp.compr) {
for (uint32_t pair = 0; pair < 2u; pair++) {
if ((exp.en & (3u << (pair * 2u))) == 0u) {
continue;
}
raw[component] = state.builder.AllocateId();
state.builder.AddFunction(spv::OpBitFieldUExtract, TypeU32(state), raw[component],
packed, ConstantU32(state, lane * 16u),
ConstantU32(state, 16));
const auto packed = ExportRawComponent(ctx, data, pair);
for (uint32_t lane = 0; lane < 2u; lane++) {
const auto component = pair * 2u + lane;
if (((exp.en >> component) & 1u) == 0u) {
continue;
}
raw[component] = state.builder.AllocateId();
state.builder.AddFunction(spv::OpBitFieldUExtract, TypeU32(state), raw[component],
packed, ConstantU32(state, lane * 16u),
ConstantU32(state, 16));
}
}
}
} else {
for (uint32_t component = 0; component < 4u; component++) {
if (((exp.en >> component) & 1u) != 0u) {
raw[component] = ExportRawComponent(ctx, data, component);
} else {
for (uint32_t component = 0; component < 4u; component++) {
if (((exp.en >> component) & 1u) != 0u) {
raw[component] = ExportRawComponent(ctx, data, component);
}
}
}
data = state.builder.AllocateId();
state.builder.AddFunction(spv::OpCompositeConstruct, TypeU32Vector(state, 4), data,
raw[0], raw[1], raw[2], raw[3]);
}
if (uint_output) {
const auto vector = state.builder.AllocateId();
state.builder.AddFunction(spv::OpCompositeConstruct, TypeU32Vector(state, 4), vector,
raw[0], raw[1], raw[2], raw[3]);
return vector;
}
uint32_t f32[4] {};
for (uint32_t component = 0; component < 4u; component++) {
f32[component] = state.builder.AllocateId();
state.builder.AddFunction(spv::OpBitcast, TypeF32(state), f32[component], raw[component]);
return data;
}
const auto vector = state.builder.AllocateId();
state.builder.AddFunction(spv::OpCompositeConstruct, TypeF32Vector(state, 4), vector, f32[0],
f32[1], f32[2], f32[3]);
state.builder.AddFunction(spv::OpBitcast, TypeF32Vector(state, 4), vector, data);
return vector;
}
@@ -461,7 +468,7 @@ void EmitSetAttribute(ValueEmitContext& ctx, const IR::Inst& inst) {
auto& state = ctx.state;
const auto& exp = ctx.Export(inst);
const auto exec = ctx.Arg(inst, 1);
if (state.program.stage == ShaderType::Pixel && exp.vm && state.requirements.pixel_valid_mask &&
if (state.program.stage == ShaderType::Pixel && exp.vm && state.program.info.pixel_valid_mask &&
state.pixel_valid_mask_variable != 0) {
const auto value = state.builder.AllocateId();
state.builder.AddFunction(spv::OpSelect, TypeU32(state), value, exec, ConstantU32(state, 1),
@@ -752,14 +752,11 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
const auto EmitGather = [&](uint32_t mip) {
const auto sampled = MakeSampledImage(state, mem.resource, sampler_id, mip);
const auto sample = state.builder.AllocateId();
std::vector<uint32_t> words {
static_cast<uint32_t>(dref ? spv::OpImageDrefGather : spv::OpImageGather), result_type,
sample, sampled, coord, component_or_dref};
if (operand_mask != 0u) {
words.push_back(operand_mask);
words.push_back(offset);
}
state.builder.AddFunction(words);
const std::array operands {operand_mask, offset};
state.builder.AddFunction(
dref ? spv::OpImageDrefGather : spv::OpImageGather, result_type,
sample, sampled, coord, component_or_dref,
std::span(operands).first(operand_mask != 0u ? 2u : 0u));
return sample;
};
const auto sample = image.mip_mode == IR::ImageMipMode::Dynamic && image.mip_count > 1u
@@ -791,24 +788,25 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
dref_value = AddressF32(ctx, mem, *address, layout.dref);
}
}
uint32_t operand_mask = 0;
std::vector<uint32_t> operands;
std::array<uint32_t, 3> operands {};
size_t operand_count = 0;
if (HasFlag(mem, Decoder::ImageSampleFlagDerivative)) {
operand_mask |= spv::ImageOperandsGradMask;
operands.push_back(
CoordF32(ctx, mem, *address, layout.grad_x, dimension_info.spatial_components));
operands.push_back(
CoordF32(ctx, mem, *address, layout.grad_y, dimension_info.spatial_components));
operands[0] = spv::ImageOperandsGradMask;
operands[1] = CoordF32(ctx, mem, *address, layout.grad_x, dimension_info.spatial_components);
operands[2] = CoordF32(ctx, mem, *address, layout.grad_y, dimension_info.spatial_components);
operand_count = 3;
} else if (explicit_lod) {
operand_mask |= spv::ImageOperandsLodMask;
auto lod = ZeroF32(state);
if (HasFlag(mem, Decoder::ImageSampleFlagLod) && layout.lod != NoImageComponent) {
lod = AddressF32(ctx, mem, *address, layout.lod);
}
operands.push_back(lod);
operands[0] = spv::ImageOperandsLodMask;
operands[1] = lod;
operand_count = 2;
} else if (layout.bias != NoImageComponent) {
operand_mask |= spv::ImageOperandsBiasMask;
operands.push_back(AddressF32(ctx, mem, *address, layout.bias));
operands[0] = spv::ImageOperandsBiasMask;
operands[1] = AddressF32(ctx, mem, *address, layout.bias);
operand_count = 2;
}
const auto sampler_id = LoadSamplerDescriptor(state, mem.sampler);
const auto EmitSample = [&](uint32_t resource, uint32_t array_index = 0u) {
@@ -818,16 +816,10 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
ImageDimensionInfoFor(candidate.dimension).coordinate_components,
candidate.cube);
const auto sampled = MakeSampledImage(state, resource, sampler_id, 0u, array_index);
const auto sample = state.builder.AllocateId();
std::vector<uint32_t> sample_operands {result_type, sample, sampled, coord};
if (dref) {
sample_operands.push_back(dref_value);
}
if (operand_mask != 0u) {
sample_operands.push_back(operand_mask);
sample_operands.insert(sample_operands.end(), operands.begin(), operands.end());
}
state.builder.AddFunction(opcode, sample_operands);
const auto sample = state.builder.AllocateId();
state.builder.AddFunction(opcode, result_type, sample, sampled, coord,
std::span(&dref_value, dref ? 1u : 0u),
std::span<const uint32_t>(operands).first(operand_count));
return sample;
};
if (image.indirect_root != mem.resource) {
@@ -867,7 +859,6 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
const auto resource = image.indirect_resources[ordinal];
const auto& candidate = state.program.info.images[resource];
const auto kind = *IR::DescriptorBindingForImage(candidate);
const auto& binding = *IR::FindBinding(state.program.bindings, kind);
if (!runs.empty()) {
auto& run = runs.back();
const auto& first = state.program.info.images[run.resource];
@@ -875,17 +866,16 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
if (candidate.dimension == first.dimension && candidate.cube == first.cube &&
IR::DescriptorBindingForImage(first) == kind &&
candidate.mip_count == 1u && first.mip_count == 1u &&
(ordinal == 1u || (next_slot < binding.resources.size() &&
binding.resources[next_slot] == resource))) {
(ordinal == 1u || candidate.descriptor_index == next_slot)) {
// Native roots precede appended children; only the root may have a slot gap.
if (ordinal == 1u)
run.slot_bias = ResourceForDescriptor(state, kind, resource) - ordinal;
run.slot_bias = candidate.descriptor_index - ordinal;
++run.count;
continue;
}
}
runs.push_back({ordinal, 1u, resource,
ResourceForDescriptor(state, kind, resource) - ordinal});
candidate.descriptor_index - ordinal});
}
const auto EmitRun = [&](uint32_t index) {
const auto& run = runs[index];
@@ -893,8 +883,7 @@ void EmitImage(ValueEmitContext& ctx, const IR::Inst& inst) {
auto slot = Binary(state, spv::OpIAdd, TypeU32(state), selected,
ConstantU32(state, run.slot_bias));
if (run.first == 0u) {
const auto kind = *IR::DescriptorBindingForImage(image);
const auto root_slot = ResourceForDescriptor(state, kind, mem.resource);
const auto root_slot = image.descriptor_index;
if (root_slot != run.slot_bias) {
const auto is_root = Binary(state, spv::OpIEqual, TypeBool(state), selected,
ConstantU32(state, 0u));
@@ -23,13 +23,8 @@ uint32_t EmitConvertF16F32(EmitterState& state, uint32_t arg0);
inline constexpr auto EmitConvertF32F16 = EmitF16BitsToF32;
uint32_t EmitConvertS32F32(EmitterState& state, uint32_t arg0);
uint32_t EmitConvertU32F32(EmitterState& state, uint32_t arg0);
template <IR::Type type>
uint32_t EmitConvertSigned32ToFloat(EmitterState& state, uint32_t arg0) {
const auto signed_value = Unary(state, spv::OpBitcast, TypeI32(state), arg0);
return EmitNative<spv::OpConvertSToF, type>(state, signed_value);
}
inline constexpr auto EmitConvertF32S32 = EmitConvertSigned32ToFloat<IR::Type::F32>;
inline constexpr auto EmitConvertF64S32 = EmitConvertSigned32ToFloat<IR::Type::F64>;
EMIT_NATIVE(ConvertF32S32, OpConvertSToF, F32, uint32_t)
EMIT_NATIVE(ConvertF64S32, OpConvertSToF, F64, uint32_t)
uint32_t EmitConvertF32F64(EmitterState& state, uint32_t arg0);
uint32_t EmitConvertF64F32(EmitterState& state, uint32_t arg0);
EMIT_NATIVE(ConvertF32U32, OpConvertUToF, F32, uint32_t)
@@ -69,7 +64,7 @@ EMIT_NATIVE(IMul64, OpIMul, U64, uint32_t, uint32_t)
EMIT_NATIVE(IAddCarry32, OpIAddCarry, U32x2, uint32_t, uint32_t)
uint32_t EmitSMulHi(EmitterState& state, uint32_t arg0, uint32_t arg1);
uint32_t EmitUMulHi(EmitterState& state, uint32_t arg0, uint32_t arg1);
uint32_t EmitIAbs32(EmitterState& state, uint32_t arg0);
inline constexpr auto EmitIAbs32 = EmitGlsl<GLSLstd450SAbs, IR::Type::U32, uint32_t>;
EMIT_NATIVE(ShiftLeftLogical32, OpShiftLeftLogical, U32, uint32_t, uint32_t)
EMIT_NATIVE(ShiftRightLogical32, OpShiftRightLogical, U32, uint32_t, uint32_t)
EMIT_NATIVE(ShiftRightArithmetic32, OpShiftRightArithmetic, U32, uint32_t, uint32_t)
@@ -96,13 +91,13 @@ EMIT_NATIVE(BitwiseAnd64, OpBitwiseAnd, U64, uint32_t, uint32_t)
EMIT_NATIVE(BitReverse32, OpBitReverse, U32, uint32_t)
EMIT_NATIVE(BitCount32, OpBitCount, U32, uint32_t)
uint32_t EmitBitCount64(EmitterState& state, uint32_t arg0);
uint32_t EmitFindILsb32(EmitterState& state, uint32_t arg0);
uint32_t EmitFindUMsb32(EmitterState& state, uint32_t arg0);
inline constexpr auto EmitFindILsb32 = EmitGlsl<GLSLstd450FindILsb, IR::Type::U32, uint32_t>;
inline constexpr auto EmitFindUMsb32 = EmitGlsl<GLSLstd450FindUMsb, IR::Type::U32, uint32_t>;
uint32_t EmitFindUMsb64(EmitterState& state, uint32_t arg0);
uint32_t EmitSMin32(EmitterState& state, uint32_t arg0, uint32_t arg1);
uint32_t EmitSMax32(EmitterState& state, uint32_t arg0, uint32_t arg1);
uint32_t EmitUMin32(EmitterState& state, uint32_t arg0, uint32_t arg1);
uint32_t EmitUMax32(EmitterState& state, uint32_t arg0, uint32_t arg1);
inline constexpr auto EmitSMin32 = EmitGlsl<GLSLstd450SMin, IR::Type::U32, uint32_t, uint32_t>;
inline constexpr auto EmitSMax32 = EmitGlsl<GLSLstd450SMax, IR::Type::U32, uint32_t, uint32_t>;
inline constexpr auto EmitUMin32 = EmitGlsl<GLSLstd450UMin, IR::Type::U32, uint32_t, uint32_t>;
inline constexpr auto EmitUMax32 = EmitGlsl<GLSLstd450UMax, IR::Type::U32, uint32_t, uint32_t>;
uint32_t EmitSMinTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2);
uint32_t EmitSMaxTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2);
uint32_t EmitUMinTri32(EmitterState& state, uint32_t arg0, uint32_t arg1, uint32_t arg2);
@@ -162,7 +157,7 @@ inline constexpr auto EmitFPOrdLessThanEqual32 = EmitFloatCompare32<spv::OpFOrdL
inline constexpr auto EmitFPUnordLessThanEqual32 = EmitFloatCompare32<spv::OpFUnordLessThanEqual>;
inline constexpr auto EmitFPOrdGreaterThanEqual32 = EmitFloatCompare32<spv::OpFOrdGreaterThanEqual>;
inline constexpr auto EmitFPUnordGreaterThanEqual32 = EmitFloatCompare32<spv::OpFUnordGreaterThanEqual>;
uint32_t EmitFPIsNan32(EmitterState& state, uint32_t arg0);
EMIT_NATIVE(FPIsNan32, OpIsNan, U1, uint32_t)
inline constexpr auto EmitFPCmpClass32 = EmitClassMaskF32;
inline constexpr auto EmitFPCmpClass16 = EmitClassMaskF16;
EMIT_NATIVE(FPAdd32, OpFAdd, F32, uint32_t, uint32_t)
@@ -6,7 +6,6 @@
#include "graphics/shader/recompiler/BufferFormat.h"
#include "graphics/shader/recompiler/backend/spirv/SpirvBuilder.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
#include <algorithm>
@@ -58,42 +57,36 @@ constexpr std::array<ImageDimensionInfo, 7> ImageDimensions {{
const ImageDimensionInfo& ImageDimensionInfoFor(ImageDimension dimension);
struct SpirvRequirements {
bool bvh = false;
bool subgroup_ballot = false;
bool subgroup_barrier = false;
bool subgroup_shuffle = false;
bool subgroup_local_invocation_id = false;
bool compute_derivatives = false;
bool image_gather_extended = false;
bool function_lds = false;
bool function_scratch = false;
bool pixel_valid_mask = false;
bool buffer_int64_atomics = false;
bool buffer_u8 = false;
bool buffer_u16 = false;
bool shared_int64_atomics = false;
bool coherent_buffers = false;
bool float64 = false;
struct BufferDefinition {
uint32_t variable = 0;
uint32_t type = 0;
uint32_t element_type = 0;
uint32_t pointer_type = 0;
uint32_t element_pointer_type = 0;
};
SpirvRequirements AnalyzeProgramRequirements(const IR::Program& program);
struct ImageDefinition {
uint32_t variable = 0;
uint32_t type = 0;
uint32_t pointer_type = 0;
uint32_t sampled_type = 0;
};
struct EmitterState {
EmitterState(const IR::Program& program_, ShaderStageInputInfo input_info_)
EmitterState(IR::Program& program_, ShaderStageInputInfo input_info_)
: builder(program_.stage == ShaderType::Mesh ? 0x00010400u : 0x00010300u),
program(program_), input_info(input_info_),
requirements(AnalyzeProgramRequirements(program_)) {
program(program_), input_info(input_info_) {
if (ShaderWorkgroupInput(program.stage, input_info) != nullptr) {
lds_storage_class = spv::StorageClassWorkgroup;
}
if (IR::FindBinding(program.bindings, IR::DescriptorBindingKind::SharedMemory) != nullptr) {
if (program.info.uses_lds && program.stage == ShaderType::Compute &&
input_info.compute != nullptr && input_info.compute->lds_storage) {
lds_storage_class = spv::StorageClassStorageBuffer;
}
}
Builder builder;
const IR::Program& program;
IR::Program& program;
ShaderStageInputInfo input_info;
uint32_t void_type = 0;
uint32_t bool_type = 0;
@@ -113,13 +106,9 @@ struct EmitterState {
uint32_t tess_inner_variable = 0;
uint32_t tess_patch_base = 0;
const SpirvRequirements requirements;
uint32_t lane_count = 1;
uint32_t lane_half = 0;
uint32_t storage_buffer_variable = 0;
uint32_t storage_buffer_u8_variable = 0;
uint32_t storage_buffer_u16_variable = 0;
uint32_t storage_buffer_u64_variable = 0;
std::array<BufferDefinition, 4> storage_buffers {};
std::array<uint32_t, IR::ShaderInfo::MaxBuffers> memory_byte_offsets {};
uint32_t bda_pagetable_variable = 0;
uint32_t fault_buffer_variable = 0;
@@ -135,8 +124,10 @@ struct EmitterState {
uint32_t lds_variable = 0;
uint32_t lds_u64_variable = 0;
std::array<uint32_t, 2> scratch_variable {};
std::array<uint32_t, IR::ImageBindingCount> image_variables {};
std::array<ImageDefinition, IR::ImageBindingCount> images {};
uint32_t sampler_variable = 0;
uint32_t sampler_type = 0;
uint32_t sampler_pointer_type = 0;
uint32_t main_func = 0;
uint32_t mesh_guest_func = 0;
uint32_t mesh_allocation = 0;
@@ -160,7 +151,6 @@ struct EmitterState {
std::vector<InputBinding> inputs;
std::vector<OutputBinding> outputs;
std::vector<uint32_t> interface_variables;
std::unordered_map<const IR::Block*, uint32_t> labels;
};
uint32_t TypeVoid(EmitterState& state);
@@ -180,8 +170,6 @@ uint32_t TypeI32Vector(EmitterState& state, uint32_t components);
uint32_t TypeF32Vector(EmitterState& state, uint32_t components);
uint32_t TypePointer(EmitterState& state, spv::StorageClass storage_class, uint32_t pointee);
uint32_t TypeFunction(EmitterState& state);
uint32_t TypeStorageBufferElement(EmitterState& state, uint32_t bits);
uint32_t TypeStorageBufferPointer(EmitterState& state, uint32_t bits = 32);
uint32_t TypeStorageBufferElementPointer(EmitterState& state, uint32_t bits = 32);
uint32_t TypePhysicalU32Pointer(EmitterState& state);
uint32_t TypePushConstantElementPointer(EmitterState& state);
@@ -255,12 +243,10 @@ struct ValueEmitContext {
const IR::Inst* ImageAddress(IR::Value value);
const IR::MemoryInfo& Memory(const IR::Inst& inst) const;
const IR::ExportInfo& Export(const IR::Inst& inst) const;
uint32_t Label(const IR::Block* block) const;
[[noreturn]] void Fail(const char* reason) const;
[[noreturn]] void Fail(const IR::Inst& inst, const char* reason) const;
EmitterState& state;
std::unordered_map<const IR::Inst*, uint32_t> definitions;
const std::unordered_map<const IR::Inst*, uint32_t>* dispatcher_spills = nullptr;
std::unordered_map<const IR::Inst*, std::pair<uint32_t, uint32_t>> dispatcher_block_loads;
uint32_t scratch_u32_variable = 0;
@@ -316,9 +302,6 @@ uint32_t EmitSubgroupLocalInvocationId(EmitterState& state);
IR::DescriptorBindingKind kind, uint32_t resource,
const char* reason);
uint32_t ResourceForDescriptor(const EmitterState& state, IR::DescriptorBindingKind kind,
uint32_t resource);
uint32_t DescriptorElementPointer(EmitterState& state, uint32_t result_ptr_type,
uint32_t variable_id, uint32_t array_index,
IR::DescriptorBindingKind kind, uint32_t resource,
@@ -413,7 +396,6 @@ uint32_t EmitShaderDataDwordLoad(EmitterState& state, uint32_t dword_index);
uint32_t StorageBufferPackedStride(const EmitterState& state, const IR::MemoryInfo& mem);
Prospero::BufferFormat StorageBufferFormat(const EmitterState& state, const IR::MemoryInfo& mem);
uint32_t StorageBufferElementBits(const IR::Program& program, const IR::MemoryInfo& mem);
void EmitMemoryOffsets(EmitterState& state);
@@ -435,7 +417,7 @@ MemoryResourceAccess PrepareMemoryResourceAccess(EmitterState& state, const IR::
MemoryResourceAccess PrepareStorageBufferResourceAccess(EmitterState& state,
const IR::MemoryInfo& mem,
uint32_t variable, uint32_t pointer_type);
BufferDefinition& buffer);
uint32_t EmitMemoryElementInBounds(EmitterState& state, const MemoryResourceAccess& access,
uint32_t index);
@@ -489,10 +471,6 @@ uint32_t EmitSubConstantMinusU32(EmitterState& state, uint32_t constant, uint32_
uint32_t EmitF32ToF16RtzBits(EmitterState& state, uint32_t f32);
uint32_t EmitMinMaxU32Value(EmitterState& state, uint32_t lhs, uint32_t rhs, bool max_value);
uint32_t EmitMinMaxI32Value(EmitterState& state, uint32_t lhs, uint32_t rhs, bool max_value);
inline constexpr auto EmitBitcastF32ToU32 = EmitNative<spv::OpBitcast, IR::Type::U32, uint32_t>;
inline constexpr auto EmitBitcastU32ToF32 = EmitNative<spv::OpBitcast, IR::Type::F32, uint32_t>;
@@ -251,8 +251,7 @@ uint32_t LoadBda(ValueEmitContext& ctx, uint32_t address, uint32_t active, uint3
uint32_t ByteAddress(ValueEmitContext& ctx, const IR::Inst& inst, const IR::MemoryInfo& mem) {
if (mem.kind == IR::ResourceKind::Buffer) {
const auto resource = ResourceForDescriptor(ctx.state, IR::DescriptorBindingKind::Buffers,
mem.resource);
const auto resource = ctx.state.program.info.buffers.at(mem.resource).descriptor_index;
const auto prefix = ctx.state.memory_byte_offsets[resource];
const auto address = Binary(ctx.state, spv::OpIAdd, TypeU32(ctx.state),
BufferByteAddress(ctx, inst, mem), prefix);
@@ -333,7 +332,8 @@ uint32_t LoadSubwordInBounds(ValueEmitContext& ctx, const MemoryResourceAccess&
uint32_t value;
if (resource.element_bits < 32u) {
const auto loaded = ctx.state.builder.AllocateId();
ctx.state.builder.AddFunction(spv::OpLoad, TypeStorageBufferElement(ctx.state, bits), loaded,
ctx.state.builder.AddFunction(spv::OpLoad,
ctx.state.storage_buffers[bits == 8u ? 0 : 1].element_type, loaded,
EmitMemoryElementPointer(ctx.state, resource, index),
resource.memory_access);
value = Unary(ctx.state, spv::OpUConvert, TypeU32(ctx.state), loaded);
@@ -438,7 +438,8 @@ void StoreSubwordInBounds(ValueEmitContext& ctx, const IR::MemoryInfo& mem,
if (resource.element_bits < 32u) {
ctx.state.builder.AddFunction(
spv::OpStore, pointer,
Unary(ctx.state, spv::OpUConvert, TypeStorageBufferElement(ctx.state, bits), data),
Unary(ctx.state, spv::OpUConvert,
ctx.state.storage_buffers[bits == 8u ? 0 : 1].element_type, data),
resource.memory_access);
return;
}
@@ -1200,8 +1201,8 @@ uint32_t EmitBufferAtomic64(ValueEmitContext& ctx, const IR::Inst& inst) {
auto& state = ctx.state;
return EmitValueOrDefaultIfCondition(
state, ctx.Arg(inst, inst.NumArgs() - 1), TypeU64(state), ConstantU64(state, 0), [&]() {
const auto resource = PrepareStorageBufferResourceAccess(
state, mem, state.storage_buffer_u64_variable, TypeStorageBufferPointer(state, 64));
const auto resource =
PrepareStorageBufferResourceAccess(state, mem, state.storage_buffers[3]);
const auto byte_address = ByteAddress(ctx, inst, mem);
const auto index = Binary(state, spv::OpShiftRightLogical, TypeU32(state), byte_address,
ConstantU32(state, 3u));
@@ -102,23 +102,15 @@ Prospero::BufferFormat StorageBufferFormat(const EmitterState& state, const IR::
return state.program.info.buffers[mem.resource].descriptor_format;
}
uint32_t StorageBufferElementBits(const IR::Program& program, const IR::MemoryInfo& mem) {
if (!mem.formatted) return mem.data_bits;
const auto format = mem.typed ? Format::DecodeTBufferFormat(mem.data_format, mem.number_format)
: program.info.buffers[mem.resource].descriptor_format;
const auto info = Format::GetFormatInfo(format);
return info.type == Format::ComponentType::Unknown || info.packed_bitfield
? 32u : info.component_bits[0];
}
void EmitMemoryOffsets(EmitterState& state) {
for (uint32_t i = 0; i < state.program.bindings.memory_offset_count; i++) {
const auto count = state.program.bindings.descriptor_counts[static_cast<size_t>(IR::DescriptorBindingKind::Buffers)];
for (uint32_t i = 0; i < count; i += 4u) {
const auto word =
EmitShaderDataDwordLoad(state, state.program.bindings.memory_offset_dword + i / 4u);
const auto shift = ConstantU32(state, (i % 4u) * 8u);
state.memory_byte_offsets[i] = EmitBinaryU32(
state, spv::OpBitwiseAnd, EmitBinaryU32(state, spv::OpShiftRightLogical, word, shift),
ConstantU32(state, 0xffu));
for (uint32_t lane = 0; lane < std::min(4u, count - i); ++lane) {
state.memory_byte_offsets[i + lane] = EmitNative<spv::OpBitFieldUExtract, IR::Type::U32>(
state, word, ConstantU32(state, lane * 8u), ConstantU32(state, 8u));
}
}
}
@@ -146,7 +138,7 @@ void EnsureLdsStorage(EmitterState& state) {
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationAliased);
return variable;
};
if (state.requirements.shared_int64_atomics) {
if (state.program.info.shared_int64_atomics) {
state.lds_variable = define(TypeU32(state), 4u);
state.lds_u64_variable = define(TypeU64(state), 8u);
state.builder.AddName(state.lds_u64_variable, "lds_qwords");
@@ -160,20 +152,22 @@ void EnsureLdsStorage(EmitterState& state) {
MemoryResourceAccess PrepareStorageBufferResourceAccess(EmitterState& state,
const IR::MemoryInfo& mem,
uint32_t variable,
uint32_t pointer_type) {
if (variable == 0) {
BufferDefinition& buffer) {
if (buffer.variable == 0) {
ExitDescriptorBindingFailure(state, IR::DescriptorBindingKind::Buffers, mem.resource,
"storage buffer descriptor array was not emitted");
}
if (buffer.pointer_type == 0) {
buffer.pointer_type = TypePointer(state, spv::StorageClassStorageBuffer, buffer.type);
}
const auto array_index =
ResourceForDescriptor(state, IR::DescriptorBindingKind::Buffers, mem.resource);
state.program.info.buffers.at(mem.resource).descriptor_index;
MemoryResourceAccess access {
.kind = mem.kind,
.memory_access = mem.coherent ? spv::MemoryAccessVolatileMask : spv::MemoryAccessMaskNone};
access.object_pointer = state.builder.AllocateId();
state.builder.AddFunction(spv::OpAccessChain, pointer_type, access.object_pointer, variable,
ConstantU32(state, array_index));
state.builder.AddFunction(spv::OpAccessChain, buffer.pointer_type, access.object_pointer,
buffer.variable, ConstantU32(state, array_index));
access.byte_offset = state.memory_byte_offsets[array_index];
access.length = state.builder.AllocateId();
state.builder.AddFunction(spv::OpArrayLength, TypeU32(state), access.length,
@@ -214,12 +208,14 @@ MemoryResourceAccess PrepareMemoryResourceAccess(EmitterState& state, const IR::
case IR::ResourceKind::ScalarBuffer:
case IR::ResourceKind::Buffer: {
const auto bits = mem.kind == IR::ResourceKind::Buffer
? StorageBufferElementBits(state.program, mem) : 32u;
const auto variable = bits == 8u ? state.storage_buffer_u8_variable
: bits == 16u ? state.storage_buffer_u16_variable
: state.storage_buffer_variable;
access = PrepareStorageBufferResourceAccess(
state, mem, variable, TypeStorageBufferPointer(state, bits));
? IR::StorageBufferElementBits(state.program, mem) : 32u;
auto* buffer = &state.storage_buffers[2];
if (bits == 8u) {
buffer = &state.storage_buffers[0];
} else if (bits == 16u) {
buffer = &state.storage_buffers[1];
}
access = PrepareStorageBufferResourceAccess(state, mem, *buffer);
access.element_bits = bits;
return access;
}
@@ -245,7 +241,7 @@ uint32_t EmitMemoryElementPointer(EmitterState& state, const MemoryResourceAcces
TypeStorageBufferElementPointer(state, 32));
}
const auto pointer = state.builder.AllocateId();
if (access.kind == IR::ResourceKind::Lds && state.requirements.shared_int64_atomics) {
if (access.kind == IR::ResourceKind::Lds && state.program.info.shared_int64_atomics) {
state.builder.AddFunction(spv::OpAccessChain, TypeU32ElementPointer(state, storage_class),
pointer, access.object_pointer, ConstantU32(state, 0), index);
return pointer;
@@ -120,32 +120,39 @@ uint32_t TypeFunction(EmitterState& state) {
return state.function_type;
}
uint32_t TypeStorageBufferElement(EmitterState& state, uint32_t bits) {
return bits == 32u ? TypeU32(state)
: bits == 64u ? TypeU64(state)
: state.builder.Type(spv::OpTypeInt, bits, 0);
}
uint32_t StorageRuntimeArrayType(EmitterState& state, uint32_t bits) {
return state.builder.DecoratedType(
spv::OpTypeRuntimeArray,
{{spv::OpDecorate, {spv::DecorationArrayStride, bits / 8u}}},
TypeStorageBufferElement(state, bits));
}
uint32_t StorageBufferType(EmitterState& state, uint32_t bits = 32) {
return state.builder.DecoratedType(spv::OpTypeStruct,
{{spv::OpMemberDecorate, {0, spv::DecorationOffset, 0}},
{spv::OpDecorate, {spv::DecorationBlock}}},
StorageRuntimeArrayType(state, bits));
}
uint32_t TypeStorageBufferPointer(EmitterState& state, uint32_t bits) {
return TypePointer(state, spv::StorageClassStorageBuffer, StorageBufferType(state, bits));
uint32_t StorageBufferType(EmitterState& state, BufferDefinition& buffer, uint32_t bits = 32) {
if (buffer.type == 0) {
if (bits == 32u) {
buffer.element_type = TypeU32(state);
} else if (bits == 64u) {
buffer.element_type = TypeU64(state);
} else {
buffer.element_type = state.builder.Type(spv::OpTypeInt, bits, 0);
}
const auto array = state.builder.DecoratedType(
spv::OpTypeRuntimeArray,
{{spv::OpDecorate, {spv::DecorationArrayStride, bits / 8u}}}, buffer.element_type);
buffer.type = state.builder.DecoratedType(
spv::OpTypeStruct, {{spv::OpMemberDecorate, {0, spv::DecorationOffset, 0}},
{spv::OpDecorate, {spv::DecorationBlock}}}, array);
}
return buffer.type;
}
uint32_t TypeStorageBufferElementPointer(EmitterState& state, uint32_t bits) {
return TypePointer(state, spv::StorageClassStorageBuffer, TypeStorageBufferElement(state, bits));
auto* buffer = &state.storage_buffers[2];
if (bits == 8u) {
buffer = &state.storage_buffers[0];
} else if (bits == 16u) {
buffer = &state.storage_buffers[1];
} else if (bits == 64u) {
buffer = &state.storage_buffers[3];
}
if (buffer->element_pointer_type == 0) {
buffer->element_pointer_type =
TypePointer(state, spv::StorageClassStorageBuffer, buffer->element_type);
}
return buffer->element_pointer_type;
}
uint32_t TypePhysicalU32Pointer(EmitterState& state) {
@@ -200,6 +207,43 @@ uint32_t F32ArrayType(EmitterState& state, uint32_t count) {
}
void DefineDescriptors(EmitterState& state) {
using Kind = IR::DescriptorBindingKind;
auto& info = state.program.info;
auto& bindings = state.program.bindings;
auto& counts = bindings.descriptor_counts;
auto& buffer_count = counts[static_cast<size_t>(Kind::Buffers)];
for (uint32_t index = 0; index < info.buffers.size(); ++index) {
info.buffers[index].descriptor_index = (info.live_buffers & (uint64_t{1} << index)) != 0
? buffer_count++ : UINT32_MAX;
}
std::array<const IR::ImageResource*, IR::ImageBindingCount> image_types {};
for (auto& image: info.images) {
const auto kind = IR::DescriptorBindingForImage(image);
if (!kind.has_value() || image.mip_count == 0u ||
(image.mip_mode != IR::ImageMipMode::Dynamic && image.mip_count != 1u)) {
EXIT("shader image has an invalid descriptor type or mip count");
}
auto& count = counts[static_cast<size_t>(*kind)];
image.descriptor_index = count;
if (count == 0) image_types[IR::ImageBindingIndex(*kind)] = &image;
count += image.mip_count;
}
counts[static_cast<size_t>(Kind::Samplers)] = static_cast<uint32_t>(info.samplers.size());
counts[static_cast<size_t>(Kind::Gds)] = info.uses_gds;
counts[static_cast<size_t>(Kind::SharedMemory)] =
info.uses_lds && state.lds_storage_class == spv::StorageClassStorageBuffer;
counts[static_cast<size_t>(Kind::BdaPagetable)] = info.uses_dma;
counts[static_cast<size_t>(Kind::FaultBuffer)] = info.uses_dma;
counts[static_cast<size_t>(Kind::FlattenedSrt)] = info.uses_flattened_srt;
bindings.memory_offset_dword = static_cast<uint32_t>(info.user_data_registers.size());
if (info.uses_dispatch_threads) {
bindings.dispatch_thread_dword = bindings.memory_offset_dword;
bindings.memory_offset_dword += 3u;
}
bindings.push_data_start_dword =
IR::PushData::StartFor(bindings.push_data_start_dword, bindings.ShaderDataDwords());
counts[static_cast<size_t>(Kind::ShaderData)] =
bindings.ShaderDataDwords() != 0 && !bindings.UsesPushData();
if (state.program.bindings.UsesPushData() || state.program.stage == ShaderType::Mesh) {
const auto type = PushConstantBlockType(state);
state.push_constant_variable = state.builder.DefineGlobalVariable(
@@ -207,7 +251,10 @@ void DefineDescriptors(EmitterState& state) {
state.builder.AddName(type, "BufferResource");
state.builder.AddName(state.push_constant_variable, "vsharp");
}
for (const auto& binding: state.program.bindings.descriptors) {
for (uint32_t index = 0; index < counts.size(); ++index) {
if (counts[index] == 0) continue;
bindings.descriptor_mask |= uint64_t{1} << index;
const auto kind = static_cast<Kind>(index);
const auto Define = [&](uint32_t type, const char* name,
spv::StorageClass storage = spv::StorageClassStorageBuffer) {
const auto variable =
@@ -215,69 +262,66 @@ void DefineDescriptors(EmitterState& state) {
state.builder.AddName(variable, name);
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationDescriptorSet, 0);
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationBinding,
IR::NativeBinding(state.program.stage, binding.kind));
IR::NativeBinding(state.program.stage, kind));
return variable;
};
const auto ArrayType = [&](uint32_t type) {
return state.builder.Type(
spv::OpTypeArray, type,
ConstantU32(state, static_cast<uint32_t>(binding.resources.size())));
ConstantU32(state, counts[index]));
};
switch (binding.kind) {
case IR::DescriptorBindingKind::Buffers:
state.storage_buffer_variable =
Define(ArrayType(StorageBufferType(state)), "buffers");
if (state.requirements.buffer_u8) {
state.storage_buffer_u8_variable =
Define(ArrayType(StorageBufferType(state, 8)), "buffers_u8");
}
if (state.requirements.buffer_u16) {
state.storage_buffer_u16_variable =
Define(ArrayType(StorageBufferType(state, 16)), "buffers_u16");
}
if (state.requirements.buffer_int64_atomics) {
state.storage_buffer_u64_variable =
Define(ArrayType(StorageBufferType(state, 64)), "buffers_u64");
}
for (const auto variable: {state.storage_buffer_variable, state.storage_buffer_u8_variable,
state.storage_buffer_u16_variable, state.storage_buffer_u64_variable}) {
if (variable == 0) continue;
if (state.storage_buffer_u8_variable != 0 || state.storage_buffer_u16_variable != 0 ||
state.storage_buffer_u64_variable != 0) {
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationAliased);
switch (kind) {
case IR::DescriptorBindingKind::Buffers: {
const std::array used {info.buffer_u8, info.buffer_u16, true, info.buffer_int64_atomics};
constexpr std::array names {"buffers_u8", "buffers_u16", "buffers", "buffers_u64"};
for (uint32_t alias = 0; alias < used.size(); ++alias) {
if (!used[alias]) continue;
auto& buffer = state.storage_buffers[alias];
const auto type = StorageBufferType(state, buffer, 8u << alias);
buffer.variable = Define(ArrayType(type), names[alias]);
if (info.buffer_u8 || info.buffer_u16 || info.buffer_int64_atomics) {
state.builder.AddAnnotation(spv::OpDecorate, buffer.variable, spv::DecorationAliased);
}
// RDNA2 stores publish to L2 even without GLC; every alias of the buffer
// must participate in visibility for cache-bypassing polling loads.
if (state.requirements.coherent_buffers) {
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationCoherent);
if (info.coherent_buffers) {
state.builder.AddAnnotation(spv::OpDecorate, buffer.variable, spv::DecorationCoherent);
}
}
break;
}
case IR::DescriptorBindingKind::BdaPagetable:
state.bda_pagetable_variable = Define(StorageBufferType(state, 64), "bda_pagetable");
state.bda_pagetable_variable =
Define(StorageBufferType(state, state.storage_buffers[3], 64), "bda_pagetable");
break;
case IR::DescriptorBindingKind::FaultBuffer:
state.fault_buffer_variable = Define(StorageBufferType(state), "fault_buffer");
state.fault_buffer_variable =
Define(StorageBufferType(state, state.storage_buffers[2]), "fault_buffer");
break;
case IR::DescriptorBindingKind::ShaderData:
state.shader_data_storage_variable =
Define(StorageBufferType(state), "shader_data");
Define(StorageBufferType(state, state.storage_buffers[2]), "shader_data");
break;
case IR::DescriptorBindingKind::FlattenedSrt:
state.flattened_srt_variable = Define(StorageBufferType(state), "flattened_srt");
state.flattened_srt_variable =
Define(StorageBufferType(state, state.storage_buffers[2]), "flattened_srt");
break;
case IR::DescriptorBindingKind::Samplers:
state.sampler_variable = Define(ArrayType(state.builder.Type(spv::OpTypeSampler)),
"samplers", spv::StorageClassUniformConstant);
state.sampler_type = state.builder.Type(spv::OpTypeSampler);
state.sampler_pointer_type = TypePointer(
state, spv::StorageClassUniformConstant, state.sampler_type);
state.sampler_variable = Define(ArrayType(state.sampler_type), "samplers",
spv::StorageClassUniformConstant);
break;
case IR::DescriptorBindingKind::Gds:
state.gds_variable = Define(StorageBufferType(state), "gds");
state.gds_variable = Define(StorageBufferType(state, state.storage_buffers[2]), "gds");
break;
case IR::DescriptorBindingKind::SharedMemory:
state.lds_variable = Define(StorageBufferType(state), "lds_dwords");
state.lds_variable = Define(StorageBufferType(state, state.storage_buffers[2]), "lds_dwords");
state.builder.AddAnnotation(spv::OpDecorate, state.lds_variable, spv::DecorationCoherent);
if (state.requirements.shared_int64_atomics) {
state.lds_u64_variable = Define(StorageBufferType(state, 64), "lds_qwords");
if (state.program.info.shared_int64_atomics) {
state.lds_u64_variable =
Define(StorageBufferType(state, state.storage_buffers[3], 64), "lds_qwords");
state.builder.AddAnnotation(spv::OpDecorate, state.lds_u64_variable, spv::DecorationCoherent);
for (const auto variable: {state.lds_variable, state.lds_u64_variable}) {
state.builder.AddAnnotation(spv::OpDecorate, variable, spv::DecorationAliased);
@@ -285,13 +329,16 @@ void DefineDescriptors(EmitterState& state) {
}
break;
default: {
EXIT_IF(IR::ImageBindingResourceClass(binding.kind) ==
EXIT_IF(IR::ImageBindingResourceClass(kind) ==
IR::ImageResourceClass::None);
const auto& image = state.program.info.images.at(binding.resources.front());
const auto name = "image_" + std::to_string(static_cast<uint32_t>(binding.kind));
state.image_variables[IR::ImageBindingIndex(binding.kind)] =
Define(ArrayType(ImageType(state, image)), name.c_str(),
spv::StorageClassUniformConstant);
const auto& image = *image_types[IR::ImageBindingIndex(kind)];
const auto name = "image_" + std::to_string(static_cast<uint32_t>(kind));
auto& definition = state.images[IR::ImageBindingIndex(kind)];
definition.type = ImageType(state, image);
definition.variable = Define(ArrayType(definition.type), name.c_str(),
spv::StorageClassUniformConstant);
definition.pointer_type = TypePointer(
state, spv::StorageClassUniformConstant, definition.type);
if (image.dimension == ImageDimension::Dim1D ||
image.dimension == ImageDimension::Dim1DArray) {
state.builder.RequireCapability(image.resource_class ==
@@ -335,11 +382,11 @@ uint32_t ConstantU64(EmitterState& state, uint64_t value) {
uint32_t ConstantU32CompositeZero(EmitterState& state, uint32_t components) {
EXIT_IF(components < 1u || components > 4u);
const auto zero = ConstantU32(state, 0);
const auto zero = ConstantU32(state, 0);
if (components == 1u) return zero;
std::vector<uint32_t> values(components, zero);
const std::array values {zero, zero, zero, zero};
return state.builder.Constant(spv::OpConstantComposite, TypeU32Composite(state, components),
values);
std::span(values).first(components));
}
uint32_t GlslStd450(EmitterState& state) {
@@ -437,7 +484,7 @@ void DefineInputs(EmitterState& state) {
state.inputs.push_back({{kind, 0, components, name}});
}
};
if (IR::FindBinding(state.program.bindings, IR::DescriptorBindingKind::SharedMemory) != nullptr) {
if (state.program.info.uses_lds && state.lds_storage_class == spv::StorageClassStorageBuffer) {
add_builtin(IR::StageInputKind::WorkgroupId, 3, "gl_WorkGroupID");
add_builtin(IR::StageInputKind::NumWorkgroups, 3, "gl_NumWorkGroups");
}
@@ -524,7 +571,7 @@ void DefineInputs(EmitterState& state) {
builtin);
}
}
if (state.requirements.subgroup_local_invocation_id) {
if (state.program.info.subgroup_local_invocation_id) {
const auto variable = DefineInterfaceVariable(state, TypeU32(state), spv::StorageClassInput,
"gl_SubgroupInvocationID");
state.subgroup_local_invocation_id_variable = variable;
@@ -647,10 +694,10 @@ void DefineModule(EmitterState& state) {
DefineOutputs(state);
DefineTessellationInterfaces(state);
DefineDescriptors(state);
if (state.requirements.function_lds) {
if (state.program.info.function_lds) {
state.lds_variable = state.builder.AllocateId();
}
if (state.requirements.function_scratch) {
if (state.program.info.function_scratch) {
for (uint32_t half = 0; half < state.lane_count; half++) {
state.scratch_variable[half] = state.builder.AllocateId();
}
@@ -675,11 +722,11 @@ void DefineModule(EmitterState& state) {
state.builder.RequireCapability(spv::CapabilityShader);
state.builder.RequireCapability(spv::CapabilitySignedZeroInfNanPreserve);
if (state.requirements.buffer_u8) {
if (state.program.info.buffer_u8) {
state.builder.RequireExtension("SPV_KHR_8bit_storage");
state.builder.RequireCapability(spv::CapabilityStorageBuffer8BitAccess);
}
if (state.requirements.buffer_u16) {
if (state.program.info.buffer_u16) {
state.builder.RequireCapability(spv::CapabilityStorageBuffer16BitAccess);
}
if (state.program.info.uses_dma) {
@@ -687,11 +734,11 @@ void DefineModule(EmitterState& state) {
state.builder.RequireCapability(spv::CapabilityPhysicalStorageBufferAddresses);
state.builder.RequireExtension("SPV_KHR_physical_storage_buffer");
}
if (state.requirements.buffer_int64_atomics || state.requirements.shared_int64_atomics) {
if (state.program.info.buffer_int64_atomics || state.program.info.shared_int64_atomics) {
state.builder.RequireCapability(spv::CapabilityInt64);
state.builder.RequireCapability(spv::CapabilityInt64Atomics);
}
if (state.requirements.shared_int64_atomics &&
if (state.program.info.shared_int64_atomics &&
state.lds_storage_class == spv::StorageClassWorkgroup) {
state.builder.RequireVersion(0x00010400u);
state.builder.RequireExtension("SPV_KHR_workgroup_memory_explicit_layout");
@@ -714,21 +761,21 @@ void DefineModule(EmitterState& state) {
if (InputVariableForKind(state, IR::StageInputKind::SampleId) != 0) {
state.builder.RequireCapability(spv::CapabilitySampleRateShading);
}
if (state.requirements.image_gather_extended) {
if (state.program.info.image_gather_extended) {
state.builder.RequireCapability(spv::CapabilityImageGatherExtended);
}
if (state.lane_count == 2 || state.requirements.subgroup_barrier ||
state.requirements.subgroup_ballot || state.requirements.subgroup_shuffle ||
state.requirements.subgroup_local_invocation_id) {
if (state.lane_count == 2 || state.program.info.subgroup_barrier ||
state.program.info.subgroup_ballot || state.program.info.subgroup_shuffle ||
state.program.info.subgroup_local_invocation_id) {
state.builder.RequireCapability(spv::CapabilityGroupNonUniform);
}
if (state.lane_count == 2 || state.requirements.subgroup_ballot) {
if (state.lane_count == 2 || state.program.info.subgroup_ballot) {
state.builder.RequireCapability(spv::CapabilityGroupNonUniformBallot);
}
if (state.requirements.subgroup_shuffle) {
if (state.program.info.subgroup_shuffle) {
state.builder.RequireCapability(spv::CapabilityGroupNonUniformShuffle);
}
if (state.requirements.compute_derivatives && state.program.stage == ShaderType::Compute) {
if (state.program.info.compute_derivatives && state.program.stage == ShaderType::Compute) {
state.builder.RequireCapability(spv::CapabilityComputeDerivativeGroupQuadsKHR);
state.builder.RequireExtension("SPV_KHR_compute_shader_derivatives");
}
@@ -751,7 +798,7 @@ void DefineModule(EmitterState& state) {
// contract prevents host compilers from treating synthesized IEEE values as finite.
state.builder.AddExecutionMode(state.main_func, spv::ExecutionModeSignedZeroInfNanPreserve,
32u);
if (state.requirements.float64) {
if (state.program.info.float64) {
EXIT_NOT_IMPLEMENTED(state.program.stage == ShaderType::Compute &&
state.input_info.compute->float_mode != 0xc0);
// Use native rounding for MODE=0xc0, consistent with ordinary FP32 arithmetic.
@@ -763,8 +810,8 @@ void DefineModule(EmitterState& state) {
// state.builder.AddExecutionMode(state.main_func, spv::ExecutionModeDenormPreserve, 64u);
}
if (const auto* cs = ShaderWorkgroupInput(state.program.stage, state.input_info)) {
uint32_t local_x = state.requirements.compute_derivatives ? 2u : 1u;
uint32_t local_y = state.requirements.compute_derivatives ? 2u : 1u;
uint32_t local_x = state.program.info.compute_derivatives ? 2u : 1u;
uint32_t local_y = state.program.info.compute_derivatives ? 2u : 1u;
uint32_t local_z = 1u;
local_x = cs->threads_num[0] != 0u ? cs->threads_num[0] : local_x;
local_y = cs->threads_num[1] != 0u ? cs->threads_num[1] : local_y;
@@ -772,7 +819,7 @@ void DefineModule(EmitterState& state) {
if (state.lane_count == 2) {
local_x = ((local_x * local_y * local_z + 63u) / 64u) * 32u;
local_y = local_z = 1u;
if (state.requirements.compute_derivatives) {
if (state.program.info.compute_derivatives) {
local_y = local_x / 2u;
local_x = 2u;
}
@@ -791,14 +838,14 @@ void DefineModule(EmitterState& state) {
state.builder.AddExecutionMode(state.main_func, spv::ExecutionModeEarlyFragmentTests);
}
}
if (state.requirements.compute_derivatives && state.program.stage == ShaderType::Compute) {
if (state.program.info.compute_derivatives && state.program.stage == ShaderType::Compute) {
state.builder.AddExecutionMode(state.main_func, spv::ExecutionModeDerivativeGroupQuadsKHR);
}
state.builder.AddName(state.main_func, "main");
if (state.requirements.function_lds) {
if (state.program.info.function_lds) {
state.builder.AddName(state.lds_variable, "lds_dwords");
}
if (state.requirements.function_scratch) {
if (state.program.info.function_scratch) {
for (uint32_t half = 0; half < state.lane_count; half++) {
state.builder.AddName(state.scratch_variable[half], "scratch_dwords");
}
@@ -83,63 +83,50 @@ void StoreDispatcherPhiEdge(ValueEmitContext& ctx, const DispatcherFunctionState
}
}
const IR::Block* TargetBlock(const IR::Program& program, uint32_t id) {
const auto found = std::ranges::find_if(
program.block_info, [&](const IR::BlockInfo& info) { return info.id == id; });
if (found == program.block_info.end()) {
return nullptr;
}
return program.blocks[static_cast<size_t>(found - program.block_info.begin())];
}
void EmitReturn(ValueEmitContext& ctx) {
EmitKillIfPixelValidMaskInactive(ctx.state);
ctx.state.builder.AddFunction(spv::OpReturn);
}
void EmitStructuredTerminator(ValueEmitContext& ctx, const IR::Block* block,
const IR::BlockInfo& info) {
const auto& program = ctx.state.program;
const auto& term = info.terminator;
void EmitStructuredTerminator(ValueEmitContext& ctx, const IR::Block* block) {
const auto& term = block->terminator;
const auto emit_merge = [&]() {
if (term.loop_header) {
const auto* merge = TargetBlock(program, term.merge_block);
const auto* cont = TargetBlock(program, term.continue_block);
const auto* merge = term.merge_block;
const auto* cont = term.continue_block;
if (merge != nullptr && cont != nullptr) {
ctx.state.builder.AddFunction(spv::OpLoopMerge, ctx.Label(merge), ctx.Label(cont),
spv::LoopControlMaskNone);
ctx.state.builder.AddFunction(spv::OpLoopMerge, merge->Definition(),
cont->Definition(), spv::LoopControlMaskNone);
}
} else if (term.kind == CFG::TerminatorKind::ConditionalBranch &&
term.merge_block != UINT32_MAX) {
if (const auto* merge = TargetBlock(program, term.merge_block); merge != nullptr) {
ctx.state.builder.AddFunction(spv::OpSelectionMerge, ctx.Label(merge),
spv::SelectionControlMaskNone);
}
term.merge_block != nullptr) {
ctx.state.builder.AddFunction(spv::OpSelectionMerge, term.merge_block->Definition(),
spv::SelectionControlMaskNone);
}
};
switch (term.kind) {
case CFG::TerminatorKind::Branch: {
const auto* target = TargetBlock(program, term.true_block);
const auto* target = term.true_block;
if (target == nullptr) {
EmitReturn(ctx);
return;
}
emit_merge();
ctx.state.builder.AddFunction(spv::OpBranch, ctx.Label(target));
ctx.state.builder.AddFunction(spv::OpBranch, target->Definition());
return;
}
case CFG::TerminatorKind::ConditionalBranch: {
const auto* true_block = TargetBlock(program, term.true_block);
const auto* false_block = TargetBlock(program, term.false_block);
if (true_block == nullptr || false_block == nullptr || info.condition.IsEmpty()) {
const auto* true_block = term.true_block;
const auto* false_block = term.false_block;
if (true_block == nullptr || false_block == nullptr || block->condition.IsEmpty()) {
EmitReturn(ctx);
return;
}
const auto condition = ctx.Def(info.condition);
const auto condition = ctx.Def(block->condition);
emit_merge();
ctx.state.builder.AddFunction(spv::OpBranchConditional, condition,
ctx.Label(true_block), ctx.Label(false_block));
true_block->Definition(), false_block->Definition());
return;
}
default: EmitReturn(ctx); return;
@@ -147,8 +134,7 @@ void EmitStructuredTerminator(ValueEmitContext& ctx, const IR::Block* block,
}
void EmitDispatcherTarget(ValueEmitContext& ctx, const DispatcherFunctionState& dispatcher,
const IR::Block* from, uint32_t target) {
const auto* block = TargetBlock(ctx.state.program, target);
const IR::Block* from, const IR::Block* block) {
if (block != nullptr) {
StoreDispatcherPhiEdge(ctx, dispatcher, from, block);
if (ctx.other_half != nullptr) {
@@ -158,41 +144,37 @@ void EmitDispatcherTarget(ValueEmitContext& ctx, const DispatcherFunctionState&
}
uint32_t EmitDispatcherNextPc(ValueEmitContext& ctx, const DispatcherFunctionState& dispatcher,
const IR::Block* block, const IR::BlockInfo& info) {
const auto& term = info.terminator;
const IR::Block* block) {
const auto& term = block->terminator;
switch (term.kind) {
case CFG::TerminatorKind::Branch:
EmitDispatcherTarget(ctx, dispatcher, block, term.true_block);
return ConstantU32(ctx.state, term.true_block);
return ConstantU32(ctx.state, term.true_block->id);
case CFG::TerminatorKind::ConditionalBranch: {
EmitDispatcherTarget(ctx, dispatcher, block, term.true_block);
EmitDispatcherTarget(ctx, dispatcher, block, term.false_block);
const auto selected = ctx.state.builder.AllocateId();
ctx.state.builder.AddFunction(
spv::OpSelect, TypeU32(ctx.state), selected, ctx.Def(info.condition),
ConstantU32(ctx.state, term.true_block), ConstantU32(ctx.state, term.false_block));
ctx.state.builder.AddFunction(spv::OpSelect, TypeU32(ctx.state), selected,
ctx.Def(block->condition),
ConstantU32(ctx.state, term.true_block->id),
ConstantU32(ctx.state, term.false_block->id));
return selected;
}
case CFG::TerminatorKind::IndirectBranch: {
for (const auto target: term.indirect_targets) {
for (const auto* target: block->ImmSuccessors()) {
EmitDispatcherTarget(ctx, dispatcher, block, target);
}
uint32_t selected = ConstantU32(ctx.state, UINT32_MAX);
if (!info.indirect_target.IsEmpty()) {
const auto selector = ctx.Def(info.indirect_target);
const auto& values = term.indirect_selector_code != UINT32_MAX
? term.indirect_selector_values
: term.indirect_target_pcs;
const auto& targets = term.indirect_selector_code != UINT32_MAX
? term.indirect_selector_targets
: term.indirect_targets;
for (size_t index = 0; index < std::min(values.size(), targets.size()); index++) {
if (!block->indirect_target.IsEmpty()) {
const auto selector = ctx.Def(block->indirect_target);
for (const auto& branch: term.cases) {
const auto match = ctx.state.builder.AllocateId();
const auto next = ctx.state.builder.AllocateId();
ctx.state.builder.AddFunction(spv::OpIEqual, TypeBool(ctx.state), match,
selector, ConstantU32(ctx.state, values[index]));
selector, ConstantU32(ctx.state, branch.value));
ctx.state.builder.AddFunction(spv::OpSelect, TypeU32(ctx.state), next, match,
ConstantU32(ctx.state, targets[index]), selected);
ConstantU32(ctx.state, branch.target->id),
selected);
selected = next;
}
}
@@ -267,7 +249,7 @@ void EmitStructuredInstruction(ValueEmitContext& ctx, StructuredFunctionState& s
}
for (size_t index = 0; index < inst.NumArgs(); index++) {
const auto* predecessor = inst.PhiBlock(index);
if (predecessor == nullptr || !ctx.state.labels.contains(predecessor)) {
if (predecessor == nullptr || predecessor->Definition() == 0) {
ctx.Fail(inst, "has a predecessor outside the structured function");
}
}
@@ -301,7 +283,7 @@ void EmitDispatcherInstruction(ValueEmitContext& ctx, const DispatcherFunctionSt
template <typename EmitInstruction>
void EmitBlock(ValueEmitContext& ctx, const IR::Block* block, EmitInstruction&& emit_instruction) {
ctx.state.current_block = block;
EmitLabel(ctx.state, ctx.Label(block));
EmitLabel(ctx.state, block->Definition());
bool emitted_non_phi = false;
for (const auto& inst: *block) {
if (inst.GetOpcode() == IR::ValueOpcode::Phi) {
@@ -342,14 +324,14 @@ void PatchStructuredPhis(ValueEmitContext& ctx, StructuredFunctionState& structu
void EmitStructuredFunction(ValueEmitContext& ctx) {
const auto& program = ctx.state.program;
StructuredFunctionState structured;
ctx.state.builder.AddFunction(spv::OpBranch, ctx.Label(program.blocks.front()));
ctx.state.builder.AddFunction(spv::OpBranch, program.blocks.front()->Definition());
for (size_t index = 0; index < program.blocks.size(); index++) {
const auto* block = program.blocks[index];
EmitBlock(ctx, block, [&](ValueEmitContext& lane, const IR::Inst& inst) {
EmitStructuredInstruction(lane, structured, inst);
});
structured.block_exit_labels.emplace(block, ctx.state.current_label);
EmitStructuredTerminator(ctx, block, program.block_info[index]);
EmitStructuredTerminator(ctx, block);
}
PatchStructuredPhis(ctx, structured);
}
@@ -357,12 +339,11 @@ void EmitStructuredFunction(ValueEmitContext& ctx) {
void EmitDispatcherFunction(ValueEmitContext& ctx, const DispatcherFunctionState& dispatcher) {
auto& state = ctx.state;
const auto* entry = state.program.blocks.front();
state.builder.AddFunction(spv::OpBranch, ctx.Label(entry));
state.builder.AddFunction(spv::OpBranch, entry->Definition());
EmitBlock(ctx, entry, [&](ValueEmitContext& lane, const IR::Inst& inst) {
EmitDispatcherInstruction(lane, dispatcher, inst);
});
const auto initial_pc =
EmitDispatcherNextPc(ctx, dispatcher, entry, state.program.block_info.front());
const auto initial_pc = EmitDispatcherNextPc(ctx, dispatcher, entry);
const auto initial_parent = state.current_label;
state.builder.AddFunction(spv::OpBranch, dispatcher.header_label);
@@ -384,8 +365,8 @@ void EmitDispatcherFunction(ValueEmitContext& ctx, const DispatcherFunctionState
spv::SelectionControlMaskNone);
std::vector<uint32_t> words {spv::OpSwitch, pc, dispatcher.after_switch_label};
for (size_t index = 1; index < state.program.blocks.size(); index++) {
words.push_back(state.program.block_info[index].id);
words.push_back(ctx.Label(state.program.blocks[index]));
words.push_back(state.program.blocks[index]->id);
words.push_back(state.program.blocks[index]->Definition());
}
state.builder.AddFunction(words);
std::vector<uint32_t> next_pc_words {spv::OpPhi, TypeU32(state), next_pc,
@@ -396,8 +377,7 @@ void EmitDispatcherFunction(ValueEmitContext& ctx, const DispatcherFunctionState
[&](ValueEmitContext& lane, const IR::Inst& inst) {
EmitDispatcherInstruction(lane, dispatcher, inst);
});
const auto selected = EmitDispatcherNextPc(ctx, dispatcher, state.program.blocks[index],
state.program.block_info[index]);
const auto selected = EmitDispatcherNextPc(ctx, dispatcher, state.program.blocks[index]);
next_pc_words.push_back(selected);
next_pc_words.push_back(state.current_label);
state.builder.AddFunction(spv::OpBranch, dispatcher.after_switch_label);
@@ -551,23 +531,23 @@ uint32_t ValueEmitContext::Shuffle(const IR::Inst& inst, size_t index, uint32_t
}
uint32_t ValueEmitContext::Result(const IR::Inst& inst) {
if (const auto found = definitions.find(&inst); found != definitions.end()) {
return found->second;
if (const auto id = inst.Definition(half); id != 0) {
return id;
}
const auto id = state.builder.AllocateId();
definitions.emplace(&inst, id);
inst.SetDefinition(half, id);
return id;
}
uint32_t ValueEmitContext::Define(const IR::Inst& inst, uint32_t value) {
if (const auto found = definitions.find(&inst); found != definitions.end()) {
if (found->second != value) {
if (const auto id = inst.Definition(half); id != 0) {
if (id != value) {
state.builder.AddFunction(spv::OpCopyObject, TypeId(state, inst.GetType()),
found->second, value);
id, value);
}
return found->second;
return id;
}
definitions.emplace(&inst, value);
inst.SetDefinition(half, value);
return value;
}
@@ -595,10 +575,6 @@ const IR::ExportInfo& ValueEmitContext::Export(const IR::Inst& inst) const {
return state.program.export_info.at(inst.Flags<IR::ExportFlags>().index);
}
uint32_t ValueEmitContext::Label(const IR::Block* block) const {
return state.labels.at(block);
}
[[noreturn]] void ValueEmitContext::Fail(const char* reason) const {
EXIT("SPIR-V emission failed: hash=0x%016" PRIx64 " stage=%u reason=%s\n",
state.program.shader_hash, static_cast<unsigned>(state.program.stage), reason);
@@ -622,13 +598,16 @@ void EmitProgram(EmitterState& state) {
high.half = 1;
}
std::optional<DispatcherFunctionState> dispatcher;
if (state.program.stage == ShaderType::Pixel && state.requirements.pixel_valid_mask) {
if (state.program.stage == ShaderType::Pixel && state.program.info.pixel_valid_mask) {
state.pixel_valid_mask_variable = state.builder.AllocateId();
state.builder.AddName(state.pixel_valid_mask_variable, "pixel_valid_mask_active");
}
for (const auto* block: program.blocks) {
const auto label = state.builder.AllocateId();
state.labels.emplace(block, label);
// Re-emission may use a different host subgroup width and ID allocation order.
for (const auto& inst: *block) {
inst.ResetDefinitions();
}
block->SetDefinition(state.builder.AllocateId());
}
if (state.program.dispatcher_fallback) {
auto& dispatch = dispatcher.emplace();
@@ -669,8 +648,8 @@ void EmitProgram(EmitterState& state) {
}
}
for (size_t index = 0; index < program.blocks.size(); index++) {
mark_cross_block(program.block_info[index].condition, program.blocks[index]);
mark_cross_block(program.block_info[index].indirect_target, program.blocks[index]);
mark_cross_block(program.blocks[index]->condition, program.blocks[index]);
mark_cross_block(program.blocks[index]->indirect_target, program.blocks[index]);
}
dispatch.header_label = state.builder.AllocateId();
dispatch.select_label = state.builder.AllocateId();
@@ -687,28 +666,23 @@ void EmitProgram(EmitterState& state) {
}
DefineGetBdaPointer(state);
DefineBvhIntersect(state);
for (const auto* block: program.blocks) {
if (std::ranges::any_of(*block, [](const IR::Inst& inst) {
return inst.GetOpcode() == IR::ValueOpcode::SwizzleU32;
})) {
ctx.scratch_u32_variable = state.builder.AllocateId();
if (state.lane_count == 2) {
high.scratch_u32_variable = state.builder.AllocateId();
}
break;
if (program.info.uses_swizzle) {
ctx.scratch_u32_variable = state.builder.AllocateId();
if (state.lane_count == 2) {
high.scratch_u32_variable = state.builder.AllocateId();
}
}
state.builder.AddFunction(spv::OpFunction, TypeVoid(state),
state.mesh_guest_func != 0 ? state.mesh_guest_func : state.main_func,
spv::FunctionControlMaskNone, TypeFunction(state));
EmitLabel(state, state.entry_label);
if (state.requirements.function_lds) {
if (state.program.info.function_lds) {
state.builder.AddFunction(
spv::OpVariable,
TypeU32ArrayPointer(state, spv::StorageClassFunction, LdsDwordCount(state)),
state.lds_variable, spv::StorageClassFunction);
}
if (state.requirements.function_scratch) {
if (state.program.info.function_scratch) {
for (uint32_t half = 0; half < state.lane_count; half++) {
state.builder.AddFunction(
spv::OpVariable,
@@ -8,7 +8,6 @@
#include <deque>
#include <list>
#include <map>
#include <set>
#include <span>
namespace Libs::Graphics::ShaderRecompiler::CFG {
@@ -568,9 +567,8 @@ bool ResolveSetpcTargets(const Decoder::Program& program, uint32_t setpc_index,
return false;
}
bool IsValidTarget(uint32_t target, const std::set<uint32_t>& instruction_pcs, uint32_t first_pc,
uint32_t end_pc) {
return target == end_pc || (target >= first_pc && instruction_pcs.contains(target));
bool IsValidTarget(uint32_t target, std::span<const Instruction> instructions, uint32_t end_pc) {
return target == end_pc || std::ranges::binary_search(instructions, target, {}, &Instruction::pc);
}
std::vector<uint32_t> AllBlockIds(uint32_t count) {
@@ -1455,9 +1453,7 @@ Graph BuildGraph(const Decoder::Program& program) {
const auto first_pc = program.instructions.front().pc;
const auto end_pc = ProgramEndPc(program);
std::set<uint32_t> instruction_pcs;
for (const auto& inst: program.instructions) {
instruction_pcs.insert(inst.pc);
if (inst.opcode == Opcode::UNSUPPORTED) {
ExitBuildFailure(
graph, FailureKind::UnsupportedInstruction, UINT32_MAX,
@@ -1466,23 +1462,21 @@ Graph BuildGraph(const Decoder::Program& program) {
}
}
std::set<uint32_t> labels;
labels.insert(first_pc);
labels.insert(end_pc);
std::vector<uint32_t> labels {first_pc, end_pc};
std::map<uint32_t, SetpcTargetInfo> setpc_targets;
for (uint32_t i = 0; i < program.instructions.size(); i++) {
const auto& inst = program.instructions[i];
const auto next_pc = InstructionEndPc(inst);
if (Decoder::IsDirectBranch(inst.opcode)) {
if (!IsValidTarget(inst.branch_target, instruction_pcs, first_pc, end_pc)) {
if (!IsValidTarget(inst.branch_target, program.instructions, end_pc)) {
ExitBuildFailure(graph, FailureKind::InvalidBranchTarget, UINT32_MAX,
fmt::format("branch at pc 0x{:08x} targets invalid pc 0x{:08x}",
inst.pc, inst.branch_target));
}
labels.insert(inst.branch_target);
labels.push_back(inst.branch_target);
if (next_pc <= end_pc) {
labels.insert(next_pc);
labels.push_back(next_pc);
}
} else if (inst.opcode == Opcode::S_SETPC_B64) {
SetpcTargetInfo target_info;
@@ -1495,30 +1489,32 @@ Graph BuildGraph(const Decoder::Program& program) {
? std::span<const uint32_t>(target_info.target_pcs)
: std::span<const uint32_t>(&target_info.target, 1);
for (const auto target: target_pcs) {
if (!IsValidTarget(target, instruction_pcs, first_pc, end_pc)) {
if (!IsValidTarget(target, program.instructions, end_pc)) {
ExitBuildFailure(
graph, FailureKind::InvalidBranchTarget, UINT32_MAX,
fmt::format("S_SETPC_B64 at pc 0x{:08x} targets invalid pc 0x{:08x}",
inst.pc, target));
}
labels.insert(target);
labels.push_back(target);
}
setpc_targets.emplace(inst.pc, std::move(target_info));
if (next_pc <= end_pc) {
labels.insert(next_pc);
labels.push_back(next_pc);
}
} else if (inst.opcode == Opcode::S_ENDPGM) {
labels.insert(next_pc);
labels.push_back(next_pc);
}
}
std::vector<uint32_t> sorted_labels(labels.begin(), labels.end());
for (uint32_t i = 0; i < sorted_labels.size(); i++) {
const auto start = sorted_labels[i];
SortUnique(labels);
graph.blocks.reserve(labels.size());
for (uint32_t i = 0; i < labels.size(); i++) {
const auto start = labels[i];
if (start > end_pc) {
continue;
}
if (start != end_pc && !instruction_pcs.contains(start)) {
const auto begin = std::ranges::lower_bound(program.instructions, start, {}, &Instruction::pc);
if (start != end_pc && (begin == program.instructions.end() || begin->pc != start)) {
ExitBuildFailure(
graph, FailureKind::InvalidLabel, UINT32_MAX,
fmt::format("CFG label does not start on an instruction: 0x{:08x}", start));
@@ -1527,11 +1523,8 @@ Graph BuildGraph(const Decoder::Program& program) {
BasicBlock block;
block.id = static_cast<uint32_t>(graph.blocks.size());
block.start_pc = start;
block.end_pc = i + 1u < sorted_labels.size() ? sorted_labels[i + 1u] : end_pc;
block.inst_begin = static_cast<uint32_t>(
std::lower_bound(program.instructions.begin(), program.instructions.end(), start,
[](const Instruction& inst, uint32_t pc) { return inst.pc < pc; }) -
program.instructions.begin());
block.end_pc = i + 1u < labels.size() ? labels[i + 1u] : end_pc;
block.inst_begin = static_cast<uint32_t>(begin - program.instructions.begin());
block.inst_end = static_cast<uint32_t>(
std::lower_bound(program.instructions.begin(), program.instructions.end(), block.end_pc,
[](const Instruction& inst, uint32_t pc) { return inst.pc < pc; }) -
@@ -1541,10 +1534,10 @@ Graph BuildGraph(const Decoder::Program& program) {
graph.entry_block = 0;
std::map<uint32_t, uint32_t> pc_to_block;
for (const auto& block: graph.blocks) {
pc_to_block.emplace(block.start_pc, block.id);
}
const auto block_at_pc = [&](uint32_t pc) {
const auto found = std::ranges::lower_bound(graph.blocks, pc, {}, &BasicBlock::start_pc);
return found != graph.blocks.end() && found->start_pc == pc ? found->id : UINT32_MAX;
};
for (auto& block: graph.blocks) {
block.terminator = {};
@@ -1566,7 +1559,7 @@ Graph BuildGraph(const Decoder::Program& program) {
block.terminator.indirect_selector_code = target_info.selector_code;
for (const auto target_pc: target_info.target_pcs) {
block.terminator.indirect_target_pcs.push_back(target_pc);
block.terminator.indirect_targets.push_back(pc_to_block.at(target_pc));
block.terminator.indirect_targets.push_back(block_at_pc(target_pc));
}
const auto selector_count = std::min(target_info.selector_values.size(),
target_info.selector_target_pcs.size());
@@ -1574,35 +1567,35 @@ Graph BuildGraph(const Decoder::Program& program) {
block.terminator.indirect_selector_values.push_back(
target_info.selector_values[i]);
block.terminator.indirect_selector_targets.push_back(
pc_to_block.at(target_info.selector_target_pcs[i]));
block_at_pc(target_info.selector_target_pcs[i]));
}
} else {
block.terminator.kind = TerminatorKind::Branch;
block.terminator.condition = BranchCondition::Always;
block.terminator.true_block = pc_to_block.at(target_info.target);
block.terminator.true_block = block_at_pc(target_info.target);
}
} else if (last.opcode == Opcode::S_BRANCH) {
block.terminator.kind = TerminatorKind::Branch;
block.terminator.condition = BranchCondition::Always;
block.terminator.true_block = pc_to_block.at(last.branch_target);
block.terminator.true_block = block_at_pc(last.branch_target);
} else if (Decoder::IsConditionalBranch(last.opcode)) {
block.terminator.kind = TerminatorKind::ConditionalBranch;
block.terminator.condition = ConditionForOpcode(last.opcode);
block.terminator.true_block = pc_to_block.at(last.branch_target);
const auto fallthrough = pc_to_block.find(next_pc);
if (fallthrough == pc_to_block.end()) {
block.terminator.true_block = block_at_pc(last.branch_target);
const auto fallthrough = block_at_pc(next_pc);
if (fallthrough == UINT32_MAX) {
ExitBuildFailure(
graph, FailureKind::MissingFallthrough, block.id,
fmt::format("conditional branch at pc 0x{:08x} has no fallthrough block",
last.pc));
}
block.terminator.false_block = fallthrough->second;
block.terminator.false_block = fallthrough;
} else {
auto next = pc_to_block.find(block.end_pc);
if (next != pc_to_block.end() && block.end_pc != block.start_pc) {
const auto next = block_at_pc(block.end_pc);
if (next != UINT32_MAX && block.end_pc != block.start_pc) {
block.terminator.kind = TerminatorKind::Branch;
block.terminator.condition = BranchCondition::Always;
block.terminator.true_block = next->second;
block.terminator.true_block = next;
} else {
block.terminator.kind = TerminatorKind::Return;
}
@@ -997,7 +997,8 @@ constexpr Vop2SdwaRule VOP2_SDWA_RULES[] = {
{SdwaSelAll(), SdwaSelAll(), SdwaSelAll(), true, false},
{SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), false, false},
{SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), false, false},
{SdwaSelWords() | SdwaSelFull(), SdwaSelAll(), SdwaSelAll(), true, false},
// Bitwise: WriteRawU32 inserts a byte or word destination like any other partial write.
{SdwaSelAll(), SdwaSelAll(), SdwaSelAll(), true, false},
};
static_assert(sizeof(VOP2_SDWA_RULES) / sizeof(VOP2_SDWA_RULES[0]) ==
static_cast<size_t>(Vop2SdwaProfile::Count));
@@ -735,7 +735,7 @@ void Translator::WriteCompareResult(const Decoder::Operand& operand, IR::U1 valu
}
void Translator::AddBranchCondition(const CFG::Graph& graph, const CFG::BasicBlock& source,
IR::BlockInfo& info) {
IR::Block& info) {
const auto native_condition = [&](CFG::BranchCondition kind) -> IR::U1 {
IR::U1 condition;
switch (kind) {
@@ -1023,45 +1023,67 @@ IR::Program TranslateProgram(const Decoder::Program& decoded, const CFG::Graph&
result.block_storage.reserve(cfg.blocks.size() + 1u);
result.blocks.reserve(cfg.blocks.size() + 1u);
result.block_info.reserve(cfg.blocks.size() + 1u);
const auto max_id = std::ranges::max_element(cfg.blocks, {}, [](const CFG::BasicBlock& block) {
return block.id;
})->id;
if (max_id == UINT32_MAX) {
EXIT("cannot allocate a typed entry block id");
}
CFG::Terminator terminator;
terminator.kind = CFG::TerminatorKind::Branch;
terminator.true_block = cfg.blocks.front().id;
result.block_storage.push_back(std::make_unique<IR::Block>());
result.blocks.push_back(result.block_storage.back().get());
result.block_info.push_back({max_id + 1u, cfg.blocks.front().start_pc,
cfg.blocks.front().start_pc, std::move(terminator)});
auto& entry = *result.blocks.front();
entry.id = max_id + 1u;
entry.start_pc = entry.end_pc = cfg.blocks.front().start_pc;
entry.terminator.kind = CFG::TerminatorKind::Branch;
std::unordered_map<uint32_t, size_t> block_indices;
block_indices.reserve(cfg.blocks.size());
for (const auto& source_block: cfg.blocks) {
if (!block_indices.emplace(source_block.id, result.blocks.size()).second) {
EXIT("CFG contains duplicate block id %u", source_block.id);
std::unordered_map<uint32_t, IR::Block*> blocks_by_id;
blocks_by_id.reserve(cfg.blocks.size());
for (const auto& source: cfg.blocks) {
auto& block = *result.block_storage.emplace_back(std::make_unique<IR::Block>());
if (!blocks_by_id.emplace(source.id, &block).second) {
EXIT("CFG contains duplicate block id %u", source.id);
}
result.block_storage.push_back(std::make_unique<IR::Block>());
result.blocks.push_back(result.block_storage.back().get());
result.block_info.push_back(
{source_block.id, source_block.start_pc, source_block.end_pc, source_block.terminator});
result.blocks.push_back(&block);
block.id = source.id;
block.start_pc = source.start_pc;
block.end_pc = source.end_pc;
}
for (const auto& source_block: cfg.blocks) {
const auto source_index = block_indices.at(source_block.id);
for (const auto successor: source_block.successors) {
const auto target = block_indices.find(successor);
if (target == block_indices.end()) {
EXIT("CFG block %u has unknown successor %u", source_block.id, successor);
const auto target_block = [&](uint32_t id) -> IR::Block* {
return id == UINT32_MAX ? nullptr : blocks_by_id.at(id);
};
entry.terminator.true_block = target_block(cfg.blocks.front().id);
for (const auto& source: cfg.blocks) {
auto& block = *blocks_by_id.at(source.id);
const auto& term = source.terminator;
block.terminator.kind = term.kind;
block.terminator.true_block = target_block(term.true_block);
block.terminator.false_block = target_block(term.false_block);
block.terminator.merge_block = target_block(term.merge_block);
block.terminator.continue_block = target_block(term.continue_block);
block.terminator.loop_header = term.loop_header;
block.terminator.indexed = term.indirect_selector_code != UINT32_MAX;
const auto& values =
block.terminator.indexed ? term.indirect_selector_values : term.indirect_target_pcs;
const auto& targets =
block.terminator.indexed ? term.indirect_selector_targets : term.indirect_targets;
if (values.size() != targets.size()) {
EXIT("CFG block %u has inconsistent indirect targets", source.id);
}
block.terminator.cases.reserve(values.size());
for (size_t index = 0; index < values.size(); ++index) {
block.terminator.cases.push_back({values[index], target_block(targets[index])});
}
for (const auto successor: source.successors) {
const auto target = blocks_by_id.find(successor);
if (target == blocks_by_id.end()) {
EXIT("CFG block %u has unknown successor %u", source.id, successor);
}
result.blocks[source_index]->AddBranch(result.blocks[target->second]);
block.AddBranch(target->second);
}
}
{
result.blocks.front()->AddBranch(result.blocks.at(block_indices.at(cfg.blocks.front().id)));
IR::IREmitter entry_ir(result.blocks.front());
entry.AddBranch(entry.terminator.true_block);
IR::IREmitter entry_ir(&entry);
const auto builtin = [&](IR::StageInputKind kind, uint32_t component = 0u) {
return IR::U32(
entry_ir.Emit(IR::ValueOpcode::GetBuiltin,
@@ -1331,10 +1353,10 @@ IR::Program TranslateProgram(const Decoder::Program& decoded, const CFG::Graph&
}
}
const bool flush_f32_inputs = options.stage == ShaderType::Compute &&
(options.input_info.compute->float_mode & 0x10u) == 0;
(options.input_info.compute->float_mode & 0x10u) == 0;
for (const auto& cfg_block: cfg.blocks) {
const auto typed_index = block_indices.at(cfg_block.id);
Translator translator(result, result.blocks[typed_index], vector_limit, flush_f32_inputs);
auto* block = blocks_by_id.at(cfg_block.id);
Translator translator(result, block, vector_limit, flush_f32_inputs);
for (uint32_t index = cfg_block.inst_begin; index < cfg_block.inst_end; index++) {
const auto& instruction = decoded.instructions[index];
if (IsCodeTableLoad(cfg, instruction.pc)) {
@@ -1357,7 +1379,7 @@ IR::Program TranslateProgram(const Decoder::Program& decoded, const CFG::Graph&
}
translator.TranslateInstruction(instruction);
}
translator.AddBranchCondition(cfg, cfg_block, result.block_info[typed_index]);
translator.AddBranchCondition(cfg, cfg_block, *block);
}
IR::ValidateProgram(result, false);
return result;
@@ -17,7 +17,8 @@ public:
void TranslateInstruction(const Decoder::Instruction& inst);
void TranslateEmbeddedFetch(const Decoder::Instruction& inst, uint32_t attribute,
uint32_t component_count, const ShaderBufferResource& resource);
void AddBranchCondition(const CFG::Graph& graph, const CFG::BasicBlock& source, IR::BlockInfo& info);
void AddBranchCondition(const CFG::Graph& graph, const CFG::BasicBlock& source,
IR::Block& info);
private:
const Decoder::Operand& SourceAt(const Decoder::Instruction& inst, uint32_t index);
+25
View File
@@ -1,5 +1,6 @@
#pragma once
#include "graphics/shader/recompiler/frontend/cfg/ShaderCFG.h"
#include "graphics/shader/recompiler/ir/Reg.h"
#include "graphics/shader/recompiler/ir/Value.h"
@@ -37,6 +38,8 @@ public:
[[nodiscard]] const InstructionList& Instructions() const;
[[nodiscard]] std::span<Block* const> ImmPredecessors() const;
[[nodiscard]] std::span<Block* const> ImmSuccessors() const;
[[nodiscard]] uint32_t Definition() const { return definition; }
void SetDefinition(uint32_t value) const { definition = value; }
void SsaSeal();
[[nodiscard]] bool IsSsaSealed() const;
@@ -47,6 +50,27 @@ public:
const_iterator end() const;
bool empty() const;
struct SwitchCase {
uint32_t value;
Block* target;
};
struct Terminator {
CFG::TerminatorKind kind = CFG::TerminatorKind::Return;
Block* true_block = nullptr;
Block* false_block = nullptr;
Block* merge_block = nullptr;
Block* continue_block = nullptr;
std::vector<SwitchCase> cases;
bool indexed = false;
bool loop_header = false;
};
uint32_t id = 0;
uint32_t start_pc = 0;
uint32_t end_pc = 0;
Terminator terminator;
Value condition;
Value indirect_target;
std::array<Value, NumScalarRegs> ssa_sreg_values {};
std::array<Value, NumScalarRegs> ssa_thread_bit_sreg_values {};
std::array<Value, NumScalarRegs> ssa_sreg_mask_tags {};
@@ -57,6 +81,7 @@ private:
std::vector<Block*> predecessors;
std::vector<Block*> successors;
bool ssa_sealed = false;
mutable uint32_t definition = 0;
};
using BlockList = std::vector<Block*>;
+37 -40
View File
@@ -1,5 +1,6 @@
#include "common/assert.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/BufferFormat.h"
#include <fmt/format.h>
#include <map>
@@ -202,6 +203,16 @@ Value ResolveActiveU32(Value value, Value active) {
return {};
}
uint32_t StorageBufferElementBits(const Program& program, const MemoryInfo& memory) {
if (!memory.formatted) return memory.data_bits;
const auto format = memory.typed
? Format::DecodeTBufferFormat(memory.data_format, memory.number_format)
: program.info.buffers[memory.resource].descriptor_format;
const auto info = Format::GetFormatInfo(format);
return info.type == Format::ComponentType::Unknown || info.packed_bitfield
? 32u : info.component_bits[0];
}
bool HasShaderMemoryWrites(const Program& program) {
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
@@ -219,12 +230,11 @@ bool HasShaderMemoryWrites(const Program& program) {
}
void ValidateProgram(const Program& program, bool require_ssa) {
if (program.blocks.size() != program.block_info.size() ||
program.blocks.size() != program.block_storage.size()) {
if (program.blocks.size() != program.block_storage.size()) {
return Fail("value IR block storage is inconsistent");
}
std::unordered_map<const Block*, size_t> block_indices;
std::unordered_map<uint32_t, const Block*> blocks_by_id;
std::unordered_set<uint32_t> block_ids;
std::unordered_map<const Inst*, size_t> instruction_positions;
for (size_t block_index = 0; block_index < program.blocks.size(); block_index++) {
const auto* block = program.blocks[block_index];
@@ -235,10 +245,10 @@ void ValidateProgram(const Program& program, bool require_ssa) {
if (!block_indices.emplace(block, block_index).second) {
return Fail("value IR block pointer is duplicated");
}
if (program.block_info[block_index].id == UINT32_MAX) {
if (block->id == UINT32_MAX) {
return Fail("value IR block uses the reserved exit id");
}
if (!blocks_by_id.emplace(program.block_info[block_index].id, block).second) {
if (!block_ids.insert(block->id).second) {
return Fail("value IR block id is duplicated");
}
size_t position = 0;
@@ -283,15 +293,14 @@ void ValidateProgram(const Program& program, bool require_ssa) {
}
std::unordered_set<const Block*> expected_successors;
const auto add_target = [&](uint32_t id) {
const auto found = blocks_by_id.find(id);
if (found == blocks_by_id.end()) {
const auto add_target = [&](const Block* target) {
if (!block_indices.contains(target)) {
return false;
}
expected_successors.insert(found->second);
expected_successors.insert(target);
return true;
};
const auto& terminator = program.block_info[block_index].terminator;
const auto& terminator = block->terminator;
const auto validate_control_value = [&](Value value, Type type) {
if (value.IsEmpty() || value.GetType() != type) {
return false;
@@ -309,44 +318,32 @@ void ValidateProgram(const Program& program, bool require_ssa) {
if (!add_target(terminator.true_block) || !add_target(terminator.false_block)) {
return Fail("value IR conditional branch target is missing");
}
if (!validate_control_value(program.block_info[block_index].condition, Type::U1)) {
if (!validate_control_value(block->condition, Type::U1)) {
return Fail("value IR conditional branch condition is invalid");
}
break;
case CFG::TerminatorKind::IndirectBranch: {
if (!validate_control_value(program.block_info[block_index].indirect_target,
Type::U32)) {
if (!validate_control_value(block->indirect_target, Type::U32)) {
return Fail("value IR indirect branch selector is invalid");
}
std::unordered_set<uint32_t> indirect_targets;
for (const auto target: terminator.indirect_targets) {
if (!indirect_targets.insert(target).second) {
return Fail("value IR indirect branch target is duplicated");
std::unordered_set<uint32_t> values;
for (const auto& branch: terminator.cases) {
if (!values.insert(branch.value).second) {
return Fail("value IR indirect selector value is duplicated");
}
if (!add_target(target)) {
if (!add_target(branch.target)) {
return Fail("value IR indirect branch target is missing");
}
}
if (terminator.indirect_selector_values.size() !=
terminator.indirect_selector_targets.size()) {
return Fail("value IR indirect selector table is inconsistent");
}
for (const auto target: terminator.indirect_selector_targets) {
const auto found = blocks_by_id.find(target);
if (found == blocks_by_id.end() ||
!expected_successors.contains(found->second)) {
return Fail("value IR indirect selector target is not a CFG successor");
}
}
break;
}
case CFG::TerminatorKind::Return:
case CFG::TerminatorKind::Unsupported: break;
}
if ((terminator.merge_block != UINT32_MAX &&
!blocks_by_id.contains(terminator.merge_block)) ||
(terminator.continue_block != UINT32_MAX &&
!blocks_by_id.contains(terminator.continue_block))) {
if ((terminator.merge_block != nullptr &&
!block_indices.contains(terminator.merge_block)) ||
(terminator.continue_block != nullptr &&
!block_indices.contains(terminator.continue_block))) {
return Fail("value IR structured control target is missing");
}
if (successors != expected_successors) {
@@ -653,7 +650,7 @@ void ValidateProgram(const Program& program, bool require_ssa) {
}
}
}
const auto& info = program.block_info[block_index];
const auto& info = *program.blocks[block_index];
if (!info.condition.IsEmpty() && !control_dominates(info.condition, block)) {
return Fail("value IR branch condition definition does not dominate its use");
}
@@ -664,9 +661,9 @@ void ValidateProgram(const Program& program, bool require_ssa) {
}
void ResolveControlFlowIdentities(Program& program) {
for (auto& info: program.block_info) {
info.condition = info.condition.Resolve();
info.indirect_target = info.indirect_target.Resolve();
for (auto* block: program.blocks) {
block->condition = block->condition.Resolve();
block->indirect_target = block->indirect_target.Resolve();
}
}
@@ -701,9 +698,9 @@ std::string ProgramToString(const Program& program) {
std::string text;
for (size_t block_index = 0; block_index < program.blocks.size(); block_index++) {
text += fmt::format("Block ${} pc=0x{:08x}..0x{:08x}\n", block_index,
program.block_info[block_index].start_pc,
program.block_info[block_index].end_pc);
text +=
fmt::format("Block ${} pc=0x{:08x}..0x{:08x}\n", block_index,
program.blocks[block_index]->start_pc, program.blocks[block_index]->end_pc);
for (const auto& inst: *program.blocks[block_index]) {
const auto type = inst.GetType();
if (type != Type::Void) {
+40 -32
View File
@@ -104,6 +104,7 @@ struct BufferResource {
static constexpr uint32_t NoIndirectBuffer = UINT32_MAX;
uint32_t source = 0;
uint32_t descriptor_index = UINT32_MAX;
uint32_t first_use_pc = 0;
uint32_t max_byte_extent = 0;
uint32_t packed_stride = 0;
@@ -131,6 +132,7 @@ struct ImageResource {
static constexpr uint32_t NoIndirectImage = UINT32_MAX;
uint32_t source = 0;
uint32_t descriptor_index = UINT32_MAX;
uint32_t first_use_pc = 0;
ImageResourceClass resource_class = ImageResourceClass::None;
Prospero::TextureNumericClass numeric_class = Prospero::TextureNumericClass::Unsupported;
@@ -424,23 +426,16 @@ DescriptorBindingForImage(const ImageResource& image) {
return static_cast<DescriptorBindingKind>(base + dimension);
}
struct DescriptorBinding {
DescriptorBindingKind kind = DescriptorBindingKind::Buffers;
std::vector<uint32_t> resources;
bool operator==(const DescriptorBinding& other) const = default;
};
struct BindingLayout {
uint32_t push_data_start_dword = PushData::NoStart;
uint32_t dispatch_thread_dword = PushData::NoStart;
uint32_t memory_offset_dword = 0;
uint32_t memory_offset_count = 0;
std::vector<uint32_t> user_data_registers;
std::vector<DescriptorBinding> descriptors;
struct Bindings {
std::array<uint32_t, static_cast<size_t>(DescriptorBindingKind::Count)> descriptor_counts {};
uint64_t descriptor_mask = 0;
uint32_t push_data_start_dword = PushData::NoStart;
uint32_t dispatch_thread_dword = PushData::NoStart;
uint32_t memory_offset_dword = 0;
[[nodiscard]] uint32_t ShaderDataDwords() const {
return memory_offset_dword + (memory_offset_count + 3u) / 4u;
return memory_offset_dword +
(descriptor_counts[static_cast<size_t>(DescriptorBindingKind::Buffers)] + 3u) / 4u;
}
[[nodiscard]] bool UsesPushData() const {
return push_data_start_dword != PushData::NoStart;
@@ -451,7 +446,7 @@ struct BindingLayout {
}
}
bool operator==(const BindingLayout& other) const = default;
bool operator==(const Bindings& other) const = default;
};
struct ShaderInfo {
@@ -466,24 +461,38 @@ struct ShaderInfo {
std::vector<SampledResourcePair> sampled_pairs;
std::vector<StageInput> inputs;
std::vector<StageOutput> outputs;
std::vector<uint32_t> user_data_registers;
uint64_t live_buffers = 0;
std::array<uint8_t, 32> vertex_fetch_components {};
int32_t vertex_offset_sgpr = -1;
int32_t instance_offset_sgpr = -1;
bool has_bitwise_xor = false;
bool uses_dma = false;
bool has_bitwise_xor = false;
bool uses_dma = false;
bool uses_swizzle = false;
bool uses_lds = false;
bool uses_gds = false;
bool uses_flattened_srt = false;
bool uses_dispatch_threads = false;
bool bvh = false;
bool subgroup_ballot = false;
bool subgroup_barrier = false;
bool subgroup_shuffle = false;
bool subgroup_local_invocation_id = false;
bool compute_derivatives = false;
bool image_gather_extended = false;
bool function_lds = false;
bool function_scratch = false;
bool pixel_valid_mask = false;
bool buffer_int64_atomics = false;
bool buffer_u8 = false;
bool buffer_u16 = false;
bool shared_int64_atomics = false;
bool coherent_buffers = false;
bool float64 = false;
bool operator==(const ShaderInfo& other) const = default;
};
struct BlockInfo {
uint32_t id = 0;
uint32_t start_pc = 0;
uint32_t end_pc = 0;
CFG::Terminator terminator;
Value condition;
Value indirect_target;
};
struct DescriptorSource {
struct IndirectDescriptor {
struct SelectorRead {
@@ -542,7 +551,7 @@ struct CompiledShaderInfo {
uint32_t param_export_mask = 0;
bool has_address_writes = false;
ShaderInfo info;
BindingLayout bindings;
Bindings bindings;
};
struct UniformFillPlan {
@@ -580,7 +589,6 @@ struct ResourcePlan {
std::vector<DescriptorSource> descriptor_sources;
std::vector<ResourceBlock> control_flow;
std::vector<SrtRead> srt_reads;
std::vector<uint8_t> clean_flat_slots;
bool requires_specialization_memory = false;
bool capture_specialization_reads = false;
bool srt_plan_complete = false;
@@ -614,7 +622,6 @@ struct Program: ResourcePlan {
bool dispatcher_fallback = false;
CFG::FailureKind cfg_failure_kind = CFG::FailureKind::None;
std::string fallback_reason;
std::vector<BlockInfo> block_info;
struct ScalarWrite { uint32_t pc; ScalarReg reg; };
std::vector<ScalarWrite> scalar_writes;
// Typed memory and export instructions reference shader-local metadata by dense index.
@@ -622,11 +629,12 @@ struct Program: ResourcePlan {
std::vector<ExportInfo> export_info;
bool has_address_writes = false;
bool shader_info_complete = false;
BindingLayout bindings;
bool binding_layout_complete = false;
Bindings bindings;
};
uint32_t StorageBufferElementBits(const Program& program, const MemoryInfo& memory);
std::string ProgramToString(const Program& program);
bool HasShaderMemoryWrites(const Program& program);
+8 -1
View File
@@ -129,6 +129,12 @@ public:
}
return evaluation_index;
}
// Backend definitions belong to one emission. Wave64 emulation needs one per half.
[[nodiscard]] uint32_t Definition(uint32_t half) const { return definitions[half]; }
void SetDefinition(uint32_t half, uint32_t definition) const {
definitions[half] = definition;
}
void ResetDefinitions() const { definitions = {}; }
void SetParent(Block* block);
void SetArg(size_t index, Value value);
@@ -165,6 +171,7 @@ private:
uint8_t num_args;
bool live = false;
mutable uint32_t evaluation_index = UINT32_MAX;
mutable std::array<uint32_t, 2> definitions {};
uint64_t flags;
Block* parent = nullptr;
union {
@@ -175,6 +182,6 @@ private:
std::vector<Use> uses;
};
static_assert(sizeof(Inst) <= 112, "Inst operand storage unintentionally increased");
static_assert(sizeof(Inst) <= 120, "Inst operand and backend storage unintentionally increased");
} // namespace Libs::Graphics::ShaderRecompiler::IR
@@ -1,197 +0,0 @@
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "common/assert.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include <algorithm>
#include <array>
#include <utility>
namespace Libs::Graphics::ShaderRecompiler::IR {
namespace {
[[noreturn]] void BindingFail(const char* message) {
EXIT("shader binding layout failed: %s", message);
std::abort();
}
void CollectShaderData(const Program& program, BindingLayout& layout) {
std::array<bool, NumScalarRegs> registers {};
bool uses_dispatch_threads = false;
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
if (!inst.HasUses()) {
continue;
}
uses_dispatch_threads |= inst.GetOpcode() == ValueOpcode::GetDispatchThreadExtent;
if (inst.GetOpcode() != ValueOpcode::GetUserData) {
continue;
}
if (inst.Arg(0).GetType() != Type::ScalarReg) {
BindingFail("typed shader contains an invalid user-data register");
}
const auto index = RegIndex(inst.Arg(0).ScalarRegister());
if (index >= NumScalarRegs) {
BindingFail("typed shader contains an invalid user-data register");
}
registers[index] = true;
}
}
for (uint32_t index = 0; index < registers.size(); index++) {
if (registers[index]) {
layout.user_data_registers.push_back(index);
}
}
layout.memory_offset_dword = static_cast<uint32_t>(layout.user_data_registers.size());
if (uses_dispatch_threads) {
layout.dispatch_thread_dword = layout.memory_offset_dword;
layout.memory_offset_dword += 3u;
}
}
void AddBinding(BindingLayout& layout, DescriptorBindingKind kind,
std::vector<uint32_t> resources = {}) {
layout.descriptors.push_back({kind, std::move(resources)});
}
} // namespace
SharedMemoryResources CollectMemoryResources(const Program& program, std::vector<uint32_t>& buffers) {
std::array<bool, ShaderInfo::MaxBuffers> live_buffers {};
SharedMemoryResources shared;
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
const auto op = inst.GetOpcode();
if (BufferAccessOf(op) == BufferAccess::None &&
SharedAccessOf(op) == SharedAccess::None &&
AddressOpcodeInfoOf(op).access == AddressAccess::None) {
continue;
}
const auto index = inst.Flags<MemoryFlags>().index;
if (index >= program.memory_info.size()) {
BindingFail("typed shader contains invalid memory metadata");
}
const auto& memory = program.memory_info[index];
if (memory.planning_only) {
continue;
}
shared.lds |= memory.kind == ResourceKind::FlatLocal;
if (SharedAccessOf(op) != SharedAccess::None) {
if (memory.kind != ResourceKind::Lds && memory.kind != ResourceKind::Gds) {
BindingFail("typed shader contains invalid shared-memory metadata");
}
shared.gds |= memory.kind == ResourceKind::Gds;
shared.lds |= memory.kind == ResourceKind::Lds;
} else if (memory.kind == ResourceKind::Buffer || memory.kind == ResourceKind::ScalarBuffer) {
EXIT_IF(memory.resource >= program.info.buffers.size());
live_buffers.at(memory.resource) = true;
for (const auto child: program.info.buffers[memory.resource].indirect_resources) {
live_buffers.at(child) = true;
}
}
}
}
for (uint32_t i = 0; i < program.info.buffers.size(); i++) {
if (live_buffers.at(i)) {
buffers.push_back(i);
}
}
return shared;
}
bool UsesFlattenedSrt(const Program& program) {
const auto uses_mapping = [](const auto& resource) {
return resource.indirect_root != UINT32_MAX;
};
return std::ranges::any_of(program.blocks, [](const Block* block) {
return std::ranges::any_of(*block, [](const Inst& inst) {
return inst.GetOpcode() == ValueOpcode::ReadConst;
});
}) || std::ranges::any_of(program.info.buffers, uses_mapping) ||
std::ranges::any_of(program.info.images, uses_mapping);
}
void AllocateBindings(Program& program, uint32_t push_data_start_dword, bool lds_storage) {
if (!program.shader_info_complete || program.binding_layout_complete) {
EXIT("shader binding layout failed: %s", !program.shader_info_complete
? "shader info is not ready"
: "binding layout already allocated");
}
BindingLayout next;
std::vector<uint32_t> buffers;
const auto shared = CollectMemoryResources(program, buffers);
CollectShaderData(program, next);
next.memory_offset_count = static_cast<uint32_t>(buffers.size());
next.push_data_start_dword =
PushData::StartFor(push_data_start_dword, next.ShaderDataDwords());
if (!buffers.empty()) {
// Draw binding accesses this first group directly when memory_offset_count is nonzero.
AddBinding(next, DescriptorBindingKind::Buffers, std::move(buffers));
}
std::array<std::vector<uint32_t>, ImageBindingCount> image_groups;
for (uint32_t i = 0; i < program.info.images.size(); i++) {
const auto kind = DescriptorBindingForImage(program.info.images[i]);
if (!kind.has_value()) {
EXIT("shader binding layout failed: image %u has an invalid binding class", i);
}
const auto group = ImageBindingIndex(*kind);
if (group >= image_groups.size()) {
EXIT("shader binding layout failed: image %u has an unmapped binding class", i);
}
auto& resources = image_groups[group];
const auto dynamic = program.info.images[i].mip_mode == ImageMipMode::Dynamic;
const auto count = dynamic ? program.info.images[i].mip_count : 1u;
if (count == 0u || (!dynamic && program.info.images[i].mip_count != 1u)) {
EXIT("shader binding layout failed: image %u has invalid specialized mip count %u", i,
program.info.images[i].mip_count);
}
resources.insert(resources.end(), count, i);
}
for (uint32_t i = 0; i < image_groups.size(); i++) {
if (!image_groups[i].empty()) {
AddBinding(next, static_cast<DescriptorBindingKind>(FirstImageBinding + i),
std::move(image_groups[i]));
}
}
if (!program.info.samplers.empty()) {
std::vector<uint32_t> resources(program.info.samplers.size());
for (uint32_t i = 0; i < resources.size(); i++) {
resources[i] = i;
}
AddBinding(next, DescriptorBindingKind::Samplers, std::move(resources));
}
if (shared.gds) {
AddBinding(next, DescriptorBindingKind::Gds);
}
if (shared.lds && lds_storage) {
AddBinding(next, DescriptorBindingKind::SharedMemory);
}
if (program.info.uses_dma) {
AddBinding(next, DescriptorBindingKind::BdaPagetable);
AddBinding(next, DescriptorBindingKind::FaultBuffer);
}
if (UsesFlattenedSrt(program)) {
AddBinding(next, DescriptorBindingKind::FlattenedSrt);
}
if (next.ShaderDataDwords() != 0 && !next.UsesPushData()) {
AddBinding(next, DescriptorBindingKind::ShaderData);
}
program.bindings = std::move(next);
program.binding_layout_complete = true;
}
const DescriptorBinding* FindBinding(const BindingLayout& layout, DescriptorBindingKind kind) {
for (const auto& binding: layout.descriptors) {
if (binding.kind == kind) {
return &binding;
}
}
return nullptr;
}
} // namespace Libs::Graphics::ShaderRecompiler::IR
@@ -1,23 +0,0 @@
#ifndef EMULATOR_INCLUDE_EMULATOR_GRAPHICS_SHADER_RECOMPILER_BINDINGLAYOUT_H_
#define EMULATOR_INCLUDE_EMULATOR_GRAPHICS_SHADER_RECOMPILER_BINDINGLAYOUT_H_
#include "graphics/shader/recompiler/ir/ShaderIR.h"
namespace Libs::Graphics::ShaderRecompiler::IR {
void AllocateBindings(Program& program, uint32_t push_data_start_dword = 0,
bool lds_storage = false);
struct SharedMemoryResources {
bool lds = false;
bool gds = false;
};
SharedMemoryResources CollectMemoryResources(const Program& program, std::vector<uint32_t>& buffers);
bool UsesFlattenedSrt(const Program& program);
const DescriptorBinding* FindBinding(const BindingLayout& layout, DescriptorBindingKind kind);
} // namespace Libs::Graphics::ShaderRecompiler::IR
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_SHADER_RECOMPILER_BINDINGLAYOUT_H_ */
@@ -215,6 +215,24 @@ void FoldInstruction(Block& block, Block::iterator instruction,
return;
case ValueOpcode::SelectU32:
case ValueOpcode::SelectF32: FoldSelect(inst); return;
case ValueOpcode::FPCmpClass32: {
const auto mask = Arg(inst, 1);
if (!IsImmediate(mask, Type::U32)) {
return;
}
switch (mask.U32() & 0x3ffu) {
case 0u: Replace(inst, Value(false)); break;
case 0x3ffu: Replace(inst, Value(true)); break;
case 3u: {
const auto result = block.PrependNewInst(
instruction, ValueOpcode::FPIsNan32, {Arg(inst, 0)});
Replace(inst, Value(&*result));
break;
}
default: break;
}
return;
}
case ValueOpcode::BitFieldInsert: {
const auto base = Arg(inst, 0);
const auto insert = Arg(inst, 1);
@@ -173,32 +173,28 @@ const DescriptorSource* Source(const ResourcePlan& program, uint32_t source) {
return &program.descriptor_sources[source];
}
void MarkCleanFlatSlots(const ResourcePlan& program, const DescriptorSource* source,
std::span<uint8_t> slots, Value extra = {}) {
void MarkCleanReads(const std::unordered_set<const Inst*>& planned_reads,
const DescriptorSource* source, Value extra = {}) {
if (source == nullptr && extra.IsEmpty()) {
return;
}
std::vector<Value> pending;
std::vector<Value> pending;
if (source != nullptr) {
pending.assign(source->dwords.begin(), source->dwords.begin() + source->dword_count);
}
if (!extra.IsEmpty()) pending.push_back(extra);
std::vector<const Inst*> visited;
while (!pending.empty()) {
auto value = pending.back().Resolve();
auto* inst = pending.back().Resolve().TryInstruction();
pending.pop_back();
const auto* inst = value.TryInstruction();
if (inst == nullptr || std::ranges::find(visited, inst) != visited.end()) {
continue;
}
visited.push_back(inst);
if (inst->GetOpcode() == ValueOpcode::ReadConst) {
const auto slot = inst->Arg(1).Resolve();
if (slot.IsImmediate() && slot.GetType() == Type::U32 && slot.U32() < slots.size()) {
slots[slot.U32()] = 1u;
pending.push_back(program.srt_reads[slot.U32()].value);
}
continue;
if (planned_reads.contains(inst)) {
auto flags = inst->Flags<SrtReadFlags>();
flags.clean = 1u;
inst->SetFlags(flags);
}
for (size_t arg = 0; arg < inst->NumArgs(); arg++) {
pending.push_back(inst->Arg(arg));
@@ -684,30 +680,29 @@ bool BuildSamplerPlan(const ShaderInfo& base, SamplerPlan& plan) {
template <typename Predicate>
static std::vector<ResourceBlock> ResourceControlFlow(const Program& program, const Predicate& predicate) {
// The renderer checks captured scalar reads against final buffer and image write ranges.
if (program.blocks.size() != program.block_info.size() || program.has_address_writes) {
if (program.has_address_writes) {
return {};
}
std::unordered_map<uint32_t, uint32_t> indices;
for (uint32_t i = 0; i < program.block_info.size(); i++) {
if (!indices.emplace(program.block_info[i].id, i).second) {
std::unordered_map<const Block*, uint32_t> indices;
for (uint32_t i = 0; i < program.blocks.size(); i++) {
if (!indices.emplace(program.blocks[i], i).second) {
return {};
}
}
std::vector<ResourceBlock> blocks(program.blocks.size());
for (uint32_t i = 0; i < blocks.size(); i++) {
auto& block = blocks[i];
const auto& info = program.block_info[i];
const auto& info = *program.blocks[i];
const auto& terminator = info.terminator;
std::vector<uint32_t> successors;
const std::array targets {terminator.true_block, terminator.false_block};
std::span<Block* const> successors;
switch (terminator.kind) {
case CFG::TerminatorKind::Branch: successors.push_back(terminator.true_block); break;
case CFG::TerminatorKind::Branch: successors = std::span(targets).first(1); break;
case CFG::TerminatorKind::ConditionalBranch:
successors = {terminator.true_block, terminator.false_block};
successors = targets;
block.condition = info.condition;
break;
case CFG::TerminatorKind::IndirectBranch:
successors = terminator.indirect_targets;
break;
case CFG::TerminatorKind::IndirectBranch: successors = info.ImmSuccessors(); break;
case CFG::TerminatorKind::Return: break;
default: return {};
}
@@ -828,17 +823,16 @@ static std::optional<std::array<uint64_t, 3>> FillIndex(Value value, uint32_t ax
}
static UniformFillPlan AnalyzeUniformFill(const Program& program) {
if (program.stage != ShaderType::Compute || program.blocks.empty() ||
program.blocks.size() != program.block_info.size() || program.info.uses_dma ||
if (program.stage != ShaderType::Compute || program.blocks.empty() || program.info.uses_dma ||
!program.info.samplers.empty()) {
return {};
}
std::unordered_set<uint32_t> visited;
uint32_t index = 0;
const Inst* store = nullptr;
std::unordered_set<const Block*> visited;
const auto* block = program.blocks.front();
const Inst* store = nullptr;
for (;;) {
if (!visited.insert(index).second) return {};
for (const auto& inst: *program.blocks[index]) {
if (block == nullptr || !visited.insert(block).second) return {};
for (const auto& inst: *block) {
if (AddressOpcodeInfoOf(inst.GetOpcode()).access != AddressAccess::None) return {};
if (!inst.MayHaveSideEffects()) continue;
if (store != nullptr || (BufferAccessOf(inst.GetOpcode()) != BufferAccess::Write &&
@@ -846,12 +840,10 @@ static UniformFillPlan AnalyzeUniformFill(const Program& program) {
return {};
store = &inst;
}
const auto& term = program.block_info[index].terminator;
const auto& term = block->terminator;
if (term.kind == CFG::TerminatorKind::Return) break;
if (term.kind != CFG::TerminatorKind::Branch) return {};
const auto next = std::ranges::find(program.block_info, term.true_block, &BlockInfo::id);
if (next == program.block_info.end()) return {};
index = static_cast<uint32_t>(next - program.block_info.begin());
block = term.true_block;
}
if (store == nullptr || visited.size() != program.blocks.size()) return {};
for (const auto& buffer: program.info.buffers) {
@@ -945,6 +937,12 @@ ResourcePlan ExtractResourcePlan(const Program& program) {
if (source == nullptr) {
return value;
}
if (source->GetOpcode() == ValueOpcode::ReadConst) {
const auto slot = source->Arg(1).Resolve();
EXIT_IF(!slot.IsImmediate() || slot.GetType() != Type::U32 ||
slot.U32() >= program.srt_reads.size());
return Clone(program.srt_reads[slot.U32()].value);
}
if (source->GetOpcode() == ValueOpcode::ReadFirstLane &&
ValidateRuntimeValue(program, source->Arg(0))) {
// Uniform values need no new EXEC context; preserve their shared evaluation memo.
@@ -965,6 +963,11 @@ ResourcePlan ExtractResourcePlan(const Program& program) {
}
auto& target =
plan.value_storage.emplace_back(source->GetOpcode(), source->Flags<uint64_t>());
if (source->GetOpcode() == ValueOpcode::LoadAddressU32 ||
source->GetOpcode() == ValueOpcode::ReadConstBuffer ||
source->GetOpcode() == ValueOpcode::LoadBufferU32) {
target.SetFlags(SrtReadFlags {.index = source->Flags<MemoryFlags>().index});
}
cloned.emplace(source, &target);
if (source->GetOpcode() == ValueOpcode::Phi) {
for (size_t index = 0; index < source->NumArgs(); index++) {
@@ -978,12 +981,15 @@ ResourcePlan ExtractResourcePlan(const Program& program) {
return Value(&target);
};
bool capture_indirect_reads = false;
plan.descriptor_sources.reserve(program.descriptor_sources.size());
for (const auto& source: program.descriptor_sources) {
auto& target = plan.descriptor_sources.emplace_back();
target.dword_count = source.dword_count;
target.indirect_descriptor = source.indirect_descriptor;
if (target.indirect_descriptor.has_value()) {
plan.requires_specialization_memory = true;
capture_indirect_reads = true;
target.indirect_descriptor->key_count = Clone(target.indirect_descriptor->key_count);
target.indirect_descriptor->selector_first =
Clone(target.indirect_descriptor->selector_first);
@@ -1034,34 +1040,39 @@ ResourcePlan ExtractResourcePlan(const Program& program) {
for (uint32_t i = 0; i < plan.uniform_fill.fill.words; ++i) {
plan.uniform_fill.values[i] = Clone(plan.uniform_fill.values[i]);
}
plan.clean_flat_slots.resize(plan.srt_reads.size());
bool capture_indirect_reads = false;
for (const auto& source: plan.descriptor_sources) {
if (!source.indirect_descriptor.has_value()) continue;
const auto& indirect = *source.indirect_descriptor;
plan.requires_specialization_memory = true;
capture_indirect_reads = true;
if (indirect.selector)
MarkCleanFlatSlots(plan, Source(plan, indirect.selector->source), plan.clean_flat_slots);
MarkCleanFlatSlots(plan, nullptr, plan.clean_flat_slots, indirect.selector_mask);
MarkCleanFlatSlots(plan, nullptr, plan.clean_flat_slots, indirect.selector_first);
MarkCleanFlatSlots(plan, nullptr, plan.clean_flat_slots, indirect.key_count);
if (indirect.sources.empty()) {
MarkCleanFlatSlots(plan, Source(plan, indirect.table_source), plan.clean_flat_slots);
} else {
for (const auto candidate: indirect.sources) {
MarkCleanFlatSlots(plan, Source(plan, candidate), plan.clean_flat_slots);
}
}
}
plan.capture_specialization_reads |= capture_indirect_reads || !plan.control_flow.empty();
if (plan.capture_specialization_reads) {
// Every former flat alias now targets its retained raw read. Mark those reads
// on the normalized graph, so discarded Phi/EXEC dependencies stay discarded.
std::unordered_set<const Inst*> planned_reads;
for (const auto& read: plan.srt_reads) {
auto* inst = read.value.ResolveInstruction();
EXIT_IF(inst == nullptr || (inst->GetOpcode() != ValueOpcode::LoadAddressU32 &&
inst->GetOpcode() != ValueOpcode::ReadConstBuffer));
planned_reads.insert(inst);
}
for (const auto& source: plan.descriptor_sources) {
if (!source.indirect_descriptor.has_value()) continue;
const auto& indirect = *source.indirect_descriptor;
if (indirect.selector)
MarkCleanReads(planned_reads, Source(plan, indirect.selector->source));
MarkCleanReads(planned_reads, nullptr, indirect.selector_mask);
MarkCleanReads(planned_reads, nullptr, indirect.selector_first);
MarkCleanReads(planned_reads, nullptr, indirect.key_count);
if (indirect.sources.empty()) {
MarkCleanReads(planned_reads, Source(plan, indirect.table_source));
} else {
for (const auto candidate: indirect.sources) {
MarkCleanReads(planned_reads, Source(plan, candidate));
}
}
}
// Clean writable addresses prove that shader writes cannot overlap captured reads.
for (const auto& buffer: plan.info.buffers) {
if (buffer.written) MarkCleanFlatSlots(plan, Source(plan, buffer.source), plan.clean_flat_slots);
if (buffer.written) MarkCleanReads(planned_reads, Source(plan, buffer.source));
}
for (const auto& image: plan.info.images) {
if (image.written) MarkCleanFlatSlots(plan, Source(plan, image.source), plan.clean_flat_slots);
if (image.written) MarkCleanReads(planned_reads, Source(plan, image.source));
}
}
if (capture_indirect_reads) plan.resource_tracking_complete &= !program.has_address_writes;
@@ -1086,7 +1097,7 @@ bool MaterializeResources(const ResourcePlan& program, const SrtRuntime& runtime
observed.read_memory = CaptureOrdinaryRead;
}
SrtWalker clean(program, CleanRuntime(observed));
SrtWalker walker(program, observed, program.clean_flat_slots,
SrtWalker walker(program, observed,
capture_reads || program.requires_specialization_memory ? &clean : nullptr);
if (!walker.RefreshFlatBuffer(snapshot.flattened_srt)) {
return false;
@@ -1229,8 +1240,7 @@ bool MaterializeResources(const ResourcePlan& program, const SrtRuntime& runtime
}
void ApplyResourceSpecialization(Program& program, const ResourceSpecialization& specialization) {
EXIT_IF(!program.resource_tracking_complete || program.shader_info_complete ||
program.binding_layout_complete);
EXIT_IF(!program.resource_tracking_complete || program.shader_info_complete);
EXIT_IF(program.info.buffers.size() > specialization.buffers.size() ||
program.info.images.size() > specialization.images.size());
@@ -613,8 +613,7 @@ private:
value = value.Resolve();
const auto* phi = value.TryInstruction();
if (m_shader_writes || phi == nullptr || phi->GetOpcode() != ValueOpcode::Phi ||
phi->NumArgs() != 2u || phi->NumPhiBlocks() != 2u || phi->GetType() != Type::U32 ||
m_program.blocks.size() != m_program.block_info.size()) {
phi->NumArgs() != 2u || phi->NumPhiBlocks() != 2u || phi->GetType() != Type::U32) {
return value;
}
const auto* merge = phi->Parent();
@@ -637,7 +636,7 @@ private:
if (branch == merge || branch->ImmSuccessors().size() != 2u) {
return value;
}
std::array<uint32_t, 2> target_ids;
std::array<const Block*, 2> targets;
for (uint32_t arm = 0; arm < 2; arm++) {
const auto* incoming = phi->PhiBlock(arm);
if (incoming == merge ||
@@ -648,29 +647,20 @@ private:
incoming->ImmSuccessors()[0] != merge))) {
return value;
}
const auto* target = incoming == branch ? merge : incoming;
const auto it = std::ranges::find(m_program.blocks, target);
if (it == m_program.blocks.end()) {
return value;
}
target_ids[arm] = m_program.block_info[it - m_program.blocks.begin()].id;
targets[arm] = incoming == branch ? merge : incoming;
}
const auto branch_it = std::ranges::find(m_program.blocks, branch);
if (branch_it == m_program.blocks.end()) {
return value;
}
const auto& info = m_program.block_info[branch_it - m_program.blocks.begin()];
const auto& info = *branch;
const auto& term = info.terminator;
if (term.kind != CFG::TerminatorKind::ConditionalBranch ||
!((term.true_block == target_ids[0] && term.false_block == target_ids[1]) ||
(term.false_block == target_ids[0] && term.true_block == target_ids[1])) ||
!((term.true_block == targets[0] && term.false_block == targets[1]) ||
(term.false_block == targets[0] && term.true_block == targets[1])) ||
!ValidateRuntimeValue(m_program, info.condition, RuntimeValueType::Integer) ||
!ValidateRuntimeValue(m_program, phi->Arg(0)) ||
!ValidateRuntimeValue(m_program, phi->Arg(1))) {
return value;
}
// Retain a host expression; replacing the GPU Phi would break SSA dominance.
const auto true_arg = term.true_block == target_ids[0] ? 0u : 1u;
const auto true_arg = term.true_block == targets[0] ? 0u : 1u;
auto& selected = m_program.value_storage.emplace_back(ValueOpcode::SelectU32);
selected.SetArg(0, info.condition);
selected.SetArg(1, phi->Arg(true_arg));
@@ -852,9 +842,9 @@ private:
if (entry.source.dwords[word].Resolve() == Value(read))
entry.source.dwords[word] = flat;
}
for (auto& info: m_program.block_info) {
if (info.condition.Resolve() == Value(read)) info.condition = flat;
if (info.indirect_target.Resolve() == Value(read)) info.indirect_target = flat;
for (auto* block: m_program.blocks) {
if (block->condition.Resolve() == Value(read)) block->condition = flat;
if (block->indirect_target.Resolve() == Value(read)) block->indirect_target = flat;
}
if (keep) {
if (resolved->planning_handle.IsEmpty()) {
@@ -1022,9 +1012,6 @@ private:
};
bool NonzeroOnEntry(Value value, const Block* block) const {
if (m_program.blocks.size() != m_program.block_info.size()) {
return false;
}
// Each unique predecessor must execute before this use. Stop at joins: an
// unrelated comparison is not a bound on FindILsb's zero-input sentinel.
for (size_t depth = 0; block != nullptr && depth < m_program.blocks.size(); ++depth) {
@@ -1098,15 +1085,11 @@ private:
}
std::optional<EdgePredicate> ConditionalEdge(const Block* from, const Block* to) const {
const auto position = std::ranges::find(m_program.blocks, from);
const auto target = std::ranges::find(m_program.blocks, to);
if (position == m_program.blocks.end() || target == m_program.blocks.end()) return {};
const auto& info = m_program.block_info[position - m_program.blocks.begin()];
const auto& term = info.terminator;
const auto id = m_program.block_info[target - m_program.blocks.begin()].id;
const auto& term = from->terminator;
if (term.kind != CFG::TerminatorKind::ConditionalBranch ||
(term.true_block == id) == (term.false_block == id)) return {};
EdgePredicate edge {info.condition, term.true_block == id};
(term.true_block == to) == (term.false_block == to))
return {};
EdgePredicate edge {from->condition, term.true_block == to};
while (const auto* inst = edge.condition.Resolve().TryInstruction()) {
if (inst->GetOpcode() == ValueOpcode::LogicalNot) {
edge.positive = !edge.positive;
@@ -1470,9 +1453,9 @@ private:
const Inst* BoundedLoop(Value key, const Block* use, const auto& accepts_bound) const {
const auto* phi = key.Resolve().TryInstruction();
if (phi == nullptr || phi->GetOpcode() != ValueOpcode::Phi ||
phi->GetType() != Type::U32 || phi->NumArgs() != 2u || phi->NumPhiBlocks() != 2u ||
m_program.blocks.size() != m_program.block_info.size()) return {};
if (phi == nullptr || phi->GetOpcode() != ValueOpcode::Phi || phi->GetType() != Type::U32 ||
phi->NumArgs() != 2u || phi->NumPhiBlocks() != 2u)
return {};
const Block* increment_block = nullptr;
uint32_t initial_arm = 0;
for (uint32_t initial = 0; initial < 2u; ++initial) {
@@ -1904,22 +1887,13 @@ private:
}
bool ImpossibleSwitchEdge(const Block* from, const Block* to) const {
const auto from_it = std::ranges::find(m_program.blocks, from);
const auto to_it = std::ranges::find(m_program.blocks, to);
if (from_it == m_program.blocks.end() || to_it == m_program.blocks.end() ||
m_program.block_info.size() != m_program.blocks.size()) return false;
const auto& info = m_program.block_info[from_it - m_program.blocks.begin()];
const auto& term = info.terminator;
if (term.kind != CFG::TerminatorKind::IndirectBranch ||
term.indirect_selector_code == UINT32_MAX ||
term.indirect_selector_values.empty() ||
term.indirect_selector_values.size() != term.indirect_selector_targets.size()) return false;
const auto target = m_program.block_info[to_it - m_program.blocks.begin()].id;
const auto maximum = SelectorMaximum(info.indirect_target);
const auto& term = from->terminator;
if (term.kind != CFG::TerminatorKind::IndirectBranch || !term.indexed) return false;
const auto maximum = SelectorMaximum(from->indirect_target);
bool found = false;
for (size_t i = 0; i < term.indirect_selector_values.size(); ++i) {
if (term.indirect_selector_targets[i] != target) continue;
if (term.indirect_selector_values[i] <= maximum) return false;
for (const auto& branch: term.cases) {
if (branch.target != to) continue;
if (branch.value <= maximum) return false;
found = true;
}
return found;
@@ -75,128 +75,11 @@ void ValidateOptions(const Program& program, ShaderStageInputInfo input_info) {
}
}
void ValidateValueReferences(const Program& program, ShaderStageInputInfo input_info) {
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
switch (inst.GetOpcode()) {
case ValueOpcode::GetAttribute: {
if (!inst.Arg(0).IsImmediate() || inst.Arg(0).GetType() != Type::U32 ||
!inst.Arg(1).IsImmediate() || inst.Arg(1).GetType() != Type::U32) {
return Fail("typed attribute reference is not constant");
}
if ((program.stage == ShaderType::Vertex ||
program.stage == ShaderType::Local) &&
(inst.Arg(1).U32() >= 4u ||
inst.Arg(0).U32() >=
static_cast<uint32_t>(input_info.vertex->resources_num))) {
return Fail("vertex input reference is out of range");
}
if (program.stage == ShaderType::Pixel &&
(inst.Arg(1).U32() >= 4u ||
inst.Arg(0).U32() >= input_info.pixel->input_num)) {
return Fail("pixel input reference is out of range");
}
break;
}
case ValueOpcode::GetInterpolationParameter: {
if (program.stage != ShaderType::Pixel || !inst.Arg(0).IsImmediate() ||
inst.Arg(0).GetType() != Type::U32 || !inst.Arg(1).IsImmediate() ||
inst.Arg(1).GetType() != Type::U32 || !inst.Arg(2).IsImmediate() ||
inst.Arg(2).GetType() != Type::U32) {
return Fail("interpolation parameter reference is invalid");
}
if (inst.Arg(0).U32() >= input_info.pixel->input_num ||
inst.Arg(1).U32() >= 4u || inst.Arg(2).U32() >= 3u) {
return Fail("interpolation parameter reference is out of range");
}
break;
}
case ValueOpcode::GetBuiltin: {
if (!inst.Arg(0).IsImmediate() || inst.Arg(0).GetType() != Type::U32 ||
!inst.Arg(1).IsImmediate() || inst.Arg(1).GetType() != Type::U32) {
return Fail("typed builtin reference is not constant");
}
const auto kind = static_cast<StageInputKind>(inst.Arg(0).U32());
const auto component = inst.Arg(1).U32();
switch (kind) {
case StageInputKind::PackedAncillary:
return Fail("packed pixel ancillary input has an unsupported live use");
case StageInputKind::Layer:
case StageInputKind::SampleId:
if (program.stage != ShaderType::Pixel || component != 0u) {
return Fail("typed pixel scalar input is invalid");
}
break;
case StageInputKind::VertexIndex:
case StageInputKind::InvocationId:
case StageInputKind::PrimitiveId:
case StageInputKind::InstanceIndex:
case StageInputKind::FrontFacing:
case StageInputKind::LocalInvocationIndex:
if (component != 0u) {
return Fail("typed scalar builtin component is out of range");
}
break;
case StageInputKind::FragCoord:
if (component >= 4u) {
return Fail("typed fragment-coordinate component is out of range");
}
break;
case StageInputKind::BaryCoordSmooth:
case StageInputKind::BaryCoordSmoothSample:
case StageInputKind::BaryCoordSmoothCentroid:
case StageInputKind::BaryCoordNoPerspective:
if (component >= 2u) {
return Fail("typed barycentric component is out of range");
}
break;
case StageInputKind::TessCoord:
case StageInputKind::WorkgroupId:
case StageInputKind::LocalInvocationId:
case StageInputKind::GlobalInvocationId:
if (component >= 3u) {
return Fail("typed invocation builtin component is out of range");
}
break;
case StageInputKind::Parameter:
default: return Fail("typed builtin kind is invalid");
}
break;
}
case ValueOpcode::SetAttribute: {
const auto index = inst.Flags<ExportFlags>().index;
if (index >= program.export_info.size()) {
return Fail("typed export metadata index is out of range");
}
const auto& exp = program.export_info[index];
if (exp.kind == ExportTargetKind::Position && exp.index != 0 && exp.en != 0 &&
program.stage != ShaderType::Vertex && program.stage != ShaderType::Mesh &&
program.stage != ShaderType::TessellationEvaluation) {
return Fail("auxiliary position export requires a vertex, mesh, or "
"tessellation evaluation shader");
}
break;
}
default: break;
}
}
}
}
void CollectVertexInputs(const Program& program, const ShaderVertexInputInfo* vertex,
ShaderInfo& info) {
void CollectVertexInputs(const ShaderVertexInputInfo* vertex, ShaderInfo& info,
const std::array<uint8_t, 32>& used_components) {
AddInput(info, StageInputKind::VertexIndex, 0, 1, "gl_VertexIndex");
AddInput(info, StageInputKind::InstanceIndex, 0, 1, "gl_InstanceIndex");
uint32_t used_components[ShaderVertexInputInfo::RES_MAX] = {};
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
if (inst.GetOpcode() == ValueOpcode::GetAttribute) {
const auto attr = inst.Arg(0).U32();
const auto chan = inst.Arg(1).U32();
used_components[attr] = std::max(used_components[attr], chan + 1u);
}
}
}
for (uint32_t attr = 0; attr < static_cast<uint32_t>(vertex->resources_num); attr++) {
if (used_components[attr] != 0) {
AddInput(info, StageInputKind::Parameter, attr, used_components[attr],
@@ -205,37 +88,25 @@ void CollectVertexInputs(const Program& program, const ShaderVertexInputInfo* ve
}
}
void CollectPixelInputs(const Program& program, const ShaderPixelInputInfo* pixel,
ShaderInfo& info) {
void CollectPixelInputs(const ShaderPixelInputInfo* pixel, ShaderInfo& info,
std::array<bool, 32>& per_vertex,
const std::array<uint8_t, 32>& used_components) {
if (pixel->HasPositionInput()) {
AddInput(info, StageInputKind::FragCoord, 0, 4, "gl_FragCoord");
}
if (pixel->ps_front_face) {
AddInput(info, StageInputKind::FrontFacing, 0, 1, "gl_FrontFacing");
}
std::array<bool, 32> per_vertex {};
std::array<bool, 32> interpolated {};
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
if (inst.GetOpcode() == ValueOpcode::GetAttribute) {
interpolated[inst.Arg(0).U32()] = true;
} else if (inst.GetOpcode() == ValueOpcode::GetInterpolationParameter) {
const auto input = inst.Arg(0).U32();
const auto mode = inst.Arg(2).U32();
per_vertex[input] =
per_vertex[input] || mode < 2u || !ShaderPixelParameterIsFlat(*pixel, input);
}
}
}
// Aliases of a vertex output share one SPIR-V interface variable. If any
// alias reads raw vertices, interpolate the other aliases from those too.
// Mixed interpolation modes share raw vertices at their guest export slot.
// Rectangle expansion alone supplies separate flat and smooth outputs.
for (uint32_t input = 0; input < pixel->input_num; input++) {
for (uint32_t alias = 0; alias < pixel->input_num; alias++) {
const bool same_mode = ShaderPixelParameterIsFlat(*pixel, input) ==
ShaderPixelParameterIsFlat(*pixel, alias);
if (ShaderPixelParameterMappedLocation(*pixel, input) ==
ShaderPixelParameterMappedLocation(*pixel, alias) &&
ShaderPixelParameterIsFlat(*pixel, input) ==
ShaderPixelParameterIsFlat(*pixel, alias)) {
per_vertex[input] = per_vertex[input] || per_vertex[alias];
(pixel->parameter_mode != ShaderPixelParameterMode::Rectangle || same_mode)) {
per_vertex[input] = per_vertex[input] || per_vertex[alias] || !same_mode;
}
}
}
@@ -244,7 +115,8 @@ void CollectPixelInputs(const Program& program, const ShaderPixelInputInfo* pixe
per_vertex[input]);
}
for (uint32_t input = 0; input < pixel->input_num; input++) {
if (interpolated[input] && per_vertex[input]) {
if (used_components[input] != 0 && per_vertex[input] &&
!ShaderPixelParameterIsFlat(*pixel, input)) {
const auto kind = pixel->ps_no_perspective ? StageInputKind::BaryCoordNoPerspective
: StageInputKind::BaryCoordSmooth;
AddInput(info, kind, 0, 3,
@@ -269,147 +141,411 @@ void CollectComputeInputs(const ShaderComputeInputInfo* compute, ShaderInfo& inf
}
}
void CollectBuiltinInputs(const Program& program, ShaderInfo& info) {
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
if (program.stage == ShaderType::TessellationControl &&
inst.GetOpcode() == ValueOpcode::LaneId) {
AddInput(info, StageInputKind::InvocationId, 0, 1, "gl_InvocationID");
}
if (inst.GetOpcode() != ValueOpcode::GetBuiltin) {
continue;
}
const auto kind = static_cast<StageInputKind>(inst.Arg(0).U32());
switch (kind) {
case StageInputKind::VertexIndex:
AddInput(info, kind, 0, 1, "gl_VertexIndex");
break;
case StageInputKind::InstanceIndex:
AddInput(info, kind, 0, 1, "gl_InstanceIndex");
break;
case StageInputKind::InvocationId:
AddInput(info, kind, 0, 1, "gl_InvocationID");
break;
case StageInputKind::PrimitiveId:
AddInput(info, kind, 0, 1, "gl_PrimitiveID");
break;
case StageInputKind::TessCoord: AddInput(info, kind, 0, 3, "gl_TessCoord"); break;
case StageInputKind::FragCoord: AddInput(info, kind, 0, 4, "gl_FragCoord"); break;
case StageInputKind::FrontFacing:
AddInput(info, kind, 0, 1, "gl_FrontFacing");
break;
case StageInputKind::Layer: AddInput(info, kind, 0, 1, "gl_Layer"); break;
case StageInputKind::SampleId: AddInput(info, kind, 0, 1, "gl_SampleID"); break;
case StageInputKind::BaryCoordSmoothSample:
AddInput(info, StageInputKind::SampleId, 0, 1, "gl_SampleID");
[[fallthrough]];
case StageInputKind::BaryCoordSmooth:
case StageInputKind::BaryCoordSmoothCentroid:
AddInput(info, StageInputKind::BaryCoordSmooth, 0, 3, "gl_BaryCoordKHR");
break;
case StageInputKind::BaryCoordNoPerspective:
AddInput(info, kind, 0, 3, "gl_BaryCoordNoPerspKHR");
break;
case StageInputKind::WorkgroupId:
AddInput(info, kind, 0, 3, "gl_WorkGroupID");
break;
case StageInputKind::LocalInvocationId:
AddInput(info, kind, 0, 3, "gl_LocalInvocationID");
break;
case StageInputKind::LocalInvocationIndex:
AddInput(info, kind, 0, 1, "gl_LocalInvocationIndex");
break;
case StageInputKind::GlobalInvocationId:
AddInput(info, kind, 0, 3, "gl_GlobalInvocationID");
break;
case StageInputKind::PackedAncillary:
case StageInputKind::Parameter: break;
}
}
void AddBuiltinInput(ShaderInfo& info, StageInputKind kind) {
switch (kind) {
case StageInputKind::VertexIndex:
AddInput(info, kind, 0, 1, "gl_VertexIndex");
break;
case StageInputKind::InstanceIndex:
AddInput(info, kind, 0, 1, "gl_InstanceIndex");
break;
case StageInputKind::InvocationId:
AddInput(info, kind, 0, 1, "gl_InvocationID");
break;
case StageInputKind::PrimitiveId:
AddInput(info, kind, 0, 1, "gl_PrimitiveID");
break;
case StageInputKind::TessCoord: AddInput(info, kind, 0, 3, "gl_TessCoord"); break;
case StageInputKind::FragCoord: AddInput(info, kind, 0, 4, "gl_FragCoord"); break;
case StageInputKind::FrontFacing:
AddInput(info, kind, 0, 1, "gl_FrontFacing");
break;
case StageInputKind::Layer: AddInput(info, kind, 0, 1, "gl_Layer"); break;
case StageInputKind::SampleId: AddInput(info, kind, 0, 1, "gl_SampleID"); break;
case StageInputKind::BaryCoordSmoothSample:
AddInput(info, StageInputKind::SampleId, 0, 1, "gl_SampleID");
[[fallthrough]];
case StageInputKind::BaryCoordSmooth:
case StageInputKind::BaryCoordSmoothCentroid:
AddInput(info, StageInputKind::BaryCoordSmooth, 0, 3, "gl_BaryCoordKHR");
break;
case StageInputKind::BaryCoordNoPerspective:
AddInput(info, kind, 0, 3, "gl_BaryCoordNoPerspKHR");
break;
case StageInputKind::WorkgroupId:
AddInput(info, kind, 0, 3, "gl_WorkGroupID");
break;
case StageInputKind::LocalInvocationId:
AddInput(info, kind, 0, 3, "gl_LocalInvocationID");
break;
case StageInputKind::LocalInvocationIndex:
AddInput(info, kind, 0, 1, "gl_LocalInvocationIndex");
break;
case StageInputKind::GlobalInvocationId:
AddInput(info, kind, 0, 3, "gl_GlobalInvocationID");
break;
case StageInputKind::PackedAncillary:
case StageInputKind::Parameter: break;
}
}
void CollectOutputs(const Program& program, ShaderStageInputInfo input_info, ShaderInfo& info) {
void CollectOutput(const Program& program, ShaderStageInputInfo input_info,
ShaderInfo& info, const Inst& inst) {
const bool alpha_remap = program.stage == ShaderType::Pixel && input_info.pixel != nullptr &&
input_info.pixel->alpha_blend_source != ShaderAlphaBlendSource::None;
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
if (inst.GetOpcode() != ValueOpcode::SetAttribute) {
continue;
const auto& export_info = program.export_info[inst.Flags<ExportFlags>().index];
if (export_info.kind == ExportTargetKind::MrtZ) {
if (program.stage == ShaderType::Pixel && (export_info.en & 0x1u) != 0 &&
input_info.pixel->ps_depth_export_enable) {
AddOutput(info, StageOutputKind::Depth, 0, 0, "gl_FragDepth");
}
if (program.stage == ShaderType::Pixel && (export_info.en & 0x4u) != 0 &&
input_info.pixel->ps_sample_mask_export_enable) {
AddOutput(info, StageOutputKind::SampleMask, 0, 0, "gl_SampleMask");
}
return;
}
if (export_info.en == 0) {
return;
}
switch (export_info.kind) {
case ExportTargetKind::Position:
if (export_info.index == 0) {
AddOutput(info, StageOutputKind::Position, 0, 0, "out_position");
break;
}
const auto& export_info = program.export_info[inst.Flags<ExportFlags>().index];
if (export_info.kind == ExportTargetKind::MrtZ) {
if (program.stage == ShaderType::Pixel && (export_info.en & 0x1u) != 0 &&
input_info.pixel->ps_depth_export_enable) {
AddOutput(info, StageOutputKind::Depth, 0, 0, "gl_FragDepth");
}
if (program.stage == ShaderType::Pixel && (export_info.en & 0x4u) != 0 &&
input_info.pixel->ps_sample_mask_export_enable) {
AddOutput(info, StageOutputKind::SampleMask, 0, 0, "gl_SampleMask");
}
continue;
if (export_info.compr) {
return Fail("compressed auxiliary position export is unsupported");
}
if (export_info.en == 0) {
continue;
for (uint32_t component = 0; component < 4; component++) {
if ((export_info.en & (1u << component)) == 0) {
continue;
}
const auto output = DecodePositionExportComponent(
input_info.vertex->pa_cl_vs_out_cntl, export_info.index, component);
if (output.viewport) {
AddOutput(info, StageOutputKind::ViewportIndex, 0, 0, "gl_ViewportIndex");
}
if (output.point_size) {
AddOutput(info, StageOutputKind::PointSize, 0, 0, "gl_PointSize");
}
if (output.layer) {
AddOutput(info, StageOutputKind::Layer, 0, 0, "gl_Layer");
}
if (output.clip_distance != UINT32_MAX) {
AddOutput(info, StageOutputKind::ClipDistance, output.clip_distance, 0,
"gl_ClipDistance");
}
if (output.cull_distance != UINT32_MAX) {
AddOutput(info, StageOutputKind::CullDistance, output.cull_distance, 0,
"gl_CullDistance");
}
}
switch (export_info.kind) {
case ExportTargetKind::Position:
if (export_info.index == 0) {
AddOutput(info, StageOutputKind::Position, 0, 0, "out_position");
break;
}
if (export_info.compr) {
return Fail("compressed auxiliary position export is unsupported");
}
for (uint32_t component = 0; component < 4; component++) {
if ((export_info.en & (1u << component)) == 0) {
continue;
}
const auto output = DecodePositionExportComponent(
input_info.vertex->pa_cl_vs_out_cntl, export_info.index, component);
if (output.viewport) {
AddOutput(info, StageOutputKind::ViewportIndex, 0, 0, "gl_ViewportIndex");
}
if (output.point_size) {
AddOutput(info, StageOutputKind::PointSize, 0, 0, "gl_PointSize");
}
if (output.layer) {
AddOutput(info, StageOutputKind::Layer, 0, 0, "gl_Layer");
}
if (output.clip_distance != UINT32_MAX) {
AddOutput(info, StageOutputKind::ClipDistance, output.clip_distance, 0,
"gl_ClipDistance");
}
if (output.cull_distance != UINT32_MAX) {
AddOutput(info, StageOutputKind::CullDistance, output.cull_distance, 0,
"gl_CullDistance");
}
}
break;
case ExportTargetKind::Parameter:
AddOutput(info, StageOutputKind::Parameter, export_info.index,
export_info.index, fmt::format("out_param_{}", export_info.index));
break;
case ExportTargetKind::Mrt: {
if (alpha_remap && export_info.index != 0) {
break;
}
const auto slot = input_info.pixel->dual_source_blending
? 0u : ShaderPixelExportTarget(input_info.pixel->target_shader_mask,
export_info.index);
if (slot >= 8) {
break;
}
AddOutput(info, StageOutputKind::Mrt, export_info.index, slot,
fmt::format("out_mrt_{}", export_info.index));
if (alpha_remap) {
AddOutput(info, StageOutputKind::Mrt, 1, 0, "out_mrt_1");
break;
case ExportTargetKind::Parameter:
AddOutput(info, StageOutputKind::Parameter, export_info.index,
export_info.index, fmt::format("out_param_{}", export_info.index));
break;
case ExportTargetKind::Mrt: {
if (alpha_remap && export_info.index != 0) {
break;
}
const auto slot = input_info.pixel->dual_source_blending
? 0u : ShaderPixelExportTarget(input_info.pixel->target_shader_mask,
export_info.index);
if (slot >= 8) {
break;
}
AddOutput(info, StageOutputKind::Mrt, export_info.index, slot,
fmt::format("out_mrt_{}", export_info.index));
if (alpha_remap) {
AddOutput(info, StageOutputKind::Mrt, 1, 0, "out_mrt_1");
}
break;
}
default: break;
}
}
struct InputUsage {
std::array<uint8_t, 32> components {};
std::array<bool, 32> per_vertex {};
std::array<bool, NumScalarRegs> user_data {};
std::array<StageInputKind, static_cast<size_t>(StageInputKind::Parameter)> builtins {};
uint32_t builtin_mask = 0;
uint32_t builtin_count = 0;
void AddBuiltin(StageInputKind kind) {
const auto bit = 1u << static_cast<uint32_t>(kind);
if ((builtin_mask & bit) == 0) {
builtin_mask |= bit;
builtins[builtin_count++] = kind;
}
}
};
void Visit(Program& program, ShaderStageInputInfo input_info, InputUsage& inputs,
const Inst& inst) {
auto& info = program.info;
const auto op = inst.GetOpcode();
const auto buffer_access = BufferAccessOf(op);
const auto address_access = AddressOpcodeInfoOf(op).access;
const auto shared_access = SharedAccessOf(op);
info.float64 |= inst.GetType() == Type::F64;
info.buffer_int64_atomics |= buffer_access == BufferAccess::Atomic &&
inst.GetType() == Type::U64;
if (buffer_access != BufferAccess::None || address_access != AddressAccess::None ||
shared_access != SharedAccess::None) {
const auto index = inst.Flags<MemoryFlags>().index;
if (index >= program.memory_info.size()) {
return Fail("typed shader contains invalid memory metadata");
}
const auto& memory = program.memory_info[index];
const auto kind = memory.kind;
if (!memory.planning_only) {
info.uses_lds |= kind == ResourceKind::FlatLocal;
if (shared_access != SharedAccess::None) {
info.uses_lds |= kind == ResourceKind::Lds;
info.uses_gds |= kind == ResourceKind::Gds;
} else if (kind == ResourceKind::Buffer || kind == ResourceKind::ScalarBuffer) {
if (memory.resource >= info.buffers.size() || memory.resource >= ShaderInfo::MaxBuffers) {
return Fail("buffer operation has invalid resource metadata");
}
info.live_buffers |= uint64_t{1} << memory.resource;
for (const auto child: info.buffers[memory.resource].indirect_resources) {
if (child >= info.buffers.size()) {
return Fail("indirect buffer operation has invalid resource metadata");
}
break;
info.live_buffers |= uint64_t{1} << child;
}
default: break;
}
}
if (address_access != AddressAccess::None) {
if (kind == ResourceKind::Scratch || kind == ResourceKind::FlatLocal) {
if (program.scratch_dwords == 0) {
return Fail("scratch operation has no per-thread storage");
}
info.function_scratch = true;
info.function_lds |= kind == ResourceKind::FlatLocal &&
program.stage != ShaderType::Compute && program.stage != ShaderType::Mesh;
} else if (address_access == AddressAccess::Write) {
return Fail("writable FLAT/GLOBAL addresses require GPU ownership tracking");
} else {
info.coherent_buffers |= memory.coherent;
}
}
if (buffer_access != BufferAccess::None) {
if (kind == ResourceKind::IndirectBuffer && op != ValueOpcode::ReadConstBuffer) {
info.subgroup_local_invocation_id = true;
}
if (kind == ResourceKind::Buffer) {
info.coherent_buffers |= memory.coherent;
if (memory.resource >= info.buffers.size()) {
return Fail("buffer operation has invalid resource metadata");
}
const auto bits = StorageBufferElementBits(program, memory);
info.buffer_u8 |= bits == 8u;
info.buffer_u16 |= bits == 16u;
if ((info.buffers[memory.resource].packed_stride & (1u << 20u)) != 0u) {
if (program.stage != ShaderType::Compute) {
return Fail("buffer ADD_TID is only valid for compute shaders");
}
info.subgroup_local_invocation_id = true;
}
}
}
if (shared_access != SharedAccess::None) {
if (kind != ResourceKind::Lds && kind != ResourceKind::Gds) {
return Fail("shared operation has invalid resource kind");
}
if (shared_access == SharedAccess::Atomic && SharedComponentCount(op) == 2u) {
if (kind != ResourceKind::Lds || program.stage != ShaderType::Compute) {
return Fail("64-bit shared atomics require compute LDS");
}
info.shared_int64_atomics = true;
}
info.function_lds |= kind == ResourceKind::Lds &&
program.stage != ShaderType::Compute && program.stage != ShaderType::Mesh;
if (shared_access == SharedAccess::Append || shared_access == SharedAccess::Consume) {
info.subgroup_ballot = true;
info.subgroup_shuffle = true;
info.subgroup_local_invocation_id = true;
}
}
}
switch (inst.GetOpcode()) {
case ValueOpcode::GetAttribute: {
if (!inst.Arg(0).IsImmediate() || inst.Arg(0).GetType() != Type::U32 ||
!inst.Arg(1).IsImmediate() || inst.Arg(1).GetType() != Type::U32) {
return Fail("typed attribute reference is not constant");
}
if ((program.stage == ShaderType::Vertex ||
program.stage == ShaderType::Local) &&
(inst.Arg(1).U32() >= 4u ||
inst.Arg(0).U32() >=
static_cast<uint32_t>(input_info.vertex->resources_num))) {
return Fail("vertex input reference is out of range");
}
if (program.stage == ShaderType::Pixel &&
(inst.Arg(1).U32() >= 4u ||
inst.Arg(0).U32() >= input_info.pixel->input_num)) {
return Fail("pixel input reference is out of range");
}
if (program.stage == ShaderType::Vertex || program.stage == ShaderType::Local ||
program.stage == ShaderType::Pixel) {
auto& components = inputs.components[inst.Arg(0).U32()];
components = std::max(components, static_cast<uint8_t>(inst.Arg(1).U32() + 1u));
}
break;
}
case ValueOpcode::GetInterpolationParameter: {
if (program.stage != ShaderType::Pixel || !inst.Arg(0).IsImmediate() ||
inst.Arg(0).GetType() != Type::U32 || !inst.Arg(1).IsImmediate() ||
inst.Arg(1).GetType() != Type::U32 || !inst.Arg(2).IsImmediate() ||
inst.Arg(2).GetType() != Type::U32) {
return Fail("interpolation parameter reference is invalid");
}
if (inst.Arg(0).U32() >= input_info.pixel->input_num ||
inst.Arg(1).U32() >= 4u || inst.Arg(2).U32() >= 3u) {
return Fail("interpolation parameter reference is out of range");
}
const auto input = inst.Arg(0).U32();
inputs.per_vertex[input] |= inst.Arg(2).U32() < 2u ||
!ShaderPixelParameterIsFlat(*input_info.pixel, input);
break;
}
case ValueOpcode::GetBuiltin: {
if (!inst.Arg(0).IsImmediate() || inst.Arg(0).GetType() != Type::U32 ||
!inst.Arg(1).IsImmediate() || inst.Arg(1).GetType() != Type::U32) {
return Fail("typed builtin reference is not constant");
}
const auto kind = static_cast<StageInputKind>(inst.Arg(0).U32());
const auto component = inst.Arg(1).U32();
switch (kind) {
case StageInputKind::PackedAncillary:
return Fail("packed pixel ancillary input has an unsupported live use");
case StageInputKind::Layer:
case StageInputKind::SampleId:
if (program.stage != ShaderType::Pixel || component != 0u) {
return Fail("typed pixel scalar input is invalid");
}
break;
case StageInputKind::VertexIndex:
case StageInputKind::InvocationId:
case StageInputKind::PrimitiveId:
case StageInputKind::InstanceIndex:
case StageInputKind::FrontFacing:
case StageInputKind::LocalInvocationIndex:
if (component != 0u) {
return Fail("typed scalar builtin component is out of range");
}
break;
case StageInputKind::FragCoord:
if (component >= 4u) {
return Fail("typed fragment-coordinate component is out of range");
}
break;
case StageInputKind::BaryCoordSmooth:
case StageInputKind::BaryCoordSmoothSample:
case StageInputKind::BaryCoordSmoothCentroid:
case StageInputKind::BaryCoordNoPerspective:
if (component >= 2u) {
return Fail("typed barycentric component is out of range");
}
break;
case StageInputKind::TessCoord:
case StageInputKind::WorkgroupId:
case StageInputKind::LocalInvocationId:
case StageInputKind::GlobalInvocationId:
if (component >= 3u) {
return Fail("typed invocation builtin component is out of range");
}
break;
case StageInputKind::Parameter:
default: return Fail("typed builtin kind is invalid");
}
inputs.AddBuiltin(kind);
break;
}
case ValueOpcode::SetAttribute: {
const auto index = inst.Flags<ExportFlags>().index;
if (index >= program.export_info.size()) {
return Fail("typed export metadata index is out of range");
}
const auto& exp = program.export_info[index];
if (exp.kind == ExportTargetKind::Position && exp.index != 0 && exp.en != 0 &&
program.stage != ShaderType::Vertex && program.stage != ShaderType::Mesh &&
program.stage != ShaderType::TessellationEvaluation) {
return Fail("auxiliary position export requires a vertex, mesh, or "
"tessellation evaluation shader");
}
info.pixel_valid_mask |= program.stage == ShaderType::Pixel && exp.vm;
CollectOutput(program, input_info, info, inst);
break;
}
case ValueOpcode::GetUserData: {
if (!inst.HasUses()) break;
if (inst.Arg(0).GetType() != Type::ScalarReg) {
return Fail("typed shader contains an invalid user-data register");
}
const auto index = RegIndex(inst.Arg(0).ScalarRegister());
if (index >= NumScalarRegs) {
return Fail("typed shader contains an invalid user-data register");
}
inputs.user_data[index] = true;
break;
}
case ValueOpcode::GetDispatchThreadExtent:
info.uses_dispatch_threads |= inst.HasUses();
break;
case ValueOpcode::ReadConst: info.uses_flattened_srt = true; break;
case ValueOpcode::BitwiseXor32: info.has_bitwise_xor = true; break;
case ValueOpcode::StoreCompletion: info.subgroup_barrier = true; break;
case ValueOpcode::BvhIntersect: info.bvh = true; break;
case ValueOpcode::ConditionRef:
info.subgroup_ballot |=
inst.Flags<CFG::BranchCondition>() != CFG::BranchCondition::ScalarInstruction;
break;
case ValueOpcode::Ballot: info.subgroup_ballot = true; break;
case ValueOpcode::DppMoveU32:
case ValueOpcode::ReadFirstLane:
case ValueOpcode::ReadLane: {
info.subgroup_ballot = true;
info.subgroup_shuffle = true;
if (inst.GetOpcode() == ValueOpcode::DppMoveU32) {
info.subgroup_local_invocation_id = true;
}
break;
}
case ValueOpcode::DppUpdateU32:
case ValueOpcode::WriteLane: {
info.subgroup_ballot = true;
info.subgroup_local_invocation_id = true;
break;
}
case ValueOpcode::SwizzleU32:
info.uses_swizzle = true;
[[fallthrough]];
case ValueOpcode::Permlane16U32:
case ValueOpcode::PermuteU32:
case ValueOpcode::BpermuteU32: {
info.subgroup_ballot = true;
info.subgroup_shuffle = true;
info.subgroup_local_invocation_id = true;
break;
}
case ValueOpcode::LaneId:
if (program.stage == ShaderType::TessellationControl) {
inputs.AddBuiltin(StageInputKind::InvocationId);
} else {
info.subgroup_local_invocation_id = true;
}
break;
case ValueOpcode::ImageQueryLod: info.compute_derivatives = true; break;
case ValueOpcode::ImageGatherRaw:
info.image_gather_extended = true;
break;
default: break;
}
}
@@ -421,29 +557,49 @@ void CollectShaderInfo(Program& program, ShaderStageInputInfo input_info) {
: "shader info already collected");
}
ValidateOptions(program, input_info);
ValidateValueReferences(program, input_info);
auto& next = program.info;
next.inputs.clear();
next.outputs.clear();
next.has_bitwise_xor =
std::any_of(program.blocks.begin(), program.blocks.end(), [](const auto* block) {
return std::any_of(block->begin(), block->end(), [](const auto& inst) {
return inst.GetOpcode() == ValueOpcode::BitwiseXor32;
});
});
auto& info = program.info;
if (info.buffers.size() > ShaderInfo::MaxBuffers) {
return Fail("buffer resource limit exceeded");
}
info.inputs.clear();
info.outputs.clear();
info.user_data_registers.clear();
info.live_buffers = 0;
info.has_bitwise_xor = info.uses_lds = info.uses_gds = info.uses_flattened_srt =
info.uses_dispatch_threads = info.uses_swizzle = false;
info.bvh = info.subgroup_ballot = info.subgroup_barrier = info.subgroup_shuffle =
info.subgroup_local_invocation_id = info.compute_derivatives = info.image_gather_extended =
info.function_lds = info.function_scratch = info.pixel_valid_mask = info.buffer_int64_atomics =
info.buffer_u8 = info.buffer_u16 = info.shared_int64_atomics = info.coherent_buffers = info.float64 = false;
InputUsage inputs;
for (const auto* block: program.blocks) {
for (const auto& inst: *block) {
Visit(program, input_info, inputs, inst);
}
}
for (uint32_t index = 0; index < inputs.user_data.size(); index++) {
if (inputs.user_data[index]) info.user_data_registers.push_back(index);
}
const auto uses_mapping = [](const auto& resource) {
return resource.indirect_root != UINT32_MAX;
};
info.uses_flattened_srt |= std::ranges::any_of(info.buffers, uses_mapping) ||
std::ranges::any_of(info.images, uses_mapping);
switch (program.stage) {
case ShaderType::Vertex:
case ShaderType::Local: CollectVertexInputs(program, input_info.vertex, next); break;
case ShaderType::Local: CollectVertexInputs(input_info.vertex, info, inputs.components); break;
case ShaderType::TessellationControl:
case ShaderType::TessellationEvaluation:
case ShaderType::Mesh: break;
case ShaderType::Pixel: CollectPixelInputs(program, input_info.pixel, next); break;
case ShaderType::Compute: CollectComputeInputs(input_info.compute, next); break;
case ShaderType::Pixel:
CollectPixelInputs(input_info.pixel, info, inputs.per_vertex, inputs.components);
break;
case ShaderType::Compute: CollectComputeInputs(input_info.compute, info); break;
default: return Fail("unsupported shader stage for info collection");
}
CollectBuiltinInputs(program, next);
CollectOutputs(program, input_info, next);
for (uint32_t i = 0; i < inputs.builtin_count; i++) {
AddBuiltinInput(info, inputs.builtins[i]);
}
program.shader_info_complete = true;
}
@@ -5,8 +5,7 @@
namespace Libs::Graphics::ShaderRecompiler::IR {
// Completes the immutable shader interface after resource tracking. On failure Program::info and
// all completion state remain unchanged.
// Collects the shader interface, resource usage and instruction features after resource tracking.
void CollectShaderInfo(Program& program, ShaderStageInputInfo input_info);
} // namespace Libs::Graphics::ShaderRecompiler::IR
@@ -324,10 +324,9 @@ private:
} // namespace
SrtWalker::SrtWalker(const ResourcePlan& program, const SrtRuntime& runtime,
std::span<const uint8_t> clean_flat_slots, SrtWalker* clean_evaluator,
Value active_mask)
: m_program(program), m_runtime(runtime), m_clean_flat_slots(clean_flat_slots),
m_clean_evaluator(clean_evaluator), m_active_mask(active_mask.Resolve()),
SrtWalker* clean_evaluator, Value active_mask)
: m_program(program), m_runtime(runtime), m_clean_evaluator(clean_evaluator),
m_active_mask(active_mask.Resolve()),
m_context(AcquireContext(program)) {}
SrtWalker::~SrtWalker() { --m_program.evaluation_depth; }
@@ -455,7 +454,7 @@ bool SrtWalker::EvaluateExtract(const Inst& inst, uint64_t& result) {
}
bool SrtWalker::EvaluateRawRead(const Inst& inst, uint64_t& result) {
const auto flags = inst.Flags<MemoryFlags>();
const auto flags = inst.Flags<SrtReadFlags>();
if (flags.index >= m_program.memory_info.size()) {
return false;
}
@@ -557,9 +556,8 @@ bool SrtWalker::EvaluateInst(const Inst& inst, uint64_t& result) {
return EvaluateWide(inst.Arg(0), result);
}
const auto clean_runtime = CleanRuntime(m_runtime);
SrtWalker clean_active(m_program, clean_runtime, {}, nullptr, inst.Arg(1));
SrtWalker active(m_program, m_runtime, m_clean_flat_slots, &clean_active,
inst.Arg(1));
SrtWalker clean_active(m_program, clean_runtime, nullptr, inst.Arg(1));
SrtWalker active(m_program, m_runtime, &clean_active, inst.Arg(1));
return active.EvaluateWide(inst.Arg(0), result);
}
case ValueOpcode::BitCastU32F32:
@@ -573,25 +571,14 @@ bool SrtWalker::EvaluateInst(const Inst& inst, uint64_t& result) {
result = static_cast<uint32_t>(a) |
(static_cast<uint64_t>(static_cast<uint32_t>(b)) << 32u);
return true;
case ValueOpcode::ReadConst: {
const auto slot = inst.Arg(1).Resolve();
if (!slot.IsImmediate() || slot.GetType() != Type::U32 ||
slot.U32() >= m_program.srt_reads.size()) {
return false;
}
if (slot.U32() < m_clean_flat_slots.size() &&
m_clean_flat_slots[slot.U32()] != 0u && m_clean_evaluator != nullptr) {
return m_clean_evaluator->EvaluateWide(m_program.srt_reads[slot.U32()].value,
result);
}
return EvaluateWide(m_program.srt_reads[slot.U32()].value, result);
}
case ValueOpcode::LoadAddressU32:
case ValueOpcode::ReadConstBuffer:
case ValueOpcode::LoadBufferU32:
if (IsRawRead(m_program, inst)) {
if (inst.GetOpcode() == ValueOpcode::LoadBufferU32 && m_clean_evaluator != nullptr &&
m_clean_evaluator->m_active_mask == m_active_mask) {
if (m_clean_evaluator != nullptr &&
(inst.Flags<SrtReadFlags>().clean != 0u ||
(inst.GetOpcode() == ValueOpcode::LoadBufferU32 &&
m_clean_evaluator->m_active_mask == m_active_mask))) {
return m_clean_evaluator->EvaluateWide(Value(const_cast<Inst*>(&inst)), result);
}
return EvaluateRawRead(inst, result);
@@ -934,8 +921,8 @@ bool SrtWalker::RefreshFlatBuffer(std::vector<uint32_t>& flat) {
const auto refresh = [&](uint32_t slot) {
if (slot >= m_program.srt_reads.size()) return false;
const auto& read = m_program.srt_reads[slot];
const bool clean = read.flat_offset < m_clean_flat_slots.size() &&
m_clean_flat_slots[read.flat_offset] != 0u;
const auto* inst = read.value.ResolveInstruction();
const bool clean = inst != nullptr && inst->Flags<SrtReadFlags>().clean != 0u;
if (clean && (m_clean_evaluator == nullptr || m_runtime.read_specialization_memory == nullptr))
return false;
auto& evaluator = clean ? *m_clean_evaluator : *this;
@@ -20,6 +20,13 @@ struct SrtRuntime {
std::span<const uint32_t> workgroup_counts;
};
// Retained reads no longer need the guest instruction PC. A clean read evaluates
// its address and bounds through the strict reader as well as the final DWORD.
struct SrtReadFlags {
uint32_t index = 0;
uint32_t clean = 0;
};
enum class RuntimeValueType { Any, Integer };
bool ValidateRuntimeValue(const ResourcePlan& program, Value value,
@@ -31,8 +38,7 @@ SrtRuntime CleanRuntime(SrtRuntime runtime);
class SrtWalker {
public:
SrtWalker(const ResourcePlan& program, const SrtRuntime& runtime,
std::span<const uint8_t> clean_flat_slots = {}, SrtWalker* clean_evaluator = nullptr,
Value active_mask = {});
SrtWalker* clean_evaluator = nullptr, Value active_mask = {});
~SrtWalker();
SrtWalker(const SrtWalker&) = delete;
SrtWalker& operator=(const SrtWalker&) = delete;
@@ -54,7 +60,6 @@ private:
const ResourcePlan& m_program;
SrtRuntime m_runtime;
std::span<const uint8_t> m_clean_flat_slots;
SrtWalker* m_clean_evaluator = nullptr;
Value m_active_mask;
ResourcePlan::EvaluationContext& m_context;
+1
View File
@@ -663,6 +663,7 @@ void BuildStageStaticKey(const ShaderPixelInputInfo& info, std::vector<uint32_t>
key.push_back(static_cast<uint32_t>(info.ps_front_face));
key.push_back(static_cast<uint32_t>(info.ps_ancillary));
key.push_back(static_cast<uint32_t>(info.ps_no_perspective));
key.push_back(static_cast<uint32_t>(info.parameter_mode));
key.push_back(static_cast<uint32_t>(info.ps_pixel_kill_enable));
key.push_back(static_cast<uint32_t>(info.ps_depth_export_enable));
key.push_back(static_cast<uint32_t>(info.ps_sample_mask_export_enable));
+2
View File
@@ -159,6 +159,7 @@ struct ShaderComputeInputInfo: ShaderWorkgroupInputInfo {
};
enum class ShaderAlphaBlendSource : uint8_t { None, SourceAlpha, SourceAlphaOne, SourceAlphaZero };
enum class ShaderPixelParameterMode : uint8_t { FirstVertex, LastVertex, Rectangle };
struct ShaderPixelInputInfo {
uint32_t interpolator_settings[32] = {0};
@@ -180,6 +181,7 @@ struct ShaderPixelInputInfo {
bool ps_front_face = false;
bool ps_ancillary = false;
bool ps_no_perspective = false;
ShaderPixelParameterMode parameter_mode = ShaderPixelParameterMode::FirstVertex;
bool ps_pixel_kill_enable = false;
bool ps_depth_export_enable = false;
bool ps_sample_mask_export_enable = false;
+4 -2
View File
@@ -30,6 +30,9 @@ uint32_t ShaderPixelParameterMappedLocation(const ShaderPixelInputInfo& info, ui
uint32_t ShaderPixelParameterLocation(const ShaderPixelInputInfo& info,
std::span<const uint32_t> active_inputs, uint32_t input) {
if (info.parameter_mode != ShaderPixelParameterMode::Rectangle) {
return ShaderPixelParameterMappedLocation(info, input);
}
std::array<bool, 32> used_locations {};
for (const auto active_input: active_inputs) {
used_locations[ShaderPixelParameterMappedLocation(info, active_input)] = true;
@@ -42,8 +45,7 @@ uint32_t ShaderPixelParameterLocation(const ShaderPixelInputInfo& info,
auto& location = group_locations[group];
if (location == UINT32_MAX) {
location = mapped;
// Smooth and custom interpolation read the same vertex output. Only
// differing flat/smooth rectangle outputs need separate locations.
// Rectangle expansion broadcasts flat aliases through separate outputs.
if (group_locations[group ^ 1u] != UINT32_MAX) {
location = 0;
while (location < used_locations.size() && used_locations[location]) {
+112 -57
View File
@@ -24,8 +24,12 @@ namespace Libs::Controller {
LIB_NAME("Pad", "Pad");
constexpr int PAD_ERROR_INVALID_ARG = -2137915391; /* 0x80920001 */
constexpr int PAD_ERROR_INVALID_HANDLE = -2137915389; /* 0x80920003 */
constexpr int PAD_ERROR_INVALID_ARG = -2137915391; /* 0x80920001 */
constexpr int PAD_ERROR_INVALID_HANDLE = -2137915389; /* 0x80920003 */
constexpr int PAD_ERROR_ALREADY_OPENED = -2137915388; /* 0x80920004 */
constexpr int PAD_ERROR_NOT_INITIALIZED = -2137915387; /* 0x80920005 */
constexpr int PAD_ERROR_NO_HANDLE = -2137915384; /* 0x80920008 */
constexpr int PAD_HANDLE = 1;
constexpr uint32_t RUMBLE_DURATION_MS = 0xffff;
constexpr uint32_t RELEASE_FLUSH_MS = 50;
@@ -107,6 +111,12 @@ public:
KYTY_CLASS_NO_COPY(GameController);
int InitPort();
int OpenPort(int user_id, int type, int index);
int GetPortHandle(int user_id, int type, int index);
int ClosePort(int handle);
int CheckPortHandle(int handle);
void Connect(int id);
void Disconnect(int id);
void Button(int id, uint32_t button, bool down);
@@ -131,6 +141,9 @@ public:
private:
static constexpr uint32_t STATES_MAX = 64;
enum class PortState { Uninitialized, Closed, Open };
int CheckPortHandleLocked(int handle) const;
void CheckActive();
void AddState();
@@ -140,6 +153,7 @@ private:
void UpdateTriggerEffectState();
Common::Mutex m_mutex;
PortState m_port_state = PortState::Uninitialized;
std::vector<int> m_connected_ids;
int m_active_id = -1;
bool m_connected = false;
@@ -925,21 +939,71 @@ int GetActiveControllerId() {
return g_controller != nullptr ? g_controller->GetActiveControllerId() : -1;
}
int KYTY_SYSV_ABI PadInit() {
PRINT_NAME();
static bool PadOpenArgsAreValid(int user_id, int type, int index) {
// The host keyboard and gamepad feed one standard port for the configured user.
return user_id == Config::GetUserId() && type == 0 && index == 0;
}
int GameController::InitPort() {
Common::LockGuard lock(m_mutex);
if (m_port_state == PortState::Uninitialized) {
m_port_state = PortState::Closed;
}
return OK;
}
static bool PadOpenArgsAreValid(int user_id, int type, int index) {
constexpr int user_id_system = 0xff;
constexpr int port_type_standard = 0;
constexpr int port_type_special = 2;
constexpr int port_type_remote = 16;
const bool personal_port =
user_id == Config::GetUserId() && (type == port_type_standard || type == port_type_special);
const bool system_remote_control = user_id == user_id_system && type == port_type_remote;
return index == 0 && (personal_port || system_remote_control);
int GameController::OpenPort(int user_id, int type, int index) {
Common::LockGuard lock(m_mutex);
if (m_port_state == PortState::Uninitialized) {
return PAD_ERROR_NOT_INITIALIZED;
}
if (!PadOpenArgsAreValid(user_id, type, index)) {
return PAD_ERROR_INVALID_ARG;
}
if (m_port_state == PortState::Open) {
return PAD_ERROR_ALREADY_OPENED;
}
m_port_state = PortState::Open;
return PAD_HANDLE;
}
int GameController::GetPortHandle(int user_id, int type, int index) {
Common::LockGuard lock(m_mutex);
if (m_port_state == PortState::Uninitialized) {
return PAD_ERROR_NOT_INITIALIZED;
}
return m_port_state == PortState::Open && PadOpenArgsAreValid(user_id, type, index) ? PAD_HANDLE
: PAD_ERROR_NO_HANDLE;
}
int GameController::CheckPortHandleLocked(int handle) const {
if (m_port_state == PortState::Uninitialized) {
return PAD_ERROR_NOT_INITIALIZED;
}
return m_port_state == PortState::Open && handle == PAD_HANDLE ? OK : PAD_ERROR_INVALID_HANDLE;
}
int GameController::CheckPortHandle(int handle) {
Common::LockGuard lock(m_mutex);
return CheckPortHandleLocked(handle);
}
int GameController::ClosePort(int handle) {
Common::LockGuard lock(m_mutex);
if (const auto error = CheckPortHandleLocked(handle); error != OK) {
return error;
}
m_port_state = PortState::Closed;
return OK;
}
int PadCheckHandle(int handle) {
return g_controller->CheckPortHandle(handle);
}
int KYTY_SYSV_ABI PadInit() {
PRINT_NAME();
return g_controller->InitPort();
}
int KYTY_SYSV_ABI PadOpen(int user_id, int type, int index, const void* param) {
@@ -951,15 +1015,7 @@ int KYTY_SYSV_ABI PadOpen(int user_id, int type, int index, const void* param) {
"\t param = 0x%016" PRIx64 "\n",
user_id, type, index, reinterpret_cast<uint64_t>(param));
constexpr int pad_error_invalid_arg = -2137915391; /* 0x80920001 */
if (!PadOpenArgsAreValid(user_id, type, index)) {
return pad_error_invalid_arg;
}
int handle = 1;
return handle;
return g_controller->OpenPort(user_id, type, index);
}
int KYTY_SYSV_ABI PadGetHandle(int user_id, int type, int index) {
@@ -970,20 +1026,20 @@ int KYTY_SYSV_ABI PadGetHandle(int user_id, int type, int index) {
"\t index = %d\n",
user_id, type, index);
constexpr int pad_error_device_no_handle = -2137915384; /* 0x80920008 */
return g_controller->GetPortHandle(user_id, type, index);
}
if (!PadOpenArgsAreValid(user_id, type, index)) {
return pad_error_device_no_handle;
}
return 1;
int KYTY_SYSV_ABI PadClose(int handle) {
PRINT_NAME();
LOGF("\t handle = %d\n", handle);
return g_controller->ClosePort(handle);
}
int KYTY_SYSV_ABI PadSetMotionSensorState(int handle, bool enable) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
LOGF("\t enable = %s\n", (enable ? "true" : "false"));
@@ -995,8 +1051,8 @@ int KYTY_SYSV_ABI PadSetMotionSensorState(int handle, bool enable) {
int KYTY_SYSV_ABI PadSetAngularVelocityDeadbandState(int handle, bool enable) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
LOGF("\t enable = %s\n", (enable ? "true" : "false"));
@@ -1007,8 +1063,8 @@ int KYTY_SYSV_ABI PadSetAngularVelocityDeadbandState(int handle, bool enable) {
int KYTY_SYSV_ABI PadResetOrientation(int handle) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
g_controller->ResetOrientation();
@@ -1018,18 +1074,17 @@ int KYTY_SYSV_ABI PadResetOrientation(int handle) {
int KYTY_SYSV_ABI PadGetControllerInformation(int handle, PadControllerInformation* info) {
PRINT_NAME();
int connected_count = 0;
bool connected = false;
g_controller->GetConnectionInfo(&connected, &connected_count);
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (info == nullptr) {
return PAD_ERROR_INVALID_ARG;
}
int connected_count = 0;
bool connected = false;
g_controller->GetConnectionInfo(&connected, &connected_count);
std::memset(info, 0, sizeof(*info));
info->touch_pixel_density = 44.86f;
@@ -1048,8 +1103,8 @@ int KYTY_SYSV_ABI PadGetControllerInformation(int handle, PadControllerInformati
int KYTY_SYSV_ABI PadIsRemoteController(int handle, bool* is_remote) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (is_remote == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1062,8 +1117,8 @@ int KYTY_SYSV_ABI PadIsRemoteController(int handle, bool* is_remote) {
int KYTY_SYSV_ABI PadReadState(int handle, PadData* data) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (data == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1084,8 +1139,8 @@ int KYTY_SYSV_ABI PadRead(int handle, PadData* data, int num) {
PRINT_NAME();
EXIT_NOT_IMPLEMENTED(num < 1 || num > 64);
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (data == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1116,8 +1171,8 @@ int KYTY_SYSV_ABI PadRead(int handle, PadData* data, int num) {
int KYTY_SYSV_ABI PadSetVibration(int handle, const PadVibrationParam* param) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (param == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1135,8 +1190,8 @@ int KYTY_SYSV_ABI PadSetVibration(int handle, const PadVibrationParam* param) {
int KYTY_SYSV_ABI PadResetLightBar(int handle) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
return OK;
@@ -1145,8 +1200,8 @@ int KYTY_SYSV_ABI PadResetLightBar(int handle) {
int KYTY_SYSV_ABI PadSetLightBar(int handle, const PadLightBarParam* param) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (param == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1160,8 +1215,8 @@ int KYTY_SYSV_ABI PadSetLightBar(int handle, const PadLightBarParam* param) {
int KYTY_SYSV_ABI PadSetTriggerEffect(int handle, const PadTriggerEffectParam* param) {
PRINT_NAME();
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (param == nullptr) {
return PAD_ERROR_INVALID_ARG;
@@ -1175,8 +1230,8 @@ int KYTY_SYSV_ABI PadGetTriggerEffectState(int handle, PadTriggerEffectStateInfo
LOGF("\t handle = %d\n", handle);
if (handle != 1) {
return PAD_ERROR_INVALID_HANDLE;
if (const auto error = PadCheckHandle(handle); error != OK) {
return error;
}
if (info == nullptr) {
return PAD_ERROR_INVALID_ARG;
+3
View File
@@ -81,9 +81,12 @@ enum class Setting { SpeakerVolume, VibrationIntensity, TriggerEffectIntensity }
void CycleSetting(Setting setting);
float GetSettingScale(Setting setting);
int PadCheckHandle(int handle);
int KYTY_SYSV_ABI PadInit();
int KYTY_SYSV_ABI PadOpen(int user_id, int type, int index, const void* param);
int KYTY_SYSV_ABI PadGetHandle(int user_id, int type, int index);
int KYTY_SYSV_ABI PadClose(int handle);
int KYTY_SYSV_ABI PadSetMotionSensorState(int handle, bool enable);
int KYTY_SYSV_ABI PadSetAngularVelocityDeadbandState(int handle, bool enable);
int KYTY_SYSV_ABI PadResetOrientation(int handle);
+14 -13
View File
@@ -19,6 +19,7 @@
#include <cstdint>
#include <cstring>
#include <deque>
#include <filesystem>
#include <limits>
#include <mutex>
#include <string>
@@ -48,7 +49,7 @@ struct ResolvedPathInfo {
uint32_t file_id = 0xffffffffu;
uint64_t file_size = 0;
bool is_dir = false;
std::string host_path;
std::filesystem::path host_path;
};
static std::mutex g_mutex;
@@ -56,7 +57,7 @@ static uint32_t g_next_submission_id =
static uint32_t g_next_file_id = 1;
static std::unordered_map<uint32_t, bool> g_submissions;
static std::condition_variable g_submission_cv;
static std::unordered_map<uint32_t, std::string> g_files;
static std::unordered_map<uint32_t, std::filesystem::path> g_files;
static std::unordered_map<uint32_t, uint64_t> g_file_sizes;
static std::unordered_map<std::string, ResolvedPathInfo> g_resolved_paths;
@@ -113,7 +114,7 @@ static int ReadGuestCString(uint64_t addr, char* out, size_t out_size) {
return LibKernel::KERNEL_ERROR_ENAMETOOLONG;
}
static bool TryGetHostPath(uint32_t file_id, std::string* out) {
static bool TryGetHostPath(uint32_t file_id, std::filesystem::path* out) {
std::scoped_lock lock(g_mutex);
const auto it = g_files.find(file_id);
if (it == g_files.end()) {
@@ -133,7 +134,7 @@ static bool TryGetHostFileSize(uint32_t file_id, uint64_t* out) {
return true;
}
static int GetHostPathStat(const std::string& host_path, LibKernel::FileSystem::FileStat* st) {
static int GetHostPathStat(const std::filesystem::path& host_path, LibKernel::FileSystem::FileStat* st) {
if (st == nullptr) {
return LibKernel::KERNEL_ERROR_EINVAL;
}
@@ -194,7 +195,7 @@ static int ResolveOnePath(const char* guest_path, uint32_t* id, uint64_t* size)
if (!found) {
const auto real_path = LibKernel::FileSystem::GetRealFilename(path);
info.host_path = Common::PathToString(real_path);
info.host_path = real_path;
if (Common::File::IsDirectoryExisting(real_path)) {
info.is_dir = true;
@@ -225,7 +226,7 @@ static int ResolveOnePath(const char* guest_path, uint32_t* id, uint64_t* size)
}
}
if (log_missing) {
LOGF("\tAPR resolve missing path: %s -> %s\n", guest_path, info.host_path.c_str());
LOGF("\tAPR resolve missing path: %s -> %s\n", guest_path, Common::PathToString(info.host_path).c_str());
}
}
@@ -408,7 +409,7 @@ static int KYTY_SYSV_ABI GetFileStat(uint32_t file_id, LibKernel::FileSystem::Fi
return KernelSyscallResult(LibKernel::KERNEL_ERROR_EINVAL);
}
std::string host_path;
std::filesystem::path host_path;
if (!AprShared::TryGetHostPath(file_id, &host_path)) {
LOGF("\tAPR stat failed for unknown file id: 0x%08" PRIx32 "\n", file_id);
return KernelSyscallResult(LibKernel::KERNEL_ERROR_ENOENT);
@@ -430,7 +431,7 @@ static int KYTY_SYSV_ABI GetFileSize(uint32_t file_id, uint64_t* size) {
return KernelSyscallResult(LibKernel::KERNEL_ERROR_EFAULT);
}
std::string host_path;
std::filesystem::path host_path;
if (!AprShared::TryGetHostPath(file_id, &host_path)) {
LOGF("\tAPR size failed for unknown file id: 0x%08" PRIx32 "\n", file_id);
return KernelSyscallResult(LibKernel::KERNEL_ERROR_ENOENT);
@@ -1161,13 +1162,13 @@ static bool AppendCommandRecord(uint64_t command_buffer, CommandBufferState::Com
return CommitCommandBufferRecord(command_buffer, &state, record_size);
}
static int ReadHostFileToGuest(const std::string& host_path, uint64_t file_offset,
static int ReadHostFileToGuest(const std::filesystem::path& host_path, uint64_t file_offset,
uint64_t destination, uint64_t size, uint64_t* bytes_read);
static int ExecuteCommand(const CommandBufferState::Command& entry) {
if (const auto* payload = std::get_if<CommandBufferState::ReadFileCommand>(&entry.data)) {
const auto& command = *payload;
std::string host_path;
std::filesystem::path host_path;
if (!AprShared::TryGetHostPath(command.file_id, &host_path)) {
LOGF("\tAPR submit failed for unknown file id: 0x%08" PRIx32 "\n", command.file_id);
return LibKernel::KERNEL_ERROR_ENOENT;
@@ -1178,7 +1179,7 @@ static int ExecuteCommand(const CommandBufferState::Command& entry) {
command.size, &bytes_read);
if (result != OK) {
LOGF("\tAPR submit read failed: id=0x%08" PRIx32 ", result=0x%08" PRIx32 ", path=%s\n",
command.file_id, static_cast<uint32_t>(result), host_path.c_str());
command.file_id, static_cast<uint32_t>(result), Common::PathToString(host_path).c_str());
return result;
}
} else if (const auto* payload =
@@ -1380,7 +1381,7 @@ static bool AdvanceCommandBuffer(uint64_t command_buffer, uint64_t record_size)
return CommitCommandBufferRecord(command_buffer, &state, record_size);
}
static int ReadHostFileToGuest(const std::string& host_path, uint64_t file_offset,
static int ReadHostFileToGuest(const std::filesystem::path& host_path, uint64_t file_offset,
uint64_t destination, uint64_t size, uint64_t* bytes_read) {
if (bytes_read == nullptr) {
return LibKernel::KERNEL_ERROR_EINVAL;
@@ -1396,7 +1397,7 @@ static int ReadHostFileToGuest(const std::string& host_path, uint64_t file_offse
Common::File file;
if (!file.Open(host_path, Common::File::Mode::Read)) {
LOGF("\tAPR read missing host file: %s\n", host_path.c_str());
LOGF("\tAPR read missing host file: %s\n", Common::PathToString(host_path).c_str());
return LibKernel::KERNEL_ERROR_ENOENT;
}
const auto file_size = file.Size();
+17 -15
View File
@@ -78,6 +78,8 @@ constexpr int SCE_FONT_TEXT_PARSER_RESULT_TERMINATE = 0;
constexpr int SCE_FONT_TEXT_PARSER_RESULT_ERROR = -1;
constexpr int SCE_FONT_WRITING_FORM_HORIZONTAL = 0x10;
constexpr int FONT_BITMAP_MAX_DIM = 128;
// TrueType glyphs are rasterized at the requested size; the limit only rejects absurd scales.
constexpr int FONT_GLYPH_MAX_DIM = 4096;
constexpr int FONT_ERROR_INVALID_PARAMETER = static_cast<int>(0x80460002u);
constexpr int FONT_ERROR_INVALID_FONT_HANDLE = static_cast<int>(0x80460005u);
constexpr int FONT_ERROR_NO_SUPPORT_CODE = static_cast<int>(0x80460041u);
@@ -206,7 +208,7 @@ struct FontState {
const uint8_t* ttf_data;
uint32_t ttf_size;
bool has_ttf;
std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX_DIM> fallback_image;
std::vector<uint8_t> fallback_image;
FontTransImage trans_image;
};
@@ -227,7 +229,7 @@ struct GlyphState {
uint32_t code;
int attribute;
FontGlyphMetrics metrics;
std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX_DIM> image;
std::vector<uint8_t> image;
FontTransImage trans_image;
};
@@ -851,8 +853,8 @@ static bool get_stb_metrics(FontState* font, uint32_t code, FontGlyphMetrics* me
return true;
}
static bool init_stb_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX_DIM>* image,
FontTransImage* trans_image, uint32_t code, FontState* font) {
static bool init_stb_image(std::vector<uint8_t>* image, FontTransImage* trans_image, uint32_t code,
FontState* font) {
if (image == nullptr || trans_image == nullptr || !ensure_stb_font(font)) {
return false;
}
@@ -868,12 +870,12 @@ static bool init_stb_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP
stbtt_GetCodepointBitmapBox(&font->font_info, static_cast<int>(code), scale_x, scale_y, &x0, &y0,
&x1, &y1);
image->fill(0);
const auto width = static_cast<uint32_t>(std::clamp(x1 - x0, 0, FONT_BITMAP_MAX_DIM));
const auto height = static_cast<uint32_t>(std::clamp(y1 - y0, 0, FONT_BITMAP_MAX_DIM));
const auto width = static_cast<uint32_t>(std::clamp(x1 - x0, 0, FONT_GLYPH_MAX_DIM));
const auto height = static_cast<uint32_t>(std::clamp(y1 - y0, 0, FONT_GLYPH_MAX_DIM));
// Keep at least one byte so that blank glyphs still get a valid address.
image->assign(std::max<size_t>(static_cast<size_t>(width) * height, 1), 0);
trans_image->address = image->data();
trans_image->width_byte = FONT_BITMAP_MAX_DIM;
trans_image->width_byte = width;
trans_image->image_width = width;
trans_image->image_height = height;
@@ -882,7 +884,7 @@ static bool init_stb_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP
}
stbtt_MakeCodepointBitmap(&font->font_info, image->data(), static_cast<int>(width),
static_cast<int>(height), FONT_BITMAP_MAX_DIM, scale_x, scale_y,
static_cast<int>(height), static_cast<int>(width), scale_x, scale_y,
static_cast<int>(code));
return true;
@@ -939,8 +941,8 @@ static uint32_t normalize_bitmap_font_code(uint32_t code) {
return static_cast<uint32_t>('?');
}
static void init_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX_DIM>* image,
FontTransImage* trans_image, uint32_t code, const FontState* font) {
static void init_image(std::vector<uint8_t>* image, FontTransImage* trans_image, uint32_t code,
const FontState* font) {
EXIT_NOT_IMPLEMENTED(image == nullptr);
EXIT_NOT_IMPLEMENTED(trans_image == nullptr);
@@ -948,11 +950,11 @@ static void init_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX
return;
}
image->fill(0);
code = normalize_bitmap_font_code(code);
const auto width = static_cast<uint32_t>(scaled_font_width(font));
const auto height = static_cast<uint32_t>(scaled_font_height(font));
image->assign(static_cast<size_t>(width) * height, 0);
#if LIBAVUTIL_VERSION_MAJOR < 61
const auto* vga_font = avpriv_vga16_font;
#else
@@ -965,13 +967,13 @@ static void init_image(std::array<uint8_t, FONT_BITMAP_MAX_DIM * FONT_BITMAP_MAX
for (uint32_t x = 0; x < width; x++) {
const uint32_t src_x = std::min<uint32_t>((x * 8u) / width, 7u);
if ((row & (0x80u >> src_x)) != 0) {
(*image)[y * FONT_BITMAP_MAX_DIM + x] = 0xff;
(*image)[y * width + x] = 0xff;
}
}
}
trans_image->address = image->data();
trans_image->width_byte = FONT_BITMAP_MAX_DIM;
trans_image->width_byte = width;
trans_image->image_width = width;
trans_image->image_height = height;
}
+73 -3
View File
@@ -1497,13 +1497,43 @@ namespace LibNpCommerce {
LIB_VERSION("NpCommerce", 1, "NpCommerce", 1, 1);
constexpr int COMMERCE_STATUS_NONE = 0;
constexpr int COMMERCE_STATUS_INITIALIZED = 1;
constexpr int COMMERCE_STATUS_NONE = 0;
constexpr int COMMERCE_STATUS_INITIALIZED = 1;
constexpr int COMMERCE_STATUS_FINISHED = 3;
constexpr int COMMERCE_RESULT_USER_CANCELED = 1;
constexpr int COMMERCE_ERROR_NOT_INITIALIZED = static_cast<int>(0x80B80003u);
constexpr int COMMERCE_ERROR_ALREADY_INITIALIZED = static_cast<int>(0x80B80004u);
constexpr int COMMERCE_ERROR_NOT_FINISHED = static_cast<int>(0x80B80005u);
constexpr int COMMERCE_ERROR_PARAM_INVALID = static_cast<int>(0x80B8000Au);
constexpr int COMMERCE_ERROR_ARG_NULL = static_cast<int>(0x80B8000Du);
static int g_commerce_status = COMMERCE_STATUS_NONE;
struct CommerceDialogParam {
uint8_t base_param[48];
int32_t size;
int32_t user_id;
int32_t mode;
uint32_t service_label;
const char* const* targets;
uint32_t num_targets;
uint32_t padding;
uint64_t features;
void* user_data;
uint8_t reserved[32];
};
struct CommerceDialogResult {
int32_t result;
bool authorized;
void* user_data;
uint8_t reserved[32];
};
static_assert(sizeof(CommerceDialogParam) == 128 && offsetof(CommerceDialogParam, user_data) == 88);
static_assert(sizeof(CommerceDialogResult) == 48 && offsetof(CommerceDialogResult, user_data) == 8);
static int g_commerce_status = COMMERCE_STATUS_NONE;
static CommerceDialogResult g_commerce_result {};
static int KYTY_SYSV_ABI NpCommerceDialogInitialize() {
PRINT_NAME();
@@ -1520,9 +1550,47 @@ static int KYTY_SYSV_ABI NpCommerceDialogTerminate() {
return COMMERCE_ERROR_NOT_INITIALIZED;
}
g_commerce_status = COMMERCE_STATUS_NONE;
g_commerce_result = {};
return OK;
}
static int KYTY_SYSV_ABI NpCommerceDialogOpen(const CommerceDialogParam* param) {
PRINT_NAME();
if (g_commerce_status == COMMERCE_STATUS_NONE) {
return COMMERCE_ERROR_NOT_INITIALIZED;
}
if (param == nullptr) {
return COMMERCE_ERROR_ARG_NULL;
}
g_commerce_result = {};
if (param->size == sizeof(CommerceDialogParam)) {
g_commerce_result.user_data = param->user_data;
}
g_commerce_status = COMMERCE_STATUS_FINISHED;
if (param->size != sizeof(CommerceDialogParam) || param->mode < 0 || param->mode > 5 ||
std::ranges::any_of(param->reserved, [](uint8_t value) { return value != 0; })) {
g_commerce_result.result = COMMERCE_ERROR_PARAM_INVALID;
return COMMERCE_ERROR_PARAM_INVALID;
}
g_commerce_result.result = COMMERCE_RESULT_USER_CANCELED;
return OK;
}
static int KYTY_SYSV_ABI NpCommerceDialogGetResult(CommerceDialogResult* result) {
PRINT_NAME();
if (g_commerce_status == COMMERCE_STATUS_NONE) {
return COMMERCE_ERROR_NOT_INITIALIZED;
}
if (result == nullptr) {
return COMMERCE_ERROR_ARG_NULL;
}
if (g_commerce_status != COMMERCE_STATUS_FINISHED) {
return COMMERCE_ERROR_NOT_FINISHED;
}
*result = g_commerce_result;
return result->result;
}
static int KYTY_SYSV_ABI NpCommerceDialogUpdateStatus() {
PRINT_NAME();
@@ -1532,6 +1600,8 @@ static int KYTY_SYSV_ABI NpCommerceDialogUpdateStatus() {
LIB_DEFINE(InitNet_1_NpCommerce) {
LIB_FUNC("0aR2aWmQal4", NpCommerceDialogInitialize);
LIB_FUNC("m-I92Ab50W8", NpCommerceDialogTerminate);
LIB_FUNC("DfSCDRA3EjY", NpCommerceDialogOpen);
LIB_FUNC("r42bWcQbtZY", NpCommerceDialogGetResult);
LIB_FUNC("LR5cwFMMCVE", NpCommerceDialogUpdateStatus);
}
+10 -20
View File
@@ -19,7 +19,7 @@ static int KYTY_SYSV_ABI PadSetVibrationMode(int handle, int mode) {
"\t mode = %d\n",
handle, mode);
return 0;
return Controller::PadCheckHandle(handle);
}
static int KYTY_SYSV_ABI PadSetVibrationTriggerEffectWeakWhileEmbeddedMicInUse(bool enabled) {
@@ -66,11 +66,10 @@ static int KYTY_SYSV_ABI
PadDeviceClassGetExtendedInformation(int handle, PadDeviceClassExtendedInformation* info) {
PRINT_NAME();
constexpr int pad_error_invalid_handle = -2137915389; /* 0x80920003 */
constexpr int pad_error_invalid_arg = -2137915391; /* 0x80920001 */
constexpr int pad_error_invalid_arg = -2137915391; /* 0x80920001 */
if (handle != 1) {
return pad_error_invalid_handle;
if (const auto error = Controller::PadCheckHandle(handle); error != OK) {
return error;
}
if (info == nullptr) {
return pad_error_invalid_arg;
@@ -85,16 +84,15 @@ static int KYTY_SYSV_ABI PadDeviceClassParseData(int handle, const Controller::P
PadDeviceClassData* class_data) {
PRINT_NAME();
constexpr int pad_error_invalid_handle = -2137915389; /* 0x80920003 */
constexpr int pad_error_invalid_arg = -2137915391; /* 0x80920001 */
constexpr int pad_error_invalid_arg = -2137915391; /* 0x80920001 */
LOGF("\t handle = %d\n"
"\t data = 0x%016" PRIx64 "\n"
"\t class_data = 0x%016" PRIx64 "\n",
handle, reinterpret_cast<uint64_t>(data), reinterpret_cast<uint64_t>(class_data));
if (handle != 1) {
return pad_error_invalid_handle;
if (const auto error = Controller::PadCheckHandle(handle); error != OK) {
return error;
}
if (data == nullptr || class_data == nullptr) {
return pad_error_invalid_arg;
@@ -122,15 +120,7 @@ static int KYTY_SYSV_ABI PadSetTiltCorrectionState(int handle, bool enabled) {
"\t enabled = %d\n",
handle, enabled ? 1 : 0);
return 0;
}
static int KYTY_SYSV_ABI PadClose(int handle) {
PRINT_NAME();
LOGF("\t handle = %d\n", handle);
return OK;
return Controller::PadCheckHandle(handle);
}
// PPSA02385
@@ -154,7 +144,7 @@ static int KYTY_SYSV_ABI PadGetExtControllerInformationStub(int handle, void* in
"\t info = 0x%016" PRIx64 "\n",
handle, reinterpret_cast<uint64_t>(info));
return OK;
return Controller::PadCheckHandle(handle);
}
static int KYTY_SYSV_ABI PadUnknownN3kSX62fgNo(uint64_t arg0, uint64_t arg1, uint64_t arg2,
@@ -197,7 +187,7 @@ LIB_DEFINE(InitPad_1) {
LIB_FUNC("znaWI0gpuo8", Controller::PadGetTriggerEffectState);
LIB_FUNC("DscD1i9HX1w", Controller::PadResetLightBar);
LIB_FUNC("RR4novUEENY", Controller::PadSetLightBar);
LIB_FUNC("6ncge5+l5Qs", PadClose);
LIB_FUNC("6ncge5+l5Qs", Controller::PadClose);
LIB_FUNC("n3kSX62fgNo", PadUnknownN3kSX62fgNo);
}
+51
View File
@@ -110,6 +110,8 @@ struct Controller {
now = 1000;
haptics_handles_rumble = false;
Initialize();
Check(PadInit() == 0 && PadOpen(Config::GetUserId(), 0, 0, nullptr) == 1,
"the standard controller port did not open");
Connect(1);
Check(GetSettingScale(Setting::SpeakerVolume) ==
Config::GetControllerSpeakerVolume() / 50.0f &&
@@ -143,6 +145,54 @@ void SetRumble(uint8_t large, uint8_t small) {
Check(PadSetVibration(1, &param) == 0, "vibration request failed");
}
void TestPadPortOwnership() {
Initialize();
const int user = Config::GetUserId();
constexpr auto not_initialized = static_cast<int32_t>(0x80920005u);
constexpr auto already_opened = static_cast<int32_t>(0x80920004u);
constexpr auto invalid_handle = static_cast<int32_t>(0x80920003u);
constexpr auto no_handle = static_cast<int32_t>(0x80920008u);
PadData data {};
Check(PadOpen(user, 0, 0, nullptr) == not_initialized &&
PadGetHandle(user, 0, 0) == not_initialized &&
PadReadState(1, &data) == not_initialized,
"an uninitialized library exposed a controller port");
Check(PadInit() == 0 && PadGetHandle(user, 0, 0) == no_handle,
"initializing the library opened a controller port");
const int handle = PadOpen(user, 0, 0, nullptr);
Check(handle == 1 && PadInit() == 0 && PadGetHandle(user, 0, 0) == handle,
"reinitializing the library lost the open port");
for (int duplicate = 0; duplicate < 4; duplicate++) {
const int rejected = PadOpen(user, 0, 0, nullptr);
Check(rejected == already_opened,
"opening one pad five times exposed duplicate connected controllers");
Check(PadClose(rejected) == invalid_handle && PadGetHandle(user, 0, 0) == handle,
"closing a rejected duplicate released the original owner");
}
Check(PadGetHandle(user + 1, 0, 0) == no_handle &&
PadGetHandle(user, 2, 0) == no_handle && PadGetHandle(user, 0, 1) == no_handle,
"a different port tuple aliased the open standard pad");
SetButton(HOST_INPUT_CONTROLLER_ID, PAD_BUTTON_CROSS, true);
Check(PadReadState(handle, &data) == 0 && data.connected &&
data.buttons == PAD_BUTTON_CROSS,
"keyboard Cross did not reach the owned port");
SetButton(HOST_INPUT_CONTROLLER_ID, PAD_BUTTON_CROSS, false);
Connect(1);
SetButton(1, PAD_BUTTON_CROSS, true);
Check(PadReadState(handle, &data) == 0 && data.buttons == PAD_BUTTON_CROSS,
"the physical controller did not share the owned port");
Disconnect(1);
Check(PadGetHandle(user, 0, 0) == handle && PadOpen(user, 0, 0, nullptr) == already_opened,
"host disconnection released guest port ownership");
Check(PadClose(handle) == 0 && PadGetHandle(user, 0, 0) == no_handle &&
PadReadState(handle, &data) == invalid_handle &&
PadRead(handle, &data, 1) == invalid_handle && PadClose(handle) == invalid_handle,
"a closed port retained a usable handle");
Check(PadOpen(user, 0, 0, nullptr) == handle && PadReadState(handle, &data) == 0,
"a closed port could not reopen");
Shutdown();
}
void TestSettingCycles() {
Controller controller;
for (auto setting:
@@ -503,6 +553,7 @@ void TestTriggerEffectState() {
int main() {
Config::Initialize();
TestPadPortOwnership();
TestSettingCycles();
TestGlobalControllerLevels();
// Each following test initializes another controller and requires strong defaults.
+90
View File
@@ -62,6 +62,10 @@ namespace Libs::LibNpWebApi2 {
void InitNet_1_NpWebApi2(Loader::SymbolDatabase *symbols);
}
namespace Libs::LibNpCommerce {
void InitNet_1_NpCommerce(Loader::SymbolDatabase *symbols);
}
namespace {
namespace FileSystem = Libs::LibKernel::FileSystem;
@@ -385,6 +389,91 @@ void TestAioBatches() {
Check(FileSystem::KernelClose(fd) == OK, "close AIO read fixture");
}
void TestNpCommerceDialog() {
Loader::SymbolDatabase symbols;
Libs::LibNpCommerce::InitNet_1_NpCommerce(&symbols);
const auto find = [&](const char *nid) {
const auto *symbol = symbols.FindByNid(nid, Loader::SymbolType::Func);
Check(symbol != nullptr, "NpCommerce dialog exports resolve");
return symbol->vaddr;
};
using Call = int (KYTY_SYSV_ABI *)();
using Open = int (KYTY_SYSV_ABI *)(const void *);
using GetResult = int (KYTY_SYSV_ABI *)(void *);
const auto initialize = reinterpret_cast<Call>(find("0aR2aWmQal4"));
const auto terminate = reinterpret_cast<Call>(find("m-I92Ab50W8"));
const auto update = reinterpret_cast<Call>(find("LR5cwFMMCVE"));
const auto open = reinterpret_cast<Open>(find("DfSCDRA3EjY"));
const auto get_result = reinterpret_cast<GetResult>(find("r42bWcQbtZY"));
struct BaseParam {
uint64_t size;
uint8_t reserved[36];
uint32_t magic;
};
struct Param {
BaseParam base;
int32_t size, user_id, mode;
uint32_t service_label;
const char *const *targets;
uint32_t num_targets, padding;
uint64_t features;
void *user_data;
uint8_t reserved[32];
} param {};
struct Result {
int32_t result;
bool authorized;
void *user_data;
uint8_t reserved[32];
} result {};
static_assert(sizeof(Param) == 128 && offsetof(Param, user_data) == 88);
static_assert(sizeof(Result) == 48 && offsetof(Result, user_data) == 8);
param.base.size = sizeof(BaseParam);
param.base.magic = static_cast<uint32_t>(0xC0D1A109u + reinterpret_cast<uintptr_t>(&param.base));
param.size = sizeof(param);
param.user_id = 1;
param.user_data = &symbols;
Check(open(&param) == static_cast<int>(0x80B80003u) &&
get_result(&result) == static_cast<int>(0x80B80003u),
"NpCommerce dialog calls require initialization");
Check(initialize() == OK && update() == 1 &&
get_result(&result) == static_cast<int>(0x80B80005u),
"NpCommerce dialog result requires completion");
Check(open(nullptr) == static_cast<int>(0x80B8000Du) && update() == 1 &&
get_result(nullptr) == static_cast<int>(0x80B8000Du),
"NpCommerce dialog rejects null arguments");
Check(open(&param) == OK && update() == 3, "NpCommerce dialog finishes without a store");
param.user_data = nullptr;
result.authorized = true;
Check(get_result(&result) == 1 && result.result == 1 && !result.authorized &&
result.user_data == &symbols,
"NpCommerce cancellation returns the original user data");
param.mode = 5;
Check(open(&param) == OK && get_result(&result) == 1 && !result.authorized &&
result.user_data == nullptr,
"NpCommerce dialog can reopen with fresh result data");
const auto invalid_param = [&] {
Check(open(&param) == static_cast<int>(0x80B8000Au) && update() == 3 &&
get_result(&result) == static_cast<int>(0x80B8000Au) &&
result.result == static_cast<int>(0x80B8000Au) && !result.authorized,
"NpCommerce dialog reports invalid parameters in the completion result");
};
param.mode = 99;
invalid_param();
param.mode = -1;
invalid_param();
param.mode = 0;
param.size = 64;
invalid_param();
param.size = sizeof(param);
param.reserved[0] = 1;
invalid_param();
param.reserved[0] = 0;
Check(open(&param) == OK && get_result(&result) == 1 && terminate() == OK && update() == 0 &&
get_result(&result) == static_cast<int>(0x80B80003u),
"NpCommerce dialog recovers from invalid parameters and terminates");
}
void TestNpWebApi2Memory() {
Loader::SymbolDatabase symbols;
Libs::LibNpWebApi2::InitNet_1_NpWebApi2(&symbols);
@@ -1644,6 +1733,7 @@ int main(int, char**) {
CheckSocketWakeup();
CheckEtherAddressFormatting();
TestNpWebApi2Memory();
TestNpCommerceDialog();
graphics.reset();
subsystems.Destroy();
+120 -19
View File
@@ -27,12 +27,12 @@ AddValueBlock(Libs::Graphics::ShaderRecompiler::IR::Program &program) {
auto block = std::make_unique<Block>();
auto *result = block.get();
program.blocks.push_back(result);
program.block_info.push_back({.id = 0});
result->id = static_cast<uint32_t>(program.blocks.size() - 1u);
program.block_storage.push_back(std::move(block));
return *result;
}
Libs::Graphics::ShaderRecompiler::IR::ResourcePlan SrtPlan(uint64_t address) {
Libs::Graphics::ShaderRecompiler::IR::Program SrtProgram(uint64_t address) {
using namespace Libs::Graphics::ShaderRecompiler::IR;
Program program;
program.stage = Libs::Graphics::ShaderType::Compute;
@@ -62,7 +62,7 @@ Libs::Graphics::ShaderRecompiler::IR::ResourcePlan SrtPlan(uint64_t address) {
source.dwords[1] = Value(0u);
source.dword_count = 2;
program.descriptor_sources.push_back(source);
return ExtractResourcePlan(program);
return program;
}
Libs::Graphics::ShaderRecompiler::IR::ResourcePlan UnbasedFlatPlan() {
@@ -152,7 +152,7 @@ Libs::Graphics::ShaderRecompiler::IR::Program MixedSamplerProgram() {
void TestMappedSrtUsesDirectReaderByDefault() {
using namespace Libs::Graphics::ShaderRecompiler::IR;
const uint32_t dword = 0x12345678;
auto plan = SrtPlan(reinterpret_cast<uint64_t>(&dword));
auto plan = ExtractResourcePlan(SrtProgram(reinterpret_cast<uint64_t>(&dword)));
uint32_t specialization_reads = 0;
const SrtRuntime runtime{.userdata = &specialization_reads,
.read_specialization_memory =
@@ -170,7 +170,7 @@ void TestMappedSrtUsesDirectReaderByDefault() {
void TestIntegerRuntimeValueFollowsSrtReads() {
using namespace Libs::Graphics::ShaderRecompiler::IR;
auto plan = SrtPlan(0x10000);
auto plan = SrtProgram(0x10000);
const auto root = plan.descriptor_sources.front().dwords[0];
Check(ValidateRuntimeValue(plan, root, RuntimeValueType::Integer),
"integer SRT read was rejected");
@@ -217,6 +217,98 @@ void TestIntegerRuntimeValueFollowsSrtReads() {
"cyclic SRT read-first-lane dependency was accepted");
}
void TestSrtAliasesRetainReadPolicy() {
using namespace Libs::Graphics::ShaderRecompiler::IR;
auto program = SrtProgram(0x1000u);
auto &block = *program.blocks[0];
const auto append_read = [&](uint32_t address) {
auto &handle = block.AppendNewInst(ValueOpcode::GetAddressResource,
{Value(address), Value(0u)});
auto &read = block.AppendNewInst(ValueOpcode::LoadAddressU32,
{Value(&handle), Value(0u), Value(0u), Value(true)});
// Different reads deliberately share MemoryInfo but have different guest PCs.
read.SetFlags(MemoryFlags{.index = 0, .pc = address});
const auto slot = static_cast<uint32_t>(program.srt_reads.size());
program.srt_reads.push_back({Value(&read), slot});
auto &srt = block.AppendNewInst(ValueOpcode::GetSrtResource);
return Value(&block.AppendNewInst(ValueOpcode::ReadConst,
{Value(&srt), Value(slot)}));
};
const auto ordinary = append_read(0x2000u);
const auto pointer = append_read(0x3000u);
program.srt_reads[0].value.Instruction()->Arg(0).Instruction()->SetArg(0, pointer);
auto &mask = block.AppendNewInst(ValueOpcode::INotEqual32, {ordinary, Value(0u)});
auto &first = block.AppendNewInst(ValueOpcode::ReadFirstLane,
{Value(9u), Value(&mask)});
program.descriptor_sources.push_back({.dwords = {Value(&first), Value(0u)},
.dword_count = 2});
DescriptorSource indirect;
indirect.indirect_descriptor.emplace().sources = {0, 1};
program.descriptor_sources.push_back(indirect);
auto plan = ExtractResourcePlan(program);
for (const auto &inst : plan.value_storage) {
Check(inst.GetOpcode() != ValueOpcode::ReadConst &&
inst.GetOpcode() != ValueOpcode::GetSrtResource,
"retained plan still contains flattened-read aliases");
}
Check(plan.descriptor_sources[0].dwords[0] == plan.srt_reads[0].value &&
plan.srt_reads[0].value.Instruction()->Flags<SrtReadFlags>().clean == 1u &&
plan.srt_reads[1].value.Instruction()->Flags<SrtReadFlags>().clean == 0u &&
plan.srt_reads[2].value.Instruction()->Flags<SrtReadFlags>().clean == 1u,
"alias normalization lost read identity or retained discarded EXEC provenance");
struct Reads { uint32_t strict = 0; uint32_t ordinary = 0; bool dirty = false; } reads;
const SrtRuntime runtime{
.read_memory = +[](void *data, uint64_t address, std::span<uint32_t> words) {
++static_cast<Reads *>(data)->ordinary;
if (address == 0x2000u) words[0] = 0x2222u;
else if (address == 0x3000u) words[0] = 0x8000u;
else if (address == 0x8000u) words[0] = 0xeeeeu;
else return false;
return true;
},
.userdata = &reads,
.read_specialization_memory = +[](void *data, uint64_t address,
std::span<uint32_t> words) {
auto &reads = *static_cast<Reads *>(data);
++reads.strict;
if (reads.dirty) return false;
if (address == 0x3000u) words[0] = 0x1000u;
else if (address == 0x1000u) words[0] = 0x1111u;
else return false;
return true;
}};
std::vector<uint32_t> flat;
DescriptorValue descriptor;
{
SrtWalker clean(plan, CleanRuntime(runtime));
SrtWalker walker(plan, runtime, &clean);
Check(walker.RefreshFlatBuffer(flat) &&
flat == std::vector<uint32_t>{0x1111u, 0x2222u, 0x1000u} &&
walker.EvaluateDescriptor(0, descriptor) && descriptor.dwords[0] == 0x1111u &&
reads.strict == 2 && reads.ordinary == 1,
"normalized aliases changed nested strict reads or repeated a shared read");
}
Check(!SrtWalker(plan, runtime).RefreshFlatBuffer(flat),
"strict flat read accepted a missing clean evaluator");
auto no_reader = runtime;
no_reader.read_specialization_memory = nullptr;
{
SrtWalker clean(plan, CleanRuntime(no_reader));
Check(!SrtWalker(plan, no_reader, &clean).RefreshFlatBuffer(flat),
"strict flat read accepted a missing strict reader");
}
Check(SrtWalker(plan, runtime).EvaluateDescriptor(0, descriptor) &&
descriptor.dwords[0] == 0xeeeeu && reads.strict == 2 && reads.ordinary == 3,
"direct descriptor evaluation without a clean evaluator changed read domains");
reads.dirty = true;
{
SrtWalker clean(plan, CleanRuntime(runtime));
Check(!SrtWalker(plan, runtime, &clean).RefreshFlatBuffer(flat) &&
reads.strict == 3 && reads.ordinary == 3,
"dirty strict pointer fell back to an ordinary read");
}
}
void TestUniformVectorDescriptorRead() {
using namespace Libs::Graphics::ShaderRecompiler::IR;
Program program;
@@ -285,16 +377,24 @@ void TestUniformVectorDescriptorRead() {
}
count.SetArg(4, Value(false));
auto &inactive = block.AppendNewInst(ValueOpcode::ReadFirstLane, {Value(&count), Value(false)});
program.descriptor_sources.push_back({.dwords = {Value(&inactive)}, .dword_count = 1});
reads.strict = 0;
uint32_t result = 99;
Check(SrtWalker(program, runtime).Evaluate(Value(&inactive), result) &&
result == 0 && reads.strict == 0 && reads.ordinary == 0,
"literal false EXEC read vector memory");
{
const auto plan = ExtractResourcePlan(program);
Check(SrtWalker(plan, runtime).Evaluate(plan.descriptor_sources.back().dwords[0], result) &&
result == 0 && reads.strict == 0 && reads.ordinary == 0,
"literal false EXEC read vector memory");
}
count.SetArg(4, Value(true));
handle.SetArg(3, Value(0x204u));
Check(SrtWalker(program, runtime).Evaluate(Value(&count), result) &&
result == 0 && reads.strict == 0 && reads.ordinary == 0,
"invalid vector buffer format read memory");
program.descriptor_sources.back().dwords[0] = Value(&count);
{
const auto plan = ExtractResourcePlan(program);
Check(SrtWalker(plan, runtime).Evaluate(plan.descriptor_sources.back().dwords[0], result) &&
result == 0 && reads.strict == 0 && reads.ordinary == 0,
"invalid vector buffer format read memory");
}
handle.SetArg(3, Value(0x16204u));
count.SetArg(1, Value(&lane));
Check(!ValidateRuntimeValue(program, Value(&count)),
@@ -362,16 +462,16 @@ void TestWrittenDescriptorUsesStrictReaderOnce() {
{Value(&offset), Value(4u)});
auto &store_block = AddValueBlock(program);
AddValueBlock(program);
program.block_info[0].condition = Value(&condition);
program.block_info[0].terminator.kind =
program.blocks[0]->condition = Value(&condition);
program.blocks[0]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[0].terminator.true_block = 1;
program.block_info[0].terminator.false_block = 2;
program.block_info[1].id = 1;
program.block_info[1].terminator.kind =
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[0]->terminator.false_block = program.blocks[2];
program.blocks[1]->id = 1;
program.blocks[1]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::Return;
program.block_info[2].id = 2;
program.block_info[2].terminator.kind =
program.blocks[2]->id = 2;
program.blocks[2]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::Return;
program.memory_info.push_back({.kind = ResourceKind::Buffer, .resource = 0});
auto &output = store_block.AppendNewInst(ValueOpcode::GetBufferResource,
@@ -567,6 +667,7 @@ void DbgExit(int) { std::abort(); }
int main() {
TestMappedSrtUsesDirectReaderByDefault();
TestIntegerRuntimeValueFollowsSrtReads();
TestSrtAliasesRetainReadPolicy();
TestUniformVectorDescriptorRead();
TestExactReciprocalDescriptorArithmetic();
TestUnbasedFlatCacheHitMaterializes();
+255 -267
View File
@@ -1,6 +1,5 @@
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/recompiler/ir/passes/BindingLayout.h"
#include "graphics/shader/recompiler/ir/passes/ConstantPropagation.h"
#include "graphics/shader/recompiler/ir/passes/DeadCodeElimination.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
@@ -59,8 +58,8 @@ struct Fixture {
auto *result = storage.get();
program.block_storage.push_back(std::move(storage));
program.blocks.push_back(result);
program.block_info.push_back(
{.id = static_cast<uint32_t>(program.block_info.size())});
program.blocks.back()->id =
static_cast<uint32_t>(program.blocks.size() - 1u);
return result;
}
@@ -127,8 +126,8 @@ struct Fixture {
}
void PlanAndTrack() {
for (size_t index = 0; index < program.block_info.size(); ++index) {
const auto condition = program.block_info[index].condition;
for (size_t index = 0; index < program.blocks.size(); ++index) {
const auto condition = program.blocks[index]->condition;
if (!condition.IsEmpty())
Emit(ValueOpcode::Reference, {condition}, 0, program.blocks[index]);
}
@@ -476,9 +475,11 @@ void TestWaterfallImageTable() {
{guard, fixture->Emit(ValueOpcode::IEqual32, {image_words[word], local})});
}
}
fixture->program.block_info[0].condition = guard;
fixture->program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch, .true_block = 1u, .false_block = 2u};
fixture->program.blocks[0]->condition = guard;
fixture->program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = fixture->program.blocks[1u],
.false_block = fixture->program.blocks[2u]};
const auto sample = [&](Value image, Block *block) {
const auto sampler = fixture->Sampler({Value(0u), Value(0u), Value(0u), Value(0u)});
MemoryInfo memory;
@@ -590,19 +591,20 @@ void TestGuardedDirectImageTable() {
if (guard != Guard::Plain && guard != Guard::SccNonZero) {
condition = fixture.Emit(ValueOpcode::ConditionRef, {condition}, kind);
}
fixture.program.block_info[0].condition = condition;
fixture.program.block_info[0].terminator = {
fixture.program.blocks[0]->condition = condition;
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = guard == Guard::Zero ? 1u : 4u,
.false_block = guard == Guard::Zero ? 4u : 1u};
.true_block = fixture.program.blocks[guard == Guard::Zero ? 1u : 4u],
.false_block = fixture.program.blocks[guard == Guard::Zero ? 4u : 1u]};
for (uint32_t block = 1; block < 4; ++block) {
fixture.program.block_info[block].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = block + 1u};
fixture.program.blocks[block]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[block + 1u]};
}
fixture.program.block_info[4].terminator = {
fixture.program.blocks[4]->terminator = {
.kind = guard == Guard::Bypass ? CFG::TerminatorKind::Branch
: CFG::TerminatorKind::Return,
.true_block = 3u};
: CFG::TerminatorKind::Return,
.true_block = fixture.program.blocks[3u]};
const auto srt = fixture.Address(fixture.UserData(0), fixture.UserData(1));
std::array<Value, 2> pointer;
for (uint32_t word = 0; word < pointer.size(); ++word) {
@@ -768,27 +770,40 @@ void TestBoundedComputeImageLoop() {
if (variant == Variant::ExitBypass) exit->AddBranch(body);
if (variant == Variant::IncrementBypass || variant == Variant::PreviousBound)
exit->AddBranch(latch);
fixture.program.block_info[0].terminator = {
.kind = variant == Variant::EntryBypass ? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = 1u, .false_block = 2u};
fixture.program.block_info[1].terminator = {
fixture.program.blocks[0]->terminator = {
.kind = variant == Variant::EntryBypass
? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u],
.false_block = fixture.program.blocks[2u]};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = variant == Variant::TrueEdge ? 2u : 4u,
.false_block = variant == Variant::TrueEdge ? 4u : 2u};
fixture.program.block_info[2].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 3u};
fixture.program.block_info[3].terminator = {
.kind = variant == Variant::PreviousBound ? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = 1u, .false_block = 5u};
fixture.program.block_info[4].terminator = {
.true_block =
fixture.program.blocks[variant == Variant::TrueEdge ? 2u : 4u],
.false_block =
fixture.program.blocks[variant == Variant::TrueEdge ? 4u : 2u]};
fixture.program.blocks[2]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[3u]};
fixture.program.blocks[3]->terminator = {
.kind = variant == Variant::PreviousBound
? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u],
.false_block = final_exit};
fixture.program.blocks[4]->terminator = {
.kind = final_exit != exit || variant == Variant::ExitBypass ||
variant == Variant::IncrementBypass
? CFG::TerminatorKind::Branch : CFG::TerminatorKind::Return,
.true_block = final_exit != exit || variant == Variant::IncrementBypass ? 3u : 2u};
? CFG::TerminatorKind::Branch
: CFG::TerminatorKind::Return,
.true_block =
fixture.program.blocks[final_exit != exit ||
variant == Variant::IncrementBypass
? 3u
: 2u]};
if (final_exit != exit)
fixture.program.block_info[5].terminator.kind = CFG::TerminatorKind::Return;
fixture.program.blocks[5]->terminator.kind =
CFG::TerminatorKind::Return;
}
auto &phi = header->AppendNewInst(ValueOpcode::Phi, {},
@@ -860,14 +875,16 @@ void TestBoundedComputeImageLoop() {
block->AddBranch(fixture.program.blocks[yes]);
block->AddBranch(fixture.program.blocks[no]);
const auto inverse = fixture.Emit(ValueOpcode::LogicalNot, {predicate}, 0, block);
fixture.program.block_info[index].condition =
fixture.program.blocks[index]->condition =
fixture.Emit(ValueOpcode::ConditionRef, {inverse}, kind, block);
fixture.program.block_info[index].terminator = {
fixture.program.blocks[index]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = yes, .false_block = no};
.true_block = fixture.program.blocks[yes],
.false_block = fixture.program.blocks[no]};
};
fixture.program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 1u};
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u]};
branch(1u, 4u, 5u, active, CFG::BranchCondition::ExecZero);
branch(5u, variant == Variant::GuardedDiamondBypass ? 7u : 4u, 6u,
diamond ? execute : mask,
@@ -878,13 +895,15 @@ void TestBoundedComputeImageLoop() {
fixture.Emit(ValueOpcode::IEqual32, {local, Value(1u)}, 0, guard)}, 0, guard);
branch(6u, 3u, 2u, image_guard, CFG::BranchCondition::ExecZero);
body->AddBranch(latch);
fixture.program.block_info[2].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 3u};
fixture.program.blocks[2]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[3u]};
branch(3u, 4u, 7u, next_active, CFG::BranchCondition::ExecZero);
increment->AddBranch(header);
fixture.program.block_info[7].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 1u};
fixture.program.block_info[4].terminator.kind = CFG::TerminatorKind::Return;
fixture.program.blocks[7]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u]};
fixture.program.blocks[4]->terminator.kind = CFG::TerminatorKind::Return;
} else {
const auto active = initial_active;
const auto allowed = fixture.Emit(variant == Variant::Disjunction
@@ -900,14 +919,15 @@ void TestBoundedComputeImageLoop() {
: CFG::BranchCondition::ExecZero, header);
if (variant == Variant::Nonzero || variant == Variant::WrongPolarity)
condition = fixture.Emit(ValueOpcode::LogicalNot, {condition}, 0, header);
fixture.program.block_info[1].condition = condition;
fixture.program.blocks[1]->condition = condition;
if (variant == Variant::PreviousBound) {
previous->AddPhiOperand(entry, Value(false));
previous->AddPhiOperand(latch, in_range);
const auto continuing = fixture.Emit(ValueOpcode::LogicalOr,
{in_range, Value(previous)}, 0, latch);
fixture.program.block_info[3].condition = fixture.Emit(ValueOpcode::ConditionRef,
{continuing}, CFG::BranchCondition::SccNonZero, latch);
fixture.program.blocks[3]->condition =
fixture.Emit(ValueOpcode::ConditionRef, {continuing},
CFG::BranchCondition::SccNonZero, latch);
}
}
const auto step = fixture.Emit(ValueOpcode::IAdd32,
@@ -1068,21 +1088,27 @@ void TestUniformizedMaterialImageKeys() {
choose->AddBranch(done);
if (first_lane) {
sample_entry->AddBranch(first_lane_loop ? sample_header : sample);
fixture.program.block_info[sample_entry_id].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = first_lane_loop ? 9u : 7u};
fixture.program.blocks[sample_entry_id]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[first_lane_loop ? 9u : 7u]};
}
if (first_lane_loop) {
sample_header->AddBranch(sample);
sample->AddBranch(sample_header);
fixture.program.block_info[9].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 7u,
.merge_block = 8u, .continue_block = 7u, .loop_header = true};
fixture.program.blocks[9]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[7u],
.merge_block = fixture.program.blocks[8u],
.continue_block = fixture.program.blocks[7u],
.loop_header = true};
}
sample->AddBranch(done);
fixture.program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 1u};
fixture.program.block_info[1].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 2u};
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u]};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[2u]};
const auto enabled = fixture.Emit(
ValueOpcode::INotEqual32, {fixture.UserData(5u), Value(0u)}, 0, entry);
const auto local = fixture.Emit(ValueOpcode::GetBuiltin,
@@ -1098,23 +1124,26 @@ void TestUniformizedMaterialImageKeys() {
ValueOpcode::Phi, {}, static_cast<uint64_t>(Type::U32));
const auto mask = Value(&mask_phi);
const auto active = Value(&active_phi);
fixture.program.block_info[2].condition = branch(
fixture.Emit(ValueOpcode::LogicalNot, {active}, 0, inactive),
variant == Variant::WrongExit ? CFG::BranchCondition::ExecNonZero
: CFG::BranchCondition::ExecZero, inactive);
fixture.program.block_info[2].terminator = {
fixture.program.blocks[2]->condition =
branch(fixture.Emit(ValueOpcode::LogicalNot, {active}, 0, inactive),
variant == Variant::WrongExit ? CFG::BranchCondition::ExecNonZero
: CFG::BranchCondition::ExecZero,
inactive);
fixture.program.blocks[2]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 5u, .false_block = 3u};
.true_block = fixture.program.blocks[5u],
.false_block = fixture.program.blocks[3u]};
const auto nonzero = fixture.Emit(
ValueOpcode::INotEqual32, {Value(0u), mask}, 0, sentinel);
const auto bit_guard = fixture.Emit(
ValueOpcode::LogicalAnd, {active, nonzero}, 0, sentinel);
fixture.program.block_info[3].condition = branch(
fixture.Emit(ValueOpcode::LogicalNot, {bit_guard}, 0, sentinel),
CFG::BranchCondition::ExecZero, sentinel);
fixture.program.block_info[3].terminator = {
fixture.program.blocks[3]->condition =
branch(fixture.Emit(ValueOpcode::LogicalNot, {bit_guard}, 0, sentinel),
CFG::BranchCondition::ExecZero, sentinel);
fixture.program.blocks[3]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 5u, .false_block = 4u};
.true_block = fixture.program.blocks[5u],
.false_block = fixture.program.blocks[4u]};
const auto first = fixture.Emit(ValueOpcode::FindILsb32, {mask}, 0, bit);
const auto position = fixture.Emit(
ValueOpcode::BitwiseAnd32, {first, Value(31u)}, 0, bit);
@@ -1131,11 +1160,12 @@ void TestUniformizedMaterialImageKeys() {
{mask, removed}, 0, bit);
const auto continuation = fixture.Emit(
ValueOpcode::LogicalAnd, {bit_guard, active_on_entry}, 0, bit);
fixture.program.block_info[4].condition = branch(
continuation, CFG::BranchCondition::ExecNonZero, bit);
fixture.program.block_info[4].terminator = {
fixture.program.blocks[4]->condition =
branch(continuation, CFG::BranchCondition::ExecNonZero, bit);
fixture.program.blocks[4]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 1u, .false_block = 5u};
.true_block = fixture.program.blocks[1u],
.false_block = fixture.program.blocks[5u]};
active_phi.AddPhiOperand(entry, active_on_entry);
active_phi.AddPhiOperand(bit, continuation);
mask_phi.AddPhiOperand(entry, fixture.UserData(4u));
@@ -1160,8 +1190,9 @@ void TestUniformizedMaterialImageKeys() {
ValueOpcode::SGreaterThan32, {Value(32u), index}, 0, merge);
const auto material_guard = fixture.Emit(
ValueOpcode::LogicalAnd, {active_on_entry, below}, 0, merge);
fixture.program.block_info[5].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 6u};
fixture.program.blocks[5]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[6u]};
const auto scaled = fixture.Emit(
ValueOpcode::ShiftLeftLogical32, {index, Value(4u)}, 0, choose);
const auto selected_scale = fixture.Emit(
@@ -1203,12 +1234,13 @@ void TestUniformizedMaterialImageKeys() {
sample_active_phi->AddPhiOperand(sample_entry, initial);
sample_active = Value(sample_active_phi);
}
fixture.program.block_info[6].condition = branch(
variant == Variant::EmptyFirstEntry ? enabled : initial,
CFG::BranchCondition::ExecNonZero, choose);
fixture.program.block_info[6].terminator = {
fixture.program.blocks[6]->condition =
branch(variant == Variant::EmptyFirstEntry ? enabled : initial,
CFG::BranchCondition::ExecNonZero, choose);
fixture.program.blocks[6]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = sample_entry_id, .false_block = 8u};
.true_block = fixture.program.blocks[sample_entry_id],
.false_block = fixture.program.blocks[8u]};
}
auto *key_block = first_lane ? sample : choose;
const auto key = first_lane
@@ -1220,12 +1252,13 @@ void TestUniformizedMaterialImageKeys() {
const auto sample_guard = fixture.Emit(
ValueOpcode::LogicalAnd, {sample_active, compared}, 0, key_block);
if (!first_lane) {
fixture.program.block_info[6].condition = branch(
fixture.program.blocks[6]->condition = branch(
fixture.Emit(ValueOpcode::LogicalNot, {sample_guard}, 0, choose),
CFG::BranchCondition::ExecZero, choose);
fixture.program.block_info[6].terminator = {
fixture.program.blocks[6]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 8u, .false_block = 7u};
.true_block = fixture.program.blocks[8u],
.false_block = fixture.program.blocks[7u]};
}
const auto table_offset = fixture.Emit(
ValueOpcode::IAdd32,
@@ -1273,17 +1306,19 @@ void TestUniformizedMaterialImageKeys() {
if (variant == Variant::WideningFirstBackedge)
remaining = fixture.Emit(ValueOpcode::LogicalOr, {remaining, enabled}, 0, sample);
sample_active_phi->AddPhiOperand(sample, remaining);
fixture.program.block_info[7].condition = branch(
fixture.program.blocks[7]->condition = branch(
variant == Variant::WrongFirstBackedge ? sample_guard : remaining,
CFG::BranchCondition::ExecNonZero, sample);
fixture.program.block_info[7].terminator = {
fixture.program.blocks[7]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 9u, .false_block = 8u};
.true_block = fixture.program.blocks[9u],
.false_block = fixture.program.blocks[8u]};
} else {
fixture.program.block_info[7].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 8u};
fixture.program.blocks[7]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[8u]};
}
fixture.program.block_info[8].terminator.kind = CFG::TerminatorKind::Return;
fixture.program.blocks[8]->terminator.kind = CFG::TerminatorKind::Return;
const auto output = fixture.Emit(
ValueOpcode::GetBufferResource,
{fixture.UserData(8u), fixture.UserData(9u),
@@ -1557,10 +1592,10 @@ void TestComputeBufferFill() {
};
const auto Run = [](Options options) {
Fixture fixture;
fixture.program.block_info[0].terminator.kind =
fixture.program.blocks[0]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::Return;
if (options.branch) {
fixture.program.block_info[0].terminator.kind = Libs::Graphics::
fixture.program.blocks[0]->terminator.kind = Libs::Graphics::
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
}
const auto buffer =
@@ -1759,17 +1794,18 @@ void TestRuntimeUnsignedMinDescriptor() {
{descriptor, Value(0u), Value(0u), Value(0u), Value(true)},
fixture.AddMemory(memory, 0x330));
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
std::array<uint32_t, 1> user_data{0xffffffffu};
SrtRuntime runtime{.user_data = user_data};
DescriptorValue value;
const auto source = fixture.program.info.buffers[0].source;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(source, value) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(source, value) &&
value.dwords[3] == 0x100u,
"runtime descriptor unsigned minimum did not clamp its first operand");
user_data[0] = 0x80u;
Check(
SrtWalker(fixture.program, runtime).EvaluateDescriptor(source, value) &&
SrtWalker(plan, runtime).EvaluateDescriptor(source, value) &&
value.dwords[3] == 0x80u,
"runtime descriptor unsigned minimum did not preserve its first operand");
}
@@ -1876,6 +1912,7 @@ void TestSampleAdjustSamplerScratch() {
{image, shift_sampler, fixture.ImageAddress()},
fixture.AddMemory(memory, 0x200));
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
const auto source = fixture.program.info.samplers[0].source;
const auto stored = fixture.program.descriptor_sources[source]
@@ -1887,12 +1924,12 @@ void TestSampleAdjustSamplerScratch() {
std::array<uint32_t, 4> user_data{4u, 1u, 2u, 0x80000abcu};
SrtRuntime runtime{.user_data = user_data};
DescriptorValue descriptor;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(source, descriptor) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(source, descriptor) &&
descriptor.dwords[3] == 0x80000abcu,
"SampleAdjust canonicalization lost sampler border fields");
const auto shift_source = fixture.program.info.samplers[
shift_sampler.Instruction()->Flags<uint32_t>()].source;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(shift_source, descriptor) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(shift_source, descriptor) &&
std::ranges::equal(std::span(descriptor.dwords).first(4),
std::array{0x36u, 0xfff000u, 0x02500000u, 0u}),
"SampleAdjust shift-only scratch was not reduced to its zero border payload");
@@ -2009,12 +2046,8 @@ void TestFmaskLoadSpecialization() {
"inactive FMASK load did not preserve the execution mask");
ShaderComputeInputInfo compute{};
CollectShaderInfo(fixture.program, {.compute = &compute});
AllocateBindings(fixture.program);
const auto kind = DescriptorBindingForImage(fixture.program.info.images[0]);
Check(kind.has_value() &&
FindBinding(fixture.program.bindings, *kind)->resources ==
std::vector<uint32_t>{0},
"FMASK allocated an ordinary image descriptor");
Check(DescriptorBindingForImage(fixture.program.info.images[0]).has_value(),
"FMASK removal lost the ordinary image class");
user_data[8] = 0;
user_data[1] = static_cast<uint32_t>(Prospero::BufferFormat::k8UInt) << 20u;
ResourceSpecialization rebound;
@@ -2152,14 +2185,9 @@ void TestDynamicStorageMipTracking() {
"base-1 through last-3 dynamic storage range was not specialized");
ShaderComputeInputInfo compute{};
CollectShaderInfo(fixture.program, {.compute = &compute});
AllocateBindings(fixture.program);
const auto storage_kind = DescriptorBindingForImage(images[0]);
Check(storage_kind.has_value(), "storage image has no descriptor binding");
const auto *storage_binding =
FindBinding(fixture.program.bindings, *storage_kind);
Check(storage_binding != nullptr &&
storage_binding->resources == std::vector<uint32_t>({0, 1, 1, 1}),
"dynamic storage mip descriptors were not expanded consecutively");
Check(DescriptorBindingForImage(images[0]) == DescriptorBindingForImage(images[1]) &&
images[0].mip_count == 1 && images[1].mip_mode == ImageMipMode::Dynamic,
"dynamic storage mip metadata did not retain one homogeneous image class");
Fixture null_fixture;
const auto null_handle = null_fixture.Image(
@@ -2249,6 +2277,7 @@ void TestSrtFlatteningAndRuntimeMemoization() {
SrtRuntime runtime{.user_data = user_data,
.read_memory = ReadTestMemory,
.userdata = &memory};
auto plan = ExtractResourcePlan(fixture.program);
DescriptorValue descriptor;
std::vector<uint32_t> flat;
const uint32_t request = fixture.program.info.buffers[0].source;
@@ -2256,27 +2285,25 @@ void TestSrtFlatteningAndRuntimeMemoization() {
SrtWalker walker(plan, runtime);
return walker.EvaluateDescriptor(request, descriptor) && walker.RefreshFlatBuffer(flat);
};
Check(refresh(fixture.program), "typed runtime source evaluation failed");
Check(refresh(plan), "typed runtime source evaluation failed");
Check(descriptor.dwords[0] == 0xdeadbeefu &&
flat == std::vector<uint32_t>{0xdeadbeefu} && memory.reads == 1,
"descriptor and flat SRT evaluation did not share one memoized read");
memory.reads = 0;
memory.words[1] = 0x12345678u;
Check(refresh(fixture.program) && descriptor.dwords[0] == 0x12345678u &&
Check(refresh(plan) && descriptor.dwords[0] == 0x12345678u &&
flat == std::vector<uint32_t>{0x12345678u} && memory.reads == 1,
"repeated runtime evaluation reused stale scalar memory");
memory.reads = 0;
memory.fail_after = 0;
Check(!refresh(fixture.program), "unavailable scalar memory was accepted");
Check(!refresh(plan), "unavailable scalar memory was accepted");
memory.fail_after = UINT32_MAX;
Check(refresh(fixture.program) && descriptor.dwords[0] == 0x12345678u && memory.reads == 1,
Check(refresh(plan) && descriptor.dwords[0] == 0x12345678u && memory.reads == 1,
"failed runtime evaluation left a value marked as visiting");
auto detached = ExtractResourcePlan(fixture.program);
Check(refresh(detached), "detached resource plan did not evaluate");
auto moved = std::move(detached);
auto moved = std::move(plan);
memory.reads = 0;
memory.words[1] = 0x87654321u;
Check(refresh(moved) && descriptor.dwords[0] == 0x87654321u && memory.reads == 1,
@@ -2284,10 +2311,8 @@ void TestSrtFlatteningAndRuntimeMemoization() {
ShaderComputeInputInfo compute{};
CollectShaderInfo(fixture.program, {.compute = &compute});
AllocateBindings(fixture.program);
Check(FindBinding(fixture.program.bindings,
DescriptorBindingKind::FlattenedSrt) != nullptr,
"flattened typed SRT reads did not receive a binding");
Check(fixture.program.info.uses_flattened_srt,
"flattened typed SRT reads were not collected");
}
void TestDynamicSrtReadRemainsExplicit() {
@@ -2308,6 +2333,7 @@ void TestDynamicSrtReadRemainsExplicit() {
{descriptor, Value(0u), Value(0u), Value(0u), Value(true)},
fixture.AddMemory(buffer, 8));
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
Check(fixture.program.srt_reads.empty() &&
read.ResolveInstruction()->GetOpcode() == ValueOpcode::LoadAddressU32 &&
@@ -2321,26 +2347,15 @@ void TestDynamicSrtReadRemainsExplicit() {
.read_memory = ReadTestMemory,
.userdata = &memory};
DescriptorValue value;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(fixture.program.info.buffers[0].source, value) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(fixture.program.info.buffers[0].source, value) &&
value.dwords[0] == 0xabcdef01u && memory.reads == 1,
"dynamic typed scalar descriptor source was not evaluated");
ShaderComputeInputInfo compute{};
CollectShaderInfo(fixture.program, {.compute = &compute});
AllocateBindings(fixture.program);
Check(FindBinding(fixture.program.bindings,
DescriptorBindingKind::FlattenedSrt) == nullptr &&
FindBinding(fixture.program.bindings,
DescriptorBindingKind::BdaPagetable) != nullptr &&
FindBinding(fixture.program.bindings,
DescriptorBindingKind::FaultBuffer) != nullptr,
"dynamic scalar read received the wrong resource bindings");
Check(fixture.program.bindings.memory_offset_dword ==
fixture.program.bindings.user_data_registers.size() &&
fixture.program.bindings.memory_offset_count == 1u &&
fixture.program.bindings.ShaderDataDwords() ==
fixture.program.bindings.memory_offset_dword + 1u,
"unified memory-offset layout is inconsistent");
Check(!fixture.program.info.uses_flattened_srt && fixture.program.info.uses_dma &&
fixture.program.info.live_buffers == 1u,
"dynamic scalar read received the wrong resource usage");
}
void TestWritableDescriptorPhi() {
@@ -2390,16 +2405,20 @@ ResourcePlan ConditionalSamplerPlan(bool diamond, bool reverse, bool reverse_phi
alternate->AddBranch(merge);
if (diamond) {
initial->AddBranch(merge);
fixture.program.block_info[1].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = merge_id};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[merge_id]};
}
fixture.program.block_info[0].terminator = {
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = reverse ? alternate_id : initial_target,
.false_block = reverse ? initial_target : alternate_id};
fixture.program.block_info[alternate_id].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = merge_id};
fixture.program.block_info[merge_id].terminator.kind =
.true_block =
fixture.program.blocks[reverse ? alternate_id : initial_target],
.false_block =
fixture.program.blocks[reverse ? initial_target : alternate_id]};
fixture.program.blocks[alternate_id]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[merge_id]};
fixture.program.blocks[merge_id]->terminator.kind =
CFG::TerminatorKind::Return;
const auto control =
fixture.Buffer({Value(0x2000u), Value(0u), Value(200u), Value(0u)});
@@ -2410,7 +2429,7 @@ ResourcePlan ConditionalSamplerPlan(bool diamond, bool reverse, bool reverse_phi
if (nonuniform) {
flag = fixture.Emit(ValueOpcode::LaneId);
}
fixture.program.block_info[0].condition =
fixture.program.blocks[0]->condition =
fixture.Emit(ValueOpcode::SGreaterThanEqual32, {flag, Value(0u)});
if (writable) {
MemoryInfo memory;
@@ -2498,7 +2517,7 @@ void TestConditionalSamplerPhi() {
memory.fail_address = 0x1000 + (first == 448 / 4 ? 480u : 448u);
DescriptorValue selected;
SrtWalker clean(plan, CleanRuntime(runtime));
Check(SrtWalker(plan, runtime, {}, &clean).EvaluateDescriptor(source, selected),
Check(SrtWalker(plan, runtime, &clean).EvaluateDescriptor(source, selected),
"conditional sampler did not survive detached plan lifetime");
for (uint32_t word = 0; word < 4; ++word) {
Check(selected.dwords[word] == memory.words[first + word],
@@ -2510,13 +2529,13 @@ void TestConditionalSamplerPhi() {
no_clean_reader.read_specialization_memory = nullptr;
{
SrtWalker clean(plan, CleanRuntime(no_clean_reader));
Check(!SrtWalker(plan, no_clean_reader, {}, &clean).EvaluateDescriptor(source, selected),
Check(!SrtWalker(plan, no_clean_reader, &clean).EvaluateDescriptor(source, selected),
"conditional sampler used unchecked memory for its predicate");
}
memory.fail_address = 0x2000 + 196;
{
SrtWalker clean(plan, CleanRuntime(runtime));
Check(!SrtWalker(plan, runtime, {}, &clean).EvaluateDescriptor(source, selected),
Check(!SrtWalker(plan, runtime, &clean).EvaluateDescriptor(source, selected),
"conditional sampler ignored unavailable coherent predicate memory");
}
}
@@ -2567,6 +2586,7 @@ void TestFiniteImagePhiCycle() {
fixture.Emit(ValueOpcode::ImageSampleRaw, {image, sampler, fixture.ImageAddress()},
fixture.AddMemory(memory, 4));
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
const auto source = fixture.program.info.images[0].source;
const auto &finite = fixture.program.descriptor_sources[source].indirect_descriptor;
Check(finite && finite->sources.size() == 2,
@@ -2581,7 +2601,7 @@ void TestFiniteImagePhiCycle() {
"finite image key did not preserve the loop-carried selection");
std::array<uint32_t, 16> data;
for (uint32_t i = 0; i < data.size(); ++i) data[i] = 100u + i;
SrtWalker walker(fixture.program, SrtRuntime{.user_data = data});
SrtWalker walker(plan, SrtRuntime{.user_data = data});
for (uint32_t candidate = 0; candidate < 2; ++candidate) {
DescriptorValue descriptor;
Check(walker.EvaluateDescriptor(finite->sources[candidate], descriptor),
@@ -2610,22 +2630,21 @@ void TestFiniteImageBitScanSentinel() {
{Value(uint32_t(StageInputKind::LocalInvocationIndex)), Value(0u)});
const auto condition = fixture.Emit(nonzero ? ValueOpcode::INotEqual32 : ValueOpcode::IEqual32,
{mask, Value(0u)});
auto &guard = fixture.program.block_info[0];
auto &guard = *fixture.program.blocks[0];
guard.condition = condition;
guard.terminator.kind = CFG::TerminatorKind::ConditionalBranch;
guard.terminator.true_block = 1;
guard.terminator.false_block = 4;
guard.terminator.true_block = dispatch;
guard.terminator.false_block = exit;
fixture.block = dispatch;
const auto first = fixture.Emit(ValueOpcode::FindILsb32, {mask});
const auto minimum = fixture.Emit(ValueOpcode::UMin32, {first, Value(32u)});
const auto selector = fixture.Emit(ValueOpcode::ShiftLeftLogical32, {minimum, Value(2u)});
fixture.Emit(ValueOpcode::ReferenceU32, {selector});
auto &table = fixture.program.block_info[1];
auto &table = *fixture.program.blocks[1];
table.indirect_target = selector;
table.terminator.kind = CFG::TerminatorKind::IndirectBranch;
table.terminator.indirect_selector_code = 0;
table.terminator.indirect_selector_values = {0u, 128u};
table.terminator.indirect_selector_targets = {2u, 3u};
table.terminator.indexed = true;
table.terminator.cases = {{0u, load}, {128u, merge}};
std::array<Value, 8> words;
for (uint32_t word = 0; word < words.size(); ++word) {
auto &phi = merge->AppendNewInst(ValueOpcode::Phi, {}, uint64_t(Type::U32));
@@ -2681,10 +2700,11 @@ void TestLoopCycleEnteredThroughRuntimeValue() {
ConstantPropagationPass(fixture.program.blocks);
RemoveIdentities(fixture.program.blocks);
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
const std::array<uint32_t, 1> user_data{0x4000u};
DescriptorValue descriptor;
Check(fixture.program.info.buffers.size() == 1u &&
SrtWalker(fixture.program, {.user_data = user_data}).EvaluateDescriptor(
SrtWalker(plan, {.user_data = user_data}).EvaluateDescriptor(
fixture.program.info.buffers[0].source, descriptor) &&
descriptor.dwords[0] == user_data[0],
"runtime-rooted invariant loop lost its buffer source");
@@ -2710,11 +2730,12 @@ void TestInvariantLoopPhi() {
{handle, Value(0u), Value(0u), Value(0u), Value(true)},
fixture.AddMemory(memory, 4), loop);
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
std::array<uint32_t, 1> user_data{0x12345678u};
SrtRuntime runtime{.user_data = user_data};
DescriptorValue descriptor;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(fixture.program.info.buffers[0].source, descriptor) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(fixture.program.info.buffers[0].source, descriptor) &&
descriptor.dwords[0] == user_data[0],
"loop-invariant descriptor phi was not evaluated through typed SSA");
}
@@ -2761,21 +2782,25 @@ void TestBoundedRelativeRegisterWrites() {
header->AddBranch(body);
header->AddBranch(exit);
body->AddBranch(header);
fixture.program.block_info[0].terminator = {
.kind = variant == Variant::EntryBypass ? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = 1u, .false_block = 2u};
fixture.program.block_info[1].terminator = {
fixture.program.blocks[0]->terminator = {
.kind = variant == Variant::EntryBypass
? CFG::TerminatorKind::ConditionalBranch
: CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u],
.false_block = fixture.program.blocks[2u]};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 2u, .false_block = 3u};
fixture.program.block_info[2].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 1u};
fixture.program.block_info[3].terminator.kind = CFG::TerminatorKind::Return;
.true_block = fixture.program.blocks[2u],
.false_block = fixture.program.blocks[3u]};
fixture.program.blocks[2]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[1u]};
fixture.program.blocks[3]->terminator.kind = CFG::TerminatorKind::Return;
const auto lane = fixture.Emit(ValueOpcode::GetBuiltin,
{Value(static_cast<uint32_t>(StageInputKind::LocalInvocationId)), Value(0u)});
const auto enabled = fixture.Emit(ValueOpcode::ULessThan32, {lane, Value(32u)});
fixture.program.block_info[0].condition = enabled;
fixture.program.blocks[0]->condition = enabled;
const auto chunk = fixture.Emit(ValueOpcode::SelectU32,
{enabled, Value(64u), Value(512u)});
const auto initial_bound = fixture.Emit(ValueOpcode::ShiftRightLogical32,
@@ -2787,8 +2812,9 @@ void TestBoundedRelativeRegisterWrites() {
auto &records = header->AppendNewInst(ValueOpcode::Phi, {}, uint64_t(Type::U32));
const auto in_range = fixture.Emit(ValueOpcode::ULessThan32,
{Value(&counter), Value(&bound)}, 0, header);
fixture.program.block_info[1].condition = fixture.Emit(
ValueOpcode::ConditionRef, {in_range}, CFG::BranchCondition::VccNonZero, header);
fixture.program.blocks[1]->condition =
fixture.Emit(ValueOpcode::ConditionRef, {in_range},
CFG::BranchCondition::VccNonZero, header);
const auto m0 = fixture.Emit(ValueOpcode::BitwiseAnd32,
{fixture.Emit(ValueOpcode::ShiftLeftLogical32,
{Value(&counter), Value(1u)}, 0, body), Value(255u)}, 0, body);
@@ -2822,10 +2848,11 @@ void TestBoundedRelativeRegisterWrites() {
continue;
}
fixture.PlanAndTrack();
auto plan = ExtractResourcePlan(fixture.program);
const std::array<uint32_t, 3> user_data{0x1000u, 0u, 72u};
SrtRuntime runtime{.user_data = user_data};
DescriptorValue descriptor;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(
Check(SrtWalker(plan, runtime).EvaluateDescriptor(
fixture.program.info.buffers[0].source, descriptor) &&
descriptor.dwords[2] == 72u,
"impossible relative writes left a false descriptor dependency");
@@ -2970,26 +2997,27 @@ ResourcePlan ConditionalBufferPlan(ConditionalBufferUse use) {
auto *done = fixture.AddBlock();
auto *condition_block = entry;
uint32_t condition_index = 0;
fixture.program.block_info[0].id = 11;
fixture.program.block_info[1].id = 27;
fixture.program.block_info[2].id = 42;
fixture.program.blocks[0]->id = 11;
fixture.program.blocks[1]->id = 27;
fixture.program.blocks[2]->id = 42;
if (use == ConditionalBufferUse::Loop) {
condition_block = fixture.AddBlock();
condition_index = 3;
fixture.program.block_info[3].id = 55;
fixture.program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 55};
fixture.program.blocks[3]->id = 55;
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = condition_block};
entry->AddBranch(condition_block);
}
condition_block->AddBranch(optional);
condition_block->AddBranch(done);
optional->AddBranch(use == ConditionalBufferUse::Loop ? condition_block : done);
fixture.program.block_info[condition_index].terminator = {
fixture.program.blocks[condition_index]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 27, .false_block = 42};
fixture.program.block_info[1].terminator = {
.true_block = optional,
.false_block = done};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = use == ConditionalBufferUse::Loop ? 55u : 42u};
.true_block = use == ConditionalBufferUse::Loop ? condition_block : done};
const auto control = fixture.Buffer(
{fixture.UserData(0), fixture.UserData(1), fixture.UserData(2),
@@ -3005,8 +3033,8 @@ ResourcePlan ConditionalBufferPlan(ConditionalBufferUse use) {
phi.AddPhiOperand(optional, Value(1u));
flag = Value(&phi);
}
fixture.program.block_info[condition_index].condition =
fixture.Emit(ValueOpcode::INotEqual32, {flag, Value(0u)}, 0, condition_block);
fixture.program.blocks[condition_index]->condition = fixture.Emit(
ValueOpcode::INotEqual32, {flag, Value(0u)}, 0, condition_block);
const auto payload = fixture.Buffer(
{fixture.UserData(4), fixture.UserData(5), fixture.UserData(6),
@@ -3084,11 +3112,14 @@ void TestGuardedScalarDescriptorReads() {
entry->AddBranch(optional);
entry->AddBranch(done);
optional->AddBranch(done);
fixture.program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch, .true_block = 1, .false_block = 2};
fixture.program.block_info[1].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 2};
fixture.program.block_info[2].terminator.kind = CFG::TerminatorKind::Return;
fixture.program.blocks[0]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = fixture.program.blocks[1],
.false_block = fixture.program.blocks[2]};
fixture.program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::Branch,
.true_block = fixture.program.blocks[2]};
fixture.program.blocks[2]->terminator.kind = CFG::TerminatorKind::Return;
const auto control = fixture.Buffer({fixture.UserData(0), fixture.UserData(1),
fixture.UserData(2), fixture.UserData(3)});
MemoryInfo scalar;
@@ -3100,7 +3131,7 @@ void TestGuardedScalarDescriptorReads() {
{Value(static_cast<uint32_t>(StageInputKind::LocalInvocationId)), Value(0u)});
const auto varying = fixture.Emit(ValueOpcode::INotEqual32, {lane, Value(0u)});
const auto enabled = fixture.Emit(ValueOpcode::INotEqual32, {flag, Value(0u)});
fixture.program.block_info[0].condition =
fixture.program.blocks[0]->condition =
fixture.Emit(ValueOpcode::LogicalAnd, {varying, enabled});
const auto root = fixture.Address(fixture.UserData(4), Value(0u));
const auto Load = [&](Block *block) {
@@ -3209,15 +3240,16 @@ void TestConditionalIndirectImageMaterialization() {
body->AddBranch(done);
const auto flag = fixture->Emit(ValueOpcode::GetUserData,
{Value(static_cast<ScalarReg>(8))}, 0, entry);
fixture->program.block_info[1].condition = fixture->Emit(
ValueOpcode::INotEqual32, {flag, Value(0u)}, 0, entry);
fixture->program.block_info[1].terminator = {
fixture->program.blocks[1]->condition =
fixture->Emit(ValueOpcode::INotEqual32, {flag, Value(0u)}, 0, entry);
fixture->program.blocks[1]->terminator = {
.kind = CFG::TerminatorKind::ConditionalBranch,
.true_block = 0, .false_block = 2};
fixture->program.block_info[0].terminator = {
.kind = CFG::TerminatorKind::Branch, .true_block = 2};
.true_block = fixture->program.blocks[0],
.false_block = fixture->program.blocks[2]};
fixture->program.blocks[0]->terminator = {.kind = CFG::TerminatorKind::Branch,
.true_block =
fixture->program.blocks[2]};
std::swap(fixture->program.blocks[0], fixture->program.blocks[1]);
std::swap(fixture->program.block_info[0], fixture->program.block_info[1]);
fixture->PlanAndTrack();
auto plan = ExtractResourcePlan(fixture->program);
std::array<uint32_t, 9> user_data{
@@ -3240,7 +3272,7 @@ void TestConditionalIndirectImageMaterialization() {
"taken indirect image branch did not require its descriptor table");
}
void TestShaderInfoAndBindingLayout() {
void TestShaderInfoCollection() {
Fixture fixture;
const auto handle = fixture.Buffer(
{fixture.UserData(3), fixture.UserData(4), Value(64u), Value(0u)}, 4);
@@ -3270,15 +3302,9 @@ void TestShaderInfoAndBindingLayout() {
StageInputKind::GlobalInvocationId,
"typed shader values were not reflected in shader info");
AllocateBindings(fixture.program);
Check(FindBinding(fixture.program.bindings, DescriptorBindingKind::Buffers) !=
nullptr &&
FindBinding(fixture.program.bindings, DescriptorBindingKind::Gds) !=
nullptr &&
FindBinding(fixture.program.bindings,
DescriptorBindingKind::ShaderData) == nullptr &&
fixture.program.bindings.UsesPushData(),
"typed resources were not assigned native bindings");
Check(fixture.program.info.live_buffers == 1u &&
fixture.program.info.uses_gds && !fixture.program.info.uses_lds,
"typed resources were not collected in shader info");
Check(NativeBinding(ShaderType::Compute, DescriptorBindingKind::Buffers) ==
static_cast<uint32_t>(DescriptorBindingKind::Buffers) &&
NativeBinding(ShaderType::Vertex, DescriptorBindingKind::Buffers) ==
@@ -3287,9 +3313,8 @@ void TestShaderInfoAndBindingLayout() {
static_cast<uint32_t>(DescriptorBindingKind::Count) +
static_cast<uint32_t>(DescriptorBindingKind::Buffers),
"fixed stage binding ranges are inconsistent");
Check(fixture.program.bindings.user_data_registers ==
std::vector<uint32_t>({3u, 4u}),
"binding layout did not collect live typed user-data values");
Check(fixture.program.info.user_data_registers == std::vector<uint32_t>({3u, 4u}),
"shader info did not collect live typed user-data values");
}
void TestImageBindingAbi() {
@@ -3414,60 +3439,27 @@ void TestImageBindingAbi() {
Check(Invalid(image), "float atomic image received a descriptor binding");
}
void TestGraphicsPushConstantLayout() {
const auto AddUserData = [](Fixture &fixture, uint32_t count) {
for (uint32_t index = 0; index < count; index++) {
fixture.Emit(ValueOpcode::ReferenceU32, {fixture.UserData(index)});
}
fixture.program.shader_info_complete = true;
};
uint32_t cursor = 0;
Fixture pixel(ShaderType::Pixel);
AddUserData(pixel, 4);
AllocateBindings(pixel.program, cursor);
Check(
pixel.program.bindings.UsesPushData() &&
pixel.program.bindings.push_data_start_dword == 0 &&
FindBinding(pixel.program.bindings,
DescriptorBindingKind::ShaderData) == nullptr,
"pixel shader did not start the shared push-data block");
pixel.program.bindings.AdvancePushData(cursor);
void TestLiveShaderDataCollection() {
Fixture fixture;
fixture.UserData(7);
fixture.Emit(ValueOpcode::ReferenceU32, {fixture.UserData(5)});
fixture.Emit(ValueOpcode::ReferenceU32, {fixture.UserData(2)});
fixture.Emit(ValueOpcode::ReferenceU32, {fixture.UserData(5)});
fixture.Emit(ValueOpcode::GetDispatchThreadExtent, {Value(1u)});
fixture.PlanAndTrack();
ShaderComputeInputInfo compute{};
CollectShaderInfo(fixture.program, {.compute = &compute});
Check(fixture.program.info.user_data_registers == std::vector<uint32_t>({2u, 5u}) &&
!fixture.program.info.uses_dispatch_threads,
"shader data collection retained unused values or duplicated registers");
Fixture vertex(ShaderType::Vertex);
AddUserData(vertex, 9);
AllocateBindings(vertex.program, cursor);
Check(vertex.program.bindings.UsesPushData() &&
vertex.program.bindings.push_data_start_dword == 4,
"vertex shader did not follow pixel data in the shared push-data block");
vertex.program.bindings.AdvancePushData(cursor);
Check(cursor == 13, "graphics push-data cursor advanced incorrectly");
Fixture edge(ShaderType::Pixel);
AddUserData(edge, NativePushConstantSize / sizeof(uint32_t));
AllocateBindings(edge.program);
Check(edge.program.bindings.UsesPushData() &&
FindBinding(edge.program.bindings,
DescriptorBindingKind::ShaderData) == nullptr,
"the full shared push-data block did not fit");
Fixture spill(ShaderType::Pixel);
AddUserData(spill, 20);
AllocateBindings(spill.program, cursor);
Check(
!spill.program.bindings.UsesPushData() &&
spill.program.bindings.push_data_start_dword == PushData::NoStart &&
FindBinding(spill.program.bindings,
DescriptorBindingKind::ShaderData) != nullptr,
"a stage that exceeded the remaining shared push data did not spill to storage");
const auto spill_layout = spill.program.bindings;
spill.program.bindings.AdvancePushData(cursor);
Check(cursor == 13, "a spilled stage consumed shared push-data space");
Fixture repeated_spill(ShaderType::Pixel);
AddUserData(repeated_spill, 20);
AllocateBindings(repeated_spill.program, 20);
Check(repeated_spill.program.bindings == spill_layout,
"storage fallback retained an irrelevant attempted push-data position");
const auto extent = fixture.Emit(ValueOpcode::GetDispatchThreadExtent, {Value(0u)});
fixture.Emit(ValueOpcode::ReferenceU32, {extent});
fixture.program.shader_info_complete = false;
CollectShaderInfo(fixture.program, {.compute = &compute});
Check(fixture.program.info.uses_dispatch_threads &&
fixture.program.info.user_data_registers == std::vector<uint32_t>({2u, 5u}),
"shader data recollection lost a live dispatch extent or retained stale registers");
}
void TestResourceLimitIsTransactional() {
@@ -3489,12 +3481,8 @@ void TestResourceLimitIsTransactional() {
"compute shader did not retain all 64 distinct buffers");
ShaderComputeInputInfo compute{};
CollectShaderInfo(accepted.program, {.compute = &compute});
AllocateBindings(accepted.program);
const auto *binding = FindBinding(accepted.program.bindings,
DescriptorBindingKind::Buffers);
Check(binding != nullptr && binding->resources.size() == 64u &&
accepted.program.bindings.memory_offset_count == 64u,
"compute shader binding layout truncated the 64 buffers");
Check(accepted.program.info.live_buffers == UINT64_MAX,
"shader info truncated the 64 live buffers");
Fixture fixture;
MemoryInfo memory;
@@ -3596,9 +3584,9 @@ int main() {
Run("guarded scalar descriptor reads", TestGuardedScalarDescriptorReads);
Run("conservative buffer reachability", TestConservativeBufferReachability);
Run("conditional indirect image", TestConditionalIndirectImageMaterialization);
Run("shader info and bindings", TestShaderInfoAndBindingLayout);
Run("shader info collection", TestShaderInfoCollection);
Run("image binding ABI", TestImageBindingAbi);
Run("graphics push constants", TestGraphicsPushConstantLayout);
Run("live shader data collection", TestLiveShaderDataCollection);
Run("resource limit", TestResourceLimitIsTransactional);
Run("malformed memory kinds", TestMalformedMemoryKindsRejected);
} catch (const std::exception &exception) {
+23 -19
View File
@@ -3,6 +3,7 @@
#include "graphics/shader/recompiler/ir/passes/DeadCodeElimination.h"
#include "graphics/shader/recompiler/ir/passes/ReadLaneElimination.h"
#include "graphics/shader/recompiler/ir/passes/ResourceTracking.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
#include "graphics/shader/recompiler/ir/passes/SrtWalker.h"
#include <cstdlib>
@@ -45,7 +46,7 @@ struct Fixture {
program.block_storage.push_back(std::make_unique<Block>());
auto *block = program.block_storage.back().get();
program.blocks.push_back(block);
program.block_info.push_back({.id = index});
program.blocks.back()->id = index;
}
}
@@ -131,7 +132,7 @@ void TestImmediateFlatteningAndGvn() {
TestMemory memory_image{{{0x1020u, 0xfeedbeefu}}};
SrtRuntime runtime{.read_memory = ReadMemory, .userdata = &memory_image};
std::vector<uint32_t> flat;
Check(SrtWalker(fixture.program, runtime).RefreshFlatBuffer(flat), "flattened SRT walk failed");
Check(SrtWalker(ExtractResourcePlan(fixture.program), runtime).RefreshFlatBuffer(flat), "flattened SRT walk failed");
Check(flat == std::vector<uint32_t>{0xfeedbeefu} && memory_image.reads == 1,
"flattened SRT did not evaluate its canonical read once");
}
@@ -147,7 +148,7 @@ void TestRawScalarComponentAlignment() {
TestMemory memory_image{{{0x1000u, 0x12345678u}}};
SrtRuntime runtime{.read_memory = ReadMemory, .userdata = &memory_image};
std::vector<uint32_t> flat;
Check(SrtWalker(fixture.program, runtime).RefreshFlatBuffer(flat), "raw scalar SRT walk failed");
Check(SrtWalker(ExtractResourcePlan(fixture.program), runtime).RefreshFlatBuffer(flat), "raw scalar SRT walk failed");
Check(
flat == std::vector<uint32_t>{0x12345678u} && memory_image.reads == 1,
"raw scalar base, immediate, and offset were not aligned independently");
@@ -202,7 +203,7 @@ void TestNestedSrtWalk() {
TestMemory memory_image{{{0x1000u, 0x2000u}, {0x2000u, 0xabcdef01u}}};
SrtRuntime runtime{.read_memory = ReadMemory, .userdata = &memory_image};
std::vector<uint32_t> flat;
Check(SrtWalker(fixture.program, runtime).RefreshFlatBuffer(flat), "nested SRT walk failed");
Check(SrtWalker(ExtractResourcePlan(fixture.program), runtime).RefreshFlatBuffer(flat), "nested SRT walk failed");
Check(flat == std::vector<uint32_t>({0x2000u, 0xabcdef01u}),
"nested typed SRT reads were not evaluated in dependency order");
}
@@ -225,7 +226,7 @@ void TestShaderBaseAndUserData() {
SrtRuntime runtime{.user_data = user_data,
.shader_base = 0x12345678abcdef00ull};
DescriptorValue result;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(0, result),
Check(SrtWalker(ExtractResourcePlan(fixture.program), runtime).EvaluateDescriptor(0, result),
"shader-relative descriptor evaluation failed");
Check(result.dword_count == 3 && result.dwords[0] == 0xabcdef00u &&
result.dwords[1] == 0x12345678u && result.dwords[2] == 0x24u,
@@ -250,7 +251,7 @@ void TestCarryAndBitFields() {
{.dwords = {low, high, inserted, sign}, .dword_count = 4});
DescriptorValue result;
Check(SrtWalker(fixture.program, {}).EvaluateDescriptor(0, result),
Check(SrtWalker(ExtractResourcePlan(fixture.program), {}).EvaluateDescriptor(0, result),
"carry and bit-field descriptor evaluation failed");
Check(result.dwords[0] == 1u && result.dwords[1] == 1u &&
result.dwords[2] == 0x89abcdefu && result.dwords[3] == 0xffffffffu,
@@ -273,11 +274,12 @@ void TestInvariantAndDivergentPhi() {
fixture.program.descriptor_sources.push_back(
{.dwords = {Value(&divergent)}, .dword_count = 1});
const auto plan = ExtractResourcePlan(fixture.program);
DescriptorValue result;
Check(SrtWalker(fixture.program, {}).EvaluateDescriptor(0, result) &&
Check(SrtWalker(plan, {}).EvaluateDescriptor(0, result) &&
result.dwords[0] == 7u,
"loop-invariant typed phi was rejected");
Check(!SrtWalker(fixture.program, {}).EvaluateDescriptor(1, result),
Check(!SrtWalker(plan, {}).EvaluateDescriptor(1, result),
"divergent phi was accepted");
}
@@ -374,7 +376,7 @@ void TestRuntime64BitDescriptorOps() {
{.dwords = {low, high}, .dword_count = 2});
DescriptorValue result;
Check(SrtWalker(fixture.program, {}).EvaluateDescriptor(0, result) &&
Check(SrtWalker(ExtractResourcePlan(fixture.program), {}).EvaluateDescriptor(0, result) &&
result.dwords[0] == 0xabcdu && result.dwords[1] == 0x1235u,
"64-bit typed descriptor arithmetic evaluation is incorrect");
}
@@ -447,12 +449,13 @@ void TestUniformFirstLaneSamplerLod() {
std::array<uint32_t, 7> user_data{};
user_data[6] = 3u;
SrtRuntime runtime{.user_data = user_data};
const auto plan = ExtractResourcePlan(fixture.program);
DescriptorValue result;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(0, result) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(0, result) &&
result.dwords[0] == 0x00300300u,
"uniform sampler LOD evaluated incorrectly");
user_data[6] = 20u;
Check(SrtWalker(fixture.program, runtime).EvaluateDescriptor(0, result) &&
Check(SrtWalker(plan, runtime).EvaluateDescriptor(0, result) &&
result.dwords[0] == 0x00ffffffu,
"uniform sampler LOD clamp evaluated incorrectly");
}
@@ -487,11 +490,12 @@ void TestFloatComparisonDescriptorInputs() {
fixture.program.descriptor_sources.push_back(
{.dwords = {low, high}, .dword_count = 2});
const auto plan = ExtractResourcePlan(fixture.program);
for (size_t index = 0; index < inputs.size(); index++) {
const auto &input = inputs[index];
const std::array user_data{input.bits};
DescriptorValue result;
Check(SrtWalker(fixture.program, {.user_data = user_data})
Check(SrtWalker(plan, {.user_data = user_data})
.EvaluateDescriptor(0, result) &&
result.dwords[0] == (flush && index < 4 ? 1 : input.less_equal) &&
result.dwords[1] == (flush && index < 4 ? 1 : input.greater_equal),
@@ -566,7 +570,7 @@ void TestConstantBufferBounds() {
TestMemory memory_image{{{0x3000u, 0x12345678u}, {0x300cu, 0xa5a5a5a5u}}};
SrtRuntime runtime{.read_memory = ReadMemory, .userdata = &memory_image};
std::vector<uint32_t> flat;
Check(SrtWalker(fixture.program, runtime).RefreshFlatBuffer(flat) == test.valid,
Check(SrtWalker(ExtractResourcePlan(fixture.program), runtime).RefreshFlatBuffer(flat) == test.valid,
"constant-buffer walk misaligned or wrapped its offset components");
Check(test.valid ? flat == std::vector<uint32_t>{test.expected} : memory_image.reads == 0,
"constant-buffer walk read the wrong word or accessed an out-of-bounds address");
@@ -620,14 +624,14 @@ void TestControlFlowValueSurvivesReadLaneFolding() {
entry->AddBranch(taken);
entry->AddBranch(other);
auto &entry_info = fixture.program.block_info[0];
auto &entry_info = *fixture.program.blocks[0];
entry_info.terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
entry_info.terminator.true_block = 1;
entry_info.terminator.false_block = 2;
fixture.program.block_info[1].terminator.kind =
entry_info.terminator.true_block = taken;
entry_info.terminator.false_block = other;
fixture.program.blocks[1]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::Return;
fixture.program.block_info[2].terminator.kind =
fixture.program.blocks[2]->terminator.kind =
Libs::Graphics::ShaderRecompiler::CFG::TerminatorKind::Return;
const auto undef = fixture.Emit(ValueOpcode::UndefU32);
@@ -690,7 +694,7 @@ void TestUndefinedRuntimeValueFails() {
fixture.program.descriptor_sources.push_back(
{.dwords = {undef}, .dword_count = 1});
DescriptorValue result;
Check(!SrtWalker(fixture.program, {}).EvaluateDescriptor(0, result),
Check(!SrtWalker(ExtractResourcePlan(fixture.program), {}).EvaluateDescriptor(0, result),
"undefined typed descriptor source was accepted");
}
File diff suppressed because it is too large. Load diff
+262 -248
View File
@@ -21,6 +21,7 @@
#include "graphics/shader/recompiler/ir/passes/DeadCodeElimination.h"
#include "graphics/shader/recompiler/ir/passes/ReadLaneElimination.h"
#include "graphics/shader/recompiler/ir/passes/ResourceTracking.h"
#include "graphics/shader/recompiler/ir/passes/ResourceMaterialization.h"
#include "graphics/shader/recompiler/ir/passes/ShaderInfoCollection.h"
#include "graphics/shader/recompiler/ir/passes/SrtWalker.h"
#include "graphics/shader/recompiler/ir/passes/SsaRewrite.h"
@@ -1294,10 +1295,14 @@ void TestNormalizedImageContracts() {
"Vulkan image-view compatibility classes diverged from production");
}
void TestSpirvRequirementsAnalysis() {
void TestShaderFeatureCollection() {
using namespace ShaderRecompiler::IR;
ShaderPixelInputInfo pixel_info{};
ShaderComputeInputInfo compute_info{};
ShaderVertexInputInfo vertex_info{};
Program program;
program.resource_tracking_complete = true;
program.stage = ShaderType::Pixel;
program.block_storage.push_back(std::make_unique<Block>());
auto *block = program.block_storage.back().get();
@@ -1318,8 +1323,8 @@ void TestSpirvRequirementsAnalysis() {
block->AppendNewInst(ValueOpcode::SetAttribute, {Value(0u), Value(true)});
export_value.SetFlags(ExportFlags{.index = 0});
const auto requirements =
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(program);
CollectShaderInfo(program, {.pixel = &pixel_info});
const auto& requirements = program.info;
Check(requirements.subgroup_ballot && requirements.subgroup_shuffle &&
requirements.subgroup_local_invocation_id &&
requirements.compute_derivatives &&
@@ -1328,8 +1333,9 @@ void TestSpirvRequirementsAnalysis() {
"consolidated SPIR-V requirements missed an IR dependency");
program.stage = ShaderType::Compute;
const auto compute_requirements =
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(program);
program.shader_info_complete = false;
CollectShaderInfo(program, {.compute = &compute_info});
const auto& compute_requirements = program.info;
Check(!compute_requirements.function_lds &&
!compute_requirements.pixel_valid_mask,
"stage-specific SPIR-V requirements leaked into compute");
@@ -1339,19 +1345,22 @@ void TestSpirvRequirementsAnalysis() {
[&] {
program.stage = ShaderType::Pixel;
shared.SetFlags(MemoryFlags{.index = 1});
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(program);
program.shader_info_complete = false;
CollectShaderInfo(program, {.pixel = &pixel_info});
},
"invalid shared-memory requirement metadata did not terminate analysis");
ExpectFatal(
[&] {
program.stage = ShaderType::Pixel;
export_value.SetFlags(ExportFlags{.index = 1});
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(program);
program.shader_info_complete = false;
CollectShaderInfo(program, {.pixel = &pixel_info});
},
"invalid export requirement metadata did not terminate analysis");
#endif
Program add_tid;
add_tid.resource_tracking_complete = true;
add_tid.stage = ShaderType::Compute;
add_tid.info.buffers.resize(1);
add_tid.info.buffers[0].packed_stride = 1u << 20u;
@@ -1367,8 +1376,8 @@ void TestSpirvRequirementsAnalysis() {
ValueOpcode::LoadBufferU32,
{Value(&buffer), Value(0u), Value(0u), Value(0u), Value(true)});
load.SetFlags(MemoryFlags{.index = 0});
const auto add_tid_requirements =
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(add_tid);
CollectShaderInfo(add_tid, {.compute = &compute_info});
const auto& add_tid_requirements = add_tid.info;
Check(add_tid_requirements.subgroup_local_invocation_id &&
!add_tid_requirements.subgroup_ballot &&
!add_tid_requirements.subgroup_shuffle,
@@ -1378,20 +1387,24 @@ void TestSpirvRequirementsAnalysis() {
ExpectFatal(
[&] {
add_tid.memory_info[0].resource = 1;
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(add_tid);
add_tid.shader_info_complete = false;
CollectShaderInfo(add_tid, {.compute = &compute_info});
},
"invalid buffer requirement metadata did not terminate analysis");
ExpectFatal(
[&] {
add_tid.stage = ShaderType::Vertex;
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(add_tid);
add_tid.shader_info_complete = false;
CollectShaderInfo(add_tid, {.vertex = &vertex_info});
},
"graphics buffer ADD_TID did not terminate requirements analysis");
#endif
Program empty;
const auto empty_requirements =
ShaderRecompiler::Spirv::Emitter::AnalyzeProgramRequirements(empty);
empty.stage = ShaderType::Compute;
empty.resource_tracking_complete = true;
CollectShaderInfo(empty, {.compute = &compute_info});
const auto& empty_requirements = empty.info;
Check(!empty_requirements.subgroup_ballot &&
!empty_requirements.subgroup_shuffle &&
!empty_requirements.subgroup_local_invocation_id &&
@@ -1916,13 +1929,14 @@ void TestSopkCompareImmediateExtension() {
};
auto translated =
TranslateProgram(shader, MakeCompileOptions(ShaderType::Vertex));
const auto branch = std::ranges::find_if(
translated.program.block_info, [](const auto &block) {
return block.terminator.kind == CFG::TerminatorKind::ConditionalBranch;
const auto branch =
std::ranges::find_if(translated.program.blocks, [](const auto *block) {
return block->terminator.kind ==
CFG::TerminatorKind::ConditionalBranch;
});
Check(branch != translated.program.block_info.end(),
Check(branch != translated.program.blocks.end(),
"captured LUT parameter export branch was lost");
const auto condition = branch->condition.Resolve();
const auto condition = (*branch)->condition.Resolve();
Check(condition.IsImmediate() && condition.U1() == (counts != 6u),
"captured LUT unsigned compare suppressed a live parameter export");
}
@@ -5209,25 +5223,13 @@ void TestNewShaderRecompilerScalarMemoryBindingDomains() {
raw_options.user_data = raw_user_data;
auto raw = RecompileForTest(raw_shader, raw_options);
const auto *address_binding = ShaderRecompiler::IR::FindBinding(
raw.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::BdaPagetable);
const auto *fault_binding = ShaderRecompiler::IR::FindBinding(
raw.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::FaultBuffer);
const auto& raw_counts = raw.program.bindings.descriptor_counts;
using BindingKind = ShaderRecompiler::IR::DescriptorBindingKind;
Check(raw.program.info.uses_dma && raw.program.info.buffers.empty() &&
address_binding != nullptr && fault_binding != nullptr &&
address_binding->resources.empty() &&
fault_binding->resources.empty() &&
ShaderRecompiler::IR::FindBinding(
raw.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::Buffers) ==
nullptr &&
ShaderRecompiler::IR::FindBinding(
raw.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::FlattenedSrt) ==
nullptr &&
raw.program.bindings.memory_offset_count == 0u,
raw_counts[static_cast<size_t>(BindingKind::BdaPagetable)] == 1 &&
raw_counts[static_cast<size_t>(BindingKind::FaultBuffer)] == 1 &&
raw_counts[static_cast<size_t>(BindingKind::Buffers)] == 0 &&
raw_counts[static_cast<size_t>(BindingKind::FlattenedSrt)] == 0,
"raw scalar load did not use only the DMA domain");
Check(count_live_memory_ops(
raw.program, ShaderRecompiler::IR::ValueOpcode::LoadAddressU32,
@@ -5255,23 +5257,14 @@ void TestNewShaderRecompilerScalarMemoryBindingDomains() {
buffer_options.user_data = buffer_user_data;
auto buffer = RecompileForTest(buffer_shader, buffer_options);
const auto *buffer_binding = ShaderRecompiler::IR::FindBinding(
buffer.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::Buffers);
const auto& buffer_counts = buffer.program.bindings.descriptor_counts;
Check(buffer.program.info.buffers.size() == 1u &&
buffer.program.info.buffers[0].scalar &&
buffer.program.info.buffers[0].descriptor_index == 0 &&
!buffer.program.info.uses_dma &&
buffer_binding != nullptr &&
buffer_binding->resources == std::vector<uint32_t>{0u} &&
ShaderRecompiler::IR::FindBinding(
buffer.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::BdaPagetable) ==
nullptr &&
ShaderRecompiler::IR::FindBinding(
buffer.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::FlattenedSrt) ==
nullptr &&
buffer.program.bindings.memory_offset_count == 1u,
buffer_counts[static_cast<size_t>(BindingKind::Buffers)] == 1 &&
buffer_counts[static_cast<size_t>(BindingKind::BdaPagetable)] == 0 &&
buffer_counts[static_cast<size_t>(BindingKind::FlattenedSrt)] == 0,
"descriptor scalar load did not use only the buffer domain");
Check(count_live_memory_ops(
buffer.program, ShaderRecompiler::IR::ValueOpcode::ReadConstBuffer,
@@ -6066,8 +6059,7 @@ void TestNewShaderRecompilerImageLoad2DMsaa() {
const auto binding_kind = ShaderRecompiler::IR::DescriptorBindingForImage(
result.program.info.images[0]);
Check(binding_kind.has_value() &&
ShaderRecompiler::IR::FindBinding(result.program.bindings,
*binding_kind) != nullptr,
result.program.bindings.descriptor_counts[static_cast<size_t>(*binding_kind)] != 0,
"2D-MSAA image did not receive a multisampled descriptor binding");
Check(SpirvContainsTypeImage(result.spirv, 1, 0, 1, 1),
"SPIR-V binary does not contain a multisampled 2D image type");
@@ -7232,14 +7224,8 @@ void TestNewShaderRecompilerUnbasedFlatUsesBda() {
auto result = RecompileForTest(shader, options);
Check(result.program.info.uses_dma &&
ShaderRecompiler::IR::FindBinding(
result.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::BdaPagetable) !=
nullptr &&
ShaderRecompiler::IR::FindBinding(
result.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::FaultBuffer) !=
nullptr,
result.program.bindings.descriptor_counts[static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::BdaPagetable)] != 0 &&
result.program.bindings.descriptor_counts[static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::FaultBuffer)] != 0,
"unbased FLAT did not compile through BDA");
CheckSpirvBinaryValidates(result.spirv);
}
@@ -7978,6 +7964,8 @@ void TestNewShaderRecompilerStructuredU64Phi() {
use.Emit(IR::ValueOpcode::CompositeExtractU64,
{IR::Value(&phi), IR::Value(1u)});
program.shader_info_complete = false;
ShaderRecompiler::IR::CollectShaderInfo(program, options.input_info);
auto spirv = ShaderRecompiler::Spirv::EmitProgram(program, options.input_info);
CheckSpirvBinaryValidates(spirv);
const auto before = MeasureSpirv(result.spirv);
@@ -9841,6 +9829,8 @@ void TestNewShaderRecompilerDispatcherSpillsU32x3() {
IR::IREmitter use(prologue_program.blocks[1]);
use.Emit(IR::ValueOpcode::CompositeExtractU32x3, {vector, IR::Value(2u)});
prologue_program.shader_info_complete = false;
IR::CollectShaderInfo(prologue_program, options.input_info);
auto spirv = ShaderRecompiler::Spirv::EmitProgram(prologue_program,
options.input_info);
CheckSpirvBinaryValidates(spirv);
@@ -9866,6 +9856,8 @@ void TestNewShaderRecompilerDispatcherSpillsU32x3() {
const auto sum = use.Emit(IR::ValueOpcode::IAdd32, {high, IR::Value(7u)});
use.Emit(IR::ValueOpcode::IAdd32, {sum, middle});
program.shader_info_complete = false;
IR::CollectShaderInfo(program, options.input_info);
auto spirv = ShaderRecompiler::Spirv::EmitProgram(program,
options.input_info);
CheckSpirvBinaryValidates(spirv);
@@ -9902,6 +9894,8 @@ void TestNewShaderRecompilerPlanningOnlyLoads() {
read.Instruction()->SetFlags(IR::MemoryFlags{.index = 1});
ir.Emit(IR::ValueOpcode::ReferenceU32, {load});
ir.Emit(IR::ValueOpcode::ReferenceU32, {read});
program.shader_info_complete = false;
IR::CollectShaderInfo(program, options.input_info);
const auto spirv = Spirv::EmitProgram(program, options.input_info);
CheckSpirvBinaryValidates(spirv);
Check(spirv == result.spirv,
@@ -10035,6 +10029,8 @@ void TestNewShaderRecompilerNativeU64Translation() {
ir.Emit(IR::ValueOpcode::ShiftRightArithmetic64, {base, IR::Value(count)});
}
program.shader_info_complete = false;
ShaderRecompiler::IR::CollectShaderInfo(program, options.input_info);
auto spirv = ShaderRecompiler::Spirv::EmitProgram(program, options.input_info);
CheckSpirvBinaryValidates(spirv);
const auto source = DisassembleSpirvBinary(spirv);
@@ -10072,6 +10068,8 @@ void TestNewShaderRecompilerNativeU64Translation() {
use.Emit(IR::ValueOpcode::CompositeExtractU64, {vector, IR::Value(1u)});
spirv.clear();
program.shader_info_complete = false;
ShaderRecompiler::IR::CollectShaderInfo(program, options.input_info);
spirv = ShaderRecompiler::Spirv::EmitProgram(program, options.input_info);
CheckSpirvBinaryValidates(spirv);
const auto before = MeasureSpirv(result.spirv);
@@ -10432,13 +10430,13 @@ void TestMeshExportStorage() {
CheckSpirvBinaryValidates(result.spirv);
const auto &layout = result.program.bindings;
Check(layout.UsesPushData() == (push_data_start == PushData::MeshDrawDwordCount) &&
layout.memory_offset_count == 1,
layout.descriptor_counts[static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::Buffers)] == 1,
"mesh shader did not retain its user-data placement and buffer offset");
const auto reg = std::ranges::find(layout.user_data_registers, 13u);
Check(reg != layout.user_data_registers.end(), "mesh shader lost user s13");
const auto reg = std::ranges::find(result.program.info.user_data_registers, 13u);
Check(reg != result.program.info.user_data_registers.end(), "mesh shader lost user s13");
const auto source = DisassembleSpirvBinary(result.spirv);
for (const auto dword :
{static_cast<uint32_t>(reg - layout.user_data_registers.begin()),
{static_cast<uint32_t>(reg - result.program.info.user_data_registers.begin()),
layout.memory_offset_dword}) {
const auto operand =
std::string(layout.UsesPushData() ? "vsharp" : "shader_data") +
@@ -10568,14 +10566,16 @@ void TestMergedShaderUserDataSnapshot() {
const auto &program = translated.program;
Check(program.info.buffers.size() == 2 && program.srt_reads.size() == 4,
"merged shader lost front user SGPRs or the back-stage SRT load");
const auto plan = IR::ExtractResourcePlan(program);
for (const auto *params : {&first, &second}) {
const auto user_data = std::span(params->user_data).first(params->user_data_count);
const IR::SrtRuntime runtime{.user_data = user_data,
.read_memory = ReadHostTestMemory};
IR::SrtWalker walker(plan, runtime);
IR::DescriptorValue front_descriptor, back_descriptor;
const auto &table = params == &first ? first_table : second_table;
Check(IR::SrtWalker(program, runtime).EvaluateDescriptor(program.info.buffers[0].source, front_descriptor) &&
IR::SrtWalker(program, runtime).EvaluateDescriptor(program.info.buffers[1].source, back_descriptor) &&
Check(walker.EvaluateDescriptor(program.info.buffers[0].source, front_descriptor) &&
walker.EvaluateDescriptor(program.info.buffers[1].source, back_descriptor) &&
std::equal(user_data.begin() + 8, user_data.end(),
front_descriptor.dwords.begin()) &&
std::equal(table.begin(), table.end(), back_descriptor.dwords.begin()),
@@ -11084,14 +11084,12 @@ void TestNewShaderRecompilerSetpcJumpTable() {
Check((result.ir_dump.find("mode=dispatcher") != std::string::npos),
"S_SETPC_B64 jump table did not select dispatcher fallback");
const auto jump = std::find_if(
result.program.block_info.begin(),
result.program.block_info.end(), [](const auto &block) {
return !block.terminator.indirect_targets.empty();
});
Check(jump != result.program.block_info.end() &&
jump->terminator.indirect_targets.size() == 2,
result.program.blocks.begin(), result.program.blocks.end(),
[](const auto *block) { return !block->terminator.cases.empty(); });
Check(jump != result.program.blocks.end() &&
(*jump)->ImmSuccessors().size() == 2,
"S_SETPC_B64 jump table did not retain both targets");
Check(jump->terminator.indirect_selector_values.size() == 2,
Check((*jump)->terminator.cases.size() == 2,
"S_SETPC_B64 jump table did not retain selector mapping");
Check(
(result.ir_dump.find("SLoadDword") == std::string::npos),
@@ -11188,31 +11186,21 @@ void TestNewShaderRecompilerSetpcDwordJumpTable() {
Check((result.ir_dump.find("mode=dispatcher") != std::string::npos),
"subtractive S_SETPC_B64 table did not select dispatcher fallback");
const auto jump = std::find_if(
result.program.block_info.begin(),
result.program.block_info.end(), [](const auto &block) {
return !block.terminator.indirect_targets.empty();
});
const auto block_pc = [&](uint32_t id) {
const auto block =
std::find_if(result.program.block_info.begin(),
result.program.block_info.end(),
[=](const auto &info) { return info.id == id; });
return block != result.program.block_info.end() ? block->start_pc
: UINT32_MAX;
};
Check(jump != result.program.block_info.end() &&
jump->terminator.indirect_targets.size() == 2 &&
jump->terminator.indirect_target_pcs ==
std::vector<uint32_t>({0x38u, 0x40u}) &&
jump->terminator.indirect_selector_values ==
std::vector<uint32_t>({0u, 4u, 8u}) &&
jump->terminator.indirect_selector_targets.size() == 3 &&
block_pc(jump->terminator.indirect_selector_targets[0]) == 0x38u &&
block_pc(jump->terminator.indirect_selector_targets[1]) == 0x40u &&
block_pc(jump->terminator.indirect_selector_targets[2]) == 0x38u &&
result.program.blocks.begin(), result.program.blocks.end(),
[](const auto *block) { return !block->terminator.cases.empty(); });
Check(jump != result.program.blocks.end() &&
(*jump)->ImmSuccessors().size() == 2 &&
(*jump)->terminator.indexed &&
(*jump)->terminator.cases.size() == 3 &&
(*jump)->terminator.cases[0].value == 0u &&
(*jump)->terminator.cases[0].target->start_pc == 0x38u &&
(*jump)->terminator.cases[1].value == 4u &&
(*jump)->terminator.cases[1].target->start_pc == 0x40u &&
(*jump)->terminator.cases[2].value == 8u &&
(*jump)->terminator.cases[2].target->start_pc == 0x38u &&
std::ranges::none_of(
result.program.block_info,
[](const auto &info) { return info.start_pc == 0x04u; }),
result.program.blocks,
[](const auto *block) { return block->start_pc == 0x04u; }),
"subtractive S_SETPC_B64 table targets or selector mapping changed");
Check((result.ir_dump.find("SLoadDword") == std::string::npos),
"subtractive S_SETPC_B64 table load reached normal IR");
@@ -11803,7 +11791,6 @@ void TestFinalSsaRejectsRegisterStatePseudos() {
Program program;
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.emplace_back();
IREmitter ir(program.blocks.front());
if (setter) {
ir.SetScalarReg(ScalarReg{0}, U32(Value(1u)));
@@ -11843,7 +11830,7 @@ void TestValuePhiValidation() {
program.block_storage.push_back(std::make_unique<Block>());
auto *block = program.block_storage.back().get();
program.blocks.push_back(block);
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
return block;
};
@@ -11851,17 +11838,17 @@ void TestValuePhiValidation() {
auto *left = add_block(1);
auto *right = add_block(2);
auto *join = add_block(3);
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[0].terminator.true_block = 1;
program.block_info[0].terminator.false_block = 2;
program.block_info[0].condition = Value(true);
program.block_info[1].terminator.kind =
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[0]->terminator.false_block = program.blocks[2];
program.blocks[0]->condition = Value(true);
program.blocks[1]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[1].terminator.true_block = 3;
program.block_info[2].terminator.kind =
program.blocks[1]->terminator.true_block = program.blocks[3];
program.blocks[2]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[2].terminator.true_block = 3;
program.blocks[2]->terminator.true_block = program.blocks[3];
entry->AddBranch(left);
entry->AddBranch(right);
left->AddBranch(join);
@@ -11911,12 +11898,12 @@ void TestValuePhiValidation() {
program.block_storage.push_back(std::make_unique<Block>());
auto *block = program.block_storage.back().get();
program.blocks.push_back(block);
program.block_info.push_back({.id = duplicate_id ? 0u : id});
program.blocks.back()->id = duplicate_id ? 0u : id;
}
if (!duplicate_id) {
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[0].terminator.true_block = 1;
program.blocks[0]->terminator.true_block = program.blocks[1];
}
expect_invalid(program,
"malformed Value CFG did not terminate IR validation");
@@ -11929,14 +11916,14 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 2; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[0].terminator.true_block = 1;
program.block_info[1].terminator.kind =
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[1]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[1].terminator.true_block = 0;
program.blocks[1]->terminator.true_block = program.blocks[0];
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[1]->AddBranch(program.blocks[0]);
expect_invalid(program,
@@ -11947,17 +11934,15 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 3; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[0].terminator.true_block = 1;
auto &indirect = program.block_info[1];
program.blocks[0]->terminator.true_block = program.blocks[1];
auto &indirect = *program.blocks[1];
indirect.terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::IndirectBranch;
indirect.terminator.indirect_targets = {1};
indirect.terminator.indirect_selector_values = {0};
indirect.terminator.indirect_selector_targets = {2};
indirect.terminator.cases = {{0, program.blocks[2]}};
indirect.indirect_target = Value(0u);
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[1]->AddBranch(program.blocks[1]);
@@ -11969,7 +11954,7 @@ void TestValuePhiValidation() {
Program program;
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = UINT32_MAX});
program.blocks.back()->id = UINT32_MAX;
expect_invalid(program,
"reserved exit block id did not terminate IR validation");
}
@@ -11978,27 +11963,29 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 2; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[0].terminator.true_block = 1;
auto &indirect = program.block_info[1];
program.blocks[0]->terminator.true_block = program.blocks[1];
auto &indirect = *program.blocks[1];
indirect.terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::IndirectBranch;
indirect.terminator.indirect_targets = {1, 1};
indirect.terminator.cases = {{0, program.blocks[1]},
{0, program.blocks[1]}};
indirect.indirect_target = Value(0u);
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[1]->AddBranch(program.blocks[1]);
expect_invalid(program,
"duplicate indirect targets did not terminate IR validation");
expect_invalid(
program,
"duplicate indirect selector values did not terminate IR validation");
}
{
Program program;
for (uint32_t id = 0; id < 2; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
expect_invalid(program,
"unreachable block did not terminate IR validation");
@@ -12008,7 +11995,6 @@ void TestValuePhiValidation() {
program.block_storage.push_back(std::make_unique<Block>());
auto *block = program.block_storage.back().get();
program.blocks.push_back(block);
program.block_info.push_back({.id = 0});
auto &use =
block->AppendNewInst(ValueOpcode::IAdd32, {Value(1u), Value(2u)});
auto &definition =
@@ -12022,17 +12008,17 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 4; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[0].terminator.true_block = 1;
program.block_info[0].terminator.false_block = 2;
program.block_info[0].condition = Value(true);
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[0]->terminator.false_block = program.blocks[2];
program.blocks[0]->condition = Value(true);
for (uint32_t id : {1u, 2u}) {
program.block_info[id].terminator.kind =
program.blocks[id]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[id].terminator.true_block = 3;
program.blocks[id]->terminator.true_block = program.blocks[3];
}
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[0]->AddBranch(program.blocks[2]);
@@ -12050,17 +12036,17 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 4; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[0].terminator.true_block = 1;
program.block_info[0].terminator.false_block = 2;
program.block_info[0].condition = Value(true);
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[0]->terminator.false_block = program.blocks[2];
program.blocks[0]->condition = Value(true);
for (uint32_t id : {1u, 2u}) {
program.block_info[id].terminator.kind =
program.blocks[id]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[id].terminator.true_block = 3;
program.blocks[id]->terminator.true_block = program.blocks[3];
}
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[0]->AddBranch(program.blocks[2]);
@@ -12081,23 +12067,23 @@ void TestValuePhiValidation() {
for (uint32_t id = 0; id < 5; id++) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
program.block_info[0].terminator.kind =
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[0].terminator.true_block = 1;
program.block_info[0].terminator.false_block = 2;
program.block_info[0].condition = Value(true);
program.block_info[1].terminator.kind =
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[0]->terminator.false_block = program.blocks[2];
program.blocks[0]->condition = Value(true);
program.blocks[1]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[1].terminator.true_block = 3;
program.block_info[2].terminator.kind =
program.blocks[1]->terminator.true_block = program.blocks[3];
program.blocks[2]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[2].terminator.true_block = 3;
program.block_info[2].terminator.false_block = 4;
program.block_info[3].terminator.kind =
program.blocks[2]->terminator.true_block = program.blocks[3];
program.blocks[2]->terminator.false_block = program.blocks[4];
program.blocks[3]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[3].terminator.true_block = 4;
program.blocks[3]->terminator.true_block = program.blocks[4];
program.blocks[0]->AddBranch(program.blocks[1]);
program.blocks[0]->AddBranch(program.blocks[2]);
program.blocks[1]->AddBranch(program.blocks[3]);
@@ -12106,7 +12092,7 @@ void TestValuePhiValidation() {
program.blocks[3]->AddBranch(program.blocks[4]);
auto &condition = program.blocks[1]->AppendNewInst(ValueOpcode::IEqual32,
{Value(1u), Value(2u)});
program.block_info[2].condition = Value(&condition);
program.blocks[2]->condition = Value(&condition);
expect_invalid(program,
"non-dominating branch condition did not terminate IR "
"validation");
@@ -12123,19 +12109,22 @@ void TestWave32MaskProjection() {
for (uint32_t id = 0; id < 3; ++id) {
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.push_back({.id = id});
program.blocks.back()->id = id;
}
auto *entry = program.blocks[0];
auto *loop = program.blocks[1];
entry->AddBranch(loop);
loop->AddBranch(loop);
loop->AddBranch(program.blocks[2]);
program.block_info[0].terminator.kind = ShaderRecompiler::CFG::TerminatorKind::Branch;
program.block_info[0].terminator.true_block = 1;
program.block_info[1].terminator.kind = ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.block_info[1].terminator.true_block = 1;
program.block_info[1].terminator.false_block = 2;
program.block_info[2].terminator.kind = ShaderRecompiler::CFG::TerminatorKind::Return;
program.blocks[0]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Branch;
program.blocks[0]->terminator.true_block = program.blocks[1];
program.blocks[1]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::ConditionalBranch;
program.blocks[1]->terminator.true_block = program.blocks[1];
program.blocks[1]->terminator.false_block = program.blocks[2];
program.blocks[2]->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Return;
IREmitter e(entry);
const auto zero = U32(Value(0u));
const auto raw = e.GetUserData(static_cast<ScalarReg>(0));
@@ -12189,7 +12178,7 @@ void TestWave32MaskProjection() {
mask.AddPhiOperand(loop, remaining);
predicate.AddPhiOperand(entry, initial_predicate);
predicate.AddPhiOperand(loop, remaining_predicate);
program.block_info[1].condition = remaining_predicate;
program.blocks[1]->condition = remaining_predicate;
ConstantPropagationPass(program.blocks, wave_size);
if (wave_size == 32u) {
@@ -12238,7 +12227,6 @@ void TestU64ShiftConstantPropagation() {
Program shifts;
shifts.block_storage.push_back(std::make_unique<Block>());
shifts.blocks.push_back(shifts.block_storage.back().get());
shifts.block_info.emplace_back();
IREmitter shift_ir(shifts.blocks.front());
struct ShiftCase {
Value value;
@@ -12310,9 +12298,7 @@ void TestNativeWideValueValidation() {
Program program;
program.block_storage.push_back(std::make_unique<Block>());
program.blocks.push_back(program.block_storage.back().get());
program.block_info.emplace_back();
program.block_info.front().id = 0;
program.block_info.front().terminator.kind =
program.blocks.front()->terminator.kind =
ShaderRecompiler::CFG::TerminatorKind::Return;
return program;
};
@@ -12840,7 +12826,7 @@ void TestNewShaderRecompilerExpPixelOutputs() {
options.input_info.pixel = &default_info;
auto partial_result = RecompileForTest(partial_shader, options);
const auto partial_source = DisassembleSpirvBinary(partial_result.spirv);
Check(SpirvSourceHasInstructionOperand(partial_source, "OpBitcast",
Check(SpirvSourceHasInstructionOperand(partial_source, "OpCompositeConstruct",
"%uint_1065353216"),
"disabled float alpha export did not default to 1.0f bits");
CheckSpirvBinaryValidates(partial_result.spirv);
@@ -13242,8 +13228,9 @@ void TestLogicalAlphaBlendExport() {
const auto source = DisassembleSpirvBinary(guest.spirv);
Check(source.find("OpDecorate %out_mrt_1 Location 0") != std::string::npos &&
source.find("OpDecorate %out_mrt_1 Index 1") != std::string::npos &&
CountSourceOccurrences(source, "OpStore %out_mrt_0 ") == 1 &&
CountSourceOccurrences(source, "OpStore %out_mrt_1 ") == 1 &&
SpirvInstructionOpcodeCount(guest.spirv, 81u) == 8u,
!SpirvContainsVectorShuffle(guest.spirv, {3u, 3u, 3u, 3u}),
"guest dual-source export was replaced by synthetic alpha");
}
@@ -13317,72 +13304,55 @@ void TestNewShaderRecompilerNativeBindingPlan() {
options.user_data = user_data;
auto result = RecompileForTest(shader, options);
const auto *buffers = ShaderRecompiler::IR::FindBinding(
result.program.bindings, BindingKind::Buffers);
const ShaderRecompiler::IR::DescriptorBinding *sampled = nullptr;
const ShaderRecompiler::IR::DescriptorBinding *storage = nullptr;
const ShaderRecompiler::IR::DescriptorBinding *atomic_storage = nullptr;
const auto& counts = result.program.bindings.descriptor_counts;
std::optional<BindingKind> sampled;
std::optional<BindingKind> storage;
std::optional<BindingKind> atomic_storage;
for (const auto &image : result.program.info.images) {
const auto kind = ShaderRecompiler::IR::DescriptorBindingForImage(image);
if (!kind.has_value()) {
continue;
}
const auto *binding =
ShaderRecompiler::IR::FindBinding(result.program.bindings, *kind);
if (image.resource_class ==
ShaderRecompiler::IR::ImageResourceClass::Sampled) {
sampled = binding;
Check(kind.has_value() && image.descriptor_index == 0,
"native image did not receive its descriptor slot");
if (image.resource_class == ShaderRecompiler::IR::ImageResourceClass::Sampled) {
sampled = kind;
} else if (image.atomic) {
atomic_storage = binding;
atomic_storage = kind;
} else {
storage = binding;
storage = kind;
}
}
const auto *samplers = ShaderRecompiler::IR::FindBinding(
result.program.bindings, BindingKind::Samplers);
Check(buffers != nullptr && buffers->resources.size() == 2,
"native binding plan did not allocate scalar/vector buffers");
Check(sampled != nullptr && sampled->resources.size() == 1,
"native binding plan did not allocate the sampled image");
Check(storage != nullptr && storage->resources.size() == 1,
"native binding plan did not allocate the float storage image");
Check(atomic_storage != nullptr && atomic_storage->resources.size() == 1,
"native binding plan did not allocate the atomic storage image");
Check(samplers != nullptr && samplers->resources.size() == 1,
"native binding plan did not allocate the sampler");
Check(counts[static_cast<size_t>(BindingKind::Buffers)] == 2 &&
result.program.info.buffers[0].descriptor_index == 0 &&
result.program.info.buffers[1].descriptor_index == 1,
"native emission did not allocate scalar/vector buffer slots");
Check(sampled && counts[static_cast<size_t>(*sampled)] == 1,
"native emission did not allocate the sampled image");
Check(storage && counts[static_cast<size_t>(*storage)] == 1,
"native emission did not allocate the float storage image");
Check(atomic_storage && counts[static_cast<size_t>(*atomic_storage)] == 1,
"native emission did not allocate the atomic storage image");
Check(counts[static_cast<size_t>(BindingKind::Samplers)] == 1,
"native emission did not allocate the sampler");
Check(SpirvContainsOpcode(result.spirv, 86),
"SPIR-V binary does not combine separate image/sampler descriptors");
Check(SpirvHasDecorationValue(result.spirv, 33u,
ShaderRecompiler::IR::NativeBinding(
result.program.stage, buffers->kind)),
"SPIR-V lacks storage-buffer Binding decoration");
Check(SpirvHasDecorationValue(result.spirv, 33u,
ShaderRecompiler::IR::NativeBinding(
result.program.stage, sampled->kind)),
"SPIR-V lacks sampled-image Binding decoration");
Check(SpirvHasDecorationValue(result.spirv, 33u,
ShaderRecompiler::IR::NativeBinding(
result.program.stage, samplers->kind)),
"SPIR-V lacks sampler Binding decoration");
for (const auto kind : {BindingKind::Buffers, *sampled, BindingKind::Samplers}) {
Check(SpirvHasDecorationValue(result.spirv, 33u,
ShaderRecompiler::IR::NativeBinding(result.program.stage, kind)),
"SPIR-V lacks a resource Binding decoration");
}
Check(SpirvDecorationValueCount(result.spirv, 34u, 0u) ==
result.program.bindings.descriptors.size(),
static_cast<size_t>(std::popcount(result.program.bindings.descriptor_mask)),
"SPIR-V resources do not all use descriptor set zero");
CheckSpirvBinaryValidates(result.spirv);
#if KYTY_PLATFORM != KYTY_PLATFORM_WINDOWS
ExpectFatal(
[&] {
auto malformed = RecompileForTest(shader, options);
auto malformed_buffers = std::ranges::find_if(
malformed.program.bindings.descriptors, [](const auto &binding) {
return binding.kind == BindingKind::Buffers;
});
malformed_buffers->resources.clear();
(void)ShaderRecompiler::Spirv::EmitProgram(malformed.program,
options.input_info);
},
"malformed native binding topology did not terminate SPIR-V emission");
const auto bindings = result.program.bindings;
result.program.bindings = {};
result.program.info.buffers[0].descriptor_index = UINT32_MAX;
Check(ShaderRecompiler::Spirv::EmitProgram(result.program, options.input_info) == result.spirv &&
result.program.bindings == bindings &&
result.program.info.buffers[0].descriptor_index == 0,
"repeated emission did not rebuild descriptor slots deterministically");
#if KYTY_PLATFORM != KYTY_PLATFORM_WINDOWS
ShaderRecompiler::IR::Inst *buffer_handle = nullptr;
for (auto *block : result.program.blocks) {
const auto found = std::ranges::find_if(*block, [](const auto &inst) {
@@ -13764,7 +13734,7 @@ void TestTypedDescriptorRealCarryAndScalarLoads() {
ShaderRecompiler::IR::SrtRuntime carry_runtime{carry_user_data, shader_base,
nullptr, nullptr};
ShaderRecompiler::IR::DescriptorValue carry_value;
Check(ShaderRecompiler::IR::SrtWalker(carry_ir, carry_runtime).EvaluateDescriptor(carry_source_index, carry_value) &&
Check(ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(carry_ir), carry_runtime).EvaluateDescriptor(carry_source_index, carry_value) &&
carry_value.dwords[0] == static_cast<uint32_t>(expected_pc) &&
carry_value.dwords[1] == static_cast<uint32_t>(expected_pc >> 32u),
"S_GETPC shader-base or add/addc carry evaluation was incorrect");
@@ -13834,7 +13804,7 @@ void TestTypedDescriptorRealCarryAndScalarLoads() {
TypedDescriptorSource(inline_sampler_ir,
inline_sampler_ir.info.samplers[0].source) !=
nullptr &&
ShaderRecompiler::IR::SrtWalker(inline_sampler_ir, runtime).EvaluateDescriptor(inline_sampler_ir.info.samplers[0].source, sampler) &&
ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(inline_sampler_ir), runtime).EvaluateDescriptor(inline_sampler_ir.info.samplers[0].source, sampler) &&
sampler.dwords[0] == 0 && sampler.dwords[1] == 0x00fff000u &&
sampler.dwords[2] == 0x09500000u && sampler.dwords[3] == 0,
"real inline sampler construction was unresolved or evaluated "
@@ -13870,7 +13840,7 @@ void TestSrtWalkerRealSmemTranslation() {
std::vector<uint32_t> flat;
const ShaderRecompiler::IR::SrtRuntime runtime{user_data, 0, ReadSrtHostDword,
nullptr};
const auto walked = ShaderRecompiler::IR::SrtWalker(ir, runtime).RefreshFlatBuffer(flat);
const auto walked = ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(ir), runtime).RefreshFlatBuffer(flat);
Check(walked, "SRT walk failed");
Check(flat.size() == table.size() &&
std::equal(flat.begin(), flat.end(), table.begin()),
@@ -13899,7 +13869,7 @@ void TestSrtWalkerVccBaseTranslation() {
const ShaderRecompiler::IR::SrtRuntime runtime{user_data, 0, ReadSrtHostDword,
nullptr};
std::vector<uint32_t> flat;
Check(ShaderRecompiler::IR::SrtWalker(ir, runtime).RefreshFlatBuffer(flat), "SRT walk failed");
Check(ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(ir), runtime).RefreshFlatBuffer(flat), "SRT walk failed");
Check(flat.size() == table.size() &&
std::equal(flat.begin(), flat.end(), table.begin()),
"typed SSA lost an SMEM base copied through VCC");
@@ -13927,14 +13897,15 @@ void TestSrtWalkerRealSBufferTranslation() {
std::vector<uint32_t> flat;
const ShaderRecompiler::IR::SrtRuntime runtime{user_data, 0, ReadSrtHostDword,
nullptr};
const auto walked = ShaderRecompiler::IR::SrtWalker(ir, runtime).RefreshFlatBuffer(flat);
const auto plan = ShaderRecompiler::IR::ExtractResourcePlan(ir);
const auto walked = ShaderRecompiler::IR::SrtWalker(plan, runtime).RefreshFlatBuffer(flat);
Check(walked, "SRT walk failed");
Check(flat.size() == 4 &&
std::equal(flat.begin(), flat.end(), table.begin()),
"real S_BUFFER_LOAD walk did not align offset components independently");
user_data[10] = 3 * sizeof(uint32_t);
const auto bounds_walked = ShaderRecompiler::IR::SrtWalker(ir, runtime).RefreshFlatBuffer(flat);
const auto bounds_walked = ShaderRecompiler::IR::SrtWalker(plan, runtime).RefreshFlatBuffer(flat);
Check(!bounds_walked, "real S_BUFFER_LOAD walk ignored descriptor bounds");
CheckFlattenedReadSlots(
ir, 4, "real S_BUFFER_LOAD patch used the wrong flat offsets");
@@ -13949,7 +13920,7 @@ void TestSrtWalkerRealSBufferTranslation() {
static_cast<uint32_t>(std::size(negative_shader)),
negative_ir);
user_data[10] = sizeof(table);
Check(!ShaderRecompiler::IR::SrtWalker(negative_ir, runtime).RefreshFlatBuffer(flat),
Check(!ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(negative_ir), runtime).RefreshFlatBuffer(flat),
"real S_BUFFER_LOAD walk accepted a negative immediate");
}
@@ -13988,7 +13959,7 @@ void TestScalarMemorySourcesCapturedBeforeWrites() {
const ShaderRecompiler::IR::SrtRuntime runtime{
user_data, 0, ReadSrtHostRangeDword, &range};
std::vector<uint32_t> flat;
Check(ShaderRecompiler::IR::SrtWalker(ir, runtime).RefreshFlatBuffer(flat), "SRT walk failed");
Check(ShaderRecompiler::IR::SrtWalker(ShaderRecompiler::IR::ExtractResourcePlan(ir), runtime).RefreshFlatBuffer(flat), "SRT walk failed");
Check(flat.size() == table.size() &&
std::equal(flat.begin(), flat.end(), table.begin()),
"overlapping scalar-memory load did not capture its sources before "
@@ -14499,12 +14470,32 @@ void TestGraphicsPushConstantPlacement() {
RecompileForTest(shader, options, nullptr, nullptr, PushDataStart);
Check(placed.program.bindings.UsesPushData() &&
placed.program.bindings.push_data_start_dword == PushDataStart &&
ShaderRecompiler::IR::FindBinding(placed.program.bindings,
BindingKind::ShaderData) == nullptr,
placed.program.bindings.descriptor_counts[static_cast<size_t>(BindingKind::ShaderData)] == 0,
"pixel shader did not use its assigned shared push-data position");
Check(SpirvHasMemberDecorationValue(placed.spirv, OffsetDecoration, 0),
"SPIR-V push-data block did not use the canonical zero base");
CheckSpirvBinaryValidates(placed.spirv);
auto exact = RecompileForTest(shader, options, nullptr, nullptr, 28);
Check(exact.program.bindings.UsesPushData() &&
exact.program.bindings.push_data_start_dword == 28 &&
exact.program.bindings.ShaderDataDwords() == 4,
"exact-fit shader data did not use the remaining push space");
uint32_t cursor = 28;
exact.program.bindings.AdvancePushData(cursor);
Check(cursor == 32, "exact-fit shader data advanced the cursor incorrectly");
CheckSpirvBinaryValidates(exact.spirv);
auto spill = RecompileForTest(shader, options, nullptr, nullptr, 29);
auto full = RecompileForTest(shader, options, nullptr, nullptr, 32);
Check(!spill.program.bindings.UsesPushData() &&
spill.program.bindings.descriptor_counts[static_cast<size_t>(BindingKind::ShaderData)] == 1 &&
spill.program.bindings == full.program.bindings && spill.spirv == full.spirv,
"insufficient push space did not produce canonical storage-backed data");
cursor = 29;
spill.program.bindings.AdvancePushData(cursor);
Check(cursor == 29, "spilled shader data consumed push space");
CheckSpirvBinaryValidates(spill.spirv);
}
void TestNewShaderRecompilerUnsupportedMemoryDecode() {
@@ -14689,17 +14680,44 @@ void TestCompilerStageInputOwnership() {
#endif
}
void TestSpirvEmissionOwnsRequirements() {
void TestSpirvRepeatedEmission() {
using namespace ShaderRecompiler;
const uint32_t shader[] = {EncodeSopp(0x01)};
const auto options = MakeCompileOptions(ShaderType::Compute);
ShaderComputeInputInfo compute{};
compute.threads_num[0] = 64;
compute.threads_num[1] = compute.threads_num[2] = 1;
auto options = MakeCompileOptions(ShaderType::Compute);
options.wave_size = 64;
options.input_info.compute = &compute;
auto compiled = RecompileForTest(shader, options);
IR::IREmitter emitter(compiled.program.blocks.front());
emitter.Emit(IR::ValueOpcode::LaneId, {});
const auto lane = emitter.Emit(IR::ValueOpcode::LaneId, {});
const auto value = emitter.Emit(IR::ValueOpcode::IAdd32, {lane, IR::Value(1u)});
const auto first = emitter.Emit(IR::ValueOpcode::ReadFirstLane,
{value, IR::Value(true)});
emitter.Emit(IR::ValueOpcode::IAdd32, {first, lane});
compiled.program.shader_info_complete = false;
IR::CollectShaderInfo(compiled.program, options.input_info);
const auto binary = Spirv::EmitProgram(compiled.program, options.input_info);
CheckSpirvBinaryValidates(binary);
Check((DisassembleSpirvBinary(binary).find("BuiltIn SubgroupLocalInvocationId") != std::string::npos),
"emission reused requirements from an earlier IR version");
"collection missed the subgroup requirement of the added IR");
Check(Spirv::EmitProgram(compiled.program, options.input_info) == binary,
"repeated emission reused IDs from the previous module");
compute.host_subgroup_size = 32;
const auto split = Spirv::EmitProgram(compiled.program, options.input_info);
CheckSpirvBinaryValidates(split);
Check(split != binary &&
Spirv::EmitProgram(compiled.program, options.input_info) == split,
"split-wave emission did not reset both halves' definitions");
compute.host_subgroup_size = 64;
Check(Spirv::EmitProgram(compiled.program, options.input_info) == binary,
"native-wave re-emission retained split-wave definitions");
compute.host_subgroup_size = 32;
Check(Spirv::EmitProgram(compiled.program, options.input_info) == split,
"split-wave re-emission retained native-wave definitions");
}
void TestRepeatedExportsHaveOneInterface() {
@@ -14756,9 +14774,7 @@ void TestNewShaderRecompilerSpirvSizeBaselines() {
const auto dead_gds_result =
compile("dead-gds", dead_gds,
{.words = 64, .instructions = 20, .labels = 4, .branches = 3});
Check(ShaderRecompiler::IR::FindBinding(
dead_gds_result.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::Gds) == nullptr,
Check(dead_gds_result.program.bindings.descriptor_counts[static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::Gds)] == 0,
"dead GDS load retained a descriptor binding");
const uint32_t structured_phi[] = {
@@ -14899,9 +14915,7 @@ void TestNewShaderRecompilerSpirvSizeBaselines() {
wide_gds_metrics.workgroup_variables == 0u &&
wide_gds_metrics.array_lengths == 1u,
"native wide GDS fixture did not share one entry-dominating length");
Check(ShaderRecompiler::IR::FindBinding(
wide_gds_result.program.bindings,
ShaderRecompiler::IR::DescriptorBindingKind::Gds) != nullptr,
Check(wide_gds_result.program.bindings.descriptor_counts[static_cast<size_t>(ShaderRecompiler::IR::DescriptorBindingKind::Gds)] != 0,
"live native wide GDS access did not allocate its descriptor binding");
const auto count_shared = [](const auto &result,
ShaderRecompiler::IR::ValueOpcode opcode) {
@@ -14995,11 +15009,11 @@ int main() {
TestShaderBufferResourceSize();
TestNativeShaderResourceDependencies();
TestNormalizedImageContracts();
TestSpirvRequirementsAnalysis();
TestShaderFeatureCollection();
TestTypedSpirvSerialization();
TestDeferredSpirvPhiPatching();
TestCompilerStageInputOwnership();
TestSpirvEmissionOwnsRequirements();
TestSpirvRepeatedEmission();
TestRepeatedExportsHaveOneInterface();
TestNewShaderRecompilerSpirvSizeBaselines();
TestDemandDrivenSpirvDeclarations();