ADD-ON ReShade 6.8.0 CRASHES w/ OptiScaler DLSS-G (RDR2 DX12); Standard DOESN'T

  • AlgonquinSasquatch
  • Topic Author
More
1 week 2 days ago - 1 week 2 days ago #1 by AlgonquinSasquatch ADD-ON ReShade 6.8.0 CRASHES w/ OptiScaler DLSS-G (RDR2 DX12); Standard DOESN'T was created by AlgonquinSasquatch
Full add-on ReShade 6.8.0 causes reproducible D3D12 access violation with OptiScaler DLSS-G; standard ReShade worksSummaryI have encountered a consistently reproducible crash in Red Dead Redemption 2 (DirectX 12) when combining OptiScaler's native NVIDIA DLSS Frame Generation output with ReShade 6.8.0 (full add-on support).The game starts normally, but crashes while loading a story-mode save, immediately after the first DLSS evaluation begins.After extensive testing, I discovered that replacing only the full-add-on ReShade DLL with the standard ReShade DLL eliminates the crash entirely, without changing any OptiScaler settings, NVIDIA components, or graphics settings.The failing ReShade binary produces a repeatable
Code:
0xC0000005
access violation at offset
Code:
+0x13131B
. Two separate crash dumps identify the same faulting instruction.Crucially, the crash occurs even when no RenoDX or other external ReShade add-ons are loaded.System and software configurationComponentDetailsGameRed Dead Redemption 2, version
Code:
1.0.1491.50
Graphics APIDirectX 12Operating systemWindows 11, build
Code:
10.0.26300
GPUNVIDIA GeForce RTX 5070 Ti, 16 GBNVIDIA driver
Code:
621.14
Display resolution2560 × 1440OptiScaler
Code:
10.0.0-dev (2fff542e)
, build
Code:
20261112_154934
NVIDIA DLSS upscaler
Code:
310.9.16
NVIDIA Streamline
Code:
2.14.16.b742680bf
Streamline host SDK
Code:
2.11.1
Crashing ReShade
Code:
6.8.0.2155
, full add-on supportWorking ReShade
Code:
6.8.0.2158
, standard editionOptiScaler is installed as
Code:
winmm.dll
, and ReShade is loaded through OptiScaler's built-in integration.The following configuration is identical between the final crashing and successful runs:
Code:
[Upscalers] Dx12Upscaler = dlss [FrameGen] Enabled = true FGInput = Upscaler FGOutput = dlssg FGNvngxReplacement = None [Hotfix] CreateD3D12DeviceForLuma = false [Plugins] LoadReShade = true
The ReShade DLL filename is
Code:
d3d12-ReShade64.dll
in both runs.Exact crash informationThe latest failing run generated the crash dump:
Code:
OptiScaler_crash_C0000005_20261112_234617.dmp
Analysis of this dump indicates:
  • Exception:
    Code:
    0xC0000005
    — access violation
  • Faulting module:
    Code:
    d3d12-ReShade64.dll
  • Faulting offset:
    Code:
    +0x13131B
  • Faulting instruction:
    Code:
    add rbx, qword ptr [rcx+0x18]
  • RCX register:
    Code:
    0x0000000000000000
  • Attempted memory read:
    Code:
    0x0000000000000018
  • Call-stack context: NVIDIA
    Code:
    nvngx_dlss.dll
    ,
    Code:
    _nvngx.dll
    , and OptiScaler's
    Code:
    winmm.dll
    are involved in the crashing thread's execution path.
This is a null-pointer dereference within ReShade's loaded module.An earlier independent crash dump,
Code:
OptiScaler_crash_C0000005_20261112_233434.dmp
, also identifies exactly the same ReShade offset (
Code:
[b]+0x13131B[/b]
) and faulting instruction, although the register state differs.The crash is therefore highly reproducible at the instruction level.This establishes where the exception occurs, but does not conclusively establish which component originally supplied or corrupted the invalid pointer.What happens immediately before the crashThe last relevant OptiScaler log entries from the final failing run are:
Code:
[23:46:17.880043] DLSSFeatureDx12::InitDLSS _CreateFeature result: NVSDK_NGX_Result_Success [23:46:17.880118] DLSSFeature::ReadVersion DLSS v310.9.16 loaded. [23:46:17.897519] NVSDK_NGX_D3D12_EvaluateFeature [23:46:17.897555] DLSSG_Dx12::CreateContext [23:46:17.899789] DLSSG_Dx12::EvaluateState FGchanged [23:46:17.900531] DLSSFeature::ProcessEvaluateParams Render Size: 1706x960, Target Size: 2560x1440, Display Size: 2560x1440
The game subsequently crashes.No successful
Code:
[b]DLSSG_Dx12::Dispatch Result: Ok[/b]
entries occur in the failing run.NVIDIA DLSS feature creation succeeds, and Streamline initializes successfully before the crash. The failure occurs when DLSS evaluation begins.Decisive A/B comparisonI then replaced the full-add-on ReShade DLL with the standard ReShade 6.8.0 DLL, retaining the same injection filename and every other configuration setting.TestFull add-on ReShadeStandard ReShadeInternal version
Code:
6.8.0.2155
Code:
6.8.0.2158
OptiScalerSame buildSame buildOptiScaler INIIdenticalIdenticalNVIDIA DLSS version
Code:
310.9.16
Code:
310.9.16
NVIDIA Streamline
Code:
2.14.16
Code:
2.14.16
Native NVIDIA DLSS-GEnabledEnabledReShade injection filename
Code:
d3d12-ReShade64.dll
SameExternal RenoDX addonNot loadedNot loadedStory save loadingCrashesSuccessfulSuccessful DLSS-G dispatches03,616The successful run's OptiScaler log records:
Code:
[00:00:43.847123] DLSSG_Dx12::Dispatch Result: Ok ... [00:01:11.293143] DLSSG_Dx12::Dispatch Result: Ok
There are 3,616 successful DLSS-G dispatch results across this interval, followed by normal game shutdown and OptiScaler unloading.This verifies that the standard ReShade build is not merely allowing the game to launch: genuine NVIDIA DLSS-G is operating successfully alongside ReShade.The log also confirms that DLSS
Code:
310.9.16
, previously suspected as a possible contributor, works in this configuration.Important caveat: Since the two ReShade DLLs also have different internal build numbers (
Code:
2155
versus
Code:
2158
), this comparison establishes that the binaries behave differently, but does not yet prove that the full-add-on compilation option alone causes the defect.Additional experiments already performedThe following tests were conducted with the full-add-on build:
  1. Exclusive fullscreen versus borderless: Both crash. OptiScaler confirms borderless operation through
    Code:
    SetFullscreenState Fullscreen: 0
    .
  2. ReShade loaded as
    Code:
    [b]dxgi.dll[/b]
    ,
    Code:
    [b]ReShade64.dll[/b]
    , or
    Code:
    [b]d3d12-ReShade64.dll[/b]
    : Crashes persist. The OptiScaler-mediated ReShade filenames are confirmed in the corresponding logs.
  3. ReShade D3D12 initialization workaround: Setting
    Code:
    CreateD3D12DeviceForLuma=true
    successfully creates the D3D12 device before loading ReShade, but does not prevent the crash.
  4. RenoDX removed: The same crash occurs without the RenoDX DLSS neural-rendering add-on loaded. The crash dump does not list its module.
  5. Frame generation disabled: RDR2 and ReShade work when NVIDIA DLSS-G output is disabled.
  6. Intel XeFG substituted for DLSS-G: RDR2 loads and runs with ReShade present. One successful run records 2,693 successful XeFG dispatches.
  7. OptiScaler with NVIDIA DLSS-G but without ReShade: Previously tested successfully over extended gameplay.
These results substantially narrow the fault to an interaction involving ReShade's implementation, OptiScaler, and the NVIDIA DLSS-G/NGX execution path.Suspected area of incompatibilityThe evidence points toward a D3D12 object, resource, or descriptor-handling incompatibility in the ReShade/OptiScaler/NVIDIA NGX interaction.The specific pointer dereference at ReShade offset
Code:
+0x13131B
is the strongest lead.However, I cannot determine from the minidumps alone whether the invalid pointer originates inside ReShade, is passed by OptiScaler/NVIDIA NGX, or results from an earlier interaction between their D3D12 hooks.The full-add-on build's additional interception capabilities are a possible contributing factor, but that remains a hypothesis.Steps to reproduce
  1. Install RDR2 and configure it for DirectX 12 with NVIDIA DLSS upscaling.
  2. Install OptiScaler
    Code:
    10.0.0-dev (2fff542e)
    as
    Code:
    winmm.dll
    .
  3. Configure
    Code:
    FGInput=Upscaler
    ,
    Code:
    FGOutput=dlssg
    ,
    Code:
    FGNvngxReplacement=None
    , and
    Code:
    LoadReShade=true
    .
  4. Install ReShade 6.8.0 with full add-on support, without installing any external ReShade add-ons.
  5. Rename its DLL to
    Code:
    d3d12-ReShade64.dll
    and place it alongside OptiScaler.
  6. Launch RDR2 and attempt to load a story-mode save.
  7. Observe the access violation during the first DLSS evaluation.
  8. Exit the game and replace only the ReShade DLL with the standard ReShade 6.8.0 build, retaining the same filename.
  9. Launch the same save again. In my environment, it now loads and native NVIDIA DLSS-G dispatches successfully.
Expected versus actual behaviorExpected: Full-add-on ReShade should coexist with OptiScaler's native NVIDIA DLSS-G output at least as reliably as standard ReShade when no external add-ons are loaded.Actual: The tested full-add-on ReShade build consistently crashes at the same instruction during DLSS evaluation, while the standard build works.Questions for maintainersI would particularly appreciate assistance with the following:
  • Can the instruction at ReShade
    Code:
    +0x13131B
    in build
    Code:
    6.8.0.2155
    be mapped to its corresponding source function?
  • Does the full-add-on build alter D3D12 resource wrapping, descriptor handling, or NVIDIA NGX interaction in a manner that could explain this null-pointer dereference?
  • Is this a known issue with OptiScaler's Streamline-based DLSS-G implementation and the full-add-on ReShade build?
  • Has a subsequent ReShade D3D12 fix addressed this code path?
  • Is there an updated full-add-on build or recommended integration approach that retains unrestricted add-on functionality without triggering the crash?
Available diagnostic filesCrashing run:
  • Code:
    OptiScaler(20261112-224646).ini
  • Code:
    OptiScaler(20261112-224646).log
  • Code:
    ReShade.log1
  • Code:
    OptiScaler_crash_C0000005_20261112_234617.dmp
Additional independent crash:
  • Code:
    OptiScaler(20261112-223512).ini
  • Code:
    OptiScaler(20261112-223514).log
  • Code:
    OptiScaler_crash_C0000005_20261112_233434.dmp
Successful comparison:
  • Code:
    OptiScaler(20261112-230158).ini
  • Code:
    OptiScaler(20261112-230204).log
  • Code:
    ReShade(1).log1
The crash dumps are approximately 198 MB each and can be provided privately if required.Thank you for looking into this. 
Last edit: 1 week 2 days ago by AlgonquinSasquatch. Reason: FULL DEV REPORT

Please Log in or Create an account to join the conversation.

  • PhoebeG93
More
5 days 2 hours ago #2 by PhoebeG93 Replied by PhoebeG93 on topic ADD-ON ReShade 6.8.0 CRASHES w/ OptiScaler DLSS-G (RDR2 DX12); Standard DOESN'T
Same crash here - root cause found, plus a working workaround for 6.8.0.2155 (add-on build)

I hit exactly this crash and traced it through four crash dumps, the ReShade 6.8.0 source and the OptiScaler source. Short version: the add-on build decodes every CPU descriptor handle passed to ClearUnorderedAccessViewUint/Float as if it were a ReShade-encoded handle, with no validity check. In OptiScaler's DLSS-G mode, NVIDIA NGX ends up passing handles from heaps ReShade never saw, so ReShade reads a garbage heap entry and crashes. A small validity check fixes it: with a patched DLL, the story save that crashed every time now loads, and DLSS-G 3x runs alongside ReShade and its add-ons.

Setup
  • Red Dead Redemption 2 1.0.1491.50, DX12, 2560x1440 HDR, RTX 5080, driver 621.24, Windows 11
  • OptiScaler 10.0.0-dev (1e568bb3, nightly 20261116) as winmm.dll: Dx12Upscaler=dlss, FGInput=upscaler, FGOutput=dlssg, FGNvngxReplacement=None, [DLSSG] InterpolationCount=2, LoadReshade=true
  • Streamline 2.14.16, DLSS 310.9.16
  • ReShade 6.8.0.2155 with full add-on support, loaded by OptiScaler as d3d12-ReShade64.dll (also crashed as d3d12.dll and as an .asi). Add-ons: RealityFX and RenoDX DLSS5. Three of my four crashes happened before the RenoDX add-on was installed, so it isn't needed to trigger this.

What the dumps show
All four dumps are 0xC0000005 in the ReShade DLL at RVA 0x1311DB, or 0x13131B for its sibling (image base 0x180000000). The instruction is the same one you found: add rbx, [rcx+0x18]. Unwound with the on-disk images, the stack is:
Code:
ReShade  D3D12GraphicsCommandList::ClearUnorderedAccessViewUint / ...Float   <- fault NGX DLSS (nvngx_dlss.dll / driver model 160_E658700.bin) _nvngx.dll  NVSDK_NGX_D3D12_EvaluateFeature OptiScaler (winmm.dll) RDR2.exe    NVSDK_NGX_D3D12_EvaluateFeature_C   <- game's DLSS upscaler evaluate
The two functions are vtable slots 49 and 50 of ReShade's command list wrapper (vtable RVA 0x435B30): ClearUnorderedAccessViewUint at 0x131150 and ClearUnorderedAccessViewFloat at 0x131290. At the fault, r8 (the ViewCPUHandle) held values like 0x1850D410C0, 0x1817E94260, 0x18391CF1C0 and 0x1859D48120. Those look like real driver CPU descriptor addresses, not ReShade-encoded handles. Decoded as ReShade handles they give heap index ~0x181-0x185, which points at a garbage or NULL slot (your rcx=0 is the NULL case).

Root cause, ReShade side
In d3d12_command_list.cpp, both Clear UAV hooks do this under RESHADE_ADDON >= 2:
Code:
ViewCPUHandle = _device->convert_to_original_cpu_descriptor_handle(ViewCPUHandle);
In d3d12_impl_device.cpp the decode is:
Code:
heap_index = (handle.ptr >> heap_index_start) & 0xFFFFFFF;   // heap_index_start = 28 on x64 assert(heap_index < _descriptor_heaps.size() && _descriptor_heaps[heap_index] != nullptr);   // compiled out in release return { _descriptor_heaps[heap_index]->_orig_base_cpu_handle.ptr + (handle.ptr & (((1ull << heap_index_start) - 1) ^ 0x7)) };
_descriptor_heaps is a concurrent_vector, and its operator[] does no bounds check. _orig_base_cpu_handle sits at heap+0x18, which is the faulting [rcx+0x18]. Only CBV_SRV_UAV and SAMPLER heaps created through ReShade's wrapped device get encoded handles of the form (heapIndex << 28) | type | (shaderVisible << 2). Any handle from a heap created on an unwrapped device gets decoded as garbage. No ini option controls this; it depends only on the RESHADE_ADDON compile setting. That explains why the standard build works. Current main has the same code.

Why only with OptiScaler DLSS-G (not XeFG, not without ReShade)
In DLSS-G mode, OptiScaler starts Streamline with the native device: it deliberately unwraps ReShade's proxy (Util::GetReShadeOriginal in StreamlineProxy::SetD3D12Device) before slSetD3DDevice. sl.common then initialises NGX on that native device. My OptiScaler log shows "Init_Ext ... Device: 4A64280" from sl.common, and later the game's own init with ReShade's proxy device "C56FCD60". NGX most likely creates its internal descriptor heaps on the first (native) device. Meanwhile the game's DLSS evaluate still passes the game's ReShade-wrapped command list, so NGX's internal ClearUnorderedAccessView calls carry native handles straight into ReShade's hook.
With XeFG output, Streamline is never started, so NGX only ever sees ReShade's device, its heaps get encoded and nothing breaks. NGX is closed source, so the "NGX builds its heaps on the native device" part is inferred from the handle values and the logs, but everything else is visible in source.

Suggested fix for ReShade (sketch, not compiled)
Validate before decoding, and pass unknown handles through unchanged:
Code:
D3D12_CPU_DESCRIPTOR_HANDLE device_impl::convert_to_original_cpu_descriptor_handle(D3D12_CPU_DESCRIPTOR_HANDLE handle) const {     constexpr SIZE_T offset_mask = ((1ull << heap_index_start) - 1) ^ 0x7;     const size_t heap_index = (handle.ptr >> heap_index_start) & 0xFFFFFFF;     if (heap_index < _descriptor_heaps.size())         if (D3D12DescriptorHeap *const heap = _descriptor_heaps[heap_index];             heap != nullptr && (handle.ptr & ~offset_mask) == heap->_internal_base_cpu_handle.ptr)             return { heap->_orig_base_cpu_handle.ptr + (handle.ptr & offset_mask) };     return handle; // not from a ReShade-wrapped heap (e.g. created on the unwrapped device) }
Optionally, also check that the offset is below NumDescriptors * increment, and skip add-on events for handles that aren't recognised. The same unchecked decode exists in the device hooks: Create{ConstantBuffer,ShaderResource,UnorderedAccess}View, CreateSampler(2), CopyDescriptors(Simple) and the newer Device methods. D3D12GraphicsCommandList::SetDescriptorHeaps also static_casts every heap to D3D12DescriptorHeap* and reads ->_orig, so a native heap passed there is unsafe too.

Workaround: byte patch for 6.8.0.2155 add-on build only
This only applies to the exact file with sha256 0cee63f9c9f13f3ac909c5b4903f4dbb4b719a7ab3b4f13b0deaf83c814b94f7 (5,592,064 bytes). After the patch: 7c23717d876ad2a47e0eb938355a4678e2c9867d234ded522cd725625a0adf4d.
It does the check above in both Clear UAV hooks: index < vector size, segment allocated, entry non-NULL, and the type/flag bits match the heap's encoded base. Otherwise it passes the handle through. It also makes SetDescriptorHeaps unwrap only objects whose vtable is ReShade's heap wrapper. The code caves sit in zero slack at the end of .text. No relocations are touched and no branch targets fall inside the patched ranges. File offsets (= RVA - 0x1000 + 0x400 for .text):
Code:
0x000228  .text VirtualSize  7C 78 3D 00 -> 00 7A 3D 00 0x1305B9  ClearUAVUint      48 8B 8E A8 02 00 00 -> E9 C2 76 2A 00 90 90 0x1306F9  ClearUAVFloat     48 8B 8E A8 02 00 00 -> E9 F2 75 2A 00 90 90 0x12EC2A  SetDescriptorHeaps 48 8B 48 08 48 89 0C D3 -> E9 41 91 2A 00 90 90 90 0x3D7C80  cave A (98 bytes, was zeros): 48 8B 8E A8 02 00 00 48 3B AE A0 02 00 00 73 38 48 83 E2 FE 48 29 D5 40 30 F6 48 8B 04 C1 48 83 F8 3F 76 27 48 8B 0C E8 48 85 C9 74 1E 48 8B 41 28 4C 31 C0 48 BA 07 00 00 F0 FF FF FF FF 48 85 D0 75 08 E9 0A 89 D5 FF 40 30 F6 4C 89 C3 48 8D 0D ED FF FF FF E9 F8 88 D5 FF 00 00 00 00 00 00 00 00 0x3D7CF0  cave B (98 bytes, was zeros): 48 8B 8E A8 02 00 00 48 3B AE A0 02 00 00 73 38 48 83 E2 FE 48 29 D5 40 30 F6 48 8B 04 C1 48 83 F8 3F 76 27 48 8B 0C E8 48 85 C9 74 1E 48 8B 41 28 4C 31 C0 48 BA 07 00 00 F0 FF FF FF FF 48 85 D0 75 08 E9 DA 89 D5 FF 40 30 F6 4C 89 C3 48 8D 0D ED FF FF FF E9 C8 89 D5 FF 00 00 00 00 00 00 00 00 0x3D7D70  cave C (30 bytes, was zeros): 48 8D 0D 19 D7 05 00 48 39 08 75 06 48 8B 48 08 EB 03 48 89 C1 48 89 0C D3 E9 A4 6E D5 FF
Cave A disassembled (cave B is the same, resuming in the Float function at 0x131312):
Code:
mov rcx,[rsi+0x2A8]        ; segment table cmp rbp,[rsi+0x2A0]        ; heap index < vector size? jae pass and rdx,-2 / sub rbp,rdx / xor sil,sil   ; original instructions mov rax,[rcx+rax*8]        ; segment cmp rax,0x3F / jbe pass2   ; segment allocated? mov rcx,[rax+rbp*8]        ; heap entry test rcx,rcx / je pass2 mov rax,[rcx+0x28] / xor rax,r8 / mov rdx,0xFFFFFFFFF0000007 / test rax,rdx / jne pass2 jmp 0x1311D2               ; valid: original decode continues pass:  xor sil,sil pass2: mov rbx,r8          ; pass the native handle through        lea rcx,[zero_qword-0x18]   ; so the original "add rbx,[rcx+0x18]" adds 0        jmp 0x1311D2
I'm not attaching a modified DLL. The bytes above let anyone reproduce and check it.

Result
With the patched DLL, the story save that crashed on the first DLSS frame every time loads normally. DLSS-G 3x runs with ReShade, the Reality Redemption preset, RealityFX and RenoDX DLSS5 all active, and I played until I quit without a crash. That's one session so far.

Caveats
  • This is an unofficial binary patch for one specific build; a proper source fix is the real solution.
  • Add-ons now get the raw native handle in the clear_unordered_access_view events instead of a resolvable view. Mine didn't mind.
  • The device-level decode sites aren't patched. They weren't on the crash path, because NGX uses the native device for those calls.
  • A native handle whose upper bits happen to match a live heap index and type would still be misread, causing a wrong clear rather than a crash. This is very unlikely with real driver addresses.

Possible fix on the OptiScaler side (for reference)
Either initialise Streamline/NGX with the same device the game's NGX calls use, or unwrap the command list (Util::GetReShadeOriginal) before forwarding EvaluateFeature to NGX when NGX was initialised on the native device. Then NGX's internal commands never pass through ReShade's wrapper.
The following user(s) said Thank You: AlgonquinSasquatch

Please Log in or Create an account to join the conversation.