Possible calling convention issue with ID3D11Blob (vkd3d) on linux x64 using dxvk

[txt231]

Summary

Having an issue running the d3d sample using dxvk.

The main issue stems from ID3D10Blob returning garbage.
From investigating it looks like the virtual functions used in ID3D10Blob from vkd3d uses microsoft calling conventon, while im presuming it gets called using a linux one.

NOTE: images are not the same process instance. If it was it should have returned, 0x5600192d8970 for data ptr and 0x258 for size.

Steps to reproduce

• Platform: Desktop Linux(arch) x86_64
• Framework Version: net7.0, Not sure if rider uses mono or dotnet to compile.
• API: D3D11(DXVK), GLFW window
• API Version: D3D_FEATURE_LEVEL_11_0

1. Copied the code from the d3d11 sample into a rider c# project.
2. install nuget libraries (for dxvk with GLFW window)
3. run code

Comments

Issue isnt techinically directly in this project as its in vkd3d this project uses, specfiically with STDMETHODCALLTYPE

[Perksey]

Thank you for reporting this!

I discussed this with @tannergooding from the .NET team, and it seems that there is an expectation that if you're running on Linux you're using cdecl and not stdcall. As a result of this, the .NET runtime has made no attempt to support this on Linux. In addition, it appears that vkd3d is primarily geared towards WINE usage, which means we have to take special care to use it outside of WINE.

So, our options:

1. Remove our vkd3d support and mandate that users wanting to use D3D11 on Linux use WINE as intended by Valve
2. Roll a custom build of the binaries that use Cdecl. This isn't future proof as our APIs are declared to use stdcall, and support for this could one day light up in the runtime and break everything.
3. Implement native trampolines to bridge the calling conventions. This has the same problem as option 2.

Thoughts @Beyley?

[tannergooding]

This isn't future proof as our APIs are declared to use stdcall, and support for this could one day light up in the runtime and break everything.

This is unlikely because it'd break existing code targeting Stdcall. If Linux support ever came, it'd likely be via a new calling convention (much as happend to cause CallConvMemberFunction to exist in addition to CallConvThisCall)

[tannergooding]

Notably the ask for such runtime support is dotnet/runtime#37548.

[txt231]

So, our options:

my ideas when writing the issue was:

1. force overriding macros using compiler flags, if that is even possible.
2. patch files using existing build script. Would probably allow for easier usage of vkd3d repo untill breaking changes happen. Breaking changes would probably be unlikely, but that depends on how the patch itself is implemented.
3. vkd3d submodule this project is using is already owned by a maintainer. so modifying it seems like an option, but might come at the cost of more matenance and annoyances when updating.
4. similar to option 3. Add an ifdef to vkd3d to disable usage of ms_abi, and try to get it upstreamed for easier maintability. Enable/disable with compiler flags.

Im generally unfamiliar with management of projects, so the ideas might seem unreasonable.

---

would be sad if support is removed as the reason i had a peek at this project is the "advertised" dxvk support.

[Perksey]

Yeah I'm personally on team "add our own builds that override the STDMETHODCALLTYPE macro" but this is Beyley's area

[Beyley]

Thank you for reporting this!

I discussed this with @tannergooding from the .NET team, and it seems that there is an expectation that if you're running on Linux you're using cdecl and not stdcall. As a result of this, the .NET runtime has made no attempt to support this on Linux. In addition, it appears that vkd3d is primarily geared towards WINE usage, which means we have to take special care to use it outside of WINE.

So, our options:

1. Remove our vkd3d support and mandate that users wanting to use D3D11 on Linux use WINE as intended by Valve

2. Roll a custom build of the binaries that use Cdecl. This isn't future proof as our APIs are declared to use stdcall, and support for this could one day light up in the runtime and break everything.

3. Implement native trampolines to bridge the calling conventions. This has the same problem as option 2.

Thoughts @Beyley?

So theres a bit of misunderstanding here, vkd3d does not have anything to do with D3D11 specifically, at least in this case. All it provides is the HLSL shader compiler (which does not compile to SPIRV), even without vkd3d, D3D11 would still work all the same, since we only pull in vkd3d for D3DCompile. DXVK on native linux is officially supported (and used) by Valve.

Since we statically link vkd3d into our custom shader compiler (that wraps over vkd3d), we dont have to worry about system vkd3d mucking it up (our custom library is named differently) and we can make any changes needed, patching vkd3d is the obvious option here, and likely the way to go, i didnt realize that even though D3DCompile was cdecl, the functions pointers returned in the d3dblob were not cdecl, since the d3d11 example code all ran fine.

The solution is just "make a patch to remove the ms_abi declaration from VKD3D, and apply that patch when building vkd3d", that will then get statically pulled in by the shader compiler, and it should work, i dont currently have the time to work on Silk, but its an easy PR id be happy to review if you would like to take it up.

[alexrp]

I think #1993 fixes this - it has the ms_abi removal in the Vkd3d submodule.