Debugging the Shader Compiling Process

Debugging the Shader Compiling Process

https://www.unrealengine.com/en-US/tech-blog/debugging-the-shader-compiling-process

Intro

During development, it’s a good idea to take a look at what exactly UE4 is sending to the platform’s shader compiler. This post will allow you to debug any issues associated with it. 

Enable CVars to allow dumping intermediate shaders

On your ConsoleVariables.ini file (usually located at Engine/Config/ConsoleVariables.ini), enable these cvars:

Build ShaderCompileWorker in Debug

By default, UnrealBuildTool (UBT) will generate projects for tools to always compile in Development. To build them in debug, change your solution properties (Visual Studio: Build->Configuration Manager) for ShaderCompileWorker to Debug_Program:

Generate intermediate files

At this point, you’ll want to generate the files you’ll be able to debug; enabling the cvars will allow subsequent compilations to dump out the generated files; to force a rebuild of all shaders, add a space or a change to Engine/Shaders/Common.usf, and re-run the editor. This will recompile shaders and dump all the intermediate files in your Project/Saved/ShaderDebugInfo folder.

If you’re debugging a particular material, you can edit the material in the editor and then apply or save, this will dump the shader files again.

Structure of folders for dumped shaders

Let’s analyze the full path for a dumped file:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

The root path of your project:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

The root path for dumping shaders:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

Now for every shader format/platform, you’ll find a subfolder, in this case PC D3D Shader Model 5:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

Now we get a folder per material name, and a special one called Global. In this case, we are in the M_Egg material:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

Shaders are grouped in maps ordered by vertex factories (which mostly end up corresponding to a mesh/component type); in this case you have the Local Vertex Factory:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf

Finally, the next path is the permutation of features; because we had enabled r.DumpShaderDebugShortNames=1 earlier, the names are compacted (to reduce path length). If we had this set to 0, the full path would look like:

D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\FLocalVertexFactory\TBasePassPSFNoLightMapPolicy\BasePassPixelShader.usf

Finally in that folder there will at least be a batch file, a text file and a usf file. This usf file is the final shader code that will go to the platform’s compiler, run after the preprocessor. The batch file is a way to call the platform compiler to look at intermediate code. The text file (named DirectCompile.txt) contains the command line for debugging with ShaderCompileWorker (below).

Using ShaderCompileWorker for debugging

Starting on 4.11 we added the feature for ShaderCompileWorker (SCW) to be able to debug the call to the platform compiler; the command line is:

PathToGeneratedUsfFile -directcompile -format=ShaderFormat -ShaderType -entry=EntryPoint {plus platform specific switches}

• PathToGeneratedUsfFile is the final usf file from the ShaderDebugInfo folder
• ShaderFormat is the shader platform format you want to debug (in our case PCD3D_SM5)
• ShaderType is one of vs/ps/gs/hs/ds/cs which correspond to Vertex/Pixel/Geometry/Hull/Domain/Compute shader type
• EntryPoint is the function name of the entry point for this shader in the usf file

 

For example, D:\UE4\Samples\Games\TappyChicken\Saved\ShaderDebugInfo\PCD3D_SM5\M_Egg\LocalVF\BPPSFNoLMPolicy\BasePassPixelShader.usf -format=PCD3D_SM5 -ps -entry=Main

This command line can also be easily copied if you enabled the r.DumpShaderDebugWorkerCommandLine=1 cvar, which will dump a file called DirectCompile.txt next to the generated USF file. This feature is available since 4.13.

At this point you can now stick a breakpoint into the CompileD3D11Shader() function on D3D11ShaderCompiler.cpp, run SCW with the command line and you should be able to step into how we call the platform compiler.

For more information on the subject, check out these two pages on the Unreal Engine documentation:

• Shader Development
• HLSL Cross Compiler