I don&#x27;t disagree with you but let&#x27;s keep in mind that this is a low level library, not a user-facing product, and it does nothing by itself. So it&#x27;s like complaining that SDL or DXVK don&#x27;t have screenshots.

thank you! so needed. there is a special place in hell for folks that don&#x27;t attach screenshots to fundamentally graphic projects.

I need to emphasise that this is dramatically different. Comparing the first clip of OoT to [1], I see:
- Higher framerate
- Real-time lighting
- Real-time shadows
- Higher render resolution
- Water ripples&#x2F;splashes (unsure about these being new)
- Widescreen, motion blur, etc.If your contention is &quot;they&#x27;re not an improvement on the graphics, just changes&quot;, then we can agree to disagree, but they&#x27;re otherwise incredibly noticeable![1]: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nySI72vRl_4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nySI72vRl_4</a>

The improvement is in the shadows. Link’s real-time shadow on the ground, the self-shadowing of the roof of the house that Luigi emerges from, the shadows the ropes cast on Kirby… the N64 could not do these things even at coarse resolution due to limited texture bandwidth.

It would be an interesting feedback loop to capture a screenshot of a game (sm64), determine the green texture is grass, update the texture realtime, take another screenshot, and iterate, having the game flesh itself out realtime as it’s being played.

&gt; Doesn&#x27;t seem to be a noticeable improvement from N64-era graphicsright, because that&#x27;s the not point, the point is to be like N64-era graphics at 60fps and greater, with widescreen, motion blur, and other things that have high engagement amongst gamers (whether the self-proclaimed gamers like them or not)

Doesn&#x27;t seem to be a noticeable improvement from N64-era graphics.It probably needs generative AI based upscaling to high resolution meshes, textures and realistic materials to actually achieve a quality improvement.

Here&#x27;s a 2022 video that shows it in action<a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;Games&#x2F;comments&#x2F;v42611&#x2F;dario_on_twitter_id_like_to_reveal_that_rt64_the&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;Games&#x2F;comments&#x2F;v42611&#x2F;dario_on_twit...</a>

And discovered you could disable cache writeback to have a region of fast RAM without bus contention, and ways to use the hidden 9th bit in each byte which is only usd by certain graphics framebuffer operations.I don&#x27;t think he uses either one in serious code, but if he did, good luck emulating it.

Just about a Fabrice Bellard level of competence there

For those who haven&#x27;t heard, Kaze is the guy who optimized Mario 64&#x27;s code, fixed some physics bugs, etc., claims roughly 6x speedup, so that he can make a mod &#x2F; ROM-hack that runs smoother despite having better graphics: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=t_rzYnXEQlE" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=t_rzYnXEQlE</a>

I guess the big test will be running Kaze Emanuar’s code through it.

That&#x27;s not quite how ubsershaders work. They&#x27;re a &quot;fallback&quot; shader in the sense that they rather inefficiently implement the entire pipeline, but they do implement the entire pipeline. The shader being compiled in another thread will be more efficient as it uses only the logic needed for whatever configuration the game is calling up. But the visual result is identical in the ubsershaders case, that&#x27;s the whole point. If you want, and your host system is powerful enough, you can turn ubsershaders on all the time and disable the entire compilation thread and associated cache.I believe the term was coined by Dolphin team, who did a pretty good high level writeup of the feature here:<a href="https:&#x2F;&#x2F;dolphin-emu.org&#x2F;blog&#x2F;2017&#x2F;07&#x2F;30&#x2F;ubershaders&#x2F;" rel="nofollow">https:&#x2F;&#x2F;dolphin-emu.org&#x2F;blog&#x2F;2017&#x2F;07&#x2F;30&#x2F;ubershaders&#x2F;</a>

So how about:1. A global, networked shader cache — where when any instance of the emulator encounters a new shader, it compiles it, and then pushes the KV-pair (ROM hash, target platform, console shader object-code hash)=(target-platform shader object-code) into some KV server somewhere; and some async process comes along periodically to pack all so-far-submitted KV entries with a given (ROM hash, target platform) prefix into shader-cache packfiles. On first load of a game, the emulator fetches the packfile if it exists, and loads the KV pairs from it into the emulator&#x27;s local KV cache. (In theory, the emulator could also offer the option to fetch global-shader-cache-KV-store &quot;WAL segment&quot; files — chunks of arbitrary global-shader-cache KV writes — as they&#x27;re published on a 15-minute-ly basis. Or KV entries for given (ROM hash, target) prefixes could be put into message-queue topics named after those prefixes, to which running instances of the emulator could subscribe. These optimizations might be helpful when e.g. many people are playing a just-released ROMhack, where no single person has yet run through the whole game to get it in the cache yet. Though, mind you, the ROMhack&#x27;s shaders could already be cached into the global store before release, if the ROMhacker used the emulator during development... or if they knew about this, and were considerate enough to use some tool created by the emulator dev to explicitly compile + submit their raw shader project files into the global KV store.)2. Have the emulator (or some separate tool) &quot;mine out&quot; all the [statically-specified] shaders embedded from the ROM, as a one-time process. (Probably not just a binwalk, because arbitrary compression. Instead, think: a concolic execution of the ROM, that is looking for any call to the &quot;load main-memory region into VRAM as shader&quot; GPU instruction — where there is a symbolically-emulated memory with regions that either have concrete or abstract values. If the RAM region referenced in this &quot;load as shader&quot; instruction is statically determinable — and the memory in that region has a statically-determinable value on a given code-path — then capture that RAM region.) Precompile all shaders discovered this way create a &quot;perfect&quot; KV cachefile for the game. Publish this into a DHT (or just a central database) under the ROM&#x27;s hash. (Think: OpenSubtitles.org)Mind you, I think the best strategy would actually combine the two approaches — solution #2 can virtually eliminate stutter with a single pre-processing step, but it doesn&#x27;t allow for caching of dynamically-procedurally-generated shaders. Solution #1 still has stutter for at least one player, one time, for each encountered shader — but it handles the case of dynamic shaders.

In those gl&#x2F;dx games (built for non-specific hardware) all the textures and shaders are compiled either during the game&#x27;s build OR before you get into the scene. Many console systems, particularly Nintendo, do that precompilation specifically for the hardware GPU that is inside the console. That is not known to the emulator in advance (unless someone publishes a shader compilation alongside the rom...) so when the shader is referenced in the scene, it needs to be compiled in runtime to work on the emulated graphics system (translated from nintendo-hardware shader code to direct-x, vulkan or openGL then further into the vendor-specific shader)Most modern emulators implement a shader cache which stores those shaders as they are encountered so that this &quot;compilation stutter&quot; only happens once per shader - but modern titles can have hundreds or thousands of shaders and that means on a playthrough you&#x27;re pretty much encountering it consistently. Breath of the Wild was one that stands out as a game where you basically had to run it with precompiled shader caches as it was borderline unplayable without it.Ubershaders act like fallback shaders - using an off the shelf precompiled &quot;particle&quot; shader vs the actual one, while the actual one is compiled for use next time - this prevents the stutter at a cost of visual fidelity. If you see an explosion in a game, it will be a generic explosion shader vs the actual one used in the game, until it is available in the shader cache.

Having worked on some of the latter part of that era of GPUs, the &quot;frontend&quot; of the shader compiler was a pretty small fraction of the total time cost, most of it was in the later optimization passes that often extremely hardware specific (so not really possible at the level of DXBC). Especially as hardware started to move away from the assumptions used in designing it.I think a big part of the user-visible difference in stutter is simply the expected complexity of shaders and number of different shaders in an &quot;average&quot; scene - they&#x27;re 100s of times larger, and CPUs aren&#x27;t 100s of times faster (and many of the optimization algorithms used are more-than-linear in terms of time vs the input too)Modern DXIL and SPIR-V are at a similar level of abstraction to DXBC, and certainly don&#x27;t &quot;solve&quot; stutter.

It was, in fact, a problem. DX11 and earlier tried to solve it with DXBC, an intermediate bytecode format that all drivers could consume. The driver would only need to lower the bytecode to the GPU’s ISA, which is much faster than a full compilation from HLSL. Prior to the emergence of Vulkan, OpenGL didn’t ever try to solve this; GLSL was always the interface to the driver. (Nowadays SPIR-V support is available to OpenGL apps if your driver implements the GL_ARB_gl_spirv extension, and of course DXIL has replaced DXBC.)Compilation stutters was perhaps less noticeable in the DX9&#x2F;OpenGL 3 era because shaders were less capable, and games relied more on fixed functionality which was implemented directly in the driver. Nowadays, a lot of the legacy API surface is actually implemented by dynamically written and compiled shaders, so you can get shader compilation hitches even when you aren’t using shaders at all.In the N64 era of consoles, games would write ISA (“microcode”) directly into the GPU’s shared memory, usually via a library. In Nintendo’s case, SGI provided two families of libraries called “Fast3D” and “Turbo3D”. You’d call functions to build a “display list”, which was just a buffer full of instructions that did the math you wanted the GPU to do.

Epic published a very well written article explaining what the problem is, what makes it so much worse in modern games, and why it&#x27;s a uniquely difficult problem to solve.<a href="https:&#x2F;&#x2F;www.unrealengine.com&#x2F;en-US&#x2F;tech-blog&#x2F;game-engines-and-shader-stuttering-unreal-engines-solution-to-the-problem" rel="nofollow">https:&#x2F;&#x2F;www.unrealengine.com&#x2F;en-US&#x2F;tech-blog&#x2F;game-engines-an...</a>

Compiling shaders directly from a high level representation to the GPU ISA only really happens on consoles.In DirectX on PC, shaders have been compiled into an intermediate form going back to Direct3D 8. All of these intermediate forms are lowered into an ISA-specific instruction set by the drivers.This final compilation step is triggered lazily when a draw happens, so if you are working on a &quot;modern&quot; engine that uses thousands of different material types your choices to handle this are to a) endure a hiccup as these shaders are compiled the first time they are used, b) force compilation at a load stage (usually by doing like a 1x1 pixel draw), or c) restructure the shader infrastructure by going to a megashader or similar.

Older games used precompiled shaders. These are inaccessible to the game devs and usually handled by the hardware makers, so the platform OEM for consoles and the video card OEM on PCs. Game devs have begged for the ability to write their own shaders for years and finally got it with DX11 and Vulkan. And that&#x27;s when things went to hell. Instead of the shaders being written and compiled for the specific hardware, they now have to be compiled for your GPU at run time. It&#x27;s a messy and imperfect process. EA, Ubisoft or anyone else is never going to have the same level of understanding of a GPU that Nvidia or AMD will have. Often the stuttering is due to the shaders having to be recompiled in game, something that never happened before.

It was, but it was made worse by DX12&#x2F;Vulkan. Previously shader stages were separate and fixed pipeline settings were loose bag of knobs. If that corresponded to hardware stages, great, everybody was happy! However, if there was mismatch between API and hardware (for example, some fixed pipeline state wasn&#x27;t really hardware state but lowered to shader code) then driver needed to hack around it. If that could be done by patching the shader, it was ugly, but it worked fine. But if it required recompilation, then stutters occurred and it couldn&#x27;t be prevented by application because it was all hidden from it. The solution was thought to encapsulate everything in pipeline state objects. If you could do that, excellent, no stutters no matter on what hardware it runs. It has issues however: in reality applications (especially emulators) don&#x27;t always know beforehand which states they will need, previous ugly driver tricks that sometimes could have avoided recompilation by patching shaders are no longer possible, and even if the hardware does have easily switchable state API doesn&#x27;t expose that leading to combinatorial explosion of state objects that was previously unnecessary. Some of that was rolled back by introducing API extensions that allow more granular switches of partial state.

The oldest N64 emulators predate programmable shaders. They mapped possible configurations of the N64 GPU onto configurations of the host GPU. But this is really hard and some configurations are just impossible. It&#x27;s basically a huge list of tricks and special cases. For reference, I think it was like 16K lines for the color combiner, which basically does (A-B)*C+D.

To answer that question in the context of Vulkan I highly recommend reading the proposal document for VK_EXT_shader_object which contains a great explanation in the problem statement about how modern graphics APIs ended up in this situation:<a href="https:&#x2F;&#x2F;github.com&#x2F;KhronosGroup&#x2F;Vulkan-Docs&#x2F;blob&#x2F;main&#x2F;proposals&#x2F;VK_EXT_shader_object.adoc">https:&#x2F;&#x2F;github.com&#x2F;KhronosGroup&#x2F;Vulkan-Docs&#x2F;blob&#x2F;main&#x2F;propos...</a>The gist of it is that graphics APIs like DX11 were designed around the pipelines being compiled in pieces, each piece representing a different stage of the pipeline. These pieces are then linked together at runtime just before the draw call. However the pieces are rarely a perfect fit requiring the driver to patch them or do further compilation, which can introduce stuttering.In an attempt to further reduce stuttering and to reduce complexity for the driver Vulkan did away with these piece-meal pipelines and opted for monolithic pipeline objects. This allowed the application to pre-compile the full pipeline ahead of time alleviating the driver from having to piece the pipeline together at the last moment.If implemented correctly you can make a game with virtually no stuttering. DOOM (2016) is a good example where the number of pipeline variants was kept low so it could all be pre-compiled and its gameplay greatly benefits from the stutter-free experience.This works great for a highly specialized engine with a manageable number of pipeline variants, but for more versatile game engines and for most emulators pre-compiling all pipelines is untenable, the number of permutations between the different variations of each pipeline stage is simply too great. For these applications there was no other option than to compile the full pipeline on-demand and cache the result, making the stutter worse than before since there is no ability to do piece-meal compilation of the pipeline ahead of time.This gets even worse for emulators that attempt to emulate systems where the pipeline is implemented in fixed-function hardware rather than programmable shaders. On those systems the games don&#x27;t compile any piece of the pipeline, the game simply writes to a few registers to set the pipeline state right before the draw call. Even piece-meal compilation won&#x27;t help much here, thus ubershaders were used instead to emulate a great number of hardware states in a single pipeline.

It happened all the time. I&#x27;ve played plenty of D3D11 games where clicking to shoot my gun would stutter the first time it happened as it was compiling shaders for the bullet fire particle effects.Driver caches mean that after everything gets &quot;prewarmed&quot;, it won&#x27;t happen again.

&gt; Old Nintendo consoles like the N64 or the GameCube&#x2F;Wii didn&#x27;t have programmable shaders.The N64 did in fact have a fully programmable pipeline. [1] At boot, the game initialized the RSP (the N64’s GPU) with “microcode”, which was a program that implemented the RSP’s graphics pipeline. During gameplay, the game uploaded “display lists” of opcodes to the GPU which the microcode interpreted. (I misspoke earlier by referring to these opcodes as “microcode”.) For most of the console’s lifespan, game developers chose between two families of vendor-authored microcode: Fast3D and Turbo3D. Toward the end, some developers (notably Factor5) wrote their own microcode.[1]: <a href="https:&#x2F;&#x2F;www.copetti.org&#x2F;writings&#x2F;consoles&#x2F;nintendo-64&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.copetti.org&#x2F;writings&#x2F;consoles&#x2F;nintendo-64&#x2F;</a>

I think there is some confusion about &quot;ubershaders&quot; in the context of emulators in particular. Old Nintendo consoles like the N64 or the GameCube&#x2F;Wii didn&#x27;t have programmable shaders. Instead, it was a mostly fixed-function pipeline but you could configure some stages of it to kind of somewhat fake &quot;programmable&quot; shaders with this configurable pipeline, at least to some degree. Now the problem is, you have no idea what any particular game is going to do, right until the moment it writes a specific configuration value into a specific GPU register, which instantly configures the GPU to do whatever the game wants it to do from that very moment onwards. There literally is no &quot;shader&quot; stored in the ROM, it&#x27;s just code configuring (parts of) the GPU directly.That&#x27;s not how any modern GPU works though. Instead, you have to emulate this semi-fixed-function pipeline with shaders. Emulators try to generate shader code for the current GPU configuration and compile it, but that takes time and can only be done after the configuration was observed for the first time. This is where &quot;Ubershaders&quot; enter the scene: they are a single huge shader which implements the complete configurable semi-fixed-function pipeline, so you pass in the configuration registers to the shader and it acts accordingly. Unfortunately, such shaders are huge and slow, so you don&#x27;t want to use them unless it&#x27;s necessary. The idea is then to prepare &quot;ubershaders&quot; as fallback, use them whenever you see a new configuration, compile the real shader and cache it, and use the compiled shader once it&#x27;s available instead of the ubershader, to improve performance again. A few years ago, the developers of the Dolphin emulator (GameCube&#x2F;Wii) wrote an extensive blog post about how this works: <a href="https:&#x2F;&#x2F;de.dolphin-emu.org&#x2F;blog&#x2F;2017&#x2F;07&#x2F;30&#x2F;ubershaders&#x2F;" rel="nofollow">https:&#x2F;&#x2F;de.dolphin-emu.org&#x2F;blog&#x2F;2017&#x2F;07&#x2F;30&#x2F;ubershaders&#x2F;</a>Only starting with the 3DS&#x2F;Wii U, Nintendo consoles finally got &quot;real&quot; programmable shaders, in which case you &quot;just&quot; have to translate them to whatever you need for your host system. You still won&#x27;t know which shaders you&#x27;ll see until you observe the transfer of the compiled shader code to the emulated GPU. After all, the shader code is compiled ahead of time to GPU instructions, usually during the build process of the game itself. At least for Nintendo consoles, there are SDK tools to do this. This, of course, means, there is no compilation happening on the console itself, so there is no stutter caused by shader compilation either. Unlike in an emulation of such a console, which has to translate and recompile such shaders on the fly.&gt; How come this was never a problem for older [...] emulators?Older emulators had highly inaccurate and&#x2F;or slow GPU emulation, so this was not really a problem for a long time. Only once the GPU emulations became accurate enough with dynamically generated shaders for high performance, the shader compilation stutters became a real problem.

&gt; Uses ubershaders to guarantee no stutters due to pipeline compilation.I may sound way out of the loop here, but... How come this was never a problem for older dx9&#x2F;dx11&#x2F;GL games and emulators?

Its a rather old sample, but from the same author and project (somewhat)<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Yl7WvabHfk4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Yl7WvabHfk4</a>the OG &quot;less compressed&quot; video was on a twitter post that i couldn&#x27;t find right now.

<pre><code> Work in Progress
 
 Emulator Support (Plugin) and Ray Tracing (RT) are not available in this repository yet.</code></pre>

RT64: N64 graphics renderer in emulators and native ports