Throughout my career in animation, I dipped in and out of doing Research and Development. Now, the line between R&D and a TD can be a little bit blurry at a studio; the major differentiators for R&D, imo, are a focus on supporting every show in production and a bigger scope of projects.
For example, as a TD I whipped up a couple of neat tricks (see TD Life Pt 2) that saw use on other shows, but they were made with a single show in mind, and they weren’t terribly large applications. Compare that to a developer of an internal application like DreamWorks’ award winning Premo, which is a massive animation package. That’s not to say that R&D groups don’t do tasks with a more narrow scope as well; for example, FX R&D often writes simulation nodes in Houdini that are comparatively much smaller. However, a big difference there is the depth of time spent on research; a TD will rarely have much time to spend on a novel technique, given how busy support on shows tends to be.
To sum it up, R&D has the opportunity to pursue greater breadth and depth compared to a TD.
There’s two major areas my time in R&D at DWA was spent.
Texture Synthesis Acceleration
We had an awesome tool at DreamWorks called texture synthesis, which could generate a seamless, non-repetitive texture on a mesh across UV islands. Artists could even control the direction and scale of the synthesized texture through painting simple texture maps. This was a really powerful tool, especially once we integrated it directly into Mari.
Unfortunately, it was a bit slow. The initial code was written by a mathematician, and R&D hadn’t had the chance to look for optimization opportunities, so the TD department loaned me out to R&D for a few months to take a look. It was a pretty massive codebase, so a lot of the first month was just spent documenting and understanding how the codebase worked, and identifying potential areas for optimization.
I generated a list of four approaches, and then spent the following months testing them out. I ultimately discovered two major optimizations, one of which I’ll discuss here; certain attributes had to be calculated per face, and in the existing codebase they were calculated three times each time the tool was called. Now, technically, there were different calculations being done that made this non-obvious that there was a core, re-usable calculation. I cached the result to a texture, and that gave about a ~30% performance boost to running the tool for the first time on a model. Since the cache was an external file, there was a further performance gain as long as the underlying model hadn’t changed on subsequent runs.
Deep-Learning Powered Denoising
Later on in my career, I returned to R&D for a couple of loan outs, and then ultimately started a Machine Learning department at DreamWorks Animation. There were a lot of cool small projects and PoCs that I did in this role, but the major one was authoring a denoiser for Moonray, powered by deep learning. This began well before most people had heard of diffusion or transformer models, so it wasn’t a well solved usecase at the time.
A denoiser is a tool that takes a noisy render from a monte-carlo raytracer, and fills in the missing details left over from the noisy sampling. Traditionally, this was done by calculating one or more convolutional kernels to set the value of an untraced pixel based on the pixels around it. This is a critical step in the process, as rendering until there’s no noise (known as convergence) can be very expensive on some scenes. Without deep learning, denoising typically requires some degree of artist intervention to tune the parameters of the tool, and occasionally requires manual paint fix.
This was a really interesting yet challenging project. There were existing papers for denoising, but we found they didn’t apply well to Moonray renders for a variety of reasons. We also faced hardware challenges; I was ultimately training on relatively weak Quadro based systems, instead of a system dedicated for training deep neural networks. Despite these limitations, I had some success in training a system that took advantage of multiple AOVs that were standard to render out of Moonray, and could handle hair and volumes on single images nicely.
Looking back, I’m proud both of the novel architecture of the denoiser, which allowed it to generalize to shows with different styles and renders with different sampling patterns from a relatively small amount of data quite well, as well as the actual quality of the code. It was quite modular, making running experiments very easy, and I wrote an extension for tensorflow to directly handle EXR without decreasing bitdepth and supporting arbitrary resolution, something that was not widely available in open source tools at the time.
Regrettably, I left DWA to pursue an opportunity at Spire Animation before I could extend the denoiser to work on video, and for that reason I believe it never saw production use. It’s a touch frustrating, as I spent at least a year and a half on the project, but I believe that may have happened anyways as deep learning powered denoising advanced as a field and open-source solutions became available. I still deeply cherish the experience, which taught me not only a great deal about deep learning, but also helped me learn how to translate my R&D efforts into real dollar impacts, which is incredibly useful for engineers to better understand technical strategy.