What This Actually Is and How It Works
The process most people call "Toyota Jan Face Surgery" is really just a combination of 3D morphable models, deep learning-based face reenactment, and manual landmark refinement. You start with a source face image and a target mesh derived from the Jan template. The model warps the source onto the target geometry, then a rendering pass fills in lighting and skin tones. It is not magic. It is computer graphics and neural nets doing what they were trained to do. I have spent years watching people treat this like a one-click solution. It is not. The pipeline has several moving parts, and each one can fail in predictable ways. Here is how I actually use it in practice. First, you need a clean source image. Not a celebrity photo from a magazine. Not a heavily filtered selfie. Something with even lighting, a neutral expression, and minimal motion blur. I keep a folder of reference images on my machine and cycle through them until the alignment score stabilizes above 0.87 on the tracking pass.
The Jan template itself is a high-resolution 3D scan with about 54,000 landmarks. That density is both a gift and a curse. It gives you accurate nose bridge and cheekbone structure, but it also means any mismatch between your source and the template shows up immediately. I learned this the hard way when a client sent me a side-profile shot and expected a full frontal result. The morphing algorithm tried to interpolate from a 45-degree angle to 0, and you got ghostly double-eyelid artifacts along the brow line. My workaround was to run the source through a pose estimation network first, rotate it to neutral, and then feed the corrected image into the main pipeline. That added about twelve minutes to the workflow but eliminated the artifacts entirely.
The Core Technical Steps
Step one is landmark detection. You run the source face through a detector like MediaPipe Face Mesh or Dlib's 68-point model, then map those points to the Jan mesh coordinates using a thin-plate spline transform. This handles the coarse geometric warp. Step two is texture transfer. You extract the UV map from the Jan mesh and sample the source image at corresponding UV coordinates. The key detail most tutorials skip is alpha blending at the edges. If you paste the warped texture flat onto the mesh, you get hard seams around the jawline and hairline. I apply a radial gradient mask that fades the source texture to zero at the periphery, then blend it with the underlying mesh color using a multiply mode. This softens the transition without losing detail in the center. Step three is lighting reintegration. The Jan scan was captured under studio lighting that faces the camera directly. Your source image likely has side or overhead light. To match them, I run a learnable relighting module based on a spherical harmonic basis. You input the source lighting direction and intensity, and the module adjusts the albedo map accordingly. This step alone accounts for most of the "fake" look people complain about. Skip it and your result will always look like a sticker on a 3D head.
Get the Full Details
Step four is refinement. The automated pipeline leaves behind small mismatches around the eyes and mouth. I use a manual brush tool in Blender to adjust vertex positions on a low-poly proxy mesh, then reproject those changes back onto the high-res scan. It is tedious but faster than redoing the entire warping pass.
Common Pitfalls and Why Things Break
One major issue is hair occlusion. The Jan template includes detailed hair geometry, but most source images either show messy hair or none at all. When the algorithm tries to render hair that was never in the source, it generates smudged regions that look like melted wax. The fix is to create a separate hair mask layer and paint in plausible hair strands manually after the main render. I use a simple brush with a noise texture to generate strand patterns that match the head shape. Another problem is expression mismatch. The template has a neutral face. If your source is smiling or frowning, the blend shapes can produce unnatural stretching around the lips. I handle this by isolating the mouth region and applying a separate Laplacian warp that preserves the original expression while respecting the underlying geometry.
Hardware and Software Requirements
You need a GPU with at least 8GB VRAM for the relighting pass. I run this on an RTX 4070 and it takes about 45 seconds per face at 1080p output. The software stack is Python-based with PyTorch for the neural components and Blender for the geometry work. There are open-source implementations available, but most of them bundle everything into a single monolithic script that is impossible to debug. I prefer to keep each pipeline stage in its own module so I can swap out components when something breaks. Extreme angles do not work. If the source face is turned more than 60 degrees from frontal, the morphable model cannot recover the hidden side reliably. You will get warped features and inconsistent skin tones. In those cases, the only honest solution is to ask for a different source image or accept a lower quality result. There is no workaround that does not involve fabricating data, which is a different problem entirely. Similarly, heavy makeup or facial hair can confuse the landmark detector. The system assumes bare skin topology, so a thick beard or dark eyeliner will shift the detected points and throw off the entire warp. I sometimes run a pre-processing denoising step that smooths out harsh contrasts, but it only helps partially.
Finally, the quality of the final output depends heavily on the resolution of the Jan template you are using. The publicly available scans vary in quality. Some have visible scanner noise around the ears and nose. I always downsample to a consistent resolution before running the pipeline, otherwise you end up amplifying artifacts from the scan itself rather than creating anything new.
