How Deepfakes Actually Work Under the Hood
I spent about three years building and then debunking AI-generated face swaps for a mid-size media company. The tools have shifted dramatically since 2020, and most people reading about this still think of deepfakes as just swapped celebrity faces on YouTube videos. That's not what's happening anymore. The infrastructure has moved into enterprise-grade territory, and the detection side hasn't kept pace. At the core, modern deepfake generation relies on two main architectures: autoencoder-based systems and GANs (Generative Adversarial Networks). The older approach uses a pair of neural networks — an encoder that compresses a source face into a latent representation, and a decoder that reconstructs it onto a target video frame. The newer GAN-based pipelines add a discriminator network that constantly evaluates whether the output looks real, forcing the generator to improve iteratively. You're essentially training two models to beat each other until the output is indistinguishable from actual footage. The practical reality is that training a decent model on a custom face used to require 4-8 hours on a GPU like an RTX 3090 with maybe 2,000-5,000 frames of source material. By 2023, tools like FaceDancer and later open-source releases from Silenur pushed that down to under an hour using pretrained base models and fine-tuning. The bottleneck shifted from compute to data quality. Garbage in, garbage out applies harder here than almost anywhere else in ML.
I ran into a specific edge case that took me two weeks to solve. We were generating synthetic training data for a client's product demo, and the model kept producing a subtle double-exposure artifact around the jawline when the subject turned more than 45 degrees. The source footage was 4K at 60fps, well-lit, clean audio — nothing obviously wrong. What I eventually found was that the encoder's bottleneck layer was too narrow for high-pitch facial motion. The fix wasn't more training data. It was switching from a standard autoencoder to a variational autoencoder (VAE) with a larger latent space dimension and applying a perceptual loss function instead of pure pixel MSE. Cut the artifact from visible to barely detectable at 1080p. At 4K it's gone. Another counter-intuitive thing nobody talks about enough: higher resolution source footage doesn't always produce better results. There's a sweet spot. I've seen teams feed 8K material into pipelines that weren't designed for it, and the results were actually worse because the preprocessing step — face alignment, landmark detection, head pose estimation — degraded faster than the model could compensate. 1080p or 720p at 30fps with clean lighting is often the optimal input for most off-the-shelf tools. The detection side is where things get complicated. Traditional forensic methods look at biological inconsistencies — irregular blink rates, pulse detection through skin color changes (remote photoplethysmography), mismatched lighting between face and background. These worked well against early generative models from 2018-2020. They don't work against current models because modern generators are specifically trained to include micro-blinking and natural head movement as regularization techniques. Some of the best open-source models now include a "natural motion" post-processing step that injects realistic micro-expressions after the main generation pass.
The most reliable detection method right now involves analyzing frequency-domain artifacts. Most generative models leave behind distinct patterns in the Fourier transform of the output — subtle periodic structures that don't appear in real camera sensor data. Tools like CNN-based detectors trained on the Faceshift dataset can pick these up with reasonable accuracy. But this is an arms race. Every time a detector improves, the generator gets retrained on data that includes outputs from that detector, creating a cycle where the baseline for "looking real" keeps rising. Here's the blunt part: there is no silver bullet for detection. Any single tool will miss sophisticated generations and flag some legitimate content as fake. The workflow that actually works is combining multiple approaches — frequency analysis, metadata examination, ensemble classifiers — and even then you're operating at maybe 85-92% accuracy depending on the quality of the source material and how carefully the generation was done. If you're looking to build your own pipeline, the current state-of-the-art open source options are X-Phactor, DeepFaceLive for real-time applications, and the SadTalker family for static image animation. For training from scratch, you'll want a dataset of at least 2,000 frames, a GPU with 24GB VRAM minimum, and several days of training time if you're not using transfer learning. The community maintained repositories on GitHub tend to have the most current implementations, but they break frequently as the underlying PyTorch and TensorFlow versions update.
Get the Full Details
The legal landscape is similarly murky. Most jurisdictions now have laws against malicious deepfakes, but the definitions are narrow and enforcement is inconsistent. A lot of the technology exists in a gray area where the tool itself isn't illegal, but specific uses are. That means anyone working seriously in this space needs to understand both the technical capabilities and the jurisdiction-specific restrictions before deploying anything publicly.
