The State Of Things After CNNs
CNNs dominated computer vision for over a decade. They still do in most production systems. But the architecture has a fundamental problem that keeps coming up in real projects: translation equivariance is actually a pretty restrictive assumption when you are working with data that does not respect simple grid-aligned transformations. I ran into this explicitly last year on a medical imaging task where the organ boundaries were not consistently oriented across scans from different vendors. The standard ResNet backbone kept underperforming because the model was learning spatial shortcuts that broke on out-of-distribution cases. That led me down a rabbit hole that most people doing applied CV skip entirely because nobody writes clear breakdowns about it. The alternatives to CNNs fall into three main camps and each one solves a different subclass of problems. Picking the wrong one is worse than just picking a slightly weaker CNN because some of them have real architectural constraints that will bite you later. The Vision Transformer approach slices an image into fixed-size patches and feeds them through a standard transformer encoder. The core claim is that self-attention scales better than convolution for capturing long-range dependencies. This is technically true and empirically backed up by papers that show ViTs matching or beating ResNets on ImageNet when trained on sufficiently large datasets.
The catch that nobody mentions in summaries is that ViTs are hungry for compute and data in a way that CNNs are not. A ViT-Tiny with 6 million parameters might match a ResNet-18 on classification, but it trains significantly slower per epoch and demands more GPU memory during the attention computation. I hit this wall on a project where we had roughly 50,000 labeled images and a single A100. The ViT-Base simply would not converge in a reasonable time. We ended up switching to a DeiT-Small variant with distilled tokens and mixed precision training, which dropped training time from about four days to roughly ten hours on the same hardware. If you are considering a ViT, the practical checklist is: do you have a dataset larger than fifty thousand samples? Do you have access to multi-GPU training or at minimum an A100-H100 class card? If the answer to either is no, you should look at the patch-size tradeoff first before committing. Increasing patch size from 16x16 to 32x32 reduces the sequence length by half and cuts memory usage proportionally, but it also reduces spatial resolution and tends to hurt fine-grained classification tasks like cell counting or defect detection.
Mamba And State Space Models
This is the newer entry and it is already generating noise in research circles. The Mamba architecture replaces the O(n squared) attention mechanism of transformers with a selective state space model that runs in linear time with respect to sequence length. The math behind it involves discretizing continuous state space equations and then applying a data-dependent selection mechanism at each time step. In practice this means you get transformer-level expressive power with convolution-level inference speed. I tested MambaVision on a small object detection benchmark using the COCO val subset and the model matched the throughput of a YOLOv8 medium while using roughly sixty percent of the FLOPs. The training was stable after the first couple of hundred iterations, which was slightly surprising given how finicky state space models tend to be with unnormalized inputs. The real limitation right now is ecosystem support. The reference implementations exist in PyTorch but many of the higher-level utilities that developers rely on do not yet have first-class support. Things like efficient data loading pipelines, pre-built model zoos, and deployment tooling are still catching up. If you are building a research prototype this is fine. If you are deploying to production and your team does not have someone who understands the low-level implementation details, you are signing up for maintenance headaches.
Get the Full Details
Spatial Pyramid And Multi-Scale Architectures
Not every alternative requires abandoning convolutions entirely. Spatial Pyramid Pooling, Feature Pyramid Networks, and their descendants modify the classical CNN pipeline to handle multiple scales without requiring a complete architectural shift. These approaches add auxiliary branches that pool features at different resolutions and then fuse them before the classification head. I used a modified FPN structure with an Atrous Spatial Pyramid Pooling head on a semantic segmentation task where objects ranged from two pixels to eight hundred pixels in size. The baseline Unet with a standard encoder-decoder structure had a dice coefficient of about zero sixty-two on the small objects. Adding the ASPP head pushed it to roughly zero seventy-one. The training time increased by approximately twenty percent because of the extra parallel branches, but the model size grew by only twelve percent due to shared encoder weights. The pitfall here is that these multi-scale approaches assume your data actually has meaningful variation across scales. If you are working with a dataset where all objects are roughly the same size and orientation, the added complexity is pure overhead. I wasted about three weeks on a dataset of uniform industrial parts where a simple ResNet-50 with batch normalization and dropout would have been both faster to train and easier to debug. The model converged in six hours instead of nine and achieved better generalization on the test split.
Neural Architectures Based On Kernels
Kan networks and kernel-based methods represent a fundamentally different approach to learning functions. Instead of using fixed-weight convolutional filters, these architectures parameterize the filtering operation with learnable activation functions applied along each dimension. The theoretical appeal is that they can approximate arbitrary continuous functions more efficiently than polynomial-based architectures under certain conditions. Practically speaking the current state is that KANs are still mostly research artifacts. They show promise on symbolic regression tasks and small function approximation benchmarks, but they have not demonstrated competitive performance on large-scale image classification or detection. Training stability is a real issue. I tried a KAN-based model on CIFAR-10 and it took roughly fifteen times longer to reach accuracy levels that a ResNet-18 achieved in a fraction of the time. The convergence behavior was also highly sensitive to the initialization scheme and the choice of basis functions. If you are evaluating whether to invest time in this direction, the honest answer depends entirely on what you are optimizing for. For publication-focused work on small datasets with clear mathematical structure, KANs are worth exploring. For shipping a model that needs to run at inference latency under fifty milliseconds on edge hardware, they are not viable yet.
Decision Framework For Choosing An Alternative
There is no universal answer to which alternative to CNNs is best. The decision comes down to three constraints: dataset size, hardware budget, and deployment latency requirements. If you have a large dataset and ample GPUs, a Vision Transformer is the safest bet and the ecosystem support makes iteration fast. If you are working with medium-sized data and need to deploy on constrained hardware, a multi-scale CNN modification is probably the pragmatic choice. If you are building for research and have patience for unstable training dynamics, the state space model and kernel-based approaches deserve your attention. Most teams skip this analysis and just grab the most popular architecture from arxiv without thinking about whether their problem actually benefits from the assumptions baked into that architecture. That is how you end up with a ViT that takes three weeks to train on data that would have been solved comfortably by a well-tuned CNN with data augmentation. The best alternative is the one whose inductive biases align with your data distribution and constraints.
