what is a representation actually made of?
A representation, in the applied sense, is a bounded set of numbers inside a network that the readout layer treats as the identity of an input. That is the working definition. It is narrower than "features" and broader than "embedding": it is whatever the classifier or forecasting head causally depends on. The question I keep coming back to in production work is which part of that bundle is doing the work, and which part is dead weight we are paying GPU-hours to compute.
Three papers dropped on arXiv on 15 June 2026 that, read together, sketch a coherent answer. They do not cite each other and they live in different sub-fields, but they triangulate the same claim: the meaningful substrate of a learned model is usually thinner, more geometric, and less digital than the network around it suggests. I want to walk through them as one argument, because that is how they read once you put them on the same desk.
paper one: identity rides on phase, not magnitude
Oppenheim and Lim showed in 1981 that a natural image stays recognizable when you reconstruct it from its Fourier phase alone. The magnitude, the part that says "how much of each frequency," carries almost nothing of what the image is. Alper Yıldırım's internal Oppenheim–Lim test asks whether trained classifiers reproduce this asymmetry inside their hidden layers, and the test is causal, which is what matters in practice: take two images, transplant the phase of one onto the magnitude of the other at a chosen layer, and record which image the prediction follows.
In PRISM2D, GFNet, and ViT-B/16 the prediction follows the phase or sign donor. Deleting all image-specific magnitude barely moves accuracy. ResNet-50 looks like an exception at first — transplanting sign after its ReLUs does nothing — but intervening before the ReLU exposes a strong latent sign code in the late blocks. The architectures share a phase/sign identity code, but rectification and readout geometry change the basis in which that code is exposed.
This is not a curiosity. If you are shipping a vision classifier and your magnitude channel is largely dispensable to the readout, the practical implications fall out fast. Robustness audits should be phase-targeted, not pixel-targeted. Quantization budgets should protect sign and phase first; magnitude can usually take the hit. The texture–shape gap between CNNs and attention models, which has been folklore for years, gets a mechanistic account: same identity code, different exposure. In production that means the choice between a CNN and a ViT for an image task is not really a choice about "what the model sees" — it is a choice about how a shared phase code is decoded.
paper two: if the operator is linear, why is the substrate digital?
The second paper is HAMON, and it asks an even more pointed question. HAMON starts from a fact the time-series community has been quietly uncomfortable with for two years: simple linear and frequency-domain models remain competitive with transformers on long-horizon forecasting. The mechanistic evidence keeps suggesting the core forecasting operator is low-complexity and approximately linear. So the authors raise what they call a substrate-level question. If the operator is linear, does it need to be a learned digital temporal mixer at all?
Their answer is a passive diffractive optical core. Historical values are encoded onto an optical aperture, future positions are left dark, and cascaded trainable phase masks with free-space diffraction shape the forecast directly in the output field. At inference, a single passive optical propagation pass produces the prediction. No digital sequence-mixing layer runs.
The results are honest, which I appreciate. HAMON outperforms the strongest digital baselines on ETTm2 at all horizons and on ETTh2 at all but the longest, improving MSE by up to 14% — and crucially, consistently across horizons, not at cherry-picked points. It is competitive on Weather. It trails on Traffic and Electricity, which are high-channel-count. That trailing is the trade-off you should expect: passive optics scales beautifully in spatial dimensions of the aperture, painfully in the number of independent channels you have to multiplex.
For a practitioner this is interesting in a very specific way. It defines a concrete target. If a real diffractive device exists with the right phase-mask precision, the inference cost of long-horizon forecasting for a single-channel or low-channel series collapses to the cost of letting light cross a few centimeters. That is not a paper about replacing your transformer tomorrow. It is a paper saying: the operator you are paying a transformer to approximate is so close to linear that a passive piece of glass can do it. Whether you ever build the glass is a capex question; whether the operator is that simple is now an empirical answer.
paper three: the data was never flat
The third paper is a 79-page survey by Choi and colleagues on shape space analysis. Shape space, in their working definition, is the mathematical space that remains once you quotient out translations, rotations, and scale from a geometric object — what is left is variability in geometry itself. Subcellular morphology, primate tooth evolution, anthropometric scans, medical surfaces: all of these live on nonlinear manifolds, and treating them as flat vectors loses the structure that carries the scientific signal.
The applied takeaway is simple and uncomfortable. Most of the production ML stack — Euclidean distances, mean pooling, additive embeddings, k-NN on raw vectors — assumes flatness. For a lot of tabular and language data that is fine, because the relevant variability really is approximately linear in the chosen basis. For geometric data it is not, and the survey organizes the field around the pipeline you actually need in practice: shape representation and parameterization, robust geodesic metrics, statistics on the resulting manifold, and geometry-aware learning. If you are building anything that compares 3D scans, medical surfaces, or biological shapes, the cost of pretending the data is flat is paid in silently degraded metrics that look fine on a leaderboard and lie when a clinician looks at them.
the same argument, three times
What ties these together is a single claim about where representation lives. Paper one says: inside a vision classifier, identity is carried by a phase/sign code, not by the bulk of the activations. Paper two says: for long-horizon forecasting, the operator is so close to linear that the substrate need not be digital. Paper three says: the data itself has a geometric structure that the flat vector spaces of standard ML throw away.
In each case the working model has more machinery than the underlying signal requires. The phase code is a low-dimensional structure hiding inside billions of parameters. The forecasting operator is an approximately linear map hiding inside a transformer. The shape space is a manifold hiding inside a vector. The applied consequence is the same in all three: you are paying for capacity you do not use, and the capacity you do not use is also where the failure modes hide.
There is a fourth signal worth putting next to these. The April 2026 tutorial on energy-based dynamical models by Montanari, Bullo, Krotov, and Motter pushes the same direction from the other side — it asks what computation looks like when you stop treating it as feedforward layers with backprop and start treating it as gradient flow on an energy landscape. Continuous-time Hopfield networks, dense associative memory, oscillator-based optimizers, proximal-descent dynamics. The recurring theme is that scalability, robustness, and energy efficiency tend to arrive when the computational substrate matches the structure of the problem, instead of being a general-purpose digital approximator running on top of it.
what I would actually do with this
If I were greenfielding a vision pipeline today, I would run a phase-transplant probe on whatever backbone I shipped, as a routine diagnostic. It is a few hundred lines of code, it tells you which layer the identity code lives in, and it tells you what your quantization and adversarial budgets should protect. If I were running long-horizon forecasting at scale, I would not be building optical hardware, but I would treat HAMON's result as license to push much harder on linear and frequency-domain baselines before reaching for a transformer — and to measure my forecasting heads against the hypothesis that the operator is approximately linear. If I were touching geometric data of any kind, I would stop pretending Euclidean distance is free.
None of these are revolutions. They are corrections to a working assumption — that more capacity, more parameters, and more digital flexibility are the path. The signal from June 2026 is that the path is often shorter, thinner, and more structured than the architecture suggests. In production that translates into the same thing it always does: cheaper inference, better calibration, and fewer surprises at the long tail.
Sources
- The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers — arXiv · cs.AI · 2026-06-15
- HAMON: Passive Optical Sequence Mixing for Long-Horizon Forecasting — arXiv · cs.AI · 2026-06-15
- Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis — arXiv · cs.LG · 2026-06-15
- [2604.05042] Energy-Based Dynamical Models for Neurocomputation, Learning, and Optimization — Manual / ad-hoc · 2026-06-17