What does it actually mean to pretrain a foundation model on a signal that is mostly noise? The industry has a default answer, and for images and text it works: chop the input into tokens, hide some of them, train the model to reconstruct what you hid. That recipe — masked reconstruction, the family that includes masked autoencoders (MAE) — has become the reflex for building any self-supervised foundation model (FM), a large model pretrained on unlabeled data so it can later be fine-tuned for many downstream tasks. The CoCoT-EEG paper argues that when the signal is electroencephalogram (EEG) — the electrical activity of the brain measured from the scalp — that reflex is the wrong one.
Start with why reconstruction works elsewhere, because that tells you why it breaks here. When you mask a patch of an image and ask the model to fill it in, the missing pixels are highly predictable from their neighbors: natural images are dense, redundant, and high signal-to-noise. Reconstructing them forces the model to learn structure. EEG is the opposite kind of data. As the paper puts it, EEG has high noise amplitude and information confined to limited dimensions such as narrow frequency bands (source). The relevant neural signal lives in a small slice of the spectrum; most of the amplitude is muscle artifact, line noise, and drift. If you train a model to reconstruct the raw waveform, you spend most of its capacity reproducing noise. The objective rewards fidelity to exactly the part of the signal you do not care about.
This is a trade-off that shows up any time you copy a recipe across modalities without checking the assumptions baked into it. The MAE assumption is the input is dense and the reconstruction target is informative. Break that assumption and the pretraining objective actively fights you.
CoCoT's answer is two design decisions, and both are worth understanding in engineering terms rather than as a benchmark headline. First, the pretraining objective changes from reconstruction to contrastive learning — instead of asking "reproduce the hidden signal," you ask the model to pull two augmented views of the same EEG segment close together in representation space and push different segments apart. Contrastive objectives do not require the model to reconstruct noise; they only require it to tell segments apart. For a low-signal-to-noise modality that is a much better-posed problem. The literature review of the paper frames this directly as a challenge to the reconstruction-pretraining industry standard.
Second, the input layers. CoCoT does not feed raw samples straight into a transformer. It uses multiscale temporal convolution input layers before the transformer encoder blocks (source). This matters because the informative content of EEG is spread across different timescales — fast oscillations and slow rhythms carry different information — and a bank of convolutions at multiple scales is a cheap, well-understood way to extract band-limited temporal features before attention ever runs. The idea of pairing multiscale convolution with a transformer for EEG is not brand new; earlier work applied multiscale convolutional transformers to EEG classification. What CoCoT adds is the contrastive pretraining on top and the demonstration that the combination scales as a foundation model.
The results are where an applied engineer should pay attention, because they change the build-versus-pretrain calculus. CoCoT matches or beats state-of-the-art reconstruction-pretrained models across benchmark decoding tasks with heterogeneous electrode configurations — different headsets, different channel counts (source). That heterogeneity is not a footnote. In production, you rarely control the hardware: one clinic has a 64-channel medical cap, a consumer BCI ships with 8 dry electrodes, a research rig has something else entirely. A model that only works on one montage is a lab demo. A model that decodes across configurations is deployable.
The second result is the one I find more useful. CoCoT trained from scratch — no pretraining at all — outperforms previous single-task decoding models and even rivals the pretrained models (source). Read that as an engineer: the architecture itself, the multiscale-conv front end feeding a transformer, is doing a large share of the work. Pretraining adds data efficiency, but you are not forced into the massive-corpus, massive-compute regime just to get a competitive decoder. If you have a modest labeled dataset and a single well-defined task — say, a motor-imagery classifier for a rehabilitation exercise — you can train this from scratch and ship it. That lowers the barrier for teams without a foundation-model budget.
I have shipped systems where a biosensor stream drives an adaptive experience in real time, so the practical constraint I keep coming back to is latency and hardware. A multiscale-convolution front end is friendly here: convolutions are cheap, parallelize well, and the band-limited features they produce are exactly what a downstream classifier needs, so you can often keep the model small enough to run near the edge rather than round-tripping every window to a server. The contrastive-pretraining story is a separate budget line — you pay that cost once, offline — and the from-scratch result tells you it is optional, not mandatory.
Step back to the systemic point, because it is bigger than EEG. A foundation-model recipe encodes assumptions about the statistics of the data it was designed for. Text and images are dense and redundant, so masked reconstruction is a good self-supervised objective for them. When a field imports that recipe wholesale into a domain with different statistics — sparse information, low signal-to-noise, structure confined to narrow bands — it inherits assumptions that no longer hold. The paper's own ablations, varying architecture and pretraining objective, are essentially a controlled test of which assumption was doing the damage (source). That is the discipline worth copying: before you adopt a pretraining objective, ask what property of the source domain made it work, and check whether your domain has that property.
The same question applies far outside brain-computer interfaces. Satellite spectra, financial tick data, industrial sensor telemetry — these are all low-redundancy, noise-heavy modalities where the interesting signal sits in a thin band. The reflex to reach for a masked-reconstruction FM should trigger the same audit every time: is my data dense enough for reconstruction to teach the model anything, or am I about to train an expensive noise-reproducer? CoCoT is one careful answer for one modality. The transferable lesson is the audit itself.
One caveat worth stating plainly: this is a single arXiv preprint, not yet peer-reviewed, and the strong claims — matching state of the art, rivaling pretrained models from scratch — rest on the authors' own benchmark suite. The direction is well-motivated and the ablations are the right kind of evidence, but I would want independent replication on an out-of-distribution montage before treating the from-scratch result as a general guarantee rather than a promising, benchmark-bound finding.
Sources
- CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding — arXiv · cs.LG · 2026-07-10