What does it take to turn subjective human suffering into objective, actionable data? In clinical psychiatry, diagnostics have historically relied on patient self-reports and clinical interviews. These methods, while valuable, are inherently vulnerable to subjective bias, cognitive distortions, and the varying empirical judgment of different practitioners. To build systematic, objective diagnostics, we must look to the body itself. Specifically, we must look to multimodal biosensing.
Multimodal biosensing is the simultaneous acquisition and processing of distinct physiological signal modalities to capture a richer, more reliable representation of an organism's biological state. In the context of mental health, this means looking at both the electrical and vascular footprints of the brain. A recent pilot study by Sakurai et al. (2026) implements an end-to-end machine learning framework for depressive state classification by fusing electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). This combination is highly strategic, but bringing such a system into production exposes engineering challenges that go far beyond academic model training.
The physics of the fusion
To understand why we fuse these two specific sensors, we must define what they measure. EEG records the postsynaptic potentials of cortical pyramidal neurons via electrodes placed on the scalp. It possesses exceptional temporal resolution—capturing changes in millisecond intervals—but suffers from poor spatial resolution due to the volume conduction effects of the skull. It is also highly susceptible to muscle and motion artifacts.
In contrast, fNIRS is an optical imaging modality that measures cortical hemodynamic responses. By emitting near-infrared light into the scalp and measuring the backscattered light, it calculates concentration changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR). This vascular response correlates with localized neural activity (neurovascular coupling). While fNIRS provides excellent spatial localization of active cortical regions, its temporal resolution is fundamentally limited; the hemodynamic response operates on a delay of several seconds.
By combining them, we build a system that covers both temporal and spatial gaps. EEG tells us exactly when the brain is responding, and fNIRS tells us precisely where the metabolic activity is occurring. In practice, this dual-window is essential for identifying latent depressive states that may go unnoticed by patients themselves, or for differentiating depression from early-stage dementia in aging populations where clinical symptoms frequently overlap Sakurai et al. (2026).
The production bottleneck: asymmetric temporal alignment
When you transition from a static dataset to a live, production-grade system, the first architectural wall you hit is temporal alignment. EEG and fNIRS operate on entirely different sampling frequencies. A standard clinical or wearable EEG device streams data at 250 Hz to 1000 Hz. A portable fNIRS headband, however, typically samples at 1 Hz to 10 Hz.
In an end-to-end machine learning pipeline, you cannot simply concatenate these vectors. If your system relies on early fusion (combining features at the input layer), you must downsample the EEG features or upsample the fNIRS signals. Downsampling EEG risks discarding critical high-frequency wave bands (such as beta and gamma waves, which are vital for emotion and cognitive load classification). Upsampling fNIRS via interpolation introduces synthetic temporal dependencies that do not reflect physical hemodynamic realities.
An applied solution is feature-level temporal windowing. Instead of raw signal fusion, we extract statistical, spectral, and connectivity features over synchronized sliding windows. For EEG, this means calculating power spectral density (PSD) across specific bands (alpha, beta, theta) over a two-second window. For fNIRS, we calculate the slope, mean, and variance of HbO and HbR over that same window. These engineered feature vectors, now temporally matched, are fed into the classification model. This architecture maintains the integrity of both physical signals while keeping inference latency within acceptable bounds.
Real-time artifact rejection
In a laboratory pilot study, subjects sit still in dark, quiet rooms. In a clinical production environment, patients move, blink, swallow, and speak. Every blink generates an electrooculographic (EOG) spike in the frontal EEG channels that is orders of magnitude larger than the underlying neural signals. Every jaw clench introduces massive high-frequency electromyographic (EMG) noise.
In my experience building the BioVR platform—where we stream real-time biosensor telemetry to dynamically adapt cognitive exercise difficulty—the primary engineering bottleneck is never the model's accuracy on static validation sets. The bottleneck is the pipeline's resilience to real-world noise. If your pipeline does not filter out motion artifacts before inference, your machine learning model will classify muscle tension and blink rates, not depressive states.
To solve this in a production pipeline, we must implement online artifact rejection. This requires running a fast, lightweight preprocessing loop before inference. We use a combination of bandpass filtering (e.g., 0.5–45 Hz for EEG) and online Independent Component Analysis (ICA) or Artifact Subspace Reconstruction (ASR) to isolate and subtract ocular and muscular components. For fNIRS, we must apply a moving average filter or bandpass filter (0.01–0.1 Hz) to eliminate cardiac pulsation (typically ~1 Hz) and respiration noise (typically ~0.2 Hz). This preprocessing must happen in a low-latency, streaming pipeline—often built with specialized serialization protocols over TCP—to ensure the classification output remains responsive.
The trade-off of calibration
Every human brain is physically different. Variations in skull thickness, hair density, and baseline metabolic rates mean that a model trained on a general population will perform poorly on a new individual. This is the problem of covariate shift.
The academic approach often glosses over this by reporting cross-validation accuracy on a small, homogenous cohort. The applied engineering approach requires a strict decision on calibration. Do we deploy a subject-independent model that works immediately out of the box with lower accuracy, or do we require a 10-minute calibration session to train a subject-specific model?
For clinical applications, a hybrid transfer learning approach is the most viable compromise. We can train a deep neural network on a large, multi-subject database to learn generalizable, hierarchical representations of neurovascular coupling. When a new patient sits down, we run a short, 3-minute resting-state baseline task. We use this calibration data to fine-tune only the top classification layers of our model. This minimizes patient setup time while maximizing diagnostic accuracy, finding the sweet spot between clinical utility and technical precision.
Sources
- End-to-End Machine Learning for Depressive State Classification via EEG and fNIRS — arXiv · q-bio.NC · 2026-06-10