Abstract
Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain–speech alignment grounds neural activity to a single anchor modality—audio or text—despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail.
We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio–text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.
Method
Fig. 1. Overview of BAT-CLIP. A trainable brain encoder fθ, initialized from the brain foundation model DIVER-1, maps neural windows to brain embeddings zB. Frozen AudioCLIP encoders provide audio anchors zA and text anchors zT. BAT-CLIP optimizes a trimodal contrastive objective LTotal to align neural embeddings with the joint audio–text representation space.
- Alignment prior. The brain encoder is initialized from DIVER-1, an iEEG foundation model pretrained with self-supervised masked reconstruction, then adapted with a lightweight two-layer MLP projection to a shared d=1024 space with ℓ2 normalization. The pretrained weights act as a prior that curbs overfitting on the ~30 min of paired brain–audio–text data available for alignment.
- Shared audio–linguistic space. Audio waveforms and transcripts are encoded by the pretrained AudioCLIP audio/text encoders, kept frozen so the brain encoder matches a stable, well-regularized manifold in the low-data regime.
- Trimodal contrastive alignment. Symmetric Brain–Audio and Brain–Text cross-entropy losses are combined with equal weight into LTotal = ½(LBA + LBT), aligning brain embeddings to both anchors at once.
Results
Overall decoding performance
| Method | Type | Content | Onset | POS | GPT Surprisal | Word Emb | Whisper | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AUC | Δ | AUC | Δ | AUC | Δ | AUC | Δ | AvgAUC | Δ | Acc | Δ | ||
| CNN (Baseline)† | Sup | 0.5498 | – | 0.7519 | – | 0.5653 | – | 0.5019 | – | 0.6260 | – | 0.6634 | – |
| BrainBERT | SSL | 0.5505 | +0.13% | 0.6341 | -15.67% | 0.5325 | -5.80% | 0.5157 | +2.75% | 0.5533 | -11.61% | 0.5805 | -12.49% |
| PopT | SSL | 0.4950 | -9.97% | 0.4654 | -38.10% | 0.5031 | -11.00% | 0.5022 | +0.06% | 0.5002 | -20.10% | 0.5014 | -24.42% |
| DIVER-1 | SSL | 0.5645 | +2.67% | 0.7729 | +2.79% | 0.5656 | +0.05% | 0.5155 | +2.71% | 0.6450 | +3.04% | 0.7022 | +5.85% |
| BA-CLIP | Align | 0.5840 | +6.31% | 0.6560 | -12.61% | 0.5673 | +0.32% | 0.5169 | +3.02% | 0.6468 | +4.26% | 0.7141 | +7.75% |
| BT-CLIP | Align | 0.5813 | +5.82% | 0.6562 | -12.56% | 0.5709 | +0.97% | 0.5203 | +3.69% | 0.6510 | +4.94% | 0.7157 | +7.98% |
| BAT-CLIP (Ours) | Align | 0.5819 | +5.96% | 0.6589 | -12.18% | 0.5740 | +1.50% | 0.5229 | +4.21% | 0.6512 | +5.01% | 0.7162 | +8.09% |
Table 1. Overall decoding performance averaged over 9 subjects. Δ is the relative gain over the supervised CNN baseline. †CNN uses full fine-tuning; all other methods use linear probing. Bold = best, underline = second-best per column. BAT-CLIP is the best aligned model on 5 of 6 probe tasks, and lifts most probes above the unaligned DIVER-1 backbone (Onset excepted).
Effect of SSL pretraining (initialization)
| Method | Init. | Content | Onset | POS | GPT Surprisal | Word Emb | Whisper | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AUC | Δ | AUC | Δ | AUC | Δ | AUC | Δ | AvgAUC | Δ | Acc | Δ | ||
| BA-CLIP | Scratch | 0.5671 | +3.14% | 0.7792 | +3.75% | 0.5624 | -0.35% | 0.5143 | +2.55% | 0.6059 | -2.33% | 0.6609 | +0.05% |
| DIVER-1 | 0.5840 | +6.31% | 0.6560 | -12.61% | 0.5673 | +0.32% | 0.5169 | +3.02% | 0.6468 | +4.26% | 0.7141 | +7.75% | |
| BT-CLIP | Scratch | 0.5650 | +2.85% | 0.7822 | +4.19% | 0.5635 | -0.18% | 0.5120 | +2.10% | 0.6296 | +1.52% | 0.6607 | -0.02% |
| DIVER-1 | 0.5813 | +5.82% | 0.6562 | -12.56% | 0.5709 | +0.97% | 0.5203 | +3.69% | 0.6510 | +4.94% | 0.7157 | +7.98% | |
| BAT-CLIP | Scratch | 0.5586 | +1.70% | 0.7600 | +0.94% | 0.5480 | -2.94% | 0.5119 | +2.09% | 0.6123 | -1.32% | 0.6463 | -2.36% |
| DIVER-1 | 0.5819 | +5.96% | 0.6589 | -12.18% | 0.5740 | +1.50% | 0.5229 | +4.21% | 0.6512 | +5.01% | 0.7162 | +8.09% | |
Table 2. CLIP-aligned models trained from scratch vs. initialized from pretrained DIVER-1. Bold marks the stronger initialization per column within each method. Pretrained DIVER-1 initialization is key to realizing the gains from trimodal alignment; onset is the exception, where timing-critical transient cues favor training from scratch.
Embedding-space clustering
| Method | Cosine Coherence (↑) | Separation Ratio (↑) |
|---|---|---|
| DIVER-1 (SSL) | 0.0004 | 0.95 |
| BA-CLIP | 0.0789 | 1.09 |
| BT-CLIP | 0.0954 | 1.04 |
| BAT-CLIP (Ours) | 0.1572 | 1.23 |
Table 3. Sentence-level clustering of the word embeddings in Fig. 2 (Sub03). Higher is better; BAT-CLIP yields the tightest intra-sentence grouping and strongest inter-sentence separation.
BibTeX
@inproceedings{kim2026batclip,
title = {BAT-CLIP: Trimodal Alignment of Brain, Audio and Text},
author = {Kim, Suhyun and Han, Jinmo and Han, Danny Dongyeop and
Lee, Ahhyun Lucy and Lee, Jewoon and Gwon, Yonghyeon and
Paris, Zach and Chung, Chun Kee and Bahk, Saewoong and
Kim, Nam Soo and Hwang, Seong Jae and Cha, Jiook},
booktitle = {IEEE International Workshop on Machine Learning for Signal Processing (MLSP)},
year = {2026}
}