| Title:
|
Language-Model Surprisal and Human N400 Responses: A Reproducible Multi-Dataset EEG Evaluation
|
| Author(s):
|
1
|
| Subject:
|
Artificial Intelligence and Brain Mapping
|
|
|
|
Presenter Author:
|
|
| Name:
|
Saleh EstakiOrgani
|
|
|
|
Author:
|
|
| Name 1:
|
Saleh EstakiOrgani - Corresponding - Presenter
|
| Affiliation 1:
|
School of Behavioral Sciences and Mental Health ,Iran University of Medical Sciences, Tehran, Iran
|
|
|
| Name 2:
|
|
| Affiliation 2:
|
|
|
|
| Name 3:
|
|
| Affiliation 3:
|
|
|
|
| Name 4:
|
|
| Affiliation 4:
|
|
|
|
| Name 5:
|
|
| Affiliation 5:
|
|
|
|
| Name 6:
|
|
| Affiliation 6:
|
|
|
|
| Name 7:
|
|
| Affiliation 7:
|
|
|
|
| Name 8:
|
|
| Affiliation 8:
|
|
|
|
| Name 9:
|
|
| Affiliation 9:
|
|
|
|
| Name 10:
|
|
| Affiliation 10:
|
|
|
|
Abstract
|
Original Research
|
|
introduction :
|
Language-model surprisal is frequently compared with the human N400 response, but parallel experimental effects, predictive alignment, and shared neural mechanisms are often conflated. We developed a reproducible framework to separate these questions and evaluated whether causal language-model prediction measures show controlled semantic sensitivity and explain held-out variation in human electroencephalographic responses.
|
|
Methods:
|
We conducted secondary analyses of two public, de-identified EEG datasets. In ERP CORE, all 39 available participants were processed using a versioned automated pipeline, with 34 retained under primary quality-control rules. The human semantic effect was defined as the unrelated-minus-related mean voltage at CPz from 300 to 500 ms. Separately, the same 100 prime–target pairs were scored using exact teacher-forced probabilities from pinned SmolLM2-135M, GPT-2, and Qwen2.5-0.5B models. Naturalistic alignment was evaluated using DERCo article 0, comprising 9,029 word-level observations from 20 participants and 559 scored items. Crossed Bayesian models and participant- and item-held-out prediction were used to estimate conditional associations while controlling for lexical and human-derived predictors.
|
|
Results:
|
Unrelated targets elicited a larger, more-negative N400: −3.669 µV, 95% confidence interval [−4.319, −3.019]. Unrelated primes also increased target surprisal by 3.057 nats [2.355, 3.759] for SmolLM2, 1.628 [1.127, 2.129] for GPT-2, and 3.517 [2.606, 4.428] for Qwen2.5. In DERCo, standardized language-model surprisal showed a conditional association of −0.215 µV per standard deviation, 95% highest-density interval [−0.369, −0.055]. However, held-out improvement was only approximately 0.015 µV in root-mean-square error, and mean held-out R² remained near zero.
|
|
Conclusion:
|
Language models and human EEG showed directionally parallel sensitivity to semantic context, and model surprisal exhibited a statistically visible conditional association with N400-window voltage. Nevertheless, practical held-out prediction was weak. These findings support carefully bounded predictive alignment, not a shared mechanism or a causal language-model-to-brain relationship.
|
| Attachment:
|
|
| Keywords:
|
Electroencephalography; Evoked Potentials; Artificial Intelligence; Large Language Models; Natural Language Processing; Humans
|
| Key message:
|
|
| Do you want your abstract to be indexed, if this is selected as one of the top 100
articles?
|
