Study Finds Deep Biases in Standard Measures of Social Cognition

According to a recent paper published in Psychological Medicine, one of the most widely used tests for measuring social cognition is biased toward well-educated Caucasians. Laura Germine, PhD, technical director of the McLean Institute for Technology in Psychiatry and director of the Laboratory for Brain and Cognitive Health Technology, said that the study looked at the impact of social, cultural, and demographic factors on the Reading the Mind in the Eyes Test (RMET), a test designed to assess the human ability to understand other people’s mental states. She and her colleagues found that the RMET “may not be an accurate measure of social cognitive abilities in non-white or less-educated groups.”

The paper, “Social Cognition or Social Class and Culture? On the Interpretation of Differences in Social Cognitive Performance,” was co-authored by Germine, David Dodell-Feder, PhD, of the University of Rochester, and Kerry J. Ressler, MD, PhD, McLean Hospital’s chief scientific officer.

The researchers examined data—collected from TestMyBrain.org, Germine’s online research platform—from over 40,000 people. The results were from on an online version of the RMET, a test that asks participants to look at the eye region of a series of faces and determine the emotions expressed in those faces. The RMET is used to test social cognition in people with conditions such as autism and schizophrenia, as well as in other populations. This data was compared to data on other tests of social and non-social cognition.

Germine said that many who work in the field of social cognition have concerns about the RMET because of a lack of diversity in the images and use of complex vocabulary. “The majority of faces in the test are of white Northern Europeans,” she said. “The women are heavily made up and are mostly young, while the men span a wide range of ages.” Also, she said, the words used to describe the facial expressions in the test—words such as “aghast”—may not be understood by less-educated test takers.

Through an examination of test scores across a diverse range of people, the researchers found that people with non-Caucasian backgrounds and people with lower levels of education showed poor performance on the test. One of the “most concerning” findings for Germine and her colleagues involved RMET results for African/black men. “We found that men who reported African descent had RMET scores that were as low as the scores of people with schizophrenia,” she reported. “The idea that this might reflect a true difference in social cognition is absurd and entirely implausible. These results strongly suggest serious cultural and racial bias in the test.”

Findings such as these are “deeply problematic,” Germine said. “These tests are supposed to be testing social reasoning,” she stated. “They are not supposed to be testing what culture you grew up in, your racial or ethnic background, or how big a vocabulary you have.”

Through the paper, Germine said the researchers are “trying to get the word out that cultural and social diversity are major factors in measuring social cognition, and that results from tests like the RMET should not be taken at face value.”

Based on their findings, the researchers are calling on the research community to design tests that are more diverse and representative. Also, Germine said, “when it comes to designing tests of social cognition and social understanding, the one-size-fits-all approach doesn’t adequately generalize across the broad range of human experiences and social contexts.”