The effect of McGurk: why can your eyes affect what you hear?

When you watch a badly duplicated foreign film, it's very easy to see that the sounds that fly out of the mouths of actors do not match the movements of their lips.

In other words, even when our vision and hearing are stimulated simultaneously during the film, our brain must do a really good job, paying attention to the correspondence of the movement of the lips and sounds.

What is the McGurk effect

But the brain can also be misled. In an intriguing illusion known as the McGurk effect, observing the movements of a person's lips can deceive the brain to hear the wrong sound.

The effect of McGurk occurs when there is a conflict between visual speech, that is, the movement of someone's mouth and lips, and auditory speech, represented by sounds that a person hears. And this can lead to the perception of a completely different message.

Research using a computer model

In a new study, neurologists from the Baylor College of Medicine in Houston tried to offer a quantitative explanation of the causes of the McGurk effect. They developed a computer model that was able to accurately predict when the McGurk effect should or should not occur in humans, according to the results published on February 16 in the journal PLOS Computational Biology.

In demonstrating the McGurk effect used in the study, the participant was asked to leave his eyes closed while listening to a video that shows a person producing his "ba-ba-ba" sounds. Then this participant was asked to open his eyes and carefully look at the person's mouth in the video, but without sound. For the participant, the visual effects looked as if a person were saying "ha-ha-ha." At the last stage of the experiment, the same video was played, but this time the sound was turned on, and the participant was offered to watch it. Sensitive to the effect of McGurk people reported that the third time they heard the sound of "yes-da-da", which did not match either the auditory or visual signals seen earlier.

This is because the brain is trying to decide which, in his opinion, the sound is closer to what he sees visually. When the participant closed his eyes again, and scientists reproduced the sound of the video, he again heard the original sound of "ba-ba-ba".

The effect was first described in an experiment conducted in 1976 by psychologists Harry McGurcom and John McDonald. He showed that the visual information provided by the movements of the mouth can influence and redefine what a person thinks he is hearing.

Predicting Illusions

"The McGurk effect is a powerful, multi-sensory illusion," said co-author John Magnotti, a doctoral student in the Department of Neurosurgery at Baylor. "The brain perceives auditory and visual speech and unites them to form something new."

When people communicate face to face, the brain is engaged in complex activities, as it tries to decide how to combine the movements of the lips with the sounds of speech that they hear, said Magnotti.

In the study, scientists tried to understand why the brain could better fold some syllables to interpret a sound that was heard correctly, while others did not.

Cause-effect conclusion

To understand this, their model relied on an idea known as a cause-effect conclusion, or a process in which the human brain decides whether the sound and visual sounds of speech were produced by the same source. Other researchers have developed models to help predict when the McGurk effect may occur, but this new study is the first to include a causal relationship in its calculation, said Magnotti. Factoring in causal reasoning may have improved the accuracy of the new model over previous ones, which also had to predict illusions.

To test the accuracy of their prediction model, the researchers collected 60 participants and asked them to listen to a pair of auditory and visual speech from one speaker. Then the participants were asked to decide what sound they heard: "ba", "yes" or "ha".

Their results showed that the model developed can reliably predict when most of the participants in the experiment will experience the McGurk effect. But, as expected in their calculation, people were identified who were not susceptible to this, said Magnotti.

It's interesting to note that when the same test was conducted with students in China, and not in the United States, it was proved that the McGurk effect works in other languages.

Practical application of used computer models

Magnotti said that, in his opinion, the computer models developed for this study may also have practical application. For example, this model can be useful for companies that create computers that help to recognize speech, such as Google Home or Amazon Echo.

If these technologies had cameras, they could integrate the movement of the people's lips into what they say to improve the accuracy of their speech recognition systems.

The model can also help children with cochlear implants, improving the researchers' understanding of the mechanism of how visual speech affects what a person hears, said Magnotti.