McGurk effect

From Wikipedia, the free encyclopedia
Jump to: navigation, search

The McGurk effect shows how hearing and vision are used for speech perception. Named after the man who found it, Harry McGurk, it says that people do not hear speech with only their ears, they use their other senses too. The McGurk effect may be experienced when watching a video of a person saying /ga/ with a sound-recording saying /ba/. When this is done, a third sound is heard: /da/.

The McGurk effect is robust: that is, it still works even if a person know about it. This is different from certain optical illusions, which do not work anymore once a person can see it.

Overview[change | change source]

The McGurk effect describes a phenomenon that shows how speech perception is not dependent solely on auditory information. Visual information in the form of reading the lips is also taken into account and combined with the auditory information that is heard to produce the final stimuli perceived. This can get particularly interesting when the auditory information of one sound, paired with the spoken lips of another sound, ultimately combine to form the perception of a third different sound.[1]

Explanation[change | change source]

When humans perceive speech, they not only take in auditory information but also visual information as well in the form of reading lips, facial expression, and other body bodily cues. Usually, these two sources of information are consistent with each other so the brain simply combines them to form one unified perceived stimuli. When the McGurk effect is tested and the incoming information from the ears and eyes differ, the brian tries to make sense of the contradictory stimuli which ultimately results in a fusion of both.[2] In humans, information received from the eyes dominates other sensory modalities, including audition, so for instance when 'ba' is heard and 'ga' is seen, the resulting stimuli is heard is 'da'.[3] The resulting stimuli is what happens when the brain tries to make sense of the two different sets of information.

History[change | change source]

Harry McGurk[change | change source]

Initially, Harry McGurk's (February 23 1936 – April 17 1998) field of psychological research was human development, focusing especially on infant behavior. In the early 1970's, interested in how infants perceived the world, he ran numerous infant perception studies. Specifically, his early work dealt with unimodal visual perception by infants.[4] He found that infants sensory information was "combined cross-modally as the result of operations on the environment or more generally, experience."[5] At the same time, two researchers by the name of Aronson and Rosenbloom published a paper in Science stating that infants' spatial perception occurs within a common auditory-visual space. This view by Aronson and Rosenbloom was for the most part consistent with the currently accepted view at the time which was that all the senses were essentially unified as one. This view though, contradicted what McGurk had found which was that "the unity of the senses, especially in speech, is a product of experience." In response to Aronson and Rosenbloom's difference in findings, McGurk and Macdonald decided to run a similar test in 1974 which tested 1-, 4-, and 7-month old infants in a similar fashion. Once again, the results from McGurk and Macdonalds' experiment did not confirm Aronson and Rosenbloom's findings.[6]

What McGurk and Macdonald decided to do next though set the stage for the accidental discovery of what is now known as the McGurk effect. After rejecting Aronson and Rosenbloom's ideas about intermodal spatial dislocation, they decided to test the unity of the senses hypothesis by pitting the senses against each other.[5]

"As documented in the accompanying address, McGurk took videotapes of productions of /ba/ and /ga/ and had them dubbed to produce not just the matching ba-voice/ba-lips, and the ga-voice/ga-lips sequences, but also conflicting ba-voice/ga-lips and ga-voice/ba-lips sequences. What is not documented in McGurk’s address is his displeasure when the videotapes returned from the auditory-visual centre at Surrey. Harry said he thought the technician had made a mistake, and that he told the technician so in no uncertain terms. He soon realized however that there had been no mistake and that he was experiencing something quite remarkable, a “da” percept when presented with an auditory [ba] and visual [ga], and a “bga” percept from an auditory [ga] and visual [ba]." [5] [7]

After this monumental realization in 1976, McGurk and Macdonald decided to scrap their intermodal conflict studies with infants and instead publish their findings of visual speech perception in a paper titled "Hearing Lips and Seeing Voices" in the journal Nature.[8]

Similar findings by others[change | change source]

Interestingly enough, around the same time that Harry McGurk and John Macdonald discovered what is now known as the McGurk effect, another British researcher also stumbled upon this phenomenon. Barbara Dodd discovered a similar effect with audio-visual speech interpretation but instead it was with the visual cue of 'hole' and the audio cue of 'tough' which ultimately generated the audio perception of 'towel'.[9] In 1987 Barbara Dodd along with Ruth Campbell then published these findings along with other contributions from researchers also studying audio-visual speech perception in an edited book titled Hearing by Eye: The Psychology of Lip-Reading.[10] These discoveries in audio-visual speech perception ultimately changed the way scientists and researchers view the interaction of different senses in the brain.

Infants[change | change source]

Infants also show signs similar to the McGurk effect. Obviously you cannot ask the infant what they hear since they cannot verbally communicate yet but by measuring certain variables such as their attention to audiovisual stimuli, effects similar to the McGurk effect can be seen. Very soon after infants are born, sometimes even within minutes of birth, they are able to imitate adult facial movements; an important first step in audiovisual speech perception. Next comes the ability to recognize lip movements and speech sounds a couple of weeks after birth. Evidence of the McGurk effect is not visible though until about 4 months of age,[11][12] with a much stronger presence at around 5 months after birth.[13][14] To test this effect on infants, infants are first habituated to a stimuli. Once the stimuli gets changed, the infant exhibits an effect similar to the McGurk effect. As infants grow older and continue to develop, the McGurk effect also becomes more prominent as visual cues start to override purely auditory information in audiovisual speech perception.

Effect in other languages and cultures[change | change source]

Although the McGurk effect has been primarily studied in English because of it's origins in English speaking countries, research has now spread to others countries with different languages as well. In particular, the comparison between English and Japanese has been prominent. Research has shown that the McGurk effect is much more prominent in English listeners compared to Japanese listeners.[15] One strong hypothesis for this is the difference between cultures and how each culture behaves and interacts. Japanese culture is notable for being politeness and avoiding direct eye or face contact when interacting.[16]

This phenomenon has also been studied in French Canadian children and adults with similar findings. When compared to adults though, children tend to show less susceptibility to the McGurk effect since their primary sense of speech perception is dominated by auditory information. This is evident in children scoring lower on lip reading tasks when compared to adults. Nevertheless, the McGurk effect was present in certain contexts but the effects were much more variable than when the tests were run on adults.[17]

Broader impact on society[change | change source]

Although the McGurk effect's importance may seem isolated to just psychological researchers and scientists, this phenomenon has been expanded to everyday audio and visual speech perception as well. Two researchers by the name of Wareham and Wright conducted a study in 2005 that may suggest that the McGurk effect can influence how everyday speech is perceived. This is especially important in witness testimony where the observations and accounts of the witness are usually expected to be accurate and correct. With this information, witness testimonies now must be interpreted with the notion that the witness may be unaware of their own inaccurate perception.[18]

References[change | change source]

  1. Nath, AR.; Beauchamp, MS. (Jan 2012). "A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion.". Neuroimage 59 (1): 781-7. doi:10.1016/j.neuroimage.2011.07.024 . PMC 3196040 . PMID 21787869 .
  2. O’Shea, M. (2005). The Brain: A Very Short Introduction. Oxford University Press
  3. Colin, C., Radeau, M. & Deltenre, P. (2011). Top-down and bottom-up modulation of audiovisual integration in speech. European Journal of Cognitive Psychology, 17(4), 541-560
  4. McGurk, H. Infant discrimination of orientation. Journal of Experimental Child Psychology, 14, 151-164.
  5. 5.0 5.1 5.2 "HARRY McGURK AND THE McGURK EFFECT". http://www.isca-speech.org/archive_open/archive_papers/avsp98/av98_001.pdf. Retrieved 2013-11-27.
  6. McGurk, H., & Lewis, M. (1974) Space perception in early infancy: Perception within a common auditory space? Science, 186, 649-650.
  7. McGurk, H., (1988) Developmental Psychology and the Vision of Speech. Inaugural Professorial Lecture, University of Surrey
  8. McGurk, H., & MacDonald J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
  9. Dodd, B. (1977). The role of vision in the perception of speech. Perception, 6, 31-40.
  10. Dodd, B. & Campbell, R. (Eds.) (1987) Hearing by eye: The psychology of lip-reading. Hillsdale, New Jersey: Lawrence Erlbaum.
  11. Bristow, D., Dehaene-Lambertz, G., Mattout, J., Soares, C., Gliga, T., Baillet, S. & Mangin, J.F. (2009). Hearing faces: How the infant brain matches the face it sees with the speech it hears. Journal of Cognitive Neuroscience, 21(5), 905-921
  12. Burnham, D. & Dodd, B. (2004). Auditory-Visual Speech Integration by Prelinguistic Infants: Perception of an Emergent Consonant in the McGurk Effect. Developmental Psychobiology, 45(4), 204-220
  13. Rosenblum, L.D. (2010). See what I’m saying: The extraordinary powers of our five senses. New York, NY: W. W. Norton & Company Inc.
  14. Rosenblum, L.D., Schmuckler, M.A. & Johnson, J.A. (1997). The McGurk effect in infants. Perception & Psychophysics, 59(3), 347-357
  15. Hisanaga, S., Sekiyama, K., Igasaki, T. & Murayama, N. (2009). Audiovisual speech perception in Japanese and English: Inter-language differences examined by event-related potentials. Retrieved from http://www.isca-speech.org/archive_open/avsp09/papers/av09_038.pdf
  16. Sekiyama, K. (1997). Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception and Psychophysics 59(1), 73-80
  17. [http://www.zas.gwz-berlin.de/fileadmin/material/ZASPiL_Volltexte/zp40/zaspil40-dupont-menard-aubin.pdf "STUDIES OF THE MCGURK EFFECT: IMPLICATIONS FOR THEORIES OF SPEECH PERCEPTION"]. http://www.zas.gwz-berlin.de/fileadmin/material/ZASPiL_Volltexte/zp40/zaspil40-dupont-menard-aubin.pdf. Retrieved 2013-11-27.
  18. Schmid, G., Thielmann, A. & Ziegler, W. (2009). The influence of visual and auditory information on the perception of speech and non-speech oral movements in patients with left hemisphere lesions. Clinical Linguistics and Phonetics, 23(3), 208-221

Other websites[change | change source]

Video examples