Apparently, when we listen to someone we achieve quite a bit of listening through our eyes and 'reading' of the mouth and face, or so says my current reading of Becoming Fluent. A demonstration of the effect of seeing on hearing is quite impressive through something called the McGurk Effect*, where we watch someone saying a series of sounds and then listen to the same track but with our eyes closed. Try it, play this short video while watching the person closely:
Now play the same video but close your eyes:
Most people will hear: da da da da when you watch the mouth and ba ba ba ba when you listen only to the video.
What is going on? The audio track is not from the movements. He is actually making the mouth movements for ga ga ga ga but the brain corrects with the visual information. We learn from an early age to use visual information in deciphering certain sounds.
Now that the effect is described, do you think you mind will process the info differently? Not likely - try it again.
This is a deeply learned effect.
So, here is my question to the smart people of the forum - how do you think these type of clues are processed or mis-processed with long periods of study of a new language via dubbed films where the video and the sound track don't match?
My guess is that two things are going on:
a) you aren't getting the visual clues that help differentiate the sounds being made, when you are most uncertain about how something is being pronounced you are likely to watch intently to get a better read but the information your mind seeks won't be there.
b) you may be untraining your capacity to visually differentiate, spending a lot of time with visual input that doesn't match the language may impact your discernment capacity in real life.
As I now love to use L2 dubbed material (with L2 subtitle too) I'm trying to be careful to not look too much at the lips of someone when they are speaking. Reading a subtitle may help.
*references for this effect:
https://www.nature.com/nature/journal/v ... 746a0.html
https://en.wikipedia.org/wiki/McGurk_effect
People of all languages rely to some extent on visual information in speech perception, but the intensity of the McGurk effect can change between languages. Dutch, English, Spanish, German and Italian language listeners experience a robust McGurk effect, while it is weaker for Japanese and Chinese listeners. Most research on the McGurk effect between languages has been conducted between English and Japanese. There is a smaller McGurk effect in Japanese listeners than in English listeners. The cultural practice of face avoidance in Japanese people may have an effect on the McGurk effect, as well as tone and syllabic structures of the language. This could also be why Chinese listeners are less susceptible to visual cues, and similar to Japanese, produce a smaller effect than English listeners. Studies have also shown that Japanese listeners do not show a developmental increase in visual influence after the age of six, as English children do.
Japanese listeners are more able to identify an incompatibility between the visual and auditory stimulus than English listeners are. This result could be in relation to the fact that in Japanese, consonant clusters do not exist. In noisy environments where speech is unintelligible, however, people of all languages resort to using visual stimuli and are then equally subject to the McGurk effect. The McGurk effect works with speech perceivers of every language for which it has been tested.