Considerations on using dubbed material and the McGurk Effect

General discussion about learning languages
User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2016
Joined: Sun Jul 26, 2015 7:21 pm
Location: Germany and France
Languages: Spanish, English, French trilingual - actively studying German (B2/C1), Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 4987
Contact:

Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 7:49 pm

The McGurk Effect is not a sexy pickle.

Apparently, when we listen to someone we achieve quite a bit of listening through our eyes and 'reading' of the mouth and face, or so says my current reading of Becoming Fluent. A demonstration of the effect of seeing on hearing is quite impressive through something called the McGurk Effect*, where we watch someone saying a series of sounds and then listen to the same track but with our eyes closed. Try it, play this short video while watching the person closely:



Now play the same video but close your eyes:



Most people will hear: da da da da when you watch the mouth and ba ba ba ba when you listen only to the video.

What is going on? The audio track is not from the movements. He is actually making the mouth movements for ga ga ga ga but the brain corrects with the visual information. We learn from an early age to use visual information in deciphering certain sounds.

Now that the effect is described, do you think you mind will process the info differently? Not likely - try it again.
This is a deeply learned effect.



So, here is my question to the smart people of the forum - how do you think these type of clues are processed or mis-processed with long periods of study of a new language via dubbed films where the video and the sound track don't match?

My guess is that two things are going on:

a) you aren't getting the visual clues that help differentiate the sounds being made, when you are most uncertain about how something is being pronounced you are likely to watch intently to get a better read but the information your mind seeks won't be there.

b) you may be untraining your capacity to visually differentiate, spending a lot of time with visual input that doesn't match the language may impact your discernment capacity in real life.

As I now love to use L2 dubbed material (with L2 subtitle too) I'm trying to be careful to not look too much at the lips of someone when they are speaking. Reading a subtitle may help.

*references for this effect:
https://www.nature.com/nature/journal/v ... 746a0.html
https://en.wikipedia.org/wiki/McGurk_effect

People of all languages rely to some extent on visual information in speech perception, but the intensity of the McGurk effect can change between languages. Dutch, English, Spanish, German and Italian language listeners experience a robust McGurk effect, while it is weaker for Japanese and Chinese listeners. Most research on the McGurk effect between languages has been conducted between English and Japanese. There is a smaller McGurk effect in Japanese listeners than in English listeners. The cultural practice of face avoidance in Japanese people may have an effect on the McGurk effect, as well as tone and syllabic structures of the language. This could also be why Chinese listeners are less susceptible to visual cues, and similar to Japanese, produce a smaller effect than English listeners. Studies have also shown that Japanese listeners do not show a developmental increase in visual influence after the age of six, as English children do.
Japanese listeners are more able to identify an incompatibility between the visual and auditory stimulus than English listeners are. This result could be in relation to the fact that in Japanese, consonant clusters do not exist. In noisy environments where speech is unintelligible, however, people of all languages resort to using visual stimuli and are then equally subject to the McGurk effect. The McGurk effect works with speech perceivers of every language for which it has been tested.
15 x
Tagged posts: Language Method Resource
Please feel free to correct me in any language, critique my posts, challenge my thoughts.
I am inconsistency incarnate.
Go study! Publisher of Syriac, Aramaic, Hebrew alphabet apps at http://alphabetsnow.zyntx.com

User avatar
Xenops
Brown Belt
Posts: 1047
Joined: Mon Nov 30, 2015 10:33 pm
Location: Boston
Languages: English (N), Japanese (approx. N5), Nansha (constructing).
x 1931
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Xenops » Sun Jun 25, 2017 8:44 pm

I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)
0 x

User avatar
LesRonces
Green Belt
Posts: 329
Joined: Mon May 15, 2017 7:48 am
Languages: FR B1¾
x 539

Re: Considerations on using dubbed material and the McGurk Effect

Postby LesRonces » Sun Jun 25, 2017 9:05 pm

Wow that's insane. I got dadada and bababa.
2 x

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2016
Joined: Sun Jul 26, 2015 7:21 pm
Location: Germany and France
Languages: Spanish, English, French trilingual - actively studying German (B2/C1), Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 4987
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 9:09 pm

Xenops wrote:I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)


Certainly you won't expect that type of effect with anime - but do the Japanese watch anime from birth?
I would think it has more to do with differentiating phoneme acquisition (absence of consonant clusters) such as the /l/ and /r/ issues...
1 x
Tagged posts: Language Method Resource
Please feel free to correct me in any language, critique my posts, challenge my thoughts.
I am inconsistency incarnate.
Go study! Publisher of Syriac, Aramaic, Hebrew alphabet apps at http://alphabetsnow.zyntx.com

vonPeterhof
Blue Belt
Posts: 532
Joined: Sat Aug 08, 2015 1:55 am
Languages: Russian (N), English (C2), Japanese (~C1), German (~B2), Kazakh (~B1), Norwegian (~A2)
Studying: Thai, Sanskrit, Akkadian, Turoyo, Polish
Language Log: viewtopic.php?f=15&t=1237
x 1625
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby vonPeterhof » Sun Jun 25, 2017 9:26 pm

zenmonkey wrote:
Xenops wrote:I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)


Certainly you won't expect that type of effect with anime - but do the Japanese watch anime from birth?
I would think it has more to do with differentiating phoneme acquisition (absence of consonant clusters) such as the /l/ and /r/ issues...

I'm reminded of a conversation I had with my (now ex-)cousin-in-law about her and my cousin's daughter, who wouldn't speak a single word well past the age of three and would instead communicate via elaborate pantomime. My in-law blamed it on her watching so many cartoons, the majority of which were dubbed from other languages, thus making it harder to associate shapes of the mouth with the sounds of speech (and even in Russian cartoons it's not very common to carefully imitate mouth shapes to match the dialogue). She didn't use the term "McGurk Effect", but since she's a psychology major I believe she may have been aware of it. I thought she was exaggerating the issue, but I didn't really have any data to refute her hypothesis. Anyway, the girl did start speaking at some point either before or shortly after turning four and the last time I saw her she spoke very fluently.

As a more relevant response, I don't think there's a huge difference in the amount of animation Japanese and American kids watch (I'd also argue that the popularity of anime among Japanese adults also tends to be exaggerated, but that's a whole other topic). The "face avoidance" and phonemic structure explanations make more sense to me.
3 x

User avatar
PeterMollenburg
Black Belt - 2nd Dan
Posts: 2503
Joined: Wed Jul 22, 2015 11:54 am
Location: Australia
Languages: English (N), French (B2-certified), Dutch (High A2?), Spanish (~A1), German (long-forgotten 99%), Norwegian (just started in 2020)
Language Log: https://forum.language-learners.org/vie ... 15&t=12510
x 5328

Re: Considerations on using dubbed material and the McGurk Effect

Postby PeterMollenburg » Sun Jun 25, 2017 9:43 pm

I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.

Edit: I replayed these videos, and apparently I'm normal ;) it's da da da da on the first and ba ba ba ba on the second for me too.
Last edited by PeterMollenburg on Sun Jun 25, 2017 9:48 pm, edited 1 time in total.
0 x

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2016
Joined: Sun Jul 26, 2015 7:21 pm
Location: Germany and France
Languages: Spanish, English, French trilingual - actively studying German (B2/C1), Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 4987
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 9:46 pm

PeterMollenburg wrote:I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.


It's the same video (posted twice).
Watch it once, paying attention with eyes open.
Watch it a second time, with eyes closed.

Order doesn't matter.

What did you hear each time?
1 x
Tagged posts: Language Method Resource
Please feel free to correct me in any language, critique my posts, challenge my thoughts.
I am inconsistency incarnate.
Go study! Publisher of Syriac, Aramaic, Hebrew alphabet apps at http://alphabetsnow.zyntx.com

User avatar
PeterMollenburg
Black Belt - 2nd Dan
Posts: 2503
Joined: Wed Jul 22, 2015 11:54 am
Location: Australia
Languages: English (N), French (B2-certified), Dutch (High A2?), Spanish (~A1), German (long-forgotten 99%), Norwegian (just started in 2020)
Language Log: https://forum.language-learners.org/vie ... 15&t=12510
x 5328

Re: Considerations on using dubbed material and the McGurk Effect

Postby PeterMollenburg » Sun Jun 25, 2017 9:48 pm

zenmonkey wrote:
PeterMollenburg wrote:I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.


It's the same video (posted twice).
Watch it once, paying attention with eyes open.
Watch it a second time, with eyes closed.

Order doesn't matter.

What did you hear each time?


see my edit above
0 x

User avatar
Steve
Yellow Belt
Posts: 79
Joined: Wed Jan 06, 2016 7:53 am
Location: US (Wisconsin)
Languages: English(N), Spanish(B1), German(A2), Ancient Greek(B1 to B2), ASL(beginner), Hebrew(beginner), Chinese(beginner)
x 369

Re: Considerations on using dubbed material and the McGurk Effect

Postby Steve » Sun Jun 25, 2017 10:04 pm

Interesting. When watching, I was hearing 3 different pairs of sounds. The first two of which kept shifting on me and the final pair being a clear da-da. I tried this several times.

I was unable to lock onto the first two pairs of sounds. It was like they kept shifting on me and changing each time I listened. I was hearing combinations of different consonants sort of like ga-la, la-ga, ga-ka, etc. But it was like there was a dissonance I clearly felt that something was not right. I kept hearing strange consonants best described as l, g, k, and t (weird hearing voiced/unvoiced consonants). The first two pairs never sounded like da-da to me.

Listening without watching was always a clear ba-ba.
1 x

User avatar
Serpent
Black Belt - 3rd Dan
Posts: 3520
Joined: Sat Jul 18, 2015 10:54 am
Location: Moskova
Languages: heritage
Russian (native); Belarusian, Polish

fluent or close: Finnish+ (certified C1), English; Portuguese, Spanish, German+, Italian+
learning: Croatian+, Ukrainian, Czech; Romanian+, Galician; Danish, Swedish
exploring: Latin, Karelian, Catalan, Dutch, Chaucer's English
+ means exploring the dialects/variants
x 4718
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Serpent » Sun Jun 25, 2017 10:50 pm

Honestly I'm so used to the lips not matching in dubbed L1 movies that I simply don't expect them to match.

I definitely didn't hear da the first time, it sounded like a weird non-English consonant a few times and then the last one was a clear ba. With my eyes closed it was a mix of ba and argh I don't remember, ga? When trying again I did hear dada with my eyes open.
1 x
LyricsTraining now has Finnish and Polish :)
Corrections welcome


Return to “General Language Discussion”

Who is online

Users browsing this forum: dicentra8, munyag and 2 guests