Considerations on using dubbed material and the McGurk Effect

General discussion about learning languages
User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 7:49 pm

The McGurk Effect is not a sexy pickle.

Apparently, when we listen to someone we achieve quite a bit of listening through our eyes and 'reading' of the mouth and face, or so says my current reading of Becoming Fluent. A demonstration of the effect of seeing on hearing is quite impressive through something called the McGurk Effect*, where we watch someone saying a series of sounds and then listen to the same track but with our eyes closed. Try it, play this short video while watching the person closely:



Now play the same video but close your eyes:



Most people will hear: da da da da when you watch the mouth and ba ba ba ba when you listen only to the video.

What is going on? The audio track is not from the movements. He is actually making the mouth movements for ga ga ga ga but the brain corrects with the visual information. We learn from an early age to use visual information in deciphering certain sounds.

Now that the effect is described, do you think you mind will process the info differently? Not likely - try it again.
This is a deeply learned effect.



So, here is my question to the smart people of the forum - how do you think these type of clues are processed or mis-processed with long periods of study of a new language via dubbed films where the video and the sound track don't match?

My guess is that two things are going on:

a) you aren't getting the visual clues that help differentiate the sounds being made, when you are most uncertain about how something is being pronounced you are likely to watch intently to get a better read but the information your mind seeks won't be there.

b) you may be untraining your capacity to visually differentiate, spending a lot of time with visual input that doesn't match the language may impact your discernment capacity in real life.

As I now love to use L2 dubbed material (with L2 subtitle too) I'm trying to be careful to not look too much at the lips of someone when they are speaking. Reading a subtitle may help.

*references for this effect:
https://www.nature.com/nature/journal/v ... 746a0.html
https://en.wikipedia.org/wiki/McGurk_effect

People of all languages rely to some extent on visual information in speech perception, but the intensity of the McGurk effect can change between languages. Dutch, English, Spanish, German and Italian language listeners experience a robust McGurk effect, while it is weaker for Japanese and Chinese listeners. Most research on the McGurk effect between languages has been conducted between English and Japanese. There is a smaller McGurk effect in Japanese listeners than in English listeners. The cultural practice of face avoidance in Japanese people may have an effect on the McGurk effect, as well as tone and syllabic structures of the language. This could also be why Chinese listeners are less susceptible to visual cues, and similar to Japanese, produce a smaller effect than English listeners. Studies have also shown that Japanese listeners do not show a developmental increase in visual influence after the age of six, as English children do.
Japanese listeners are more able to identify an incompatibility between the visual and auditory stimulus than English listeners are. This result could be in relation to the fact that in Japanese, consonant clusters do not exist. In noisy environments where speech is unintelligible, however, people of all languages resort to using visual stimuli and are then equally subject to the McGurk effect. The McGurk effect works with speech perceivers of every language for which it has been tested.
15 x
I am a leaf on the wind, watch how I soar

User avatar
Xenops
Brown Belt
Posts: 1444
Joined: Mon Nov 30, 2015 10:33 pm
Location: Boston
Languages: English (N), Danish (A2), Japanese (rusty), Nansha (constructing)
On break: Japanese (approx. N4), Norwegian (A2)
Language Log: https://forum.language-learners.org/vie ... 15&t=16797
x 3559
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Xenops » Sun Jun 25, 2017 8:44 pm

I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)
0 x
Check out my comic at: https://atannan.com/

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 9:09 pm

Xenops wrote:I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)


Certainly you won't expect that type of effect with anime - but do the Japanese watch anime from birth?
I would think it has more to do with differentiating phoneme acquisition (absence of consonant clusters) such as the /l/ and /r/ issues...
1 x
I am a leaf on the wind, watch how I soar

vonPeterhof
Blue Belt
Posts: 879
Joined: Sat Aug 08, 2015 1:55 am
Languages: Russian (N), English (C2), Japanese (~C1), German (~B2), Kazakh (~B1), Norwegian (~A2)
Studying: Kazakh, Mandarin, Coptic
Language Log: viewtopic.php?f=15&t=1237
x 2833
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby vonPeterhof » Sun Jun 25, 2017 9:26 pm

zenmonkey wrote:
Xenops wrote:I wonder if anime contributes to the lack of the McGurk effect: it seems that the Japanese watch more animation than Americans do (maybe?)


Certainly you won't expect that type of effect with anime - but do the Japanese watch anime from birth?
I would think it has more to do with differentiating phoneme acquisition (absence of consonant clusters) such as the /l/ and /r/ issues...

I'm reminded of a conversation I had with my (now ex-)cousin-in-law about her and my cousin's daughter, who wouldn't speak a single word well past the age of three and would instead communicate via elaborate pantomime. My in-law blamed it on her watching so many cartoons, the majority of which were dubbed from other languages, thus making it harder to associate shapes of the mouth with the sounds of speech (and even in Russian cartoons it's not very common to carefully imitate mouth shapes to match the dialogue). She didn't use the term "McGurk Effect", but since she's a psychology major I believe she may have been aware of it. I thought she was exaggerating the issue, but I didn't really have any data to refute her hypothesis. Anyway, the girl did start speaking at some point either before or shortly after turning four and the last time I saw her she spoke very fluently.

As a more relevant response, I don't think there's a huge difference in the amount of animation Japanese and American kids watch (I'd also argue that the popularity of anime among Japanese adults also tends to be exaggerated, but that's a whole other topic). The "face avoidance" and phonemic structure explanations make more sense to me.
3 x

User avatar
PeterMollenburg
Black Belt - 3rd Dan
Posts: 3229
Joined: Wed Jul 22, 2015 11:54 am
Location: Australia
Languages: English (N), French (B2-certified), Dutch (High A2?), Spanish (~A1), German (long-forgotten 99%), Norwegian (false starts in 2020 & 2021)
Language Log: https://forum.language-learners.org/vie ... 15&t=18080
x 8029

Re: Considerations on using dubbed material and the McGurk Effect

Postby PeterMollenburg » Sun Jun 25, 2017 9:43 pm

I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.

Edit: I replayed these videos, and apparently I'm normal ;) it's da da da da on the first and ba ba ba ba on the second for me too.
Last edited by PeterMollenburg on Sun Jun 25, 2017 9:48 pm, edited 1 time in total.
0 x

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby zenmonkey » Sun Jun 25, 2017 9:46 pm

PeterMollenburg wrote:I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.


It's the same video (posted twice).
Watch it once, paying attention with eyes open.
Watch it a second time, with eyes closed.

Order doesn't matter.

What did you hear each time?
1 x
I am a leaf on the wind, watch how I soar

User avatar
PeterMollenburg
Black Belt - 3rd Dan
Posts: 3229
Joined: Wed Jul 22, 2015 11:54 am
Location: Australia
Languages: English (N), French (B2-certified), Dutch (High A2?), Spanish (~A1), German (long-forgotten 99%), Norwegian (false starts in 2020 & 2021)
Language Log: https://forum.language-learners.org/vie ... 15&t=18080
x 8029

Re: Considerations on using dubbed material and the McGurk Effect

Postby PeterMollenburg » Sun Jun 25, 2017 9:48 pm

zenmonkey wrote:
PeterMollenburg wrote:I know I used to lip read when I was a kid, or to be correct, watch peoples' mouths as they spoke. Somewhere along the lines I stopped doing it, perhaps as a teenager or even slightly before my teens. I think maybe it felt intrusive to pay attention to peoples' mouths, and I still don't do it now. I find it almost strange now to deliberately look at someone's mouth while they speak, but I do wonder if I am indeed still doing it subconsciously in snippets here and there as a guide during speech, but maybe much less than most. I couldn't do this test properly- I didn't understand what I was supposed to be doing or thinking I was hearing, and played them out of order by accident.


It's the same video (posted twice).
Watch it once, paying attention with eyes open.
Watch it a second time, with eyes closed.

Order doesn't matter.

What did you hear each time?


see my edit above
0 x

User avatar
Steve
Orange Belt
Posts: 169
Joined: Wed Jan 06, 2016 7:53 am
Location: US (Wisconsin)
Languages: English(N), Spanish (Intermediate), Ancient Greek (Intermediate), Hebrew (Beginner), ASL (Beginner), German (Beginner)
x 801
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Steve » Sun Jun 25, 2017 10:04 pm

Interesting. When watching, I was hearing 3 different pairs of sounds. The first two of which kept shifting on me and the final pair being a clear da-da. I tried this several times.

I was unable to lock onto the first two pairs of sounds. It was like they kept shifting on me and changing each time I listened. I was hearing combinations of different consonants sort of like ga-la, la-ga, ga-ka, etc. But it was like there was a dissonance I clearly felt that something was not right. I kept hearing strange consonants best described as l, g, k, and t (weird hearing voiced/unvoiced consonants). The first two pairs never sounded like da-da to me.

Listening without watching was always a clear ba-ba.
1 x

User avatar
Serpent
Black Belt - 3rd Dan
Posts: 3657
Joined: Sat Jul 18, 2015 10:54 am
Location: Moskova
Languages: heritage
Russian (native); Belarusian, Polish

fluent or close: Finnish (certified C1), English; Portuguese, Spanish, German, Italian
learning: Croatian+, Ukrainian; Romanian, Galician; Danish, Swedish; Estonian
exploring: Latin, Karelian, Catalan, Dutch, Czech, Latvian
x 5179
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Serpent » Sun Jun 25, 2017 10:50 pm

Honestly I'm so used to the lips not matching in dubbed L1 movies that I simply don't expect them to match.

I definitely didn't hear da the first time, it sounded like a weird non-English consonant a few times and then the last one was a clear ba. With my eyes closed it was a mix of ba and argh I don't remember, ga? When trying again I did hear dada with my eyes open.
1 x
LyricsTraining now has Finnish and Polish :)
Corrections welcome

User avatar
Ani
Brown Belt
Posts: 1433
Joined: Mon Mar 14, 2016 8:58 am
Location: Alaska
Languages: English (N), speaks French, Russian & Icelandic (beginner)
x 3840
Contact:

Re: Considerations on using dubbed material and the McGurk Effect

Postby Ani » Mon Jun 26, 2017 2:59 am

I was only able to hear ba ba and see the mouth movement for ga ga, right from the beginning :/ I tried several times. It might be because my speaker is off to the side, but it might also be teaching children reading and spelling with the LIPS system. It is a focus on the movement and shape of the mouth when pronouncing a word to understand phonemes and give help for spelling. It was evident that the mouth movement was neither ba nor da because of the lift of the back of the tongue.
3 x
But there's no sense crying over every mistake. You just keep on trying till you run out of cake.


Return to “General Language Discussion”

Who is online

Users browsing this forum: No registered users and 2 guests