Can one calculate comprehension?

General discussion about learning languages
s_allard
Blue Belt
Posts: 969
Joined: Sat Jul 25, 2015 3:01 pm
Location: Canada
Languages: French (N), English (N), Spanish (C2 Cert.), German (B2 Cert)
x 2305

Can one calculate comprehension?

Postby s_allard » Thu Apr 27, 2017 3:08 pm

I regularly see contributors here give percentages for their level of comprehension of samples of their target language. Very often it's something like: I can understand about 95% of a certain television series. I've always wondered how people arrive at these figures and I've tried to reproduce them myself.

This all came to a head when I was working with my Spanish tutor recently and I realized on two occasions that I perfectly knew all the words in the text but I didn't have the full meaning because I misinterpreted a few items. In fact, my understanding of the entire text was thrown off by misunderstanding a particular word. Here is one of the two samples from the Spanish newspaper El mundo:

Iba para ingeniero químico y hoy tiene 27.000 empleados a su cargo. Así es el hombre más rico de India.

None of the words were particularly unknown to me but my tutor and I got into a discussion of "Iba para ingeniero..." and more particularly the use of the well-known word para here. I quickly realized that I didn't really understand the usage here. So, what percentage of the entire message did I understand? Considering that there are 20 words in the text, did I really understand 95% of the message? Or given that the use of para here is key to understanding the whole text, in fact I did understand something but not the full meaning. But I can't say that I understood nothing.

In the old HTLAL forum I argued that it is impossible to quantify comprehension. What we can quantify is the number of words known or unknown in a sample. Thus we speak of text coverage. I've often seen reference to adequate vs inadequate comprehension but rarely to percentages of comprehension. This is why I have always proposed a three-level scale of comprehension: nothing, something, full. I wonder what people think.
Last edited by s_allard on Thu Apr 27, 2017 5:31 pm, edited 1 time in total.
5 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6330
Contact:

Re: Can one calculate comprehension?

Postby emk » Thu Apr 27, 2017 3:34 pm

In a nutshell: The DELF examiners certainly believe they can meaningfully award you 20 out of 25 possible points for comprehension. And there's no reason you can't do something similar at home, if you think it's worth your time.

s_allard wrote:I regularly see contributors here give percentages for their level of comprehension of samples of their target language. Very often it's something like: I can understand about 95% of a certain television series. I've always wondered how people arrive at these figures and I've tried to reproduce them myself.

"95%" is just a fancy way of saying "19 out of 20."

It's pretty much meaningless to say something like this without also saying how well understand that fraction of the material. For example, you might say:

  • "I could follow about 80% of the plot." This might mean that out of every 10 scenes, you basically knew what was going on in 8 of them, but the other two left you lost.
  • "I understood 95% of the sentences in the dialog well enough that I could have transcribed it correctly, including all the clitic pronouns, and explained what every sentence meant." This means that out of 300 lines of dialog in the TV show, there were 15 where you couldn't write them down and explain them.
  • "99% of the vocabulary was either known to me, or trivially guessable from context." This might mean that you read 10 pages, with an average of 350 words each, and you marked every unknown word and looked it up later. You looked up a total of 35 words for 10 pages.
  • "I scored 78% on my DELF B2." This means that the DELF graders applied their detailed grading standards to speaking, writing, reading and listening, and awarded me 78 out of 100 points.
  • If you use Anki for intensive listening, you might say, "When I first heard the cards from that TV show, I marked 50% of them as 'correct' the first time I heard them, based on how well my understanding corresponded to the L2 and L1 subtitles on the back of the card."
In fact, every language teacher does this with their students: They make up a grading standard, they explain it, and then they go through with a red pen. Then they give you a percentage grade.

The important thing is to explain the grading standard. When I saw "followed 40%", that's much weaker than "understood 40%". The former implies I understood the dialog well enough to know what was going on, and the later implies that I could write it down accurately and explain it in detail. And of course there's a subjective element! But there's also a subjective element when a teacher grades a paper, or when a CEFR examiner evaluates your speaking skills. So you do the best you can, and realize that a 78% from one teacher might have been 75% or 85% if you took the test from a different teacher in a different city. And yet, despite the lack of perfectly rigorous and reproducible measurement, teachers still grade tests, and the education system still works.

If you find, however, that this approach makes no sense for you, then don't use it, and ignore other people when they use it.
11 x

YtownPolyglot
Orange Belt
Posts: 227
Joined: Thu Apr 14, 2016 9:42 pm
Location: Ohio, USA
Languages: English (N), French (C1), Spanish (B2), German (B2), Italian (B1)
On the wishlist: Norwegian, Portuguese, Russian, Dutch....
x 311

Re: Can one calculate comprehension?

Postby YtownPolyglot » Thu Apr 27, 2017 4:41 pm

The problem with putting a number on it is that comprehension is not just understanding a certain number of words. If you don't understand "iba para ingeniero," your problem is not likely to be with any one of those three particular words. It's a matter of how they work together to form meaning.

The other problem is that we typically use context to figure out meaning. I take this phrase to mean that he was going to study engineering or he was studying to become an engineer and all these other things happened. Either one of these interpretations seems likely.

El Mundo probably uses the sort of language that is useful to anyone functioning in Spain. On the other hand, not everything is written like a newspaper. I feel like I've got a good grasp of a language if I can understand jokes. It's possible to understand 95% of the surface vocabulary or all of it, and not get the joke. The first time I understood a Brazilian meme in Portuguese, I felt like I had really turned a corner.
3 x
Native language: English
Other languages: French (C1), Spanish (B3), German (B2), Italian (B1)
Wish list: Portuguese, Dutch, Russian, Norwegian....

s_allard
Blue Belt
Posts: 969
Joined: Sat Jul 25, 2015 3:01 pm
Location: Canada
Languages: French (N), English (N), Spanish (C2 Cert.), German (B2 Cert)
x 2305

Re: Can one calculate comprehension?

Postby s_allard » Thu Apr 27, 2017 5:29 pm

emk wrote:In a nutshell: The DELF examiners certainly believe they can meaningfully award you 20 out of 25 possible points for comprehension. And there's no reason you can't do something similar at home, if you think it's worth your time.

s_allard wrote:I regularly see contributors here give percentages for their level of comprehension of samples of their target language. Very often it's something like: I can understand about 95% of a certain television series. I've always wondered how people arrive at these figures and I've tried to reproduce them myself.

"95%" is just a fancy way of saying "19 out of 20."

It's pretty much meaningless to say something like this without also saying how well understand that fraction of the material. For example, you might say:

  • "I could follow about 80% of the plot." This might mean that out of every 10 scenes, you basically knew what was going on in 8 of them, but the other two left you lost.
  • "I understood 95% of the sentences in the dialog well enough that I could have transcribed it correctly, including all the clitic pronouns, and explained what every sentence meant." This means that out of 300 lines of dialog in the TV show, there were 15 where you couldn't write them down and explain them.
  • "99% of the vocabulary was either known to me, or trivially guessable from context." This might mean that you read 10 pages, with an average of 350 words each, and you marked every unknown word and looked it up later. You looked up a total of 35 words for 10 pages.
  • "I scored 78% on my DELF B2." This means that the DELF graders applied their detailed grading standards to speaking, writing, reading and listening, and awarded me 78 out of 100 points.
  • If you use Anki for intensive listening, you might say, "When I first heard the cards from that TV show, I marked 50% of them as 'correct' the first time I heard them, based on how well my understanding corresponded to the L2 and L1 subtitles on the back of the card."
In fact, every language teacher does this with their students: They make up a grading standard, they explain it, and then they go through with a red pen. Then they give you a percentage grade.

The important thing is to explain the grading standard. When I saw "followed 40%", that's much weaker than "understood 40%". The former implies I understood the dialog well enough to know what was going on, and the later implies that I could write it down accurately and explain it in detail. .../quote]
I agree that there are all sorts of indirect indicators that one can use when attempting to measure comprehension of samples of spoken or written language. I can count the words I've never seen before and calculate a percentage. Or I can certainly count the number of lines of TV dialog that I can't make out. But the question is whether this equates to measuring how well I understand or not.

If someone says I read a text and underlined 150 out of 5000 distinct words as unknown, for a percentage of 3% unknown words that's a fact I don't dispute. Does that mean the person understood 97% of the text? I say not necessarily. It depends on the nature of those 150 words. If those are critical vocabulary words or even grammatical components, a lot of the meaning can be thrown off.

This is of course a major problem with the so-called cognate discount. The fact that French and English have about 40% of vocabulary in common doesn't mean that an English-speaker can understand 40% of a French conversation.

This is precisely why I gave a specific example from Spanish. This example is something I could transcribe perfectly. 100% of the vocabulary is known to me, or at least I thought so. Does that mean I understood 100% of the text? It turns out I didn't. After discussion with my tutor I realized that my interpretation of three common words "... iba para ingeniero..." was wrong. What then was my initial understanding then? More like 85%?

In fact my understanding was totally off from the intention of the author. And the only reason I realized this is because the tutor caught my mistake. I thought I understood but in reality I didn't.

This is the fundamental problem of designing oral comprehension tests. Is it as simple as trying to count the words that the candidate does not know in the listening sample? Obviously not. Designers will try to create questions that try to indicate deeper understanding. And does getting a perfect score on the multiple-choice questions mean you understand the text 100% ? That's for another debate.

Note: I wrote this post before seeing Ytownpolyglot's excellent post. Sorry for the redundancies.
2 x

User avatar
sfuqua
Black Belt - 1st Dan
Posts: 1642
Joined: Sun Jul 19, 2015 5:05 am
Location: san jose, california
Languages: Bad English: native
Samoan: speak, but rusty
Tagalog: imperfect, but use all the time
Spanish: read
French: read some
Japanese: beginner, obsessively studying
Language Log: https://forum.language-learners.org/vie ... =15&t=9248
x 6299

Re: Can one calculate comprehension?

Postby sfuqua » Thu Apr 27, 2017 6:34 pm

I've always used a standard of about how many words were unfamiliar when I have ever tried to figure out my comprehension.
Of course it is possible to understand almost every word and still miss the point completely, or it is possible to miss 30% of the word and still comprehend a passage, so I agree with OP's idea.
2 x
荒海や佐渡によこたふ天の川

the rough sea / stretching out towards Sado / the Milky Way
Basho[1689]

Sometimes Japanese is just too much...

aaleks
Blue Belt
Posts: 884
Joined: Thu Apr 13, 2017 7:04 pm
Languages: Russian (N)
x 1910

Re: Can one calculate comprehension?

Postby aaleks » Thu Apr 27, 2017 6:59 pm

s_allard wrote:This is why I have always proposed a three-level scale of comprehension: nothing, something, full. I wonder what people think.

I think there should be at least one more level in between “something” and “full”. For example, I can’t say I fully understand all I read or listen to in English, but still I understand more than just something.
2 x

s_allard
Blue Belt
Posts: 969
Joined: Sat Jul 25, 2015 3:01 pm
Location: Canada
Languages: French (N), English (N), Spanish (C2 Cert.), German (B2 Cert)
x 2305

Re: Can one calculate comprehension?

Postby s_allard » Fri Apr 28, 2017 6:49 am

YtownPolyglot wrote:The problem with putting a number on it is that comprehension is not just understanding a certain number of words. If you don't understand "iba para ingeniero," your problem is not likely to be with any one of those three particular words. It's a matter of how they work together to form meaning.

The other problem is that we typically use context to figure out meaning. I take this phrase to mean that he was going to study engineering or he was studying to become an engineer and all these other things happened. Either one of these interpretations seems likely. ....

This is exactly the problem encountered. I knew every one of these words but I had never encountered this combination. How could one calculate the percentage comprehension? And if one doesn't understand these three words, the rest of the sentence loses much of its meaning even though all the words are known. In fact, the interpretations of these words include:
1. He was going to study to become a chemical engineer but began these studies.
2. He studied to be an engineer but did not finish.
3. He finished his studies in engineering but never practiced the profession.

It's only later in the article that we learn that option 3 in the right one. But the point is that while we can use all sorts of conceptual devices to count or measure some indicator that we believe is correlated with understanding, the fact is we can't measure understanding. Counting words is notoriously inaccurate.
0 x

User avatar
tastyonions
Black Belt - 1st Dan
Posts: 1576
Joined: Sat Jul 18, 2015 5:39 pm
Location: Dallas, TX
Languages: EN (N), FR, ES, DE, IT, PT, NL, EL
x 3867

Re: Can one calculate comprehension?

Postby tastyonions » Fri Apr 28, 2017 9:50 am

Maybe something like "percentage of clauses understood" would give a more precise idea. Even beginning learners of French understand the words "prendre" and "la bouteille" but many wouldn't understand "prendre de la bouteille."
1 x

Cainntear
Black Belt - 3rd Dan
Posts: 3469
Joined: Thu Jul 30, 2015 11:04 am
Location: Scotland
Languages: English(N)
Advanced: French,Spanish, Scottish Gaelic
Intermediate: Italian, Catalan, Corsican
Basic: Welsh
Dabbling: Polish, Russian etc
x 8664
Contact:

Re: Can one calculate comprehension?

Postby Cainntear » Fri Apr 28, 2017 4:19 pm

s_allard wrote:I regularly see contributors here give percentages for their level of comprehension of samples of their target language. Very often it's something like: I can understand about 95% of a certain television series. I've always wondered how people arrive at these figures and I've tried to reproduce them myself.

It's just an estimate. I agree that using percentages has a pseudo-scientific feel to it for something very, very subjective, but I can't find it in me to be bothered when people do it (or even ask me to do it, even).
Thus we speak of text coverage.

Yes, but then we hit the old problem of "what gets measured gets managed". Text coverage is easy to measure, but if we use it as our measure, we risk tricking ourselves into placing too much emphasis on word lists and memorisation in our teaching.
1 x

qeadz
Green Belt
Posts: 298
Joined: Thu Jul 21, 2016 11:37 pm
Languages: English (N), Korean (~A2)
x 400

Re: Can one calculate comprehension?

Postby qeadz » Fri Apr 28, 2017 5:43 pm

The issue of course being that you don't know what you don't know. So the more you don't know, the less you can trust what you think you know. However you think you know what you think you know because of the first part; namely that you don't know what you don't know :?

Obviously words known has to play into it somehow, but there are plenty of texts in my Korean now for which I do know every word in the sentence or even the paragraph yet I'm a little confused as to the meaning. In some cases it may be who is doing what to who, or in other cases whether the author is saying something is a good thing or a bad thing.

IMHO a percentage means very little. I strongly suspect that many people who would say they catch "about 50%" or "60%" of whats going on are rating themselves far higher than I would were I in their shoes. Regardless of what is being measured with those percentages, there is sufficient room in the unknown to completely alter the known.

My test for Korean has been to take an unseen transcribed dialog from TalkToMeInKorean, write down in English what I think they are saying and then compare with Hyunwoo's excellent English translations. I just make my own judgement as to how wrong I feel I got it.

There'd be no way to place any kind of measurement system that wouldn't have downfalls on that and reliably say: I got 70%. To come up with "70%", I'd have to have some way to itemize understanding and say "I understood 7 out of 10 'meanings' in this text!".

What about areas where my translation is 'in the ballpark' but not quite right. Do I get half a point?
0 x


Return to “General Language Discussion”

Who is online

Users browsing this forum: lichtrausch and 2 guests