Can one calculate comprehension?

[Iversen]

I vaguely remember an experiment where somebody removed (or inverted?) all negations in a text by the French philosopher Derrida and asked some francophone test persons with academical credentials to guess where they should have been. Or asked them what the heck the meaning of the text was supposed to be. And it seemed that nobody had a clue as to what Derrida had tried to express, with or without negations, in the original version or with negations strewn around in all the wrong places.

So comprehension is a relative thing.

The nearest thing I come to quantifying comprehension is when I do vocabulary counts. Following the lead given by Rdearman earlier in this thread there are cases where the meaning is absolutely crystal clear (and I may even know some context and/or background information concerning the word), and there are other cases where I don't have a clue - not even with a suitable context. In between there are cases where I can guess the meaning because I know some whetherthe word in question also can be used in other ways not known to me. On agood day I might believe that my judgments were roughly equally untrustworthy across languages, but even that isn't a certain fact - I definitely guess better in strong languages than in weak ones, both due to my larger preexistant wordstock and because I 'understand' contexts better in a strong languages. So in essence I am measuring my degree of understanding using the very same degree of understanding as my yardstick, and logicians have never been happy with circular arguments.

Now move the discussion to texts... With words we at least had standardized and reasonably comprehensive dictionaries, but not so with whole sentences. Here I only have my gut feelings (which in the case of words also could be said about estimates of my active vocabulary), but that's actually better than nothing. I can't trust that I have a comprehension yardstick with zero at one end and total blissfull savvy at the other (since I can't even guess at what I have missed), but I can count the cases where I - on an extremely shaky foundation - estimate my comprehension to for instance somewhere around 80%. The 80% are nonsense, but my claim to understand 80% is a countable entity on which you can do statistical analyses.

[aaleks]

Let's systematize this. If I take a major newspaper in the target language - let's say the New York Times in English, Le monde in French - an A1 level learner will understand 10%, A2 20%, B1 30%, B2 60% C1 80% and C2 100%. It actually makes sense. My take on this is that understanding the language of something like a newspaper or any kind of native materials such as TV series that we've been talking about here is basically an all or nothing thing. I see no difference between 10% and 60% in terms of appreciating a sample. Even understanding 80% of a newspaper is for me basically pointless.

Comprehension is not only a language issue. What and how well we understand depends no less on our other knowledge, experience etc. If I read some newspaper article about stock exchange my understanding would be the same no matter what the language it’s written in - Russian (native), English (passive skills about C1), German (learned in school and a little after college), French (a little basic grammar and handful words), Spanish (handful phrases), Italian (have never studied) so on.

Btw, there’s no difference when someone claim that he understand 30 or 50% and any statement about CEFR levels without taking an exam. That’s the same subjective self-assessment. But language learning isn’t mathematics after all

[Cainntear]

Casual reading of my first post would show that I propose a three level scale for describing degree of comprehension of language (and not movies): nothing, something and full.

That's all well and good, but that still doesn't give us any means of grading material relatively -- there's no way of identifying "harder" or "easier" material on that scale. And you clearly wouldn't propose grading difficulty by vocabulary coverage, as you've already made it clear you think that's a non-starter.
....

This question of grading material is irrelevant here. The CEFR scale does the job perfectly and is widely used in the European book publishing industry for language learning.

This question is only irrelevant if we make certain assumptions about what we're discussing, and I (and most other posters) clearly am making different assumptions from you. I am assuming that any measure or estimate of understanding is intended as a means of gauging progress, and therefore requires some degree of comparability. "None, some, full" does not offer much comparability, so I don't understand what purpose you intend your scale for -- you have not explained it yet, and I'm afraid no-one's going to understand your point until you do.

I campaigned strenuously on the old HTLAL site for the use of the CEFR model to replace all the vague, misleading and, most importantly, meaningless words for describing one's language skills. Today, since everybody on this site uses the same model we have some standard idea of what the speaker means when they use the CEFR proficiency terms.

As leosmith says, the CEFR itself is extremely vague. It is an improvement on what went before it because there is now more commonality in our vagueness, but it is still very, very vague.

This means that the CEFR is not a measure, but an estimate. This is perfectly fine, because language is not strictly measurable, and any statement of ability will always be an estimate. A score on a language exam is a measure only against that exam -- as an indicator of overall language proficiency it can only ever be an estimate. In fact, any exam in any subject is an estimate, and most of us are perfectly happy with the existence of exams.

In fact, I have no objection to someone claiming B2 or C1 understanding in a language. It gives me a relatively clear understanding of what the person can do in the language. But 30% understanding of a recording? What does that mean?

Not a lot, admittedly. But don't we normally, as teachers, try to get a figure for student's understanding of a text when we set comprehension questions? Admittedly, we target our questions at certain parts of the text, but only because direct measure of comprehension is impossible. But on the other hand, for a student reading a text of their own choosing, with no accompanying comprehension questions, they're hardly able to estimate what marks they would get in a comprehension test. So they come up with a vague, woolly figure, and there's nothing wrong with that.

Asking how people come up with their estimates is the wrong way to go about it, because nobody is directly measuring their comprehension, or counting exactly how many words they look up, or even counting how many sentences they understand, so sorry leosmith...

On the other hand, when I say I understand 30% of a recording, that means that for every 10 sentences I understand 3 perfectly.

... I don't believe you. I believe you just have a "feeling" and stick a number on it, and that's perfectly fine. (There have been studies suggesting that teachers' gut reactions to student essays are as reliable at determining final grades are as good as, if not better than, careful and conscious marking with a red pen.)

I think it is utterly pointless to try to argue explicit rules about a completely implicit process, and constantly asking people how they come up with their percentages is about as pointless as asking them how they decide which words to use when they talk -- both are subconscious processes, and no-one can describe them usefully.

But I'm not against percentages. Far from it, I can see a very useful application. If I read a book of 50,000 words or word tokens, let's say there are 10,000 distinct words. I'll skip the various issues around what is a word. Of these distinct words, 500 are completely unknown to me. I have to look them up in the dictionary. This means that my word coverage without the dictionary is 95%. Is that my percentage understanding? No, these are two different concepts. Word coverage and understanding are not the same thing. Can I get an adequate understanding of the book with only 95% coverage? This is very debatable.

Again, you're back on coverage, and you're simultaneously arguing for and against it. As you say, coverage is not a measure of comprehension.

The fundamental problem is that comprehension is not measurable, but we need to track it as it is a basic component of language ability.
If there is no measure, there can only be an estimate. People's percentages are an estimate. What's wrong with that.

[s_allard]

...

The fundamental problem is that comprehension is not measurable, but we need to track it as it is a basic component of language ability.
If there is no measure, there can only be an estimate. People's percentages are an estimate. What's wrong with that.

Who would believe that Cainntear and I actually agree on something? When people use percentages of comprehension, they are estimating or guesstimating. If that makes them feel good, fine.

My objection is within the context of discussions or debates when we are trying to use terms that we can agree on to some extent. This is why the CEFR has become so popular. Of course, there is a vagueness of definitions of the various categories, but we all adhere more or less to this framework. So, we don't waste time trying to figure out what you mean by "high beginner - low intermediate fluency" when you can say B1. The next step of course is to actually be certified as B1. This is the essence of the whole CEFR revolution. All of the European Union now has a standard framework for language assessment and testing. Compare this to the situation in North America where we still have a hodgepodge of various systems.

The objection I have to the use of percentages to estimate comprehension is that it lends an air of pseudo-science to something that has no scientific foundation. To me, 90%, 95%, 100% comprehension are all the same because this is by definition such a vague notion. In fact, it's more at the other end that I have a real problem with these percentages. I just don't see how 30%, 40% or even 50% comprehension is fundamentally different from 0%. Is 20% understanding really better than nothing? I don't believe so. Now, some bright person might say that 50% comprehension means you perfectly understand every other page of a book or the first half and not a word of the second half.

When it comes to television and movies, I think part of the problem is that people mix up all the visual clues and pleasures with the actual comprehension of the words being used. For our purposes here, it is more interesting to work with audio-only materials.

[aaleks]

Is 20% understanding really better than nothing? I don't believe so.

Maybe it’s just cultural differences but to me nothing means zero when 20% means something. Some time ago on Internet I’ve come across an old French series which I had watched as teenager. It’s original version, i.e. in French. As I already wrote above I know a little French, really just a little, so it will be fair to say I can understand about 20% of what I hear. But at the same time I can follow a story of every episode because, first of all, I’ve seen the series in Russian already and have known the plot. Besides, a plot of an episode usually is very simple and the actors laugh, yell, cry, gesticulate, make big eyes, drop jaws etc. at the right moments. That all helps to understand what is going on. If I want to describe how well or bad I understand the show what should I say? That I’ve understood nothing? But it won’t be true since I’m able to understand not only action but some of words too.

[Aozora]

The objection I have to the use of percentages to estimate comprehension is that it lends an air of pseudo-science to something that has no scientific foundation. To me, 90%, 95%, 100% comprehension are all the same because this is by definition such a vague notion. In fact, it's more at the other end that I have a real problem with these percentages. I just don't see how 30%, 40% or even 50% comprehension is fundamentally different from 0%. Is 20% understanding really better than nothing? I don't believe so. Now, some bright person might say that 50% comprehension means you perfectly understand every other page of a book or the first half and not a word of the second half.

I think you're the only one who would say under 50% comprehension is no better than understanding nothing, or that 90% is the same as 100%. We're all in the process of learning our target languages, so at some point we will have poor comprehension and as we progress we will understand more. No one here is satisfied with 20% comprehension, but it's useful to know we are improving and understanding more than before.

When someone says they understood 40% of a tv series and later understood 97%, the takeaway is that they greatly improved. And that's useful for other learners who might use the same method. We know it's an estimation and not a precise scientific calculation. In my opinion it's a better description of one's level than "I understood something before, and now I understand something, but more than before."

[s_allard]

The objection I have to the use of percentages to estimate comprehension is that it lends an air of pseudo-science to something that has no scientific foundation. To me, 90%, 95%, 100% comprehension are all the same because this is by definition such a vague notion. In fact, it's more at the other end that I have a real problem with these percentages. I just don't see how 30%, 40% or even 50% comprehension is fundamentally different from 0%. Is 20% understanding really better than nothing? I don't believe so. Now, some bright person might say that 50% comprehension means you perfectly understand every other page of a book or the first half and not a word of the second half.

I think you're the only one who would say under 50% comprehension is no better than understanding nothing, or that 90% is the same as 100%. We're all in the process of learning our target languages, so at some point we will have poor comprehension and as we progress we will understand more. No one here is satisfied with 20% comprehension, but it's useful to know we are improving and understanding more than before.

When someone says they understood 40% of a tv series and later understood 97%, the takeaway is that they greatly improved. And that's useful for other learners who might use the same method. We know it's an estimation and not a precise scientific calculation. In my opinion it's a better description of one's level than "I understood something before, and now I understand something, but more than before."

I like this post because it clearly states the prevailing wisdom. The person saying they went from 40% to 97% is just saying "that they greatly improved". Well, if that's what they want to say, why not just say that? Why put a precise figure like 40% when it is not based on anything except some sort of hunch. And how can one justify 97% except by pure guessing.

The funny thing is that I see what people are getting at. The percentages are not real measurements, they are just a way of expressing a perception. Let's say I'm learning a language and when I listen to a conversation in the language I have an inkling, a gist, an idea of what people are saying. It's not as if I don't understand a thing. But for sure I don't understand everything. I'm not very sure. So, I'll put 40%. Why 40% and not 43%? Well, it's a gut feeling.

That's exactly how this works most of the time. Those figures are just pure guesses masquerading as numerical values. There is no justification for 40% other than "I say so". Admittedly some people do try to provide some type of method based on counting words and sentences but I know of very few people who have actually done this. I have tried and I won't write about it here because people will accuse me of keeping the thread going forever.

I think you're the only one who would say under 50% comprehension is no better than understanding nothing, or that 90% is the same as 100%.

Let first say that my original statement was: "I just don't see how 30%, 40% or even 50% comprehension is fundamentally different from 0%." What I'm saying is that where language is concerned there is a threshold that I'll put at 50% below which the overall communication does not make much sense.

Let me give a concrete example. I have never studied Portuguese but it is close to Spanish and has some things in common with French. So, today, I tried to read the following first paragraph from a Brazilian newspaper on the web:

Na tarde desta segunda-feira (1°), data em que se comemora o Dia Internacional dos Trabalhadores, manifestantes se reuniram na Cinelândia, no Centro do Rio de Janeiro, para protestar contra as reformas trabalhistas e da Previdência, em trâmite no Congresso Nacional, além da opressão policial vista durante o ato da última sexta-feira (28). Com a presença de centrais sindicais e lideranças políticas de partidos de oposição ao presidente Michel Temer, o ato reuniu milhares de trabalhadores que empunhados de bandeiras e faixas entoavam gritos de ordem.

Because of Spanish and French, I have a basic idea that the some workers are protesting for International Worker's Day in the centre of Rio de Janeiro. But there are a few words here and there where I have no clue and I really don't know how Portuguese grammar works. So, I'm kind of guessing. So, how much do I understand? I think I have some idea of what's happening. Let's say I feel that I understand 50%. It sounds good but the reality is that I'm looking at bits and pieces of words and phrases here and there. If asked to translate this paragraph, I would say: "There is some kind of demonstration going on related to International Workers' Day. There is mention of politicians and unions." Did this already take place or is it going to take place? I don't know. There is a section ..., em trâmite no Congresso Nacional, além da opressão policial vista durante o ato da última sexta-feira (28)." where I don't have a clue.

The fact is that the understanding that I do have, the 50%, is in my mind hardly better than nothing because there is so much key information missing. But I really don't want to make a big deal out of 50% being equal to zero. Remember my scale is: nothing, something, full. So, 50% could be the threshold for something.

[Xmmm]

Again we see no attempt to calculate the percentage of comprehension. Instead, we get the suggestion that 30% understanding corresponds to the feeling that one gets by attempting to read a graded reader that is two CEFR levels above one's current level.

Why are you using "we" in this context? Who is "we"?

Let's systematize this. If I take a major newspaper in the target language - let's say the New York Times in English, Le monde in French - an A1 level learner will understand 10%, A2 20%, B1 30%, B2 60% C1 80% and C2 100%. It actually makes sense. My take on this is that understanding the language of something like a newspaper or any kind of native materials such as TV series that we've been talking about here is basically an all or nothing thing. I see no difference between 10% and 60% in terms of appreciating a sample. Even understanding 80% of a newspaper is for me basically pointless.

Thank you for your concession.

I feel like you have two different points in this thread:

1. It didn't make sense to you to estimate comprehension at low levels
2. It doesn't make sense to you to read or watch TV at low levels of comprehension

With regard to #2 you make be right in theory. But adult hobbyists don't want to spend months or years using graded readers. They would rather struggle through Le Monde and look up words. Besides, there isn't an infinite supply of graded readers anyway, so what should people do?

"In theory, there is no difference between theory and practice. But in practice there is." -- Jan L. A. van de Snepscheut

[reineke]

The CEFR Global scale includes a set of descriptors that were drawn from a wider bank of illustrative descriptors. The descriptors that relate to receptive skills are grouped around topics such as Overall Listening Comprehension, Understanding Interaction between Native Speakers. Listening as a Member of a Live Audience, Listening to Announcements & Instructions Listening to Radio & Audio Recordings, Watching TV & Film etc.

With regards to broadcast media, B1 "can follow many films in which visuals and action carry much of the storyline, and which are delivered clearly in straightforward language.
Can catch the main points in TV programmes on familiar topics when the delivery is relatively slow and clear.
Can understand a large part of many TV programmes on topics of personal interest such as interviews, short lectures, and news reports when the delivery is relatively slow and clear."

B2 "Can understand most TV news and current affairs programmes.
Can understand documentaries, live interviews, talk shows, plays and the majority of films in standard dialect."

C1 can "follow films employing a considerable degree of slang and idiomatic usage". C1 can also "understand a wide range of recorded and broadcast audio material, including some non-standard usage, and identify finer points of detail including implicit attitudes and relationships between speakers" .

My interpretation of the "percentage" scale:

<20% Comprehension of individual words and phrases.
20-50% Spotty comprehension. This is A2 territory. Sorry.
50%-80% Global comprehension. Increasing ability to pick up details. B1
80%-98% Selective comprehension. The devil is in the detail. B2-C1
99%-100% Detailed comprehension. C1-C2

Someone mentioning 80%-90% comprehension could be referring to general listening or reading comprehension or something more specific (ie "Buffy" ). If former is the case, I find that the "vague" CEFR scale may prove more helpful and accurate than a personal percentage scale. I wouldn't completely write off the percentage scale but I find it more useful when describing progress from A1 through B1.

The higher the percentage, the more important it becomes to include additional explanatory language. For instance, someone's idea of 95% comprehension may mostly apply to being able to understand the gist of a TV show.

If you are investing time and effort to create a detailed record of your language learning consider adding some descriptive language to these percentages. Also, take advantage of the more detailed CEFR descriptors (which, btw, are getting expanded).

To give you an idea why estimating comprehension is a tricky business :

"Bonk’s (2000) subjects listened to four passages on unfamiliar topics with varying levels of lexical familiarity (as measured by subjects’ ability to correctly record items in a dictation test). His findings indicate the complexity of the relationship between lexical knowledge and comprehension: although higher dictation scores (and therefore lexical familiarity) were associated with better comprehension, some subjects were able to achieve quite good comprehension with a lexical knowledge of less than 75%. Others, however, could not achieve the same level of comprehension with even 100% lexical knowledge."

"Just as native-speaker listeners are able to switch from top-down to bottom-up processing and vice versa according to whether a topic is unfamiliar or not, or whether the listening makes lexical, grammatical or phonological demands (Vanderplank, 1988), L2 listeners need to know when to switch the focus of their listening, or how to apply knowledge strategically."

Strategy clusters and sources of knowledge in French L2 listening comprehension. Innovation in Language Learning and Teaching, Graham, S. J., Santos, D. and Vanderplank, R. (2010) 4 (1). pp. 1-20

[s_allard]

...

B2 "Can understand most TV news and current affairs programmes.
Can understand documentaries, live interviews, talk shows, plays and the majority of films in standard dialect."

C1 can "follow films employing a considerable degree of slang and idiomatic usage". C1 can also "understand a wide range of recorded and broadcast audio material, including some non-standard usage, and identify finer points of detail including implicit attitudes and relationships between speakers" .

My interpretation of the "percentage" scale:

<20% Comprehension of individual words and phrases.
20-50% Spotty comprehension. This is A2 territory. Sorry.
50%-80% Global comprehension. Increasing ability to pick up details. B1
80%-98% Selective comprehension. The devil is in the detail. B2-C1
99%-100% Detailed comprehension. C1-C2

Someone mentioning 80%-90% comprehension could be referring to general listening or reading comprehension or something more specific (ie "Buffy" ). If former is the case, I find that the "vague" CEFR scale may prove more helpful and accurate than a personal percentage scale. I wouldn't completely write off the percentage scale but I find it more useful when describing progress from A1 through B1.

The higher the percentage, the more important it becomes to include additional explanatory language. For instance, someone's idea of 95% comprehension may mostly apply to being able to understand the gist of a TV show.

...

A very interesting post and even a relevant citation. Kudos to reineke. What I find interesting in the descriptors for B2 is the wording "can understand". There is no reference to degrees or percentages of understanding. For testing purposes, one will be scored using a questionnaire of course, but the description of the skill simply assumes that the candidate has an adequate understanding.

As to reineke's interpretation of the percentage scale, I had suggested something quite similar. This scale assumes that we are working with the same materials; for example, a newspaper. A2 understands 20-50%, B1 50-80%, etc. This might work. But I don't think this is how people have been using these percentages here. When people say they understand a TV program at 97%, I'm not sure that are thinking that they have a C1 level understanding. To be explored further.