Measuring Efficiency of Learning Techniques

General discussion about learning languages
orlandohill
Yellow Belt
Posts: 50
Joined: Sun Oct 02, 2022 11:54 pm
Languages: English (N)
x 97

Measuring Efficiency of Learning Techniques

Postby orlandohill » Wed Oct 05, 2022 4:32 pm

Is there a commonly accepted methodology for measuring the efficiency of language learning techniques?

I'm interested in quantifying the efficiency of my use of Listening-Reading (L-R) and related techniques. I was thinking to measure increase in vocabulary, listening comprehension and reading comprehension, and relate it to the time spent using the techniques.

I could take CEFR type tests, but in addition I'd like to get more fine-grained data.

I'll mainly be doing this for my own benefit, but it's possible that my data could be useful to others.

The methodology would need to be readily applicable to the less mainstream languages that currently interest me the most.

The methodology that I have in mind at the moment would be something like this.

1) Measure vocabulary recognition in isolation. Take the top Xk frequent words, and mark those that I believe I know. Compare against a bilingual dictionary whether I know the correct, partially correct, or incorrect definition. I understand that languages have many words that aren't word-for-word translations. This would just be a rough measure for early-stage learning.

2) Take a random sample of pages from books in different domains and at different reading levels. For example, 10 random pages from the Harry Potter series, 10 from The Lord of the Rings, and 10 from Sapiens. Starting at the next complete sentence of each page, grade each page with Refold's 0-6 comprehension level.

3) Do similar to 2), but take random samples from audio books.

For steps 2 and 3, I would note my prior familiarity with the material.

I would record how many minutes I used each technique each day. Combined with before and after tests, that would give a measure of the effectiveness of the techniques when applied on that learning schedule.
2 x

dlt529
White Belt
Posts: 27
Joined: Sun Aug 28, 2022 1:16 pm
Languages: English (N), Spanish (Adv), Portuguese (Intermediate)
Language Log: https://forum.language-learners.org//vi ... 15&t=18534
x 126

Re: Measuring Efficiency of Learning Techniques

Postby dlt529 » Thu Oct 06, 2022 9:45 pm

As a data-driven person myself, I appreciate the want to collect data to determine effectiveness. However, collecting data on a single person doesn't really tell you much. By checking the frequency list, you're likely to learn and remember some words, maybe not a lot. By using texts you already know, and returning regularly to them, you're likely to improve your comprehension just by doing the task, regardless of study techniques or time. And at the end of the data, data on one subject doesn't tell us much.

People have always learned languages, regardless of how they've studied them. I'd recommend that you focus on enjoyment instead of efficiency. If you enjoy the L-R reading method, go ahead and do it. Language learning for most people is a marathon, not the sprint, so you might as well use techniques that you enjoy and don't worry so much about crossing the finish line the fastest.
6 x

orlandohill
Yellow Belt
Posts: 50
Joined: Sun Oct 02, 2022 11:54 pm
Languages: English (N)
x 97

Re: Measuring Efficiency of Learning Techniques

Postby orlandohill » Fri Oct 07, 2022 1:30 am

Thanks for your feedback.

I don't think my proposed vocabulary test would do much to directly improve my vocabulary in subsequent tests. I would only look up the definition of words that I believe I know, so at most it might turn some false positives into positive results.

I have access to enough books and audio books to be able to test comprehension exclusively on unfamiliar material. I'll make sure to avoid resampling previously tested pages and audio.
1 x

Spaceman
White Belt
Posts: 41
Joined: Thu Sep 29, 2022 12:49 pm
Languages: English (N), Spanish (Intermediate Receptive), German (Rusty), French (Very Rusty)
x 160

Re: Measuring Efficiency of Learning Techniques

Postby Spaceman » Fri Oct 07, 2022 1:00 pm

I think there's two problems with your proposed technique. One is the problem of bias. There is a huge amount of gray area in whether you know a word and should mark it as such. If you want to have improved and want to have a good score, you'll be more likely to lean "know." This will be even more pronounced if you're learning a related language with a lot of cognates. Then using the Refold scale is even more subjective. It's also not very granular and will be hard to show any improvement on after a while. Counting unknown words/phrases on the pages might be a good additional data point.

The other problem is randomness in how understandable a given book page (or 10 pages) is. A page that's all action is so much easier to understand than a page of description. Dialogue can be the hardest or the easiest thing depending on the author. The book I'm reading now has a character that is completely incomprehensible to me (he's an eccentric book lover). Statistically speaking you'll likely need quite a bit more than 10 pages to make a valid comparison.
4 x

orlandohill
Yellow Belt
Posts: 50
Joined: Sun Oct 02, 2022 11:54 pm
Languages: English (N)
x 97

Re: Measuring Efficiency of Learning Techniques

Postby orlandohill » Fri Oct 07, 2022 4:15 pm

I'm mostly interested in my 'before' test results.

For the vocabulary test, I would do dictionary lookups for each word that I believe I know. I think the real issue is that words often have more than one definition.

Perhaps the test of vocabulary in isolation could be the median percentage of definitions correctly produced for each word. I would still filter out words that I don't believe I know. Those words would automatically get a score of 0%.

I understand that the Refold comprehension scale is subjective, though for my own purposes it's probably sufficient. I like the idea of adding specific numerical thresholds to each level.

A more objective measure might be to test transcription and translation skills, though comparing the result of a translation with a model answer would still require human interpretation.

Good point about the qualitative difference between action and dialogue in fiction. If I were to use fiction, I would likely choose books where the difficulty of compreheding dialogue was fairly even.

Perhaps an improved sampling method would be to take 5 books, and sample each book 10 times. Each sample starts at a random page number, and collects the next 100 words of dialogue, and 100 words of action, or 200 words for a non-fiction book, extended to the end of the current sentence. Audio book samples could use 1 or 2 minutes instead of 100 or 200 words. That would result in 10000+ words, and 100+ minutes of audio. That would make for a very time consuming test.

I think I'll aim for one 5th of the sample material, and sample from a single book. That would still be 2000+ words and 20 minutes of audio.
1 x

emcuttsy
Posts: 3
Joined: Fri Oct 21, 2022 7:41 pm
Languages: English (N), German (C1), Italian (beginner), Spanish (beginner)
x 15

Re: Measuring Efficiency of Learning Techniques

Postby emcuttsy » Fri Oct 21, 2022 7:59 pm

The trouble here is that you need to define "efficacy" before you can even begin to try to measure it.

What is efficacy? Number of new words acquired per [time]? Reduction per [time] in number of grammatical errors in a written text of X words prepared in Y minutes? Reduction per [time] in the number of grammatical errors in a spontaneous speech over Z minutes? Increase in reading speed over X time? Increase in typing speed over X time? Increase per [time] in subjective scoring of fluency by X native speakers after a conversation of Y minutes? Time to achieve a given CEFR score?

Messy stuff. Every study you look at will look at something different.

All that said — someone I know on Twitter tracks his progress in a way I find very cool. He measures his reading speed and the number of words he doesn't know in a text every month. The text varies so it's not a really an objective measure in the strictest sense. But over the years it's yielded a pretty interesting curve. His reading speed in French leveled off at a high level but plateaued and isn't really approaching his level in English (he's a big reader in English though, too) despite continued work.
3 x

User avatar
Iversen
Black Belt - 4th Dan
Posts: 4768
Joined: Sun Jul 19, 2015 7:36 pm
Location: Denmark
Languages: Monolingual travels in Danish, English, German, Dutch, Swedish, French, Portuguese, Spanish, Catalan, Italian, Romanian and (part time) Esperanto
Ahem, not yet: Norwegian, Afrikaans, Platt, Scots, Russian, Serbian, Bulgarian, Albanian, Greek, Latin, Irish, Indonesian and a few more...
Language Log: viewtopic.php?f=15&t=1027
x 14962

Re: Measuring Efficiency of Learning Techniques

Postby Iversen » Sat Oct 22, 2022 3:53 pm

It's a hard problem because people react differently to different techniques, and because different techniques serve different purposes. I have done some tests of the efficiency of my own wordlist system, and I can say with certainty that it functions well for me - but for those who arfre more context dependent it might not work. Besides it primarily serves to build a passive vocabulary which is excellent for reading (and a good preparatoion for writing), but if you purpose is to learn the things you need for a smalltalk session in a café that might be an irrelevant criterion. Here it might be more efficient to prelearn some useful spoken phrases and pay less attention to the written language.
2 x

s_allard
Blue Belt
Posts: 969
Joined: Sat Jul 25, 2015 3:01 pm
Location: Canada
Languages: French (N), English (N), Spanish (C2 Cert.), German (B2 Cert)
x 2305

Re: Measuring Efficiency of Learning Techniques

Postby s_allard » Sun Oct 23, 2022 11:48 pm

emcuttsy wrote:The trouble here is that you need to define "efficacy" before you can even begin to try to measure it.

What is efficacy? Number of new words acquired per [time]? Reduction per [time] in number of grammatical errors in a written text of X words prepared in Y minutes? Reduction per [time] in the number of grammatical errors in a spontaneous speech over Z minutes? Increase in reading speed over X time? Increase in typing speed over X time? Increase per [time] in subjective scoring of fluency by X native speakers after a conversation of Y minutes? Time to achieve a given CEFR score?

Messy stuff. Every study you look at will look at something different.

All that said — someone I know on Twitter tracks his progress in a way I find very cool. He measures his reading speed and the number of words he doesn't know in a text every month. The text varies so it's not a really an objective measure in the strictest sense. But over the years it's yielded a pretty interesting curve. His reading speed in French leveled off at a high level but plateaued and isn't really approaching his level in English (he's a big reader in English though, too) despite continued work.


I’m sympathetic to the idea of developing metrics to measure the effectiveness of language learning techniques. The problem is in what to measure and how. Given the major methodological obstacles, most, if not all, efforts at quantifying language proficiency gravitate to measuring quantities of words known in some manner.

The assumption here is that the word is a basic unit of language and therefore can be counted as some sort of proxy of proficiency. This idea is immediately beset with complex problems of defining what is a word and what it means to know and use a word. I won’t wade into that debate except to say that counting the number of words « known » or, more likely, merely recognized is a very poor way of determining overall language proficiency. In this regard, it should be pointed out that no where in the CEFR model is there mention of vocabulary size.

The big picture question is what is the purpose of trying to measure effectiveness. I would think that the idea is to be able to objectively choose the best methods, tools and techniques for our learning goals. Since, in my opinion, this is something of a fool’s errand, I think the best thing is to simply list all these methods, tools and techniques and then systematically experiment with each one to see which ones work best for you.

So maybe Anki works great for you whereas I like paper word cards because I can manipulate them physically. Some people like the word lists à la iversen ; then go for it. Other people are big on comprehensible input. Have a go at shadowing. And so on. The real issue is exploring various options and finding what is the right fit for you.
2 x


Return to “General Language Discussion”

Who is online

Users browsing this forum: Beli Tsar, emk and 2 guests