Study on Language Improvement with Native Materials

[jeff_lindqvist]

... I can try to think about a template for the spreadsheet...

Please address "listening while doing housework etc" and "paying attention on and off when listening" in your spreadsheet. I did not understand rdearman's answer to my question:

If you're not paying attention it doesn't count.

Isn't it as simple as that? Count the time you're paying attention.

From How much language study do you do?:

If I have L2 radio in the background for, say, 8 hours a day that I may or may not pay attention to, how do I count it?

Full attention - eight hours, "half" attention - four hours, no attention - nothing.

[smallwhite]

Isn't it as simple as that? Count the time you're paying attention.
Full attention - eight hours, "half" attention - four hours, no attention - nothing.

That's one way of doing it. A very complicated way to maintain for 6 months when I can listen to things for more than 10 hours a day as I do all sorts of different activities. Another way not excluded by rdearman's reply, is to count 100% even for partial attention. Movies and songs aren't 100% speech either but we count the full length anyway; no need to complicate radio-listening. Yet another way of doing it, which is what I do, is to not record background listening time at all.

All are valid options and I'd like to know which one to use.

[zenmonkey]

I'm posting a draft tracking sheet here, for comments.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

I've added the request for "attention" to be included.

Please consider that this is a draft tracking sheet. It will be modified based on input (particularly rdearman's).
In particular, take a look at the list of activities and let me know of activities you use, that you see missing.

[Systematiker]

I'm posting a draft tracking sheet here, for comments.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

I've added the request for "attention" to be included.

Please consider that this is a draft tracking sheet. It will be modified based on input (particularly rdearman's).
In particular, take a look at the list of activities and let me know of activities you use, that you see missing.

You may want an activity specified for L-R stuff, broken down into which combination of L1-L2 or just the L2 audio with L2 text. Especially the latter will form a part of what I'll do, as I'll try the films and subs2srs thing, but I'll probably spend more time reading familiar text while listening to audio of the same.

[rdearman]

I was having a little thing about this, and perhaps the best way to think about tracking is the results. What improvements do we see over time. So for example if we what to report on the result of increase in comprehension. This is why I suggested the comprehension scale of 1-9. Ideally both sets of participants should see an increase in comprehension over time. Although this is "self measured" it should increase. Perhaps another would be speed of reading. So having amount of time read and number of words allows us to derive a reading speed, which you would hope would increase and improve.

Smallwhites suggestion about attention when used with comprehension should indicate an increase in understanding with less attention paid. E.g. you can devote less attention to get a similar amount of comprehension as you improve.

So what additional metric could we track? I'm especially interested in metrics for people who'd be in the control group and for example might be using a course or grammar book?

I think zenmonkey has covered the basics, time spent, comprehension, attention, wordcounts, length of audio/video in time. Any other sugestions?

[zenmonkey]

I'm posting a draft tracking sheet here, for comments.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

I've added the request for "attention" to be included.

Please consider that this is a draft tracking sheet. It will be modified based on input (particularly rdearman's).
In particular, take a look at the list of activities and let me know of activities you use, that you see missing.

You may want an activity specified for L-R stuff, broken down into which combination of L1-L2 or just the L2 audio with L2 text. Especially the latter will form a part of what I'll do, as I'll try the films and subs2srs thing, but I'll probably spend more time reading familiar text while listening to audio of the same.

I've add:
L-R - Listening L2
L-R - Listening L2 reading L1
L-R - Listening - reading L2
L-R - Listening - reading L2 and mimicking
L-R - non specified / mixed

Is that ok?

[zenmonkey]

I was having a little thing about this, and perhaps the best way to think about tracking is the results. What improvements do we see over time. So for example if we what to report on the result of increase in comprehension. This is why I suggested the comprehension scale of 1-9. Ideally both sets of participants should see an increase in comprehension over time. Although this is "self measured" it should increase. Perhaps another would be speed of reading. So having amount of time read and number of words allows us to derive a reading speed, which you would hope would increase and improve.

Smallwhites suggestion about attention when used with comprehension should indicate an increase in understanding with less attention paid. E.g. you can devote less attention to get a similar amount of comprehension as you improve.

So what additional metric could we track? I'm especially interested in metrics for people who'd be in the control group and for example might be using a course or grammar book?

I think zenmonkey has covered the basics, time spent, comprehension, attention, wordcounts, length of audio/video in time. Any other sugestions?

For the course book - I tend to do three things:

course book - study new
course book - exercises/drills
course book - study review

which reminds me - I forgot to pop in FSI material

[smallwhite]

I was having a little thing about this, and perhaps the best way to think about tracking is the results. What improvements do we see over time. So for example if we what to report on the result of increase in comprehension. This is why I suggested the comprehension scale of 1-9. Ideally both sets of participants should see an increase in comprehension over time. Although this is "self measured" it should increase. Perhaps another would be speed of reading. So having amount of time read and number of words allows us to derive a reading speed, which you would hope would increase and improve.

(1) Do the end-of-study self/Dialang tests not measure our increased ability already? Do you really need to see our improvement from one study session to the next as well? (Improvement per 25 minutes for Pomodoro users). If not (then don't request such data) and if seeing monthly improvement is good enough, how about getting us to do self/Dialang tests once a month instead?

(2) I don't see any way of calculating reading speed from the spreadsheet. It asks for total pages of the book, but it doesn't know how much of the book I've read, or how many times I've read each sentence.

(3) If that's how you're interpreting my data, then you're either interpreting it wrong or you're collecting it wrong.

(3a) I apply i+1 when I choose my books so comprehension will theoretically always be 95% which is my comfort level. That does not mean my reading skills are not improving.

(3b) You cannot derived from the spreadsheet my change in comprehension from one book to the next when you don't know how difficult my books are.

(3c) You cannot derive from the spreadsheet my change in comprehension from the beginning of a book to its end when you don't know how fast I'm reading it. I comprehend more when I read more slowly, and less when I read more quickly.

(3d) Reading speed is a choice, a factor, not a result for me, unless you're explicitly testing me on it. I can read as fast as the next person, but outside of tests I choose to read very slowly, especially when I'm between A0 and B2 which your study covers because I am still learning, because reading is my learning tool and not my speed reading contest, I'm trying to assimilate the grammar and the vocabulary.

(3e) Basically, book difficulty and reading speed are my ways of fine-tuning a given book to create i+1. They are factors to my improvement. If you take them as results of my improvement, you're misinterpreting my data.


I normally only analyse financial data and I've never had to explain the setup process to anyone so all that is probably incomprehensible or even wrong. I suggest you read up on data collection for research, or at least database design which is sort of within IT and probably an easier read for you. It would make the whole process much easier for you. Your professor can maybe recommend a book?

[smallwhite]

I'm posting a draft tracking sheet here, for comments.

Any other sugestions?

Under the current format, I'm asked "Comprehension" and "Attention" even when the activity is "Memrise" or "Note review". I think you want different fields for different activities.

I'm posting a draft tracking sheet here, for comments.

Any other sugestions?

You may want an "Other" activity for each type of activity that is relevant to the study -
Reading other
Listening other
Vocabulary other
etc
and probably no all-encompassing "Other" field.

I'm posting a draft tracking sheet here, for comments.

Any other sugestions?

I spend ~25% of my desk time on admin work and paperwork. Do I input those, and how?
* Formatting and filing downloaded material
* Googling for learning material, reading Amazon reviews
* Splitting, combining or rearranging notes and vocabulary decks
* etc

[zenmonkey]

I was having a little thing about this, and perhaps the best way to think about tracking is the results. What improvements do we see over time. So for example if we what to report on the result of increase in comprehension. This is why I suggested the comprehension scale of 1-9. Ideally both sets of participants should see an increase in comprehension over time. Although this is "self measured" it should increase. Perhaps another would be speed of reading. So having amount of time read and number of words allows us to derive a reading speed, which you would hope would increase and improve.

(1) Do the end-of-study self/Dialang tests not measure our increased ability already? Do you really need to see our improvement from one study session to the next as well? (Improvement per 25 minutes for Pomodoro users). If not (then don't request such data) and if seeing monthly improvement is good enough, how about getting us to do self/Dialang tests once a month instead?

(2) I don't see any way of calculating reading speed from the spreadsheet. It asks for total pages of the book, but it doesn't know how much of the book I've read, or how many times I've read each sentence.

(3) If that's how you're interpreting my data, then you're either interpreting it wrong or you're collecting it wrong.

(3a) I apply i+1 when I choose my books so comprehension will theoretically always be 95% which is my comfort level. That does not mean my reading skills are not improving.

(3b) You cannot derived from the spreadsheet my change in comprehension from one book to the next when you don't know how difficult my books are.

(3c) You cannot derive from the spreadsheet my change in comprehension from the beginning of a book to its end when you don't know how fast I'm reading it. I comprehend more when I read more slowly, and less when I read more quickly.

(3d) Reading speed is a choice, a factor, not a result for me, unless you're explicitly testing me on it. I can read as fast as the next person, but outside of tests I choose to read very slowly, especially when I'm between A0 and B2 which your study covers because I am still learning, because reading is my learning tool and not my speed reading contest, I'm trying to assimilate the grammar and the vocabulary.

(3e) Basically, book difficulty and reading speed are my ways of fine-tuning a given book to create i+1. They are factors to my improvement. If you take them as results of my improvement, you're misinterpreting my data.


I normally only analyse financial data and I've never had to explain the setup process to anyone so all that is probably incomprehensible or even wrong. I suggest you read up on data collection for research, or at least database design which is sort of within IT and probably an easier read for you. It would make the whole process much easier for you. Your professor can maybe recommend a book?

When I grow up, I want to be smallwhite.

Excellent points.

I would suggest that we do not use any activity elements to evaluate levels of improvement without an explicit question/evaluation. If you want to know if my reading has improved over the last month - ask, I can then go back and see - "oh, yeah, I was reading at a level x (book y) and now..." rather than pages or speed from the tracking. My reading varies from device use (kindle vs computer vs paper), environment, intensive vs extensive, etc... I could add "pages read" but

a) it is inconsistent with the reported words per book, we'd have to add pages per book
b) I would not suggest we add words read - spending time calculating that after each session will drive us bananas.

Thinking it through , I'll add an activity field 'if reading, pages read' and also capture pages per book.

Under the current format, I'm asked "Comprehension" and "Attention" even when the activity is "Memrise" or "Note review". I think you want different fields for different activities.

I'm not going to add new fields unless we have specifics for other activities suggested and agreed. However, I'll add "NR" to both fields to allow people to mark it if they feel it is not relevant or do not want to provide a value.

Updated: https://docs.google.com/spreadsheets/d/ ... sp=sharing