Dialang validity and reliability?

All about language programs, courses, websites and other learning resources
Tomás
Blue Belt
Posts: 554
Joined: Sat Oct 10, 2015 9:48 pm
Languages: English (N). Currently studying Spanish (intermediate), French (false beginner).
x 661

Dialang validity and reliability?

Postby Tomás » Sun Jun 05, 2016 8:20 pm

I have nine months of studying Spanish with at least a half hour of listening per day. I have no trouble following the radio and TV news and telenovelas, and so I'm pretty certain my listening is at the intermediate level.

I took the Dialang Spanish vocab test last night, and then again this morning. My score varied by more than 300 points out of a 1,000 between the two tests, indicating extremely poor test-retest reliability.

I took the listening test today and scored A1. The program scored me wrong on questions for writing answers with incorrect capitalization and punctuation. That would make sense on a writing test. It makes no sense at all for a listening comprehension test.

It also scored me wrong on a question about two old men talking about the post Civil War period. I chose the "complaining" response; the correct answer was "discussing". However, they were talking about how poor and hungry they were, and about how dangerous it was to live under Franco being the reason so many people emigrated to Mexico. That sounds like "complaining" to me. There were other questions like that, where I would have gotten it wrong even in my native language due to poor item construction on the part of the test-makers.

In summary, I do not think this is either a valid or a reliable test.
3 x

User avatar
Serpent
Black Belt - 3rd Dan
Posts: 3657
Joined: Sat Jul 18, 2015 10:54 am
Location: Moskova
Languages: heritage
Russian (native); Belarusian, Polish

fluent or close: Finnish (certified C1), English; Portuguese, Spanish, German, Italian
learning: Croatian+, Ukrainian; Romanian, Galician; Danish, Swedish; Estonian
exploring: Latin, Karelian, Catalan, Dutch, Czech, Latvian
x 5179
Contact:

Re: Dialang validity and reliability?

Postby Serpent » Sun Jun 05, 2016 9:51 pm

It's very similar to the actual CEFR exams. You do need to focus on the details both in the questions and answers. And from what I remember, in that question most things were negative but there was something neutral or positive, ie it wasn't 100% complaining. Maybe that Mexico fact is what makes it count as discussing? Anyway, that's not a poorly designed question, that's deliberate. (The real problem is that you're expected to pick out the details just like you do in writing, and you can only listen once. But again, that does happen at the actual exams too)
Listening tasks at exams often involve actually writing, yes. You're expected to write down/summarize accurately what you hear, other than stuff like proper names.
As for the placement test, it's best not to take it too often, otherwise you'll be remembering your previous answers and trying to improve your score, not actually answering the question "is this a real word?". The words are the same every time, so if you give the same answers, you'll get an identical score. And again, you need to focus on details. Most "fake" words are not random but based on the existing words you may have heard somewhere.

I agree, the test is not perfect, and especially writing is hard to measure. But it does give you a reasonable idea of what to expect at a CEFR exam. Also, if you do the checklist and overestimate your skills, you'll get harder questions.
7 x
LyricsTraining now has Finnish and Polish :)
Corrections welcome

User avatar
reineke
Black Belt - 3rd Dan
Posts: 3570
Joined: Wed Jan 06, 2016 7:34 pm
Languages: Fox (C4)
Language Log: https://forum.language-learners.org/vie ... =15&t=6979
x 6554

Re: Dialang validity and reliability?

Postby reineke » Mon Jun 06, 2016 3:43 am

None of these tests is perfect. I never saw that sentence but anything to do with Franco is likely not simple "complaining". Try the Cervantes placement test. It lets you listen to the audio at least two times.

http://ave.cervantes.es/prueba_nivel/registro/test_de_clasificacion.php
Last edited by reineke on Wed Oct 31, 2018 2:37 am, edited 1 time in total.
1 x

Cainntear
Black Belt - 3rd Dan
Posts: 3468
Joined: Thu Jul 30, 2015 11:04 am
Location: Scotland
Languages: English(N)
Advanced: French,Spanish, Scottish Gaelic
Intermediate: Italian, Catalan, Corsican
Basic: Welsh
Dabbling: Polish, Russian etc
x 8662
Contact:

Re: Dialang validity and reliability?

Postby Cainntear » Mon Jun 06, 2016 2:02 pm

Yes, but it's not a great leap to think "complaining" means "saying something was bad". Teacher brain tells me that "complaining" in this case would mean "making a complaint", but it's one of the things I would feel I had to teach specifically in exam technique classes.
2 x

User avatar
Cola
White Belt
Posts: 32
Joined: Sat Mar 11, 2017 4:16 am
Location: United States
Languages: English (N), Studying Dutch
x 38

DIALANG

Postby Cola » Sun Dec 10, 2017 6:57 am

About DIALANG
DIALANG is a language diagnosis system developed by many European higher education institutions. It reports your level of skill against the Common European Framework (CEF) for language learning. DIALANG is not an exam. DIALANG does not issue certificates.

DIALANG’s languages are Danish, Dutch, English, Finnish, French, German, Greek, Icelandic, Irish-gaelic, Italian, Norwegian, Portuguese, Spanish and Swedish. DIALANG has instructions and tests in all these languages.


Have any of you used this website to test your language skills? Do you believe it to be a reliable measurement of your skill level? I'm very curious, please let me know. :D

http://www.lancaster.ac.uk/researchenterprise/dialang/DIALANG_PAGE_WebVersion.htm
Last edited by Serpent on Sun Dec 10, 2017 7:54 am, edited 1 time in total.
Reason: mod note - threads merged
1 x
Please correct even the smallest language mistakes I make. I really appreciate it!

User avatar
smallwhite
Black Belt - 2nd Dan
Posts: 2386
Joined: Mon Jul 06, 2015 6:55 am
Location: Hong Kong
Languages: Native: Cantonese;
Good: English, French, Spanish, Italian;
Mediocre: Mandarin, German, Swedish, Dutch.
.
x 4876

Re: DIALANG

Postby smallwhite » Mon Dec 11, 2017 5:34 am

Cola wrote:Have any of you used this website to test your language skills? Do you believe it to be a reliable measurement of your skill level?

Yes and Yes.
It has never given me scores that are lower than my expectations so I love it ;)
3 x
Dialang or it didn't happen.

User avatar
Cola
White Belt
Posts: 32
Joined: Sat Mar 11, 2017 4:16 am
Location: United States
Languages: English (N), Studying Dutch
x 38

Re: Dialang validity and reliability?

Postby Cola » Tue Dec 12, 2017 3:37 am

Thanks for your response smallwhite! :)

Does anyone else wanna share their experiences?

I scored much higher than I thought I would, so I was just wondering how accurate it is. :oops:

By the way @Serpent sorry that you had to merge my thread; I had searched for "Dialang" but got no results so I thought no one had made a thread about it yet. Funnily enough when I search now I'm getting lots of results. :oops:
1 x
Please correct even the smallest language mistakes I make. I really appreciate it!

User avatar
smallwhite
Black Belt - 2nd Dan
Posts: 2386
Joined: Mon Jul 06, 2015 6:55 am
Location: Hong Kong
Languages: Native: Cantonese;
Good: English, French, Spanish, Italian;
Mediocre: Mandarin, German, Swedish, Dutch.
.
x 4876

Re: Dialang validity and reliability?

Postby smallwhite » Tue Dec 12, 2017 8:30 am

Cola wrote:Does anyone else wanna share their experiences?

I scored much higher than I thought I would, so I was just wondering how accurate it is. :oops:

Many of us have taken the Dialang but I'm not sure people would be eager to answer your question. You just have to look through the search results. Many if not most comments about tests and exams tell you how the test was too strict, the sound was too poor, the day was too bad, the X was too Y, and - the score was too low. So if you got higher than expected scores, congratulate yourself.

Here's another website you can try: http://eclexam.eu/languages/
2 x
Dialang or it didn't happen.

User avatar
tiia
Blue Belt
Posts: 735
Joined: Tue Mar 15, 2016 11:52 pm
Location: Finland
Languages: German (N), English (?), Finnish (C1), Spanish (B2??), Swedish (B2)
Language Log: viewtopic.php?t=2374
x 1997

Re: Dialang validity and reliability?

Postby tiia » Tue Dec 12, 2017 11:01 am

It's been quite some time since I've last tried Dialang. Despite having heard that it's great for testing your level, I do agree that it might have tendencies to overestimate your level. It's always worth to read the descriptions when it shows your results.

I tested my skills in Finnish twice (or even more often) and in English once. The results in Finnish were (skill: first test - second test):

Reading: C1 - C1
Listening: B2 - B2
Writing: C1 - C1
Grammar: B2 - B2
Vocabulary: B1 - C1

Unfortunately I didn't add the date, when I wrote down the test results. So I don't know exactly when I have taken them, but if I remember correctly, I started reading books between those tests, something one can absolutely see in the increased vocabulary size.
I still would not put myself into the C-range when it comes to reading or writing - B2 yes, but C? I don't know.

More surprisingly were the results in English. I probably did the test more or less at the same time as the first Finnish test. And some of the results were lower than their Finnish counterpart. (I didn't write them down.) Quite certainly that was just not true back then.* In writing there were some tasks, where you had to choose the most appropriate response out of four. It felt like the responses in Finnish had larger differences than in English, therefore the answer in Finnish was often more than obvious, even if you might not have been able to produce the sentence yourself.

Perhaps it's time to do the test again, as the second test result might be from 2015, maybe even from the second half of 2014.


*Nowadays I wouldn't be so sure anymore as I read far more fictional books in Finnish than in English. Depending on what area/subject is tested I wouldn't be surprised about a higher level in Finnish than in English.
4 x
Corrections for entries written in Finnish, Spanish or Swedish are welcome.
Project 30+X: 25 / 30

User avatar
MorkTheFiddle
Black Belt - 2nd Dan
Posts: 2114
Joined: Sat Jul 18, 2015 8:59 pm
Location: North Texas USA
Languages: English (N). Read (only) French and Spanish. Studying Ancient Greek. Studying a bit of Latin. Once studied Old Norse. Dabbled in Catalan, Provençal and Italian.
Language Log: https://forum.language-learners.org/vie ... 11#p133911
x 4824

Re: Dialang validity and reliability?

Postby MorkTheFiddle » Tue Dec 12, 2017 7:59 pm

FWIW, I answered 25 of 28 questions correctly on the English exam. As a native English speaker, perhaps I should have done better. The test consumed a bit of time, so I did not try the others.
I did score 1000 on the placement test.
2 x
Many things which are false are transmitted from book to book, and gain credit in the world. -- attributed to Samuel Johnson


Return to “Language Programs and Resources”

Who is online

Users browsing this forum: No registered users and 2 guests