Italki Language Test

General discussion about learning languages
User avatar
lysi
Yellow Belt
Posts: 98
Joined: Sat Mar 27, 2021 1:34 pm
Location: Ontario, Canada
Languages: English (N), French, Mandarin (Beginner)
x 300

Italki Language Test

Postby lysi » Thu Apr 08, 2021 4:14 pm

I got an email from Italki a few days ago about their new online automated language assessments. I know they had an English test for a long time but I haven't seen any information at all about their new test for 7 languages.

https://www.italki.com/languageassessment

Here's some quotes from the site:

Reading, Grammar, and Vocabulary Test
Our multiple-choice adaptive placement assessment is designed to provide individualized evaluation through a combination of grammar, reading, and vocabulary questions. As students proceed through the assessment, the difficulty of the following questions will be adapted based on the previous responses. By offering multiple questions targeted at the student’s level, the assessment can accurately evaluate students’ abilities very quickly. Additionally, the psychometrics used in the scoring algorithm can control for lucky guesses or inadvertent errors that deviate from the expected response model. Once the algorithm is confident in the student’s ability, the assessment ends and the student will receive a score.


Speaking Test
Our fully-automated speaking assessment utilizes artificial intelligence to evaluate students’ speaking ability through the use of innovative linguistic methods, such as elicited imitation (EI). Elicited imitation, which is a fancy way to say “listen and repeat”, is a research-proven method to efficiently predict test-takers' oral language proficiency. It measures how effectively test-takers' brains can process or “chunk” language information. Higher proficiency in the language enables a test-taker to process longer and more complex sentences.


Has anyone done it? Any thoughts on this?

I'm surprised that I haven't seen this mentioned anywhere and I really can't find any talk about it online, other than 1 post from a month ago on the language learning subreddit that has 7 replies. Looking at the internet archive it seems they added it on the 22nd of January of this year.
4 x

Cavesa
Black Belt - 4th Dan
Posts: 4960
Joined: Mon Jul 20, 2015 9:46 am
Languages: Czech (N), French (C2) English (C1), Italian (C1), Spanish, German (C1)
x 17566

Re: Italki Language Test

Postby Cavesa » Thu Apr 08, 2021 6:00 pm

I've just taken it.

I got the email too a few days ago. I was curious. And as the readers of my log know, I've been trying to push myself to finally take a Spanish exam for years! It is also good, because I had no clue about my real level after a longer gap (that's why I paid for this).

The testing method looks rather interesting, even though limited in some ways. Btw here is what italki writes about the exam on the support board https://support.italki.com/hc/en-us/art ... ssessment- but it is not that informative either. It is a very new thing, so you won't probably find many reviews online. I haven't found any.

So, what was it like and what were the results. The overall grade was C1, with C1 for the "grammar part" and B2 for the "oral part". As you see, my main question "should I prepare for DELE B2 or C1?" rests unresolved :-D

The first exam is on speaking. As a short video explains, the first part is based on research showing that an average person holds 7 items in the short term memory. When you're beginning, it's just 7 sounds in the language. As you improve, you remember bigger and bigger chunks and are able to repeat them. The test serves you longer and longer, and more complex, sentences to repeat. Not easy, but I still see how the AI grades that.

The second part of the first exam are 60s answers to open questions. I did well in the first two, in my opinion, but really badly in the third one. I will not tell you the questions, to not spoil it and affect your results, should you decide to take it. This is something I am not really sure about, how does the AI grade your speaking, what is being evaluated and how.

I got 7.6 exam points, which is B2. They also offer equivalents on other scales, namely Advanced Low - Advanced Mid ACTEL, 6.5 - 7
IELTS, 22-25 Toefl, 170-180 TOEIC. As you can see, the exam is originally meant to test English and only later adapted for the other languages. Not sure whether it affects the quality.

I think this exam was surprisingly correct, I think I am B2ish, even though with rather unusual set of strengths and weaknesses. I think it serves well for one's orientation, but of course a normal oral DELE exam is the golden standard.

The second exam is on grammar and reading. Multiple choice. The website tells you it will take 15-30 minutes. I finished in 2 minutes.

They say it is well done and talk about how it eliminates guesses and so on. Well, I had expected a much longer and more complex exam, truth be told. I don't know, whether I should trust the result, which was 897.0 points and therefore C1. No idea, whether the test goes up to C2 (for example SIELE doesn't), or whether the questions are selected based on the result from the first exam (to not annoy an advanced learner with basic questions and the opposite). I found the reading bits very easy, and most of the grammar questions too. No idea where I lost points, or what is the maximum (so how many things I had probably wrong).

I don't know how reliable this result is. I know my writing (of normal texts) is most probably not C1, no idea whether it is even B2. My comprehension is C1 or the listening one even C2. My grammar and vocab? Not sure.

I had expected to be disappointed with the oral exam (we all know that oral computer graded exams are rather controversial) and satisfied with the grammar one. It's the opposite. The oral exam is a very pleasant surprise and in my opinion does have value for your orientation. I have more doubts about the grammar one. I'd say the best online exam for Spanish and French grammar is on Kwiziq, but it doesn't test other stuff, and isn't available for other languages (but it is free!).

No regrets concerning the money, I think I got the value, especially in the speaking part, and have ideas on how to continue. In spite of all the previous failed plans, I'd like to take a real Spanish exam this year. Finally.
13 x

jmar257
Green Belt
Posts: 268
Joined: Tue Apr 07, 2020 2:41 pm
Location: USA
Languages: English (N), español (Intermediate), français (Lower Intermediate), deutsch (beginner)
Language Log: https://forum.language-learners.org/vie ... hp?t=15645
x 680

Re: Italki Language Test

Postby jmar257 » Thu Apr 08, 2021 6:30 pm

Cavesa wrote:I'd say the best online exam for Spanish and French grammar is on Kwiziq, but it doesn't test other stuff, and isn't available for other languages (but it is free!).

No regrets concerning the money, I think I got the value, especially in the speaking part, and have ideas on how to continue. In spite of all the previous failed plans, I'd like to take a real Spanish exam this year. Finally.

Great, in-depth review. I have some time in a couple of weekends and I may spring for the iTalki test, just to see where I'm at. I've tried Kwiziq before and liked it but didn't love it, but may be convinced to shell out for a few months to see how much I think I will stick with it long term (although I have enough on my plate at the moment, really).

Edit: How long did it take? I saw you said one portion took 2 minutes, but overall.
3 x
Feel free to give me corrections in any of my target languages!

Cavesa
Black Belt - 4th Dan
Posts: 4960
Joined: Mon Jul 20, 2015 9:46 am
Languages: Czech (N), French (C2) English (C1), Italian (C1), Spanish, German (C1)
x 17566

Re: Italki Language Test

Postby Cavesa » Thu Apr 08, 2021 7:12 pm

The oral part took me 16 minutes, which is what it was supposed to take (more or less). As it is more structured (you listen to stuff and then record stuff), it's impossible to do it much faster. Really, I think my fast completion of the grammar test is a flaw in design, it is really suspiciously short.

I'd like to hear from someone testing at the A2 or B1 levels, as I'd say the exam is likely to be targeted towards that public. I could try that, but I don't feel like spending the same amount of money to test either my Italian nor my German right now.

But comparing my experience to someone at those levels might give us a much more complete picture.
4 x

jmar257
Green Belt
Posts: 268
Joined: Tue Apr 07, 2020 2:41 pm
Location: USA
Languages: English (N), español (Intermediate), français (Lower Intermediate), deutsch (beginner)
Language Log: https://forum.language-learners.org/vie ... hp?t=15645
x 680

Re: Italki Language Test

Postby jmar257 » Thu Apr 08, 2021 7:21 pm

Cavesa wrote:The oral part took me 16 minutes, which is what it was supposed to take (more or less). As it is more structured (you listen to stuff and then record stuff), it's impossible to do it much faster. Really, I think my fast completion of the grammar test is a flaw in design, it is really suspiciously short.

I'd like to hear from someone testing at the A2 or B1 levels, as I'd say the exam is likely to be targeted towards that public. I could try that, but I don't feel like spending the same amount of money to test either my Italian nor my German right now.

But comparing my experience to someone at those levels might give us a much more complete picture.

I would bet my French would be at those levels but I wanted to see where it graded my Spanish--I wouldn't be shocked if my oral skills were B1 or so, but since I can read novels without a dictionary I wouldn't expect reading to be that low. That said, I also don't feel like spending $60 on that in a short span, and in general don't see the utility in (paid) tests at the lower levels. I'll definitely post here in a couple weeks when I get around to doing the test, or before if I get the chance.
3 x
Feel free to give me corrections in any of my target languages!

jmar257
Green Belt
Posts: 268
Joined: Tue Apr 07, 2020 2:41 pm
Location: USA
Languages: English (N), español (Intermediate), français (Lower Intermediate), deutsch (beginner)
Language Log: https://forum.language-learners.org/vie ... hp?t=15645
x 680

Re: Italki Language Test

Postby jmar257 » Sat Apr 24, 2021 4:55 pm

As promised, I took the Spanish test today. I didn't watch the clock strictly but all in it tookmaybe half an hour? It tells you in the recap how long each section took, 12 minutes for speaking and 9 for grammar for a total of 21 minutes. I'm going to quote Cavesa's (quite thorough) review below and add my 2 cents to each portion to compare/contrast our experiences. But first, results: for the speaking portion I got 6.9 (nice), with the CEFR equivalent given B1-B2. Okay, as I expected pretty much, since I seldom practice it. That said, I did have an iTalki session right before where I spoke Spanish for half an hour, so that helped (I figured it I was going to get this test done this weekend, might as well while Spanish was still buzzing in my head). Description: Individuals at this level can participate in conversations on familiar topics in everyday situations by asking and answering a variety of questions. They can usually talk about events and experiences in various time frames. They can usually handle social interactions when there is an unexpected complication. Sounds about right, although as for the last part I don't think I could handle that too gracefully. Ah, there's always improvement to be had. On grammar I got a 608, or B2. To be clear, the grammar section is more about understanding grammar and picking correctly in multiple choice questions as opposed to getting a written or spoken text checked for grammatical accuracy, so this sounds about right.

Cavesa wrote:So, what was it like and what were the results. The overall grade was C1, with C1 for the "grammar part" and B2 for the "oral part". As you see, my main question "should I prepare for DELE B2 or C1?" rests unresolved :-D
I'm not planning on testing any time soon but the test didn't really clear much up for me either, I knew that my production lagged wayyyy behind my receptive skills. That said, the reinforcement was nice (i.e., I'm not deluding myself too much).

Cavesa wrote:The first exam is on speaking. As a short video explains, the first part is based on research showing that an average person holds 7 items in the short term memory. When you're beginning, it's just 7 sounds in the language. As you improve, you remember bigger and bigger chunks and are able to repeat them. The test serves you longer and longer, and more complex, sentences to repeat. Not easy, but I still see how the AI grades that.
Yeah, I was wondering during this about how it grades partial answers, i.e., if I bungle the first part but get the last part correct, or vice versa, do I get credit? I really fell off on the end of this portion.

Cavesa wrote:The second part of the first exam are 60s answers to open questions. I did well in the first two, in my opinion, but really badly in the third one. I will not tell you the questions, to not spoil it and affect your results, should you decide to take it. This is something I am not really sure about, how does the AI grade your speaking, what is being evaluated and how.
I was the opposite, I bungled the first one (favorite book to read--mainly because I kept getting stumped on how to describe thrillers in Spanish and got caught up in that when I probably could've just described the plot--whatever). The last two I was able to flow on, and actually got cut off on the last one by the time limit. There were still hiccups and circumlocution, but I was able to keep talking on the given subject.

Cavesa wrote:I think this exam was surprisingly correct, I think I am B2ish, even though with rather unusual set of strengths and weaknesses. I think it serves well for one's orientation, but of course a normal oral DELE exam is the golden standard.
Yeah, agree here. I think it's pretty correct for me, but I kind of knew my productive skills need polish.

Cavesa wrote:They say it is well done and talk about how it eliminates guesses and so on. Well, I had expected a much longer and more complex exam, truth be told. I don't know, whether I should trust the result, which was 897.0 points and therefore C1. No idea, whether the test goes up to C2 (for example SIELE doesn't), or whether the questions are selected based on the result from the first exam (to not annoy an advanced learner with basic questions and the opposite). I found the reading bits very easy, and most of the grammar questions too. No idea where I lost points, or what is the maximum (so how many things I had probably wrong).
Pretty much the exact same complaints, I'm not sure which were wrong or why but I didn't feel like the questions were super difficult.

Cavesa wrote:I don't know how reliable this result is. I know my writing (of normal texts) is most probably not C1, no idea whether it is even B2. My comprehension is C1 or the listening one even C2. My grammar and vocab? Not sure.
I definitely would not think my writing or speaking is B2, but to be fair I got B1-B2 on the speaking portion. My reading/listening skills are definitely leagues ahead of my other skills, and reading even more so than listening.

Cavesa wrote:I had expected to be disappointed with the oral exam (we all know that oral computer graded exams are rather controversial) and satisfied with the grammar one. It's the opposite. The oral exam is a very pleasant surprise and in my opinion does have value for your orientation. I have more doubts about the grammar one. I'd say the best online exam for Spanish and French grammar is on Kwiziq, but it doesn't test other stuff, and isn't available for other languages (but it is free!).

No regrets concerning the money, I think I got the value, especially in the speaking part, and have ideas on how to continue. In spite of all the previous failed plans, I'd like to take a real Spanish exam this year. Finally.

I'll agree largely with this with the exception I've barely used Kwiziq, but I didn't hate it. I just don't want to pay what they want me to. That said, I took the free test a while back and recall it being decent.

Apologies for the novel-length post.

Edit: It would've been cool to have been able to do this before the SC and then after. Ah well.
7 x
Feel free to give me corrections in any of my target languages!

Cavesa
Black Belt - 4th Dan
Posts: 4960
Joined: Mon Jul 20, 2015 9:46 am
Languages: Czech (N), French (C2) English (C1), Italian (C1), Spanish, German (C1)
x 17566

Re: Italki Language Test

Postby Cavesa » Mon Jun 14, 2021 2:26 pm

I've just finished the test in German, and I feel like calling the whole thing bullshit. I had thought that the test would be more precise at the lower levels, but I am more convinced now that it seriously overestimates the learner's skills due to several big design flaws, which I have already described the last time.

As I am now at the beginning of a few months of intensive learning to get a needed level and pass an exam, I had hoped to take it at the beginning and the end, and see the progress.

My real level right now is A1ish active skills, B1ish passive skills. My result was B1 speaking and B2 "grammar".

The first problem was a bug that made the test unusuable at first. To the company's credit, the support reacted fast and made the test work in a few hours. However, it was late in the night already and I didn't have time, so I really finished it a week or so later. I didn't get to retake the exam, I got to continue it and I don't know how this affected my results. It probably has affected them.

I've described the spoken exam the last time. I got 6.3 points and the level B1. I am convinced the first exercise has a lot of merit actually, it really was much harder in German than in Spanish for me, as expected. I think that part of the whole exam is the most valuable one.

However, the free speaking part? I wouldn't be graded B1 by any living person with at least superficial knowledge of the scale. I wasn't even comfortable with understanding the questions, and I was far bellow what I am used to as B1. I'd really be interested in knowing how does the computer grade the free speaking. Perhaps it measures for how long you speak? I tried to say at least something and keep trying. Perhaps it looks for some key words and/or key grammar features used? I used for example the past tense but not sure whether correctly and the whole thing was very poor in info and cohesion.

The "grammar" test was still short (5 minutes this time). And I think having an "I have no clue" option would improve the exam a lot! Like one quarter of the test is on reading comprehension and vocab, not on grammar. And the rest are just multiple choice questions, therefore easy. I guessed a few times, because I really didn't know and had just some more probable candidates, so that may have inflated my score too.

I am nowhere near B2, and reading the description and comparing it to my real skills would made me laugh, if I hadn't paid 26 euros for such a lousy test.
9 x

User avatar
lysi
Yellow Belt
Posts: 98
Joined: Sat Mar 27, 2021 1:34 pm
Location: Ontario, Canada
Languages: English (N), French, Mandarin (Beginner)
x 300

Re: Italki Language Test

Postby lysi » Tue Jun 15, 2021 5:29 am

Thank you for your detailed reviews and experiences with the Italki language test, Cavesa and Jmar. This thread is now on the 6th spot for the google search "Italki language test" (which explains how this thread has 2000 views) and by far the most informative and insightful google result for the test thanks to you.
5 x


Return to “General Language Discussion”

Who is online

Users browsing this forum: Slowpoke and 2 guests