A New Receptive Vocabulary Size Test for French
Abstract:
Researchers have developed several tests of receptive vocabulary knowledge suitable for use with learners of English, but options are few for learners of French. This situation motivated the authors to create a new vocabulary size measure for French, the
Test de la taille du vocabulaire (TTV). The measure is closely modelled on Nation’s (1983) Vocabulary Levels Test (VLT) and follows the guidelines written by Schmitt, Schmitt, and Clapham (2001). Initially, a pilot version was trialled with 63 participants; then an improved
version was administered to 175 participants at four proficiency levels. Results attest to the TTV’s validity: mean scores across the four frequency sections decreased as the tested words became less frequent, and more proficient learner groups outperformed less proficient groups. The TTV in its current form is intended to be of practical use to teachers and learners, but it is also expected to evolve; ideas for future improvements are discussed.
An important characteristic of receptive size measures is the frequency-informed selection of test items. Analyses of large corpora
have resulted in lists of the most frequent word families or lemmas of a language (a lemma differs from a family in that it includes a headword and its inflections but does not include derived forms).Frequency-informed size testing is based on the assumption that the most frequent words of a language will be learned early, while less frequently encountered words will be learned later. An overview of research by Milton (2009) provides evidence that this assumption is
sound:
But it is also clear that in the cases of certain words and certain learners, learning may not always follow a strict frequency order. Research by Bardel, Gudmundson, and Lindqvist (2012) shows that Swedish-speaking learners of French can readily recognize some infrequent French words (e.g., e´lectrique, vocation) due to their resemblance to words that have been borrowed into Swedish.
These researchers also identified
“thematic” words that are rather infrequent in French generally, but are likely to be known to learners
because they occur frequently in classroom input. Nonetheless, frequency appears to be a powerful factor in vocabulary learning across
groups of learners. Milton’s (2009) analysis of learnability factors found the frequency of a word to be a much stronger predictor of its being learned than cognateness, word length, and part of speech.
It is important to note that the number of L2 word families that a
learner knows receptively is just one of several measurable dimensions of vocabulary knowledge (see Nation [2013] for an overview).
Instruments used by researchers interested in the acquisition of French, for example, include measures of lexical diversity in speech production (Tidball & Treffers-Daller, 2007), “depth” measures that assess learners’ ability to recognize
collocates and other word associations (Bogaards, 2000; Greidanus, Bogaards, van der Linden, Nienhuis, & de Wolf, 2004), and profiling software that identifies proportions of
advanced lexis in speech samples (Bardel et al., 2012). A study by Forsberg Lundell and Lindqvist (2014) uses
innovative measures of productive collocation ability and lexicopragmatic knowledge. Generally, the instruments mentioned above have proved their usefulness in answering questions about relationships between different kinds of lexical knowledge and effective ways
of distinguishing between groups of varying proficiency levels. But most of them target fairly advanced university learners of French, and
they are accessible mainly to researchers. In our view, there is a need for a new, freely available French vocabulary size test suited to assessing learners of a wide range of proficiencies. It is also important that the test be easy to administer and that it produce readily interpretable scores. Our size test was designed with these practical goals in mind.
What are some of the typical vocabulary sizes identified using these instruments? An investigation using the VST, reported by
Nation (2013), found that learners of English who were able to perform adequately in undergraduate studies at an English-medium university had vocabulary sizes of 5,000 to 6,000 word families. Learners studying at the doctoral level were found to have a vocabulary size of around 9,000 English families. Analyses of the coverage of frequency lists for a variety of text types by Nation (2006) show that learners would need knowledge of the 8,000 to 9,000 most frequent English word families to understand 98% of the vocabulary that occurs in novels written for native speakers. The 98% criterion is based on research by Schmitt, Jiang, and Grabe (2011) and others (see Nation [2013]), which indicates that knowledge of 98% of the words in a text is a reasonably good guarantee that it will be comprehended adequately. Size research has also investigated native speakers; a study by Goulden, Nation, and Read (1990) indicates that university-educated adults may know around 20,000 English word families.
What does vocabulary size research have to say about learners of French? In a study tellingly entitled “Language Lite,” Milton (2006) reports that after hundreds of hours of French study over seven years in British secondary school programs, learners were found to have a recognition vocabulary size of only 1930 lemmas (SD = 475), according to mean scores on the checklist test. Similar modest figures are reported by David (2008) in a study that also investigated secondary learners in Britain using the checklist test. In a follow-up to his study of secondary learners, Milton (2008) investigated university students. He reports that
after an additional four years of study at a British university, including a year abroad spent in France, students’ mean French vocabulary size reached 3,326 lemmas (SD = 579).
Although the research discussed above sheds some light on the amount of vocabulary needed to complete school French programs
in Britain, many other questions remain.
To our knowledge, the vocabulary sizes that learners would need to read a French novel without assistance, follow the dialogue of a movie, study at a French medium university, or achieve other learning goals they may have are largely undetermined. Corpus counts by Cobb and Horst (2004) suggests that
knowledge of words on the 2,000 most frequent French list is likely to be a powerful asset, offering a possibly even higher level of known-word coverage than in English. But to our knowledge this potential has not been investigated experimentally with L2 learners. Nor are we aware of research that specifies the number of words that native speakers of French can recognize, as indicated by their performance on a size test. One explanation for this research shortfall may be the
unavailability (until recently) of good corpus-based frequency lists for French.All these concerns informed our decision to create and trial a new French size test suitable for use in classrooms, the Test de la taille du vocabulaire (TTV). VST, which presents four answer options for each target word. A possible point in favour of the VST is its sampling of English words from all of 14 corpus-based frequency bands, which gives it the ability to test a wide range of learner knowledge. But French word lists drawn from a large modern corpus at 14 different levels of frequency were not available to us. However, we were able to take
advantage of recent work by Lonsdale and Le Bras (2009), whose list of the 5,000 most frequent lemmas is based on a 23-million word corpus of current written and spoken international French.
Unlike earlier French corpora that are based largely on written language (e.g. Baudot, 1992; Verlinde & Selva, 2001), this corpus has a substantial spoken component (50%). The Lonsdale and Le Bras list was used to construct a pilot version of the TTV following the VLT, with sections that sample the 2,000, 3,000, and 5,000 frequency levels.
The VLT also has sections that test the 10,000 level as well as Coxhead’s (2000) Academic Word List, a list of families that occur frequently in university textbooks. Like the VLT, the TTV tests words at the 10,000 level, but since the Lonsdale and Le Bras lists go only as far as the 5,000 frequency level, we turned to an older list by Baudot (1992) to create this part of the test. The decision to include the same frequency levels as the VLT was made with a view to enabling eventual comparison studies of L2 English and French vocabulary development. However, we did not include a section on the TTV that parallels the Academic Word List section on the VLT. Such a list has not been determined for French, and it may well not exist.
Research by Cobb and Horst (2004) indicates that while a distinct academic lexis (largely Greco-Latin) is characteristic of English, this is likely not the case in French or other Romance languages. Creating the pilot TTV involved first sampling the test words and distractors at random from the 2,000, 3,000, 5,000 and 10,000 frequency lists (henceforth referred to as 2K, 3K, 5K, and 10K) and creating test clusters for each level.
Interviews
Before moving to the larger study, we probed test-takers’ knowledge of tested words in individual interviews. The procedure was intended
to explore the extent to which learners actually knew words they had correctly matched to definitions on the TTV.
When the TTV in its piloted and improved form was administered to 175 learners of French at four levels of proficiency at a school in Montreal, it functioned as expected: The sections that tested less frequent words proved more difficult than sections with more frequent words; means for the four frequency sections differed significantly. Since research has shown that learners generally acquire more frequent words before they acquire less frequent ones (Milton, 2009; Nation, 1990), the frequency findings speak to the validity of the test.
Performance of individuals did not always follow this neatly descending pattern, however. For example, several learners in the advanced group scored higher on the 3K and 5K sections than on the 2K section. Milton notes similar results in his 2009 overview of vocabulary size testing. One possible explanation for this finding comes from Milton’s (2007) study, which compared learners with “normal” profiles to those with a 2K deficit and found evidence of an aptitude effect. More research of this type is needed to understand how individual learners respond to frequency in the input to which they are exposed. Also, as Bardel et al. (2012) have found, exposure to thematic classroom vocabulary and the availability of L1 cognates can facilitate the learning of infrequent words; these factors may have been in play here.
The testing also identified proficiency differences in the expected direction: the higher the proficiency level of the group, the greater the
mean scores on the test. These differences were statistically significant. The finding that greater vocabulary sizes were associated with more advanced proficiency (as determined by the school’s placement measure) lends credibility to the test and points to its potential usefulness in helping to place students in language courses.
In terms of hours of instruction, the beginning Quebec participants in the TTV study, who have completed 330 hours of study, can be seen as roughly comparable to British secondary learners in year 5, who have completed an estimated total 351 classroom hours (78 + 58.5 + 58.5 + 78 + 78), according to Milton’s figures. As shown in the first row of Table 4, the mean vocabulary size for the Quebec learners amounts to an estimated 2,699 words. This stands in marked contrast to the mean size of just 852 words reported for the British learners after a similar amount of time in class. Another comparison might be made between the low intermediate Quebec learners with 660 hours spent in class and the British learners, who are reported to have spent a total of 643.5 hours in class by the end of seven years of secondary school.
Again, the difference is large. The mean size for the Quebec learners shown in the second row of Table 4 is estimated at 4,068, while the British figure is 1,930 (with considerable variability in both groups). Arguably, these very great differences call the TTV’s measurement capabilities into serious question.
But are the Quebec figures wildly implausible? There are several reasons to think they are not. First, the TTV is designed to measure size through the 10K frequency level, while the maximum level assessed on the checklist test is 5K. This gives the Quebec learners a considerable advantage in terms of opportunities to demonstrate word knowledge. Another explanation pertains to the frequency lists used to build the two size tests. The checklist test draws on the Baudot (1992) list, which is based on a corpus of written materials, but the TTV draws for the most part on work by Lonsdale and Le Bras (2009), whose corpus contains a large spoken component (50%). In other words, the character of the lists sampled to build the measures differs considerably, and it is possible that this makes the TTV an easier test.
There are also important differences in exposure to target language input in the two learning contexts. In Milton’s study, the participants were learning French as a foreign language at school while living in an English-speaking milieu. By contrast, the TTV participants live and work in a French-speaking society, and therefore they have a great deal more exposure to target language input. Acquiring proficiency in their new language promises social and economic benefits, so the Quebec participants are likely to be motivated learners. There is also
research evidence that
intensive instruction leads to greater proficiency gains than does a distributed “drip feed” program (Serrano &Mun˜oz, 2007; White & Turner, 2005), which seems a fair characterization of the classroom situations investigated by Milton. By contrast,
Quebec francisation programs promote rapid integration...
...all of the TTV participants spent at least 12 hours per week in class; most of them spent as many as 30. This may well have given them a vocabulary learning advantage over the British learners, who appear to have attended only two or three hours of class per week during most of their seven years of study. Finally, over a third of the students who took the TTV were speakers of Romance languages and were therefore probably able to recognize many words on the test due to familiarity with cognate equivalents or near-equivalents in their first languages.
To determine the extent to which the TTV might have advantaged participants with a Romance-language background, we divided the 175 participants into three rough first language groups:
Romance language speakers, Asian language speakers, and speakers of other languages. The Romance group consisted of 67 speakers of
Portuguese, Romanian, and Spanish. The Asian group consisted of 27 speakers of Korean, Mandarin, Teochew, and Vietnamese; these East Asian languages are typologically distant from French and have not been as strongly influenced by Latin as English has been, for instance. The “other” group consisted of 81 speakers of Farsi, Russian, Tagalog, and 11 other languages (see the Participants section above). The means in these three groups were calculated for each of the four frequency sections and for the test as a whole. As can be seen in the first row of Table 5, means on the 2K section were distinctly higher in the Romance group, at 25.40 (maximum score = 30), while the means in the two other groups were both lower, at around 17.80. This pattern is also seen in the other frequency sections, with
Romance speakers outperforming the other two groups by substantial margins (and with more consistency, as the smaller standard deviations indicate). When means for total scores in the three groups were tested via a oneway ANOVA, significant differences were found, F(2, 174) = 59.11, p < .0001. Post hoc pairwise comparisons confirmed
a statistically significant advantage for the Romance speakers over both of the nonRomance groups, but
there was no statistically significant difference between the two non-Romance groups (p < .01)...
Conclusion
There is an imbalance in available corpus-based resources for vocabulary research and pedagogy in the case of L2 French, with a great deal more in the way of frequency lists, size tests, and learning activities available to those interested in English. One of the goals in creating the TTV was to help redress that imbalance by drawing on state-of-the-art corpus work in French to create an updated receptive vocabulary size measure and make it available to the teaching and research community. To this end, the TTV appears in its entirety at the testing link on Cobb’s Lextutor website (
www.lextutor.ca)
https://www.lextutor.ca/tests/batista_horst_2016.pdfhttps://www.lextutor.ca/tests/ttv/?mode=test