Humboldt-Universität zu Berlin - CALLIDUS

Research findings

The DFG-supported research project CALLIDUS presents key research results at the end of the project period, which relate both to the studies carried out and to the software development.

The studies

Four intervention studies were carried out in three Berlin secondary schools, in which 283 learners participated in 11 groups or classes. The sample is a non-randomized ad hoc sample to preserve the authentic context. On top of that, the selection of participants must be considered biased, as it depended on which teachers were willing to cooperate. Personal information about the learners was not collected.

 

Study 1: Intermediate Learners(Cicero)

Purpose: Assessment of the status quo - Vocabulary knowledge and linguistic competence in general

Groups: Test group = 27 participants, control group = 19 participants

Age: 13-15, 9th grade [+ only a test group: 16-18, 12th grade]

Level: End of the 5th year learning Latin

Study design:

  • Pre-test (45min): Testing of vocabulary knowledge and vocabulary competence in five categories (basic morphological knowledge, strategies for word association and word inference, phrases, vocabulary knowledge in context), including tasks for supported language production
  • Intervention (4 weeks, Cicero): Original literature using a special text edition of Cicero Ad Quintum fratrem I, 1; supplemented in the test group by a vocabulary intervention (tasks on basic form, polysemy, word formation, word pairs, collocations, strategies); vocabulary relevant for the tests: 172 lemmas of the 500 most frequent words of the Bamberg vocabulary
  • Post-test (45min): identical to the pre-test

Results (Data related to the study):

The very short and rather inconvenient (before the summer break) intervention study does not show any improvement in performance in the test group, as the stimulated vocabulary work obviously needs more time and should be limited more to a few selected phenomena in order to have an effect. Accordingly, the already existing (cognitive) performance differences between the two groups remain in the post-test. These are particularly noticeable when comparing certain task types: In the task type "Determine the form (of the given word)" the results in the pre-test do not differ on average (Q1-Q3 = 1-3 points out of 5) and in the post-test only slightly in favour of the more capable control group (Q1-Q3 = 1-3 (t), 2-4 (c)). However, in the task "Derive word meaning from your entire vocabulary knowledge", which is more strongly oriented towards abstraction and association, the more proficient control group already scores better in the pre-test and even better in the post-test in absolute terms (median in the post-test: 3 out of 3 points (c) compared to 1 out of 3 points (t)). In addition, interesting linguistic "errors" in the tasks for word formation and supported language production could be observed, e.g. ab-scribere as an analogy to the word formation rule from L1 (German: ab-schreiben, i.e. to copy). NB: The teacher's feedback on the task formats was very positive.

Study 2: Intermediate Learners (Ovid)

Purpose: Test of vocabulary knowledge in context

Groups: Test group = 21 participants; control group = 27 participants

Age: 14-16, 10th grade [+ only test group: 17-19, 13th grade]

Level: Start of the 6th year of learning Latin

Study design:

  • Pre-test (45min): 10 tasks on vocabulary competence in the context of a metamorphosis (Ovid Metamorphosis IV, vv. 271-388: Salmacis), including metacognitive tasks, e.g. how a basic form can be derived from a given word
  • Intervention (8 weeks, Ovid): original literature with the aid of a special text edition on Ovid Metamorphosis IV, vv. 55-166 (Pyramus and Thisbe); in the test group through vocabulary intervention (context-based tasks on metacognitive strategies and lexical representation in the mental lexicon (form, function, concept level)); vocabulary range relevant for the tests: 165 lemmata of the 500 most frequent words of the Bamberg vocabulary
  • Post-test (45min): structurally identical to the pretest, but based on a different metamorphosis (Ovid Metamorphoses X, vv. 243-297: Pygmalion)

Results (Data related to the study):

The test group proved to be the (according to the teacher not unexpectedly) weaker group in the pre-test. The average score was almost 20% below the average score of the control group. However, while the average performance of the control group in the post-test fell by 2%, the test group increased their outcome in the post-test by 11% on average. The groups thus converged in their overall performance. This result suggests that a longer-term, systematic intervention contributes to an improvement in performance, especially in weaker learning groups, due to the explicit vocabulary work. Nevertheless, there were also some questionable results when looking at the tasks and the performance separately, e.g. tasks to justify or explain one's own actions (--> metacognitive reflection). Another result that has particularly serious consequences for text work in Latin classes is that the majority of learners cannot name the basic form of a given word, which comes from the basic vocabulary of the 500 most frequent words, let alone derive this form.

Study 3: Beginners (VIVA)

Purpose: Testing a DDL concept for vocabulary acquisition

Groups: Test group 1 = 27 participants, test group 2 = 27 participants; control group 1 = 25 participants; control group 2 = 24 participants

Age: 10-11, 5th grade

Level: first year of learning Latin

Study design:

  • Pre-test (45min): Test of reading comprehension and vocabulary; German text about the Punic Wars, readability: grade 7
  • Intervention (1 year, textbook VIVA): DBR approach; materials supplement the textbook VIVA with context-based vocabulary tasks
    • Context-based introduction of the new vocabulary as a bilingual cloze text, in which German meanings of the new words are to be deduced from the context
    • Working materials both on vocabulary knowledge and strategies
    • Exercises on concepts and vocabulary
    • Vocabulary test (DDL test) with a recurring structure (--> name, identify, use, form, explain)
    • Vocabulary test as a word list after every 3 lessons (randomized and vocabulary by part of speech evenly distributed over each test)
  • Post-test (45min): identical to the pre-test

Results(Data related to the study):

As a side effect, the test methods used show that the four participating fifth grades have already become significantly more heterogeneous in their performance during their first year at secondary school. Such a development, if it continues to the same extent in the following years, can ultimately lead to such large disparities that in the end even entire grades have to be described as "left behind" and thus the bringing together of learners in the course system will clearly disadvantage the weaker ones in particular.
Now to the DDL-specific results: DDL vocabulary tests are basically also manageable for young learners. However, they need some time - as do the teachers, by the way - to get used to this new format. At the same time, the results suggest that results in DDL tests and (traditional) list tests are not correlated. However, it also implies that (cautiously put) the result in a list test does not allow a statement to be made about how well a learner will perform in text work (text comprehension, translation).

Study 4: Intermediate Learners (Vocabulary Unit in the MC)

Purpose: Comparison of context-based learning vs. word list; simulation of a Latin lesson with vocabulary focus (ca. 40min)

Groups: Group 1 = 13 participants

Age: unknown

Level: intermediate

Study design:

  • Pre-test (4min): 5 exercise types for vocabulary competence
  • Introduction and Reading comprehension (18min): German introduction to the Latin text (Cicero Ad Atticum I, 1, 8-10), marking of the predicates, reading comprehension on the basis of a bilingual version and a German translation --> preparation of the vocabulary exercise, since it must be solvable equally well for all participants, regardless of the previous lessons
  • Intervention (12min): randomised group assignment; test group = cloze text, control group = list learning
  • Post-test (4min): identical to the pre-test
  • Evaluation(--): Performance overview of completed tasks --> individual results per section and comparison between pre-test and post-test (percentage change)

Results(Data related to the study):

It was found that participants who passed their test very quickly also had a more pronounced vocabulary competence overall (measured by the number of points achieved and the lower number of errors). With regard to the actual question of whether corpus-based vocabulary learning supports vocabulary acquisition, it can be said with due caution that this methodology at least correlates with better results in the post-test.

Conclusions

  1. In the evaluation of the tests, it was helpful to have the learners assess how comprehensible they thought the task phrasing was and how difficult they thought the tasks themselves were.
  2. In order to be able to evaluate the results more precisely, in future tests the tasks must be broken down into the individual steps marked by operators, so that the scores can be assigned separately for each step.
  3. Metacognitive skills are poorly developed and must be trained much more explicitly in Latin classes.
  4. Various vocabulary work in Latin classes, which obviously helps to deal with texts more successfully, is generally appreciated by learners.
  5. The language proficiency of Latin learners is generally weak:
    • They fall back on linguistic knowledge from L1 if they are to solve e.g. word formation tasks.
    • Automatic information retrieval from the mental lexicon is rarely successful, e.g. when searching for the basic form.
    • Learners have considerable difficulty associating Latin words with other Latin words, e.g. when searching for synonyms. They almost always take the detour of L1 and are also encouraged to do so by learning word equations.
    • Learners do not get beyond the lowest level of language acquisition - formal level according to Jiang 2000 (Lexical Representation and Development in a Second Language. Applied Linguistics 21 (1), 47-77) - which, however, is by no means sufficient to understand words in context or even to be able to translate them.
  6. The chosen DDL approach refers only to intentional learning - in contrast to random learning on authentic texts - , since Latin original texts never (!) fulfil the required vocabulary coverage of 95-98% of known words for the latter. Therefore it is important to reflect on the methodical approach to solving the tasks. The work with the context - referring to the representation levels of a lexical unit as well as to the surrounding lexical units or the entire text section - has to be done explicitly.
  7. The DDL approach does not confront learners with major problems if they are made aware of the context-sensitive work with "words". Therefore, it should be seen as an enrichment of Latin classes and should be incorporated into teaching Latin as part of a newly conceptualised vocabulary work.
  8. In principle, more empirical studies should be carried out in Latin learning environments in order to develop teaching Latin on a more solid database.

 

Publications related to the studies:

  • New Insights and Methods of Vocabulary Acquisition in Latin Classes. Beyer, A. Forma y Función. 2020. Currently in peer review. (all studies in an overview)
  • Using NLP to Create Corpus-based Vocabulary Exercises in Latin Classes. Beyer, A. and Schulz, K. In: INTED2020 Proceedings, pages 1750–1757, Valencia, 2020. DOI: 10.21125/inted.2020.0562. (focus: study 4, MC)
  • Im Lateinunterricht: „cupidine kommt von cupidi und ne ist Fragepartikel.“ – Wortschatzprobleme und ihre Ursachen. Beyer, A. In: Pegasus Online-Zeitschrift 2019, 1-19. (focus: study 2, Ovid)

 

The software

A software has been developed that can be used to conduct corpus-based vocabulary studies in schools. It contains ready-made exercises for controlled experimental conditions, but also allows individual re-use by creating completely new exercises. The basic principle is always the same sequence: text selection > linguistic analysis > determine exercise design. If necessary, additional information can be retrieved (vocabulary comparison, keyword-in-context view, word fields) to identify particularly suitable text passages and vocabulary.

Text selection (corpora) and natural language processing

Numerous existing resources and methods were used for the software and meaningfully networked for the purpose of vocabulary acquisition:

  • Corpora
    • Raw texts
    • scholarly annotated texts (treebanks)
      • Pragmatic Resources of Old Indo-European Languages (PROIEL)
      • The Ancient Greek and Latin Dependency Treebank (AGLDT)
  • Natural Language Processing
    • Segmentation
    • Tokenisation
    • persistent referencing using Canonical Text Services (CTS)
    • Part-of-Speech-Tagging
    • Dependency Parsing
    • Text complexity
    • Dynamic vocabulary comparison between any text passage and reference vocabulary (e.g. the German Bamberger Wortschatz)
    • Representation Learning (Word Embeddings)

Problems occurred mainly with the quality of automatically generated annotations, e.g. on the syntax of a text passage. Here, the tool used (UDPipe) often constructs correlations incorrectly or names them incorrectly. Further measures to increase the quality of the generated learning materials for teaching purposes are currently being researched.

Text complexity

Different indicators of text complexity are displayed for each text, e.g. the number of subordinate clauses or complex syntactical constructions. For a quick overview and for better comparability, all indicators are combined into a total score, which includes, among other things, already common measures of complexity such as the lexical density.

Exercise formats: Integration, evaluation and export

The generated exercises are independent of the device used and are always interactive, i.e. they can be edited directly in digital form. Each exercise can be embedded by a short script on external websites, i.e. also in newer versions of the learning platform Moodle. The evaluation of an exercise is fully automatic and is persisted in the project's own database. This makes it possible to carry out later analyses across persons and time periods, e.g. on the effect of different exercise types on language acquisition. This link to the research of the CALLIDUS project can be broken if necessary by users exporting their exercises offline, for example as an editable Word document, as a PDF or as an XML document.

The design of the vocabulary unit

The ready-made exercises offered by the CALLIDUS project are linked to form a meaningful unit that can be mastered within one lesson (approx. 40min). It contains diagnostics (identical pre- and post-test), an introduction to the texts to be worked on and specific vocabulary training. Furthermore, the users are randomly assigned to test or control groups so that the effect of the intervention can be determined afterwards. In concrete terms, this means the effect of the exercise format (cloze vs. vocabulary list) on qualitative and quantitative aspects of vocabulary acquisition. In order to ensure that all parts of the unit are at least partially processed by the participants, a timer runs in parallel with pre-determined maximum processing times for each section. After the timer expires, the exercise environment automatically jumps to the next section so that learners do not spend too much time on particular tasks.

Prototype: Semantic Analysis

The dynamic, corpus-specific identification of abstract representations (e.g. word fields) has so far only been realized prototypically by means of a few examples. However, the basic principle can be easily transferred: For a selected text collection, a distributional semantic model is trained, which is then able to find words related in content and even find entire text passages on a given topic. This enables teachers to find suitable terms and texts to convey certain contents or to test semantic hypotheses empirically.

 

Publications related to the software:

  • Natural Language Processing for the Classics - Teacher's Toolkit. University of Kiel, virtual conference: Teaching Classics in the Digital Age, Schulz, K. June 2020. DOI: 10.5281/zenodo.3928068
  • Machina Callida. Schulz, K. February 2020. DOI: 10.5281/zenodo.3662709.
  • Development of Basic Reading Skills in Latin: A Corpus-based Tool for Computer-assisted Fluency Training. Kühnast, M., Schulz, K. and Lüdeling, A. Journal of Latin Linguistics. 2020. Currently in peer review.
  • Building a Database of Exercises for Learning Latin. Schulz, K. New Classicists. 2020. Currently in peer review. 
  • A Data-driven Platform for Creating Educational Content in Language Learning. Schulz, K.; Beyer, A.; Dreyer, M.; and Kipf, S. In Proceedings of the Conference on Digital Curation Technologies (Qurator 2020), Berlin, January 2020.
  • CALLIDUS Project—Learner Data from a Study on Latin Language Learning (Version 2.0). Schulz, K. 2020. https://doi.org/10.5281/zenodo.3601182.