Adaptive Quiz - Advanced CAT Module

The local plugin “Adaptive Quiz - Advanced CAT Module” offers extensive functionalities for implementing computer-adaptive testing (CAT) based on item response theory (IRT) in Moodle. It extends and complements the Moodle activity “Adaptive Quiz” with functions for managing a question pool, a complete CAT engine for all common IRT models, as well as numerous setting options and different strategies for test execution.

Background

Classic tests, as they are usually conducted in Moodle as a quiz, consist of a fixed selection of questions that each participant has to work on. This leads to a number of problems: Participants are regularly confronted with questions that are too easy or too difficult for them. Questions that are too easy or too difficult do not contribute any diagnostic information to the result and prove to be additional sources of error due to acceptance or concentration problems with long tests. As soon as questions are changed, added to or removed from the test, test results of the test versions are no longer comparable with each other. It is unclear how large the measurement error is in a test attempt. A practical problem with classic tests with a fixed selection of questions is the different processing speeds of individual participants. Only if a classic test is completed in full is the result meaningful. Classic tests cannot react to suspected deficits, for example, and “spontaneously ask again”.

Computer-adaptive testing (CAT) largely eliminates these disadvantages. In computer-adaptive testing, test takers are mainly presented with questions that fall within their identified ability range. Ability is assessed continuously during the test and is based on the answers given.

In contrast to traditional tests, the evaluation is not based on scores, but on statistical models of Item Response Theory (IRT). These models establish a relationship between the difficulty and discriminatory power of the individual questions and the personal ability of the test takers. Depending on the quality of the questions used, this procedure can achieve significantly shorter test times (up to 60 percent shorter) as well as more accurate results. In addition, the test results are independent of the number of questions answered individually: answering more questions leads to a more accurate result, but not to a different result. This not only allows the selection of questions to be made dynamically during the test, but also allows fixed processing times to be guaranteed by using a variable number of questions. This means that tests can be reliably carried out within the scheduled time frame.

Scope of functions

The plug-in offers you the following functions:

• Conducting adaptive tests, optionally for
  • Determining general abilities in the queried scale
  • Diagnosing the greatest deficit / greatest strength of the individual test participants
  • Adaptive or partially adaptive testing of all stored sub-scales for an ability
  • classic testing (all questions of the scale are asked) with evaluation based on the Item-Response-Theory (IRT)
  • Tests can be limited in terms of the number of questions, time and standard error to be achieved
• Use of different IRT models for
  • questions without partial scoring: 1PL Rasch model, 2PL Rasch-Birnbaum, 3PL mixed Rasch-Birnbaum
  • Questions with categorical answers or partial credit: Graded Response Model (GRM) and generalized GRM, Partial Credit Model (PCM) and generalized PCM
• Setting of differential feedback
  • in 1 to 10 ranges of the competence spectrum
  • as media-text-field as well as
  • additionally with registration in courses or groups
• Backend for managing your question pools (CAT Manager)
  • Organization of questions from your Moodle question collection for scales and sub-scales
  • Import, export and editing of question characteristics such as item difficulty and item selectivity
  • Versioning of question characteristics
  • Organization of pre-configured test settings
  • Calculation of item statistics from tests carried out in Moodle (from version 1.3, end of 2024) for the different IRT models

How to Use

Despite the fact that the plugin supports you in preparing and carrying out computer-adaptive testing and makes your work easier in the best possible way, using the plugin involves some requirements and effort. The following sections present these requirements in terms of

• installation of the plugin,
• organization of the question pool,
• test setup and
• access to feedback and statistics in the course.

Installation of the Plugin

System Requirements Settings in php:

• max_memory: 256 MB, better: 512 MB or higher
• max_executiontime: 90 s, better 180 s or higher

Necessary Moodle Plugins:

• local_wunderbyte_table
• mod_adaptivequiz (Develop-Version of Wunderbyte) and adaptivequizcatmodel_catquiz, here in the most current versions in the package

Recommended Moodle plugins:

• filter_shortcodes, needed to display feedback and statistics in course areas

During installation, a new role, CAT manager, is added to your Moodle instance. CAT managers have the necessary rights to create CAT scales and to add questions as items to these scales (see next section). For this purpose, you will have access to a separate area in the Moodle instance called CAT manager.

You can access it by clicking the “CATQuiz” button in the header of your Moodle instance at the top right. To test the functionality of the CAT plugin, you can access a prepared dummy question pool. This is how you do it:

1. Download the prepared question export simulation.xml and import it into your Moodle question catalog. To make the questions accessible everywhere in your instance, you should insert them in a directory under “System”.
2. Download the corresponding parameter file simulation.csv and import it in the CAT Manager under CAT Manager > Import as a CSV file separated by semicolons.
3. Check under “Scales” whether the simulation data pool has been correctly read in and
4. create your first adaptive test with the scale “Simulation” in a course of your choice.

Organization of the question pool

After installation, administrators will find a button labeled “CAT Quiz” in the upper right corner of their Moodle page. This button is only visible to administrators and people who have been given the “CAT Manager” role.

In this section, you can add questions from your Moodle question catalog to CAT scales, assign item parameters to individual questions, and manage test templates and versioning. You can also conveniently import a larger number of question items, including the scale structure, from a CSV file.

To ensure system-wide accessibility to your questions, we recommend that you store these questions in the question catalog under categories assigned to the system level.

Setting up tests

As soon as the first scales have been set up, you can create the “Adaptive Test” activity in the courses of your choice. Make sure that you select the option “Catquiz CAT model” under “CAT-Model”. This activates the Advanced CAT module and the functions provided by this plug-in are activated.

Now select the desired (global) scale and the corresponding subscales. Then select the desired test strategy under “Test purpose”. The following strategies are available:

• “CAT“ - questions are adaptively drawn from all selected scales. An evaluation only takes place with regard to the global scale.
• “Diagnose the lowest competency gap” or “Diagnose the greatest strength” - the test algorithm specifically searches for striking deficits or strengths in the competence range of the person being tested and returns the most significant deficit or the greatest strength.
• “Check scales in the proximal development zone“ - the test determines the current competence spectrum of the tested person and their abilities with regard to the scales located therein.
• “Check all scales” - the test will check all the scales given above with the given minimum number of questions, regardless of whether they are too easy or too difficult for the person being tested.
• “Classic test” - as with a normal Moodle quiz, all questions in the set scales are tested, but the evaluation is done using IRT.

After you have defined the scales and test strategy, set the framework conditions: minimum and maximum number of questions, accepted standard error and, if necessary, time limits. If you are unsure, it is recommended that you first accept the default settings and adjust them later if necessary.

Then insert your information for differential feedback for each of the selected scales. This can be done in up to 8 ability levels, whose cut-scores you can define as logit values accordingly. You can also insert textual feedback, enroll in one or more courses, or assign tested persons to the specified groups by entering group IDs separated by commas.

Access to feedback and statistics

Immediately after completing the test attempt, participants will see their feedback. This consists of the general text block and the differential text part or parts that have been stored according to scale and competence range. In addition, a bar with several tabs is displayed, under which additional information regarding the test result in comparison to the group and to the person's own previous test attempts is displayed. The test process and the detailed results achieved in the individual scales can also be found here.

To display feedback and statistics about the test at a later date, this information can be integrated at any point in the course using a shortcode. The shortcodes

• [catquizfeedback]
• [catquizstatitics]

are available for this purpose.

FAQ (Frequently Asked Questions)

• What is the difference to the Moodle Quiz and Adaptive Quiz?

Moodle's quizzes test classically, i.e. on the basis of scores and fixed sets of questions - with the consequences listed in the “Background” section.

Adaptive Quiz is a very rudimentary implementation of the 1PL Rasch model with only integer difficulty levels, which have to be laboriously stored as tags in the questions themselves. This plugin offers you the use of all common IRT models, including those that take into account partial credit or rate probabilities. It also has a user-friendly interface for organizing questions, scales, test settings and item parameters.

Both Moodle's Quiz and Adaptive Quiz are not able to provide differentiated feedback and independently enroll students in groups or courses based on their results. In this respect, both are only partially suitable for adaptive learning settings and differential diagnoses.

• Moodle Quiz also provides me with information on question difficulty and a selectivity index. What is the difference?

These figures are always based on the specific composition of the questions and the test takers. As soon as you change your test and add significantly more difficult or easier questions, or test a stronger or weaker group with the same test, you will get different values. These values in Moodle are therefore based on the test results achieved.

The question difficulty and the item-trennschärfe as item parameters of this plugin, however, are fixed values for the tests, to which the test results are aligned. In this way, measurements of different groups of participants with different question compositions can be compared, as long as they were carried out with the same set of item parameters.

• Where do I get the item parameters from?

Currently, you still have to estimate the item parameters yourself, e.g. in R using the software packages mirt or TAM. Later versions of the plugin will also have the ability to calculate item parameters directly in Moodle from the Moodle data.

• Where is the data processed? Is a third-party service provider involved?

All data is processed entirely on the Moodle instance on which the plugin is installed. In particular, no other third-party services are used.

• Is the plug-in really free of charge? Are there any (hidden) costs?

Yes, the plug-in and its full range of functions is (and remains) free of charge.

• Why can't I navigate back to an earlier question during the test?

The idea of computer adaptive testing is that each answer entered and submitted is used to estimate the person's ability in order to decide which question to ask next. If a past question were answered differently, the entire further course of the test would be called into question. For this reason, free navigation through the test is not possible and not allowed.

• Is it worth my while to use adaptive tests with this plugin?

The plugin supports you in the preparation and execution of computer-adaptive testing in the best possible way. Nevertheless, you will need a basic understanding of CAT and the effort involved in organizing the item pool and setting up the test.

If you are currently satisfied with the way the Moodle quizzes meet your requirements, it is not advisable to switch. However, if some of the points from the “Background” section are important to you, the advantages associated with the CAT procedure could offset the costs of getting started with this topic.

• Where can I find more information on the calculation methods used in this plugin and on the topics of CAT and IRT?

A publication specifically about this plugin is currently in preparation. Until it is published, we refer you to the basic article (Linacre 2000).