UH-What makes a good remote examination?
Suomeksi: Millainen on hyvä etätentti? På svenska: HU-Hurdan är en bra distanstentamen?
|
In this context, remote examinations refer to
|
Remote examination or another means of assessing students’ skills?
If you are ‘replacing’ an examination invigilated in a room with a remote examination, take a moment to consider whether a remote examination is the best way to measure the achievement of the course learning outcomes.
Remote examinations are only one means among many of assessing students’ skills. You can also assess learning by utilising summaries, essays, groupwork, online discussions, reports, peer and self-assessment as well as learning journals and reports.
Hold remote examinations when
- You wish students to apply the knowledge acquired on the basis of the learning material
- Students can utilise literature, the internet or other material when answering the questions
- You wish to simulate an actual professional situation (known as an authentic examination):
- Time pressure: in professional life, reports and evaluations often have to be completed by a deadline
- Students have the opportunity to ask a colleague or complete the task in a team (as groupwork)
- Students have the opportunity to google and use other sources
Also remember to take advantage of the University of Helsinki’s Yammer groups on online teaching and/or Moodle and MOOCs. Share your ideas and ask a colleague!
How do remote examinations differ from examinations held on the University premises or in the electronic Examinarium facilities?
The differences compared to examinations held in a designated venue are the following:
- Remote examinations are not invigilated:
- Students have at there disposal, for example, literature, learning materials and online sources.
- Students can also ask others for help.
- Spatial independence:
- Students complete the examination at a location of there choosing.
- Answers are produced by using a computer.
- Students complete the examination by using a device of there choosing (computer, smartphone, tablet).
- Answers written on a device help the teachers in grading, as handwriting may be illegible.
- Compared to paper-based examinations, editing and outlining answers is easier for students.
- When necessary, students’ answers can be run through the plagiarism recognition system Urkund (Urkund instructions).
What tools should be used to implement remote examinations?
The Moodle learning environment is the recommended tool for implementing online examinations at the University of Helsinki for the following reasons:
- Many courses are already using Moodle areas, which are also suited to holding examinations (students are in the Moodle area to begin with).
- Managing questions and answers is easy:
- All of the examination questions and answers pertaining to the course can be found in the course’s Moodle area (when using email, questions/answers are easily lost, particularly in the case of a large number of students).
- You can copy questions from one Moodle area to another.
- Moodle offers diverse grading and feedback tools for teachers.
- Grades can be transferred from Moodle to Excel, and Excel sheets can be submitted to those recording completed studies.
- Grades and feedback are displayed to students directly in the Assessment section in Moodle.
- Support is available
- For drawing up examinations, in the Moodle Manual for Teachers and from moodle@helsinki.fi
- For verifying the settings of examinations created by using the Assignments activity, from Examination Services
- Students have at there disposal the Moodle Manual for Students, which includes instructions on the technical aspects of examinations.
Do online examinations require more time?
Even though writing answers by using an electronic device is faster than by hand, and editing the text is easier, students may need more time especially for writing essays compared to traditional pen-and-paper examinations (Rytkönen & Myyry, 2014; Myyry & Joutsenvirta, 2015).
If examination questions activate students’ thinking, reflecting on there answers may take more time than answering questions based on recall.
Ensure the scope and number of questions is in proportion to the time available. If answering them requires more than one source/the simultaneous discussion of several subject areas, the number of questions can be lower.
As a rule, students can be prompted to define concepts and use them accurately.
Using technical devices and there functioning may make students nervous. In fact, a practice area should be created for students where they can try out taking the examination in advance. You can also experiment with holding examinations open to all by using a Moodle training course.
How are the people taking an online examination identified?
The goal is to verify students’ identity by having them submit there answers by using there University of Helsinki credentials. In the case of suspected misuse or risks pertaining to information security, credentials can be suspended. Information on credentials for students is provided, among other sources, on the compulsory Student’s Digital Skills course.
You can also approach the topic by considering how to identify the people completing essays and other learning assignments. Often, there is no certainty about this. Experienced teachers often point out that they can see if the text is not by the student in question.
What if students look for the answers in the learning materials or on the internet?
Try to design examination questions that require from students, for example, the application, analysis, comparison or combination of knowledge from different sources (see section What makes a good essay question? and examples of applied questions).
Ask students to cite sources according to the academic style of writing.
Make use of the Urkund plagiarism recognition system, which prevents direct copying from, among others, online sources. Notify students in advance of your intention to use Urkund in the examination.
What if students ask there friends for help?
Consider whether cooperation can be allowed. You can also consider holding a group examination. Cooperation is one form of learning and problem-solving, which can also be utilised in online examinations.
Attempts can be made to prevent cooperation by
- Notifying the students that cooperation is not allowed and that the answers of two or more students cannot be identical
- Mixing up the order of questions in Moodle’s Quiz activity, as well as in multiple choice examinations by mixing up the order of the alternatives. In the case of examinations taken by all course participants at the same time, this will hinder cooperation.
- Limiting the duration of the examination so that students have no time to cooperate
- Practising answering questions during the course so that those not taking the course have no advantage
- Designing the examination to require so much time and effort that it would be difficult for the student to persuade anyone to take it on there behalf
- Making the examination only part of the course assessment process, emphasising in it skills in critical thinking and knowledge construction instead of grades
Storing and sharing examination questions is extremely easy in an online environment. What can teachers do about it?
- Establish a question bank from which different questions are drawn for different students. However, this may be laborious. Than again, teachers may have such a resource already available in a non-electronic format.
- Publish the questions in advance. You can draw up, for example, 10 essay questions and notify the students that two to three of these must be answered in the examination. Or you can draw up 50 multiple choice questions, from which 10 will be drawn for the examination. Students can formulate there answers in advance, learning the subject already before the examination.
- Question banks should be created collaboratively with other teachers, even across institutional boundaries. Trade your questions! You will also gain your colleagues’ views on your examination questions.
This can also be considered from the perspective of the fact that examination questions have also been distributed in advance in conjunction with paper-based examinations. At times, teachers even allow students to take the questions with them when they leave the examination venue.
Learning is promoted when students know in advance questions that direct them to learn.
What makes a good examination question?
Ask about
- Essential/important things, see the course learning outcomes
- Things that have been taught/discussed on the course
Formulate the questions in interrogative form. In essay examinations, a ‘question’ cannot be, for example, “the Finnish climate”, as that does not contain a question.
More information in the following sections:
- What makes a good essay examination question?
- What makes a good multiple choice question?
- Lecture recordings
What makes a good essay examination question?
Check the course learning outcomes. When coordinating the outcomes and assessment, rely on Bloom’s taxonomy:
| When you want students to... | Ask them in the examination to: |
| remember | define, describe, identify, list, repeat |
| understand | classify, differentiate, explain, paraphrase |
| apply | apply, modify, choose, supplement |
| analyse | analyse, compare, combine, contrast |
| evaluate | generalise, manage, revise, plan, summarise |
| create | infer, judge, interpret, recommend, convince |
Questions that require the application of knowledge steer towards in-depth learning (the analysis and interpretation of knowledge in relation to other contexts).
Formulate questions in a way that no answer can be found directly from the learning materials or online. For example, avoid asking about individual concepts or theories, if there definitions and descriptions are easy to copy (see the examples on applied essay questions).
Whenever possible, link the application of knowledge to a topical phenomenon, which can be replaced with another in subsequent examinations.
Good applied examination answers also include the definition and precise use of concepts.
Examples of applied essay questions
| Question | What intellectual skills are required? |
| Analyse the link between constructively aligned teaching and success in learning. | Analysing, inferring |
| What is the difference between constructive alignment and approaches in teaching? | Comparing |
| How is constructive alignment related to theories on ideas of learning? | Activating previously acquired knowledge |
| On the basis of the literature you have read, design a collection of concepts that could be turned into an information package for basic education teachers. | Finding a solution, drawing connections between ideas |
| How do outsiders explain the behaviour of a student who failed to pass an examination? Consider the situation with the help of concepts pertaining to views on learning. | Analysing causality |
| What significance could constructively aligned teaching have in drawing up study plans at the department? | Analysing meanings |
| How could ideas of learning impact the work of experts in professional life? | Predicting/hypothesising |
| Come up with a case related to the University where the concept of constructively aligned teaching is utilised in improving the classroom atmosphere. | Solving problems, producing cases |
| Refute the following argument by basing your argument on the concept of constructive alignment: “Success in entrance examinations guarantees the smooth progress of university studies.” | Refuting arguments |
| How is the basic knowledge on constructively aligned teaching you have learned previously connected to the examples of utilising IT equipment in teaching presented in the Sulautuva opetus and Laadukkaasti verkossa books? | Combining information from different sources |
| Offer criticism on constructively aligned teaching from the perspective of behaviourist learning. | Critiquing |
| Apply the concept of self-efficacy beliefs to the assessment of responsibilities for personal action by a student who failed an examination. | Applying |
| Come up with three comparisons/practical examples of constructively aligned teaching. | Coming up with comparisons |
| Do you agree or disagree with the authors’ claim that constructively aligned teaching supports in-depth learning? Justify your view. | Agreeing or disagreeing |
| Analyse the link between constructively aligned teaching and students’ success in learning. | Agreeing or disagreeing |
| Is constructively aligned teaching as useful in higher education as it is in upper secondary level education? | Creating alternative realities and producing arguments that support alternative views |
| Plan a course in mathematics in accordance with the principles of constructively aligned teaching. | Planning skills |
| What analogies could constructively aligned teaching have in professional life? | Identifying and creating analogies and metaphors |
| Why is it important to take constructively aligned teaching into account when assessing learning? | Analysing how/why |
| The book illustrates the link between teaching and learning from the perspective of constructively aligned teaching. Choose two additional concepts pertaining to the link between teaching and learning from the examination literature, which are useful in analysing the example. | Presenting other perspectives |
| Assess the usefulness of the concept of constructively aligned teaching in relation to the other concepts of the book. | Appraising, justifying |
How are answers to applied questions assessed?
Consider the course learning outcomes and define the assessment criteria accordingly. Make sure students know what is expected of them and what the grounds are for there assessment. The assessment criteria must be described in detail to students at the beginning of the course. Research has shown students direct there learning largely on the basis of how there learning is assessed.
To organise assessment levels, you can use the SOLO taxonomy (Structure of the Observed Learning Outcome; Biggs & Collins, 1982), originally developed for the assessment of essay questions:
- Prestructural: fragmented, unconnected information, the student repeats what they have learned without a clear structural understanding
- Unistructural: the student only discusses one or only a few aspects relevant to the subject, nominal understanding
- Multistructural: the student explores the question from several perspectives or presents several relevant aspects, but only discussing them separately. Understanding mainly on the level of unconnected recollections
- Relational: The subject is discussed from several perspectives, which are linked together as a coherent whole. The student has understood the interconnections, but a link to the wider context is missing.
- Extended abstract: As above, but in addition the student expands the discussion outside the information given, for example, by making generalisations, exploring the matter on a conceptual level, offering different potential solutions or reflecting on the matter personally. In this case, the tractor of learning is evident, as acquired information is applied to a new context.
What makes a good multiple choice question?
Benefits of multiple choice questions include the following:
- Objectivity
- Quick and easy to grade
- A large number of questions can comprehensively cover course content
The downside to multiple choice questions is that students may guess the right answer. Guessing can be prevented by designing good questions and multiple choice alternatives.
The features of good multiple choice questions include the following:
- The question is in the interrogative form.
- The question is preceded by an introduction and necessary background information.
- The question is linked to a real-life case, such as a patient or legal case.
- The question is precise – avoid imprecise questions.
- The question does not pertain to individual facts.
- The question is not a trick question or purposely misleading.
- The structure of the question is understandable without the choices.
See further details in the section on lecture recordings.
How many multiple choice alternatives should there be and what should they be like?
The recommended number of alternatives is four. Rather than focusing on there number, focus on designing alternatives that work well.
If there are only two alternatives (e.g., right/wrong statements), guessing can produce a good score.
The alternatives should be credible, similar in substance and length, as well as ones discussed on the course.
The introduction/question can be a long one, but the alternatives should be short and concise.
It is advisable to present the alternatives in logical order, for example, in ascending order.
See further details in the section on lecture recordings (in Finnish).
How and using which tools can I draw up good short-answer questions?
See What makes a good multiple choice question? Well-formulated multiple choice questions can also be used as short-answer questions.
Both Moodle’s Quiz activity and the Exam module can be used to design short-answer questions.
See further details in the section on lecture recordings (in Finnish).
What makes a good oral examination and how are they implemented?
The situations to which oral examinations are suited include the following:
- Small groups of students
- Group examinations
- Language acquisition
Oral examinations can be carried out, for example, via Zoom or Teams ([Instructions for teaching: Educational videos and streaming your lectures]).
You can also ask students to make an audio/video recording. For this, students can use there smartphone or, for example, Zoom where PowerPoint slides or other material can be simultaneously shared. Unfortunately, large files (over 50 MB) cannot be submitted in Moodle, which makes it necessary to share recordings via, for example, OneDrive.
In the case of oral examinations, take the following into account:
- Students are not necessarily used to oral examinations, which makes it advisable to practise in advance.
- Distribute the assessment criteria to students in advance.
- Whenever possible, use two assessors (especially if no recording is made of the examination situation).
How to draw up remote examinations containing essay questions
Examinations with essay-type answers can be conducted using Moodle’s Assignments or Quiz activities. The differences between Moodle activities are described in the essay examination instructions (HY-Avoinyliopisto-Tenttiohjeet) of the Open University of the University of Helsinki (in Finnish). Support for drawing up Moodle examinations is available in the Moodle Manual for Teachers and from moodle@helsinki.fi. In addition, Examination Services provides assistance in checking the settings of completed examinations.
How to design examinations that check the answers by automated means
Remote examinations to be assessed by automated means, including multiple choice, matching and fill-in-the-blanks questions, are conducted using Moodle’s Quiz activity. Help in drawing up Moodle examinations is available in the instructions for the Quiz activity and from moodle@helsinki.fi. In addition, Examination Services provides assistance in checking the settings of completed examinations.
What are Examinarium examinations?
Examinarium examinations, also known as examinations completed in an electronic examination facility, are examinations which students take at a time and in an Examinarium facility of there choosing by using a computer. Examinarium facilities are equipped with recording cameras, while the internet access of the examination computers is restricted.
Examinarium examinations can include multiple choice, essay and fill-in-the-blank questions. Usually teachers draw up a set of questions from which questions are drawn separately for each individual student.
For Examinarium examinations, a period is determined (between one day to several months) during which students will take the examination, ranging in duration from 30 minutes to three hours, at a time of there choosing in an Examinarium facility. Support in designing Examinarium examinations is available from the Examinarium instructions on the Instructions for Teaching website and from edutech@helsinki.fi.
See also a video about a teacher’s Examinarium experiences in the lecture recordings section (in Finnish).
Lecture recordings (in Finnish)
The two lecture recordings below are almost identical in content. Browse the table of contents and watch the sections useful to you. Both are recordings of lectures by Sanna Siirilä, e-learning specialist at Educational Technology Services of the University of Helsinki.
The Akvaariorakkautta webinar also includes a video on the experiences of a teacher of Examinarium examinations.
Why draw up multiple choice questions? What makes a good multiple choice question?
Lecture recording from 2019 (MEDigi), approximately 60 minutes: https://vimeo.com/361285666 (in Finnish only)
Time (roughly) - Contents
1:30 Constructive alignment
02:10 Benefits and challenges of multiple choice examinations
02:45 Bloom’s taxonomy
03:35 Miller's pyramid
04:35 Benefits and challenges of multiple choice examinations
06:05 Strategies for passing examinations
6:55 Examples of multiple choice questions with predictable answers
16:15 Formulating multiple choice questions
19:55 Number of multiple choice alternatives
21:10 Is using right/wrong questions advisable? (No)
21:30 Formulating multiple choice alternatives
22:50 Do the incorrect alternatives have to be entirely incorrect? (No)
24:21 Formulating multiple choice alternatives
29:20 Can more than one alternative be correct?
31:15 Explain the examination rules to students, including the ban on returning to previous questions.
32:10 Questions with more than one correct alternative
33:10 Formulating multiple choice alternatives: justification
34:25 General: Question difficulty
35:15 Drawing up examinations is a process: cooperate, test and improve
37:05 Are you using the examination to measure what is supposed to be measured?
37:26 Short answers
39:39 Moodle learning analytics related to examinations
42:50 Further reading: Constructing Written Test Questions for the Basic and Clinical Sciences (https://www.researchgate.net/publication/242759434_Constructing_Written_Test_Questions_For_the_Basic_and_Clinical_Sciences) and references
44:55 Can you use ‘Don’t know’ as a multiple choice alternative?
Pedagogically good examination questions: multiple choice, short answer and essay (Examinarium question types) and a teacher's Examinarium experiences
The webinar recording Akvaariorakkautta from 14 February 2020 includes two presentations, see the table of contents: https://youtu.be/9O4OvazKoq4 (in Finnish only).
Time (roughly) - Contents
02:20 Presentation 1: Sanna Siirilä, MA (Education) and e-learning specialist, Educational Technology Services (UH): Pedagogically good examination questions: multiple choice, short answer and essay (Examinarium question types)
04:05 On examinations as learning situations, the validity and reliability of examinations
06:40 Constructive alignment and examinations, as well as Bloom’s taxonomy
09:00 Essay questions: benefits and challenges
11:52 Multiple choice questions: benefits and challenges
14:31 Formulating questions
18:33 Formulating multiple choice alternatives
20:00 Do the incorrect alternatives have to be entirely incorrect?
21:17 Formulating multiple choice alternatives
26:28 Short answers
29:08 Process of examination design
32:31 Are you measuring what is supposed to be measured?
32:44 References
34:30 Discussion: Distributing examination questions to students in advance
38:30 Presentation 2: Minna-Maarit Jaskari: Examinarium as a tool and challenge for teachers
40:10 Introduction
44:18 How to use Examinarium. Initial and final examinations, tests, maturity tests
48:03 Why Examinarium? Key benefits
55:05 Operating logic of Examinarium, choosing between subject matter and question types
59:05 Challenges: challenges related to questions
01:00:45 Challenges: short essays
01:03:10 Challenges: long essays
01:05:03 Challenges: assessment
01:06:25 Challenges: summary, analytics and permitted software
01:07:45 Concluding remarks
