CodeRunner question type

From MoodleDocs
Revision as of 08:53, 1 June 2023 by Klaus Steitz (talk | contribs) (Downloads Link-Fix)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


CodeRunner is a free open-source question-type plug-in for Moodle that can run program code submitted by students in answer to a wide range of programming questions in many different languages. It is intended primarily for use in computer programming courses although it can be used to grade any question for which the answer is text. It is normally used in Moodle's adaptive quiz mode; students paste in their code in answer to each programming question and get to see their test-case results immediately. They can then correct their code and resubmit, typically for a small penalty.

For example:

CodeRunner example 01.png

Demo site

Demo quiz

To see how CodeRunner quizzes behave, try taking the Python3 demo quiz. You'll be asked to register as a user on the site first, but this is painless and straightforward. Don't worry, we won't spam you as a consequence.

The Python quiz is recommended to give you a feel for CodeRunner's capabilities, even if you don't understand Python3; the questions could (for the most part) be written in any of the supported Languages (Python2, Python3, C, C++, Java, PHP, JavaScript, Octave).

The second quiz shows only a few simple C and Java questions.

CodeRunner Authors' Forum

A forum for exchange of question ideas and suggestions, bug-reports or anything else related to CodeRunner. Subscribe yourself to this forum if you wish to receive such news.

CodeRunner description

CodeRunner is a Moodle question type that allows teachers to run a program in order to grade a student's answer. By far the most common use of CodeRunner is in programming courses where students are asked to write program code to some specification and that code is then graded by running it in a series of tests. CodeRunner questions have also been used in other areas of computer science and engineering to grade questions in which many different correct answers are possible and a program must be used to assess correctness. However, the focus throughout most of this document will be on programming questions, where the students' code is being graded.

Regardless of the behaviour chosen for a quiz, CodeRunner questions always run in an adaptive mode, in which students can click a Checkbutton to see if their code passes the tests defined in the question. If not, students can resubmit, typically for a small penalty. In the typical 'all-or-nothing' mode, all test cases must pass if the submission is to be awarded any marks. The mark for a set of questions in a quiz is then determined primarily by which questions the student is able to solve successfully and then secondarily by how many submissions the student makes on each question. However, it is also possible to configure CodeRunner questions so that the mark is determined by how many of the tests the code successfully passes.

CodeRunner and its predecessors pycode and ccode has been in use at the University of Canterbury for about six years, running over a million student quiz question submissions in Python, C, JavaScript, PHP, Octave and Matlab. Laboratory work, assignment work and mid-semester tests in the introductory first year Python programming course (COSC121), which has around 500 students in the first semester and 300 in the second, are all assessed using CodeRunner questions. The final exams for COSC121 have also been run using Moodle/CodeRunner since November 2014.

The second year C course (ENCE260) of around 200 students makes similar use of CodeRunner using C questions and a third year Civil Engineering course (ENCN305), taught in Matlab, uses CodeRunner for all labs and for the mid-semester programming exam. Other courses using Moodle/CodeRunner include:

   EMTH171 Mathematical Modelling and Computation
   SENG02 Software Engineering I
   COSC261 Formal Languages and Compilers
   COSC367 Computational Intelligence
   ENCE360 Operating Systems
   SENG365 Web Computing Architectures

CodeRunner currently supports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, JavaScript (NodeJS), Octave and Matlab. The architecture allows easy extension to other languages.

CodeRunner can safely be used on an institutional Moodle server, provided that the sandbox software in which code is run ("Jobe") is installed on a separate machine with adequate security and firewalling. However, if CodeRunner-based quizzes are to be used for tests and final exams, a separate Moodle server is recommended, both for load reasons and so that various Moodle communication facilities, like chat and messaging, can be turned off without impacting other classes.

A single 4-core Moodle server can handle an average quiz question submission rate of about 60 quiz questions per minute while maintaining a response time of less than about 3 – 4 seconds, assuming the student code itself runs in a fraction of a second. We have run CodeRunner-based exams with nearly 300 students and experienced only light to moderate load factors on an 8-core Moodle server. The Jobe server, which runs student submissions (see below), is even more lightly loaded during such an exam.

The CodeRunner question type can be installed on any modern Moodle system (version 2.9 or later including version 3.2), on Linux, Windows and Mac. For security reasons submitted jobs are usually run on a separate machine called the "Jobe server" or "Jobe sandbox machine".