Development:Unit tests

Location: Administration > Reports > Unit tests in 1.9 or Site administration > Development > Unit tests in Moodle 2.0 onwards

The purpose of Unit tests is to evaluate the individual parts of a program (functions, and methods of classes) to make sure that each element individually does the right thing. Unit Tests can be one of the first steps in a quality control process for developing or tweaking Moodle code. The next steps will involve other forms of testing to ensure that these different parts work together properly.

The unit testing framework is based on the SimpleTest framework.

Running the unit tests in Moodle

Running the basic tests

1. Log in with an admin account.
2. Access Administration > Reports > Unit tests in 1.9 or Site administration > Development > Unit tests in Moodle 2.0 onwards
3. Click on the Reports link near the bottom of the page.
4. Click the Run tests button and wait.

This finds all the tests in Moodle and runs them. You can run a subset of the tests by entering a path (for example question/type) in the 'Only run tests in' box. Similarly, if a test fails, you get some links in the failure message to make it easy to re-run just those tests.

Writing new tests

As an example, suppose we wanted to write some tests for the string_manager class in mod/quiz/editlib.php.

Where to put the tests

The unit test report finds tests by looking for files called 'test....php' inside folders called 'simpletest'.

So, for our example, we want to create called something like mod/quiz/simpletest/testeditlib.php. The skeleton of this file should look like:

<?php /**

* Unit tests for (some of) mod/quiz/editlib.php.
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
* @package question
*/

if (!defined('MOODLE_INTERNAL')) {

   die('Direct access to this script is forbidden.'); //  It must be included from a Moodle page

}

// Make sure the code being tested is accessible. require_once($CFG->dirroot . '/mod/quiz/editlib.php'); // Include the code to test

/** This class contains the test cases for the functions in editlib.php. */ class quiz_editlib_test extends UnitTestCase {

   function test_something() {
       // Do the test here.
   }

   // ... more test methods.

} ?>

That is, you have a class called something_test, and in that class you have lots of methods called test_something. Normally, you have one test method for each particular thing you want to test, and you should try to name the function to describe what is being tested - without making the name too ridiculously long!

A test function

The a test function typically looks like

function test_move_question_up() {

   // Setup fixture

   // Exercise SUT
   $newlayout = quiz_move_question_up('1,2,0', 2);

   // Validate outcome
   $this->assertEqual($newlayout, '2,1,0');

   // Teardown fixture

}

This is the four phase test pattern. Those comments use a lot of testing jargon. The fixture is the background situation that needs to be set up before the test runs. SUT is short for 'situation under test'. This is where you call the function or method that you want to test. Then you check to see if the function did the right thing. Finally, you have to clean up the fixture you created. With luck there is nothing to do here

In this simple example, there is no setup or teardown to do. We just call the function we are testing with some sample input, and check that the return value is what we expect.

Shared setUp and tearDown methods

If all your test cases relate to the same area of code, then they may all need to same bit of fixture set up. For example, all the tests in lib/simpletest/teststringmanager.php need an instance of the string_manager class to test.

To avoid duplicating code, you can override a method called setUp() that sets up the test data. If present, this method will be called before each test method. You can write a matching tearDown() method if there is any clean-up that needs to be done after each test case has run. For example, in lib/simpletest/teststringmanager.php there are setUp and tearDown methods that do something like: public function setUp() {

   // ...
   $this->stringmanager = new string_manager(...);

}

public function tearDown() {

   $this->stringmanager = null;

} Then, each test can use $this->stringmanager without having to worry about the details of how it is set up.

Further information

The SimpleTest documentation is at: http://www.simpletest.org/.

Changes to your existing code to make it work with unit testing

The whole point of unit testing is to test each piece of functionality separately. You can only do this is by isolating that function and call it individually, perhaps after setting up a few other things.

Therefore, it is good if you can write your code to depend on as few other things as possible.

Include paths

Includes like

require_once('../../config.php'); // Won't work.

won't work. Instead, the more robust option is

require_once(dirname(__FILE__) . '/../../config.php'); // Do this.

Access to global variables

Because your code was included from within a function, you can't access global variables until you have done a global statement.

require_once(dirname(__FILE__) . '/../../config.php'); require_once($CFG->libdir . '/moodlelib.php'); // Won't work.

require_once(dirname(__FILE__) . '/../../config.php');

global $CFG; // You need this. require_once($CFG->libdir . '/moodlelib.php'); // Will work now.

Calls to global functions

Testing a class method that calls global functions can be problematic. At least, it's always complex, because we can't control what goes on in the global functions. We can't override the global functions or mock them in our unit tests. If the global functions themselves are well tested, this may not be a big problem, but most global functions are not well tested.

Bridge Pattern

If your code needs to rely extensively on some public API, you could use the bridge pattern to decouple your code from that API. This way, when you write unit tests, you can override the bridging class or mock it, and control its outputs while you focus exclusively on testing your code.

An basic example follows: Imagine that I do not trust the get_string() global function, but my code needs to use it. Initially my code has strong coupling with get_string():

class myclass {

   public function print_stuff($stuff) {
       echo get_string($stuff);
   }

}

Now let's write a bridging class to solve this coupling issue and use it instead of get_string():

class languageBridge {

   public function get_string($stuff,$module='moodle') {
       echo get_string($stuff, $module);
   }

}

class myclass {

   public $lang_bridge;
   public function __construct() {
       $this->lang_bridge = new languageBridge();
   }
   public function print_stuff($stuff) {
       echo $this->lang_bridge->get_string($stuff);
   }

}

The following is yet another example using a bridging method to decouple from the Moodle core API.

class workshop_api {

   /**
    * This is a method we want to unittest
    */
   public function get_peer_reviewers($context) {
       static $users=null;
       if (is_null($users)) {
           $users = $this->get_users_by_capability($context, 'mod/workshop:peerassess', 
                       'u.id, u.lastname, u.firstname', 'u.lastname,u.firstname', , , , , false, false, true);
       }
       return $users;
   }

   /**  
    * Bridging method to decouple from Moodle core API
    */
   protected function get_users_by_capability() {
       $args = func_get_args();
       return call_user_func_array('get_users_by_capability', $args);
   }

}

Warning: Here are some comments on the examples above expressing that the bridge pattern should be used very carefully.

• "I think that is a case of unit tests leading to worse software design, in that you are not using the standard API for something. But if you really want to unit test, I can't think of a better solution."
• "I think this can be OK if used very selectively. Unfortunately I don't think it's the solution if you want to decouple the very complex and deeply nested Moodle functions from each other"
• "...your code is designed to be part of Moodle, so decoupling from a standard Moodle API is perverse."

Unit testing in 2.0

Template:Moodle 2.0 With the Objectification of the Database libraries in Moodle 2.0, new and better approaches to Unit testing can be used. Here is a sample of a simple test case: (in course/simpletest)

require_once($CFG->dirroot . '/course/lib.php');

global $DB;
Mock::generate(get_class($DB), 'mockDB');

class courselib_test extends UnitTestCase {
    var $realDB;

    function setUp() {
        global $DB;
        $this->realDB = $DB;
        $DB           = new mockDB();
    }

    function tearDown() {
        global $DB;
        $DB = $this->realDB;
    }

    function testMoveSection() {
        global $DB;
        $course = new stdClass();
        $course->id = 1;

        $sections = array();
        for ($i = 1; $i < 11; $i++) {
            $sections[$i]          = new stdClass();
            $sections[$i]->id      = $i;
            $sections[$i]->section = $i - 1;
        }

        $DB->expectOnce('get_records', array('course_sections', array('course' => $course->id)));
        $DB->setReturnValue('get_records', $sections);
        $this->assertFalse(move_section($course, 2, 3));
    }
}

See also UnitTestCaseUsingDatabase in lib/simpletestlib.php.

Testing HTML output

(work in progress - to be documented properly once the API stabilizes)

• ContainsTagWithAttribute($tag, $attribute, $value)
• ContainsTagWithAttributes($tag, $attributes)
• ContainsTagWithContents($tag, $content)
• ContainsEmptyTag($tag)

The syntax is $this->assert(new ContainsTagWithAttribute($tag, $attribute, $value), $html);

Code coverage analysis

Note: This section is a work in progress. Please use the page comments or an appropriate moodle.org forum for any recommendations/suggestions for improvement.

Code coverage is a technique, strongly tied with software testing, that allows to check and improve the quality of the tests by measuring the degree of source code that is being covered by them. With Moodle supporting more and more tests each day (slowly towards a Test Driven Development model) we need to integrate some tool into our development process helping to analyse the quality of ours tests.

Right now (in Moodle 2.0) we are using SimpleTest, one simple and great tool to perform all the tests. Unluckily, it doesn't support code coverage analysis at all. In the other hand, other PHP unit testing products like PHPUnit, more complex and powerful, have built-in support for that technique, but switching to a new product is out from our current Roadmap plans.

So, after some readings and comparisons, Moodle will implement its own extensions to SimpleTest, in order to fulfil the main goal of having statement/line code coverage analysis working under Moodle 2.0 onwards. To achieve this, also Spike PHPCoverage, a basic code-coverage tool, will be used and extended. You can find the details of the implementation of this tool at MDL-19579.

Changes

To enable code coverage in your tests, only a few modifications need to be performed to your current code:

1. Change your test classes by adding two (public static) attributes: $includecoverage and $excludecoverage, both arrays, being used to inform to the code coverage tool (via reflection) about which source code files and dirs must be covered/skipped by the analysis.
2. Use simpletestcoveragelib.php instead of simpletestlib.php in your caller scripts.
3. Use the autogroup_test_coverage class instead of the AutoGroupTest one (see below for details) in your caller scripts.

(note that only the 1st point above is needed for new unit tests being created because both 2 and 3 (changes in caller scripts) are already implemented in Moodle and awaiting your cool unit tests.

That's all! With those 3 basic changes, you will end with a complete code coverage report available for further analysis.

API

When using code coverage within Moodle there are two alternative APIs available, both providing the same code coverage reports at the end, but doing that in a different way. Here they are:

• Internal (hidden) coverage API: This API is completely hidden beyond the Unit testing API and you won't need to know the details about it. Just perform the 1-2-3 changes described above and, after running the tests you'll get the final report available for being used immediately, without needing to perform anything in your code. His major drawback: it only can perform one "code coverage session" (a.k.a. instrumentation), so it's only suitable for testing scripts using only one unit test execution. One example of this type of unit testing is admin/report/unittest/index.php where only one (big) test-group is executed.
• External (explicit) coverage API: This API needs extra coding as long as coverage instantiating, configuration and report generation happens in the main script. It's a bit more complex but, in the other hand, it support multiple instrumentations to be performed, and gives you more control about the code coverage process. One example of this type of unit testing is admin/report/unittest/dbtest.php where multiple (one for each DB being tested) test-group are executed.

So, first of all (point 1 in prev section - usage), we need to define, for each unit test, which files / directories (relative to dirroot) we want to analyse with the tool. Here it's one example, for the dml_test unit test (/lib/dml/simpletest/testdml.php), all we need to add to these lines to the class declaration :

public static $includecoverage = array('lib/dml'); public static $excludecoverage = array('lib/dml/somedir');

By doing so, the code coverage tool will know which are the target files to perform coverage analysis/reporting and will do that for all the files (recursively) in the lib/dml' dir but excluding the lib/dml/somedir directory (recursively too). Note that both attributes are arrays so multiple paths can be specified in any of them. Also note that the dir where the UnitTest is stored is automatically excluded (usually simpletest dirs).

And, as said, that's all you need to fulfil in order to get current/new unit tests being analysed by the code coverage tool by current Moodle scripts. The documentation below is only interesting for developers wanting to create new scripts able to perform unit testing with code coverage (points 2 and 3 in prev section - usage).

Internal coverage API

$test = new autogroup_test_coverage($showsearch, $test_name, $performcoverage, $coveragename, $coveragedir); Create one new autogroup test object with code coverage support, there you can specify if you want to perform coverage (true/false), the name of the report (title) and the directory where the final report will be created (under moodledata/codecoverage). Optionally you can add more files and directories (relative to dirroot) to the list of files to be covered / ignored by using these functions (in case the defined in point 1 aren't enough). $test->add_coverage_include_path($path); $test->add_coverage_exclude_path($path); And then, affter adding a bunch of unit tests to the group, you simply invoke the test execution with code coverage support to end with a nice code coverage report under dataroot/codecoverage/$coveragedir: $test->run($unit_test_reporter); (don't forget that this API supports only one instrumentation to be performed)

And that's all!

External coverage API

$covreporter = new moodle_coverage_reporter($coveragename, $coveragedir); $covrecorder = new moodle_coverage_recorder($covreporter); Create one coverage reporter, by passing its title and output directory as parameters. $test = new autogroup_test_coverage($showsearch, $test_name, $performcoverage); Create one new autogroup test object with code coverage support, you don't need to specify the title and dir here (as already have been defined by the moodle_coverage_reporter object). Optionally you can add more files and directories (relative to dirroot) to the list of files to be covered / ignored by using these functions (in case the defined in point 1 aren't enough). $test->add_coverage_include_path($path); $test->add_coverage_exclude_path($path); Then, after adding a bunch of unit tests to the group, you simply invoke the test execution with code coverage support with: $test->run_with_external_coverage($unit_test_reporter, $covrecorder); (don't forget that this API supports multiple instrumentations to be performed)

And finally, you generate the code coverage report (under dataroot/codecoverage/$coveragedir) using: $covrecorder->generate_report(); Once more, that's all!

Final notes

• Note that there are some more methods available in the moodle_coverage_recorder class. They will allow to control starting/stopping instrumentations by hand and other minor things but they shouldn't really be used. The run() and run_with_external_coverage() methods should be enough in 99% of cases.
• Not being part of the API, but used by it, there is one overagefile.php script under admin/report/unittest responsible for serving the coverage report files from within Moodle. See current scripts in that dir to see how can be used.
• All the test execution / reporting / coverage utilities must be protected with the 'moodle/site:config' permission.

A warning

In 'xUnit Test Patterns' there is a scale of testing difficulty that goes from 1. to 6. Moodle is definitely at number 6. on that scale 'Non-object-oriented legacy software'. It then goes recommend that you don't start to learn about unit testing with that sort of software :-(

Further reading about unit testing

PHP in Action has an excellent chapter explaining unit testing in PHP with simpletest. (Although the rest of that book advocates a style of programming that is very different from the style used in Moodle.)

Pragmatic Unit Testing in Java with JUnit is also a very good introduction, despite being in the wrong programming language. JUnit and Simpletest are very similar.

xUnit Test Patterns is the ultimate unit test book. I think it teaches you everything you could learn about unit testing by reading a book. The only way to learn more would be years of experience. It has really great advice for dealing with the kind of messy problems you get in a big, real project like Moodle.