Testing of integrated issues
Note: This page is for analysing and developing our weekly testing of code added to the integration server. It is a work-in-progress. Comments or suggestions are welcome! Please use the page comments.
Testing is an important part of the Moodle development process. Wednesday has been allocated for testing, tester's first priority should be to finish testing on that day. Tester should update testing status or add comments in tracker, so that status of testing is known to iTeam.
The testing process
- Every Wednesday by 10:00 am (WST) all tests get allocated to HQ developers by iTeam.
- Developer should check mail or search tracker to see which issues are assigned to him/her for testing.
- Pull latest integration from git://git.moodle.org/integration.git
- Test issue by following Testing instructions
- Select 'Pass test' or 'Fail test' as appropriate, adding a short description of what was tested if not obvious
- If you find you cannot finish testing a particular pull request, click 'Stop testing' and let iTeam know about it.
- Failed tests will wait for assignee to respond. If the patch is provided late and there is constrained to find tester then issue will be re-opened.
- Once the fail fix is integrated, it goes back to *complete* re-testing.
- It's responsibility of tester to test the issue again, provided fix patch is not from tester. If tester provides fix patch then tester will be changed.
- Tester who passes the issue will be set as tester for the issue.
For test sessions, if you encounter a failure, please fail the issue add a comment on the issue itself. If everything's good, add a comment in the session and complete the session. You may also comment on the issue and say that testing passes on your part.
Expectation from developer and peer-reviewer
- Testing instructions are spot-on, clear and easy to perform. Please, follow the Testing instructions guide recommendations.
Expectation from iTeam
- Most (if not all) tests should be allocated by Wednesday Morning 10:00 a.m (WST). Tests that appear after this might not be tested. If tests are allocated after this point, the allocated tester should be contacted individually by private message and allowed to acknowledge.
- The iTeam member assigning tests should put a message in office chat that tests have been allocated.
- The iTeam may need to help/re-assign tests if developers are having problems.
Expectation from tester
- Testing must happen always against up to date integration.git repository (unless testing instructions include some exceptional git operation). More specifically, testing must not happen against development branches for a number of reasons (based on old core stuff, missing interdependencies with other issues or changes performed along the integration process, upgrade problems...).
- If HQ developer is not available for testing on Wednesday, this should be recorded in the calendar. If this has not been communicated in the leave calendar, then the developer or management should let iTeam know ASAP.
- Testers should try to finish testing as early as possible on Wednesdays, so when tests fail, the issue assignee has as much time as possible to resolve it.
- When a test fails, or new (related) regression found then fail test
- If tester is not sure of results or need explanation on testing instructions, then tester can either fail test with comments, or contact the assignee individually to raise the problem.
- Testers should let the iTeam know ASAP if they are facing any problems, need help or may not be able to complete their allocated tests
- For any reason (Big test, not enough time, not started testing yet), if a test is dragged to next day (Thursday) then the tester should leave comment on tracker, updating the status of testing and adding the expected time needed to finish testing.
- When a test is passed, it is recommended to add some extra information that confirms that all works as expected. This could be a browser screenshot, terminal output...
- All UI tests should be tested on currently supported themes (current themes being Boost and Clean for old versions and Boost and Classic for 3.7 and up).
Differences in test process during continuous integration periods
During continuous integration we change our schedule to produce and release fixes to bugs more quickly than the usual weekly cycle. Our goal during this period is to release a new version of master multiple times per week and we do not have preallocated testing days (wednesday). We try to keep the process more flexible during this time in order that developers who have less pressing issues than others can take the load off those concentrating on big fixes. It works best if we work together to help each other out.
It is expected that:
- Testers to proactively pick up issues from the Urgent Issues Dashboard (awaiting testing widget/filter).
- At ~12:00 (UTC-8) each day, any waiting tests will be allocated to testers (first to those who have not proactively taken any)
- Priority is given to testing issues to ensure we can release regularly
Installing a local test site from the integration repository
Moodle uses two Git repositories for its source code. Their names are moodle.git and integration.git and they live at http://git.moodle.org. All submitted patches that were accepted during the integration review go to the integration.git first. Testers use integration.git as the source of the code to test. Patches that survive testing are then promoted to moodle.git and become the part of the official Moodle weekly build.
git clone git://git.moodle.org/integration.git
as the very first command.
Changing theme to another one
Ensure you have following setting in the config (It allows changing theme from url).
Set $CFG->allowthemechangeonurl = true;
For changing to a theme named "yay" add ?theme=yay to the url.
Checking tests assigned to you
- If issues requires an Oracle and MSSQL installations for testing, and you don't have one, then please let iTeam know about this.
- Any update should be added as comment on MDL issue being tested.
- If testers pass or fail an issue by mistake, then please request iTeam to reopen it for testing.
- Testers should not be involved in the bugfixing or review process.
- If an issue cannot be fixed within a sprint and has to be reopened, the fix for sprint version should be removed and an appropriate backlog version set.