Student projects/Usability issues/T4
Usability problems found in the Fluid project
At Research of Moodle usability problems mentioned in the Fluid project one can find more information about this issue.
Usability problems found in the tracker
In order to classify the importance of the different usability problems found in the tracker, they are put in a ranking. The solution found of this issues can be found at this wiki page
Moreover, after having heard the needs of several people being with Moodle as teachers and administrators, two main issues should be improved:
Usability problems found when talking with Moodle users
Proper order of selections when using the tab key of the keyboard
This functionality would help both experience and new users. When you press the tab key it goes from one item to the help button, which is not really usable because the help button is not frequently used.
Two possible solutions would be:
- Set a tabulation preference order so that the components that are usually skipped are at the end
- Control with Javascript specific cases
IA of the gradebook
To study the information architecture of the Gradebook, it is necessary to do an open and closed card sorting. To do this studies several software has been used:
- WebSort.net: it has been used to do the card sorting online
- CardSword: it has been used to do the analysis of the card sorting (tree analysis and dendogram). It must be taken into account that we would have used this software if we would have been able to do the card sorting in a non virtual way.
During the realization of the cardsorting, some restrictions have been taken into account:
Restrictions
- The quantity of contents of each category will have between 2 and 8 targets
- Taxonomy: at most it will be structures of eight categories
- The granularity of the categories will be between 1 and 3
Test
To collaborate with this two studies please take the test online (it will take only 5 minutes).
The aspect of the two text is the following:
Open Card Sorting
Closed Card Sorting
Results of the Card Sorting
The partial results are the following. It must be taken into account, that an excel has been created with the groups that each person thinks that is the best (for that it is called open card sorting)
Closed Card Sorting
Open Card Sorting
Open Card Sorting in detail
Analysis of the results
The open card sorting is the test which we could obtain more information. Next the tree diagrams and the dendograms of the open card sorting can be appreciated:
Tree diagram
Dendogram 0% (all the information is displayed)
Dendogram 25% (the information that the 25% of the users agree is displayed)
Dendogram 40% (the information that the 40% of the users agree is displayed)
It can be seen in a more bold line the concepts that are most frequently connected by the users.
Information Architecture
The previous analysis of the results helps to decide the final categories which are:
Edit Gradebook: In this category there will be: Compute final score List of course assignments Edit course settings Grade letters: grade letters will replace letters because most people does not understand the meaning of letters. Grade scale: the same as grade letters. View Grades: 4 reports are in this group: Grader report Outcomes report Overview report User report Import/Export: As there are only 6 elements and it is not a frequently used functionality, the import / export group is in the same category.
It must be taken into consideration that the order is also important (the import/export is the last one because is not as frequently used)
Usability of the gradebook
The gradebook perhaps is the biggest Moodle usability issue although students are not affected. The gradebook, with so many options, makes difficult for the teachers to find the issue.
For instance, there is a problem when choosing and action or when selecting category items, the aggregation or editing category homework. So some usability testing should be done in the gradebook in order to improve the usability.
Some quick grading tutorials (quick grading tutorials in Spanish)have been written in order to help teachers and some complaints and some grading questions have been posted to the Moodle Forum. It must also be considered that Fluid project analyzes the grading of Moodle 1.8.2 and it is not the same as Moodle version 1.9, so it is necessary to do some usability tests in order to improve grading usability.
Next the test plan of the gradebook usability test will be described:
Objectives of the gradebook usability test
The profile of the users who will participate of the usability test are the following:
| #ID | User | Profile | Objectives |
|---|---|---|---|
| 1 | Administrator and teacher | He is really used to the administration interfaces. He has been using Moodle as an administrator and teacher for several years | Test both functionalities and improvements he thought and give some comments he could have heard from other teachers |
| 2 | Prospective teacher familiar with new technologies | He is computer scientist so he is used to new applications. He will use Moodle as a student but he has never used Moodle in a teacher role. | Usability issues that a new teacher familiar with Moodle user view and new applications can find. |
| 3 | Teacher not familiar with new technologies | She is not used to new applications. She has never used Moodle neither as a student nor a teacher. | Usability issues that a new teacher neither familiar with Moodle nor new applications can find. |
The objective of the developer is to modify the grading design with both quantitative and qualitative information of the usability test.
Definition of the tasks that will be evaluated in the usability test
All tasks will be done to all the participants of the usability test.
T1: Items weighted by their point values
Description:
- In a sample course of three students, create three assignments titled "10 point assignment", "20 point assignment", and "50 point midterm".
- Assign scores to each of the assignments (simple weighted mean of grades)
- Verify that the course grade is as you desire.
- Change various scores and confirm that the gradebook is computing the course grade in the way you want. If not, you might want to make additional changes to your gradebook setup.
T2: Items all given equal weight
Description: In some courses, the number of points(or marks) given to an item is arbitrary and is not a factor in deciding its weight in computing the course grade. In this case, each assignment is given equal weight, so a 10 point essay might be worth just as much as a 20 point essay. With this grading strategy, one would first convert each item to a percentage value and then average the percentages in computing the course total.
T3: Course grade weighted by category
Description: In many courses, the teacher may want to assign different elements of performance (such as daily work and tests) to defined percentages in determining a course grade. Moodle calls this Aggregation method of computing course grades Weighted mean of grades because the course grade is computed by weights that the teacher assigns to categories (or items).
Definition of the variables that will be tested in the usability test
The following variables will be analyzed:
- Verbal protocols in order to extract qualitative information.
- Corporal expressions in order to extract qualitative information.
- User expectations: In order to have a qualitative analysis specially of the search and visualization of the information.
- Reaction time: the time the user spend knowing how to do an activity, to orientate, etc.
- Navigation patterns: This variable includes:
- Knowledge of where is the start page
- Knowledge of the content of the pages
- Knowledge of how to find clues or more information of an item.
- Knowledge of what buttons or resources are not used
- Knowledge of the reason that this buttons or resources are not used
- Clicks and trace
- Knowledge of his/her movement all over the page
- Knowledge of how he / she goes to the next page
- Quantitative errors (success / problems)
- Time spent performing the task
| ID | Parameter | Variable | Quantitative analysis | Qualitative analysis |
|---|---|---|---|---|
| 1 | Time to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 2 | Time to know how to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 3 | Percentage of success / fail attempts | Percentage | Average and standard deviation of all users | See if there are tasks with a high percentage of failure. |
| 4 | Questions after the interview | Grading scale from 1 to 5 | Average and standard deviation of all users | Get users' comments |
| 5 | Pressed links | Number of clicks to perform a task | Average and standard deviation of all users | Analyze tasks with a great number of clicks |
| 6 | Information about help | # of times that help is used | Average and standard deviation of times that help is needed by all users. | See what tasks / users need more help |
Preparation of the scenarios, tasks, questions and materials that will be presented to the user in the usability test
- First, there will be some questions at the beginning of the usability test:
- Age
- Profession and studies
- Experience
- Is familiar with Moodle?
- In which context do you use it?
- Secondly, an application called recordmydesktop will be run in order to make a video of the desktop (to record the usability test).
- Finally, after making the test, there will be some questions:
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like gradebook interface?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think it is appropriate the number of clicks to access to the information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like the organization and the structure of the information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you understand the icons?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like interface design?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think there is enough quantity of information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think you have enough clues to navigate?
- From 1 to 5, (1 more satisfied and 5 less satisfied) overall degree of satisfaction
Results of the usability test
First usability test
- Beginning of the usability test:
- Age: 24
- Profession and studies: PhD Student in Computer Science, Msc. in Computer Science
- Experience: 1 year
- Is familiar with Moodle? Yes
- In which context do you use it? At university
- Videos of the usability test:
- Final questions of the usability test:
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like gradebook interface? 2
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think it is appropriate the number of clicks to access to the information? 2
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like the organization and the structure of the information? 2
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you understand the icons? 4
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like interface design? 3
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think there is enough quantity of information? 2
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think you have enough clues to navigate? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) overall degree of satisfaction 2
- Comments:
- Selecting the maximum grade of the courses with a scroll bar is not really usable. It would be better to allow the user to type the number.
- Categories and items is not a clear name, so the user finds it difficult to know what information he/she could find there.
- It would be far more intuitive to click on the form and put there the student marks.
- It would also be fare more intuitive to click on the last column (where the final score is automatically calculated)and putting there the formula.
- It makes the user angry to introduce the equal sign every time a calculation should be done. It would be nice that the = is used automatically.
- The way to introduce the formulas should be easier (to the the average, a weighted average etc.).
- It would be great if there would be any connection between the grading menu and the adding assignment page.
Second usability test
- Beginning of the usability test:
- Age: 54
- Profession and studies: Catalan philology
- Experience: 31 years teaching
- Is familiar with Moodle? No
- In which context do you use it? She does not use it.
- Final questions of the usability test:
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like gradebook interface? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think it is appropriate the number of clicks to access to the information? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like the organization and the structure of the information? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you understand the icons? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think there is enough quantity of information? 5
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think you have enough clues to navigate? 1
- From 1 to 5, (1 more satisfied and 5 less satisfied) overall degree of satisfaction 1
It was too difficult for her and she was lost nearly all the time. She did not know how to do even basic grading tasks. Too much complexity for her and it was too difficult to find the information
- Comments:
- The first attempt in order to introduce the marks was to click in each quadrant. This action was also realize to compute the mark average.
- She thought that she should click on the activity to compute the weighted average.
- To add the mark she went to the profile of each user and she add in the personal comments. Next she found that it was information of the user and that the marks hadn't been added.
- She found difficult to return to the main menu sometimes. Many times used the functionality of going backwards with the browser.
- When she found how to put the marks, she put the mark as a comment and when she displayed she did not see the changes. Somme message could be displayed in order to see the comments or to warn the user the that comments will not be displayed.
- She used the go to functionality but she founds nothing of weighted average
- She uses the help of the aggregation functionality
- She doesn't understand the meaning of "category". She does not see the difference with "course"
- She goes to the assignment and wants to put the mark. The assignments should be editable.
- She does not know how to calculate the average. When I explained to her she says that it is very difficult. So it is not only difficult to access the page where the average is computed but also to specify the way the average is computed.
- She used the help menu to calculate the average but she did not learn how to calculate the average (this menu was not helpful for her, perhaps it was too much information and she got lost)
Third usability test
- Beginning of the usability test:
- Age: 41
- Profession and studies: Phd Computer Science
- Experience: Researcher and teacher
- Is familiar with Moodle? Yes
- In which context do you use it? As a teacher and as a coordinator
- Final questions of the usability test:
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like gradebook interface? 3
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think it is appropriate the number of clicks to access to the information? 3
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like the organization and the structure of the information? 2
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you understand the icons? 5
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like interface design? 4
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think there is enough quantity of information? 5
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think you have enough clues to navigate? 4
- From 1 to 5, (1 more satisfied and 5 less satisfied) overall degree of satisfaction 3
- Comments:
- I would maximize the basic actions of the grading or to access to them only with a click (not a very high granularity). I would try to allow to calculate the final mark in the first page, on perhaps access to the final calculation in a click.
- Teachers often confuse the meaning of categories, and find it difficult to get used with the calculation of the final mark. A more interactive way to compute it (perhaps in the same way as calc tools (such as calc of open office or excel do)
- Teachers that are not familiar with Moodle ofter don't find the turn on editing button. Perhaps it would be a good idea to put a help button next to the turn editing on button so the functionality of the turn editing on button will be explained and perhaps they would notice the button easily. Another possibility would be to activate the turn editing mode in another way (redundancy is used in some applications such as text editors and sometimes can be useful for starters)
- Perhaps it would be useful to add a button to compute the final score of the course in the first grade page because this and the adding of the marks is the most frequently feature used in the grading system.
- It would be useful too to be able to select other types of grading system such as the Spanish or European one (MH, Excellent, Good, Passable and Fail) automatically (without having to translate the American system to the European system) although this functionality is not a critical one.
Analysis of the usability test
- Verbal protocols: it takes too long to locate the different information in the page, this page is not usable, I don't find anything.
- Corporal expressions: the user is disgusted because at first he finds difficult how to locate the information.
- User expectations: the grading functionality has far less usability than other Moodle pages / functionalities.
- Reaction time: the user spent 30 min minimum familiarizing with the grading pages (a user that is familiar with Moodle).
- Navigation patterns: This variable includes:
- Knowledge of where is the start page: easy to find
- Knowledge of the content of the pages: difficult, mainly "categories and items" and other edit labels.
- Knowledge of how to find clues or more information of an item: difficult to find without clicking on the page. Moreover, help boxes had too much information (the ones about how to calculate the final score) and it would be better to have a summery if a quick help is wanted.
- Knowledge of what buttons or resources are not used: difficult to find the "turn editing on"
- Knowledge of the reason that this buttons or resources are not used: the buttons to activate a mode should be in another colour.
- Clicks and trace: there are many clicks because the user gets lost with facility.
- Knowledge of his/her movement all over the page: once the user knows where is the information, sometimes has problems when repeating the same steps.
- Knowledge of how he / she goes to the next page: the user has to navigate through several pages: it gets lost with facility
- Quantitative errors (success / problems)
- Time spent performing the task
| ID | Parameter | Variable | Quantitative analysis |
|---|---|---|---|
| 1 | Time to do a task | Seconds | Average and standard deviation of all users |
| 2 | Time to know how to do a task | Task 1 took 30 min, task 2 10 min, task 3 10 min | Average and standard deviation of all users: desviation users 20 min (although a user could not finish the task) |
| 3 | Percentage of success / fail attempts | Percentage: user 1 and 3 100%, user 2 0% (with help 33%) | Average and standard deviation of all users 0,3 |
| 4 | Questions after the interview | Grading scale from 1 to 5: 2 was the more common value | Average and standard deviation of all users: average 2 and standard deviation 1. |
| 5 | Pressed links | Number of clicks to perform a task: really high, (like 30), they needed first to get used where things where, and they did not find some things. | Average and standard deviation of all users: more or less the same clicks (user 2 perhaps 25% more) |
| 6 | Information about help | # of times that help is used: 5 times per user | Average and standard deviation of times that help is needed by all users: user used help the same amount of times |
Conclusions
After having analyzed the results of the usability test, the following actions should be done in order to improve the usability of grading pages:
* Task #1: Change the information architecture (this is specified in the study of information architecture) * Task #2: Selecting the maximum grade of the courses with a scroll bar is not really usable. It would be better to allow the user to type the number. * Task #3: It would also be far more intuitive to click on the last column (where the final score is automatically calculated)and putting there the formula. * Task #4: It makes the user angry to introduce the equal sign every time a calculation should be done. It would be nice that the = is used automatically. * Task #5: It would be great if there would be any connection between the grading menu and the adding assignment page. When you are in the "assignments" page that I have attached, you have no way to return to the gradebook without going back. You loose the Un site/ ► CF101/ ► Grades/ ► Report/ ► Grader report/ ► Grader report. It should be some link to return to the gradebook or connect this page with the gradebook.* Task #6: There is too much information in the help menu of calculating the final grade. At least a summary would be helpful for a quick help. * Task #7: It should be easier to return to the initial grade page from some pages (this functionality should be checked). * Task #8: Calculate the final mark in the way calcul tools do (such as calc of open office or excel) Some letters and numbers should be displayed when the "turn editing on" is activated such as it is shown in image "Grades". This would make easier to calculate the final score in this page (for example (A1+B1+C1)/3 would be the average score for student 1).
* Task #9: Add a help button next to the turn editing on button so the functionality of the turn editing on button will be explained and perhaps they would notice the button easily. Another possibility would be to activate the turn editing mode in another way (redundancy is used in some applications such as text editors and sometimes can be useful for starters). * Task #10: It would be useful too to be able to select other types of grading system such as the Spanish or European one (MH, Excellent, Good, Passable and Fail) automatically (without having to translate the American system to the European system) although this functionality is not a critical one. * Task #11: A demo of how to use the gradebook would be very useful.
