Student projects/Usability issues/T4
Usability problems found in the Fluid project
At Research of Moodle usability problems mentioned in the Fluid project one can find more information about this issue.
Usability problems found in the tracker
In order to classify the importance of the different usability problems found in the tracker, they are put in a ranking. The solution found of this issues can be found at this wiki page
Moreover, after having heard the needs of several people being with Moodle as teachers and administrators, two main issues should be improved:
Usability problems found when talking with Moodle users
Properly order of selections when using the tab key of the keyboard
This functionality would help both experience and new users. Usability problems have been found mainly in two different situations:
- When you press the tab key it goes from one item to the help button, which is not really usable because the help button is not frequently used.
- When you edit a text and you press a tab in order to go to the text field and write something, the tab goes first to all the writing options (bold, italics, etc.) making it not usable for the user that wants usually to get directly to the text field.
IA of the gradebook
To study the information architecture of the Gradebook, it is necessary to do an open and closed card sorting. To collaborate with this two studies please take the test online (it will take only 5 minutes).
The aspect of the two text is the following:
Open Card Sorting
Closed Card Sorting
Results of the Card Sorting
The partial results are the following. It must be taken into account, that an excel has been created with the groups that each person thinks that is the best (for that it is called open card sorting)
Closed Card Sorting
Open Card Sorting
Open Card Sorting in detail
Usability of the gradebook
The gradebook perhaps is the biggest Moodle usability issue although students are not affected. The gradebook, with so many options, makes difficult for the teachers to find the issue.
For instance, there is a problem when choosing and action or when selecting category items, the aggregation or editing category homework. So some usability testing should be done in the gradebook in order to improve the usability.
Some quick grading tutorials have been written in order to help teachers and some complaints and some grading questions have been posted to the Moodle Forum. It must also be considered that Fluid project analyzes the grading of Moodle 1.8.2 and it is not the same as Moodle version 1.9, so it is necessary to do some usability tests in order to improve grading usability.
Next the test plan of the gradebook usability test will be described:
Objectives of the gradebook usability test
The profile of the users who will participate of the usability test are the following:
| #ID | User | Profile | Objectives |
|---|---|---|---|
| 1 | Administrator and teacher | He is really used to the administration interfaces. He has been using Moodle as an administrator and teacher for several years | Test both functionalities and improvements he thought and give some comments he could have heard from other teachers |
| 2 | Prospective teacher familiar with new technologies | He is computer scientist so he is used to new applications. He will use Moodle as a student but he has never used Moodle in a teacher role. | Usability issues that a new teacher familiar with Moodle user view and new applications can find. |
| 3 | Teacher not familiar with new technologies | She is not used to new applications. She has never used Moodle neither as a student nor a teacher. | Usability issues that a new teacher neither familiar with Moodle nor new applications can find. |
The objective of the developer is to modify the grading design with both quantitative and qualitative information of the usability test.
Definition of the tasks that will be evaluated in the usability test
All tasks will be done to all the participants of the usability test.
T1: Items weighted by their point values
Description:
- In a sample course of three students, create three assignments titled "10 point assignment", "20 point assignment", and "50 point midterm".
- Assign scores to each of the assignments (simple weighted mean of grades)
- Verify that the course grade is as you desire.
- Change various scores and confirm that the gradebook is computing the course grade in the way you want. If not, you might want to make additional changes to your gradebook setup.
T2: Items all given equal weight
Description: In some courses, the number of points(or marks) given to an item is arbitrary and is not a factor in deciding its weight in computing the course grade. In this case, each assignment is given equal weight, so a 10 point essay might be worth just as much as a 20 point essay. With this grading strategy, one would first convert each item to a percentage value and then average the percentages in computing the course total.
T3: Course grade weighted by category
Description: In many courses, the teacher may want to assign different elements of performance (such as daily work and tests) to defined percentages in determining a course grade. Moodle calls this Aggregation method of computing course grades Weighted mean of grades because the course grade is computed by weights that the teacher assigns to categories (or items).
Definition of the variables that will be tested in the usability test
The following variables will be analyzed:
- Verbal protocols in order to extract qualitative information.
- Corporal expressions in order to extract qualitative information.
- User expectations: In order to have a qualitative analysis specially of the search and visualization of the information.
- Reaction time: the time the user spend knowing how to do an activity, to orientate, etc.
- Navigation patterns: This variable includes:
- Knowledge of where is the start page
- Knowledge of the content of the pages
- Knowledge of how to find clues or more information of an item.
- Knowledge of what buttons or resources are not used
- Knowledge of the reason that this buttons or resources are not used
- Clicks and trace
- Knowledge of his/her movement all over the page
- Knowledge of how he / she goes to the next page
- Quantitative errors (success / problems)
- Time spent performing the task
| ID | Parameter | Variable | Quantitative analysis | Qualitative analysis |
|---|---|---|---|---|
| 1 | Time to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 2 | Time to know how to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 3 | Percentage of sucess / fail attempts | Percentage | Average and standard deviation of all users | See if there are tasks with a high percentage of failure. |
| 4 | Questions after the interview | Grading scale from 1 to 5 | Average and standard deviation of all users | Get users' coments |
| 5 | Pressed links | Number of clicks to perform a task | Average and standard deviation of all users | Analyze tasks with a great number of clicks |
| 6 | Information about help | # of times that help is used | Average and standard deviation of times that help is needed by all users. | See what tasks / users need more help |
Preparation of the scenarios, tasks, questions and materials that will be presented to the user in the usability test
- First, there will be some questions at the beginning of the usability test:
- Age
- Profession and studies
- Experience
- Is familiar with Moodle?
- In which context do you use it?
- Secondly, an application called recordmydesktop will be run in order to make a video of the desktop (to record the usability test).
- Finally, after making the test, there will be some questions:
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like gradebook interface?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think it is appropriate the number of clicks to access to the information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like the organization and the structure of the information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you understand the icons?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you like interface design?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think there is enough quantity of information?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you think you have enough clues to navigate?
- From 1 to 5, (1 more satisfied and 5 less satisfied) ¿do you want to comment anything?
Results of the usability test
First usability test
- Beginning of the usability test:
- Age: 24
- Profession and studies: Phd Student in Computer Science, Msc. in Computer Science
- Experience: 1 year
- Is familiar with Moodle? Yes
- In which context do you use it? At university
- Videos of the usability test:
- Final questions of the usability test:
- 2
- 2
- 2
- 4
- 3
- 2
- 1
- 2
- Comments:
- To put the maximum rate of the courses is not really usable with the scroll bar. It would be better to put the number (use a text box).
- Categories and items is not a clear name, so the user finds it difficult to know what information he/she could find there.
- It would be far more intuitive to click on the form and put there the student marks.
- It would also be fare more intuitive to click on the last colum (where the final score is automatically calculated)and putting there the formula.
- It makes the user angry to introduce the equal sign every time a calculation should be done. It would be nice that the = is used automatically.
- The way to introduce the formulas should be easier (to the the average, a weithed average etc.).
- It would be great if there would be any connection between the grading menu and the adding assignment page.
Second usability test
- Beginning of the usability test:
- Age: 54
- Profession and studies:
- Experience:
- Is familiar with Moodle?
- In which context do you use it?
- Final questions of the usability test:
- Comments:
Third usability test
- Beginning of the usability test:
- Age:
- Profession and studies:
- Experience:
- Is familiar with Moodle?
- In which context do you use it?
- Final questions of the usability test:
- Comments:
Analysis of the usability test
- Verbal protocols: it takes too long to locate the different information in the page, this page is not usable, I don't find anything.
- Corporal expressions: the user is disgusted because at first he finds difficult how to locate the information.
- User expectations: the grading functionality has far less usability than other Moodle pages / functionalities.
- Reaction time: the user spent 30 min familiarizing with the grading pages.
- Navigation patterns: This variable includes:
- Knowledge of where is the start page: easy to find
- Knowledge of the content of the pages: difficult, mainly "categories and items" and other edit labels.
- Knowledge of how to find clues or more information of an item: difficult to find withouth clicking on the page. Moreover, help boxes had too much information (the ones about how to calculate the final score) and it would be better to have a summery if a quick help is wanted.
- Knowledge of what buttons or resources are not used: difficult to find the "turn editing on"
- Knowledge of the reason that this buttons or resources are not used: the buttons to activate a mode should be in another colour.
- Clicks and trace: there are many clicks because the user gets lost with facility.
- Knowledge of his/her movement all over the page: once the user knows where is the information, sometimes has problems when repeating the same steps.
- Knowledge of how he / she goes to the next page: the user has to navigate through several pages: it gets lost with facility
- Quantitative errors (success / problems)
- Time spent performing the task
| ID | Parameter | Variable | Quantitative analysis | Qualitative analysis |
|---|---|---|---|---|
| 1 | Time to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 2 | Time to know how to do a task | Seconds | Average and standard deviation of all users | See if there are tasks where the user spends a considerable amount of time |
| 3 | Percentage of sucess / fail attempts | Percentage | Average and standard deviation of all users | See if there are tasks with a high percentage of failure. |
| 4 | Questions after the interview | Grading scale from 1 to 5 | Average and standard deviation of all users | Get users' coments |
| 5 | Pressed links | Number of clicks to perform a task | Average and standard deviation of all users | Analyze tasks with a great number of clicks |
| 6 | Information about help | # of times that help is used | Average and standard deviation of times that help is needed by all users. | See what tasks / users need more help |
