Cognitive operator: Ordering

Back to index

Operator

Complexity level

Ordering operator is of complexity 1, as there is only one arrangement action on a linear problem dimension.

Operations

Operation 1 : Define order of the input set from "LESS" to a "MOST" value of some criteria defined in teacher's requirements

Purpose

A typical exercise that precedes complex decision making is ordering things in a rough, non quantitative "more than" or "less than" feeling applied to inputs. Ordering can be a simple exercice when inputs and requirements match onto a measurable attribute, f.e. : it is simple to order historical facts based on dates. Ordering more "conceptual" predicates that are less related to measurable or valuable (thus comparable) attribute, or that have multiple valuable attributes not clearly related to teacher's requirement can become a bit hard.

Algorithms

The main algorithm in this operator is defining a comparison between two ordering from distinct users, in order to get a quantifiable metric of the divergence.

One of the start point of a usable metric is the algorithm that counts the number of permuations that is needed to start from one ordered list and to reach the other as end point. F.e. say a first student answers A,B,C,D,E, and the second answers B,A,C,D,E. One unique permutation of A,B is necessary to get list 2 from list 1. Say a third student answers F,B,C,D,E, the number of permutation (sum of absolute position shifts of each member) is 12. So here we can say student 3 is further than student 2 from student 1.

Second step for refining metric will consider position in the list 1. Inverting top members of the list, or last members has not the same meaning on decision shift : B,A,C,D,E needs to be closer to A,B,C,D,E than A,B,C,E,D. This is not rendered by our first calculation. We can refine the calculation by multiplying each local shift by the relative weight of the item in the reference user :

Say user 1 is the reference : A,B,C,D,E will be default assigned to weights : 5,4,3,2,1

Thus shift metric of user 2 is now : 1*5 + 1*4 + 0*3 + 0*2 + 0*1 = 9

Shift metric of user 3 is now : 0*5 + 0*4 + 0*3 + 1*2 + 1*1 = 3

In which we can now say that use 3 is "nearer choice" to user 1 than user 2.

More theoretical development on metric model (to be done)

Ref http://www-poleia.lip6.fr/~marcin/papers/Lesot_EGC_09.pdf

Applications

Direct Workflow

In a direct scenario, the students must order a set of "things" against a requirement the teacher has given.

Non valuated version

• Teacher provides a set of choosen inputs
• Teacher setup the exercice and writes requirement ("Order by xxxxx using yyyy")
• Student processes order
• Student checks his own order against the environment (peer trends in semi-blind mode, or direct peer answers in full view mode)
• Student report about his choice and argue in report
• Teacher gives final feedback to peers or globally

Evaluated version

• Teacher provides a set of choosen inputs
• Teacher setup the exercice and writes requirement ("Order by xxxxx using yyyy")
• Teacher makes the "teacher version" of the ordering, standing to the the "good answer'
• Student processes order
• Student checks his order against teacher's order and get shift evaluation
• Teacher feedbacks if required

Reverse Workflow

In a reverse scenario, the teacher needs order a set of proposals given by the students and give feedback about his choice.

• Students collectively input some proposals
• Teacher setup the Ordering operator
• Teacher makes ordering
• Students observe teacher's anwers
• Students can accessorily order themselves as counter submission
• Student report and argue on teacher's choice
• Teacher gives final feedback

Meta Workflow

Help to discover the concept of "comparable" and what makes things easier to compare when a valuation attribute can be found.

Implementation

Setup

absolute order: When absolute order, there is no way to accept some inputs may be equal to other.

Can replay: The participant will be able to update his answers event after trends or feedback has been given

Only for teachers :

Blindness: The degree of privacy of this operator. (defaults to the activity privacy setting).

Categorize GUI

Master part of the GUI is dedicated to let the participant order the list. First model in prototype provides a simple flat table with 'up' and 'down' controls. This might be not very operable for big lists of more than 10 inputs.

TODO : Find a drag&drop method for easier reordering / Find a nice way to handle big lists (30, 50 items)

• Blind mode: The user just orders the items. No trends.

• Semi blind mode: The user orders the items, and is given a global 'average' trends about match or no match position of each item (number of choices of this item at the same location / percentage of choices over all given answers of this item at this location). Trends are displayed once one answer is given.

• Full View mode: Same as above, but the trend is displayed BEFORE first answer is given.

An alternate method

On big lists, or lists of "things" that are not trivially comparable, an alternate method for ordering consists in reducing the global view to a narrower choice (pair comparison) and process locally all comparisons (its usually easier to compare one to one than having to place one item at a definite rank into a list, which cognitively implies a much more complex operation involving global decision).

The results of the pair comparison are aggregated to produce an output ranking order that will be proposed as the participant's answers.

Display GUI

Display GUI renders the participant ordering with some comparison possibilities, depending on privacy and other participant's answers.

• Blind mode:
  • If there is a teacher answer, the participant's answers are displayed against the teacher answer, with indication of shift amount + a global shift metric. TODO : this shift metric might be reworked and scale to generate an automated evaluative grade.
  • If there is no teacher answer, the ordering of the participant is shown agains an average compilation of all other answers.
• Semi-blind mode: Same as above, but in addition participant can see the table of all other answers with shift indicators, but anonimized.
• Full view mode: The user can pair-compare his answers with any other nominative user. He will also have access to the global metric and global shift table as a synthesis board.